--- title: "llms.txt vs robots.txt: what each file does (2026)" description: "robots.txt controls which URLs crawlers may fetch; llms.txt lists your key pages for AI tools. A side-by-side table, examples and when to use each." url: https://seenlens.com/guides/llms-txt-vs-robots-txt last_updated: 2026-11-06 --- # llms.txt vs robots.txt robots.txt tells crawlers which URLs they may fetch, and llms.txt offers AI tools a short Markdown map of your best pages. robots.txt is an IETF standard that every major search and AI crawler reads. llms.txt is a 2024 proposal that Google Search does not use. Every site needs robots.txt; llms.txt is optional. By SeenLens. Last updated: 2026-11-06 ## What is the difference between llms.txt and robots.txt? robots.txt is a set of access rules. llms.txt is a reading list. One file controls what a crawler may fetch, and the other suggests what an AI tool should read first. Neither file replaces the other. | | robots.txt | llms.txt | | --- | --- | --- | | Purpose | Allow or disallow crawling of URL paths, per crawler | Summarize the site and link to its key pages for language models | | Who reads it | Googlebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and most other crawlers | Some coding agents, SEO and GEO tools, Lighthouse; no answer engine documents that it uses it | | Format | Plain-text records: `User-agent`, `Allow`, `Disallow`, plus extras such as `Sitemap` | Markdown: an H1, an optional `>` summary, H2 sections with link lists | | Standard status | IETF standard, RFC 9309 (September 2022); in use since 1994 | Community proposal on llmstxt.org (September 2024) | | Where it lives | `/robots.txt` at the root of each host, UTF-8, `text/plain` | `/llms.txt` at the site root | | Can it block access? | Yes, for crawlers that follow it; it is not access control | No; it has no allow or disallow rules | | Effect on Google Search | Controls what Googlebot crawls | None; Google says it neither helps nor harms | ## What robots.txt does Martijn Koster defined the robots exclusion protocol in 1994. The IETF published it as RFC 9309 in September 2022. A crawler that follows the standard reads `/robots.txt` before it fetches other URLs on the host, finds the group for its user agent, and applies the `Allow` and `Disallow` rules in that group. Crawlers must parse at least the first 500 KiB of the file. ```text User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: GPTBot Disallow: / User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml ``` AI operators each publish the robots.txt token that controls their crawler. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic and Perplexity document their search crawlers the same way. The [robots.txt guide for AI crawlers](https://seenlens.com/guides/robots-txt-for-ai-crawlers) lists every token. robots.txt has limits. RFC 9309 says the rules are not a form of access authorization. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, because a person asked for the page. Use a login to protect private content. robots.txt can also carry a usage preference. A `Content-Signal` line states whether content may be used for search, AI answers and training; see [Content Signals in robots.txt](https://seenlens.com/guides/content-signals-robots-txt). ## What llms.txt does llms.txt gives a language model a short, clean starting point. The file names the site in an H1, summarizes it in one blockquote line, and lists the pages that matter in H2 sections. An agent that reads it can pick the right page without parsing navigation menus. The evidence that answer engines read llms.txt is thin. In an Ahrefs log study of 137,210 domains, 97% of llms.txt files got zero requests in May 2026, and AI retrieval bots made 1.1% of the requests that did happen. The [llms.txt evidence guide](https://seenlens.com/guides/does-llms-txt-work) covers the numbers. ## Can llms.txt block AI crawlers? No. llms.txt has no rule syntax for access. Leaving a page out of llms.txt does not stop any crawler from fetching that page. To limit AI crawlers, use one of these: - **robots.txt groups** per user agent, for crawlers that follow RFC 9309. - **Meta robots tags or the X-Robots-Tag header** with `noindex` or `nosnippet`, to keep a page out of search results and AI answers that follow those directives. - **Firewall rules**, for crawlers that ignore robots.txt. Be careful: firewalls often block the AI search crawlers you want; see [AI crawlers blocked by a firewall](https://seenlens.com/fixes/ai-crawlers-blocked-by-waf). - **Authentication**, for anything private. ## When to use each file | Task | Use | | --- | --- | | Keep a training crawler such as GPTBot off the site | robots.txt | | Make sure ChatGPT search, Claude and Perplexity can fetch your pages | robots.txt (allow their search crawlers) and firewall settings | | Tell search engines about every URL | sitemap.xml | | Give coding agents a short list of your docs pages | llms.txt | | State whether content may train AI models | robots.txt with `Content-Signal` | | Pass the Lighthouse llms.txt audit | llms.txt | | Rank in Google or appear in AI Overviews | Neither file alone; indexable pages and good content | If you publish both files, they should agree. Do not list pages in llms.txt that robots.txt disallows for the crawlers you care about. ## llms.txt vs sitemap.xml sitemap.xml is an XML list of URLs for search engines, defined at sitemaps.org. One sitemap file can hold up to 50,000 URLs and 50 MB uncompressed, and each entry needs a `loc` tag. A sitemap aims to be complete. llms.txt aims to be short: a curated set of the pages a model should read first, with a line of context for each. You can link the sitemap from robots.txt with a `Sitemap:` line. llms.txt has no such hook. ## Common mistakes - Writing `Disallow` rules inside llms.txt. No crawler reads access rules from that file. - Blocking OAI-SearchBot or PerplexityBot while meaning to block training. Block the training crawlers (GPTBot, ClaudeBot, CCBot) in their own groups instead. - A catch-all page that returns 200 for every path. Tools then think `/llms.txt` exists. Return 404 for unknown URLs. - A robots.txt that returns a 5xx error. Compliant crawlers then treat the whole site as off limits; see [how to fix a missing or broken robots.txt](https://seenlens.com/fixes/robots-txt-missing). - Expecting llms.txt to change Google rankings. Google says it does not use the file. ## How to check both files ```text curl -sI https://example.com/robots.txt curl -sI https://example.com/llms.txt curl -s https://example.com/llms.txt ``` Both requests should return 200. robots.txt should be `text/plain`. llms.txt should start with an H1 line. The [AI crawler checker](https://seenlens.com/tools/ai-crawler-checker) reads your robots.txt and shows which rule applies to each AI crawler. The llms.txt checker above validates the format and tests the links in the file. ## FAQ ### Is llms.txt the new robots.txt? No. robots.txt sets access rules that crawlers follow. llms.txt is a reading list with no rules, and Google Search does not use it. Keep robots.txt in any case. ### Do I need llms.txt if I have robots.txt? Not for search or AI answers. llms.txt is optional. Publish it if you run developer docs or your logs show agents requesting it. ### Should I reference llms.txt in robots.txt? The llms.txt proposal defines no robots.txt line for it. A `Sitemap:` line exists for sitemaps only. ### Does blocking GPTBot in robots.txt remove me from ChatGPT? No. GPTBot is OpenAI's training crawler. ChatGPT search depends on OAI-SearchBot, which has its own robots.txt group. ### Where do robots.txt and llms.txt go? Both go at the root of the site: `/robots.txt` and `/llms.txt`. robots.txt applies per host, so each subdomain needs its own file. ## Sources - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) (IETF standard, September 2022; file location, records, not access authorization, 500 KiB parse limit; checked 2026-11-06) - [llmstxt.org: The /llms.txt file](https://llmstxt.org/) (proposal by Jeremy Howard, 2024-09-03; file format; checked 2026-11-06) - [Google Search Central: Optimizing your website for generative AI features](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) (Google Search does not use llms.txt or similar files; structured data not required; no ideal page length; checked 2026-11-06) - [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) (OAI-SearchBot, GPTBot, ChatGPT-User; no llms.txt input listed; checked 2026-11-06) - [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/guides/bots) (PerplexityBot and Perplexity-User; robots.txt behavior; checked 2026-11-06) - [Anthropic: Does Anthropic crawl data from the web](https://support.claude.com/en/articles/8896518) (ClaudeBot, Claude-User, Claude-SearchBot; checked 2026-11-06) - [Ahrefs: We analyzed 137K sites, 97% of llms.txt files never get read](https://ahrefs.com/blog/llmstxt-study/) (137,210 domains, May 2026 logs, published 2026-06-15; checked 2026-11-06) - [sitemaps.org: Sitemaps XML format](https://www.sitemaps.org/protocol.html) (50,000 URLs and 50 MB per file; checked 2026-11-06)