Skip to content
SeenLens

llms.txt vs robots.txt

robots.txt tells crawlers which URLs they may fetch, and llms.txt offers AI tools a short Markdown map of your best pages. robots.txt is an IETF standard that every major search and AI crawler reads. llms.txt is a 2024 proposal that Google Search does not use. Every site needs robots.txt; llms.txt is optional.

By SeenLens · Last updated

Free, no sign-up. We fetch /llms.txt, /llms-full.txt and up to 10 linked URLs. We do not store the files.

What is the difference between llms.txt and robots.txt?

robots.txt is a set of access rules. llms.txt is a reading list. One file controls what a crawler may fetch, and the other suggests what an AI tool should read first. Neither file replaces the other.

llms.txt and robots.txt compared (checked 2026-11-06)
robots.txtllms.txt
PurposeAllow or disallow crawling of URL paths, per crawlerSummarize the site and link to its key pages for language models
Who reads itGooglebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and most other crawlersSome coding agents, SEO and GEO tools, Lighthouse; no answer engine documents that it uses it
FormatPlain-text records: User-agent, Allow, Disallow, plus extras such as SitemapMarkdown: an H1, an optional > summary, H2 sections with link lists
Standard statusIETF standard, RFC 9309 (September 2022); in use since 1994Community proposal on llmstxt.org (September 2024)
Where it lives/robots.txt at the root of each host, UTF-8, text/plain/llms.txt at the site root
Can it block access?Yes, for crawlers that follow it; it is not access controlNo; it has no allow or disallow rules
Effect on Google SearchControls what Googlebot crawlsNone; Google says it neither helps nor harms

What robots.txt does

Martijn Koster defined the robots exclusion protocol in 1994. The IETF published it as RFC 9309 in September 2022. A crawler that follows the standard reads /robots.txt before it fetches other URLs on the host, finds the group for its user agent, and applies the Allow and Disallow rules in that group. Crawlers must parse at least the first 500 KiB of the file.

robots.txt: allow AI search, opt out of training
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

AI operators each publish the robots.txt token that controls their crawler. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic and Perplexity document their search crawlers the same way. The robots.txt guide for AI crawlers lists every token.

robots.txt has limits. RFC 9309 says the rules are not a form of access authorization. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, because a person asked for the page. Use a login to protect private content.

robots.txt can also carry a usage preference. A Content-Signal line states whether content may be used for search, AI answers and training; see Content Signals in robots.txt.

What llms.txt does

llms.txt gives a language model a short, clean starting point. The file names the site in an H1, summarizes it in one blockquote line, and lists the pages that matter in H2 sections. An agent that reads it can pick the right page without parsing navigation menus.

The evidence that answer engines read llms.txt is thin. In an Ahrefs log study of 137,210 domains, 97% of llms.txt files got zero requests in May 2026, and AI retrieval bots made 1.1% of the requests that did happen. The llms.txt evidence guide covers the numbers.

Can llms.txt block AI crawlers?

No. llms.txt has no rule syntax for access. Leaving a page out of llms.txt does not stop any crawler from fetching that page. To limit AI crawlers, use one of these:

  • robots.txt groups per user agent, for crawlers that follow RFC 9309.
  • Meta robots tags or the X-Robots-Tag header with noindex or nosnippet, to keep a page out of search results and AI answers that follow those directives.
  • Firewall rules, for crawlers that ignore robots.txt. Be careful: firewalls often block the AI search crawlers you want; see AI crawlers blocked by a firewall.
  • Authentication, for anything private.

When to use each file

Which file solves which task
TaskUse
Keep a training crawler such as GPTBot off the siterobots.txt
Make sure ChatGPT search, Claude and Perplexity can fetch your pagesrobots.txt (allow their search crawlers) and firewall settings
Tell search engines about every URLsitemap.xml
Give coding agents a short list of your docs pagesllms.txt
State whether content may train AI modelsrobots.txt with Content-Signal
Pass the Lighthouse llms.txt auditllms.txt
Rank in Google or appear in AI OverviewsNeither file alone; indexable pages and good content

If you publish both files, they should agree. Do not list pages in llms.txt that robots.txt disallows for the crawlers you care about.

llms.txt vs sitemap.xml

sitemap.xml is an XML list of URLs for search engines, defined at sitemaps.org. One sitemap file can hold up to 50,000 URLs and 50 MB uncompressed, and each entry needs a loc tag. A sitemap aims to be complete. llms.txt aims to be short: a curated set of the pages a model should read first, with a line of context for each. You can link the sitemap from robots.txt with a Sitemap: line. llms.txt has no such hook.

Common mistakes

  • Writing Disallow rules inside llms.txt. No crawler reads access rules from that file.
  • Blocking OAI-SearchBot or PerplexityBot while meaning to block training. Block the training crawlers (GPTBot, ClaudeBot, CCBot) in their own groups instead.
  • A catch-all page that returns 200 for every path. Tools then think /llms.txt exists. Return 404 for unknown URLs.
  • A robots.txt that returns a 5xx error. Compliant crawlers then treat the whole site as off limits; see how to fix a missing or broken robots.txt.
  • Expecting llms.txt to change Google rankings. Google says it does not use the file.

How to check both files

Terminal
curl -sI https://example.com/robots.txt
curl -sI https://example.com/llms.txt
curl -s https://example.com/llms.txt

Both requests should return 200. robots.txt should be text/plain. llms.txt should start with an H1 line. The AI crawler checker reads your robots.txt and shows which rule applies to each AI crawler. The llms.txt checker above validates the format and tests the links in the file.

FAQ

Is llms.txt the new robots.txt?

No. robots.txt sets access rules that crawlers follow. llms.txt is a reading list with no rules, and Google Search does not use it. Keep robots.txt in any case.

Do I need llms.txt if I have robots.txt?

Not for search or AI answers. llms.txt is optional. Publish it if you run developer docs or your logs show agents requesting it.

Should I reference llms.txt in robots.txt?

The llms.txt proposal defines no robots.txt line for it. A Sitemap: line exists for sitemaps only.

Does blocking GPTBot in robots.txt remove me from ChatGPT?

No. GPTBot is OpenAI's training crawler. ChatGPT search depends on OAI-SearchBot, which has its own robots.txt group.

Where do robots.txt and llms.txt go?

Both go at the root of the site: /robots.txt and /llms.txt. robots.txt applies per host, so each subdomain needs its own file.

Sources