Skip to content
SeenLens

Does llms.txt work?

For AI search, llms.txt has no measured effect today. Google Search does not use the file, no answer engine documents that it reads it, and an Ahrefs log study of 137,210 domains found that 97% of llms.txt files got zero requests in May 2026. Coding agents and documentation tools do fetch it, and the file costs little to publish.

By SeenLens · Last updated

Free, no sign-up. We fetch /llms.txt, /llms-full.txt and up to 10 linked URLs. We do not store the files.

Short answer: who reads llms.txt?

An llms.txt file only works if a system requests it and uses it. The table lists what each major reader says or does, with the type of evidence behind each line.

Who reads llms.txt (checked 2026-11-06)
ReaderUses llms.txt?Evidence
Google Search, AI Overviews, AI ModeNoOfficial: Google says Search does not use llms.txt or similar files
ChatGPT search (OpenAI)Not documentedOfficial crawler docs list no llms.txt input. GPTBot made 4.51% of llms.txt requests in the Ahrefs logs
Perplexity, Claude (Anthropic)Not documentedOfficial crawler docs do not mention llms.txt as an input
Coding agents such as Claude CodeSome fetch itStudy: Claude-Code fetched llms.txt more often than every AI retrieval bot in the Ahrefs logs
Chrome LighthouseChecks itOfficial: the experimental Agentic Browsing category checks for the file
SEO and GEO audit toolsFetch itStudy: SEO audit tools made 21.7% of llms.txt requests, the largest group

A fetch does not prove that the file was read or used. The Ahrefs authors say so too, and they call their numbers a ceiling on real use.

Who proposed llms.txt, and what is in the file?

Jeremy Howard proposed llms.txt on llmstxt.org on 3 September 2024. The proposal is a Markdown file at /llms.txt that gives language models a short summary of a site and links to its most useful pages. llms.txt is a community proposal. No standards body such as the IETF or W3C has adopted it. Read the llms.txt definition for the short version.

Parts of an llms.txt file, from the llmstxt.org proposal
PartExampleRequired?
H1 with the site or project name# Example CoYes, the only required part
Blockquote with a short summary> Example Co sells invoicing software for small firms.No
Free text with more detailPlain paragraphs or lists, no headingsNo
H2 sections with link lists## Docs then - [Pricing](https://example.com/pricing): plans and pricesNo
An H2 section named Optional## Optional with links an agent can skipNo

The proposal also suggests a clean Markdown copy of each page at the same URL with .md added. SeenLens checks that idea separately; see how to serve Markdown to agents.

Does Google use llms.txt?

No. Google's guide on optimizing for generative AI features says you do not need new machine-readable files, AI text files or Markdown to appear in Google Search. The guide adds that creating them will neither harm nor help a site's visibility or rankings in Google Search. This covers AI Overviews and AI Mode, because both are Google Search features.

John Mueller of Google gave the same view on Reddit in April 2025, as reported by Search Engine Journal. He wrote that, as far as he knew, none of the AI services had said they use llms.txt. He also said server logs show they do not even check for it, and he compared llms.txt to the keywords meta tag: a claim by the site owner about what the site is about.

Google is not a single reader, though. Chrome Lighthouse, a Google developer tool, does check llms.txt. That audit measures agent readiness in a browser tool. It has no link to Google Search ranking.

Does ChatGPT read llms.txt?

OpenAI has not said that ChatGPT uses llms.txt. OpenAI's crawler documentation describes three agents and lists robots.txt as the control for each. It does not name llms.txt as an input for ChatGPT search. (The OpenAI docs site does point its own readers to an llms.txt index of its documentation, which is a different use.)

  • GPTBot collects content that may be used to train OpenAI models. In the Ahrefs logs, GPTBot was the most frequent named AI bot to request llms.txt, at 4.51% of requests. A training crawler that fetches a file tells you nothing about ChatGPT answers.
  • OAI-SearchBot decides whether a site can appear in ChatGPT search. All AI retrieval bots together, OAI-SearchBot included, made only 1.1% of llms.txt requests in the same logs.
  • ChatGPT-User fetches pages when a person asks ChatGPT to. It could open llms.txt if a user links to it, like any other URL.

To get into ChatGPT search, allow OAI-SearchBot in robots.txt and in your firewall. The OpenAI crawlers guide shows the rules.

What do the log studies show?

Server logs are the most direct evidence, because they show which systems request the file. The largest study is from Ahrefs. Ahrefs sells SEO tools, so treat it as a vendor study with a large sample.

Evidence on llms.txt use, by type
SourceSample and dateFindingEvidence type
Ahrefs log study137,210 domains, May 2026 logs, published 2026-06-1528% publish llms.txt. 97% of those files got zero requests. About 1.1K domains got all measured requests.Study (vendor, large sample)
Same study, by requesterSameSEO audit tools 21.7%, AI agents 10.5%, AI training crawlers 5.3%, AI assistants 2.5%, AI retrieval bots 1.1%, humans 4%Study (vendor)
Same study, missing filesSameAI bots made zero requests for llms.txt files that do not existStudy (vendor)
Originality.ai adoption countTracked sites, June 2025 to May 2026llms.txt files rose from 4,088 to 36,120 (8.8 times)Vendor claim, seen in secondary coverage
John Mueller, GoogleReddit reply, April 2025AI services do not check for llms.txt in server logsStatement by a Google employee

Adoption grows, but requests stay rare. The Ahrefs authors conclude that llms.txt is largely ineffective for AI search visibility today, that coding agents are its most plausible audience, and that the file carries a prompt-injection risk if it contains text that an agent might obey.

Your own logs are better evidence than any study. Search your access log for GET /llms.txt and group the hits by user agent. If nothing requests the file over a month, nothing uses it on your site.

Does Lighthouse check llms.txt?

Yes. Chrome Lighthouse has an Agentic Browsing category, and one of its audits checks for llms.txt at the domain root. The Chrome docs call the category experimental and based on proposed standards. Testing needs Chrome 150 or later.

  • The category gives a fraction of checks passed. It does not give a 0 to 100 score like the Performance category.
  • The other audits cover WebMCP tools, agent-relevant accessibility (names and labels, tree integrity, visibility) and Cumulative Layout Shift.
  • The Chrome docs do not say whether a missing llms.txt counts as a fail or as not applicable. Third-party write-ups disagree, so this page leaves that open.

A Lighthouse pass shows that a file exists in a readable form. It does not show that any answer engine reads the file. See how to fix Lighthouse agentic audit failures for the other audits.

Where llms.txt can still help

  • Developer documentation. Coding agents and IDE assistants look for compact context. Documentation sites such as developers.openai.com, docs.perplexity.ai and modelcontextprotocol.io point their readers to an /llms.txt index (seen on 2026-11-06).
  • Agents that a user points at your site. When a person pastes your llms.txt URL into an assistant, the assistant reads it like any page.
  • Lighthouse and audit tools. A valid file passes the Lighthouse llms.txt audit and the SeenLens llms.txt check.
  • A map for your own tools. Internal chatbots and retrieval pipelines can use the file as a curated list of the pages that matter.

llms.txt does not replace the work that AI search depends on: crawler access, pages in the search index, and clear content. Check those first with the guide to whether your site is visible to AI.

Is llms.txt worth it? Publish it if, skip it if

Decision table for llms.txt
Publish llms.txt ifSkip llms.txt for now if
You run developer docs, an API or an SDK that coding agents useYour goal is Google rankings or AI Overviews; Google Search does not use the file
Your logs show agents requesting /llms.txtYour logs show no requests and you have more urgent fixes
A client or Lighthouse audit asks for it and it takes under an hourNobody can keep the links current when pages change
You want a curated page list for your own AI toolsYour site has no stable set of key pages to list

If an agency sells llms.txt as a way to get cited by ChatGPT or Google, ask for evidence from logs. The studies above give no support for that claim.

What does publishing llms.txt cost?

The cost is small. llms.txt is one static text file. A first version takes under an hour for most sites: an H1, a one-line summary and 10 to 30 links. The ongoing cost is keeping the links correct when pages move. A file full of dead links is worse than no file for a tool that trusts it.

  • Serve the file as plain text or Markdown with status 200.
  • Use absolute URLs and check that each link returns 200.
  • Make sure unknown paths return 404. A site that returns 200 for every path makes /llms.txt look present when it is not; see how to fix soft 404s.
  • Keep instructions out of the file. Agents may read text in it as commands.

The step-by-step llms.txt fix shows a full example. SeenLens treats llms.txt as a low-weight, optional check: it is 3 of the check weights in the Agent readiness dimension, which is 15% of the score.

Check your llms.txt file

FAQ

Is llms.txt worth it?

For AI search visibility, the evidence says no: Google Search does not use it and 97% of files in the Ahrefs study got no requests. It is worth an hour if you run developer documentation or your logs show agents requesting it.

Does Google use llms.txt?

No. Google's generative AI guide says Google Search does not need these files and that they neither help nor hurt rankings. Chrome Lighthouse checks for the file, but that is a developer tool with no link to Search ranking.

Does ChatGPT read llms.txt?

OpenAI has not said so. Its crawler documentation names robots.txt as the control and lists no llms.txt input. GPTBot, the training crawler, does request llms.txt on some sites.

Is llms.txt a standard?

No. llms.txt is a community proposal published on llmstxt.org in September 2024. robots.txt, by comparison, is an IETF standard (RFC 9309).

Can llms.txt hurt my SEO?

Google says the file neither harms nor helps visibility in Google Search. The real risks are small: stale links, and text in the file that an agent could treat as instructions.

Does Lighthouse check llms.txt?

Yes. The experimental Agentic Browsing category in Lighthouse includes an llms.txt audit. It needs Chrome 150 or later and reports a fraction of checks passed.

Sources