AI readiness statistics: what a scan of 30 well-known websites found

• Crawler access is mostly open. 22 of 28 readable robots.txt files allowed all 15 crawler tokens we tested at the scanned page. 6 of 28 blocked at least one AI crawler.
• Blocking training usually means blocking search too. 5 of 28 sites blocked an AI search crawler such as OAI-SearchBot. Only 1 site blocked GPTBot while it allowed OAI-SearchBot.
• A firewall can refuse what robots.txt allows. 5 of 30 pages refused our scanner with HTTP 403 or 202. On one of them, robots.txt allowed all 13 AI crawler tokens we tested.
• AI-specific signals are rare. Content-Signal: 2 of 28. Markdown version: 6 of 25. WebMCP tools: 1 of 14.
• llms.txt is common on these sites. 14 of 27 sites served a valid llms.txt. 12 returned HTTP 404 and 1 file failed validation.
• Structured data is the weakest area. 13 of 25 pages had no JSON-LD. The median structured data score was 18 out of 100.
• Scores: median 78 out of 100 for all 30 pages, and 79 for the 25 pages that loaded.

Each share uses the pages where the check could run, so the denominators differ. A page that refused the scanner has no result for the page checks. The method and its limits are at the end of this page.

We read each site's robots.txt and applied the rule group for each crawler token to the scanned page path, as RFC 9309 describes. 28 of 30 robots.txt files were readable. The other 2 sites answered the robots.txt request with HTTP 403 or a challenge.

Most sites that block one AI crawler block several. 5 of the 6 also block at least one AI search crawler, and 4 of those 5 still allow Googlebot. 4 of the 5 block OAI-SearchBot, and OpenAI says pages opted out of OAI-SearchBot are not shown in ChatGPT search answers. Google can still index those pages. Only 1 site separated training from search for OpenAI: it blocked GPTBot and allowed OAI-SearchBot. One site blocked every crawler, Googlebot included, through its User-agent: * group.

OpenAI and Anthropic document separate tokens for training and for search. If you want AI search visibility without training use, block the training token and keep the search token allowed. The OpenAI crawler guide and the robots.txt guide for AI crawlers show the rules.

Only 10 of 28 files named any AI crawler token in a rule group. The other 18 left every AI crawler to the User-agent: * group.

The scanner flagged 4 of the 5 refusals as bot walls. Only one response carried an explicit challenge signal: an x-amzn-waf-action: challenge header with HTTP 202. Two HTTP 403 responses came with bot protection vendor headers and a body under 1 KB. One HTTP 403 had an empty body. The fifth was an HTTP 403 with a full page that said the request was blocked; it had no vendor header, and the scanner did not flag it as a bot wall.

On one of the two pages with vendor headers, robots.txt allowed all 13 AI crawler tokens tested, but the page still answered HTTP 403. The robots.txt policy and the firewall made different decisions. The scanner caps the score of a page it cannot read at 49. Our requests came from one scanner network, so a verified crawler from its operator's IP range can get a different answer. The fix for AI crawlers blocked by a firewall explains where to look in a CDN or WAF.

llms.txt was more common here than on the wider web. Ahrefs found llms.txt on 28% of 137,210 domains, and 97% of those files got no requests in a month. Well-known sites adopt new formats early, so this sample overstates adoption. The evidence on llms.txt shows no proof that the file increases AI citations.

Content-Signal lets a site state whether its content may be used for search, as AI input or for training. Only 2 of 28 sites published it. The Content Signals guide explains the three settings, and the Content Signal generator writes the line.

12 of 25 pages that loaded published JSON-LD, and all 12 blocks parsed as valid JSON. 10 of 25 passed our check for an Organization or WebSite node, which names the publisher and the site. Our scanner reads JSON-LD only. It does not detect microdata or RDFa, so a few pages may carry structured data in another format.

Google says its AI features need no special schema, and studies have not shown that JSON-LD raises AI citations. JSON-LD still gives machines exact facts, such as the publisher's name and logo. The JSON-LD fix and the Organization schema guide show what to add. The schema checker reads what a page publishes today.

The five highest scores ranged from 92 to 96. All five pages loaded without a bot wall, served their main content in the HTML, and had a title, Open Graph tags and a readable robots.txt. All five published valid JSON-LD with an Organization or WebSite node; 5 of the other 20 pages that loaded did. All five passed the llms.txt check; 8 of the other 20 did. None of the five passed the Core Web Vitals check. Across all pages, 5 of 23 pages with PageSpeed data passed it.

The score measures how ready a page is to be read by search engines, AI crawlers and agents. It does not measure whether an AI answer cites the page. How the score works lists all 43 checks and their weights.

• Sample: one URL on each of 30 well-known sites, chosen by hand to cover retail, news, documentation, developer tools, software, travel, community and public-sector sites. 27 URLs were site home pages. The sites are anonymous on this page.
• When: 7 November 2026, from 09:06 to 09:20 UTC. One scan per page.
• Rules: SeenLens rules 0.4.1, with 43 checks. The user agent comparison probe was off, so 42 checks could give a result.
• Requests: the SeenLens scanner with its own user agent, from a server in Finland. Two sites redirected to a European version of the page.
• robots.txt: each crawler token was matched against the scanned path. A crawler with no matching group falls under User-agent: *.
• Repeatability: the same input and rules version give the same score. We rescored all 30 stored results and matched every production score. Between two rule versions on the same input, 20 of the 25 pages that loaded moved by 3 points or less.

• 30 sites is a small sample, and we did not pick them at random. Treat the shares as a description of these sites, not of the web.
• We tested one page per site. Other pages on the same site can behave differently.
• A response to our scanner does not show what a verified crawler gets from its own IP range.
• The scan does not measure AI citations or traffic.

You can cite these numbers as "SeenLens, AI readiness scan of 30 well-known websites, November 2026" with a link to this page.

The free AI crawler checker reads your robots.txt for 14 crawlers and compares your page's response to a browser and four AI crawlers. The free scan on the home page runs all checks on one URL and shows your score and up to three findings.

Scan your page with SeenLens

• SeenLens scan of 30 well-known websites, rules 0.4.1, 7 November 2026 (this study; aggregate results only)
• RFC 9309: Robots Exclusion Protocol (how a crawler picks its rule group and matches paths; checked 2026-11-07)
• OpenAI: Overview of OpenAI crawlers (GPTBot, OAI-SearchBot and ChatGPT-User are separate controls; checked 2026-11-07)
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (ClaudeBot, Claude-SearchBot and Claude-User; checked 2026-11-07)
• Ahrefs: llms.txt study (28% of 137,210 domains publish llms.txt; 97% of files got no requests; published 2026-06-15)
• Google Search Central: AI features and your website (no special schema or extra requirements for AI Overviews and AI Mode; checked 2026-11-07)
• Ahrefs: Does schema markup increase AI citations? (before and after comparison found no significant lift; checked 2026-11-07)