--- title: "How the AI Visibility Score Works: 43 Checks ยท SeenLens" description: "The 43 deterministic checks behind the SeenLens score: AI crawler access, content, structured data, agent readiness and delivery, with weights." url: https://seenlens.com/how-it-works last_updated: 2026-11-06 --- # How SeenLens works: 43 checks in five dimensions SeenLens fetches one page the way an AI crawler does and runs 43 deterministic checks on the response. Each check passes, warns or fails. The results roll up into five weighted dimension scores and one score from 0 to 100. The same page always gets the same score. Last updated: 2026-11-06 ## Methodology AI search engines such as ChatGPT, Claude, Perplexity and Google AI Overviews build answers from pages their crawlers fetched. AI agents fetch pages live when a person asks them to. Both mostly read the raw HTML response and a few site files. SeenLens looks at the same things. 1. **Validate the URL.** We reject private addresses, local hosts and non-web schemes before any request is made. 2. **Fetch the page.** SeenLensBot makes a plain HTTP GET for the page and follows redirects. If the raw HTML looks like an empty JavaScript app shell, it also renders the page in a headless browser and scores what appears. 3. **Fetch site files.** It reads robots.txt, llms.txt, llms-full.txt, a Markdown version of the page if one exists, and a short list of /.well-known files that AI agents use. It also requests one made-up URL to confirm that unknown paths return 404. 4. **Run the checks.** Each rule reads the facts from those responses and returns pass, warn, fail or not applicable, with the evidence it saw. 5. **Score.** Weighted results become the dimension scores and the overall score. The rules are versioned. This page describes rules version 0.4.1. Every report records the version it was scored with. ## The five dimensions and every check Each item below is one check: what it looks at and why it matters for AI search. The paid report adds the concrete fix and the evidence for every check that does not pass. ### AI crawler access 30% of the overall score. 11 checks. - **Page reachable for plain HTTP clients.** A plain HTTP request for the page failed or returned an error. AI crawlers and agents that do not run a browser get nothing to read or cite. Evidence: Documented by crawler operators. [How to fix](https://seenlens.com/fixes/ai-crawlers-cannot-load-page#page-reachable) - **No bot wall for plain clients.** AI crawlers and agents are plain HTTP clients. When a firewall answers with a challenge page, they see nothing of the page. Evidence: Documented by crawler operators. [How to fix](https://seenlens.com/fixes/ai-crawlers-blocked-by-waf#bot-wall) - **Main content present without JavaScript.** Most AI crawlers read the raw HTML and do not run JavaScript. If the text only appears after scripts run, they see an empty or thin page. Evidence: Vendor studies, mixed results. [How to fix](https://seenlens.com/fixes/content-needs-javascript#server-rendered) - **robots.txt reachable.** Crawlers read robots.txt first. A missing file states no policy, and a 5xx error makes compliant crawlers treat the whole site as off limits. Evidence: Documented by crawler operators. [How to fix](https://seenlens.com/fixes/robots-txt-missing) - **robots.txt policy for AI crawlers.** Blocking an AI search crawler removes the site from that engine's answers and citations. Without named groups, every AI crawler falls back to the generic * rules. Evidence: Documented by crawler operators. [How to fix](https://seenlens.com/fixes/robots-txt-blocks-ai-crawlers#robots-ai-groups) - **Search engine crawlers allowed in robots.txt.** Google AI Overviews and AI Mode cite pages from Google's search index, and Copilot answers draw on Bing's index. A page that Googlebot or Bingbot may not crawl drops out of these answers. Evidence: Documented by crawler operators. [How to fix](https://seenlens.com/fixes/robots-txt-blocks-ai-crawlers#robots-search-engines) - **User-triggered agents and robots.txt.** Agents such as ChatGPT-User fetch a page because a person asked for it. Several do not treat robots.txt as binding, so the block gives a false sense of protection and may still cut you out of answers. Evidence: Documented by crawler operators. [How to fix](https://seenlens.com/fixes/robots-txt-blocks-ai-crawlers#robots-user-agents) - **Content-Signal preference.** A Content-Signal line in robots.txt tells AI systems whether the content may be used for search, answers and training. Without it the site states no preference. Evidence: Emerging agent standard. [How to fix](https://seenlens.com/fixes/add-content-signals) - **Page allows indexing and snippets.** A noindex, nosnippet or max-snippet:0 directive stops search and AI engines from listing or quoting the page. Evidence: Documented by crawler operators. [How to fix](https://seenlens.com/fixes/noindex-blocks-ai-answers) - **AI training and grounding preferences.** Bing reads ai-grounding to decide whether a page may support Copilot and Bing AI answers. Until 2027-03-01 it also reads noarchive and nocache that way; after that date they no longer limit AI use. Evidence: Documented by crawler operators. [How to fix](https://seenlens.com/fixes/bing-ai-grounding-blocked) - **Responses to AI agent user agents.** Requests that identify as AI agents were challenged or answered differently from a browser. Verified crawlers may be blocked by the firewall. Evidence: Documented by crawler operators. [How to fix](https://seenlens.com/fixes/ai-crawlers-blocked-by-waf#waf-ua-matrix) ### Content structure 25% of the overall score. 8 checks. - **Page title.** The title is the first thing search and AI engines use to decide what the page is about and how to label a citation. Evidence: Search engine guidance. [How to fix](https://seenlens.com/fixes/page-title-missing) - **Meta description.** Engines often show the meta description as the summary. Without a good one they pick text from the page, which may be boilerplate. Evidence: Search engine guidance. [How to fix](https://seenlens.com/fixes/meta-description-missing) - **Single H1 heading.** One clear H1 tells readers and machines the main topic. None or several makes the topic harder to pin down. Evidence: Search engine guidance. [How to fix](https://seenlens.com/fixes/h1-heading-missing) - **Heading hierarchy.** H3 headings without any H2 break the outline. AI systems use headings to split a page into passages they can quote. Evidence: Search engine guidance. [How to fix](https://seenlens.com/fixes/heading-levels-skipped) - **Content depth.** Google states no ideal length. A page with very little text gives engines little to understand or quote. Evidence: Vendor studies, mixed results. [How to fix](https://seenlens.com/fixes/thin-content) - **Opening states the answer.** AI answers favour pages whose first lines state the point in one short sentence. Cookie banners or vague intros push the answer down. Evidence: Vendor studies, mixed results. [How to fix](https://seenlens.com/fixes/first-paragraph-does-not-answer) - **Publication date.** A visible date helps engines judge freshness. Answers about current topics prefer content with a clear date. Evidence: Vendor studies, mixed results. [How to fix](https://seenlens.com/fixes/add-publication-date) - **Descriptive URL path.** A study of AI answers found that pages with words in the URL path were cited a little more often than pages with ids or dates. Words also tell readers what the link is before they click. Evidence: Vendor studies, mixed results. [How to fix](https://seenlens.com/fixes/url-not-descriptive) ### Structured data 15% of the overall score. 4 checks. - **JSON-LD structured data.** JSON-LD states facts about the page and the business in a form machines read without guessing. Engines use it to build rich results and entity facts. Evidence: No proven effect on AI citations. Evidence strength: weak for AI citations. Structured data still matters for Google rich results. A matched study of 1,885 pages (Ahrefs, published 2026-05-11) found no significant citation lift. [How to fix](https://seenlens.com/fixes/add-json-ld) - **JSON-LD parses.** A JSON-LD block with a syntax error is ignored completely. The facts it was meant to state are lost. Evidence: No proven effect on AI citations. Evidence strength: weak for AI citations. Broken JSON-LD is ignored by parsers, so fixing it protects rich results. No proven AI citation lift. [How to fix](https://seenlens.com/fixes/json-ld-syntax-errors) - **Entity markup (Organization, WebSite, Person or LocalBusiness).** Organization, WebSite, Person or LocalBusiness markup tells engines who is behind the site. That helps AI answers name and trust the source. Evidence: No proven effect on AI citations. Evidence strength: weak for AI citations. Organization or WebSite markup states who runs the site. No proven AI citation lift. [How to fix](https://seenlens.com/fixes/add-organization-schema) - **Open Graph tags.** Chat apps and social sites build link previews from og:title and og:description. Some AI tools read them as a summary too. Evidence: No proven effect on AI citations. [How to fix](https://seenlens.com/fixes/open-graph-tags-missing) ### Agent readiness 15% of the overall score. 12 checks. - **llms.txt.** llms.txt is an optional Markdown index of your key pages for AI tools. Lighthouse checks it; Google Search ignores it. Evidence: No proven effect on AI citations. Evidence strength: weak. Google Search does not use llms.txt. A study of 137,210 domains (Ahrefs, published 2026-06-15) found that 97% of the files got no AI requests in May 2026. Under rules 0.4.1, its weight is 1 of 17 in agent readiness. [How to fix](https://seenlens.com/fixes/add-llms-txt) - **Markdown version of the page.** A Markdown version gives agents clean text without layout code. It is optional; no major agent is confirmed to ask for it yet. Evidence: No proven effect on AI citations. [How to fix](https://seenlens.com/fixes/serve-markdown-to-agents) - **Agent catalog (ard.json or ai-catalog.json).** An agent catalog at /.well-known/ard.json (or the legacy ai-catalog.json) lists agent resources such as MCP servers. A broken or fake catalog fails the Lighthouse agentic audit. Evidence: Emerging agent standard. [How to fix](https://seenlens.com/fixes/ai-catalog-json-errors) - **API Catalog (/.well-known/api-catalog).** Agents use the API Catalog to find your APIs. A wrong content type or an HTML page in its place misleads them. Evidence: Emerging agent standard. [How to fix](https://seenlens.com/fixes/well-known-agent-files#well-known-api-catalog) - **OAuth Protected Resource Metadata.** Agents read this file to learn how to authenticate to your API. A wrong content type or an HTML page in its place breaks that step. Evidence: Emerging agent standard. [How to fix](https://seenlens.com/fixes/well-known-agent-files#well-known-oauth-protected-resource) - **OAuth Authorization Server Metadata.** Agents read this file to find your OAuth endpoints. A wrong content type or an HTML page in its place breaks sign-in for agents. Evidence: Emerging agent standard. [How to fix](https://seenlens.com/fixes/well-known-agent-files#well-known-oauth-authorization-server) - **A2A agent card.** Other agents read the A2A agent card to learn what your agent can do. A broken card hides it. Evidence: Emerging agent standard. [How to fix](https://seenlens.com/fixes/well-known-agent-files#well-known-agent-card-json) - **UCP commerce profile.** Shopping agents read the UCP profile to learn how to buy from you. A broken profile blocks agent checkout. Evidence: Emerging agent standard. [How to fix](https://seenlens.com/fixes/well-known-agent-files#well-known-ucp) - **WebMCP tools.** WebMCP lets browser agents call your search, booking or checkout functions directly. It is an early draft, so most sites do not have it yet. Evidence: Emerging agent standard. [How to fix](https://seenlens.com/fixes/add-webmcp-tools#webmcp-tools) - **WebMCP entry point.** The page registers tools on navigator.modelContext only. Agents that follow the current draft look at document.modelContext. Evidence: Emerging agent standard. [How to fix](https://seenlens.com/fixes/add-webmcp-tools#webmcp-entry-point) - **Forms annotated for WebMCP.** Forms with toolname or tooldescription attributes can be used by browser agents safely. Unannotated forms leave agents guessing. Evidence: Emerging agent standard. [How to fix](https://seenlens.com/fixes/add-webmcp-tools#webmcp-form-annotations) - **Lighthouse Agentic Browsing audit.** Lighthouse tests how well browser agents can read and use the page, for example its accessibility tree and its WebMCP tools. Evidence: Emerging agent standard. [How to fix](https://seenlens.com/fixes/lighthouse-agentic-audit-failures) ### Technical delivery 15% of the overall score. 8 checks. - **Served over HTTPS.** Browsers flag plain HTTP pages as not secure, and engines prefer HTTPS sources. Evidence: Search engine guidance. [How to fix](https://seenlens.com/fixes/serve-page-over-https) - **HTTP status 200.** Crawlers drop pages that return errors, and long redirect chains waste crawl budget and can be cut off before the page loads. Evidence: Documented by crawler operators. [How to fix](https://seenlens.com/fixes/ai-crawlers-cannot-load-page#http-status) - **Content in the initial HTML.** Our scanner had to run a browser to see the content. Crawlers that skip JavaScript see an empty shell. Evidence: Vendor studies, mixed results. [How to fix](https://seenlens.com/fixes/content-needs-javascript#render-needed) - **Unknown URLs return 404.** A catch-all 200 page creates soft 404s. Crawlers index junk URLs, and files such as llms.txt look present when they are not. Evidence: Search engine guidance. [How to fix](https://seenlens.com/fixes/soft-404-unknown-urls) - **Canonical URL.** The canonical tag tells engines which URL to index and credit. Without it, duplicates split the signals; pointing elsewhere hands credit to another site. Evidence: Search engine guidance. [How to fix](https://seenlens.com/fixes/add-canonical-url) - **XML sitemap lists the page.** Search engines find and recrawl pages faster through an XML sitemap. AI answers from Google and Bing draw on those indexes. Evidence: Search engine guidance. [How to fix](https://seenlens.com/fixes/xml-sitemap-missing) - **Core Web Vitals.** Slow loading, slow response to taps and layout shifts hurt rankings and make visitors leave. Search engines use real-user Core Web Vitals data as a ranking signal. Evidence: Search engine guidance. [How to fix](https://seenlens.com/fixes/slow-page-core-web-vitals#core-web-vitals) - **Lighthouse performance score.** A low score means the page loads slowly on a mid-range phone. Slow pages lose visitors and can time out for crawlers and agents. Evidence: Search engine guidance. [How to fix](https://seenlens.com/fixes/slow-page-core-web-vitals#performance-score) ## How scoring works - **Check weights.** Each check has a weight from 1 to 5 inside its dimension. Blocking problems, such as a robots.txt rule that shuts out AI crawlers, weigh 5. - **Statuses.** A pass earns the full weight, a warning or an informational finding earns half, and a fail earns nothing. Checks that do not apply to the page, such as OAuth metadata on a site without an API, are left out instead of counted against you. - **Dimension scores.** Points earned divided by points possible, as a number from 0 to 100. - **Overall score.** A weighted average of the dimension scores: AI crawler access 30%, Content structure 25%, Structured data 15%, Agent readiness 15%, Technical delivery 15%. - **Grades.** A is 90 or more, B is 75 or more, C is 60 or more, D is 40 or more, and F is below 40. - **Severity.** Each issue is high, medium or low, based on how much the check costs you and whether it failed or warned. Issues are listed most severe first. Scoring is deterministic. There is no AI model in the scoring path and no randomness, so the same page with the same server responses always gets the same score. If your score changes, something on the page or the server changed. | Dimension | Weight | Checks | | --- | --- | --- | | AI crawler access | 30% | 11 | | Content structure | 25% | 8 | | Structured data | 15% | 4 | | Agent readiness | 15% | 12 | | Technical delivery | 15% | 8 | A page that plain HTTP clients cannot read, because of a bot wall or a failed fetch, is capped at 49 points whatever the other checks say. A dimension needs at least two applicable checks to get a score. ## What is not tested yet One check is defined but not run in this version, because it needs an external service. Reports list it under "not tested" and it does not affect the score. The two PageSpeed checks, Core Web Vitals and Lighthouse performance score, also move to "not tested" when the PageSpeed API does not answer. SeenLens also does not test backlinks, keyword rankings, brand mentions or whether an AI engine has cited you. ## Limits - **One page per scan.** SeenLens scores the URL you enter. It does not crawl the rest of the site, so other pages may score differently. - **Public pages only.** It cannot log in, so content behind a login wall, paywall or CAPTCHA is not seen. That is also what AI crawlers see. - **A snapshot.** The score reflects the page at scan time. Run a new scan after you make changes. - **A small request budget.** A scan makes about 25 requests at most, to keep the load on your server negligible. ## How strong is the evidence behind the checks? A SeenLens score measures how ready a page is to be read. It does not measure whether any AI engine cites the page. Every check in a report carries one of five evidence levels: - **Documented by crawler operators.** The company that runs the crawler or search engine writes down how it behaves. Examples: robots.txt rules, noindex, Bing AI preferences, reachability, bot walls and HTTP status. This is the strongest level. - **Search engine guidance.** A search engine documents the signal for its own results. Examples: title, meta description, headings, canonical, HTTPS, soft 404 and Core Web Vitals. The documents describe search, and AI answers draw on the same indexes. - **Vendor studies, mixed results.** Studies by vendors point to an effect, and the results differ between studies. Examples: server rendering, word count, opening answer and publication date. - **No proven effect on AI citations.** No operator documents the signal, and the studies we found show no effect. Examples: llms.txt, Markdown delivery, JSON-LD and Open Graph. - **Emerging agent standard.** The standard is new and few agents read it today. Examples: Content-Signal, agent catalog, well-known files, WebMCP and the Lighthouse agentic audit. Three cases show the range: - **Crawler access and delivery checks.** The crawler operators document these themselves. If a crawler is blocked or gets an error, it cannot read the page. This is the strongest group. - **llms.txt.** Google says Search does not use llms.txt ([Google: optimizing your website for generative AI features](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide), checked 2026-11-06). SeenLens keeps the check because Lighthouse and some agents read the file. - **JSON-LD.** Google says structured data is not required for its generative AI features. A matched study found no significant citation lift ([Ahrefs: schema and AI citations](https://ahrefs.com/blog/schema-ai-citations/), 1,885 pages, published 2026-05-11; a vendor study). Keep JSON-LD for rich results and clear facts, not as a citation lever. ## Method changes Every report records the rules version it was scored with. This list shows what changed between versions. | Rules version | Date | Change | | --- | --- | --- | | 0.1 | 2026-11-04 | First release. Core Web Vitals was defined but not tested. | | 0.2 | 2026-11-05 | PageSpeed data added. Core Web Vitals now reads field or lab data, and a new check scores the Lighthouse performance result. | | 0.3 | 2026-11-05 | Bot wall detection added: DataDome, AWS WAF, Cloudflare, Akamai, PerimeterX and Imperva challenge pages are reported as a high priority issue. Pages that plain clients cannot read are capped at 49 points. A dimension needs two applicable checks to be scored. Heading, first paragraph and Organization extraction was corrected. | | 0.3.1 | 2026-11-06 | Robots meta parsing fixed: `max-image-preview:none` no longer counts as the blocking `none` directive, and crawler-specific values such as `googlebot: noindex` are read correctly. Only pages that use these values change score. | | 0.3.2 | 2026-11-06 | Agent catalog check follows ARD v0.91 and Lighthouse 13.6: SeenLens requests `/.well-known/ard.json` first and checks the legacy `ai-catalog.json` only when ard.json is missing. Sites without either file are not affected. | | 0.3.3 | 2026-11-06 | New check for Bing's AI preferences, announced 2026-10-30: `ai-grounding:disallow` in a robots or Bingbot meta tag or an `X-Robots-Tag` header fails, `noarchive` and `nocache` warn until Bing stops reading them as AI limits on 2027-03-01, and a training opt-out alone passes. A page with no such limit gains one passed check, which raises its total by at most 3 points; pages with a crawler access score above 70 gain 1 point or less. | | 0.3.4 | 2026-11-07 | New check: robots.txt must not block Googlebot or Bingbot for the scanned page, because Google AI Overviews and AI Mode cite pages from Google's index and Copilot answers draw on Bing's index. The report adds a table of 15 crawlers with their robots.txt result, names rules that come from Cloudflare's managed robots.txt, labels each check with its evidence strength, and adds steps to measure AI traffic. Only the new check changes scores. On the 30-site set, 22 scores stay the same, 7 rise by 1 or 2 points, and Reddit falls from 49 to 44 because its robots.txt disallows every crawler. | | 0.4.0 | 2026-11-07 | Weights now follow the evidence: llms.txt counts 1 instead of 3, Markdown delivery 2 instead of 3, and Content-Signal 1 instead of 2. A page with 100 to 299 words gets a note instead of a warning; the score value is the same. Two new checks: the URL path uses words, and an XML sitemap lists the page. The Lighthouse agentic audit, listed before but never run, now runs through PageSpeed when PageSpeed is available. On the 30-site set without the Lighthouse audit, no score moved more than 2 points: 11 stayed the same, 8 rose and 11 fell. Sites that fail some Lighthouse agentic audits can lose a few more points in agent readiness. The paid report also adds a page structure profile, an entity profile, page dates, a prompt and code to paste for each fix, quotable passages and likely sub-questions. | | 0.4.1 | 2026-11-07 | No score change. Reports now list the problems first and then what the page does well, with the evidence for each passed check. The free report shows the score, the five dimension scores and three items: the top issues or, when a page has fewer than three, what it does well. | Before 0.3 shipped, we re-scored 30 public sites with the stored scan facts (2026-11-05). 25 of the 30 scores moved by 3 points or less. The other five moved more. Four of them are sites that answered with a bot wall or a refusal: Amazon fell from 57 to 49, Booking.com from 46 to 20 and Yelp from 32 to 15, while Etsy rose from 42 to 47 because content checks on a refused page became "not tested". ## Sources - [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) (user agents and robots.txt tokens, checked 2026-11-06) - [Google: Optimizing your website for generative AI features](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) (llms.txt and structured data are not required, checked 2026-11-06) - [Ahrefs: schema and AI citations study](https://ahrefs.com/blog/schema-ai-citations/) (1,885 pages with 4,000 controls, published 2026-05-11, vendor study) - [Ahrefs: llms.txt study](https://ahrefs.com/blog/llmstxt-study/) (137,210 domains, published 2026-06-15, vendor study) ## The scanner: SeenLensBot Every request comes from SeenLensBot, which identifies itself with `SeenLensBot/0.1 (+https://seenlens.com/bot)` in its user agent and respects robots.txt rules for SeenLensBot. [Read what SeenLensBot fetches and how to block it](https://seenlens.com/bot). [Scan a page now](https://seenlens.com/)