AI search guides
These guides answer the questions site owners ask most about AI search. They cover which crawlers to allow, whether llms.txt works, how GEO differs from SEO and how to measure AI visibility. Each guide cites primary sources and shows the date it was last checked.
By SeenLens · Last updated
Start here
| Your question | Guide |
|---|---|
| Which AI crawlers should my robots.txt allow or block? | robots.txt for AI crawlers |
| What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User? | OpenAI's crawlers compared |
| Can ChatGPT and other AI search engines see my website? | Is my website visible to AI? |
| Why does my site not appear in Google AI Overviews? | Not showing in AI Overviews |
| Does llms.txt do anything? | Does llms.txt work? |
| How do GEO and AEO differ from SEO? | GEO vs AEO vs SEO |
| How do I audit a site for AI search myself? | GEO audit checklist |
| How do I measure AI visibility? | How to check AI visibility |
Each guide links to the fix guides for the problems it describes and to the free tools that test them. Crawler names link to their entries in the AI crawler list.
How we write the guides
- The answer comes first, in the opening paragraph.
- Claims about crawlers and search engines link to the operator's own documentation.
- Studies are named with their sample size and date. Vendor claims are marked as vendor claims.
- Code examples say which platform and version they are for.
- Each guide shows the date it was last checked.
Pages in this section
- Robots.txt for AI crawlers
AI crawler tokens, copy-paste robots.txt recipes (allow all, allow AI search and block training, block all), how groups match under RFC 9309, common mistakes and firewall blocks.
- OpenAI crawlers
GPTBot trains, OAI-SearchBot feeds ChatGPT search, ChatGPT-User fetches pages for users. What each does, what blocking changes and the robots.txt for each goal.
- Content Signals in robots.txt
Content Signals is Cloudflare's robots.txt extension for search, ai-input and ai-train. Syntax, defaults, who reads it today, the IETF AIPREF link and a live example.
- Is my website visible to AI?
Five things must be true for AI search to see your site: crawler access, a search index, text in the raw HTML, a page that answers the question, and brand mentions elsewhere.
- AI Overview not showing up
Why AI Overviews do not appear for a search, and a diagnosis checklist for sites that are missing from them: indexing, snippet controls, query fan-out and Search Console.
- Does llms.txt work?
Evidence-led answer on llms.txt: Google's position, the Ahrefs log study of 137K domains, ChatGPT, Lighthouse, and when the file is still worth publishing.
- llms.txt vs robots.txt
Side-by-side comparison of llms.txt and robots.txt (and sitemap.xml): purpose, readers, format, standard status, location, and when to use each.
- llms.txt vs AGENTS.md
Comparison of llms.txt, AGENTS.md, the Model Context Protocol and schema.org markup: purpose, readers, format, status, location, and which one a site needs.
- GEO vs SEO vs AEO
Plain definitions of SEO, AEO and GEO, a comparison of goals and measurement, the evidence on overlap, and a practical plan.
- GEO audit checklist
Do-it-yourself GEO audit: 43 checks grouped by crawler access, content, structured data, agent readiness and technical delivery, each with a manual test and a fix link.
- WebMCP
WebMCP lets a web page expose tools to AI agents. A dated guide to the draft API, form annotations, WebMCP vs MCP and browser support.
- Bing AI Performance
Bing Webmaster Tools AI Performance counts citations of your pages in Copilot and Bing AI answers. How to open it, export it and use grounding queries.
- Check AI visibility
Five ways to check your AI visibility, with cost, what each method measures and what it misses: prompt panel, Search Console, Bing, logs, readiness scan.
- IndexNow
IndexNow pings search engines when a page is added, changed or deleted. Setup, curl and Next.js examples, supported engines, limits and Google's position.
- Where ChatGPT gets its information
The sources behind ChatGPT answers: training data, live web search, user requests. OpenAI's bots (GPTBot, OAI-SearchBot, ChatGPT-User), the Ahrefs citation study and how to check your site.
- Can Claude search the web?
Claude web search explained: the app, the API tools and Claude Code, Anthropic's ClaudeBot, Claude-SearchBot and Claude-User, the search provider question and robots.txt controls.
- Bing AI grounding and training
Bing's two AI preferences: AI Training (robots.txt token __usage_ai-training) and AI Grounding (page level ai-grounding), with syntax, dates and a decision table.
- AI readiness statistics
Study: crawler access, llms.txt, Content-Signal, JSON-LD and bot walls on 30 well-known websites (November 2026).