AI crawlers and their user agents
This list covers the AI crawlers and fetchers that decide whether AI search can use your site in 2026. Each page names the operator, what the crawler collects, its exact robots.txt token and what blocking it changes, checked against the operator's own documentation.
By SeenLens · Last updated
Three kinds of AI crawler
AI companies now run separate user agents for separate jobs. Blocking the wrong one can remove your site from AI search answers while leaving model training untouched, or the other way round.
| Kind | What it does | Examples | What blocking it changes |
|---|---|---|---|
| Training crawler | Collects public pages to train models | GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended | Your pages leave future training data. AI search answers can still cite you |
| Search crawler | Builds the index an AI search engine answers from | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Your pages can drop out of that engine's answers and citations |
| User fetcher | Fetches a page when a user asks the assistant to | ChatGPT-User, Claude-User, Perplexity-User | The assistant cannot open your page for that user. Some fetchers do not follow robots.txt |
Check which ones your site blocks
The AI crawler checker reads your robots.txt the way each crawler does and requests your page with each crawler's user agent, so it also finds firewall rules that block a crawler your robots.txt allows. The guide to robots.txt for AI crawlers shows rules you can copy.
Pages in this section
- GPTBot
OpenAI's training crawler: user agent, IP list, robots.txt token, and how to block it while staying in ChatGPT search.
- OAI-SearchBot
OpenAI's crawler for ChatGPT search: user agent, IP list, robots.txt token, and what blocking it does.
- ChatGPT-User
OpenAI's user-triggered fetcher: user agent, IP list, and why OpenAI says robots.txt rules may not apply.
- ClaudeBot
Anthropic's training crawler: robots.txt token, IP list, Crawl-delay, and how to block it while staying in Claude search.
- Claude-SearchBot
Anthropic's search crawler and Claude-User fetcher: robots.txt tokens, IP list, and what blocking each one does.
- PerplexityBot
Perplexity's search crawler and Perplexity-User fetcher: user agents, IP lists, robots.txt behavior, WAF rules.
- Google-Extended
Google's robots.txt token for Gemini training and grounding: no user agent, no effect on Search, and how it relates to AI Overviews.
- Applebot-Extended
Apple's robots.txt token for AI training opt-out: it does not crawl, and how it differs from Applebot.
- CCBot
Common Crawl's crawler: user agent, robots.txt rule, Crawl-delay, IP list, reverse DNS check, and how to block it.
- Bingbot
Microsoft's Bing crawler: user agent strings, reverse DNS check, IP list, robots.txt rules, and its role in Copilot and AI search.