---
title: "AI Crawler List 2026: User Agents and What They Do"
description: "The AI crawlers that matter for AI search in 2026: operator, purpose, robots.txt token and what blocking each one changes, checked against official docs."
url: https://seenlens.com/bots
last_updated: 2026-11-06
---

# AI crawlers and their user agents

This list covers the AI crawlers and fetchers that decide whether AI search can use your site in 2026. Each page names the operator, what the crawler collects, its exact robots.txt token and what blocking it changes, checked against the operator's own documentation.

By SeenLens. Last updated: 2026-11-06

## Three kinds of AI crawler

AI companies now run separate user agents for separate jobs. Blocking the wrong one can remove your site from AI search answers while leaving model training untouched, or the other way round.

| Kind | What it does | Examples | What blocking it changes |
| --- | --- | --- | --- |
| Training crawler | Collects public pages to train models | GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended | Your pages leave future training data. AI search answers can still cite you |
| Search crawler | Builds the index an AI search engine answers from | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Your pages can drop out of that engine's answers and citations |
| User fetcher | Fetches a page when a user asks the assistant to | ChatGPT-User, Claude-User, Perplexity-User | The assistant cannot open your page for that user. Some fetchers do not follow robots.txt |

## Check which ones your site blocks

The [AI crawler checker](https://seenlens.com/tools/ai-crawler-checker) reads your robots.txt the way each crawler does and requests your page with each crawler's user agent, so it also finds firewall rules that block a crawler your robots.txt allows. The guide to [robots.txt for AI crawlers](https://seenlens.com/guides/robots-txt-for-ai-crawlers) shows rules you can copy.

## Pages in this section

- [GPTBot](https://seenlens.com/bots/gptbot): OpenAI's training crawler: user agent, IP list, robots.txt token, and how to block it while staying in ChatGPT search.
- [OAI-SearchBot](https://seenlens.com/bots/oai-searchbot): OpenAI's crawler for ChatGPT search: user agent, IP list, robots.txt token, and what blocking it does.
- [ChatGPT-User](https://seenlens.com/bots/chatgpt-user): OpenAI's user-triggered fetcher: user agent, IP list, and why OpenAI says robots.txt rules may not apply.
- [ClaudeBot](https://seenlens.com/bots/claudebot): Anthropic's training crawler: robots.txt token, IP list, Crawl-delay, and how to block it while staying in Claude search.
- [Claude-SearchBot](https://seenlens.com/bots/claude-searchbot): Anthropic's search crawler and Claude-User fetcher: robots.txt tokens, IP list, and what blocking each one does.
- [PerplexityBot](https://seenlens.com/bots/perplexitybot): Perplexity's search crawler and Perplexity-User fetcher: user agents, IP lists, robots.txt behavior, WAF rules.
- [Google-Extended](https://seenlens.com/bots/google-extended): Google's robots.txt token for Gemini training and grounding: no user agent, no effect on Search, and how it relates to AI Overviews.
- [Applebot-Extended](https://seenlens.com/bots/applebot-extended): Apple's robots.txt token for AI training opt-out: it does not crawl, and how it differs from Applebot.
- [CCBot](https://seenlens.com/bots/ccbot): Common Crawl's crawler: user agent, robots.txt rule, Crawl-delay, IP list, reverse DNS check, and how to block it.
- [Bingbot](https://seenlens.com/bots/bingbot): Microsoft's Bing crawler: user agent strings, reverse DNS check, IP list, robots.txt rules, and its role in Copilot and AI search.