--- title: "ClaudeBot User Agent: What It Is and How to Block It" description: "ClaudeBot is Anthropic's training crawler. See its robots.txt token, IP list and Crawl-delay support, and how to block it without losing Claude search." url: https://seenlens.com/bots/claudebot last_updated: 2026-11-06 --- # What is ClaudeBot? ClaudeBot is the web crawler Anthropic uses to collect web content that could contribute to training its generative AI models. Its robots.txt token is `ClaudeBot`. Anthropic says its bots honor robots.txt, and restricting ClaudeBot signals that a site's future material should be excluded from training. Anthropic publishes an IP list. Verified 2026-11-06. By SeenLens. Last updated: 2026-11-06 ## ClaudeBot fact sheet This page covers Anthropic's crawler. It does not cover Clawdbot, an unrelated agent whose name looks similar. | Fact | Value | | --- | --- | | Operator | Anthropic | | Purpose | Training. ClaudeBot collects web content that could potentially contribute to training Anthropic's generative AI models. | | User agent string | Anthropic's help article names the bot `ClaudeBot` and does not publish a full user agent string. Match on `ClaudeBot`. | | robots.txt token | `ClaudeBot` | | Published IP list | [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json) | | Obeys robots.txt | Yes. Anthropic: its bots respect "do not crawl" signals by honoring industry standard directives in robots.txt. It also supports the non-standard Crawl-delay extension ([Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)). | | Related bots | [Claude-SearchBot](https://seenlens.com/bots/claude-searchbot) (search), Claude-User (user requests, covered on the Claude-SearchBot page) | | Last verified | 2026-11-06 | ## What does ClaudeBot do? ClaudeBot gathers public web content for model development. Anthropic runs it next to two other bots with different jobs: Claude-SearchBot improves search results, and Claude-User fetches pages when a person asks Claude a question. Anthropic says it uses different bots to give site owners transparency and choice. Anthropic states two crawling principles: its crawling should not be intrusive or disruptive, and its bots respect anti-circumvention technologies. As an example, Anthropic says it will not try to bypass CAPTCHAs. Anthropic also says that since 2026-10-30 it recognizes a machine-readable "no AI training" signal, based on an IETF aipref working group draft. If your robots.txt names ClaudeBot, Anthropic follows that rule even when a general setting says something different. The standard is still changing, and Anthropic says it may change how it reads these settings. ## How do I block ClaudeBot in robots.txt? Anthropic's instruction is to add a Disallow rule for the bot in the robots.txt file in your top-level directory, and to repeat it for every subdomain you want to opt out: ```text User-agent: ClaudeBot Disallow: / ``` To slow ClaudeBot down instead of blocking it, use Anthropic's Crawl-delay example: ```text User-agent: ClaudeBot Crawl-delay: 1 ``` To stay visible in Claude search and opt out of training, block only ClaudeBot and allow the other two Anthropic bots: ```text User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: ClaudeBot Disallow: / ``` Do not block ClaudeBot by IP address alone. Anthropic says IP blocking may not work correctly or persistently guarantee an opt-out, because it stops the bot from reading your robots.txt file. ## Does blocking ClaudeBot affect Claude search or citations? Anthropic describes the effect of a ClaudeBot block as a signal that the site's future materials should be excluded from its AI model training datasets. The article lists search and user-directed fetching as the jobs of [Claude-SearchBot](https://seenlens.com/bots/claude-searchbot) and Claude-User. Anthropic says that disabling Claude-SearchBot may reduce your site's visibility and accuracy in user search results, and that disabling Claude-User may reduce visibility for user-directed web search. It attaches no such warning to ClaudeBot. ## How do I check whether my site blocks ClaudeBot? Use the [AI crawler checker](https://seenlens.com/tools/ai-crawler-checker) above, preselected for ClaudeBot. It shows whether robots.txt allows ClaudeBot for your page path and which line matched. It then requests the page with the ClaudeBot user agent and compares status code and response size with a browser request, which finds firewall blocks that robots.txt does not show. In your logs, search for `ClaudeBot`. Compare the source IPs with Anthropic's bots.json file. Anthropic says a request from an address on that list comes from Anthropic. ## What are the common ClaudeBot mistakes? - **Blocking one hostname only.** Anthropic asks you to repeat the robots.txt rule on every subdomain. `www.example.com` and `blog.example.com` each need the file. - **A firewall 403 while robots.txt allows ClaudeBot.** The firewall acts first. See [AI crawlers blocked by a firewall](https://seenlens.com/fixes/ai-crawlers-blocked-by-waf). - **A wildcard group.** `User-agent: *` with `Disallow: /` blocks Claude-SearchBot and Claude-User too, unless each has its own group. A crawler uses the group for its own token and falls back to `*` only if none matches ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html)). - **IP blocking instead of robots.txt.** Anthropic warns that this can stop the bot from reading your opt-out. - **Edge settings that block AI bots.** Cloudflare's AI Crawler and Agent access policy can block training crawlers at the edge. See [robots.txt blocks AI crawlers](https://seenlens.com/fixes/robots-txt-blocks-ai-crawlers). - **Mixing up ClaudeBot and Clawdbot.** The two are unrelated. Filter your logs on the exact string `ClaudeBot`. ## Which other AI bots should I know about? Each AI company runs several bots with separate robots.txt tokens. These pages cover the others. The [bots hub](https://seenlens.com/bots) lists all of them. - [GPTBot](https://seenlens.com/bots/gptbot): OpenAI's training crawler. - [OAI-SearchBot](https://seenlens.com/bots/oai-searchbot): OpenAI's crawler for ChatGPT search. - [ChatGPT-User](https://seenlens.com/bots/chatgpt-user): OpenAI's fetcher for user requests in ChatGPT. - [Claude-SearchBot](https://seenlens.com/bots/claude-searchbot): Anthropic's search crawler, plus Claude-User. - [PerplexityBot](https://seenlens.com/bots/perplexitybot): Perplexity's search crawler, plus Perplexity-User. - [Google-Extended](https://seenlens.com/bots/google-extended): Google's robots.txt token for Gemini training and grounding. - [Applebot-Extended](https://seenlens.com/bots/applebot-extended): Apple's robots.txt token for AI training. - [CCBot](https://seenlens.com/bots/ccbot): Common Crawl's open dataset crawler. ## ClaudeBot FAQ ### What is the ClaudeBot user agent? Anthropic's help article does not publish a full user agent string. It names the bot ClaudeBot, which is also the robots.txt token. Match on `ClaudeBot` and confirm the source IP against bots.json. ### How do I block ClaudeBot? Add `User-agent: ClaudeBot` and `Disallow: /` to the robots.txt file at the top level of each hostname. ### Does ClaudeBot respect robots.txt? Anthropic says its bots honor industry standard robots.txt directives and support Crawl-delay. ### Does blocking ClaudeBot remove my site from Claude search? Anthropic assigns search to Claude-SearchBot and user-directed fetches to Claude-User. A ClaudeBot block signals exclusion from training data. ### Does ClaudeBot support Crawl-delay? Yes. Anthropic says it supports the non-standard Crawl-delay extension. ## Sources - [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (bot roles, token, Crawl-delay, IP list, IP blocking warning, training signal; checked 2026-11-06) - [Anthropic: bots.json](https://claude.com/crawling/bots.json) (published IP prefixes; the file loaded on 2026-11-06) - [IETF RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) (group selection and the `*` fallback; checked 2026-11-06)