Skip to content
SeenLens

What is ClaudeBot?

ClaudeBot is the web crawler Anthropic uses to collect web content that could contribute to training its generative AI models. Its robots.txt token is ClaudeBot. Anthropic says its bots honor robots.txt, and restricting ClaudeBot signals that a site's future material should be excluded from training. Anthropic publishes an IP list. Verified 2026-11-06.

By SeenLens · Last updated

Free, no sign-up. We read robots.txt and request the page with a browser and four crawler user agents. We do not store your page text.

ClaudeBot fact sheet

This page covers Anthropic's crawler. It does not cover Clawdbot, an unrelated agent whose name looks similar.

ClaudeBot facts, from Anthropic's help article
FactValue
OperatorAnthropic
PurposeTraining. ClaudeBot collects web content that could potentially contribute to training Anthropic's generative AI models.
User agent stringAnthropic's help article names the bot ClaudeBot and does not publish a full user agent string. Match on ClaudeBot.
robots.txt tokenClaudeBot
Published IP listclaude.com/crawling/bots.json
Obeys robots.txtYes. Anthropic: its bots respect "do not crawl" signals by honoring industry standard directives in robots.txt. It also supports the non-standard Crawl-delay extension (Anthropic).
Related botsClaude-SearchBot (search), Claude-User (user requests, covered on the Claude-SearchBot page)
Last verified2026-11-06

What does ClaudeBot do?

ClaudeBot gathers public web content for model development. Anthropic runs it next to two other bots with different jobs: Claude-SearchBot improves search results, and Claude-User fetches pages when a person asks Claude a question. Anthropic says it uses different bots to give site owners transparency and choice.

Anthropic states two crawling principles: its crawling should not be intrusive or disruptive, and its bots respect anti-circumvention technologies. As an example, Anthropic says it will not try to bypass CAPTCHAs.

Anthropic also says that since 2026-10-30 it recognizes a machine-readable "no AI training" signal, based on an IETF aipref working group draft. If your robots.txt names ClaudeBot, Anthropic follows that rule even when a general setting says something different. The standard is still changing, and Anthropic says it may change how it reads these settings.

How do I block ClaudeBot in robots.txt?

Anthropic's instruction is to add a Disallow rule for the bot in the robots.txt file in your top-level directory, and to repeat it for every subdomain you want to opt out:

robots.txt: block ClaudeBot
User-agent: ClaudeBot
Disallow: /

To slow ClaudeBot down instead of blocking it, use Anthropic's Crawl-delay example:

robots.txt: Crawl-delay for ClaudeBot
User-agent: ClaudeBot
Crawl-delay: 1

To stay visible in Claude search and opt out of training, block only ClaudeBot and allow the other two Anthropic bots:

robots.txt: allow search bots, block training
User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

Do not block ClaudeBot by IP address alone. Anthropic says IP blocking may not work correctly or persistently guarantee an opt-out, because it stops the bot from reading your robots.txt file.

Does blocking ClaudeBot affect Claude search or citations?

Anthropic describes the effect of a ClaudeBot block as a signal that the site's future materials should be excluded from its AI model training datasets. The article lists search and user-directed fetching as the jobs of Claude-SearchBot and Claude-User.

Anthropic says that disabling Claude-SearchBot may reduce your site's visibility and accuracy in user search results, and that disabling Claude-User may reduce visibility for user-directed web search. It attaches no such warning to ClaudeBot.

How do I check whether my site blocks ClaudeBot?

Use the AI crawler checker above, preselected for ClaudeBot. It shows whether robots.txt allows ClaudeBot for your page path and which line matched. It then requests the page with the ClaudeBot user agent and compares status code and response size with a browser request, which finds firewall blocks that robots.txt does not show.

In your logs, search for ClaudeBot. Compare the source IPs with Anthropic's bots.json file. Anthropic says a request from an address on that list comes from Anthropic.

What are the common ClaudeBot mistakes?

  • Blocking one hostname only. Anthropic asks you to repeat the robots.txt rule on every subdomain. www.example.com and blog.example.com each need the file.
  • A firewall 403 while robots.txt allows ClaudeBot. The firewall acts first. See AI crawlers blocked by a firewall.
  • A wildcard group. User-agent: * with Disallow: / blocks Claude-SearchBot and Claude-User too, unless each has its own group. A crawler uses the group for its own token and falls back to * only if none matches (RFC 9309).
  • IP blocking instead of robots.txt. Anthropic warns that this can stop the bot from reading your opt-out.
  • Edge settings that block AI bots. Cloudflare's AI Crawler and Agent access policy can block training crawlers at the edge. See robots.txt blocks AI crawlers.
  • Mixing up ClaudeBot and Clawdbot. The two are unrelated. Filter your logs on the exact string ClaudeBot.

ClaudeBot FAQ

What is the ClaudeBot user agent?

Anthropic's help article does not publish a full user agent string. It names the bot ClaudeBot, which is also the robots.txt token. Match on ClaudeBot and confirm the source IP against bots.json.

How do I block ClaudeBot?

Add User-agent: ClaudeBot and Disallow: / to the robots.txt file at the top level of each hostname.

Does ClaudeBot respect robots.txt?

Anthropic says its bots honor industry standard robots.txt directives and support Crawl-delay.

Does blocking ClaudeBot remove my site from Claude search?

Anthropic assigns search to Claude-SearchBot and user-directed fetches to Claude-User. A ClaudeBot block signals exclusion from training data.

Does ClaudeBot support Crawl-delay?

Yes. Anthropic says it supports the non-standard Crawl-delay extension.

Sources