Skip to content
SeenLens

What is Claude-SearchBot?

Claude-SearchBot is the Anthropic crawler that analyzes online content to improve the relevance and accuracy of search results in Claude. Its robots.txt token is Claude-SearchBot. Anthropic says disabling it prevents indexing of your content for search and may reduce your visibility in user search results. Claude-User is a separate bot for user requests. Verified 2026-11-06.

By SeenLens · Last updated

Free, no sign-up. We read robots.txt and request the page with a browser and four crawler user agents. We do not store your page text.

Claude-SearchBot fact sheet

Claude-SearchBot and Claude-User facts, from Anthropic's help article
FactClaude-SearchBotClaude-User
OperatorAnthropicAnthropic
PurposeSearch index. It navigates the web to improve search result quality for users.User-triggered fetch. When a person asks Claude a question, Claude may access websites with this agent.
User agent stringNot published in Anthropic's article. Match on Claude-SearchBot.Not published in Anthropic's article. Match on Claude-User.
robots.txt tokenClaude-SearchBotClaude-User
Published IP listclaude.com/crawling/bots.jsonclaude.com/crawling/bots.json
Obeys robots.txtYes. Anthropic says its bots honor industry standard robots.txt directives.Yes, under the same Anthropic statement for its bots.
Related botsClaudeBot (training)ClaudeBot (training)
Last verified2026-11-062026-11-06

What does Claude-SearchBot do?

Claude-SearchBot crawls pages so that Claude can return better search results. Anthropic says it analyzes online content specifically to enhance the relevance and accuracy of search responses. It is the search bot among Anthropic's three bots. ClaudeBot is the training bot.

Claude-User works differently. When a person asks Claude a question, Claude may access a site with the Claude-User agent. Anthropic describes it as support for Claude users. Anthropic documents the browsers it hosts for tasks (Claude-Agent) in a separate article, which this page does not cover.

How do I allow or block Claude-SearchBot and Claude-User?

Anthropic's instruction for any of its bots is to add a Disallow rule for that bot to the robots.txt file in your top-level directory, for every subdomain you want to opt out.

robots.txt: block Claude-SearchBot
User-agent: Claude-SearchBot
Disallow: /
robots.txt: block Claude-User
User-agent: Claude-User
Disallow: /

To allow Claude search and Claude user requests while opting out of training:

robots.txt: allow search bots, block training
User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

Anthropic recommends robots.txt over IP blocking. It says IP blocking may not work correctly or persistently guarantee an opt-out, because it stops its bots from reading your robots.txt.

Does blocking Claude-SearchBot affect Claude search results and citations?

Yes, by Anthropic's account. Disabling Claude-SearchBot prevents Anthropic from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.

Disabling Claude-User prevents Anthropic from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search. Anthropic uses the words "may reduce" in both cases. It does not state that a block removes you, and it does not say which search providers feed Claude.

How do I check whether my site blocks Claude-SearchBot?

Use the AI crawler checker above, preselected for Claude-SearchBot. It reads your robots.txt and shows the group and rule that match your page path for Claude-SearchBot and Claude-User. The checker does not request your page with these two user agents. Its firewall test uses a browser, GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. If the ClaudeBot request returns a 403, a rule may also catch the other Anthropic bots, and your logs will confirm it.

In your logs, search for Claude-SearchBot and Claude-User. Compare source IPs with bots.json.

What are the common Claude-SearchBot mistakes?

  • Blocking ClaudeBot and thinking search is blocked too. Each Anthropic bot has its own token. Set all three on purpose.
  • Blocking all three because the names look alike. That removes you from Claude search and user-directed fetches.
  • A firewall 403 while robots.txt allows the bot. See AI crawlers blocked by a firewall.
  • A wildcard group. User-agent: * with Disallow: / blocks any Anthropic bot that has no group of its own. See robots.txt blocks AI crawlers.
  • Rules on one hostname only. Anthropic asks you to repeat the rule for every subdomain.

Claude-SearchBot FAQ

What is Claude-SearchBot?

Claude-SearchBot is an Anthropic crawler that navigates the web to improve search result quality for Claude users.

How do I block Claude-SearchBot with robots.txt?

Add User-agent: Claude-SearchBot and Disallow: / to the robots.txt file of each hostname. Anthropic warns that blocking by IP address alone may not work.

What is Claude-User?

Claude-User is the agent Claude uses to access a website when a person asks Claude a question. It has its own robots.txt token.

Is Claude-SearchBot used for training?

Anthropic assigns training to ClaudeBot. It describes Claude-SearchBot as a bot that improves search result quality.

Does blocking Claude-SearchBot lower my visibility in Claude?

Anthropic says it may reduce your site's visibility and accuracy in user search results.

Sources