Skip to content
SeenLens

Robots.txt for AI crawlers: what to allow and what to block

Robots.txt tells AI crawlers which parts of your site they may fetch. Each AI company runs separate crawlers for training, search and user requests, so you can block training and still appear in AI search. This guide lists the crawler tokens, gives copy-paste recipes and shows how to test them with the free checker.

By SeenLens · Last updated

Free, no sign-up. We read robots.txt and request the page with a browser and four crawler user agents. We do not store your page text.

Which AI crawlers can robots.txt control?

Most AI companies run more than one crawler. One crawler gathers training data. Another builds the search index. A third fetches a page when a user asks for it. You control each one by its robots.txt token, which is the name on the User-agent: line. The table below uses each operator's own documentation, checked on 2026-11-06.

AI crawler tokens, operators, purposes and what blocking changes (operator documentation, checked 2026-11-06)
TokenOperatorPurposeWhat blocking changes
GPTBotOpenAITrainingTells OpenAI not to use your content to train its foundation models. It does not control ChatGPT search.
OAI-SearchBotOpenAIChatGPT searchYour site is not shown in ChatGPT search answers. It can still appear as a navigational link.
ChatGPT-UserOpenAIFetches a page when a user asksOpenAI says robots.txt rules may not apply, because a user starts the request.
ClaudeBotAnthropicTrainingSignals that your future content should be left out of training data sets.
Claude-SearchBotAnthropicSearch qualityStops indexing for search. Anthropic says this may reduce your visibility in Claude search results.
Claude-UserAnthropicFetches a page when a user asksStops retrieval for user questions. Anthropic says this may reduce visibility in user-directed search.
PerplexityBotPerplexityPerplexity searchPerplexity recommends allowing it so your site can appear in its search results. It is not used for foundation model training.
Perplexity-UserPerplexityFetches a page when a user asksPerplexity says it generally ignores robots.txt, because a user requested the fetch.
Google-ExtendedGoogleGemini training and groundingNo effect on Google Search inclusion or ranking. It is a robots.txt token only and has no user agent string.
Applebot-ExtendedAppleTraining opt-outOpts your content out of training Apple's foundation models. It does not crawl, and pages can stay in Apple search results.
CCBotCommon CrawlOpen web archiveStops Common Crawl from crawling your site. The page does not say whether data already collected is removed.

Googlebot and Bingbot are search crawlers, not AI-only crawlers. Google says its Googlebot robots.txt rules are the crawl control for Search, and AI Overviews and AI Mode are part of Search. If you block Googlebot, you leave Google Search. Read why a site may not show in AI Overviews before you change those rules.

Should I block AI crawlers?

It depends on what you want. Blocking a training crawler and blocking a search crawler have different results, so decide per purpose. The table shows the trade for each goal.

Decision table: what to allow or block for each goal
Your goalAllowBlockCost or limit
Appear in ChatGPT search answersOAI-SearchBotNothingAllowing does not guarantee a mention. ChatGPT cited only about half of the URLs it retrieved in an Ahrefs study of 1.4 million prompts (2026-04-15).
Appear in Claude and Perplexity searchClaude-SearchBot, PerplexityBotNothingAI referrals are small today: 0.32% of web traffic in a SE Ranking study of 101,574 sites (2026-06-18).
Keep content out of model trainingThe search crawlersGPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBotRobots.txt is a request. It does not remove data collected before the rule, and it cannot stop a crawler that ignores it.
Keep content out of every AI productNothingAll tokens in the table aboveYour site is not shown in AI search answers. Googlebot still reads your pages for Google Search, including AI features.
Limit how much text Google shows in AI featuresGooglebotNothing in robots.txtUse nosnippet or max-snippet. Google says these also limit direct input to AI Overviews and AI Mode.

A site that sells through search and wants new customers usually allows the search crawlers. A publisher that sells its text may block the training crawlers and accept the loss of some AI mentions. Both choices are valid. SeenLens scores pages on readiness for AI search, so its checks treat a blocked search crawler as a problem and a blocked training crawler as your decision.

Robots.txt recipes you can copy

Put the file at the root of each host, for example https://example.com/robots.txt. A subdomain needs its own file. Anthropic gives the same instruction for its crawlers. Copy one recipe and edit the paths.

Recipe 1: allow everything

A site with no rule for a crawler is open to that crawler. This file says so in plain words. Use it when you want AI search and AI training to reach all pages.

robots.txt: allow all crawlers
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Recipe 2: allow AI search and block training

This is the most common choice. The search crawlers get access. The training tokens are disallowed. OpenAI gives the same pattern for OAI-SearchBot and GPTBot in its bots documentation.

robots.txt: allow AI search, block training
# AI search crawlers: allowed
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Training crawlers and training opt-out tokens: blocked
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Everyone else
User-agent: *
Allow: /

ChatGPT-User, Claude-User and Perplexity-User have no group here. They fall to the * group. Their operators say user-requested fetches may not follow robots.txt, so a rule for them may change nothing.

Recipe 3: block all AI crawlers

This recipe disallows every AI token in the table. It keeps normal search crawlers such as Googlebot and Bingbot out of the rule. Your site is no longer shown in ChatGPT search answers, and Claude and Perplexity may show it less.

robots.txt: block named AI crawlers
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Several User-agent: lines in a row share the rules that follow them. RFC 9309 allows this form. It keeps the file short.

How do robots.txt groups match a crawler?

RFC 9309 defines the Robots Exclusion Protocol. Crawlers from the AI companies above document that they read robots.txt. The rules for matching are short.

  1. A crawler looks for a group that names its token. The match ignores letter case. User-agent: gptbot matches GPTBot.
  2. If several groups name the same token, the crawler combines their rules into one group.
  3. **If no group names the token, the crawler uses the * group.** If there is no * group, nothing is restricted.
  4. **A crawler that finds its own group ignores the * group.** A rule in * does not reach a crawler that has its own group.
  5. Inside a group, the longest matching path wins. The spec calls this the most specific match.
  6. If an Allow rule and a Disallow rule match equally, the Allow rule should win. The spec says SHOULD here, so check a crawler's own documentation if the case matters.
Longest path wins
User-agent: GPTBot
Disallow: /docs/
Allow: /docs/public/

With this group, GPTBot may fetch /docs/public/guide.html because the Allow path is longer. GPTBot may not fetch /docs/internal.html.

The file itself has rules too. RFC 9309 says a 4xx response means the file is absent, so the crawler may fetch anything. A 5xx response means the file is unreachable, and the crawler must assume complete disallow. Crawlers may cache the file for up to 24 hours. Crawlers must parse at least the first 500 KiB, so keep the file small.

What are the most common robots.txt mistakes for AI crawlers?

  • **A blanket User-agent: * with Disallow: /.** This blocks every AI crawler that has no group of its own. Add a group with Allow: / for each crawler you want, such as OAI-SearchBot.
  • A wrong token. GPT-Bot does not match GPTBot. Copy tokens from the operator's page or from the table above.
  • A file on the wrong host. www.example.com and example.com are different hosts. Each needs its own robots.txt.
  • A robots.txt that errors. A 5xx response makes crawlers assume a full block, while a 404 opens the whole site. Check that /robots.txt returns 200 with the text you expect.
  • A managed robots.txt that adds rules you did not choose. Cloudflare's managed robots.txt adds Disallow rules for named crawlers: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. Read the live file after you switch it on.
  • Blocking by IP address. Anthropic advises against it. An IP block can stop the crawler from reading your robots.txt, so it cannot see your rules.
  • Blocking Googlebot to stop AI Overviews. This removes you from Google Search. Use snippet controls instead.
  • Treating robots.txt as security. RFC 9309 says the rules are not a form of access authorization. Anyone can ignore them. Protect private pages with a login.

The fix guide robots.txt blocks AI crawlers covers two SeenLens checks for these cases. If your site has no robots.txt at all, read the fix guide robots.txt is missing.

Can a firewall block AI crawlers when robots.txt allows them?

Yes. Robots.txt is a request that a crawler reads. A firewall, CDN or bot manager acts earlier: it can refuse the request by user agent, by IP address or with a challenge page. A crawler that is blocked this way never sees a page, even when robots.txt says Allow.

OpenAI recommends that you allow OAI-SearchBot and allow requests from its published IP ranges. OpenAI lists them at openai.com/searchbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json. Perplexity and Anthropic also publish IP lists. Use them in an allow rule, not in a guess based on the user agent string, because anyone can fake a user agent.

To detect a firewall block, request the same page with a browser user agent and with each AI user agent, and compare the status code and the size of the response. A 403, a 429 or a much smaller body for the AI user agent points to a block. The checker below runs this comparison. The fix guide AI crawlers blocked by a firewall explains how to fix a block.

Do AI crawlers obey robots.txt?

The operators say yes for their automatic crawlers and no or not always for user-requested fetchers. SeenLens has not tested compliance. These statements come from the operators' pages.

  • OpenAI: robots.txt controls GPTBot and OAI-SearchBot. For ChatGPT-User, OpenAI says robots.txt rules may not apply.
  • Anthropic: its page tells site owners how to block ClaudeBot, Claude-SearchBot and Claude-User with robots.txt. Anthropic supports Crawl-delay for ClaudeBot.
  • Perplexity: PerplexityBot follows robots.txt. Perplexity-User generally ignores it.
  • Google: common crawlers always respect robots.txt for automatic crawls. Google treats user-triggered fetchers separately.
  • Apple and Common Crawl: both document robots.txt control through Applebot-Extended and CCBot.

A newer option is a usage preference line such as Content-Signal. No operator above documents that its crawlers read it today. Read Content Signals in robots.txt for the details.

How do I test my robots.txt for AI crawlers?

  1. Open https://your-site/robots.txt in a browser. Confirm that it returns text and that your rules are in it.
  2. Enter your page URL in the checker above. It reads the robots.txt and evaluates the page path for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Bytespider, Googlebot and Bingbot.
  3. For each crawler, read the matched group and the rule line. Confirm that the result is the one you chose.
  4. Read the firewall comparison. It requests the page with a browser user agent and with the GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot user agents. Look for different status codes or sizes.
  5. Change the file, then wait. OpenAI says it can take about 24 hours for its search systems to adjust after a robots.txt update.
  6. Check your server logs after a day. Look for the crawler user agents and confirm that the status codes are 200.

The AI crawler checker is free and needs no account. To see all 38 SeenLens checks for one page, run the full scan.

FAQ

How do I block AI training but keep ChatGPT search?

Allow OAI-SearchBot and disallow GPTBot. OpenAI documents this pattern. Use Recipe 2 above, which also covers Claude, Perplexity, Google-Extended, Applebot-Extended and CCBot.

Does blocking GPTBot remove my site from ChatGPT?

No. OpenAI says GPTBot is for training. OAI-SearchBot controls whether your site appears in ChatGPT search answers. Read OpenAI crawlers explained.

Which AI crawlers respect robots.txt?

The operators document robots.txt control for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google's crawlers, Applebot-Extended and CCBot. User-requested fetchers such as ChatGPT-User and Perplexity-User may not follow it.

How long does a robots.txt change take to work?

OpenAI says its search systems can take about 24 hours to adjust. RFC 9309 lets crawlers cache the file for up to 24 hours. Plan for one day.

Do I need llms.txt to control AI crawlers?

No. Robots.txt controls crawler access. Google says Google Search does not use llms.txt. See llms.txt versus robots.txt.

Sources