Skip to content
SeenLens

robots.txt blocking ChatGPT: how to fix it

Your robots.txt blocks ChatGPT when a Disallow rule applies to OAI-SearchBot, either by name or through a User-agent: * group. The cause is usually a broad rule written to stop AI training. Allowing the search crawlers and blocking only the training crawlers fixes it. The edit takes 10 minutes. Crawlers pick it up in about a day.

By SeenLens · Last updated

Free, no sign-up. We read robots.txt and request the page with a browser and four crawler user agents. We do not store your page text.

robots-ai-groups: does robots.txt block AI search crawlers?

SeenLens parses your robots.txt by RFC 9309 rules. For each AI crawler token it uses the group that names the crawler. If no group names it, it uses the User-agent: * group. It then tests whether the path / is allowed, using the longest matching rule.

How the robots-ai-groups check scores robots.txt
ResultWhat SeenLens found
WarnOAI-SearchBot, Claude-SearchBot or PerplexityBot cannot read /
PassAt least one AI crawler has its own named group, and no search crawler is blocked
InfoNo named AI groups. All AI crawlers use the * group

The 13 tokens checked are GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Google-Agent, Applebot-Extended and CCBot.

The roles differ. OpenAI says OAI-SearchBot surfaces sites in ChatGPT search, and sites that opt out will not be shown in ChatGPT search answers. GPTBot is for training data. Anthropic uses ClaudeBot for training and Claude-SearchBot for search indexing. Perplexity says PerplexityBot surfaces sites in Perplexity search and is not used for model training. Blocking a training crawler does not remove you from search.

How to fix it

Add a named group for each search crawler and keep your training choice in separate groups. A named group wins over the * group, so a site wide Disallow: / under * no longer applies to them.

robots.txt
# AI search crawlers: allowed
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Training crawlers: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

# Everyone else
User-agent: *
Allow: /
  • Next.js: pass an array of rules to robots() in app/robots.ts, with userAgent, allow and disallow per rule (docs updated 2026-05-01).
  • Shopify: add the groups to a robots.txt.liquid template, outside the Liquid block that prints the default rules.
  • Cloudflare: the managed robots.txt can add Disallow: / for training crawlers. Under Security, Settings, AI Crawler and Agent access policies, set Training to Disallow and turn on Bot Preference Sync. Cloudflare says robots.txt is voluntary, and with Bot Preference Sync on, Disallow also blocks training crawlers at its edge.
  • WordPress, Nginx, Apache: edit the robots.txt file in the web root.

robots-user-agents: does robots.txt block agents that act for a user?

ChatGPT-User, Claude-User, Perplexity-User and Google-Agent fetch a page because a person asked an AI tool to open it. The check reports "info" when any of them cannot read /, and "pass" otherwise.

  • OpenAI says robots.txt rules may not apply to ChatGPT-User, because a user starts the action.
  • Perplexity says Perplexity-User generally ignores robots.txt.
  • Anthropic says its bots honor robots.txt, and that blocking Claude-User may reduce visibility in user directed web search.

A Disallow for these agents may not stop all of them, and it can cut you out of answers for the ones that obey it. To keep private content private, put it behind authentication. Remove the Disallow for these agents if you want people to be able to share your pages with an AI tool.

robots-search-engines: does robots.txt block Googlebot or Bingbot?

SeenLens tests the scanned path against the Googlebot and Bingbot rules in your robots.txt. It picks the group by RFC 9309 rules. Bingbot also follows an msnbot group when the file has no bingbot group. The check fails when either crawler cannot read the path. The evidence names the crawler, the group and the rule line. It passes when both crawlers may read the path, and when robots.txt answers 404 or 410, because a missing file sets no limits. It reports "not tested" when the robots.txt request fails or the server blocks the probe.

This matters because AI search answers build on search indexes. Google says a page must be "indexed and eligible to be shown in Google Search with a snippet" to appear as a supporting link in AI Overviews and AI Mode. A page that Googlebot cannot crawl is unlikely to meet that condition. Microsoft Copilot draws on Bing's index, so a Bingbot block limits Copilot in the same way. A rule written to stop AI training that also covers Googlebot or Bingbot removes you from search as well.

How to fix it

Find the rule that names Googlebot or Bingbot, or the User-agent: * group they fall back to, and remove or narrow its Disallow. Then keep the training crawlers blocked in their own groups.

robots.txt
# Search engines: allowed
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# Training crawlers: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

# Everyone else
User-agent: *
Allow: /

A Disallow on one path, such as /private/, is fine. The check fails only when the scanned path itself is blocked.

Cloudflare adds a managed block to robots.txt

Cloudflare can put a managed block in front of your own robots.txt. The block starts with # BEGIN Cloudflare Managed content and ends with # END Cloudflare Managed Content. SeenLens detects the block and tells you in the report when the deciding rule sits inside it. Editing your own file does not change the managed block.

Cloudflare calls the feature the robots.txt setting, and it runs through Bot Preference Sync. To change it, open the Cloudflare dashboard, go to Security Settings, filter by Bot traffic, and open AI Crawler and Agent access policies. There you turn Bot Preference Sync on or off and set the Training policy. Cloudflare says that when your origin serves a robots.txt, it prepends the managed content and serves both as one response (documentation updated 2026-11-02, checked 2026-11-07).

How to confirm the problem

curl
curl -s https://example.com/robots.txt | grep -n -i -E "user-agent|disallow|allow"

Look for Disallow: / under User-agent: * or under an AI crawler name. The embedded AI crawler checker shows the group and rule line for each crawler.

How to verify the fix

  1. Fetch the file again and confirm your edit is live.
  2. Run the AI crawler checker. OAI-SearchBot, Claude-SearchBot and PerplexityBot should show Allowed.
  3. Run the SeenLens scan again. The robots-search-engines check should pass.
  4. Open the robots.txt report in Google Search Console (under Settings). It shows the file Google fetched, when it fetched it and any errors. Use the URL Inspection tool to test one URL.
  5. Open the robots.txt Tester in Bing Webmaster Tools, enter a URL and confirm Bingbot may fetch it.
  6. Wait. OpenAI says robots.txt changes can take about 24 hours to take effect for search results.

Run the free SeenLens scan on your page

When can you ignore it?

Ignore a warning for a search crawler only if you decided to stay out of that engine. A deliberate block of training crawlers is a valid choice and does not trigger the warning.

Sources