Skip to content
SeenLens

Cloudflare blocking AI crawlers: how to fix it

A firewall blocks AI crawlers when it answers their requests with a challenge page or an HTTP 403. The cause is a bot rule that treats AI crawlers as unwanted traffic. Allowing the crawlers you want lets them read your page. The change takes 10 to 30 minutes, and you check the result in your firewall logs.

By SeenLens · Last updated

Free, no sign-up. We read robots.txt and request the page with a browser and four crawler user agents. We do not store your page text.

bot-wall: does a firewall challenge plain clients?

The check looks at the answer to the plain GET request for your page. It fails when the answer looks like a bot wall:

  • a cf-mitigated: challenge header, or an x-amzn-waf-action header with the value challenge or captcha;
  • a small page with a challenge marker from Cloudflare, DataDome, Imperva or PerimeterX;
  • HTTP 403, 429 or 503 with a body under 5 KB, or HTTP 202 with a small body.

The check passes when the page answers without a challenge. It reports "not tested" when the page returns an error but shows no bot wall signature.

curl
curl -sS -D - -o /dev/null -A "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36 SeenLensBot/0.1" https://example.com/your-page

Look for status 403, 429 or 503, or for the cf-mitigated header in the output.

waf-ua-matrix: do AI user agents get the same answer as a browser?

This check requests the page with a Chrome user agent as a baseline. It repeats the request with eight AI user agents: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. An agent counts as different when its status differs from the baseline or its answer is a challenge (status 401, 403, 429 or 503, a cf-mitigated header, or a small page with a challenge marker).

The probe is switched off in scans at rules version 0.3, so the SeenLens report lists this check as not tested. The free AI crawler checker runs the comparison with the browser, GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot user agents. The result is an observation. A copied user agent does not come from the crawler's real IP range, so a firewall that verifies bots by IP may treat it differently from the real crawler.

How to find the rule that blocks AI crawlers

  • Cloudflare: open Security, Analytics, Events. Challenged requests show the product that acted, such as Bot Fight Mode, in the Service column.
  • Nginx or Apache: search the configuration for the crawler names. A rule such as if ($http_user_agent ~* "GPTBot|ClaudeBot") { return 403; } blocks them in Nginx. RewriteCond %{HTTP_USER_AGENT} GPTBot [NC] with RewriteRule .* - [F] does it in Apache.
  • WordPress and AWS WAF: read the block log of the security plugin, host firewall or web ACL.

How to allow AI crawlers in Cloudflare

These steps follow the Cloudflare documentation checked on 2026-11-06. The AI crawler policy page shows an update date of 2 November 2026.

Step 1: check the AI crawler access policies

  1. Open the Cloudflare dashboard and go to Security, Settings. Filter by Bot traffic.
  2. Select AI Crawler and Agent access policies.
  3. Set Search and Agent to Allow (do not block). Set Training to the choice you want. Cloudflare sets all three to Allow for new domains.
  4. Select Save.

Step 2: check single crawlers

  1. Go to AI Crawl Control and open the Crawlers tab.
  2. Find the crawler, such as OAI-SearchBot or PerplexityBot.
  3. Set its Action to Allow. A Block action creates a WAF custom rule on your zone, so also check your custom rules.

Step 3: handle Bot Fight Mode

Bot Fight Mode challenges traffic that matches known bot patterns, and WAF custom rules cannot skip it. Turn it off under Security, Settings, Bot traffic, or move to Super Bot Fight Mode (Pro, Business and Enterprise plans), where a custom rule with the Skip action bypasses it.

Step 4: add a Skip rule for the crawlers you trust

OpenAI and Perplexity publish the IP ranges of their crawlers. Perplexity advises matching the user agent and its IP list, using the Skip action, and placing the rule above any rule that blocks or challenges. Fill the address list from the published JSON files, for example https://openai.com/searchbot.json.

Cloudflare custom rule expression (addresses are examples)
(http.user_agent contains "OAI-SearchBot" and ip.src in {203.0.113.0/24})
or
(http.user_agent contains "PerplexityBot" and ip.src in {198.51.100.0/24})

The Boolean field cf.client.bot is true for requests from a known good bot. Use it only when Cloudflare lists the crawler as a verified bot.

How to verify the fix

  1. Run the curl command from the bot-wall section. Expect status 200 and no cf-mitigated header.
  2. Run the AI crawler checker. The status code and size for each crawler should match the browser row.
  3. When real crawler traffic arrives, open Security, Analytics, Events. OAI-SearchBot and PerplexityBot requests should show no challenge or block.

Run the free SeenLens scan on your page

Should I block AI crawlers on Cloudflare?

It depends on the crawler. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers. Blocking a training crawler such as GPTBot keeps your content out of model training. Decide per crawler, then write the same choice into robots.txt.

How do I turn off the Cloudflare block for AI crawlers?

Go to Security, Settings, filter by Bot traffic, open AI Crawler and Agent access policies and set the actions to Allow. Then check the Crawlers tab in AI Crawl Control for single crawlers that you blocked, and check your WAF custom rules.

Why does GPTBot get a 403 from Cloudflare?

A block action in AI Crawl Control or in the access policies, a custom rule that matches the user agent, or Bot Fight Mode can return the 403 or a challenge. Security, Analytics, Events shows which one acted.

When can you ignore it?

Ignore a difference for crawlers you chose to block, such as training crawlers. Fix it when the blocked crawler is a search crawler you want to appear in.

Sources