--- title: "Fix: Cloudflare Blocking AI Crawlers (GPTBot 403)" description: "A firewall or Cloudflare bot rule can answer AI crawlers with a challenge or 403. See how to find the rule, allow the crawlers you want and verify the fix." url: https://seenlens.com/fixes/ai-crawlers-blocked-by-waf last_updated: 2026-11-06 --- # Cloudflare blocking AI crawlers: how to fix it A firewall blocks AI crawlers when it answers their requests with a challenge page or an HTTP 403. The cause is a bot rule that treats AI crawlers as unwanted traffic. Allowing the crawlers you want lets them read your page. The change takes 10 to 30 minutes, and you check the result in your firewall logs. By SeenLens. Last updated: 2026-11-06 ## bot-wall: does a firewall challenge plain clients? The check looks at the answer to the plain GET request for your page. It fails when the answer looks like a bot wall: - a `cf-mitigated: challenge` header, or an `x-amzn-waf-action` header with the value `challenge` or `captcha`; - a small page with a challenge marker from Cloudflare, DataDome, Imperva or PerimeterX; - HTTP 403, 429 or 503 with a body under 5 KB, or HTTP 202 with a small body. The check passes when the page answers without a challenge. It reports "not tested" when the page returns an error but shows no bot wall signature. ```text curl -sS -D - -o /dev/null -A "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36 SeenLensBot/0.1" https://example.com/your-page ``` Look for status 403, 429 or 503, or for the `cf-mitigated` header in the output. ## waf-ua-matrix: do AI user agents get the same answer as a browser? This check requests the page with a Chrome user agent as a baseline. It repeats the request with eight AI user agents: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. An agent counts as different when its status differs from the baseline or its answer is a challenge (status 401, 403, 429 or 503, a `cf-mitigated` header, or a small page with a challenge marker). The probe is switched off in scans at rules version 0.3, so the SeenLens report lists this check as not tested. The free [AI crawler checker](https://seenlens.com/tools/ai-crawler-checker) runs the comparison with the browser, GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot user agents. The result is an observation. A copied user agent does not come from the crawler's real IP range, so a firewall that verifies bots by IP may treat it differently from the real crawler. ## How to find the rule that blocks AI crawlers - **Cloudflare:** open Security, Analytics, Events. Challenged requests show the product that acted, such as Bot Fight Mode, in the Service column. - **Nginx or Apache:** search the configuration for the crawler names. A rule such as `if ($http_user_agent ~* "GPTBot|ClaudeBot") { return 403; }` blocks them in Nginx. `RewriteCond %{HTTP_USER_AGENT} GPTBot [NC]` with `RewriteRule .* - [F]` does it in Apache. - **WordPress and AWS WAF:** read the block log of the security plugin, host firewall or web ACL. ## How to allow AI crawlers in Cloudflare These steps follow the Cloudflare documentation checked on 2026-11-06. The AI crawler policy page shows an update date of 2 November 2026. ### Step 1: check the AI crawler access policies 1. Open the Cloudflare dashboard and go to Security, Settings. Filter by Bot traffic. 2. Select AI Crawler and Agent access policies. 3. Set Search and Agent to Allow (do not block). Set Training to the choice you want. Cloudflare sets all three to Allow for new domains. 4. Select Save. ### Step 2: check single crawlers 1. Go to AI Crawl Control and open the Crawlers tab. 2. Find the crawler, such as OAI-SearchBot or PerplexityBot. 3. Set its Action to Allow. A Block action creates a WAF custom rule on your zone, so also check your custom rules. ### Step 3: handle Bot Fight Mode Bot Fight Mode challenges traffic that matches known bot patterns, and WAF custom rules cannot skip it. Turn it off under Security, Settings, Bot traffic, or move to Super Bot Fight Mode (Pro, Business and Enterprise plans), where a custom rule with the Skip action bypasses it. ### Step 4: add a Skip rule for the crawlers you trust OpenAI and Perplexity publish the IP ranges of their crawlers. Perplexity advises matching the user agent and its IP list, using the Skip action, and placing the rule above any rule that blocks or challenges. Fill the address list from the published JSON files, for example `https://openai.com/searchbot.json`. ```text (http.user_agent contains "OAI-SearchBot" and ip.src in {203.0.113.0/24}) or (http.user_agent contains "PerplexityBot" and ip.src in {198.51.100.0/24}) ``` The Boolean field `cf.client.bot` is true for requests from a known good bot. Use it only when Cloudflare lists the crawler as a verified bot. ## How to verify the fix 1. Run the curl command from the bot-wall section. Expect status 200 and no `cf-mitigated` header. 2. Run the [AI crawler checker](https://seenlens.com/tools/ai-crawler-checker). The status code and size for each crawler should match the browser row. 3. When real crawler traffic arrives, open Security, Analytics, Events. OAI-SearchBot and PerplexityBot requests should show no challenge or block. [Run the free SeenLens scan on your page](https://seenlens.com/) ## Should I block AI crawlers on Cloudflare? It depends on the crawler. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers. Blocking a training crawler such as GPTBot keeps your content out of model training. Decide per crawler, then write the same choice into [robots.txt](https://seenlens.com/fixes/robots-txt-blocks-ai-crawlers). ### How do I turn off the Cloudflare block for AI crawlers? Go to Security, Settings, filter by Bot traffic, open AI Crawler and Agent access policies and set the actions to Allow. Then check the Crawlers tab in AI Crawl Control for single crawlers that you blocked, and check your WAF custom rules. ### Why does GPTBot get a 403 from Cloudflare? A block action in AI Crawl Control or in the access policies, a custom rule that matches the user agent, or Bot Fight Mode can return the 403 or a challenge. Security, Analytics, Events shows which one acted. ## When can you ignore it? Ignore a difference for crawlers you chose to block, such as training crawlers. Fix it when the blocked crawler is a search crawler you want to appear in. ## Related checks - [Crawlers cannot load the page](https://seenlens.com/fixes/ai-crawlers-cannot-load-page) - [robots.txt blocks AI crawlers](https://seenlens.com/fixes/robots-txt-blocks-ai-crawlers) - [OAI-SearchBot](https://seenlens.com/bots/oai-searchbot), [PerplexityBot](https://seenlens.com/bots/perplexitybot) and [GPTBot](https://seenlens.com/bots/gptbot) - [robots.txt for AI crawlers](https://seenlens.com/guides/robots-txt-for-ai-crawlers) ## Sources - [Cloudflare: AI Crawler and Agent access policies](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/) (dashboard path, actions, defaults, updated 2026-11-02) - [Cloudflare: Manage AI crawlers](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/) (Allow and Block actions, WAF rule created on Block, checked 2026-11-06) - [Cloudflare: Bot Fight Mode](https://developers.cloudflare.com/bots/get-started/bot-fight-mode/) (custom rules cannot skip it, checked 2026-11-06) - [Cloudflare: Super Bot Fight Mode](https://developers.cloudflare.com/bots/get-started/super-bot-fight-mode/) (plans, Skip action, checked 2026-11-06) - [Cloudflare: cf.client.bot field](https://developers.cloudflare.com/ruleset-engine/rules-language/fields/reference/cf.client.bot/) (Boolean for known good bots, checked 2026-11-06) - [Perplexity: crawlers](https://docs.perplexity.ai/guides/bots) (allow by user agent and IP list, Skip rule above blocking rules, checked 2026-11-06) - [OpenAI: crawlers and user agents](https://developers.openai.com/api/docs/bots) (OAI-SearchBot and IP list files, checked 2026-11-06)