AI crawler checker
The AI crawler checker shows which AI crawlers your robots.txt lets fetch a page, and whether your firewall serves them the same page as a browser. We fetch only your robots.txt and the page you enter. We do not store your page text. The result shows the rules crawlers read, not proof that a crawler visits.
By SeenLens · Last updated
Free, no sign-up. We read robots.txt and request the page with a browser and four crawler user agents. We do not store your page text.
What does the AI crawler checker test?
The checker runs three tests on the URL you enter. First, it reads robots.txt and evaluates the page path for 14 crawlers, with the matched group and rule line for each. Second, it reads any Content-Signal lines in robots.txt. Third, it requests the page five times: once with a browser user agent, and once each with the GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot user agents. It compares status codes and response sizes. A difference points to a firewall that treats AI bots differently.
| Crawler | Operator | Role |
|---|---|---|
| GPTBot | OpenAI | Training crawler |
| OAI-SearchBot | OpenAI | Feeds ChatGPT search |
| ChatGPT-User | OpenAI | User-triggered fetch |
| ClaudeBot | Anthropic | Training crawler |
| Claude-SearchBot | Anthropic | Feeds Claude search |
| Claude-User | Anthropic | User-triggered fetch |
| PerplexityBot | Perplexity | Feeds Perplexity search |
| Perplexity-User | Perplexity | User-triggered fetch |
| Google-Extended | Token for AI training and grounding, not Search | |
| Applebot-Extended | Apple | Token for Apple model training; does not crawl |
| CCBot | Common Crawl | Open web crawl archive |
| Bytespider | ByteDance | ByteDance crawler |
| Googlebot | Google Search crawler | |
| Bingbot | Microsoft | Bing Search crawler |
Google-Extended and Applebot-Extended are robots.txt tokens only. No request carries these names, so the checker reads their rules and does not fetch the page as them.
How do I read the result?
- Allowed or blocked. Each crawler shows the group and the rule line behind its result. Per RFC 9309, a crawler uses the group that names it, else the
*group, and the most specific matching rule wins. - Training and search crawlers differ. Blocking GPTBot, ClaudeBot or CCBot opts out of training. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot can remove your site from that engine's answers.
- Same status and size for all five requests. Your firewall treats AI user agents like a browser. This is the result you want.
- A 403, 429 or 503 for an AI user agent, with 200 for the browser. A firewall rule blocks the bot even when robots.txt allows it.
- Status 200 but a much smaller response. The AI user agent may receive a challenge page or an empty shell instead of your content.
- No robots.txt. A 4xx response means a crawler may access everything. A 5xx response means a compliant crawler must assume the whole site is disallowed.
A robots.txt rule is a request. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. To keep a path private, use authentication.
How do I fix a failed check?
- robots.txt blocks AI crawlers: allow the search crawlers and block only the training crawlers you want to opt out.
- Firewall or WAF blocks AI crawlers: allow verified crawlers by user agent and published IP range.
- robots.txt is missing: publish a plain-text file with status 200.
- AI crawlers cannot load the page: fix errors that a plain HTTP request meets.
- Add Content Signals: state your preference for search, AI input and training.
This robots.txt allows the search crawlers and opts out of training:
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /The robots.txt guide for AI crawlers explains each choice. OpenAI and Perplexity say a robots.txt change can take about 24 hours to apply.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
No. OpenAI says GPTBot crawls content that may train its models. It says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Block GPTBot to opt out of training, and allow OAI-SearchBot to stay in ChatGPT search. See the GPTBot page and the OAI-SearchBot page.
Do all AI crawlers follow robots.txt?
The training and search crawlers follow it, according to their operators. Agents that act for a user may not. See the note under the result guide above.
Why does a crawler get a different response than my browser?
A firewall or bot-protection rule treats the AI user agent differently. See AI crawlers blocked by a WAF.
Should I block AI crawlers?
It depends on your goal. To stay in AI search answers, allow the search crawlers. Anthropic says blocking Claude-SearchBot may reduce your visibility in search results. To keep content out of model training, block the training crawlers in their own groups.
Method and sources
SeenLensBot fetches robots.txt and the page. Read the matched group and rule line against RFC 9309. Crawler roles come from each operator's documentation. Last reviewed 2026-11-06. See how SeenLens works.
Sources
- OpenAI: Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, checked 2026-11-06)
- Anthropic: Does Anthropic crawl data from the web (Claude bots, checked 2026-11-06)
- Perplexity: Bots (PerplexityBot and Perplexity-User, checked 2026-11-06)
- Google: AI features and your website (Google-Extended scope, checked 2026-11-06)
- Apple: About Applebot (Applebot-Extended, checked 2026-11-06)
- Common Crawl: CCBot (CCBot token, checked 2026-11-06)
- RFC 9309: Robots Exclusion Protocol (matching rules, 4xx and 5xx, checked 2026-11-06)