What is CCBot?
CCBot is the web crawler of Common Crawl, a non-profit that publishes an open repository of web crawl data. Its user agent is CCBot/2.0 (https://commoncrawl.org/faq/) and its robots.txt token is CCBot. Common Crawl says CCBot checks robots.txt first and obeys Crawl-delay. Many large language models use Common Crawl data. Verified 2026-11-06.
By SeenLens · Last updated
Free, no sign-up. We read robots.txt and request the page with a browser and four crawler user agents. We do not store your page text.
CCBot fact sheet
| Fact | Value |
|---|---|
| Operator | Common Crawl, a non-profit foundation |
| Purpose | Open dataset. CCBot builds a repository of web crawl data that anyone can access and analyze. It is not a search engine or an AI product. |
| User agent string | CCBot/2.0 (https://commoncrawl.org/faq/) |
| robots.txt token | CCBot |
| Published IP list | index.commoncrawl.org/ccbot.json (IPv4 and IPv6) |
| Obeys robots.txt | Yes. Common Crawl's FAQ says CCBot checks robots.txt first and obeys the Crawl-delay parameter (Common Crawl FAQ). |
| Related bots | GPTBot, ClaudeBot and other AI training crawlers |
| Last verified | 2026-11-06 |
What does CCBot do?
CCBot crawls the public web and Common Crawl publishes the results as open data. Common Crawl says free access to the data supports collaboration and research. The data goes to whoever downloads it. Common Crawl states in its opt-out protocols article that many large language models use Common Crawl data as a large part of their text corpora, so a CCBot rule can affect AI training indirectly.
Common Crawl warns that some crawlers falsely identify as CCBot and recommends verifying user agent strings. CCBot now runs on dedicated IP address ranges with reverse DNS, except over IPv6, where reverse DNS is not yet supported.
To verify a request, run a reverse lookup on the source IP. The hostname should end in crawl.commoncrawl.org. Then run a forward lookup on that hostname and confirm it returns the same IP.
host 18.97.14.84
84.14.97.18.in-addr.arpa domain name pointer 18-97-14-84.crawl.commoncrawl.org.
host 18-97-14-84.crawl.commoncrawl.org
18-97-14-84.crawl.commoncrawl.org has address 18.97.14.84How do I block CCBot in robots.txt?
Common Crawl's instruction is to add these lines to your robots.txt:
User-agent: CCBot
Disallow: /To slow CCBot down instead, use a Crawl-delay value. Common Crawl says it obeys the Crawl-delay parameter:
User-agent: CCBot
Crawl-delay: 5To block CCBot and the AI training crawlers while keeping AI search bots:
User-agent: CCBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Common Crawl also runs an Opt-out Registry, described on its CCBot page and in a blog post. It says CCBot rechecks robots.txt periodically.
Does blocking CCBot affect AI search or citations?
Common Crawl is a dataset publisher, not a search engine, and its pages describe no search feature that a CCBot block would remove you from. AI search products such as ChatGPT search, Claude search and Perplexity use their own bots, which have separate tokens. See OAI-SearchBot, Claude-SearchBot and PerplexityBot.
The Common Crawl pages checked on 2026-11-06 do not say whether a block removes content from crawls it already published. A CCBot block applies to future crawling.
How do I check whether my site blocks CCBot?
Use the AI crawler checker above, preselected for CCBot. It reads your robots.txt and shows the group and rule line that match your page path for CCBot. Its firewall test requests the page as a browser, GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, and does not use the CCBot user agent.
In your logs, search for CCBot and verify addresses with the reverse DNS steps above or the ccbot.json file.
What are the common CCBot mistakes?
- Trusting the user agent alone. Common Crawl says other crawlers falsely use the CCBot name. Verify the IP before you act on log counts.
- Blocking GPTBot but not CCBot. An opt-out from one training crawler does not stop the dataset that other companies train on.
- A wildcard group.
User-agent: *withDisallow: /already blocks CCBot, along with every other bot that has no group. See robots.txt blocks AI crawlers. - Expecting old crawls to vanish. Common Crawl's pages do not say a block removes content that earlier crawls already contain.
- Firewall rules that block CCBot but not robots.txt. If you deny CCBot at the firewall, it also cannot read your robots.txt. See AI crawlers blocked by a firewall for the general pattern.
CCBot FAQ
What is the CCBot user agent string?
Common Crawl publishes CCBot/2.0 (https://commoncrawl.org/faq/).
How do I block CCBot?
Add User-agent: CCBot and Disallow: / to your robots.txt file.
Does CCBot obey robots.txt?
Yes. Common Crawl says CCBot checks robots.txt first and obeys the Crawl-delay parameter.
Is CCBot used for AI training?
Common Crawl publishes open data. Its blog says many large language models use Common Crawl data as a large part of their corpora. CCBot itself only collects the data.
How do I verify a request really comes from CCBot?
Do a reverse DNS lookup on the IP, check that the host is in the crawl.commoncrawl.org domain, then do a forward lookup to confirm the same IP. You can also compare the IP with ccbot.json.
Sources
- Common Crawl: CCBot (user agent string, robots.txt block, IP list, reverse DNS verification, spoofing warning; checked 2026-11-06)
- Common Crawl: FAQ (robots.txt check and Crawl-delay; checked 2026-11-06)
- Common Crawl: Balancing Discovery and Privacy, opt-out protocols (CCBot and the Robots Exclusion Protocol; LLM use of Common Crawl data; checked 2026-11-06)