Skip to content
SeenLens

What is GPTBot?

GPTBot is the web crawler OpenAI uses to collect content that may be used to train its generative AI foundation models. Its robots.txt token is GPTBot. OpenAI documents robots.txt as the control: disallowing GPTBot signals that a site's content should not be used for training. OpenAI publishes the user agent and an IP list. Verified 2026-11-06.

By SeenLens · Last updated

Free, no sign-up. We read robots.txt and request the page with a browser and four crawler user agents. We do not store your page text.

GPTBot fact sheet

GPTBot facts, from OpenAI's crawler documentation
FactValue
OperatorOpenAI
PurposeTraining. GPTBot crawls content that may be used to train OpenAI's generative AI foundation models.
User agent stringMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot (OpenAI marks this as an example; the version number may change)
robots.txt tokenGPTBot
Published IP listopenai.com/gptbot.json
Obeys robots.txtYes, as OpenAI documents it. OpenAI lists GPTBot as a robots.txt tag and states that disallowing it indicates content should not be used in training (OpenAI).
Related botsOAI-SearchBot (search), ChatGPT-User (user requests), OAI-AdsBot (ad landing page checks)
Last verified2026-11-06

What does GPTBot do?

GPTBot visits public web pages and collects their content. OpenAI says this content may be used to train its generative AI foundation models. GPTBot does not answer questions for ChatGPT users and it does not decide whether a page shows up in ChatGPT search. Other bots do those jobs.

When GPTBot fetches a robots.txt file, OpenAI says it may add a robots.txt marker to the user agent string. The marker lets you tell robots.txt requests from page requests in logs that do not show the path.

Any client can send the string GPTBot. To confirm a request is real, compare its IP address with the prefixes in OpenAI's gptbot.json file. OpenAI updates that list, so read the current file instead of copying addresses once.

How do I allow or block GPTBot in robots.txt?

To block GPTBot from the whole site, add this group to the robots.txt file at the root of your host:

robots.txt: block GPTBot
User-agent: GPTBot
Disallow: /

To allow GPTBot everywhere, either leave it out of robots.txt (if no wildcard group blocks it) or name it with an empty rule:

robots.txt: allow GPTBot
User-agent: GPTBot
Disallow:

To block only part of a site, disallow those paths:

robots.txt: block GPTBot from two folders
User-agent: GPTBot
Disallow: /members/
Disallow: /drafts/

OpenAI states that each of its robots.txt settings is independent. This group lets your pages appear in ChatGPT search and asks OpenAI not to use them for training:

robots.txt: stay in ChatGPT search, opt out of training
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Does blocking GPTBot remove my site from ChatGPT search or citations?

OpenAI's documentation says no. It gives this exact use case: a webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training.

Presence in ChatGPT search depends on OAI-SearchBot. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. OpenAI adds that if you allow both bots, it may use the results of one crawl for both purposes. OpenAI's page does not say whether a GPTBot block removes content that OpenAI already collected.

How do I check whether my site blocks GPTBot?

Use the AI crawler checker at the top of this page. It is preselected for GPTBot. It reads your robots.txt, shows the group and rule line that match your page path, and then requests the page with the GPTBot user agent. A status code or response size that differs from the browser request points to a firewall block.

You can also search your server logs for GPTBot and compare the source IPs with gptbot.json.

What are the common GPTBot mistakes?

  • Firewall returns 403 while robots.txt allows GPTBot. robots.txt is a request. A firewall or CDN rule acts first. If it denies the GPTBot user agent or OpenAI's IP addresses, GPTBot never reaches your content. See AI crawlers blocked by a firewall.
  • A wildcard group blocks everything. User-agent: * with Disallow: / blocks every bot that has no group of its own. Under RFC 9309, a crawler that finds a group for its own token uses that group, and uses the * group only when no group matches. Add a named group for each bot you want to treat differently.
  • A rule sits under the wrong User-agent line. A group runs from its User-agent lines to the next User-agent line. Rules typed after the wrong line apply to the wrong bot. Leave a blank line between groups.
  • Blocking GPTBot to leave ChatGPT search. GPTBot does not control search. Block or allow OAI-SearchBot for that.
  • CDN settings that block AI bots. Cloudflare, for example, has an AI Crawler and Agent access policy with Block and Disallow actions for training crawlers. Check these settings when GPTBot cannot reach a page that robots.txt allows. See robots.txt blocks AI crawlers and the guide to robots.txt for AI crawlers.

GPTBot FAQ

Does GPTBot respect robots.txt?

OpenAI documents GPTBot as a robots.txt controlled crawler. Disallowing the GPTBot token indicates that a site's content should not be used for training. OpenAI's page does not give a time for the change to take effect for GPTBot. For search it says about 24 hours.

Is GPTBot used for training?

Yes. OpenAI says GPTBot crawls content that may be used to train its generative AI foundation models.

What are GPTBot's IP ranges?

OpenAI publishes them as JSON at https://openai.com/gptbot.json. Read the file when you build a firewall rule, because OpenAI can change the list.

What is the GPTBot user agent string?

OpenAI's example is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot. OpenAI says the version number may change, so match on GPTBot.

Does blocking GPTBot affect ChatGPT search?

Not according to OpenAI. Search uses OAI-SearchBot, and OpenAI says each setting is independent.

Sources