--- title: "What Is GPTBot? User Agent, IP Ranges and robots.txt Rules" description: "GPTBot is OpenAI's training crawler. See its user agent, IP list URL and robots.txt token, and how to block it without leaving ChatGPT search." url: https://seenlens.com/bots/gptbot last_updated: 2026-11-06 --- # What is GPTBot? GPTBot is the web crawler OpenAI uses to collect content that may be used to train its generative AI foundation models. Its robots.txt token is `GPTBot`. OpenAI documents robots.txt as the control: disallowing GPTBot signals that a site's content should not be used for training. OpenAI publishes the user agent and an IP list. Verified 2026-11-06. By SeenLens. Last updated: 2026-11-06 ## GPTBot fact sheet | Fact | Value | | --- | --- | | Operator | OpenAI | | Purpose | Training. GPTBot crawls content that may be used to train OpenAI's generative AI foundation models. | | User agent string | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot` (OpenAI marks this as an example; the version number may change) | | robots.txt token | `GPTBot` | | Published IP list | [openai.com/gptbot.json](https://openai.com/gptbot.json) | | Obeys robots.txt | Yes, as OpenAI documents it. OpenAI lists GPTBot as a robots.txt tag and states that disallowing it indicates content should not be used in training ([OpenAI](https://developers.openai.com/api/docs/bots)). | | Related bots | [OAI-SearchBot](https://seenlens.com/bots/oai-searchbot) (search), [ChatGPT-User](https://seenlens.com/bots/chatgpt-user) (user requests), OAI-AdsBot (ad landing page checks) | | Last verified | 2026-11-06 | ## What does GPTBot do? GPTBot visits public web pages and collects their content. OpenAI says this content may be used to train its generative AI foundation models. GPTBot does not answer questions for ChatGPT users and it does not decide whether a page shows up in ChatGPT search. Other bots do those jobs. When GPTBot fetches a robots.txt file, OpenAI says it may add a `robots.txt` marker to the user agent string. The marker lets you tell robots.txt requests from page requests in logs that do not show the path. Any client can send the string `GPTBot`. To confirm a request is real, compare its IP address with the prefixes in OpenAI's [gptbot.json](https://openai.com/gptbot.json) file. OpenAI updates that list, so read the current file instead of copying addresses once. ## How do I allow or block GPTBot in robots.txt? To block GPTBot from the whole site, add this group to the robots.txt file at the root of your host: ```text User-agent: GPTBot Disallow: / ``` To allow GPTBot everywhere, either leave it out of robots.txt (if no wildcard group blocks it) or name it with an empty rule: ```text User-agent: GPTBot Disallow: ``` To block only part of a site, disallow those paths: ```text User-agent: GPTBot Disallow: /members/ Disallow: /drafts/ ``` OpenAI states that each of its robots.txt settings is independent. This group lets your pages appear in ChatGPT search and asks OpenAI not to use them for training: ```text User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Disallow: / ``` ## Does blocking GPTBot remove my site from ChatGPT search or citations? OpenAI's documentation says no. It gives this exact use case: a webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training. Presence in ChatGPT search depends on [OAI-SearchBot](https://seenlens.com/bots/oai-searchbot). OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. OpenAI adds that if you allow both bots, it may use the results of one crawl for both purposes. OpenAI's page does not say whether a GPTBot block removes content that OpenAI already collected. ## How do I check whether my site blocks GPTBot? Use the [AI crawler checker](https://seenlens.com/tools/ai-crawler-checker) at the top of this page. It is preselected for GPTBot. It reads your robots.txt, shows the group and rule line that match your page path, and then requests the page with the GPTBot user agent. A status code or response size that differs from the browser request points to a firewall block. You can also search your server logs for `GPTBot` and compare the source IPs with gptbot.json. ## What are the common GPTBot mistakes? - **Firewall returns 403 while robots.txt allows GPTBot.** robots.txt is a request. A firewall or CDN rule acts first. If it denies the GPTBot user agent or OpenAI's IP addresses, GPTBot never reaches your content. See [AI crawlers blocked by a firewall](https://seenlens.com/fixes/ai-crawlers-blocked-by-waf). - **A wildcard group blocks everything.** `User-agent: *` with `Disallow: /` blocks every bot that has no group of its own. Under RFC 9309, a crawler that finds a group for its own token uses that group, and uses the `*` group only when no group matches. Add a named group for each bot you want to treat differently. - **A rule sits under the wrong User-agent line.** A group runs from its `User-agent` lines to the next `User-agent` line. Rules typed after the wrong line apply to the wrong bot. Leave a blank line between groups. - **Blocking GPTBot to leave ChatGPT search.** GPTBot does not control search. Block or allow OAI-SearchBot for that. - **CDN settings that block AI bots.** Cloudflare, for example, has an AI Crawler and Agent access policy with Block and Disallow actions for training crawlers. Check these settings when GPTBot cannot reach a page that robots.txt allows. See [robots.txt blocks AI crawlers](https://seenlens.com/fixes/robots-txt-blocks-ai-crawlers) and the [guide to robots.txt for AI crawlers](https://seenlens.com/guides/robots-txt-for-ai-crawlers). ## Which other AI bots should I know about? Each AI company runs several bots with separate robots.txt tokens. These pages cover the others. The [bots hub](https://seenlens.com/bots) lists all of them. - [OAI-SearchBot](https://seenlens.com/bots/oai-searchbot): OpenAI's crawler for ChatGPT search. - [ChatGPT-User](https://seenlens.com/bots/chatgpt-user): OpenAI's fetcher for user requests in ChatGPT. - [ClaudeBot](https://seenlens.com/bots/claudebot): Anthropic's training crawler. - [Claude-SearchBot](https://seenlens.com/bots/claude-searchbot): Anthropic's search crawler, plus Claude-User. - [PerplexityBot](https://seenlens.com/bots/perplexitybot): Perplexity's search crawler, plus Perplexity-User. - [Google-Extended](https://seenlens.com/bots/google-extended): Google's robots.txt token for Gemini training and grounding. - [Applebot-Extended](https://seenlens.com/bots/applebot-extended): Apple's robots.txt token for AI training. - [CCBot](https://seenlens.com/bots/ccbot): Common Crawl's open dataset crawler. ## GPTBot FAQ ### Does GPTBot respect robots.txt? OpenAI documents GPTBot as a robots.txt controlled crawler. Disallowing the `GPTBot` token indicates that a site's content should not be used for training. OpenAI's page does not give a time for the change to take effect for GPTBot. For search it says about 24 hours. ### Is GPTBot used for training? Yes. OpenAI says GPTBot crawls content that may be used to train its generative AI foundation models. ### What are GPTBot's IP ranges? OpenAI publishes them as JSON at https://openai.com/gptbot.json. Read the file when you build a firewall rule, because OpenAI can change the list. ### What is the GPTBot user agent string? OpenAI's example is `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot`. OpenAI says the version number may change, so match on `GPTBot`. ### Does blocking GPTBot affect ChatGPT search? Not according to OpenAI. Search uses OAI-SearchBot, and OpenAI says each setting is independent. ## Sources - [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) (user agent string, robots.txt token, IP list URL, purpose, independence of settings; checked 2026-11-06) - [OpenAI: gptbot.json](https://openai.com/gptbot.json) (published IP prefixes; the file loaded on 2026-11-06) - [IETF RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) (how a crawler picks the group that applies; checked 2026-11-06)