OAI-SearchBot vs GPTBot vs ChatGPT-User: what each OpenAI crawler does
OpenAI runs three main crawlers, and each has one job. OAI-SearchBot finds pages for ChatGPT search. GPTBot collects content that OpenAI may use to train its models. ChatGPT-User fetches a page when a person asks ChatGPT to. To appear in ChatGPT search and stay out of training, allow OAI-SearchBot and disallow GPTBot.
By SeenLens · Last updated
Free, no sign-up. We read robots.txt and request the page with a browser and four crawler user agents. We do not store your page text.
What is the difference between the three OpenAI crawlers?
OpenAI documents each crawler on its bots page. The page treats each one as a separate control. You can allow one and block another. The table sums up what OpenAI says, checked on 2026-11-06.
| Crawler | Job | Does robots.txt apply? | What blocking does |
|---|---|---|---|
| OAI-SearchBot | Surfaces websites in ChatGPT search features | Yes | Your site is not shown in ChatGPT search answers. It can still appear as a navigational link. |
| GPTBot | Crawls content that may be used to train OpenAI foundation models | Yes | Signals that your content should not be used for training. |
| ChatGPT-User | Visits a page when a user asks ChatGPT or a Custom GPT, and for some GPT Actions | Not necessarily. OpenAI says the rules may not apply. | OpenAI says this crawler is not used to decide whether content appears in search. |
OpenAI lists a fourth crawler, OAI-AdsBot. It checks landing pages that advertisers submit for ChatGPT ads. It visits only those pages, and OpenAI says it does not use the data for training. Most sites never see it.
What does OAI-SearchBot do?
OAI-SearchBot is the crawler for ChatGPT search. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers. A blocked site can still appear as a navigational link. OpenAI recommends that you allow OAI-SearchBot if you want your site in search results, and that you allow requests from its published IP ranges.
The user agent string ends with compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot. The version number can change, so match on the token OAI-SearchBot and not on the version. OpenAI publishes the IP ranges at openai.com/searchbot.json.
Allowing OAI-SearchBot does not guarantee a mention. In an Ahrefs study of 1.4 million ChatGPT prompts (published 2026-04-15), ChatGPT cited about half of the URLs it retrieved. Cited page titles matched the prompt more closely than titles of pages that were not cited. See the OAI-SearchBot page for the full profile.
What does GPTBot do, and does blocking it remove my site from ChatGPT?
GPTBot collects content that OpenAI may use to train its generative AI foundation models. OpenAI says that disallowing GPTBot indicates your content should not be used in training. OpenAI does not list GPTBot as an input to ChatGPT search. For search, OpenAI points to OAI-SearchBot.
So blocking GPTBot does not remove your site from ChatGPT search answers. Blocking OAI-SearchBot does. Many sites block both by mistake when they copy a list of AI bots. Check your file against the recipes below.
The user agent string ends with compatible; GPTBot/1.4; +https://openai.com/gptbot. OpenAI publishes IP ranges at openai.com/gptbot.json. Read the GPTBot page for more.
What does ChatGPT-User do, and does robots.txt stop it?
ChatGPT-User visits a page because a person asked ChatGPT about it. This includes some Custom GPT actions. OpenAI says that because a user starts these visits, robots.txt rules may not apply. OpenAI also says ChatGPT-User is not used to decide whether content appears in search.
This has a practical effect. You can paste your URL into ChatGPT and watch it read your page, and the visit comes from ChatGPT-User. A successful read does not prove that OAI-SearchBot can crawl the page. Robots.txt may not apply to ChatGPT-User, but it does apply to OAI-SearchBot. Test both. The ChatGPT-User page has details, and its IP ranges are at openai.com/chatgpt-user.json.
Which robots.txt should I use for each goal?
OpenAI's guidance supports this pattern: allow search and block training. Copy it and keep the other groups in your file.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /| Goal | OAI-SearchBot | GPTBot | ChatGPT-User |
|---|---|---|---|
| Appear in ChatGPT search and allow training | Allow | Allow | Leave out |
| Appear in ChatGPT search and stay out of training | Allow | Disallow | Leave out |
| Stay out of ChatGPT search and training | Disallow | Disallow | Leave out |
The last column says Leave out because OpenAI says robots.txt may not apply to user-requested visits. Do not rely on a rule there to protect private pages. Use a login. The wider guide robots.txt for AI crawlers covers other operators and the matching rules.
How long do robots.txt changes take to work for ChatGPT?
OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust for search results. Plan for one day before you test again. OpenAI gives this timing for search. Its page gives no separate timing for GPTBot.
How do I check that these crawlers can reach my site?
- Run the checker above with your page URL. Select OAI-SearchBot, GPTBot and ChatGPT-User. Read the matched group and rule line for each.
- Read the firewall comparison. A firewall can return 403 or a challenge page to OpenAI user agents even when robots.txt allows them.
- Check your server log for the three tokens. Match the visiting IP address with the OpenAI IP list. A user agent string alone can be faked.
- If the checker shows a block you did not intend, fix the file and see what to do when robots.txt blocks AI crawlers. For a firewall block, see AI crawlers blocked by a firewall.
The AI crawler checker is free. The full scan runs 43 checks on one page.
FAQ
Is GPTBot the same as OAI-SearchBot?
No. GPTBot gathers content that may be used for training. OAI-SearchBot surfaces sites in ChatGPT search. They have separate robots.txt tokens, so you can allow one and block the other.
Do I need to allow all three OpenAI crawlers?
No. To appear in ChatGPT search, allow OAI-SearchBot. GPTBot is your training decision. For ChatGPT-User, OpenAI says robots.txt rules may not apply.
Does ChatGPT-User respect robots.txt?
OpenAI says robots.txt rules may not apply to it, because a user starts each visit. Use authentication to protect private pages.
How do I know whether a visit really comes from OpenAI?
Match the visiting IP address with the lists OpenAI publishes for each crawler. Anyone can send a fake user agent string.
Sources
- OpenAI: Overview of OpenAI crawlers (user agents, robots.txt tokens, what blocking does, IP lists, about 24 hours for search changes, OAI-AdsBot, checked 2026-11-06)
- Ahrefs: why ChatGPT cites pages (1.4 million prompts, about half of retrieved URLs cited, title match, published 2026-04-15)