--- title: "What Is OAI-SearchBot? User Agent and robots.txt Rules" description: "OAI-SearchBot is OpenAI's crawler for ChatGPT search. See its user agent, IP list and robots.txt token, and what happens to your site if you block it." url: https://seenlens.com/bots/oai-searchbot last_updated: 2026-11-06 --- # What is OAI-SearchBot? OAI-SearchBot is the crawler OpenAI uses to surface websites in ChatGPT's search features. Its robots.txt token is `OAI-SearchBot`. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI recommends allowing it. Verified 2026-11-06. By SeenLens. Last updated: 2026-11-06 ## OAI-SearchBot fact sheet | Fact | Value | | --- | --- | | Operator | OpenAI | | Purpose | Search index. It surfaces websites in search results in ChatGPT's search features. | | User agent string | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot` (OpenAI's example; the version number may change) | | User agent string for robots.txt fetches | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot` | | robots.txt token | `OAI-SearchBot` | | Published IP list | [openai.com/searchbot.json](https://openai.com/searchbot.json) | | Obeys robots.txt | Yes, as OpenAI documents it. OpenAI lists OAI-SearchBot as a robots.txt tag, and says a search change takes about 24 hours ([OpenAI](https://developers.openai.com/api/docs/bots)). | | Related bots | [GPTBot](https://seenlens.com/bots/gptbot) (training), [ChatGPT-User](https://seenlens.com/bots/chatgpt-user) (user requests), OAI-AdsBot (ad landing page checks) | | Last verified | 2026-11-06 | ## What does OAI-SearchBot do? OAI-SearchBot crawls pages so that ChatGPT can find them when it answers with search. It is a search crawler, and OpenAI keeps it separate from the training crawler [GPTBot](https://seenlens.com/bots/gptbot). OpenAI says each setting is independent of the others. OpenAI also lists OAI-AdsBot. That bot validates landing pages submitted as ads on ChatGPT. OpenAI says OAI-AdsBot visits only pages submitted as ads and that its data is not used to train generative AI foundation models. The OpenAI page lists no robots.txt token for OAI-AdsBot. OpenAI says OAI-SearchBot may add a `robots.txt` marker to its user agent string when it fetches robots.txt files. To confirm that a request is real, compare its IP address with [searchbot.json](https://openai.com/searchbot.json). ## How do I allow or block OAI-SearchBot? OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges. An explicit allow group looks like this: ```text User-agent: OAI-SearchBot Allow: / ``` To block OAI-SearchBot from the whole site: ```text User-agent: OAI-SearchBot Disallow: / ``` To stay in ChatGPT search and opt out of training, allow OAI-SearchBot and block GPTBot: ```text User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Disallow: / ``` OpenAI says it can take about 24 hours after a robots.txt update for its search systems to adjust. ## Does blocking OAI-SearchBot affect ChatGPT search and citations? Yes. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Such sites can still appear as navigational links. Blocking [GPTBot](https://seenlens.com/bots/gptbot) alone does not remove you from ChatGPT search. OpenAI documents the two bots as independent settings. Allowing OAI-SearchBot does not guarantee that ChatGPT cites a page. It only removes one reason for exclusion. ## How do I check whether my site blocks OAI-SearchBot? Run the [AI crawler checker](https://seenlens.com/tools/ai-crawler-checker) above, which is preselected for OAI-SearchBot. It shows whether your robots.txt allows the bot for your page path and which rule matched. It also requests the page with the OAI-SearchBot user agent and compares the result with a browser request. A 403 or a much smaller response means a firewall or CDN rule blocks the bot even if robots.txt allows it. In your own logs, search for `OAI-SearchBot` and check the source IPs against searchbot.json. ## What are the common OAI-SearchBot mistakes? - **robots.txt allows the bot but the firewall returns 403.** The firewall decides first. Allow the OAI-SearchBot user agent together with OpenAI's published IP ranges. See [AI crawlers blocked by a firewall](https://seenlens.com/fixes/ai-crawlers-blocked-by-waf). - **A `User-agent: *` group with `Disallow: /`.** Any bot without its own group falls back to this rule. Add a named `OAI-SearchBot` group. A crawler uses the group that matches its token and uses `*` only when nothing matches ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html)). - **Blocking GPTBot and expecting search to stop, or the reverse.** The two bots have separate tokens. Set each one on purpose. - **Managed AI bot blocking at the CDN.** Cloudflare's AI Crawler and Agent access policy has separate actions for search, agent and training behavior. A Block action for search crawlers can stop OAI-SearchBot at Cloudflare's edge. See [robots.txt blocks AI crawlers](https://seenlens.com/fixes/robots-txt-blocks-ai-crawlers). - **Expecting an instant effect.** OpenAI says a robots.txt update can take about 24 hours to reach its search systems. ## Which other AI bots should I know about? Each AI company runs several bots with separate robots.txt tokens. These pages cover the others. The [bots hub](https://seenlens.com/bots) lists all of them. - [GPTBot](https://seenlens.com/bots/gptbot): OpenAI's training crawler. - [ChatGPT-User](https://seenlens.com/bots/chatgpt-user): OpenAI's fetcher for user requests in ChatGPT. - [ClaudeBot](https://seenlens.com/bots/claudebot): Anthropic's training crawler. - [Claude-SearchBot](https://seenlens.com/bots/claude-searchbot): Anthropic's search crawler, plus Claude-User. - [PerplexityBot](https://seenlens.com/bots/perplexitybot): Perplexity's search crawler, plus Perplexity-User. - [Google-Extended](https://seenlens.com/bots/google-extended): Google's robots.txt token for Gemini training and grounding. - [Applebot-Extended](https://seenlens.com/bots/applebot-extended): Apple's robots.txt token for AI training. - [CCBot](https://seenlens.com/bots/ccbot): Common Crawl's open dataset crawler. ## OAI-SearchBot FAQ ### What is OAI-SearchBot? OAI-SearchBot is OpenAI's crawler for ChatGPT's search features. It surfaces websites in ChatGPT search results. ### What is the OAI-SearchBot user agent? OpenAI's example string ends with `compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot`. The version number may change, so match on `OAI-SearchBot`. ### Does blocking GPTBot affect ChatGPT search? No, according to OpenAI. GPTBot is for training. OAI-SearchBot controls search, and each setting is independent. ### Should I allow OAI-SearchBot? OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing its published IP ranges if you want your site to appear in ChatGPT search. ### Does OAI-SearchBot use my content for training? OpenAI assigns training to GPTBot and search to OAI-SearchBot. OpenAI also says that if you allow both, it may use the results of one crawl for both purposes. ## Sources - [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) (user agent strings, token, IP list URL, 24 hour note, OAI-AdsBot; checked 2026-11-06) - [OpenAI: searchbot.json](https://openai.com/searchbot.json) (published IP prefixes; the file loaded on 2026-11-06) - [Cloudflare: AI Crawler and Agent access policies](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/) (Search, Agent and Training actions; checked 2026-11-06) - [IETF RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) (group selection and the `*` fallback; checked 2026-11-06)