What is Applebot-Extended?

Applebot-Extended is a rule target, not a visitor. Apple crawls pages with Applebot. Apple then uses the Applebot-Extended rules in your robots.txt to decide whether it may use the crawled data to train its foundation models.

Apple states that Applebot-Extended is only used to determine how to use data crawled by Applebot. It adds that allowing Applebot-Extended will help improve its generative AI models over time.

Apple names a second AI use of Applebot data: additional context and up-to-date content when AI models generate output in Apple products, such as broad world knowledge answers in Siri and Search that may link to sources. Applebot-Extended does not control that use. Apple says publishers opt out of it with the nosnippet meta tag.

Apple's example disallows a folder for Applebot-Extended. To opt out for the whole site:

User-agent: Applebot-Extended
Disallow: /

To opt out of training and keep Apple search, name both tokens:

User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Disallow: /

To also keep a page out of the Apple AI answers described above, add the nosnippet robots directive. Apple supports it as a meta tag and as an X-Robots-Tag header:

<meta name="applebot" content="nosnippet">

Apple says webpages that disallow Applebot-Extended can still be included in search results, and that site rules for Applebot-Extended are not considered in ranking for Search.

Even if you disallow Applebot-Extended and add nosnippet, Apple says your instructions may still allow Applebot to crawl your pages, and your content remains discoverable through Spotlight, Siri, Safari and other system-wide features.

Apple states: enabling Applebot in robots.txt allows website content to appear in search results for Apple users around the world.

Use the AI crawler checker above, preselected for Applebot-Extended. It reads your robots.txt and shows the group and rule line that match your page path. It does not request your page as Applebot-Extended, because that token does not crawl.

To confirm that real Applebot visits your site, use reverse DNS. Apple says Applebot traffic is generally identified by a reverse DNS name in the *.applebot.apple.com domain, or by an IP address inside the CIDR prefixes in Apple's JSON file.

• Blocking Applebot to opt out of training. Applebot powers Apple search. Block the Applebot-Extended token for the training opt-out.
• Assuming no mention means Googlebot rules do not apply. Apple says if your robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot instructions. A Googlebot block can therefore block Applebot.
• Blocking scripts and styles. Apple says Applebot may render pages in a browser, and blocked JavaScript, CSS or XHR resources can stop it from rendering content properly.
• Expecting a crawl-delay to work. Apple says Applebot does not follow crawl-delay.
• A firewall 403 on Applebot. robots.txt cannot fix a firewall rule. See AI crawlers blocked by a firewall and robots.txt blocks AI crawlers.

Each AI company runs several bots with separate robots.txt tokens. These pages cover the others. The bots hub lists all of them.

• GPTBot: OpenAI's training crawler.
• OAI-SearchBot: OpenAI's crawler for ChatGPT search.
• ChatGPT-User: OpenAI's fetcher for user requests in ChatGPT.
• ClaudeBot: Anthropic's training crawler.
• Claude-SearchBot: Anthropic's search crawler, plus Claude-User.
• PerplexityBot: Perplexity's search crawler, plus Perplexity-User.
• Google-Extended: Google's robots.txt token for Gemini training and grounding.
• CCBot: Common Crawl's open dataset crawler.

• Apple: About Applebot (Applebot-Extended definition, no crawling, search effect, nosnippet, user agent format, reverse DNS and IP list, Googlebot fallback; Apple's page shows a published date of 2026-09-04; checked 2026-11-06)
• Apple: Applebot IP CIDRs (linked from Apple's support page; the URL returned HTTP 200 on 2026-11-06)