Skip to content
SeenLens

Content Signals in robots.txt: syntax, meaning and who reads it

Content Signals is a robots.txt extension that Cloudflare published in September 2025. A Content-Signal line says whether a site allows search, AI input and AI training. On 2026-11-06, no crawler operator I checked documents that its crawlers read the line, so treat it as a stated preference and keep Disallow rules for blocking.

By SeenLens · Last updated

What do you allow?
search

Building a search index and showing links and short excerpts in results.

ai-input

Using the content as input to an AI answer in real time (retrieval, grounding).

ai-train

Training or fine-tuning AI models.

robots.txt block
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

User-agent: __usage_ai-training
Disallow: /

Content-Signal states a preference. It does not block a crawler; use Disallow lines or firewall rules for that. The __usage_ai-training group is a preference too: it names a use of content, not a crawler. Bing AI training and grounding settings.

What is the Content Signals policy?

Cloudflare announced the Content Signals Policy on 2025-09-24. It adds a Content-Signal line to robots.txt. The line states what the site allows crawlers to do with its content. Cloudflare released the policy text under CC0, which lets anyone use it freely.

The policy also states that any restriction expressed with Content Signals is an express reservation of rights under Article 4 of EU Directive 2019/790 on copyright. This is Cloudflare's legal framing. It is not a technical control, and SeenLens gives no legal advice.

Cloudflare describes the signals as preferences and says some companies might ignore them. The signals block nothing by themselves.

What is the Content-Signal syntax?

The line sits inside a User-agent: group. It holds a comma-separated list of name=yes or name=no pairs. This example allows search and AI input and says no to training.

robots.txt with a Content-Signal line
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

The policy text in the file gives three rules. A signal set to yes means a crawler may collect the content for that use. A signal set to no means it may not. A signal you leave out means the site neither grants nor restricts that use through Content Signals.

The contentsignals.org site also shows an example with a path in front of the pairs, such as Content-Signal: /blog/ search=yes, ai-input=no, ai-train=no. Check that your tools accept this form before you use it. The Content Signal generator on this page writes the standard form.

What do search, ai-input and ai-train mean?

The three Content Signals and their meaning (Cloudflare policy text)
SignalMeaningNote
searchBuilding a search index and providing search results, such as hyperlinks and short excerpts.The policy says search does not include AI-generated search summaries.
ai-inputInputting content into one or more AI models, for example retrieval-augmented generation or grounding.This covers real-time use for generative AI search answers.
ai-trainTraining or fine-tuning AI models.Set it to no to say that you do not allow training.

Because search excludes AI summaries, a site that wants to appear in AI answers sets ai-input=yes as well. That is why the SeenLens robots.txt below says search=yes, ai-input=yes.

What are the defaults?

  • Cloudflare managed robots.txt: the 2025 blog post says customers who switch it on get the policy text and Content-Signal: search=yes, ai-train=no. Cloudflare serves no ai-input value, because it does not know the customer's preference. Cloudflare's current documentation lists a longer default that adds a use=reference value.
  • Free zones with no robots.txt: they get the policy text as comments, with no Allow, Disallow or signal lines.
  • Sites that already have a robots.txt: Cloudflare says they see no change.
  • Any other site: no signal means no stated preference.

Cloudflare said more than 3.8 million domains had the managed robots.txt switched on at the time of its blog post. Its documentation also says the managed file adds Disallow rules for named crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot. Those named rules do the blocking. The signals do not.

Do AI crawlers respect Content Signals?

No crawler operator that I checked documents support for it. On 2026-11-06 I searched the crawler pages of OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl. None of them mentions Content Signals. I did not check Bing. Cloudflare's own documentation names no crawler that obeys the signals either.

This means a Content-Signal line is a public statement of your preference. It has no proven effect on any crawler. If you need to stop a crawler, use a Disallow rule for its token, and use a firewall for crawlers that ignore robots.txt. The guide robots.txt for AI crawlers has the tokens and recipes.

Google Search Console may report a Content-Signal line as Syntax not understood. Cloudflare says it observed no effect on crawling or SEO from this report. Other robots.txt validators can also flag the line.

How do Content Signals relate to the IETF AIPREF work?

The IETF AI Preferences working group (AIPREF) is writing a standard way to express AI usage preferences. Its charter has two main parts. One is a vocabulary of preferences. The other is a way to attach the preferences to content, for example through the Robots Exclusion Protocol (RFC 9309) or an HTTP response header.

On 2026-11-06 the datatracker lists two working group drafts: draft-ietf-aipref-vocab (revision 08, 2026-09-13) and draft-ietf-aipref-attach (revision 05, 2026-08-18). Both show the state Proposed Standard as a working group document. I could not confirm that either is an RFC. Content Signals is Cloudflare's own vocabulary. contentsignals.org now describes itself as a guide to the IETF's proposed AI Preferences.

Two operators have announced plans for the IETF approach. Google's crawler documentation says Google intends to support the AIPREF training preference signal, with user-agent: __usage_ai-training, in early 2027. Anthropic says it will honor a machine-readable no-AI-training signal based on an IETF draft and is rolling it out gradually. Neither page mentions Content-Signal. Microsoft documents the same token for Bing AI Training; see Bing AI grounding and AI training.

What does a live Content-Signal robots.txt look like?

SeenLens publishes its own robots.txt at seenlens.com/robots.txt. Every group carries the same signal and the same path rules. Here is the group for GPTBot, fetched on 2026-11-06.

seenlens.com/robots.txt (group for GPTBot)
User-agent: GPTBot
Content-Signal: search=yes, ai-input=yes
Allow: /
Allow: /scan/sample
Disallow: /api/
Disallow: /scan/
  • The signal repeats in every group. A crawler reads the group that names it, so a signal in the * group does not reach a crawler with its own group.
  • `ai-train` is left out. SeenLens neither grants nor restricts training through Content Signals.
  • Real rules follow the signal. Disallow: /api/ and Disallow: /scan/ keep crawlers out of private scan results. The signal does not do that.
  • One group exists per AI crawler. The file names GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot and Bytespider.

The SeenLens scan has a check for this line. See the fix guide add Content Signals to robots.txt. The AI crawler checker also reads Content-Signal lines for any site.

Should I add Content-Signal to my robots.txt?

Add it if you want your preference on record. It costs one line per group and no operator documents a harm. Do not expect it to change crawler behavior. A signal says what you allow. A Disallow rule is what a crawler actually obeys today.

  1. Choose your three values. Most sites that want AI search set search=yes, ai-input=yes.
  2. Build the line with the generator above, then copy it into each User-agent: group.
  3. Keep Disallow rules for any crawler you want to stop.
  4. Run the AI crawler checker to confirm the file parses and the signal shows.

FAQ

What is the Cloudflare Content Signals policy?

It is a robots.txt extension that Cloudflare published on 2025-09-24. A Content-Signal line states whether a site allows search, AI input and AI training. Cloudflare released the policy text under CC0.

Do AI crawlers respect Content Signals?

No operator I checked documents it. OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl do not mention Content Signals on their crawler pages. Use Disallow rules to block.

What happens if I leave out a signal?

The policy text says you neither grant nor restrict that use through Content Signals.

Is Content Signals an internet standard?

No. It is Cloudflare's proposal. The IETF AIPREF working group is writing a standard vocabulary and a way to attach it, and its drafts were not RFCs when I checked.

Does ai-train=no stop Google from using my content in AI Overviews?

Google's documentation does not list Content Signals. For AI Overviews, Google points to Googlebot rules and snippet controls such as nosnippet and max-snippet. Google-Extended does not affect Search.

Sources