Skip to content
SeenLens

What are content signals?

Content signals are Content-Signal lines in robots.txt that state whether your content may be used for search, AI input and AI training. Cloudflare published the Content Signals Policy on 24 September 2025. The signals are preferences that no crawler has to obey, and an IETF working group is developing a related standard.

By SeenLens · Last updated

Definition

A content signal is a machine-readable statement about how a site's content may be used after it is fetched. Robots.txt rules say whether a crawler may fetch a path. A content signal says what the operator allows the crawler to do with the content. Cloudflare released the policy under a CC0 license, so anyone may use it.

How do content signals work?

The three signals in the Content Signals Policy
SignalMeaning
searchBuilding a search index and returning results such as links and short excerpts. It excludes AI-generated search summaries.
ai-inputFeeding content into AI models, for example retrieval augmented generation, grounding or other real-time use for AI search answers.
ai-trainTraining or fine-tuning AI models.
robots.txt
User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

A value of yes means the use is allowed. A value of no means the use is not allowed. A missing signal has no meaning: it does not say you have no preference, only that you did not state one in robots.txt. The example above allows search, opts out of training and states nothing about ai-input.

Who uses content signals and who does not?

  • Cloudflare. When a customer turns on the managed robots.txt feature, Cloudflare serves Content-Signal: search=yes, ai-train=no. At the time of the announcement, over 3.8 million domains had the feature on. Cloudflare does not serve an ai-input value for them.
  • Cloudflare Markdown for Agents. Converted Markdown responses carry a content-signal header. If your server sets none, the default is ai-train=yes, search=yes, ai-input=yes.
  • Crawler operators. No complete list of crawlers that honor the signals exists. Cloudflare says the signals express preferences and are not technical countermeasures, and that some companies may ignore them. Google's documentation for AI features does not mention them.
  • Standards work. The IETF AI Preferences (aipref) working group is developing a formal vocabulary on related ideas. Names may still change before it publishes.

Cloudflare's policy text also states that the signals are an express reservation of rights under Article 4 of EU Directive 2019/790. Ask a lawyer what that means for your site. For enforcement, use firewall rules.

How do I check and set content signals?

The AI crawler checker reads the Content-Signal lines in your robots.txt. The Content Signal generator writes a block for you. The fix page add Content Signals covers the scan check that reports a missing preference. Put the line inside each User-agent group.

Frequently asked questions

What does Content-Signal ai-train=no mean?

It states that you do not want your content used to train or fine-tune AI models. It is a request, not a block.

Who invented content signals?

Cloudflare published the Content Signals Policy on 24 September 2025. The IETF AI Preferences working group is developing a related standard.

Is Content-Signal an official standard?

No. It is a published policy with a CC0 license. The IETF AI Preferences work on a related standard is still in progress.

Does a content signal block a crawler?

No. Only a Disallow rule in robots.txt, or a firewall rule, blocks a crawler that complies with it. A signal states a preference.

Where do I put the Content-Signal line?

Inside a User-agent group in robots.txt, next to the Allow and Disallow lines of that group.

Sources