Skip to content
SeenLens

Content Signal generator for robots.txt

The Content Signal generator writes the robots.txt block that states whether your content may be used for search, AI input and AI training. It runs in your browser, so we fetch nothing and store nothing. This is a robots.txt tool, not an electronic signal generator. A Content Signal is a preference, and no crawler has to obey it.

By SeenLens · Last updated

What do you allow?
search

Building a search index and showing links and short excerpts in results.

ai-input

Using the content as input to an AI answer in real time (retrieval, grounding).

ai-train

Training or fine-tuning AI models.

robots.txt block
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

User-agent: __usage_ai-training
Disallow: /

Content-Signal states a preference. It does not block a crawler; use Disallow lines or firewall rules for that. The __usage_ai-training group is a preference too: it names a use of content, not a crawler. Bing AI training and grounding settings.

What does the generator write?

The generator writes one Content-Signal line for robots.txt. The line has three signals, and each one takes the value yes or no:

The three Content Signals
SignalCovers
searchBuilding a search index and returning results such as links and short excerpts. It excludes AI-generated search summaries.
ai-inputFeeding content into AI models, for example retrieval augmented generation, grounding or other real-time use for AI search answers
ai-trainTraining or fine-tuning AI models

This output allows search, allows AI input and opts out of training:

robots.txt
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Cloudflare, which published the policy on 24 September 2025, says a signal set to yes means go, a signal set to no means stop, and a missing signal has no meaning. A missing signal does not mean you have no preference. It means you did not state one in robots.txt.

How do I use the generated block?

  1. Set each signal in the form and copy the block.
  2. Paste the Content-Signal line inside each User-agent group in your robots.txt. A line outside a group does not belong to a crawler.
  3. Publish the file at /robots.txt.
  4. Run the AI crawler checker on a page. It reads your Content-Signal lines.

Keep your Allow and Disallow rules. A Content Signal does not replace them.

What does a Content Signal not do?

  • It does not enforce anything. Cloudflare writes that signals express preferences and are not technical countermeasures against scraping. Some companies may ignore them. For enforcement, use firewall rules.
  • It is not a finished standard. The IETF AI Preferences (aipref) working group is developing a formal vocabulary on similar ideas, and names may still change. See the guide to Content Signals.
  • It does not control Google Search. Google's documentation for AI features does not mention Content Signals. Googlebot and Google-Extended follow their own robots.txt rules.
  • It is not a legal contract. Cloudflare's policy text states that the signals reserve rights under Article 4 of EU Directive 2019/790. Ask a lawyer how that applies to you.

How do I fix a missing or wrong Content Signal?

Frequently asked questions

What does ai-train=no do?

It states that you do not want your content used to train or fine-tune AI models. It is a request. A crawler that does not read Content Signals can still fetch the page.

Do AI crawlers respect Content Signals?

There is no complete list. Cloudflare says the signals are preferences and that some companies may ignore them. Check each operator's documentation. The Content Signals guide tracks this.

Does ai-train=no stop AI Overviews?

No. Google's AI features page does not list Content Signals. It lists nosnippet, data-nosnippet, max-snippet and noindex as the controls that limit what Search shows.

What does Cloudflare put in the managed robots.txt?

Cloudflare serves Content-Signal: search=yes, ai-train=no for customers who turn on the managed robots.txt feature. It does not serve an ai-input signal, because it does not know those customers' preference.

Should I set ai-input to yes?

Set yes if you want AI search answers to use and cite your content. Set no if you do not. Search crawlers such as OAI-SearchBot decide inclusion in AI search, so read the AI crawler guide too.

Method and sources

The generator follows the syntax in Cloudflare's Content Signals Policy. The scan check content-signal looks for a Content-Signal line in robots.txt. Last reviewed 2026-11-06.

Sources