Skip to content
SeenLens

Content-Signal in robots.txt: how to add it

A Content-Signal line in robots.txt states whether AI systems may use your content for search, AI answers and model training. Add one line per group, such as search=yes, ai-input=yes, ai-train=no. It states a preference and does not enforce it. The edit takes about five minutes.

By SeenLens · Last updated

What do you allow?
search

Building a search index and showing links and short excerpts in results.

ai-input

Using the content as input to an AI answer in real time (retrieval, grounding).

ai-train

Training or fine-tuning AI models.

robots.txt block
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

User-agent: __usage_ai-training
Disallow: /

Content-Signal states a preference. It does not block a crawler; use Disallow lines or firewall rules for that. The __usage_ai-training group is a preference too: it names a use of content, not a crawler. Bing AI training and grounding settings.

What this check tests

SeenLens reads the Content-Signal lines in your robots.txt and checks their syntax.

How the content-signal check scores robots.txt
ResultWhat SeenLens found
InfoNo Content-Signal line
WarnA malformed entry, an unknown key, a value other than yes or no, a line outside any group, or a named AI group without the line while the * group has one
PassValid lines inside groups, and no gaps

The format comes from Cloudflare's Content Signals Policy, published on 2025-09-24. A line has the form Content-Signal: followed by comma separated pairs. The key search covers building a search index and showing results. The key ai-input covers feeding content into AI models, for example for retrieval or grounding. The key ai-train covers training or fine tuning models. The value yes permits the use, no forbids it, and an omitted key states no preference. The scanner also accepts Cloudflare's experimental use key.

How to confirm the problem

curl
curl -s https://example.com/robots.txt | grep -n -i "content-signal"

No output means the file has no line. The Content Signal generator above builds a correct block from a form.

How to fix it

Put the line inside each User-agent group. A crawler that matches a named group reads that group, so repeat the line in every named group.

robots.txt
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

User-agent: GPTBot
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /
  • Cloudflare: go to Security, Settings, filter by Bot traffic, open AI Crawler and Agent access policies, set Training to Disallow, turn on Bot Preference Sync and save. Cloudflare then writes a managed block with search=yes, ai-train=no and a use=reference extension. It puts that block before your own file.
  • Next.js: add a static app/robots.txt. From Next.js 16.3.0 the other field of a robots.ts rule can print non standard lines such as Content-Signal.
  • Shopify: edit robots.txt.liquid, and add the line to the group you want.
  • WordPress, Nginx, Apache: edit the robots.txt file in the web root.

Does Google follow Content-Signal?

No. Search Engine Roundtable reported on 2026-07-06 that John Mueller of Google said the directive has no effect on any crawler or LLM. He wrote this in a Reddit reply in r/TechSEO. The line is a published preference, and each AI company decides whether to honor it. For control that works, use named robots.txt groups for training crawlers, a firewall rule, or both. Google's own controls for AI features in Search are nosnippet, max-snippet and noindex; see the noindex fix. Bing documents a different training token, __usage_ai-training; see Bing AI grounding and AI training.

How to verify the fix

  1. Fetch robots.txt and find the line inside each group.
  2. Open the AI crawler checker. It lists the Content-Signal lines it reads.
  3. Run the SeenLens scan. The content-signal check should pass.

Run the free SeenLens scan on your page

When can you ignore it?

Skip it if you do not want to state a preference or if you rely on named groups and a firewall. The check has a low weight and reports Info when the line is absent.

Sources