---
title: "Content Signals in robots.txt: Syntax and Support"
description: "Content Signals add a Content-Signal line to robots.txt for search, ai-input and ai-train. Syntax, defaults, who reads it and how it fits IETF AIPREF."
url: https://seenlens.com/guides/content-signals-robots-txt
last_updated: 2026-11-06
---

# Content Signals in robots.txt: syntax, meaning and who reads it

Content Signals is a robots.txt extension that Cloudflare published in September 2025. A `Content-Signal` line says whether a site allows search, AI input and AI training. On 2026-11-06, no crawler operator I checked documents that its crawlers read the line, so treat it as a stated preference and keep Disallow rules for blocking.

By SeenLens. Last updated: 2026-11-06

## What is the Content Signals policy?

Cloudflare announced the Content Signals Policy on 2025-09-24. It adds a `Content-Signal` line to robots.txt. The line states what the site allows crawlers to do with its content. Cloudflare released the policy text under CC0, which lets anyone use it freely.

The policy also states that any restriction expressed with Content Signals is an express reservation of rights under Article 4 of EU Directive 2019/790 on copyright. This is Cloudflare's legal framing. It is not a technical control, and SeenLens gives no legal advice.

Cloudflare describes the signals as preferences and says some companies might ignore them. The signals block nothing by themselves.

## What is the Content-Signal syntax?

The line sits inside a `User-agent:` group. It holds a comma-separated list of `name=yes` or `name=no` pairs. This example allows search and AI input and says no to training.

```text
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
```

The policy text in the file gives three rules. A signal set to yes means a crawler may collect the content for that use. A signal set to no means it may not. A signal you leave out means the site neither grants nor restricts that use through Content Signals.

The contentsignals.org site also shows an example with a path in front of the pairs, such as `Content-Signal: /blog/ search=yes, ai-input=no, ai-train=no`. Check that your tools accept this form before you use it. The [Content Signal generator](https://seenlens.com/tools/content-signal-generator) on this page writes the standard form.

## What do search, ai-input and ai-train mean?

| Signal | Meaning | Note |
| --- | --- | --- |
| `search` | Building a search index and providing search results, such as hyperlinks and short excerpts. | The policy says search does not include AI-generated search summaries. |
| `ai-input` | Inputting content into one or more AI models, for example retrieval-augmented generation or grounding. | This covers real-time use for generative AI search answers. |
| `ai-train` | Training or fine-tuning AI models. | Set it to no to say that you do not allow training. |

Because `search` excludes AI summaries, a site that wants to appear in AI answers sets `ai-input=yes` as well. That is why the SeenLens robots.txt below says `search=yes, ai-input=yes`.

## What are the defaults?

- **Cloudflare managed robots.txt:** the 2025 blog post says customers who switch it on get the policy text and `Content-Signal: search=yes, ai-train=no`. Cloudflare serves no `ai-input` value, because it does not know the customer's preference. Cloudflare's current documentation lists a longer default that adds a `use=reference` value.
- **Free zones with no robots.txt:** they get the policy text as comments, with no Allow, Disallow or signal lines.
- **Sites that already have a robots.txt:** Cloudflare says they see no change.
- **Any other site:** no signal means no stated preference.

Cloudflare said more than 3.8 million domains had the managed robots.txt switched on at the time of its blog post. Its documentation also says the managed file adds Disallow rules for named crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot. Those named rules do the blocking. The signals do not.

## Do AI crawlers respect Content Signals?

No crawler operator that I checked documents support for it. On 2026-11-06 I searched the crawler pages of OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl. None of them mentions Content Signals. I did not check Bing. Cloudflare's own documentation names no crawler that obeys the signals either.

This means a `Content-Signal` line is a public statement of your preference. It has no proven effect on any crawler. If you need to stop a crawler, use a `Disallow` rule for its token, and use a firewall for crawlers that ignore robots.txt. The guide [robots.txt for AI crawlers](https://seenlens.com/guides/robots-txt-for-ai-crawlers) has the tokens and recipes.

Google Search Console may report a Content-Signal line as `Syntax not understood`. Cloudflare says it observed no effect on crawling or SEO from this report. Other robots.txt validators can also flag the line.

## How do Content Signals relate to the IETF AIPREF work?

The IETF AI Preferences working group (AIPREF) is writing a standard way to express AI usage preferences. Its charter has two main parts. One is a vocabulary of preferences. The other is a way to attach the preferences to content, for example through the Robots Exclusion Protocol (RFC 9309) or an HTTP response header.

On 2026-11-06 the datatracker lists two working group drafts: `draft-ietf-aipref-vocab` (revision 08, 2026-09-13) and `draft-ietf-aipref-attach` (revision 05, 2026-08-18). Both show the state Proposed Standard as a working group document. I could not confirm that either is an RFC. Content Signals is Cloudflare's own vocabulary. contentsignals.org now describes itself as a guide to the IETF's proposed AI Preferences.

Two operators have announced plans for the IETF approach. Google's crawler documentation says Google intends to support the AIPREF training preference signal, with `user-agent: __usage_ai-training`, in early 2027. Anthropic says it will honor a machine-readable no-AI-training signal based on an IETF draft and is rolling it out gradually. Neither page mentions Content-Signal. Microsoft documents the same token for Bing AI Training; see [Bing AI grounding and AI training](https://seenlens.com/guides/bing-ai-grounding-and-training).

## What does a live Content-Signal robots.txt look like?

SeenLens publishes its own robots.txt at [seenlens.com/robots.txt](https://seenlens.com/robots.txt). Every group carries the same signal and the same path rules. Here is the group for GPTBot, fetched on 2026-11-06.

```text
User-agent: GPTBot
Content-Signal: search=yes, ai-input=yes
Allow: /
Allow: /scan/sample
Disallow: /api/
Disallow: /scan/
```

- **The signal repeats in every group.** A crawler reads the group that names it, so a signal in the `*` group does not reach a crawler with its own group.
- **`ai-train` is left out.** SeenLens neither grants nor restricts training through Content Signals.
- **Real rules follow the signal.** `Disallow: /api/` and `Disallow: /scan/` keep crawlers out of private scan results. The signal does not do that.
- **One group exists per AI crawler.** The file names GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot and Bytespider.

The SeenLens scan has a check for this line. See the fix guide [add Content Signals to robots.txt](https://seenlens.com/fixes/add-content-signals). The [AI crawler checker](https://seenlens.com/tools/ai-crawler-checker) also reads Content-Signal lines for any site.

## Should I add Content-Signal to my robots.txt?

Add it if you want your preference on record. It costs one line per group and no operator documents a harm. Do not expect it to change crawler behavior. A signal says what you allow. A Disallow rule is what a crawler actually obeys today.

1. Choose your three values. Most sites that want AI search set `search=yes, ai-input=yes`.
2. Build the line with the generator above, then copy it into each `User-agent:` group.
3. Keep Disallow rules for any crawler you want to stop.
4. Run the [AI crawler checker](https://seenlens.com/tools/ai-crawler-checker) to confirm the file parses and the signal shows.

## FAQ

### What is the Cloudflare Content Signals policy?

It is a robots.txt extension that Cloudflare published on 2025-09-24. A `Content-Signal` line states whether a site allows search, AI input and AI training. Cloudflare released the policy text under CC0.

### Do AI crawlers respect Content Signals?

No operator I checked documents it. OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl do not mention Content Signals on their crawler pages. Use Disallow rules to block.

### What happens if I leave out a signal?

The policy text says you neither grant nor restrict that use through Content Signals.

### Is Content Signals an internet standard?

No. It is Cloudflare's proposal. The IETF AIPREF working group is writing a standard vocabulary and a way to attach it, and its drafts were not RFCs when I checked.

### Does ai-train=no stop Google from using my content in AI Overviews?

Google's documentation does not list Content Signals. For AI Overviews, Google points to Googlebot rules and snippet controls such as `nosnippet` and `max-snippet`. Google-Extended does not affect Search.

## Sources

- [Cloudflare: Content Signals Policy](https://blog.cloudflare.com/content-signals-policy/) (announcement 2025-09-24, signal definitions, defaults, CC0, Article 4 statement, checked 2026-11-06)
- [Content Signals](https://contentsignals.org/) (policy text, examples, description as an IETF AIPREF guide, checked 2026-11-06)
- [Cloudflare Docs: managed robots.txt](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) (defaults, named crawlers in Disallow rules, Search Console message, checked 2026-11-06)
- [IETF datatracker: AI Preferences working group](https://datatracker.ietf.org/wg/aipref/about/) (charter, vocabulary and attachment drafts, checked 2026-11-06)
- [Google: common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) (Google-Extended, planned AIPREF support in early 2027, checked 2026-11-06)
- [Anthropic: crawler documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (no-AI-training signal based on an IETF draft, checked 2026-11-06)
- [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots), [Perplexity: crawlers](https://docs.perplexity.ai/guides/bots), [Apple: About Applebot](https://support.apple.com/en-us/119829) and [Common Crawl: CCBot](https://commoncrawl.org/ccbot) (no mention of Content Signals on 2026-11-06)
- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) (group rules, checked 2026-11-06)