--- title: "What Are Content Signals? Definition and robots.txt Syntax" description: "Content signals are Content-Signal lines in robots.txt that state whether content may be used for search, AI input or AI training. See syntax and limits." url: https://seenlens.com/glossary/content-signals last_updated: 2026-11-06 --- # What are content signals? Content signals are Content-Signal lines in robots.txt that state whether your content may be used for search, AI input and AI training. Cloudflare published the Content Signals Policy on 24 September 2025. The signals are preferences that no crawler has to obey, and an IETF working group is developing a related standard. By SeenLens. Last updated: 2026-11-06 ## Definition A content signal is a machine-readable statement about how a site's content may be used after it is fetched. Robots.txt rules say whether a crawler may fetch a path. A content signal says what the operator allows the crawler to do with the content. Cloudflare released the policy under a CC0 license, so anyone may use it. ## How do content signals work? | Signal | Meaning | | --- | --- | | `search` | Building a search index and returning results such as links and short excerpts. It excludes AI-generated search summaries. | | `ai-input` | Feeding content into AI models, for example retrieval augmented generation, grounding or other real-time use for AI search answers. | | `ai-train` | Training or fine-tuning AI models. | ```text User-Agent: * Content-Signal: search=yes, ai-train=no Allow: / ``` A value of yes means the use is allowed. A value of no means the use is not allowed. A missing signal has no meaning: it does not say you have no preference, only that you did not state one in robots.txt. The example above allows search, opts out of training and states nothing about `ai-input`. ## Who uses content signals and who does not? - **Cloudflare.** When a customer turns on the managed robots.txt feature, Cloudflare serves `Content-Signal: search=yes, ai-train=no`. At the time of the announcement, over 3.8 million domains had the feature on. Cloudflare does not serve an `ai-input` value for them. - **Cloudflare Markdown for Agents.** Converted Markdown responses carry a `content-signal` header. If your server sets none, the default is `ai-train=yes, search=yes, ai-input=yes`. - **Crawler operators.** No complete list of crawlers that honor the signals exists. Cloudflare says the signals express preferences and are not technical countermeasures, and that some companies may ignore them. Google's documentation for AI features does not mention them. - **Standards work.** The IETF AI Preferences (aipref) working group is developing a formal vocabulary on related ideas. Names may still change before it publishes. Cloudflare's policy text also states that the signals are an express reservation of rights under Article 4 of EU Directive 2019/790. Ask a lawyer what that means for your site. For enforcement, use firewall rules. ## How do I check and set content signals? The [AI crawler checker](https://seenlens.com/tools/ai-crawler-checker) reads the `Content-Signal` lines in your robots.txt. The [Content Signal generator](https://seenlens.com/tools/content-signal-generator) writes a block for you. The fix page [add Content Signals](https://seenlens.com/fixes/add-content-signals) covers the scan check that reports a missing preference. Put the line inside each `User-agent` group. ## Related terms - [llms.txt](https://seenlens.com/glossary/llms-txt): a different file that lists pages and states no usage preference. - [Markdown content negotiation](https://seenlens.com/glossary/markdown-content-negotiation): the feature that adds a `content-signal` header to Markdown. - [Content Signals in robots.txt](https://seenlens.com/guides/content-signals-robots-txt) and [robots.txt for AI crawlers](https://seenlens.com/guides/robots-txt-for-ai-crawlers). - [The list of AI crawlers](https://seenlens.com/bots). ## Frequently asked questions ### What does Content-Signal ai-train=no mean? It states that you do not want your content used to train or fine-tune AI models. It is a request, not a block. ### Who invented content signals? Cloudflare published the Content Signals Policy on 24 September 2025. The IETF AI Preferences working group is developing a related standard. ### Is Content-Signal an official standard? No. It is a published policy with a CC0 license. The IETF AI Preferences work on a related standard is still in progress. ### Does a content signal block a crawler? No. Only a `Disallow` rule in robots.txt, or a firewall rule, blocks a crawler that complies with it. A signal states a preference. ### Where do I put the Content-Signal line? Inside a `User-agent` group in robots.txt, next to the `Allow` and `Disallow` lines of that group. ## Sources - [Cloudflare: Giving users choice with Cloudflare's new Content Signals Policy](https://blog.cloudflare.com/content-signals-policy/) (signals, syntax, managed robots.txt, 3.8 million domains, CC0, 24 September 2025, checked 2026-11-06) - [contentsignals.org](https://contentsignals.org/) (guide to the IETF AI Preferences work, checked 2026-11-06) - [Cloudflare docs: Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) (default content-signal header, checked 2026-11-06) - [Google: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) (no mention of Content Signals, checked 2026-11-06)