---
title: "Fix: Add Content-Signal to robots.txt (ai-train=no)"
description: "Add a Content-Signal line to robots.txt to state whether AI systems may use your content for search, answers and training. Syntax, limits and a generator."
url: https://seenlens.com/fixes/add-content-signals
last_updated: 2026-11-06
---

# Content-Signal in robots.txt: how to add it

A Content-Signal line in robots.txt states whether AI systems may use your content for search, AI answers and model training. Add one line per group, such as search=yes, ai-input=yes, ai-train=no. It states a preference and does not enforce it. The edit takes about five minutes.

By SeenLens. Last updated: 2026-11-06

## What this check tests

SeenLens reads the Content-Signal lines in your robots.txt and checks their syntax.

| Result | What SeenLens found |
| --- | --- |
| Info | No Content-Signal line |
| Warn | A malformed entry, an unknown key, a value other than yes or no, a line outside any group, or a named AI group without the line while the `*` group has one |
| Pass | Valid lines inside groups, and no gaps |

The format comes from Cloudflare's Content Signals Policy, published on 2025-09-24. A line has the form `Content-Signal:` followed by comma separated pairs. The key `search` covers building a search index and showing results. The key `ai-input` covers feeding content into AI models, for example for retrieval or grounding. The key `ai-train` covers training or fine tuning models. The value `yes` permits the use, `no` forbids it, and an omitted key states no preference. The scanner also accepts Cloudflare's experimental `use` key.

## How to confirm the problem

```text
curl -s https://example.com/robots.txt | grep -n -i "content-signal"
```

No output means the file has no line. The [Content Signal generator](https://seenlens.com/tools/content-signal-generator) above builds a correct block from a form.

## How to fix it

Put the line inside each User-agent group. A crawler that matches a named group reads that group, so repeat the line in every named group.

```text
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

User-agent: GPTBot
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /
```

- **Cloudflare:** go to Security, Settings, filter by Bot traffic, open AI Crawler and Agent access policies, set Training to Disallow, turn on Bot Preference Sync and save. Cloudflare then writes a managed block with `search=yes, ai-train=no` and a `use=reference` extension. It puts that block before your own file.
- **Next.js:** add a static `app/robots.txt`. From Next.js 16.3.0 the `other` field of a `robots.ts` rule can print non standard lines such as `Content-Signal`.
- **Shopify:** edit `robots.txt.liquid`, and add the line to the group you want.
- **WordPress, Nginx, Apache:** edit the `robots.txt` file in the web root.

## Does Google follow Content-Signal?

No. Search Engine Roundtable reported on 2026-07-06 that John Mueller of Google said the directive has no effect on any crawler or LLM. He wrote this in a Reddit reply in r/TechSEO. The line is a published preference, and each AI company decides whether to honor it. For control that works, use named robots.txt groups for training crawlers, a firewall rule, or both. Google's own controls for AI features in Search are `nosnippet`, `max-snippet` and `noindex`; see [the noindex fix](https://seenlens.com/fixes/noindex-blocks-ai-answers). Bing documents a different training token, `__usage_ai-training`; see [Bing AI grounding and AI training](https://seenlens.com/guides/bing-ai-grounding-and-training).

## How to verify the fix

1. Fetch robots.txt and find the line inside each group.
2. Open the [AI crawler checker](https://seenlens.com/tools/ai-crawler-checker). It lists the Content-Signal lines it reads.
3. Run the SeenLens scan. The content-signal check should pass.

[Run the free SeenLens scan on your page](https://seenlens.com/)

## When can you ignore it?

Skip it if you do not want to state a preference or if you rely on named groups and a firewall. The check has a low weight and reports Info when the line is absent.

## Related checks

- [robots.txt blocks AI crawlers](https://seenlens.com/fixes/robots-txt-blocks-ai-crawlers)
- [robots.txt is missing](https://seenlens.com/fixes/robots-txt-missing)
- [Content signals in robots.txt, a guide](https://seenlens.com/guides/content-signals-robots-txt)
- [Glossary: Content Signals](https://seenlens.com/glossary/content-signals)

## Sources

- [Cloudflare blog: Content Signals Policy](https://blog.cloudflare.com/content-signals-policy/) (syntax, the three signals, absent key, published 2025-09-24)
- [Cloudflare: robots.txt setting](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) (managed directives, defaults, append behaviour, checked 2026-11-06)
- [Search Engine Roundtable: Google on Content Signals](https://www.seroundtable.com/google-cloudflare-content-signals-41631.html) (Mueller comment, reported 2026-07-06)
- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) (group matching, checked 2026-11-06)
- [Next.js: robots.txt](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots) (the `other` field added in v16.3.0, updated 2026-05-01)