Skip to content
SeenLens

robots.txt not found: how to fix it

A robots.txt file is missing when /robots.txt returns 404 or a server error. A 404 means crawlers assume everything is allowed and you state no AI policy. A 5xx error is worse, because compliant crawlers may stop crawling your site. Publishing the file takes about 10 minutes.

By SeenLens · Last updated

Free, no sign-up. We read robots.txt and request the page with a browser and four crawler user agents. We do not store your page text.

What this check tests

SeenLensBot requests /robots.txt at the root of your host. The status code decides the result.

How the robots-reachable check scores each answer
AnswerResultWhat crawlers do
2xxPassThey read the file
4xx, such as 404WarnGoogle assumes no crawl restrictions. RFC 9309 says a crawler may access any resource
5xxFailRFC 9309 says a crawler must assume complete disallow
No answer or network errorWarnCrawlers cannot read a policy

Google handles a 5xx answer in steps. For the first 12 hours it stops crawling the site and keeps retrying. For up to 30 days it uses the last good copy. If there is no cached copy, it assumes no restrictions. Google caches robots.txt for up to 24 hours.

How to confirm the problem

curl
curl -sS -I -L https://example.com/robots.txt

Read the first status line. Also open the file in a private browser window. It must show plain text rules, not an HTML page. Each host needs its own file, so check www.example.com and example.com separately. The embedded AI crawler checker reads the file and shows the rules it finds.

How to fix it

Create a UTF-8 plain text file named robots.txt at the root of the host. Google reads up to 500 KiB. This starter file allows all crawlers and names your sitemap:

robots.txt
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Next.js (App Router)

Add app/robots.ts. The convention exists since Next.js 13.3 (docs updated 2026-05-01).

app/robots.ts
import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: "*", allow: "/" },
    sitemap: "https://example.com/sitemap.xml",
  };
}

Shopify

Shopify generates a default robots.txt. If yours is missing, check the domain and any proxy in front of the store. To change the rules, add a robots.txt.liquid template in the theme code editor. That template replaces the generated file, so save a copy of the current output first.

Cloudflare

With Bot Preference Sync on and a block policy set, Cloudflare serves managed rules. If your origin has no file, Cloudflare creates one. If your origin returns a file with status 200, Cloudflare puts the managed rules before it. Fix a 5xx at the origin instead.

Nginx, Apache, WordPress and static hosts

Put the file in the web root, the folder that holds index.html or, for WordPress, wp-config.php. Upload it by SFTP or the host's file manager. For a 5xx answer, read the server error log, because the file itself is rarely the cause.

How to verify the fix

  1. Run the curl command. Expect status 200 and a content type that starts with text/plain.
  2. Open the AI crawler checker and confirm it shows your rules.
  3. In Google Search Console, open the robots.txt report to see the fetched copy.

Run the free SeenLens scan on your page

When can you ignore it?

A 404 is acceptable if you want every crawler to access everything and you do not want to state an AI preference. A 5xx answer is never acceptable.

Sources