---
title: "Fix: robots.txt Not Found (404, 5xx or Missing)"
description: "A missing robots.txt states no policy, and a 5xx error can stop crawlers. See what each status means and how to publish a working robots.txt file."
url: https://seenlens.com/fixes/robots-txt-missing
last_updated: 2026-11-06
---

# robots.txt not found: how to fix it

A robots.txt file is missing when /robots.txt returns 404 or a server error. A 404 means crawlers assume everything is allowed and you state no AI policy. A 5xx error is worse, because compliant crawlers may stop crawling your site. Publishing the file takes about 10 minutes.

By SeenLens. Last updated: 2026-11-06

## What this check tests

SeenLensBot requests `/robots.txt` at the root of your host. The status code decides the result.

| Answer | Result | What crawlers do |
| --- | --- | --- |
| 2xx | Pass | They read the file |
| 4xx, such as 404 | Warn | Google assumes no crawl restrictions. RFC 9309 says a crawler may access any resource |
| 5xx | Fail | RFC 9309 says a crawler must assume complete disallow |
| No answer or network error | Warn | Crawlers cannot read a policy |

Google handles a 5xx answer in steps. For the first 12 hours it stops crawling the site and keeps retrying. For up to 30 days it uses the last good copy. If there is no cached copy, it assumes no restrictions. Google caches robots.txt for up to 24 hours.

## How to confirm the problem

```text
curl -sS -I -L https://example.com/robots.txt
```

Read the first status line. Also open the file in a private browser window. It must show plain text rules, not an HTML page. Each host needs its own file, so check `www.example.com` and `example.com` separately. The embedded AI crawler checker reads the file and shows the rules it finds.

## How to fix it

Create a UTF-8 plain text file named `robots.txt` at the root of the host. Google reads up to 500 KiB. This starter file allows all crawlers and names your sitemap:

```text
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
```

### Next.js (App Router)

Add `app/robots.ts`. The convention exists since Next.js 13.3 (docs updated 2026-05-01).

```text
import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: "*", allow: "/" },
    sitemap: "https://example.com/sitemap.xml",
  };
}
```

### Shopify

Shopify generates a default robots.txt. If yours is missing, check the domain and any proxy in front of the store. To change the rules, add a `robots.txt.liquid` template in the theme code editor. That template replaces the generated file, so save a copy of the current output first.

### Cloudflare

With Bot Preference Sync on and a block policy set, Cloudflare serves managed rules. If your origin has no file, Cloudflare creates one. If your origin returns a file with status 200, Cloudflare puts the managed rules before it. Fix a 5xx at the origin instead.

### Nginx, Apache, WordPress and static hosts

Put the file in the web root, the folder that holds `index.html` or, for WordPress, `wp-config.php`. Upload it by SFTP or the host's file manager. For a 5xx answer, read the server error log, because the file itself is rarely the cause.

## How to verify the fix

1. Run the curl command. Expect status 200 and a content type that starts with `text/plain`.
2. Open the [AI crawler checker](https://seenlens.com/tools/ai-crawler-checker) and confirm it shows your rules.
3. In Google Search Console, open the robots.txt report to see the fetched copy.

[Run the free SeenLens scan on your page](https://seenlens.com/)

## When can you ignore it?

A 404 is acceptable if you want every crawler to access everything and you do not want to state an AI preference. A 5xx answer is never acceptable.

## Related checks

- [robots.txt blocks AI crawlers](https://seenlens.com/fixes/robots-txt-blocks-ai-crawlers)
- [Add Content-Signal lines](https://seenlens.com/fixes/add-content-signals)
- [Write robots.txt for AI crawlers](https://seenlens.com/guides/robots-txt-for-ai-crawlers)
- [Crawlers cannot load the page](https://seenlens.com/fixes/ai-crawlers-cannot-load-page)

## Sources

- [Google Search Central: how Google interprets robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) (status code handling, cache, size limit, checked 2026-11-06)
- [Google Search Central: create a robots.txt file](https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt) (location per host, example, checked 2026-11-06)
- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) (unavailable and unreachable robots.txt, checked 2026-11-06)
- [Next.js: robots.txt](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots) (robots.ts, updated 2026-05-01)
- [Shopify: robots.txt.liquid](https://shopify.dev/docs/storefronts/themes/seo/robots-txt) (default file and custom template, checked 2026-11-06)
- [Cloudflare: robots.txt setting](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) (managed robots.txt behaviour, checked 2026-11-06)