Skip to content
SeenLens

ChatGPT can't access my website: how to fix it

ChatGPT cannot access your website when a plain request for your page returns an error, a bot challenge or a long redirect chain. The usual causes are a firewall rule, a server error or a wrong redirect. Fixing the cause lets AI crawlers read the page. Most fixes take under one hour.

By SeenLens · Last updated

Free, no sign-up. We read robots.txt and request the page with a browser and four crawler user agents. We do not store your page text.

Why can't ChatGPT access my website?

OpenAI runs three crawlers. OAI-SearchBot finds pages for ChatGPT search. GPTBot collects training data. ChatGPT-User fetches a page when a person asks ChatGPT to open it. All three need your server to answer a plain request with the page. Perplexity and Anthropic crawlers work the same way.

Common causes of a page that AI crawlers cannot load
What the crawler seesLikely causeWhere to fix it
HTTP 403, 429 or 503, often with a small bodyFirewall or bot protectionFirewall and Cloudflare fix
HTTP 200 and an empty pageContent needs JavaScriptJavaScript fix
HTTP 500 to 599Server error or timeoutSection below
HTTP 404, or a redirect to a login pageWrong URL, or a page behind a loginSection below
Page loads, but the crawler stays awayA robots.txt rulerobots.txt fix

page-reachable: can a plain request load the page?

What this check tests

SeenLensBot sends one GET request for your page. It uses a browser style user agent that ends with the SeenLensBot token, follows redirects and runs no JavaScript. The check passes when the answer is below HTTP 400 and no bot challenge appears. It fails when the request times out, returns HTTP 400 or higher, or returns a challenge page from a firewall.

How to confirm the problem

Run this command. It prints the final status code, the size of the answer and the number of redirects. A small size next to a 403 or 503 points to a firewall.

curl
curl -sS -o /dev/null -L -w "status %{http_code}, %{size_download} bytes, %{num_redirects} redirects\n" https://example.com/your-page

Then repeat it with -A "OAI-SearchBot" and compare. The embedded AI crawler checker above does the same comparison for several crawlers.

How to fix it

  • HTTP 403, 429 or 503: a firewall or bot rule blocks plain clients. Follow the firewall fix.
  • HTTP 500 to 599: read your server error log for the same time. Fix the failing code, database call or upstream service.
  • Timeouts: the origin answers too slowly. Check server load, cold starts and slow database queries.
  • HTTP 404: the URL is wrong or the page moved. Link to the live URL and add a 301 redirect from the old one.
  • Redirect to a login page: a crawler cannot sign in. Public pages must open without a login.

http-status: does the page return status 200?

What this check tests

How the http-status check scores the final response
ResultWhat SeenLens saw
PassStatus 200 after at most two redirects
WarnA status below 400 that is not 200 (for example 202 or 204), or more than two redirects
FailStatus 400 or higher, or no status at all

Google follows up to 10 redirect hops and treats 301 and 308 as strong signals. Other crawlers may stop sooner, so a short chain is safer.

How to fix it

Link to the final URL and send each old address to it in one hop. This Nginx example sends HTTP and www requests straight to the HTTPS apex URL:

nginx
server {
    listen 80;
    server_name example.com www.example.com;
    return 301 https://example.com$request_uri;
}

In Apache, use Redirect permanent "/" "https://example.com/" inside the virtual host. In Next.js, set permanent: true on a redirect in next.config.js; it sends status 308.

How to verify the fix

  1. Run the curl command again. Expect status 200 and at most one redirect.
  2. Run the AI crawler checker on the page. The status codes for each crawler should match the browser.
  3. Check your access log for the crawler names. On a combined log format, grep -E "OAI-SearchBot|GPTBot|ChatGPT-User" access.log | awk '{print $9}' | sort | uniq -c counts the status codes they received.

Run the free SeenLens scan on your page

When can you ignore it?

Ignore a failure on a page you want to keep private, such as a staging site, an admin page or a page behind a login. A crawler that cannot load those pages is the correct result.

Sources