How to check robots.txt

Request the file from the root of the host and read the status line together with the body: curl -sS -D - https://developer.mozilla.org/robots.txt. A usable file answers 200 with content-type: text/plain. A 404 means no restrictions apply, and a 5xx means crawlers treat the whole host as disallowed.

Why check this

Run this on staging sign-off, after any change that puts the site behind a different proxy or CDN, and after any deploy that touches routing. The failure it prevents is the file swap: a production deploy that ships the staging copy carrying Disallow: /, or a staging environment that ships the production copy and starts collecting crawler traffic on test data.

The check settles three things and no more. Whether the file is reachable at the one location crawlers look at, what status and content type it is served with, and what bytes it actually contains. It does not tell you whether a particular URL is allowed, which needs the rule matcher on the next page, and it tells you nothing about what any search engine has stored.

Prerequisites

• curl 7.0 or later. No HTTP/2 support is needed here. See the curl manual.
• The origin under test, written as the crawler sees it. A robots.txt applies to one scheme, host and port, so https://example.com, https://www.example.com and http://example.com each need their own.
• RFC 9309 for the field names, the status code rules and the parsing limit.

Steps

1. 1

   Step 1.

   Fetch the file and keep the response headers alongside it.

   curl -sS -D - -o mdn-robots.txt https://developer.mozilla.org/robots.txt \
     | grep -iE '^HTTP|^content-type|^content-length|^last-modified'
   

   HTTP/2 200 
   content-type: text/plain
   last-modified: Tue, 19 Aug 2025 16:24:09 GMT
   content-length: 119

   text/plain is the part to read. A server that answers text/html here is handing the crawler an error page with a 200 on it.

2. 2

   Step 2.

   Print the saved file with line endings shown, so nothing invisible passes.

   cat -A mdn-robots.txt
   

   User-agent: *$
   Sitemap: https://developer.mozilla.org/sitemap.xml$
   $
   Disallow: /api/$
   Disallow: /*/files/$
   Disallow: /media$

   Each $ is a bare LF. No ^M means no CRLF, and the first line starting at User-agent means no byte order mark ahead of it.

3. 3

   Step 3.

   Ask a host that publishes no rules, to see what absence looks like.

   curl -sS -o /dev/null -w '%{http_code} %{content_type} %{size_download}\n' https://example.com/robots.txt
   

   404 text/html 559

4. 4

   Step 4.

   Repeat the request on the other scheme, because that is a separate file as far as the rules go.

   curl -sS -o /dev/null -w '%{http_code} -> %{redirect_url}\n' http://developer.mozilla.org/robots.txt
   

   301 -> https://developer.mozilla.org/robots.txt

   A redirect to the same path on the secure origin is the answer you want. A redirect to the home page is not, and neither is a 200 serving different rules.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | 200 with content-type: text/plain | The rules in the body are the rules in force | Read them, then match your URLs against them | | 404 or 410 | No file, so nothing is restricted | Fine when that is the intent. On staging it is a leak | | 500, 502, 503, or a timeout | RFC 9309 section 2.3.1.4 tells crawlers to assume complete disallow | Treat as a site-wide outage, not a cosmetic defect | | 200 with content-type: text/html | A catch-all route answered instead of the file | Fix routing. The parser finds no rules and crawls everything | | 301 to a different host | The rules of the other host apply, not yours | Serve the file on the host under test |

Common mistakes

Thresholds

What to check next

• How to check if a url is blocked by robots.txt: the file is readable now, so match a real URL against its rules.
• How to check sitemap and robots txt: the Sitemap: line above points somewhere, and that target needs its own check.
• How to check x robots tag: a header on the page can restrict indexing even where robots.txt allows crawling.
• How to check if a staging site is indexable: the environment where a wrong robots.txt costs the most.
• How to check HTTP response headers with curl: the header reading technique used in step 1, on any response.

FAQ

How to check the robots txt of any website?

Append /robots.txt to the origin and request it. The file is public by definition, so no credentials and no permission are involved. Keep it to one request per host: reading somebody else's rules repeatedly is traffic they did not ask for.

How to see robots.txt of a website in a browser?

Type the URL in the address bar. The browser renders it as plain text when the content type is right, which doubles as a check: if the browser shows a styled page instead of a monospaced wall of text, the server is not sending text/plain.

How to verify robots txt after a deploy?

Compare the served bytes against the file in the repository. curl -sS https://host/robots.txt | diff - public/robots.txt prints nothing when they agree. That catches the wrong environment's file, which a visual read at a glance does not.

Does a missing robots.txt block anything?

No. RFC 9309 section 2.3.1.3 says a crawler receiving an unavailable status may access any resource on the server. A 404 is a valid, permissive answer. An unreachable server is the opposite case and is treated as a full disallow.

Verified