How to check x-robots-tag

X-Robots-Tag is an indexing directive sent as a response header, so it works on responses that carry no HTML. Read it with curl -sS -D - -o /dev/null https://host/file.pdf | grep -i x-robots-tag. A line reading x-robots-tag: noindex keeps that resource out of results. No line means no directive.

Why check this

Run this on any route that returns something other than a web page: exports, invoices, generated reports, API responses. Run it again after a CDN or proxy change, since headers are the part of a response that intermediaries rewrite.

One concrete failure it prevents. A reporting endpoint starts returning PDFs at guessable URLs. There is no HTML in the response, so a robots meta tag is not an option, and this header is the only way to keep those files out of results. A tester who checks the surrounding page and stops never looks at the file itself.

The header is also a separate channel from the HTML, so a page can be clean in view source and still carry a noindex.

Prerequisites

curl 7.0 or later. Any build works, since nothing here needs HTTP/2 locally.
Node 18 or later, for the local target below.
Google's X-Robots-Tag reference for the directive names and the user agent syntax.

Save this as index-server.cjs and start it with node index-server.cjs. It is the target for steps 1 to 4, and it is shared with How to check if a page is noindex.

// node index-server.cjs   ->   http://127.0.0.1:8791
const http = require('http');
const PORT = 8791;
const page = (head, body) =>
  `<!doctype html>\n<html lang="en">\n<head>\n<meta charset="utf-8">\n${head}\n<title>t</title>\n</head>\n<body>\n${body}\n</body>\n</html>\n`;
const PDF = Buffer.from(
  '%PDF-1.7\n1 0 obj<</Type/Catalog/Pages 2 0 R>>endobj\n' +
    '2 0 obj<</Type/Pages/Kids[3 0 R]/Count 1>>endobj\n' +
    '3 0 obj<</Type/Page/Parent 2 0 R/MediaBox[0 0 99 99]>>endobj\n' +
    'trailer<</Size 4/Root 1 0 R>>\n%%EOF\n',
  'latin1'
);
const routes = {
  '/robots.txt': [200, { 'content-type': 'text/plain' }, 'User-agent: *\nDisallow: /private/\n'],
  '/private/old-report.html': [200, {}, page('<meta name="robots" content="noindex">', '<h1>Old report</h1>')],
  '/ok.html': [200, {}, page('<link rel="canonical" href="http://127.0.0.1:8791/ok.html">', '<h1>Open</h1>')],
  '/report.pdf': [200, { 'content-type': 'application/pdf', 'x-robots-tag': 'noindex' }, PDF],
  '/api/items': [200, { 'content-type': 'application/json', 'x-robots-tag': 'noindex, nofollow' }, '{"items":[]}'],
  // Two X-Robots-Tag headers on one response, each naming a crawler.
  '/multi.html': [200, { 'x-robots-tag': ['otherbot: nofollow', 'googlebot: noindex'] }, page('', '<h1>Two rules</h1>')],
  '/head-differs.html': [200, {}, page('', '<h1>Depends on the method</h1>')],
  '/late.html': [200, {}, page('', '<p>x</p>\n<meta name="robots" content="noindex">\n' +
    '<script>document.head.insertAdjacentHTML("beforeend",' +
    '\'<meta name="robots" content="noindex,nofollow">\');</script>')],
  '/gone.html': [410, {}, page('', '<h1>Gone</h1>')],
};
http.createServer((req, res) => {
  const r = routes[req.url];
  if (!r) return res.writeHead(404, { 'content-type': 'text/html' }).end('not found');
  const h = { 'content-type': 'text/html; charset=utf-8', ...r[1] };
  // A header the app appends only while writing a body: HEAD and GET disagree.
  if (req.url === '/head-differs.html' && req.method !== 'HEAD') h['x-robots-tag'] = 'noindex';
  res.writeHead(r[0], h);
  res.end(req.method === 'HEAD' ? undefined : r[2]);
}).listen(PORT, '127.0.0.1', () => console.log('listening on ' + PORT));

Steps

1
Step 1.
Read the header on a file where a meta tag is not an option.
```
curl -sS -I http://127.0.0.1:8791/report.pdf
```
```
HTTP/1.1 200 OK
content-type: application/pdf
x-robots-tag: noindex
Date: Fri, 11 Sep 2026 20:39:34 GMT
Connection: keep-alive
Keep-Alive: timeout=5
```
Confirm the body really is not markup, so the reason for the header is on the page and not assumed.
```
curl -sS http://127.0.0.1:8791/report.pdf | head -c 9
```
```
%PDF-1.7
```
A PDF has nowhere to put a <meta> element. This header is the whole mechanism for that resource, which is the reason it exists.

Read it on a JSON response, where the same argument applies.

curl -sS -D - -o /dev/null http://127.0.0.1:8791/api/items | tr -d '\r' | grep -iE '^HTTP|^content-type|^x-robots-tag'

HTTP/1.1 200 OK
content-type: application/json
x-robots-tag: noindex, nofollow

Two rules in one comma separated value. none is the short form of this pair.

3
Step 3.
Print every instance of the header, not the first one.
```
curl -sS -D - -o /dev/null http://127.0.0.1:8791/multi.html | tr -d '\r' | grep -i '^x-robots-tag'
```
```
x-robots-tag: otherbot: nofollow
x-robots-tag: googlebot: noindex
```
Now run the same read the way it is usually written, with a head -1 on the end.
```
curl -sS -D - -o /dev/null http://127.0.0.1:8791/multi.html | tr -d '\r' | grep -i '^x-robots-tag' | head -1
```
```
x-robots-tag: otherbot: nofollow
```
The rule for Googlebot is on the second line, and the shortened pipeline never prints it. The header may repeat, and a value may name a crawler before the rules, in the form googlebot: noindex. A read that keeps one line reports the wrong verdict for every crawler but the first.
4
Step 4.
Compare a HEAD request with a GET on the same URL.
```
curl -sS -I http://127.0.0.1:8791/head-differs.html | tr -d '\r' | grep -iE '^HTTP|^x-robots-tag'
```
```
HTTP/1.1 200 OK
```
```
curl -sS -D - -o /dev/null http://127.0.0.1:8791/head-differs.html | tr -d '\r' | grep -iE '^HTTP|^x-robots-tag'
```
```
HTTP/1.1 200 OK
x-robots-tag: noindex
```
Same URL, same status, different headers. The target adds the header only when a body is written, which is how middleware is often wired. curl -I is the fastest header read and the easiest way to miss this one. Crawlers issue GET.
5
Step 5.
Read the header over HTTP/2 on a public host, then read it again with a capital letter in the pattern.
```
curl -sS -D - -o /dev/null 'https://httpbin.org/response-headers?X-Robots-Tag=noindex%2C%20nofollow' | tr -d '\r' | grep -iE '^HTTP|^content-type|robots'
```
```
HTTP/2 200 
content-type: application/json
x-robots-tag: noindex, nofollow
```
```
curl -sS -D - -o /dev/null 'https://httpbin.org/response-headers?X-Robots-Tag=noindex%2C%20nofollow' | tr -d '\r' | grep 'X-Robots-Tag'
```
```
(no output; grep exits 1)
```
The endpoint echoes the query string back as a response header, so the value is the one requested. The case of the name is not: HTTP/2 carries header names in lowercase, and curl prints what arrived. The second pipeline asks for X-Robots-Tag and finds nothing on a response that carries it. Use grep -i on every header read.
6
Step 6.
Check the absence on a real page. Save the headers with one request.
```
curl -sS -D mdn-head.txt -o mdn-body.html https://developer.mozilla.org/en-US/docs/Web/HTTP
```
```
(no output; the headers are saved to mdn-head.txt)
```
```
grep -i 'x-robots-tag' mdn-head.txt; echo "exit=$?"
```
```
exit=1
```
developer.mozilla.org sends no X-Robots-Tag on its HTTP documentation page, read on 2026-09-11. Absence is the normal result, and it means something only once step 5 has ruled out a case sensitive pattern.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | x-robots-tag: noindex | This resource is kept out of results | Confirm it is intended for this URL and this content type | | x-robots-tag: none | Equivalent to noindex, nofollow | Treat as both rules at once | | x-robots-tag: googlebot: noindex | The rule applies to that crawler only | Read every instance of the header before deciding | | Two headers with different rules | Each names its own crawler | Google applies the sum of the negative rules to a crawler that matches more than one | | Nothing, with grep -i | No header directive on this response | Check the HTML and the status as well | | Present on GET, absent on HEAD | The application adds it on the body path | Fix it at the routing layer. Crawlers use GET, so the rule does apply |

Common mistakes

Sign: A header read returns one X-Robots-Tag line and the crawler behaves as though a different rule were in force.Cause: The header may appear more than once on a single response, each instance naming its own crawler. A pipeline ending in head -1 prints whichever the server sent first, which step 3 showed is not the one that governs Googlebot. Print every instance.

Sign: curl -I shows no X-Robots-Tag and a crawl report says the page is noindex.Cause: The header is added on the response path that writes a body, so HEAD and GET return different header sets. One line of the target above reproduces it. Read headers with -D - and -o /dev/null, which sends a GET and discards the body.

Sign: A grep for X-Robots-Tag finds nothing, and the header is there.Cause: Header field names are case-insensitive, and over HTTP/2 they arrive lowercase, so a pattern written with capitals matches the documentation and not the wire. This is not specific to this header, and it silently turns every header assertion in a test suite into a pass.

Sign: The header is set correctly on the HTML page and the linked PDF is still in the index.Cause: The directive belongs to the response it is sent on, and nothing propagates from a page to the files it links. Every downloadable resource needs its own header, applied at the route or the storage layer that serves it.

What to check next

How to check if a page is noindex: the other channel, for responses that do carry HTML.
How to check if a page is indexable: combines this header with status, canonical and crawl access.
How to check HTTP response headers with curl: the header reading technique used in every step here.
How to check robots.txt: a crawl block stops the fetch, and this header is then never read.
How to check if a staging site is indexable: where a blanket header is the usual control.

FAQ

Noindex detected in x-robots-tag http header: what does it mean?

A crawler read X-Robots-Tag: noindex on that response, so it will not show the URL in results. Reproduce it with step 1 against the URL named in the report, then find the middleware, proxy or storage setting that adds it.

How to check the x-robots-tag of a PDF?

Request the PDF and read the headers without downloading it: curl -sS -D - -o /dev/null https://host/file.pdf | grep -i x-robots-tag. Use a GET, as step 4 showed, because a HEAD can return a different header set on the same URL.

Which rule wins when the header and the meta tag disagree?

Google's reference states that in the case of conflicting robots rules, the more restrictive rule applies. A header saying noindex and a meta tag saying index resolve to noindex. Read both before reporting either.

Can X-Robots-Tag target one crawler?

Yes. Write the product token, a colon, then the rules: X-Robots-Tag: googlebot: noindex. Rules sent with no user agent named apply to every crawler. Step 3 shows two such headers on one response and what a truncated read does with them.

Verified

Verified 2026-09-11by Maks Vernycurl 8.21.0node 22.23.2

Each output block is what the command above it printed on that date, on the host named in the step. Figures read from a live site move between runs. Compare the shape of the answer rather than the digits, and see the methodology for how a page is re-verified.

All crawlability and indexing checkscluster

intermediate6 minpublished 2026-09-11updated 2026-09-11Maks Verny