How to check if a page is noindex

A page is noindex when the served response carries the directive, in the X-Robots-Tag header or in a robots meta tag inside <head>. Run node robots-meta.cjs https://host/page. It prints the header value and every robots meta with the element it sits in, so placement is visible, not assumed.

Checker offline. Follow the manual steps below, they give the same answer.

Why check this

Run this on every URL a release adds or moves, and on the whole staging host before anyone outside the team sees it. Two failures it prevents point in opposite directions. A staging template ships to production with noindex still in it, and the new section never enters any index. Or the directive leaves the template while one route keeps its own copy, and an internal search page starts collecting traffic.

A directive can live in three places: the response header, the served HTML, and the DOM after scripts run. Reading one and stopping is how a page passes review and fails in production.

Prerequisites

Node 18 or later, for the reader below and for the local target.
curl 7.0 or later. Any build works, since nothing needs HTTP/2.
Google's robots meta tag reference for the directive names.
A browser with DevTools for step 5. The DOM figure is one capture on one machine.

Save this as index-server.cjs and start it with node index-server.cjs. It is the target for every step, and it is shared with How to check x-robots-tag.

// node index-server.cjs   ->   http://127.0.0.1:8791
const http = require('http');
const PORT = 8791;
const page = (head, body) =>
  `<!doctype html>\n<html lang="en">\n<head>\n<meta charset="utf-8">\n${head}\n<title>t</title>\n</head>\n<body>\n${body}\n</body>\n</html>\n`;
const PDF = Buffer.from(
  '%PDF-1.7\n1 0 obj<</Type/Catalog/Pages 2 0 R>>endobj\n' +
    '2 0 obj<</Type/Pages/Kids[3 0 R]/Count 1>>endobj\n' +
    '3 0 obj<</Type/Page/Parent 2 0 R/MediaBox[0 0 99 99]>>endobj\n' +
    'trailer<</Size 4/Root 1 0 R>>\n%%EOF\n',
  'latin1'
);
const routes = {
  '/robots.txt': [200, { 'content-type': 'text/plain' }, 'User-agent: *\nDisallow: /private/\n'],
  // Blocked in robots.txt AND carrying a noindex the crawler may never fetch.
  '/private/old-report.html': [200, {}, page('<meta name="robots" content="noindex">', '<h1>Old report</h1>')],
  '/ok.html': [200, {}, page('<link rel="canonical" href="http://127.0.0.1:8791/ok.html">', '<h1>Open</h1>')],
  '/report.pdf': [200, { 'content-type': 'application/pdf', 'x-robots-tag': 'noindex' }, PDF],
  '/api/items': [200, { 'content-type': 'application/json', 'x-robots-tag': 'noindex, nofollow' }, '{"items":[]}'],
  '/multi.html': [200, { 'x-robots-tag': ['otherbot: nofollow', 'googlebot: noindex'] }, page('', '<h1>Two rules</h1>')],
  '/head-differs.html': [200, {}, page('', '<h1>Depends on the method</h1>')],
  // Directive in <body>, plus one injected into <head> by script after parse.
  '/late.html': [200, {}, page('', '<p>x</p>\n<meta name="robots" content="noindex">\n' +
    '<script>document.head.insertAdjacentHTML("beforeend",' +
    '\'<meta name="robots" content="noindex,nofollow">\');</script>')],
  '/gone.html': [410, {}, page('', '<h1>Gone</h1>')],
};
http.createServer((req, res) => {
  const r = routes[req.url];
  if (!r) return res.writeHead(404, { 'content-type': 'text/html' }).end('not found');
  const h = { 'content-type': 'text/html; charset=utf-8', ...r[1] };
  // A header the app appends only while writing a body: HEAD and GET disagree.
  if (req.url === '/head-differs.html' && req.method !== 'HEAD') h['x-robots-tag'] = 'noindex';
  res.writeHead(r[0], h);
  res.end(req.method === 'HEAD' ? undefined : r[2]);
}).listen(PORT, '127.0.0.1', () => console.log('listening on ' + PORT));

Save this as robots-meta.cjs. It reports both wire signals and, for each meta, the element it was found in.

// node robots-meta.cjs <url>
const url = process.argv[2];
fetch(url, { redirect: 'manual' }).then(async (r) => {
  const html = await r.text();
  console.log(`status:       ${r.status}`);
  console.log(`x-robots-tag: ${r.headers.get('x-robots-tag') ?? '(absent)'}`);
  const headEnd = html.search(/<\/head>/i);
  const re = /<meta[^>]+name=["']?(robots|googlebot)["']?[^>]*>/gi;
  let m, n = 0;
  while ((m = re.exec(html))) {
    const where = headEnd !== -1 && m.index < headEnd ? 'head' : 'body';
    console.log(`meta[${++n}]:      ${m[0]}   (byte ${m.index}, inside <${where}>)`);
  }
  if (!n) console.log('meta:         (none in the served HTML)');
});

Steps

1
Step 1.
Establish what a page with no directive looks like, so the positive result later means something.
```
node robots-meta.cjs http://127.0.0.1:8791/ok.html
```
```
status:       200
x-robots-tag: (absent)
meta:         (none in the served HTML)
```
Both lines have to be read. An empty HTML result is not a verdict on its own, because the header is a second, independent channel.
2
Step 2.
Run the same reader against a page that does carry the directive.
```
node robots-meta.cjs http://127.0.0.1:8791/private/old-report.html
```
```
status:       200
x-robots-tag: (absent)
meta[1]:      <meta name="robots" content="noindex">   (byte 63, inside <head>)
```
Byte 63 and inside <head> are the two facts that matter. The directive is in the served bytes, before </head>, so any consumer that parses the response sees it without rendering anything.
3
Step 3.
Read the robots.txt of the same origin before drawing a conclusion from step 2.
```
curl -sS http://127.0.0.1:8791/robots.txt
```
```
User-agent: *
Disallow: /private/
```
The URL from step 2 sits under /private/. The reader saw the noindex only because it ignores robots.txt. A crawler that obeys the file never issues that request, so the directive is never read. Google states the case directly: "If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored." The crawl block wins by making the indexing rule unreachable, not by outranking it. Confirm which rule covers the path with How to check if a url is blocked by robots.txt.
4
Step 4.
Run the reader against a page where the directive is not in <head>.
```
node robots-meta.cjs http://127.0.0.1:8791/late.html
```
```
status:       200
x-robots-tag: (absent)
meta[1]:      <meta name="robots" content="noindex">   (byte 105, inside <body>)
meta[2]:      <meta name="robots" content="noindex,nofollow">   (byte 198, inside <body>)
```
Two hits, and neither is what it looks like. The first is a real element, in <body>. The second is not an element: it is a string literal inside a <script>, which a regex over the bytes cannot tell from markup. Only the DOM settles it.

Open the same URL in the browser and ask the DOM where each tag ended up.

[...document.querySelectorAll('meta[name="robots"]')].map(
  (m) => `content="${m.getAttribute('content')}" parent=<${m.parentElement.tagName.toLowerCase()}>`
)

dom meta[1]: content="noindex,nofollow" parent=<head>
dom meta[2]: content="noindex" parent=<body>

document.head.innerHTML: <meta charset="utf-8"> <title>t</title> <meta name="robots" content="noindex,nofollow">

The two readings disagree, and both are correct. On the wire there is one tag, in <body>. In the DOM there are two, and the one in <head> was put there by script after the parse. Anything that does not render sees only the body tag.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | meta[1] with noindex, inside <head> | The page is noindex for anything that parses the response | Confirm it is intended, then check the header too | | meta: (none in the served HTML) and x-robots-tag: (absent) | No indexing directive on this response | The page is not noindex. Other blockers may still apply | | inside <body> | The tag is outside the place the reference names | Move it into <head>, where every consumer looks | | A directive that appears only in the DOM | It was injected by script and is absent from the served bytes | Serve it in the HTML instead of adding it at runtime | | content="none" | Equivalent to noindex, nofollow | Treat exactly as a noindex, and expect links to stop being followed |

Common mistakes

Sign: A grep over the HTML finds a robots meta tag, and the page is not noindex.Cause: The match was the text of a string inside a script, or a tag in a commented-out block. Byte matching cannot distinguish markup from a string that looks like markup. Step 4 produced exactly this: two matches, one element. Confirm every hit against the DOM before acting on it.

Sign: The tag sits in the body, and a tester reports it as ignored, citing the requirement that it live in the head.Cause: Google's reference does say to place the tag in the head section, and it also says Google Search does not enforce that placement and will respect a robots meta tag in the body. The tag is therefore honoured by Google and missed by tooling that only parses the head, which is most of it. The defect is real and it is a tooling defect, not an indexing one.

Sign: A script removes the noindex after load, the DOM is clean, and the page still does not appear.Cause: Google documents that when it encounters the noindex tag it may skip rendering and JavaScript execution, so a directive removed by script may never be observed as removed. A noindex in the original HTML is the one to delete.

Sign: The page under test is blocked in robots.txt and carries a noindex, and the URL still turns up as a bare result.Cause: The two rules cancel each other. The crawl block stops the fetch, so the noindex is never read, and the URL can still be listed from external signals. Remove the robots.txt rule so the noindex can be fetched, or drop the noindex and rely on the crawl block alone.

What to check next

How to check x-robots-tag: the second channel, and the only one on a response with no HTML.
How to check if a page is indexable: this result plus status, canonical and crawl access, as one verdict.
How to check if a url is blocked by robots.txt: needed before step 2 means anything, as step 3 showed.
How to check if a staging site is indexable: where a missing noindex costs the most.
How to check if googlebot can render javascript: the fate of a directive that exists only after scripts run.

FAQ

How to check a noindex tag on a page?

Fetch the response and read the header and the HTML in one pass, as in step 1. View source shows the served bytes, while the Elements panel shows the DOM, so the two are not interchangeable on a page that rewrites its own head.

How to check the robots meta tag without a crawler tool?

curl -sS https://host/page | grep -io '<meta name="robots"[^>]*>' is enough for a single URL. It cannot tell an element from a string inside a script, which step 4 demonstrated, so treat a hit as a lead rather than a verdict.

How to prevent search engines from indexing a single page of my website?

Serve <meta name="robots" content="noindex"> in the <head> of that page, or send X-Robots-Tag: noindex on its response. Leave the URL crawlable. Adding a robots.txt rule as well stops the directive from being read.

Does a noindex page still get crawled?

Yes. noindex governs whether the page appears in results, not whether it is fetched, and it has to be fetched for the rule to be seen. Use nofollow for links and robots.txt for the fetch itself.

Verified

Verified 2026-09-11by Maks Vernynode 22.23.2curl 8.21.0Chrome 152.0.7977.76

Each output block is what the command above it printed on that date, on the host named in the step. Figures read from a live site move between runs. Compare the shape of the answer rather than the digits, and see the methodology for how a page is re-verified.

Checker: meta-tags title, description, canonical, robots meta, hreflang, Open Graph, Twitter card, viewportserver
All crawlability and indexing checkscluster

intermediate6 minpublished 2026-09-11updated 2026-09-11Maks Verny