How to check if a page is indexable

Indexable and indexed are different questions. This procedure answers the first: run node indexable.cjs https://host/page and it checks status, X-Robots-Tag, the robots meta in <head> and the canonical, then prints INDEXABLE or the rule that blocks it. Whether Google holds the page is Search Console's answer.

Checker offline. Follow the manual steps below, they give the same answer.

Why check this

Run this on every URL a release adds, on every URL a migration moves, and across a staging host before it is reachable from outside. The failure it prevents is a launch where the section is live, links resolve, the pages render, and nothing in them can enter an index because one template kept a directive or a canonical pointing at the old path.

Indexable means the response permits indexing, and every part of that is readable with a request. Indexed means a search engine has stored the page, which is a property of that engine's systems and cannot be read from the response. This procedure settles the first question and not the second.

Four signals decide indexability, and a page has to pass all four: the status of the final URL, the X-Robots-Tag header, the robots meta tag in the head, and the canonical. A fifth condition sits outside the response: the crawler has to be allowed to fetch the URL.

Prerequisites

Node 18 or later. The script below uses only the standard library.
curl 7.0 or later, for the saved copy in step 4.
The URL exactly as it is linked. A query string, a trailing slash and a scheme each make a different URL.
Google's robots meta tag reference for the directive names.
The local target from How to check if a page is noindex, started with node index-server.cjs, for steps 2 and 3.

Save this as indexable.cjs.

// node indexable.cjs <url>
// Decides whether one URL is INDEXABLE: reachable, not blocked by a directive,
// and canonical to itself. It cannot decide whether the URL is INDEXED.
const start = process.argv[2];
const bad = /(^|[\s,])(noindex|none)([\s,]|$)/i;

fetch(start, { redirect: 'follow' }).then(async (r) => {
  const html = r.headers.get('content-type')?.includes('html') ? await r.text() : '';
  const xrt = r.headers.get('x-robots-tag');
  const head = html.slice(0, html.search(/<\/head>/i) + 1 || html.length);
  const meta = [...head.matchAll(/<meta[^>]+name=["']?(?:robots|googlebot)["']?[^>]*>/gi)]
    .map((m) => /content=["']([^"']*)/i.exec(m[0])?.[1] ?? '');
  const canon = /<link[\s\S]{0,300}?rel=["']canonical["'][\s\S]{0,300}?>/i.exec(head);
  const canonHref = canon ? /href=["']([^"']+)/i.exec(canon[0])?.[1] : null;

  const fail = [];
  if (r.status !== 200) fail.push(`status ${r.status}`);
  if (xrt && bad.test(xrt)) fail.push(`x-robots-tag: ${xrt}`);
  for (const c of meta) if (bad.test(c)) fail.push(`meta robots in <head>: ${c}`);
  if (canonHref && new URL(canonHref, r.url).href !== r.url) fail.push(`canonical -> ${canonHref}`);

  console.log(`url:        ${r.url}${r.redirected ? '  (redirected from ' + start + ')' : ''}`);
  console.log(`status:     ${r.status}`);
  console.log(`x-robots:   ${xrt ?? '(absent)'}`);
  console.log(`meta head:  ${meta.length ? meta.join(' | ') : '(none in <head>)'}`);
  console.log(`canonical:  ${canonHref ?? '(none)'}`);
  console.log(`verdict:    ${fail.length ? 'NOT INDEXABLE  (' + fail.join('; ') + ')' : 'INDEXABLE'}`);
  console.log('indexed:    not decided here. Use Search Console URL Inspection.');
});

Steps

Run the check against a page that should pass, to learn the shape of a clean result.

node indexable.cjs https://developer.mozilla.org/en-US/docs/Web/HTTP

url:        https://developer.mozilla.org/en-US/docs/Web/HTTP
status:     200
x-robots:   (absent)
meta head:  (none in <head>)
canonical:  https://developer.mozilla.org/en-US/docs/Web/HTTP
verdict:    INDEXABLE
indexed:    not decided here. Use Search Console URL Inspection.

Read the last line every time. The four signals above it came off the wire. Whether this URL is in anyone's index is not among them, and no request to the site can produce it.

Run it against three URLs that each fail for a different reason.

for u in /private/old-report.html /api/items /gone.html; do
  node indexable.cjs "http://127.0.0.1:8791$u" | sed -n "1p;6p"
done

url:        http://127.0.0.1:8791/private/old-report.html
verdict:    NOT INDEXABLE  (meta robots in <head>: noindex)
url:        http://127.0.0.1:8791/api/items
verdict:    NOT INDEXABLE  (x-robots-tag: noindex, nofollow)
url:        http://127.0.0.1:8791/gone.html
verdict:    NOT INDEXABLE  (status 410)

Three mechanisms, one verdict line each. A check that reads only the HTML misses the second, and one that reads only the status misses the first two.

3
Step 3.
Run it against a page that carries a noindex and passes anyway.
```
node indexable.cjs http://127.0.0.1:8791/late.html
```
```
url:        http://127.0.0.1:8791/late.html
status:     200
x-robots:   (absent)
meta head:  (none in <head>)
canonical:  (none)
verdict:    INDEXABLE
indexed:    not decided here. Use Search Console URL Inspection.
```
The served HTML contains <meta name="robots" content="noindex">, and a script injects a second one into the head after load. Neither is in the head in the served bytes, so this check, and every parser that does not render, reports INDEXABLE. Google says it respects a robots meta tag in the body, so page and tool disagree while both behave as documented. Settle it with How to check if a page is noindex.

Save a real page and read its canonical two ways.

curl -sS -D mdn-head.txt -o mdn-body.html https://developer.mozilla.org/en-US/docs/Web/HTTP

(no output; the response is saved to mdn-body.html)

grep -io '<link rel="canonical"[^>]*>' mdn-body.html; echo "exit=$?"

exit=1

node -e "const s=require('fs').readFileSync('mdn-body.html','utf8');const m=/<link[\s\S]{0,300}?rel=[\"']canonical[\"'][\s\S]{0,300}?>/i.exec(s);console.log(m ? m[0].replace(/\s+/g,' ') : '(none)')"

<link rel="canonical" href="https://developer.mozilla.org/en-US/docs/Web/HTTP" />

The tag is there. The grep reports nothing because the element spans several lines and grep matches within one line. A canonical read this way returns "missing" on a page that has one, and a team will act on that.

5
Step 5.
Check the condition outside the response.
```
curl -sS http://127.0.0.1:8791/robots.txt
```
```
User-agent: *
Disallow: /private/
```
The first URL in step 2 is under /private/, so a crawler obeying this file never fetches it and never reads the noindex the verdict named. An INDEXABLE verdict on a disallowed URL is equally hollow. Match the path against the rules with How to check if a url is blocked by robots.txt before filing either result.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | verdict: INDEXABLE | The response permits indexing on all four signals | Confirm crawl access, then stop. This is not a statement about any index | | indexed: not decided here | Printed on every run, pass or fail | Use Search Console URL Inspection on a verified property for the indexed question | | status other than 200 | A crawler stores nothing for a non-200 response | Fix the status, or accept it if the URL is meant to be retired | | x-robots: containing noindex or none | A header directive blocks it | Find the middleware or CDN rule that sets it | | meta head: containing noindex | The HTML blocks it | Remove the tag from the template or the route | | canonical -> another URL | The page nominates a different URL as the one to index | Correct it unless the duplicate is intended | | (redirected from ...) | The verdict describes the destination, not the URL you typed | Check the chain, then rerun on both ends |

Common mistakes

Sign: An indexability tool reports a page as fine, and the team reads that as confirmation the page is in Google.Cause: No tool that only requests your URL can answer that. Indexable is read from the response; indexed is held in a search engine's systems, and the only ways to read it are Search Console's URL Inspection on a verified property, or scraping a results page, which is behind a consent screen and against the terms. Report the two separately or the report is wrong.

Sign: A canonical check reports the tag as missing and the page has one.Cause: The element is written across several lines, so a line-oriented grep never matches it. Step 4 reproduces it on developer.mozilla.org, where the single-line pattern exits 1 and a multiline read prints the tag. Every HTML assertion in a test suite that uses grep has this defect.

Sign: The verdict is INDEXABLE and the HTML plainly contains a noindex.Cause: The directive sits outside the head element in the served bytes, or it is added by script after load. Parsers that stop at the end of the head miss it, while Google states it respects a robots meta tag in the body. The page and the tooling disagree, so a passing verdict is not a clean result on its own.

Sign: The verdict looks right and describes a URL nobody linked.Cause: The request followed a redirect, so every signal was read from the destination. The output names the final URL for that reason. A source that 301s and a destination that is indexable is a normal pair, and a source that 302s to a login page is not.

What to check next

How to check if a page is noindex: the directive channel in full, including placements this check cannot see.
How to check x-robots-tag: the header channel, and the only one on responses with no HTML.
How to check if a url is blocked by robots.txt: the fifth condition, from step 5.
How to check canonical tag: what a canonical pointing elsewhere does, and when that is correct.
How to check redirect chain: needed whenever the URL tested is not the URL served.

FAQ

How to check google indexing of a website?

Not from the site. Open Search Console, verify the property, then use URL Inspection for one URL or the Pages report for the site. The only other route is reading a results page, which sits behind a consent screen. The site: operator is a sample, not a record.

How to check if google can crawl my site?

Crawling and indexing are separate. Fetch /robots.txt, match the URL against its rules, and confirm it answers 200 to a plain GET. That settles crawl access. This procedure settles what the response permits once the crawler arrives.

How to check if a website can be crawled?

Run the robots.txt match and a status check on a sample of sitemap URLs, one per template. A site that returns 200 for every path, invented ones included, is serving soft 404s.

Does an INDEXABLE verdict mean the page will appear in search results?

No. It means nothing in the response prevents indexing. A search engine still decides whether to fetch, store and show the page. The verdict is a statement about your server, not about anyone's index.

Verified

Verified 2026-09-11by Maks Vernynode 22.23.2curl 8.21.0

Each output block is what the command above it printed on that date, on the host named in the step. Figures read from a live site move between runs. Compare the shape of the answer rather than the digits, and see the methodology for how a page is re-verified.

Checker: meta-tags title, description, canonical, robots meta, hreflang, Open Graph, Twitter card, viewportserver
All crawlability and indexing checkscluster

intermediate8 minpublished 2026-09-11updated 2026-09-11Maks Verny