h2check

How to check if a staging site is indexable

Run node staging-audit.cjs http://staging.internal against a staging host you own. It prints five lines: whether credentials are required, whether robots.txt carries a blanket disallow, whether X-Robots-Tag rides on HTML and on non-HTML responses, and where the canonical resolves. Any FAIL line means the host is indexable.

Why check this

Run this the day a staging environment is created, and again on every sign-off, because the answer changes with the proxy, the CDN and the deploy that added a new route. Run it on your own hosts. Probing somebody else's staging environment is scanning, not testing.

The failure it prevents is specific. A pre-release build sits on a public hostname, a partner links to it from a ticket or a public Slack channel, and the URL enters an index with unreleased prices in the snippet. Getting it removed then takes longer than the release did.

One belief causes most of these. Teams put Disallow: / in robots.txt and treat the host as protected. That directive controls crawling, not indexing, and the two are separate. Google's own documentation says it plainly: "A page that's disallowed in robots.txt can still be indexed if linked to from other sites." The rest of this procedure shows why the fix people reach for next cannot work either.

Prerequisites

// node staging-server.cjs
//   http://127.0.0.1:8837  staging kept out of the index with robots.txt alone
//   http://127.0.0.1:8838  staging behind HTTP auth, X-Robots-Tag on every response
// Every request to 8837 is appended to staging-requests.log.
const http = require('http');
const fs = require('fs');
const LOG = 'staging-requests.log';
const html = (canonical, text) =>
  `<!doctype html>\n<html lang="en">\n<head>\n<meta charset="utf-8">\n<title>Pricing</title>\n${canonical}\n</head>\n<body><h1>${text}</h1></body>\n</html>\n`;
const PDF = Buffer.from(
  '%PDF-1.7\n1 0 obj<</Type/Catalog/Pages 2 0 R>>endobj\n' +
    '2 0 obj<</Type/Pages/Kids[3 0 R]/Count 1>>endobj\n' +
    '3 0 obj<</Type/Page/Parent 2 0 R/MediaBox[0 0 99 99]>>endobj\n' +
    'trailer<</Size 4/Root 1 0 R>>\n%%EOF\n',
  'latin1'
);

http.createServer((req, res) => {
  fs.appendFileSync(LOG, `${new Date().toISOString()} ${req.method} ${req.url}\n`);
  if (req.url === '/robots.txt') {
    res.writeHead(200, { 'content-type': 'text/plain' });
    return res.end('User-agent: *\nDisallow: /\n');
  }
  if (req.url === '/report.pdf') {
    res.writeHead(200, { 'content-type': 'application/pdf' });
    return res.end(PDF);
  }
  // The team added a directive here and left Disallow: / in place.
  if (req.url === '/promo/') res.setHeader('X-Robots-Tag', 'noindex');
  res.writeHead(200, { 'content-type': 'text/html; charset=utf-8' });
  res.end(html('<link rel="canonical" href="/pricing/">', 'Pricing (staging copy)'));
}).listen(8837, '127.0.0.1', () => console.log('leaky staging on 8837'));

const OK = 'Basic ' + Buffer.from('qa:qa').toString('base64');

http.createServer((req, res) => {
  res.setHeader('X-Robots-Tag', 'noindex, nofollow');
  if (req.headers.authorization !== OK) {
    res.writeHead(401, { 'WWW-Authenticate': 'Basic realm="staging"', 'content-type': 'text/plain' });
    return res.end('Authentication required\n');
  }
  if (req.url === '/robots.txt') {
    res.writeHead(200, { 'content-type': 'text/plain' });
    return res.end('User-agent: *\nDisallow: /\n');
  }
  if (req.url === '/report.pdf') {
    res.writeHead(200, { 'content-type': 'application/pdf' });
    return res.end(PDF);
  }
  res.writeHead(200, { 'content-type': 'text/html; charset=utf-8' });
  res.end(html('<link rel="canonical" href="https://www.example.com/pricing/">', 'Pricing (staging copy)'));
}).listen(8838, '127.0.0.1', () => console.log('hardened staging on 8838'));

Save this as staging-audit.cjs. It is the checklist, and it takes the base URL plus optional credentials.

// node staging-audit.cjs <base-url> [user:pass]
// Five checks on your own staging host. Prints PASS or FAIL per line and one verdict.
const [base, creds] = process.argv.slice(2);
const auth = creds ? { authorization: 'Basic ' + Buffer.from(creds).toString('base64') } : {};
const get = (p, h = {}) => fetch(new URL(p, base), { headers: h, redirect: 'manual' });
const row = (ok, name, detail) => {
  console.log(`  ${ok ? 'PASS' : 'FAIL'}  ${name.padEnd(14)} ${detail}`);
  return ok;
};

(async () => {
  console.log(base);
  const anon = await get('/');
  const gated = anon.status === 401 || anon.status === 403;
  row(gated, 'auth', `GET / without credentials -> ${anon.status}`);

  const r = await get('/robots.txt');
  const body = r.status === 200 ? await r.text() : '';
  const blanket = /^\s*Disallow:\s*\/\s*$/m.test(body);
  row(!blanket, 'robots.txt', blanket
    ? `${r.status}, Disallow: / stops the crawl and carries no directive`
    : `${r.status}, no blanket Disallow: /`);

  let tagged = true;
  for (const p of ['/', '/report.pdf']) {
    const res = await get(p, auth);
    const tag = res.headers.get('x-robots-tag');
    tagged = row(/noindex/i.test(tag || ''), 'x-robots-tag', `${p} -> ${tag || '(absent)'}`) && tagged;
  }

  const doc = await get('/pricing/', auth);
  const raw = (/<link[^>]+rel=["']?canonical["']?[^>]*>/i.exec(await doc.text()) || [''])[0];
  const href = (/href=["']([^"']+)["']/i.exec(raw) || [])[1];
  const abs = href ? new URL(href, new URL('/pricing/', base)).href : '(none)';
  const offHost = href ? new URL(abs).host !== new URL(base).host : false;
  row(offHost, 'canonical', `${href || '(none)'} -> ${abs}`);

  const pass = gated || tagged;
  console.log(`  VERDICT: ${pass ? 'PASS' : 'FAIL'} ${pass ? 'not indexable' : 'indexable'}`);
})();

Save this as polite-fetch.cjs. It reads robots.txt before the URL, the way a compliant crawler does.

// node polite-fetch.cjs <url>
const url = new URL(process.argv[2]);
(async () => {
  const r = await fetch(new URL('/robots.txt', url));
  const txt = r.status === 200 ? await r.text() : '';
  const rules = [];
  let active = false;
  for (const line of txt.split(/\r?\n/)) {
    const m = /^\s*(user-agent|allow|disallow)\s*:\s*(.*?)\s*$/i.exec(line);
    if (!m) continue;
    if (m[1].toLowerCase() === 'user-agent') active = m[2] === '*';
    else if (active && m[2]) rules.push([m[1].toLowerCase(), m[2]]);
  }
  const path = url.pathname + url.search;
  const hit = rules
    .filter(([, p]) => path.startsWith(p))
    .sort((a, b) => b[1].length - a[1].length || (a[0] === 'allow' ? -1 : 1))[0];
  if (hit && hit[0] === 'disallow') {
    console.log(`robots.txt: ${hit[0]}: ${hit[1]} matches ${path}`);
    console.log('not fetched, so no directive in the response is ever read');
    return;
  }
  const res = await fetch(url);
  console.log(`fetched ${res.status}  x-robots-tag: ${res.headers.get('x-robots-tag') || '(absent)'}`);
})();

Steps

  1. Step 1.

    Audit the staging host the way an anonymous visitor reaches it.

    node staging-audit.cjs http://127.0.0.1:8837

    http://127.0.0.1:8837
FAIL  auth           GET / without credentials -> 200
FAIL  robots.txt     200, Disallow: / stops the crawl and carries no directive
FAIL  x-robots-tag   / -> (absent)
FAIL  x-robots-tag   /report.pdf -> (absent)
FAIL  canonical      /pricing/ -> http://127.0.0.1:8837/pricing/
VERDICT: FAIL indexable

    Five lines, four separate leaks. Anyone with the hostname gets a 200, the PDF carries no directive, and the relative canonical resolves back to the staging host instead of naming the production URL.

  2. Step 2.

    Confirm that the one page with a directive really carries it.

    curl -sSI http://127.0.0.1:8837/promo/ | grep -i 'x-robots-tag'

    X-Robots-Tag: noindex

    This is the fix a team applies after the first report: a noindex on the response, with Disallow: / left in robots.txt.

  3. Step 3.

    Request the same URL the way a crawler does, reading robots.txt first.

    node polite-fetch.cjs http://127.0.0.1:8837/promo/

    robots.txt: disallow: / matches /promo/
not fetched, so no directive in the response is ever read

    The two directives cancel each other. The disallow stops the fetch, and the header that would have kept the URL out of the index sits in a response nobody requests.

  4. Step 4.

    Read the server's own log to see what the crawler actually asked for.

    tail -4 staging-requests.log

    2026-09-11T20:52:40.942Z GET /report.pdf
2026-09-11T20:52:40.944Z GET /pricing/
2026-09-11T20:52:41.187Z HEAD /promo/
2026-09-11T20:52:41.463Z GET /robots.txt

    The last line is the crawler's entire visit. There is no GET /promo/, so the header measured in step 2 was never delivered to the client that needed it.

  5. Step 5.

    Audit the hardened host, credentials included, so every response is inspected.

    node staging-audit.cjs http://127.0.0.1:8838 qa:qa

    http://127.0.0.1:8838
PASS  auth           GET / without credentials -> 401
PASS  robots.txt     401, no blanket Disallow: /
PASS  x-robots-tag   / -> noindex, nofollow
PASS  x-robots-tag   /report.pdf -> noindex, nofollow
PASS  canonical      https://www.example.com/pricing/ -> https://www.example.com/pricing/
VERDICT: PASS not indexable

    Two independent controls now hold: nothing gets past 401 without credentials, and anything that does carries the directive, PDF included.

  6. Step 6.

    Confirm the authentication sits in front of the whole host, not only the application routes.

    for p in / /pricing/ /report.pdf /robots.txt; do curl -sS -o /dev/null -w "%{http_code} $p\n" "http://127.0.0.1:8838$p"; done

    401 /
401 /pricing/
401 /report.pdf
401 /robots.txt

    Static files and robots.txt are the two paths most often routed around the authentication layer. Both answer 401 here.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | PASS auth on every path | Nothing reaches the content without credentials | Done. This is the only control that also hides the content | | FAIL auth and PASS x-robots-tag on both samples | Publicly readable, but every response asks to be kept out | Acceptable when staging must be reachable. Re-check after every CDN change | | FAIL robots.txt with Disallow: / | Crawling is blocked and indexing is not | Remove the blanket disallow, then serve X-Robots-Tag: noindex instead | | FAIL x-robots-tag /report.pdf | The directive is set by the app, not by the file server | Move it to the proxy or CDN so every response carries it | | FAIL canonical resolving to the staging host | The staging copy names itself as the original | Emit an absolute canonical to the production URL, or none at all |

Common mistakes

Sign: robots.txt says Disallow: / and the staging URL appears in search results anyway, with no description under the title.Cause: Disallow controls fetching, not indexing. An external link is enough to put the URL in an index, and because the crawler never fetches the page, there is no title or description to show. Adding noindex afterwards changes nothing, which steps 2 to 4 measure: the directive is on the response and the response is never requested. Remove the disallow so the noindex can be read, or put authentication in front.
Sign: Every page on staging returns X-Robots-Tag: noindex, and exported PDFs and CSVs still turn up.Cause: The header is set by the application framework, so it only exists on routes the application serves. Static files come from the web server or the CDN, which has its own header configuration. Step 5 checks a non-HTML path for that reason.
Sign: Staging pages carry a canonical, so the team treats duplication as handled.Cause: A canonical is a hint about which duplicate to prefer, not an indexing control, and a relative href resolves against the host that served it. A canonical of /pricing/ on staging resolves to the staging URL, which tells a search engine that the staging copy is the original.
Sign: Basic authentication is enabled, and a monitoring service still retrieves pages.Cause: The rule was applied to the application's location block and not to the static asset paths or to /robots.txt. Step 6 probes those paths directly instead of trusting the configuration.

What to check next

FAQ

How to prevent staging to be indexed in search engines?

Put HTTP authentication in front of the whole host. It is the only control that hides the content as well as the URL. Where staging has to be publicly reachable, send X-Robots-Tag: noindex on every response from the proxy, and leave robots.txt permissive so the directive can be read.

Is Disallow: / enough to keep staging out of Google?

No. Google's documentation states that a page disallowed in robots.txt can still be indexed when other sites link to it. The disallow also prevents the crawler from reading any noindex you add later, so it actively blocks the control that would have worked.

Can I run this against another company's staging host?

No. Run it on hosts you own or have written permission to test. The procedure sends a handful of requests, but credentials, authentication probes and a request log belong to the operator of the host, and so does the decision to fix a FAIL.

What about a staging host that is already indexed?

Fix the response first, then request removal through the search engine's own tooling. Removal needs the crawler to fetch the page and read the directive, so a Disallow: / left in place keeps the URL in the index instead of clearing it.

Verified

Verified by Maks Vernycurl 8.21.0node 22.23.2

Each output block is what the command above it printed on that date, on the host named in the step. Figures read from a live site move between runs. Compare the shape of the answer rather than the digits, and see the methodology for how a page is re-verified.

intermediate10 minpublished updated Maks Verny