How to check if a url is blocked by robots.txt

Do not match the rules by eye. Save the robots.txt, then run a matcher that follows RFC 9309: node robots-match.js robots.txt https://host/path Googlebot. It prints ALLOWED or DISALLOWED plus the one rule that decided it, which is the longest matching pattern, with Allow winning a tie.

Why check this

Run this whenever a rule is added or edited, before a release that introduces a URL shape, and when a crawl report names a URL you believed was open. Deciding by inspection is where the defects come from, because the matching rules are not the ones the eye applies.

Four rules catch people out. Patterns anchor at the start of the path, so a substring elsewhere does not count. The longest matching pattern wins whatever order the rules sit in. An Allow and a Disallow of equal length resolve to Allow. A crawler obeys exactly one group, and the * group is skipped when a more specific one exists.

Prerequisites

• Node 18 or later. The matcher uses nothing outside the standard library.
• The robots.txt saved to a file, so every test reads the same bytes. Fetch it once with How to check robots.txt.
• RFC 9309, sections 2.2.1 to 2.2.3, for the rules the matcher implements.
• A second parser for step 4: npm install robots-parser.

Save this as robots-match.js. It takes a robots file, a URL and a product token:

// node robots-match.js <robots-file> <url-or-path> [product-token]
const fs = require('fs');
const [file, arg, ua = '*'] = process.argv.slice(2);
// Rules match the path plus the query string, never the scheme or the host.
const u = /^https?:/.test(arg) ? new URL(arg) : null;
const target = u ? u.pathname + u.search : arg;
let text = fs.readFileSync(file, 'utf8');
if (text.charCodeAt(0) === 0xfeff) text = text.slice(1);   // a BOM would hide line 1
const groups = new Map();
let agents = [], inRules = false;
for (let line of text.split(/\r?\n/)) {
  line = line.replace(/#.*$/, '').trim();
  if (!line) continue;                              // a blank line never ends a group
  const m = /^([A-Za-z-]+)\s*:\s*(.*)$/.exec(line);
  if (!m) continue;
  const field = m[1].toLowerCase(), value = m[2].trim();
  if (field === 'user-agent') {
    if (inRules) { agents = []; inRules = false; }   // a rule line closed the last group
    const a = value.toLowerCase();
    agents.push(a);
    if (!groups.has(a)) groups.set(a, []);
  } else if (field === 'allow' || field === 'disallow') {
    inRules = true;
    for (const a of agents) groups.get(a).push({ type: field, pattern: value });
  }
}
// The * group is a fallback, reached only when no product token matches (RFC 9309 2.2.1).
const key = [...groups.keys()].filter((a) => a !== '*' && ua.toLowerCase().startsWith(a))
  .sort((a, b) => b.length - a.length)[0] ?? (groups.has('*') ? '*' : null);
const rules = key ? groups.get(key) : [];
const toRe = (p) => new RegExp('^' + p.replace(/[.+?^${}()|[\]\\]/g, '\\$&')
  .replace(/\*/g, '.*').replace(/\\\$$/, '$'));
let best = null;
for (const r of rules) {
  if (r.pattern === '' || !toRe(r.pattern).test(target)) continue;
  if (!best || r.pattern.length > best.pattern.length ||
      (r.pattern.length === best.pattern.length && r.type === 'allow')) best = r;
}
const verdict = best && best.type === 'disallow' ? 'DISALLOWED' : 'ALLOWED';
console.log(`${verdict}  ${target}
  group:  ${key ?? '(none)'}
  rule:   ${best ? best.type + ': ' + best.pattern : '(no rule matched)'}`);

Save this as robots-cases.txt. It holds the awkward cases one real file rarely carries at once.

User-agent: *
Disallow: /

User-agent: TestBot
Disallow: /reports/
Allow: /reports/public/
Disallow: /reports/public/draft/
Allow: /*.css$
Disallow: /assets/
Disallow: /search
Allow: /search$
Allow: /docs/beta/
Disallow: /docs/beta/

Save this as crosscheck.js, the second opinion used in step 4.

// node crosscheck.js <robots-file> <product-token> <url...>
const fs = require('fs');
const robotsParser = require('robots-parser');
const v = require('robots-parser/package.json').version;
const [file, ua, ...urls] = process.argv.slice(2);
for (const url of urls) {
  const origin = new URL(url).origin;
  const r = robotsParser(origin + '/robots.txt', fs.readFileSync(file, 'utf8'));
  const verdict = r.isAllowed(url, ua) ? 'ALLOWED   ' : 'DISALLOWED';
  console.log(`${verdict} ${new URL(url).pathname + new URL(url).search}   [robots-parser ${v}]`);
}

Steps

1. 1

   Step 1.

   Test a URL where the naive reading and the matcher disagree. The saved file contains Disallow: /api/, and this URL contains /API/.

   node robots-match.js mdn-robots.txt https://developer.mozilla.org/en-US/docs/Web/API/fetch Googlebot
   

   ALLOWED  /en-US/docs/Web/API/fetch
   group:  *
   rule:   (no rule matched)

   Someone scanning the file sees /api/ in the rule and /API/ in the URL and calls it blocked. The pattern anchors at the start of the path and compares byte for byte, so neither the position nor the case lines up. The rule does not apply.

2. 2

   Step 2.

   Test three paths on the same host where the rules do reach.

   for u in /api/v1/whoami /en-US/files/12345/x.png /mediaqueries; do
     node robots-match.js mdn-robots.txt "https://developer.mozilla.org$u" Googlebot
   done
   

   DISALLOWED  /api/v1/whoami
   group:  *
   rule:   disallow: /api/
   DISALLOWED  /en-US/files/12345/x.png
   group:  *
   rule:   disallow: /*/files/
   DISALLOWED  /mediaqueries
   group:  *
   rule:   disallow: /media

   The second result comes from a wildcard in the middle of a pattern, which prefix comparison never finds. The third comes from Disallow: /media carrying no trailing slash, so it covers any path starting with those six characters. That last path is invented, to show the reach.

3. 3

   Step 3.

   Run the six precedence cases against the local file.

   for u in /reports/public/q1.html /reports/public/draft/q1.html /assets/site.css /search '/search?q=login' /docs/beta/api.html; do
     node robots-match.js robots-cases.txt "https://t.example$u" TestBot
   done
   

   ALLOWED  /reports/public/q1.html
   group:  testbot
   rule:   allow: /reports/public/
   DISALLOWED  /reports/public/draft/q1.html
   group:  testbot
   rule:   disallow: /reports/public/draft/
   DISALLOWED  /assets/site.css
   group:  testbot
   rule:   disallow: /assets/
   ALLOWED  /search
   group:  testbot
   rule:   allow: /search$
   DISALLOWED  /search?q=login
   group:  testbot
   rule:   disallow: /search
   ALLOWED  /docs/beta/api.html
   group:  testbot
   rule:   allow: /docs/beta/

   Result three is the second disagreement. Allow: /*.css$ is in the file and the stylesheet is blocked anyway, because /assets/ is eight octets and /*.css$ is seven. Result six is the tie: Allow: /docs/beta/ and Disallow: /docs/beta/ are equal length, and RFC 9309 section 2.2.2 gives it to Allow.

4. 4

   Step 4.

   Send one URL through two product tokens, then check every verdict against a second parser.

   for ua in TestBot OtherBot; do
     node robots-match.js robots-cases.txt https://t.example/about.html $ua
   done
   

   ALLOWED  /about.html
   group:  testbot
   rule:   (no rule matched)
   DISALLOWED  /about.html
   group:  *
   rule:   disallow: /

   One file, one URL, two crawlers, opposite answers. TestBot has a group of its own, so Disallow: / under User-agent: * never reaches it.

   node crosscheck.js robots-cases.txt TestBot https://t.example/reports/public/q1.html https://t.example/assets/site.css https://t.example/search 'https://t.example/search?q=login' https://t.example/docs/beta/api.html
   

   ALLOWED    /reports/public/q1.html   [robots-parser 3.0.1]
   DISALLOWED /assets/site.css   [robots-parser 3.0.1]
   ALLOWED    /search   [robots-parser 3.0.1]
   DISALLOWED /search?q=login   [robots-parser 3.0.1]
   ALLOWED    /docs/beta/api.html   [robots-parser 3.0.1]

   Eleven cases went through both implementations, the five above and six more from steps 1 to 3, and both agreed every time. That makes the surprising verdicts trustworthy rather than a property of one script.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | ALLOWED with rule: (no rule matched) | Nothing in the applicable group touches this path | Correct by default. RFC 9309 allows a URI that no rule matches | | DISALLOWED naming a rule you did not expect | A short pattern with no trailing slash, or a wildcard, reached further than intended | Lengthen the pattern, then rerun the same URL | | group: naming a token other than the crawler you care about | A more specific group exists, so the * group is not in play | Rerun with the exact product token that crawler sends | | The verdict flips once the query string is added | A $ anchored rule stops matching as soon as anything follows | Decide which of the two shapes you meant and write both rules | | Two parsers disagree on one case | One of them is not applying longest match or the tie rule | Trust neither until the rule is rewritten so both agree |

Common mistakes

What to check next

• How to check robots.txt: the file must be reachable and served as text before any verdict means anything.
• How to check sitemap and robots txt: run this matcher over every sitemap URL to find contradictions.
• How to check if a page is noindex: a crawl block and an indexing block are separate mechanisms.
• How to view a page as googlebot: what the crawler receives once the rules let it through.
• How to check redirect chain: for when the URL under test is not the URL finally served.

FAQ

How to check robots txt is working or not?

Run one URL you expect to be blocked and one you expect to be open through the matcher. A file that parses but blocks nothing returns (no rule matched) every time, the signature of rules attached to the wrong user-agent line.

How to fix blocked by robots.txt?

Read the deciding rule out of the matcher output, then shorten what it covers or add an Allow longer in octets than the Disallow it must beat. Rerun the same URL. Deleting the rule is the other option when nothing needed blocking.

How to configure robots.txt to allow everything?

Two lines: User-agent: * followed by Disallow: with an empty value. An empty pattern matches nothing, so no rule ever applies. Removing the file has the same effect, since RFC 9309 section 2.3.1.3 treats an unavailable file as unrestricted access.

Robots.txt allow root only, disallow everything else?

User-agent: *
Allow: /$
Disallow: /

Allow: /$ is two octets against one, so the home page wins on length while every other path falls to Disallow: /. Put both shapes through the matcher before shipping it.

Does an allowed verdict mean the page will appear in search results?

No. This check covers crawling only. Indexing is decided separately, by a robots meta tag, an X-Robots-Tag header and the engine's own judgement, and none of that is readable from robots.txt. Start with How to check if a page is indexable.

Verified