How to check sitemap and robots txt

Read the Sitemap: lines out of robots.txt with grep -iE '^[[:space:]]*sitemap:' robots.txt, confirm each value is an absolute URL, fetch it, then put every URL it lists back through the same robots.txt. A sitemap advertising URLs its own robots.txt disallows is the contradiction this pairing catches.

Why check this

Run this on staging sign-off, after a domain or path migration, and after any release that adds a Disallow rule. The failure it prevents is the contradiction: the sitemap invites a crawler to fetch a URL that the same robots.txt forbids it to fetch. Nothing errors, nothing is logged, and the pages sit in a crawl report instead of being retrieved.

The second failure is quieter. A Sitemap: value left relative, or still pointing at the previous hostname after a migration, keeps the robots.txt parsing cleanly while the sitemap is never read at all.

This check covers the declaration and the agreement between the two files. It does not validate the sitemap against its schema, and it says nothing about what any search engine has stored.

Prerequisites

• curl 7.0 or later, and the robots.txt already saved with How to check robots.txt.
• Node 18 or later, plus robots-match.js from How to check if a url is blocked by robots.txt. Steps 4 and 6 call it.
• The sitemaps.org protocol for the file limits and the same-host rule.
• Two local files for steps 5 and 6, built to fail so the failing output is on record.

User-agent: *
Disallow: /media
Disallow: /*?sort=

User-agent: Googlebot
Disallow: /internal/
Sitemap: /sitemap.xml
Sitemap: https://shop.example/sitemap.xml

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://shop.example/products/1</loc></url>
  <url><loc>https://shop.example/media/hero.jpg</loc></url>
  <url><loc>https://shop.example/products?sort=price</loc></url>
  <url><loc>https://cdn.shop.example/products/2</loc></url>
  <url><loc>/products/3</loc></url>
</urlset>

Steps

1. 1

   Step 1.

   Pull every Sitemap: declaration out of the file and label it absolute or not.

   grep -iE '^[[:space:]]*sitemap:' mdn-robots.txt | sed 's/^[^:]*:[[:space:]]*//' | while read -r s; do
     case "$s" in https://*|http://*) echo "absolute  $s";; *) echo "RELATIVE  $s";; esac
   done
   

   absolute  https://developer.mozilla.org/sitemap.xml

   The -i matters. The field name is matched case-insensitively, so SITEMAP: and sitemap: are both valid and a case-sensitive grep silently reports a site as having none.

2. 2

   Step 2.

   Request the declared URL and read the status line and content type.

   curl -sS -I https://developer.mozilla.org/sitemap.xml | grep -iE '^HTTP|^content-type|^content-length|^last-modified'
   

   HTTP/2 200 
   content-type: application/xml
   last-modified: Fri, 11 Sep 2026 01:34:47 GMT
   content-length: 1282

3. 3

   Step 3.

   Download it with curl -sS https://developer.mozilla.org/sitemap.xml -o mdn-sitemap.xml, then list what it points at and count the distinct hosts.

   grep -o '<loc>[^<]*</loc>' mdn-sitemap.xml | sed 's/<[^>]*>//g' | tee locs.txt \
     | sed 's#^\(https*://[^/]*\).*#\1#' | sort | uniq -c
   

        10 https://developer.mozilla.org

   One host for ten entries. The sitemaps.org protocol requires every listed URL to use the same protocol and host as the sitemap itself, so any second line here is a finding.

4. 4

   Step 4.

   Put every URL the sitemap lists back through the robots.txt that declared it.

   while read -r u; do node robots-match.js mdn-robots.txt "$u" Googlebot | head -1; done < locs.txt
   

   ALLOWED  /sitemaps/en-us/sitemap.xml.gz
   ALLOWED  /sitemaps/es/sitemap.xml.gz
   ALLOWED  /sitemaps/fr/sitemap.xml.gz
   ALLOWED  /sitemaps/ja/sitemap.xml.gz
   ALLOWED  /sitemaps/ko/sitemap.xml.gz
   ALLOWED  /sitemaps/pt-br/sitemap.xml.gz
   ALLOWED  /sitemaps/ru/sitemap.xml.gz
   ALLOWED  /sitemaps/zh-cn/sitemap.xml.gz
   ALLOWED  /sitemaps/zh-tw/sitemap.xml.gz
   ALLOWED  /sitemaps/de/sitemap.xml.gz

   Ten of ten allowed is the passing shape. The same file disallows /api/ and /media, so this is a real agreement and not an empty rule set.

5. 5

   Step 5.

   Repeat step 1 on the local robots.txt, which carries one relative declaration.

   grep -iE '^[[:space:]]*sitemap:' robots-local.txt | sed 's/^[^:]*:[[:space:]]*//' | while read -r s; do
     case "$s" in https://*|http://*) echo "absolute  $s";; *) echo "RELATIVE  $s";; esac
   done
   

   RELATIVE  /sitemap.xml
   absolute  https://shop.example/sitemap.xml

   Ask a parser what it makes of that line.

   node -e "const fs=require('fs'),rp=require('robots-parser');console.log(JSON.stringify(rp('https://shop.example/robots.txt',fs.readFileSync('robots-local.txt','utf8')).getSitemaps()))"
   

   ["/sitemap.xml","https://shop.example/sitemap.xml"]

   The relative value comes back unresolved, exactly as written. A consumer that hands that string to a fetch call has a URL with no host on it.

6. 6

   Step 6.

   Audit the local pair that was built to fail. Count its hosts, then match every entry against its own robots.txt as two different crawlers.

   grep -o '<loc>[^<]*</loc>' sitemap-local.xml | sed 's/<[^>]*>//g' | sed 's#^\(https*://[^/]*\).*#\1#' | sort | uniq -c
   

         1 /products/3
       1 https://cdn.shop.example
       3 https://shop.example

   for ua in Googlebot bingbot; do
     echo "-- $ua"
     grep -o '<loc>[^<]*</loc>' sitemap-local.xml | sed 's/<[^>]*>//g' \
       | while read -r u; do node robots-match.js robots-local.txt "$u" "$ua" | head -1; done
   done
   

   -- Googlebot
   ALLOWED  /products/1
   ALLOWED  /media/hero.jpg
   ALLOWED  /products?sort=price
   ALLOWED  /products/2
   ALLOWED  /products/3
   -- bingbot
   ALLOWED  /products/1
   DISALLOWED  /media/hero.jpg
   DISALLOWED  /products?sort=price
   ALLOWED  /products/2
   ALLOWED  /products/3

   One sitemap, one robots.txt, and two crawlers that do not agree about it. Googlebot has a group of its own holding a single Disallow: /internal/ rule, so the two rules under User-agent: * never apply to it. Audit as Googlebot alone and the pair looks clean. That is why this step runs a token with no group of its own as well.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | One absolute line and a 200 with application/xml | The declaration is usable | Move on to the agreement check | | RELATIVE on any line | The value is not what the protocol documents, and parsers return it unresolved | Write the full URL, scheme and host included | | More than one host in the loc count | Entries on another host are outside this sitemap's scope | Split them into a sitemap served from that host | | A count line that is a bare path | A loc that does not begin with a protocol | Make it a full URL. The protocol requires one | | Any DISALLOWED in the last step | The two files contradict each other | Remove the rule or drop the URL, but not both files unchanged |

Common mistakes

Thresholds

What to check next

• How to check robots.txt: the file that carries the declaration has to be served correctly first.
• How to check if a url is blocked by robots.txt: the matcher used in steps 4 and 6, explained rule by rule.
• How to validate sitemap xml: the schema check this page deliberately leaves out.
• How to check sitemap index file: what to do with the ten children found in step 3.
• How to check sitemap lastmod: the field most often wrong once the structure is right.

FAQ

Multiple sitemap entries in robots.txt, is that allowed?

Yes. Each declaration is an independent record, so a file may carry as many as it needs, in any order. Every one is read. Check each separately, because a single unreachable entry among four is easy to miss in a page load.

Listing both sitemaps and sitemap index files in robots.txt?

That works and is common during a migration. An index and the files it points at are both valid values, and declaring a child that its parent index already lists is harmless duplication rather than an error. Fetch each declared URL once and confirm it answers 200.

Can a relative sitemap url be used in robots.txt?

The protocol documents the directive with a full absolute URL, and <loc> values must begin with the protocol. Step 5 shows a parser returning /sitemap.xml unresolved. Write the absolute form and the question disappears.

Does the position of the Sitemap line in the file matter?

No. It belongs to no user-agent group, so putting it at the top, at the bottom, or inside a group all read the same. Keeping it on its own at the top of the file only makes review easier.

Verified