How to check sitemap index file

An index is a sitemap whose root element is sitemapindex rather than urlset. Fetch it, confirm the root, then follow every entry and record what each one returns: python index-lint.py https://host/sitemap.xml. An entry pointing at a second index or at an HTML page is refused, and the file name proves nothing.

Checker offline. Follow the manual steps below, they give the same answer.

Why check this

Run this when a site crosses 50,000 URLs and the generator starts splitting output, when locales or sections each get their own file, and on every release afterwards. The failure it catches is silent: the index is reachable, it is well-formed, and one of the children it names answers with a 404, an HTML page, or a second index. The crawler reads the index, drops that child, and a whole section stops being submitted.

Checking the index alone is not enough, because an index is a list of promises about other files. Each promise has to be kept, so the check follows every entry. That costs one request per child, which is why it runs against a site you own.

Prerequisites

Python 3.8 or later. urllib and xml.etree.ElementTree are in the standard library.
Node 18 or later, for the local target in steps 3 and 4. Pick a port and confirm it is free with netstat -ano | grep 8734 before you start.
The sitemap index reference and the siteindex.xsd content model, which allows loc and lastmod inside a sitemap element and nothing else in that namespace.
Save the local target as sitemap-server.js and run node sitemap-server.js. It serves one good child, one child that is itself an index, one HTML page, and one missing file.

// node sitemap-server.js   ->  http://127.0.0.1:8734/sitemap.xml
const http = require('http');
const H = '<?xml version="1.0" encoding="UTF-8"?>\n';
const NS = 'xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"';
const B = 'http://127.0.0.1:8734';
const files = {
  '/sitemap.xml': ['application/xml', H + `<sitemapindex ${NS}>
  <sitemap><loc>${B}/sitemap-products.xml</loc><lastmod>2026-09-10</lastmod></sitemap>
  <sitemap><loc>${B}/sitemap-archive.xml</loc><lastmod>2026-09-10</lastmod></sitemap>
  <sitemap><loc>${B}/about.html</loc></sitemap>
  <sitemap><loc>${B}/sitemap-gone.xml</loc></sitemap>
</sitemapindex>\n`],
  '/sitemap-products.xml': ['application/xml', H + `<urlset ${NS}>
  <url><loc>${B}/p/1</loc><lastmod>2026-09-09</lastmod></url>
</urlset>\n`],
  '/sitemap-archive.xml': ['application/xml', H + `<sitemapindex ${NS}>
  <sitemap><loc>${B}/sitemap-2025.xml</loc></sitemap>
</sitemapindex>\n`],
  '/sitemap-nested.xml': ['application/xml', H + `<sitemapindex ${NS}>
  <sitemap>
    <loc>${B}/sitemap-products.xml</loc>
    <sitemapindex ${NS}><sitemap><loc>${B}/sitemap-archive.xml</loc></sitemap></sitemapindex>
  </sitemap>
</sitemapindex>\n`],
  '/about.html': ['text/html', '<!doctype html><html><head><title>About</title></head><body>About</body></html>'],
};
http.createServer((req, res) => {
  const f = files[req.url];
  if (!f) { res.writeHead(404, { 'content-type': 'text/plain' }); return res.end('not found'); }
  res.writeHead(200, { 'content-type': f[0] });
  res.end(f[1]);
}).listen(8734, '127.0.0.1', () => console.log('listening on 8734'));

Save this as index-lint.py. It makes one request per entry, so point it at your own site.

# python index-lint.py <index-url>
import sys, gzip, urllib.request
import xml.etree.ElementTree as ET
from urllib.parse import urlsplit

NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}"

def fetch(url):
    with urllib.request.urlopen(url, timeout=10) as r:
        body = r.read()
        if url.endswith(".gz"):
            body = gzip.decompress(body)
        return r.status, r.headers.get("content-type", ""), body

index_url = sys.argv[1]
status, ctype, body = fetch(index_url)
print("index: %s %s %d bytes" % (status, ctype, len(body)))
root = ET.fromstring(body)
print("root element:", root.tag)
if root.tag != NS + "sitemapindex":
    sys.exit("FATAL root is not %ssitemapindex" % NS)
# tSitemap allows loc then optional lastmod, nothing else in this namespace.
allowed = {NS + "loc", NS + "lastmod"}
bad = 0
for child in root:
    if child.tag != NS + "sitemap":
        print("REJECT child <%s>: sitemapindex may only contain <sitemap>" % child.tag)
        bad += 1
        continue
    for g in child:
        if g.tag.startswith(NS) and g.tag not in allowed:
            print("REJECT <%s> inside <sitemap>: only loc and lastmod are allowed" % g.tag)
            bad += 1
base = urlsplit(index_url)
scope = base.path.rsplit("/", 1)[0] + "/"
for i, sm in enumerate(root.findall(NS + "sitemap"), 1):
    loc = (sm.findtext(NS + "loc") or "").strip()
    p = urlsplit(loc)
    if p.netloc != base.netloc or not p.path.startswith(scope):
        print("%d %s REJECT outside %s%s" % (i, loc, base.netloc, scope)); bad += 1; continue
    try:
        st, ct, b = fetch(loc)
    except Exception as e:
        print("%d %s UNREACHABLE %s" % (i, loc, e)); bad += 1; continue
    try:
        tag = ET.fromstring(b).tag
    except ET.ParseError as e:
        print("%d %s NOT XML (%s) %s" % (i, loc, ct, e)); bad += 1; continue
    if tag == NS + "urlset":
        print("%d %s OK urlset, %d urls" % (i, loc, len(ET.fromstring(b).findall(NS + "url"))))
    else:
        print("%d %s REJECT root is <%s>, an index entry must point at a urlset" % (i, loc, tag)); bad += 1
print("rejections:", bad)

Steps

Fetch the file once and read its root element.

curl -sS -o sitemap.xml https://developer.mozilla.org/sitemap.xml
python -c "import xml.etree.ElementTree as ET; r=ET.parse('sitemap.xml').getroot(); print('root:', r.tag); print('children:', len(r))"

root: {http://www.sitemaps.org/schemas/sitemap/0.9}sitemapindex
children: 10

sitemapindex in the root tag is what makes this an index. The namespace is the same one a urlset uses, so the tag name is the only thing that separates the two.

Print every entry with its lastmod, before following anything.

python -c "import xml.etree.ElementTree as ET; NS='{http://www.sitemaps.org/schemas/sitemap/0.9}'; [print(s.findtext(NS+'loc'), s.findtext(NS+'lastmod')) for s in ET.parse('sitemap.xml').getroot().findall(NS+'sitemap')]"

https://developer.mozilla.org/sitemaps/en-us/sitemap.xml.gz 2026-09-11
https://developer.mozilla.org/sitemaps/es/sitemap.xml.gz 2026-09-11
https://developer.mozilla.org/sitemaps/fr/sitemap.xml.gz 2026-09-11
…
https://developer.mozilla.org/sitemaps/de/sitemap.xml.gz 2026-09-11

Ten children, one per locale, each gzipped and each carrying a date. Stop here on a site you do not own: the next step makes one request per entry, and ten of them is traffic nobody asked for.

Start the local target and follow every entry of an index built with the usual faults.

python index-lint.py http://127.0.0.1:8734/sitemap.xml

index: 200 application/xml 465 bytes
root element: {http://www.sitemaps.org/schemas/sitemap/0.9}sitemapindex
1 http://127.0.0.1:8734/sitemap-products.xml OK urlset, 1 urls
2 http://127.0.0.1:8734/sitemap-archive.xml REJECT root is <{http://www.sitemaps.org/schemas/sitemap/0.9}sitemapindex>, an index entry must point at a urlset
3 http://127.0.0.1:8734/about.html NOT XML (text/html) syntax error: line 1, column 0
4 http://127.0.0.1:8734/sitemap-gone.xml UNREACHABLE HTTP Error 404: Not Found
rejections: 3

Line 1 is the only child that is a sitemap. Line 2 is a second index, which is the nesting refusal: an index names sitemaps, and the tree stops one level down. Line 3 is an HTML page served with a 200, which a status check would have passed. Line 4 is a child that was renamed and never removed from the index.

Try the other shape of nesting, where the index element sits inside the file rather than behind a URL.

python index-lint.py http://127.0.0.1:8734/sitemap-nested.xml

index: 200 application/xml 362 bytes
root element: {http://www.sitemaps.org/schemas/sitemap/0.9}sitemapindex
REJECT <{http://www.sitemaps.org/schemas/sitemap/0.9}sitemapindex> inside <sitemap>: only loc and lastmod are allowed
1 http://127.0.0.1:8734/sitemap-products.xml OK urlset, 1 urls
rejections: 1

The refusal comes from the content model in siteindex.xsd, where the sitemap element is a sequence of one loc and an optional lastmod.

Confirm that the same file is acceptable XML, so nothing else will warn you.

curl -sS http://127.0.0.1:8734/sitemap-nested.xml | python -c "import sys,xml.etree.ElementTree as ET; r=ET.fromstring(sys.stdin.buffer.read()); print('parsed OK, root', r.tag, ', children', [c.tag.split('}')[1] for c in r])"

parsed OK, root {http://www.sitemaps.org/schemas/sitemap/0.9}sitemapindex , children ['sitemap']

A parser sees one sitemap child and reports a clean document. The inner index is invisible to it, and to any check that counts entries instead of reading the tree.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | Every entry OK urlset | Each child is a real sitemap and was reachable | Check the lastmod values next | | REJECT root is <...sitemapindex> | An entry points at another index | Flatten it. Name the leaf sitemaps directly in the top index | | NOT XML (text/html) | A page answered where a sitemap was expected | Fix the route. A 200 hid this from the monitor | | UNREACHABLE HTTP Error 404 | A child was renamed, moved, or never deployed | Regenerate the index from the files that exist | | REJECT outside host/path | A child on another host, or above the index in the tree | Move the child under the index, or move the index up | | FATAL root is not ...sitemapindex | This is a plain sitemap, whatever the file is called | Read it as a urlset instead |

Common mistakes

Sign: A crawler reports fewer submitted URLs than the index accounts for, and every file in the index answers 200.Cause: One of the children is itself a sitemap index. The file name and the content type look right, the fetch succeeds, and only the root element gives it away. Reading status codes alone cannot find this. The check has to parse each child and compare its root element against urlset.

Sign: The index is named sitemap_index.xml, so a review assumes it is an index, or it is named sitemap.xml, so a review assumes it is not.Cause: Neither name carries meaning. MDN serves a sitemap index at /sitemap.xml, and WordPress plugins serve one at /sitemap_index.xml. The root element decides, and both files share the same namespace, so a namespace check passes for either. Print the root tag before anything else.

Sign: An index lists children on a CDN host or in a parent directory, and a crawler ignores those children.Cause: Referenced sitemaps must be on the same site as the index, and in the same directory or lower. An index at /sitemaps/index.xml naming /sitemap-blog.xml points above itself and is out of scope. Move the index to the site root, which puts every path in the site under it.

Thresholds

A sitemap index may list up to 50,000 sitemaps and must be no larger than 50 MB. The index read in step 1 lists 10. Source: https://www.sitemaps.org/protocol.html

Referenced sitemaps must be hosted on the same site as the index and must sit in the same directory as the index or lower in the hierarchy. Source: https://developers.google.com/search/docs/crawling-indexing/sitemaps/large-sitemaps

What to check next

How to validate sitemap xml: run the rules check on each child the index names, once they all resolve.
How to check sitemap lastmod: the dates printed in step 2, and the formats that pass a parser but fail the schema.
How to check sitemap and robots txt: the index is the file that belongs on the Sitemap: line, not the children.
How to check robots.txt: where a crawler looks first, and the scope the directory rule is measured from.
How to check if gzip is enabled: every child in step 2 is gzipped, and the 50 MB limit applies after decompression.

FAQ

What is sitemap_index.xml?

A conventional file name for a sitemap index, used by several WordPress plugins. The name has no standing in the protocol. A file is an index when its root element is sitemapindex, whatever it is called, and MDN serves one at /sitemap.xml.

Can a sitemap index point to another sitemap index?

No. The content model in siteindex.xsd gives each sitemap element one loc and an optional lastmod, and the target of that loc is a sitemap. Steps 3 and 4 show both shapes of nesting being refused. Keep the tree two levels deep.

How many URLs can an index cover?

50,000 sitemaps times 50,000 URLs each. Sites past that use several indexes and list each one in robots.txt, because there is no third level to hold them.

Does every child need a lastmod?

No, it is optional. When present it should be the date that child file changed, which lets a crawler skip children that have not moved. This site's own index lists four children and carries none. MDN's carries the same date on all ten entries.

Verified

Verified 2026-09-11by Maks Vernycurl 8.21.0python 3.13.1node 22.23.2

Each output block is what the command above it printed on that date, on the host named in the step. Figures read from a live site move between runs. Compare the shape of the answer rather than the digits, and see the methodology for how a page is re-verified.

Checker: robots-sitemap robots.txt parse, sitemap reachability and validityserver
All crawlability and indexing checkscluster

intermediate12 minpublished 2026-09-11updated 2026-09-11Maks Verny