h2check

robots.txt and sitemap checker

Reads a site robots.txt, works out whether a given URL is allowed for a given crawler, and follows the Sitemap lines to check that each one is reachable and parses.

Checker offline. Follow the manual steps below, they give the same answer.

What it checks

  • The robots.txt status code. A 404 means everything is allowed; a 500 means well-behaved crawlers stay away entirely.
  • Which User-agent group applies, since the most specific group wins and the others are ignored rather than merged.
  • Whether a URL is allowed, by longest-match with Allow winning ties, not by the order the rules appear in.
  • Wildcards and end-of-match anchors in the paths.
  • Every Sitemap line, whether the file is reachable, and whether it parses.
  • A sitemap URL on a different host from the robots.txt that names it.

What is not stored

This checker runs on a server, and the server is not deployed yet, so the form above is disabled. The manual steps on the procedures below return the same answer from your own machine, and they are what the checker automates.

Limits

  • Allowed is not the same as indexable. A URL can pass every robots rule and still carry a noindex directive.
  • Nothing here reports whether a search engine has indexed a page. That question needs Search Console and a verified property.
  • Rules are read as written. A crawler that ignores robots.txt is not affected by any of this.

Procedures that explain the result

Put it on your own page

Link to it with the value filled in, and the checker runs as soon as the page opens.

<a href="https://h2check.org/tools/robots-sitemap/">robots.txt and sitemap checker</a>