How to check for broken links on a website

Extract every href from the page, request each URL with GET rather than HEAD, and compare the body of each 200 response against the body of a URL that cannot exist. On a local test site, a HEAD-only run called two working links broken and passed one broken link as healthy.

Why check this

Links break on a schedule you can predict: after a content migration, after a route rename, after a CMS export, and after any redirect map is rewritten. That makes this a sign-off check on staging and a scheduled check on production, not something to run once a year.

The failure it catches is the link that still answers. A help article is renamed, the old path now falls through to the application shell, and the shell answers 200 with a "page not found" panel. Uptime monitoring sees 200. Analytics sees a visit. The reader sees nothing, and no alert fires.

Prerequisites

import { createServer } from 'node:http';

const html = { 'content-type': 'text/html; charset=utf-8' };
const home = `<!doctype html><html lang="en"><meta charset="utf-8"><title>Link test</title>
<h1>Link test page</h1><ul>
<li><a href="/ok">Working page</a></li>
<li><a href="/missing">Deleted page</a></li>
<li><a href="/sale">Summer sale</a></li>
<li><a href="/report.pdf">Quarterly report</a></li>
<li><a href="/members">Members area</a></li>
<li><a href="/old">Old address</a></li></ul>`;
const notFound = `<!doctype html><html lang="en"><meta charset="utf-8"><title>Page not found</title>
<h1>Sorry, we could not find that page</h1><p><a href="/">Back to the home page</a></p>`;

createServer((req, res) => {
  switch (req.url) {
    case '/': return res.writeHead(200, html).end(home);
    case '/ok': return res.writeHead(200, html).end('<!doctype html><title>Working</title><h1>Working page</h1>');
    case '/missing': return res.writeHead(404, html).end(notFound);
    // Soft 404: the not-found page, served with 200.
    case '/sale': return res.writeHead(200, html).end(notFound);
    // Rejects HEAD, serves GET. Common on static file handlers.
    case '/report.pdf':
      if (req.method === 'HEAD') return res.writeHead(405, { allow: 'GET' }).end();
      return res.writeHead(200, { 'content-type': 'application/pdf' }).end('%PDF-1.4 stub');
    // A rule that answers any method other than GET with 403.
    case '/members':
      if (req.method === 'HEAD') return res.writeHead(403, html).end();
      return res.writeHead(200, html).end('<!doctype html><title>Members</title><h1>Members area</h1>');
    case '/old': return res.writeHead(301, { location: '/ok' }).end();
    default: return res.writeHead(404, html).end(notFound);
  }
}).listen(8731, () => console.log('link test site on http://localhost:8731/'));

Steps

  1. Step 1.

    List the links the markup declares, before any script runs.

    curl -s http://localhost:8731/ | grep -o -E 'href="[^"]+"' | sort -u
    
    href="/members"
    href="/missing"
    href="/ok"
    href="/old"
    href="/report.pdf"
    href="/sale"

    Six candidates. A real extraction also resolves relative paths against the page URL and drops mailto:, tel: and bare fragments.

  2. Step 2.

    Request each one with HEAD, which is what most link checkers send by default.

    for p in /ok /missing /sale /report.pdf /members /old; do printf '%-12s ' "$p"; curl -s -o /dev/null -w '%{http_code}\n' -I "http://localhost:8731$p"; done
    
    /ok          200
    /missing     404
    /sale        200
    /report.pdf  405
    /members     403
    /old         301

    Read as a verdict, this says one link is broken and two more are suspect.

  3. Step 3.

    Request the same six with GET and put the two columns next to each other.

    for p in /ok /missing /sale /report.pdf /members /old; do printf '%-12s HEAD %s  GET %s\n' "$p" "$(curl -s -o /dev/null -w '%{http_code}' -I http://localhost:8731$p)" "$(curl -s -o /dev/null -w '%{http_code}' http://localhost:8731$p)"; done
    
    /ok          HEAD 200  GET 200
    /missing     HEAD 404  GET 404
    /sale        HEAD 200  GET 200
    /report.pdf  HEAD 405  GET 200
    /members     HEAD 403  GET 200
    /old         HEAD 301  GET 301

    Two rows disagree. /report.pdf and /members answer 405 and 403 to HEAD and 200 to GET. A reader clicking either link gets the file. Neither answer is a statement about the URL: one comes from a handler that implements GET only, the other from a rule that filters by method.

  4. Step 4.

    Catch the links that answer 200 with an error page. Request a URL that cannot exist, keep its body, and compare every other body against it.

    node -e "
    const base = 'http://localhost:8731';
    const paths = ['/ok', '/missing', '/sale', '/report.pdf', '/members', '/old'];
    (async () => {
      const probe = await fetch(base + '/h2check-probe-' + Date.now());
      const baseline = await probe.text();
      console.log('probe:', probe.status, baseline.length, 'bytes');
      for (const p of paths) {
        const r = await fetch(base + p, { redirect: 'manual' });
        const body = await r.text();
        console.log(String(r.status).padEnd(4), p.padEnd(12), String(body.length).padStart(4), 'bytes',
          body === baseline ? 'identical to probe' : '');
      }
    })();
    "
    
    probe: 404 170 bytes
    200  /ok            58 bytes
    404  /missing      170 bytes identical to probe
    200  /sale         170 bytes identical to probe
    200  /report.pdf    13 bytes
    200  /members       58 bytes
    301  /old            0 bytes

    /sale returns 200 and the byte-for-byte not-found page. That is the link no status check finds. On a site that injects a timestamp or a request id into every page, compare the <title> or a container element instead of the whole body.

  5. Step 5.

    Follow the redirects, because a 301 is a verdict about the wrong URL.

    for p in /old /missing /sale; do printf '%-12s ' "$p"; curl -sL -o /dev/null -w 'final %{http_code}  hops %{num_redirects}  %{url_effective}\n' "http://localhost:8731$p"; done
    
    /old         final 200  hops 1  http://localhost:8731/ok
    /missing     final 404  hops 0  http://localhost:8731/missing
    /sale        final 200  hops 0  http://localhost:8731/sale

    /old is healthy: one hop to a page that exists. Record the hop count as well as the final code, since a link that works through four redirects is still a defect worth a ticket.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | 404 or 410 on GET | The link is broken | Fix the href, or redirect the old path to its replacement. | | 403 or 405 on HEAD, 200 on GET | The server rejects the method, not the URL | Re-request with GET before reporting. Switch the checker to GET. | | 200 with the not-found body | A soft 404 | The worst case: no status says so. Read How to check soft 404. | | 301 to a URL that then 404s | A redirect map points at a deleted page | Follow the chain to its end and judge the final response. | | 429 during a run | You are sending faster than the host allows | Lower concurrency, add a delay, and never point the run at a third party. |

Common mistakes

Sign: The checker reports 403 or 405 on links that open in a browser.Cause: It sent HEAD. In the run above, /report.pdf answered 405 and /members answered 403 to HEAD, and both answered 200 to GET. HEAD is cheaper, so tools default to it, and a server is free not to implement it. Verify every non-2xx with a GET before it reaches a report.
Sign: The report is clean and users still land on a not-found page.Cause: The application answers 200 for unknown routes and renders an error panel. Status codes cannot see it. Fetch a URL that cannot exist, keep the body, and flag any 200 whose body matches it.
Sign: A run against a live third-party site returns 429, or the requests stop arriving.Cause: Requesting every outbound link at speed is a crawl, and the operator pays for it. Check internal links against your own build or staging. For external links, sample rather than sweep, keep one request per host in flight, and space them out.

What to check next

FAQ

What are broken links on a website?

Links whose target no longer serves the content they promise. That includes 404 and 410 responses, hosts that no longer resolve, and pages that answer 200 with an error body. The last group is the largest on sites that render routes in JavaScript.

How do I find broken links on a website?

Crawl your own build, collect every href, and request each one with GET. Compare the response body of every 200 against a known not-found body. Do the run on staging, where a crawl costs nothing and the results are not confused by a CDN.

How do I check if a single link is broken?

Run curl -sIL -o /dev/null -w '%{http_code} %{url_effective}\n' URL. If the code is not 2xx, repeat without -I before deciding, because the server may be rejecting HEAD rather than the URL.

Do broken links show up in the browser console?

A broken <a href> produces nothing until someone clicks it. Failed subresources do log there, which is a separate check: How to check console errors on a website.

Verified

Verified by Maks Vernycurl 8.21.0node 22.23.2

Each output block is what the command above it printed on that date, on the host named in the step. Figures read from a live site move between runs. Compare the shape of the answer rather than the digits, and see the methodology for how a page is re-verified.

basic8 minpublished updated Maks Verny