Deep health check

A deep check runs each dependency's own query and reports it by name. curl -sS http://127.0.0.1:8763/health/deep returned HTTP 200 carrying "status":"degraded" and a failed queue inside the body. A monitor reading only the status code called that service up. Assert on the body, or make the endpoint answer 503.

Why check this

A shallow health check proves the process is listening. A deep check asks each dependency the question the service asks it: a real query on a real connection. Test it before a release that changes a dependency, because its failure mode is silent by construction.

The failure it prevents is a green dashboard over a broken service. The queue is gone, every order times out, and the deep endpoint returns 200 with the word degraded inside a JSON body nobody parses.

Prerequisites

• Node 22 or later. node:sqlite is experimental there, so the fixture is started with --no-warnings.
• sqlite3 on the PATH to build the database file. Version 3.50.6 built the one used here.
• curl 7.75 or later, and a free port. netstat -ano | grep 8763 prints nothing when 8763 is free.
• The Azure health endpoint monitoring pattern for the shape this check is named after.
• SQLite is the database here. A pool, a network hop and a lock wait behave differently, so read the shape of the result, not the milliseconds.

Build two database files: one with the table the service reads, one valid but empty.

sqlite3 health.db "create table orders(id integer primary key, total integer); insert into orders(total) values (100),(250),(75);"

sqlite3 empty.db "pragma user_version=1;"

Save this as deep-health.js. It exposes a shallow endpoint, a deep endpoint written the common way, and a strict one that turns a failed check into a status code.

// node --no-warnings deep-health.js   listens on http://127.0.0.1:8763
const http = require('http');
const { DatabaseSync } = require('node:sqlite');
const state = { db: 'health.db', queue: 'up' };
const send = (res, code, body) => {
  res.writeHead(code, { 'content-type': 'application/json', 'cache-control': 'no-store' });
  res.end(JSON.stringify(body, null, 1));
};
const timed = (fn) => {                     // every check reports its own latency
  const t0 = performance.now();
  try { fn(); return { status: 'ok', ms: +(performance.now() - t0).toFixed(1) }; }
  catch (e) { return { status: 'fail', ms: +(performance.now() - t0).toFixed(1), error: e.message }; }
};
const open = () => new DatabaseSync(state.db, { readOnly: true });
const collect = () => ({
  database_ping: timed(() => { const d = open(); d.prepare('select 1').get(); d.close(); }),
  database_query: timed(() => { const d = open(); d.prepare('select count(*) as n from orders').get(); d.close(); }),
  queue: timed(() => { if (state.queue !== 'up') throw new Error('connect ECONNREFUSED 127.0.0.1:6379'); })
});
http.createServer((req, res) => {
  const u = new URL(req.url, 'http://127.0.0.1');
  const p = u.searchParams;
  if (u.pathname === '/control') {          // test hook, never shipped
    if (p.has('db')) state.db = p.get('db');
    if (p.has('queue')) state.queue = p.get('queue');
    return send(res, 200, state);
  }
  if (u.pathname === '/healthz') return send(res, 200, { status: 'ok' });   // shallow
  const checks = collect();
  const failed = Object.entries(checks).filter(([, c]) => c.status !== 'ok').map(([k]) => k);
  if (u.pathname === '/health/deep')        // the common shape: 200 whatever it found
    return send(res, 200, { status: failed.length ? 'degraded' : 'ok', checks });
  if (u.pathname === '/health/deep/strict') // the shape a load balancer can act on
    return send(res, failed.length ? 503 : 200, { status: failed.length ? 'unhealthy' : 'ok', failed, checks });
  send(res, 404, { status: 'not found' });
}).listen(8763, '127.0.0.1', () => console.log('deep health fixture on 127.0.0.1:8763'));

Save this as assert-deep.mjs. It is the check a test should make when the endpoint refuses to use status codes.

// node assert-deep.mjs <url>   exits 1 when any check inside the body failed
const r = await fetch(process.argv[2]);
const body = await r.json();
const bad = Object.entries(body.checks).filter(([, c]) => c.status !== 'ok').map(([k]) => k);
console.log(`http=${r.status} body.status=${body.status} failed=${bad.length ? bad.join(',') : 'none'}`);
process.exit(r.status < 400 && bad.length === 0 ? 0 : 1);

Steps

1. 1

   Step 1.

   Read the deep endpoint with every dependency working, to record the shape.

   curl -sS http://127.0.0.1:8763/health/deep
   

   {
   "status": "ok",
   "checks": {
   "database_ping": {
    "status": "ok",
    "ms": 0.5
   },
   "database_query": {
    "status": "ok",
    "ms": 0.5
   },
   "queue": {
    "status": "ok",
    "ms": 0
   }
   }
   }

   Three named checks, each with its own latency. That per-check number is what makes the body worth returning.

2. 2

   Step 2.

   Compare what the shallow and the deep endpoint cost.

   curl -sS -o /dev/null -o /dev/null -w '%{time_total}s  %{url}\n' http://127.0.0.1:8763/healthz http://127.0.0.1:8763/health/deep
   

   0.001285s  http://127.0.0.1:8763/healthz
   0.000990s  http://127.0.0.1:8763/health/deep

   The deep endpoint came back faster than the shallow one, so the difference is under the noise of one request pair on loopback. Quote no number from here: on a real system the deep check is the one that opens sockets.

3. 3

   Step 3.

   Take the queue away, leaving the database alone.

   curl -sS "http://127.0.0.1:8763/control?queue=down"
   

   {
   "db": "health.db",
   "queue": "down"
   }

4. 4

   Step 4.

   Read the status code of all three endpoints against that one fault.

   curl -sS -o /dev/null -o /dev/null -o /dev/null -w '%{http_code}  %{url}\n' http://127.0.0.1:8763/healthz http://127.0.0.1:8763/health/deep http://127.0.0.1:8763/health/deep/strict
   

   200  http://127.0.0.1:8763/healthz
   200  http://127.0.0.1:8763/health/deep
   503  http://127.0.0.1:8763/health/deep/strict

   The deep endpoint found the fault and still answered 200. Only the strict handler turned it into a status code a load balancer acts on.

5. 5

   Step 5.

   Read what the deep endpoint put in the body instead.

   curl -sS http://127.0.0.1:8763/health/deep
   

   {
   "status": "degraded",
   "checks": {
   "database_ping": {
    "status": "ok",
    "ms": 0.3
   },
   "database_query": {
    "status": "ok",
    "ms": 0.2
   },
   "queue": {
    "status": "fail",
    "ms": 0,
    "error": "connect ECONNREFUSED 127.0.0.1:6379"
   }
   }
   }

   The failure is named, with the error text and the port. All of it sits behind a 200.

6. 6

   Step 6.

   Run the check the way an uptime monitor runs it, on the status code alone.

   curl -sf -o /dev/null http://127.0.0.1:8763/health/deep && echo "monitor: service is UP"
   

   monitor: service is UP

   --fail makes curl exit non-zero on 4xx and 5xx, and there is no 5xx here. The monitor is green while the service cannot process an order.

7. 7

   Step 7.

   Run the check that reads the body.

   node assert-deep.mjs http://127.0.0.1:8763/health/deep; echo "exit=$?"
   

   http=200 body.status=degraded failed=queue
   exit=1

   Same request, opposite verdict. A test of an endpoint that answers 200 either parses the body or tests nothing.

8. 8

   Step 8.

   Restore the queue and point the service at a database file that exists and has no tables.

   curl -sS "http://127.0.0.1:8763/control?queue=up&db=empty.db"
   

   {
   "db": "empty.db",
   "queue": "up"
   }

   This is the state after a migration that did not run, or a connection pointed at the wrong schema.

9. 9

   Step 9.

   Read the strict endpoint and compare the two database checks inside it.

   curl -sS http://127.0.0.1:8763/health/deep/strict
   

   {
   "status": "unhealthy",
   "failed": [
   "database_query"
   ],
   "checks": {
   "database_ping": {
    "status": "ok",
    "ms": 0.4
   },
   "database_query": {
    "status": "fail",
    "ms": 0.3,
    "error": "no such table: orders"
   },
   "queue": {
    "status": "ok",
    "ms": 0
   }
   }
   }

   select 1 succeeded and select count(*) from orders failed on the same connection. A database check that runs a ping proves the driver can talk to a file, not that the service can read its data.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | 200 with every check ok | The dependencies answered | Record each ms value as the baseline for the next release. | | 200 with a check fail inside | The endpoint found a fault and hid it | Return 503 for it, or make every consumer parse the body, as in step 7. | | 503 with a failed array | The endpoint is usable by a load balancer | Confirm the pool drains, then follow the named dependency. | | database_ping ok, database_query fail | The connection works and the schema does not | A migration did not run, or the connection points at the wrong database. | | The deep endpoint is slower than the request it protects | The check is doing more work than the service | Cache the result for a few seconds and serve the cached verdict. |

Common mistakes

What to check next

• How to test a health check endpoint: the shallow endpoint this one extends.
• Readiness probe vs liveness probe: which probe a dependency check belongs to, and what the wrong one costs.
• Health check timeout: a dependency that stops answering rather than failing.
• How to test connection pool exhaustion: the fault a deep check should catch and usually cannot.

FAQ

What status code should an unhealthy health check return?

503, with a body naming the failing check. It is the status for a service that cannot handle the request now, and pools act on it. A 200 carrying the word unhealthy needs every reader to parse the body.

What is a deep health check?

A health endpoint that exercises the dependencies the service needs instead of confirming the process is listening. It runs a real query and reports each result separately, with its own latency.

Which dependencies belong in a health check?

The ones a request cannot be served without. Everything else belongs in the body as information. A check that fails on an optional analytics sink takes a working service out of the pool.

Should a deep health check be reachable from the internet?

No. The body names internal hosts, ports and error text, as in step 5. Bind it to an internal listener, and confirm an anonymous request gets nothing useful.

How often should a deep check run?

Less often than it is polled. Compute the verdict on a timer and serve the cached result, so probes from every instance do not become load on the dependency.

Verified