Health check timeout

Point the probe's own timeout at the endpoint: curl -sS --max-time 3 -o /dev/null -w 'code=%{http_code} total=%{time_total}s\n' http://127.0.0.1:8763/health/nodeadline printed code=000 and exited 28 after 3.006 s. The handler was still waiting on an 8 s dependency. A health endpoint needs a deadline shorter than the probe that polls it.

Why check this

Every health handler that calls a dependency inherits that dependency's worst case. A database that answers in 2 ms most days answers in 30 seconds during a failover, and a handler with no deadline of its own waits for it. Run this check whenever a health endpoint starts touching anything, and after any change to a client library's default timeout.

The failure it prevents is a probe result that carries no information. The service is running, the endpoint is reachable, and the probe gives up before either fact reaches it. Every instance looks identically dead, the platform restarts all of them, and the logs contain no 503 to explain why.

Prerequisites

• Node 18 or later. The fixture uses http and Promise.race.
• curl 7.75 or later, for --max-time, --fail and -w.
• A free port. netstat -ano | grep 8763 prints nothing when 8763 is free.
• curl exit codes: 28 is a timeout, 22 is an HTTP error under --fail.
• The 8 s delay here is a fixed number in a fixture. A real dependency is slow by a distribution, not by a constant, so use this to test the handler's behaviour and measure the budget on your own system.

Save this as slow-health.js. One endpoint waits for the dependency however long it takes, the other gives it 2000 ms and then answers anyway.

// node slow-health.js   listens on http://127.0.0.1:8763
const http = require('http');
const state = { delayMs: 0 };                 // how long the dependency takes to answer
const send = (res, code, body) => {
  res.writeHead(code, { 'content-type': 'application/json', 'cache-control': 'no-store' });
  res.end(JSON.stringify(body));
};
const dependency = () => new Promise((resolve) => setTimeout(() => resolve('ok'), state.delayMs));
const withDeadline = (p, ms) => Promise.race([
  p,
  new Promise((_, reject) => setTimeout(() => reject(new Error(`dependency deadline ${ms} ms exceeded`)), ms))
]);
http.createServer(async (req, res) => {
  const u = new URL(req.url, 'http://127.0.0.1');
  if (u.pathname === '/control') {            // test hook, never shipped
    state.delayMs = Number(u.searchParams.get('delay') ?? 0);
    return send(res, 200, state);
  }
  const t0 = Date.now();
  if (u.pathname === '/health/nodeadline') {  // waits for the dependency, however long it takes
    await dependency();
    return send(res, 200, { status: 'ok', dependency_ms: Date.now() - t0 });
  }
  if (u.pathname === '/health/deadline') {    // gives the dependency 2000 ms, then answers anyway
    try {
      await withDeadline(dependency(), 2000);
      return send(res, 200, { status: 'ok', dependency_ms: Date.now() - t0 });
    } catch (e) {
      return send(res, 503, { status: 'unhealthy', error: e.message, waited_ms: Date.now() - t0 });
    }
  }
  send(res, 404, { status: 'not found' });
}).listen(8763, '127.0.0.1', () => console.log('slow health fixture on 127.0.0.1:8763'));

node slow-health.js & netstat -ano | grep 8763

Steps

1. 1

   Step 1.

   Record the endpoint's cost while the dependency is fast.

   curl -sS -o /dev/null -w 'code=%{http_code} total=%{time_total}s\n' http://127.0.0.1:8763/health/nodeadline
   

   code=200 total=0.008379s

   8 ms. Nothing in this response says what the handler will do when the dependency stops answering, which is why the check continues.

2. 2

   Step 2.

   Make the dependency take 8 seconds.

   curl -sS "http://127.0.0.1:8763/control?delay=8000"
   

   {"delayMs":8000}

3. 3

   Step 3.

   Repeat the first request with the timing split, and no client timeout.

   curl -sS -o /dev/null -w 'code=%{http_code} connect=%{time_connect}s ttfb=%{time_starttransfer}s total=%{time_total}s\n' http://127.0.0.1:8763/health/nodeadline
   

   code=200 connect=0.000424s ttfb=8.001732s total=8.001825s

   The connection was accepted in under a millisecond and the first byte arrived 8 seconds later. The whole wait is the handler blocking on the dependency, and it ends in 200, so the endpoint believes it succeeded.

4. 4

   Step 4.

   Give the request the deadline a probe would give it.

   curl -sS --max-time 3 -o /dev/null -w 'code=%{http_code} total=%{time_total}s\n' http://127.0.0.1:8763/health/nodeadline; echo "exit=$?"
   

   curl: (28) Operation timed out after 3005 milliseconds with 0 bytes received
   code=000 total=3.006128s
   exit=28

   No status code and no body. The handler was going to answer 200 five seconds later, and nothing about the dependency reached the probe.

5. 5

   Step 5.

   Read the endpoint that holds its own deadline, with the same 8 s dependency behind it.

   curl -sS -w '\ncode=%{http_code} total=%{time_total}s\n' http://127.0.0.1:8763/health/deadline
   

   {"status":"unhealthy","error":"dependency deadline 2000 ms exceeded","waited_ms":2008}
   code=503 total=2.009342s

   2009 ms, a status code, and the reason in the body. The dependency is exactly as slow as it was in step 3.

6. 6

   Step 6.

   Put the probe deadline back on that endpoint.

   curl -sS --max-time 3 -o /dev/null -w 'code=%{http_code} total=%{time_total}s\n' http://127.0.0.1:8763/health/deadline; echo "exit=$?"
   

   code=503 total=2.016537s
   exit=0

   The answer arrived inside the budget, so the probe reads a real verdict. Exit 0 is curl reporting a completed request, not a healthy service.

7. 7

   Step 7.

   Run the endpoint without a deadline the way a script would, with --fail and a timeout.

   curl -sf --max-time 3 -o /dev/null http://127.0.0.1:8763/health/nodeadline; echo "exit=$?"
   

   exit=28

8. 8

   Step 8.

   Run the identical command against the endpoint that has one.

   curl -sf --max-time 3 -o /dev/null http://127.0.0.1:8763/health/deadline; echo "exit=$?"
   

   exit=22

   Two exit codes for one dependency fault. 28 says the check never got an answer, 22 says the service answered and declared itself unhealthy. A script that treats every non-zero exit the same throws that difference away.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | code=000, curl exit 28 | The handler is still working on the request | Give the handler a deadline below the probe timeout. | | ttfb close to the dependency's latency | The health check waits on the dependency | Put every dependency call behind a deadline and a cached verdict. | | 503 in less than the probe timeout | The endpoint answered under its own budget | Record the number and alert if it moves toward the probe timeout. | | code=200 after 8 seconds | The handler returns success whenever it finishes | Add the timeout, then assert the total time in the test, not the status alone. | | Exit 28 on some instances only | One instance has a slow dependency, or a probe budget that is too tight | Compare time_starttransfer across instances before changing the probe. |

Thresholds

Common mistakes

What to check next

• How to test a health check endpoint: the status codes and exit codes this page times.
• Deep health check: the handler that acquires these dependency calls in the first place.
• Readiness probe vs liveness probe: which probe should see a slow dependency at all.
• How to test API timeout handling: the same deadline question on the paths that serve customers.

FAQ

What should a health endpoint do when a dependency hangs?

Answer anyway. Give each dependency call a deadline shorter than the probe's timeout, and return 503 naming the check that ran out of time, as in step 5. A late 200 is worth less than an early 503.

How long should a health check take?

Less than the timeout of whatever polls it, with margin for a loaded machine. Take the probe timeout, subtract the margin, and make that the handler's deadline. Then assert the number in a test, because it drifts.

Why does the probe time out while the service is up?

Because the endpoint has not answered yet. The connection is accepted, the process is fine, and the handler is blocked on something slow. Step 3 shows the connection completing in 0.4 ms and the first byte arriving 8 s later.

How do I tell a timeout from a 503 in a script?

By curl's exit code. Steps 7 and 8 run the same command against two endpoints: 28 for a request that never completed, 22 for a 5xx under --fail. Record which one your pipeline saw.

Should the health check retry a slow dependency?

No. A retry inside the handler multiplies the wait by the attempt count and pushes the endpoint further past the probe's deadline. Report the first failure and let the next poll be the retry.

Verified