How to check if a queue worker is running

Stop asking whether the process exists. Enqueue a canary job with a deadline and read whether the worker claimed and finished it. Against a worker stalled inside job 3, node canary.mjs 5000 reported FAIL, state is pending after 5097 ms, while its pid, its heartbeat and its /health endpoint all still said alive.

Why check this

A worker that exits gets noticed. A worker that stays up and stops working does not, because every cheap signal keeps answering: the pid is in the process table, the heartbeat timer keeps firing, the health endpoint returns 200.

Run this after a deploy, after a dependency upgrade, and when a ticket says a job never arrived. The failure it prevents: one worker holds a claim on job 3 forever, 37 jobs pile up behind it, and the dashboard stays green because nobody reads completions.

Presence and progress are different questions. Five states of one worker separate them.

Prerequisites

Node 22. The queue is a table in node:sqlite, experimental and warning on import. Step 1 shows it, the rest set NODE_NO_WARNINGS=1.
Port 8932 on 127.0.0.1 for the health endpoint, confirmed free with netstat -ano | grep 8932.
The schema. Save as queue-init.mjs.

// Creates the queue and the worker registry. state: pending | running | done.
import { DatabaseSync } from 'node:sqlite';
import { rmSync } from 'node:fs';

for (const f of ['queue.db', 'queue.db-wal', 'queue.db-shm']) rmSync(f, { force: true });
const db = new DatabaseSync('queue.db');
db.exec(`
  PRAGMA journal_mode = WAL;
  CREATE TABLE jobs (
    id          INTEGER PRIMARY KEY,
    enqueued_at INTEGER NOT NULL,
    claimed_at  INTEGER,
    claimed_by  TEXT,
    done_at     INTEGER,
    state       TEXT NOT NULL DEFAULT 'pending'
  );
  CREATE TABLE workers (
    name    TEXT PRIMARY KEY,
    pid     INTEGER NOT NULL,
    beat_at INTEGER NOT NULL
  );
`);
console.log(db.prepare('SELECT name FROM sqlite_master WHERE type = ?').all('table').map((r) => r.name).join(' '));
db.close();

The producer. Save as produce.mjs.

// produce.mjs <n>. Enqueues n jobs in one go.
import { DatabaseSync } from 'node:sqlite';

const n = Number(process.argv[2]);
const db = new DatabaseSync('queue.db');
db.exec('PRAGMA busy_timeout = 5000');
const ins = db.prepare('INSERT INTO jobs (enqueued_at) VALUES (?)');
for (let i = 0; i < n; i += 1) ins.run(Date.now());
console.log(`enqueued ${n}`);
db.close();

The worker. It publishes the three signals a real one does: a pid, a heartbeat on a timer, a health endpoint. Its two failure modes are arguments, not edits. Save as worker.mjs.

// worker.mjs <name> <mode> <service_ms> <port>
// Claims one job at a time, writes a heartbeat on a 1000 ms timer, and answers
// GET /health. mode normal works through the queue; hang stalls the third job on
// a promise that never settles; block spins the third job synchronously.
import { DatabaseSync } from 'node:sqlite';
import { createServer } from 'node:http';

const [name, mode, svcArg, portArg] = process.argv.slice(2);
const svc = Number(svcArg);
const port = Number(portArg);

const db = new DatabaseSync('queue.db');
db.exec('PRAGMA busy_timeout = 5000');
const beat = db.prepare(
  `INSERT INTO workers (name, pid, beat_at) VALUES (?, ?, ?)
     ON CONFLICT(name) DO UPDATE SET pid = excluded.pid, beat_at = excluded.beat_at`,
);
const claim = db.prepare(
  `UPDATE jobs SET state = 'running', claimed_at = ?, claimed_by = ?
     WHERE id = (SELECT id FROM jobs WHERE state = 'pending' ORDER BY id LIMIT 1)
   RETURNING id`,
);
const finish = db.prepare("UPDATE jobs SET state = 'done', done_at = ? WHERE id = ?");
const release = db.prepare(
  "UPDATE jobs SET state = 'pending', claimed_at = NULL, claimed_by = NULL WHERE state = 'running' AND claimed_by = ?",
);

process.on('SIGTERM', () => {
  console.log(`${name} SIGTERM handler ran, releasing claims`);
  release.run(name);
  process.exit(0);
});

beat.run(name, process.pid, Date.now());
setInterval(() => beat.run(name, process.pid, Date.now()), 1000);

createServer((req, res) => {
  res.writeHead(200, { 'content-type': 'application/json' });
  res.end(JSON.stringify({ status: 'ok', worker: name, pid: process.pid }));
}).listen(port, '127.0.0.1', () => console.log(`${name} pid ${process.pid} listening on ${port}`));

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
let done = 0;
for (;;) {
  const row = claim.get(Date.now(), name);
  if (!row) { await sleep(50); continue; }
  if (mode === 'hang' && done === 2) {
    console.log(`${name} stalled on job ${row.id}, awaiting a promise that never settles`);
    await new Promise(() => {});
  }
  if (mode === 'block' && done === 2) {
    console.log(`${name} blocking on job ${row.id}, synchronous spin`);
    for (;;) Math.sqrt(Math.random());
  }
  await sleep(svc);
  finish.run(Date.now(), row.id);
  done += 1;
}

The check. Three presence readings against four progress readings. Save as liveness.mjs.

// liveness.mjs <worker> <port>. Prints the three presence readings and the four
// progress readings side by side, then a verdict built only from progress.
import { DatabaseSync } from 'node:sqlite';

const name = process.argv[2];
const port = Number(process.argv[3]);
const db = new DatabaseSync('queue.db');
const w = db.prepare('SELECT pid, beat_at FROM workers WHERE name = ?').get(name);
const q = db.prepare(`
  SELECT (SELECT COUNT(*) FROM jobs WHERE state = 'pending') AS depth,
         (SELECT COUNT(*) FROM jobs WHERE state = 'running') AS running,
         (SELECT COUNT(*) FROM jobs WHERE state = 'done')    AS done,
         (SELECT MAX(claimed_at) FROM jobs)                  AS last_claim,
         (SELECT MAX(done_at) FROM jobs)                     AS last_done
`).get();

let exists = 'no';
try { process.kill(w.pid, 0); exists = 'yes'; } catch (e) { exists = e.code; }

const t0 = Date.now();
let health;
try {
  const r = await fetch(`http://127.0.0.1:${port}/health`, { signal: AbortSignal.timeout(3000) });
  health = `${r.status} in ${Date.now() - t0} ms`;
} catch (e) { health = `${e.cause?.code ?? e.name} after ${Date.now() - t0} ms`; }

const age = (t) => (t === null ? 'never' : `${Date.now() - t} ms ago`);
const row = (k, v) => console.log(`${k.padEnd(24)}${v}`);
row('pid in process table', `${w.pid} ${exists}`);
row('GET /health', health);
row('heartbeat', age(w.beat_at));
row('depth pending', String(q.depth));
row('rows in running', String(q.running));
row('jobs done', String(q.done));
row('last claim', age(q.last_claim));
row('last completion', age(q.last_done));
const waiting = q.depth + q.running;
const since = q.last_done === null ? Infinity : Date.now() - q.last_done;
const ago = q.last_done === null ? 'ever' : `for ${since} ms`;
row('verdict', waiting > 0 && since > 5000
  ? `NOT WORKING: ${waiting} jobs waiting, nothing completed ${ago}`
  : `working: ${waiting} jobs waiting, last completion ${q.last_done === null ? 'never' : `${since} ms ago`}`);
db.close();

The canary. Save as canary.mjs.

// canary.mjs <deadline_ms>. Enqueues one job and polls until it is done or the
// deadline passes. The only reading here that crosses the whole path.
import { DatabaseSync } from 'node:sqlite';

const deadline = Number(process.argv[2]);
const db = new DatabaseSync('queue.db');
db.exec('PRAGMA busy_timeout = 5000');
const id = db.prepare('INSERT INTO jobs (enqueued_at) VALUES (?) RETURNING id').get(Date.now()).id;
const read = db.prepare('SELECT state, enqueued_at, claimed_at, done_at FROM jobs WHERE id = ?');
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

const t0 = Date.now();
while (Date.now() - t0 < deadline) {
  if (read.get(id).state === 'done') break;
  await sleep(100);
}
const r = read.get(id);
console.log(`canary job ${id}, deadline ${deadline} ms`);
console.log(`  claimed   ${r.claimed_at ? `${r.claimed_at - r.enqueued_at} ms after enqueue` : 'no'}`);
console.log(`  finished  ${r.done_at ? `${r.done_at - r.enqueued_at} ms after enqueue` : 'no'}`);
console.log(`  verdict   ${r.state === 'done' ? 'PASS' : `FAIL, state is ${r.state} after ${Date.now() - t0} ms`}`);
db.close();

The shutdown harness for step 7. Save as stop-test.mjs.

// stop-test.mjs <port>. Forks the worker, lets it claim, sends SIGTERM, and reports
// what the child did on the way out and what it left behind in the queue.
import { fork } from 'node:child_process';
import { DatabaseSync } from 'node:sqlite';

const child = fork('worker.mjs', ['w1', 'normal', '3000', process.argv[2]], { stdio: 'inherit' });
child.on('exit', (code, signal) => {
  console.log(`child exited code=${code} signal=${signal}`);
  const db = new DatabaseSync('queue.db');
  const left = db.prepare("SELECT id, claimed_by FROM jobs WHERE state = 'running'").all();
  console.log(`rows left in running: ${JSON.stringify(left)}`);
  let exists = 'no';
  try { process.kill(child.pid, 0); exists = 'yes'; } catch (e) { exists = e.code; }
  console.log(`pid ${child.pid} in process table after exit: ${exists}`);
  db.close();
});
setTimeout(() => { console.log('sending SIGTERM'); child.kill('SIGTERM'); }, 2000);

Steps

1
Step 1.
Create the queue and the worker registry.
```
node queue-init.mjs
```
```
jobs workers
(node:43464) ExperimentalWarning: SQLite is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
```
Two tables, because the check needs both sides. workers is what the worker says about itself, jobs is what it did. Run export NODE_NO_WARNINGS=1 before the rest.

Start a working consumer and take the baseline reading.

node queue-init.mjs > /dev/null
node produce.mjs 40
node worker.mjs w1 normal 400 8932 > worker.log 2>&1 &
sleep 6
cat worker.log
node liveness.mjs w1 8932

enqueued 40
w1 pid 19000 listening on 8932
pid in process table    19000 yes
GET /health             200 in 45 ms
heartbeat               865 ms ago
depth pending           25
rows in running         1
jobs done               14
last claim              197 ms ago
last completion         198 ms ago
verdict                 working: 26 jobs waiting, last completion 198 ms ago

Note the pid, 19000. The verdict rests on the last completion, 198 ms ago against a 400 ms service time. The three readings above it look the same in every step but one.

Stall the same worker inside a job and repeat the reading.

powershell -Command "Stop-Process -Id 19000 -Force"
node queue-init.mjs > /dev/null
node produce.mjs 40 > /dev/null
node worker.mjs w1 hang 400 8932 > worker.log 2>&1 &
sleep 12
cat worker.log
node liveness.mjs w1 8932

w1 pid 39816 listening on 8932
w1 stalled on job 3, awaiting a promise that never settles
pid in process table    39816 yes
GET /health             200 in 57 ms
heartbeat               653 ms ago
depth pending           37
rows in running         1
jobs done               2
last claim              10924 ms ago
last completion         10925 ms ago
verdict                 NOT WORKING: 38 jobs waiting, nothing completed for 10925 ms

This is the shape that ships. The pid is there, the heartbeat is 653 ms old, /health answered 200 in 57 ms, and two jobs finished in twelve seconds. Heartbeat and endpoint both run on the event loop, which the stalled job never occupies.

4
Step 4.
Send a canary job through the stalled worker with a 5 second deadline.
```
node canary.mjs 5000
```
```
canary job 41, deadline 5000 ms
claimed   no
finished  no
verdict   FAIL, state is pending after 5097 ms
```
The canary needs no registry, no pid, no endpoint. It crosses the producer, the queue and the consumer, and answers in one number.

Block the worker's event loop instead of its work loop, and compare the presence readings.

powershell -Command "Stop-Process -Id 39816 -Force"
node queue-init.mjs > /dev/null
node produce.mjs 40 > /dev/null
node worker.mjs w1 block 400 8932 > worker.log 2>&1 &
sleep 12
cat worker.log
node liveness.mjs w1 8932

w1 pid 44208 listening on 8932
w1 blocking on job 3, synchronous spin
pid in process table    44208 yes
GET /health             TimeoutError after 3056 ms
heartbeat               14914 ms ago
depth pending           37
rows in running         1
jobs done               2
last claim              14081 ms ago
last completion         14082 ms ago
verdict                 NOT WORKING: 38 jobs waiting, nothing completed for 14083 ms

Same damage as step 3, opposite symptoms. A synchronous block starves the timer and the socket, so the heartbeat froze at 14914 ms and /health never answered. The progress rows are identical in both.

Let a healthy worker go idle, and see the stall rule refuse to fire.

powershell -Command "Stop-Process -Id 44208 -Force"
node queue-init.mjs > /dev/null
node produce.mjs 3 > /dev/null
node worker.mjs w1 normal 400 8932 > worker.log 2>&1 &
sleep 14
node liveness.mjs w1 8932
node canary.mjs 5000

pid in process table    32872 yes
GET /health             200 in 52 ms
heartbeat               797 ms ago
depth pending           0
rows in running         0
jobs done               3
last claim              13085 ms ago
last completion         12660 ms ago
verdict                 working: 0 jobs waiting, last completion 12661 ms ago

canary job 4, deadline 5000 ms
claimed   30 ms after enqueue
finished  447 ms after enqueue
verdict   PASS

Nothing completed for 12.6 seconds and the worker is fine: the quiet is an empty queue. A rule on completion age alone pages someone here; conditioned on depth + running > 0 it stays silent, and the canary settles it in 447 ms.

Kill a worker mid-job and read what it left behind.

powershell -Command "Stop-Process -Id 32872 -Force"
node queue-init.mjs > /dev/null
node produce.mjs 5 > /dev/null
node stop-test.mjs 8932
node liveness.mjs w1 8932

w1 pid 40244 listening on 8932
sending SIGTERM
child exited code=null signal=SIGTERM
rows left in running: [{"id":1,"claimed_by":"w1"}]
pid 40244 in process table after exit: ESRCH
pid in process table    40244 ESRCH
GET /health             ECONNREFUSED after 68 ms
heartbeat               1349 ms ago
depth pending           4
rows in running         1
jobs done               0
last claim              2353 ms ago
last completion         never
verdict                 NOT WORKING: 5 jobs waiting, nothing completed ever

On Windows 11 with Node 22.23.2, child.kill('SIGTERM') against a forked worker never ran the SIGTERM handler: the line it prints is absent, job 1 stayed in running with claimed_by set, and the child exited code=null signal=SIGTERM. That is the one state the presence readings catch alone.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | pid present, /health 200, heartbeat under a second, completions frozen | The work loop is stalled and the event loop is not. Step 3 | Read the claim it is holding and go to the job, not the process | | Heartbeat older than its interval and /health timing out | The event loop is blocked. Step 5 timed out at 3056 ms with a 14914 ms heartbeat | Profile the job body. A synchronous call in the handler stops everything the process publishes | | pid ESRCH, /health ECONNREFUSED, a row still in running | The worker died holding a claim. Step 7 | Requeue by claim age, not by worker name. The claim outlives the pid | | Heartbeat fresh, pid ESRCH | The registry is one interval stale. Step 7 read 1349 ms after the process was gone | Compare the heartbeat to the pid before trusting either | | Completions old, depth pending and rows in running both 0 | An idle worker, not a dead one. Step 6 went 12660 ms with nothing wrong | Gate the stall rule on work being available, then confirm with a canary | | Canary claimed but never finished | The worker claims and then stalls, so depth still drains | Set the canary deadline above the longest measured service time, or it reports its own impatience. 5000 ms against 400 ms of service here |

Common mistakes

Sign: The heartbeat is under a second old, the health endpoint returns 200, and no job has completed for ten seconds.Cause: Both signals are produced by the event loop, and an await that never settles leaves the event loop free. Step 3 wrote a heartbeat 653 ms old while the worker sat on job 3 with 37 jobs behind it. A heartbeat written by a timer reports the timer, not the work.

Sign: A worker is gone and the liveness rule calls it healthy for a few more seconds.Cause: Step 7 read the heartbeat 1349 ms after the process had left the process table. A heartbeat is only ever as fresh as its interval, so a rule with a five second threshold on a one second beat is blind for up to five seconds after the death it exists to catch.

Sign: An alert on time since the last completion fires every night at low traffic.Cause: Step 6 shows a working consumer 12660 ms after its last completion, with an empty queue, and a canary that passed in 447 ms. Silence means nothing on its own. The condition is work waiting and nothing finishing, which is why the verdict line reads depth and running before it reads a clock.

Sign: A graceful shutdown test passes locally and leaves claimed rows behind in the pipeline.Cause: On Windows 11 with Node 22.23.2, child.kill('SIGTERM') on a forked worker terminated it without running the SIGTERM handler: exit was code=null signal=SIGTERM and job 1 stayed in running. Signal delivery is not portable, so a shutdown path tested only by signal on this platform is untested.

What to check next

How to test a stuck background job: the job's side of this failure.
How to test graceful shutdown: what step 7 should have done instead.
How to check queue depth and consumer lag: the rates behind these columns.
Readiness probe vs liveness probe: why the 200 in steps 2 and 3 proves two different things.
Deep health check: an endpoint that answers for the work, not the process.

FAQ

How do I test a message queue?

Drive it end to end. Enqueue a job you control, then assert it was claimed and completed inside a deadline. The canary in step 4 is the smallest form: one insert, one poll loop, one verdict.

Is a process check enough to prove a worker is running?

No. process.kill(pid, 0) answers whether the pid exists and nothing else. It returned yes in steps 3 and 5, where two jobs finished in twelve seconds with 37 waiting.

Why does the health endpoint return 200 while the queue is stuck?

Because it is served by the event loop and the stalled job is not on it. In step 3 it answered in 57 ms. An endpoint that proves consuming has to read the queue.

How often should the canary run?

Often enough that the gap between runs is shorter than the outage you are willing to own. One run costs an insert and a poll, 447 ms on an idle worker here. Tag the row so side effects are skipped.

Verified

Verified 2026-09-12by Maks Vernynode 22.23.2node:sqlite (SQLite) 3.51.3GNU bash 5.2.15

Each output block is what the command above it printed on that date, on the host named in the step. Figures read from a live site move between runs. Compare the shape of the answer rather than the digits, and see the methodology for how a page is re-verified.

All background jobs and queues checkscluster

intermediate9 minpublished 2026-09-12updated 2026-09-12Maks Verny