How to test graceful shutdown

Stop a worker the way your deploy stops it, then read the queue row it was holding. node deploy.mjs ipc drain 30 printed finished job 1 and exited code=0, and the row read done. A stop the worker never receives leaves that row in running under a dead pid, with no outcome.

Why check this

Run this before a release that changes the worker, and after a change to how a job is claimed or acknowledged. A deploy stops every worker in the pool at the same moment, so whatever happens to one job in flight happens to all of them.

Draining and stopping are different promises. A draining worker refuses new work and finishes what it holds. A releasing worker puts the job back for somebody else. A worker that does neither leaves the claim behind. The runs below produced all three from one fixture.

Prerequisites

• Node 22. node:sqlite prints an ExperimentalWarning on every command here. See its documentation.
• No broker. The queue is one SQLite table and the workers are forked Node processes.
• One free local port for step 7. The run used 8935, checked free first.
• The queue. Save as queue.mjs.

// The queue: one SQLite table. Run `node queue.mjs` once to create and seed it.
import { DatabaseSync } from 'node:sqlite';
export const db = new DatabaseSync('queue.db');
db.exec('PRAGMA journal_mode = WAL');
db.exec('PRAGMA busy_timeout = 5000');

export const claim = (w) => db.prepare(
  `UPDATE jobs SET state='running', attempts=attempts+1, worker=?, worker_pid=?, claimed_at=?
   WHERE id=(SELECT id FROM jobs WHERE state='queued' ORDER BY id LIMIT 1)
   RETURNING id, name, chunks_total, chunks_done`).get(w, process.pid, Date.now());
export const checkpoint = (id, n) => db.prepare('UPDATE jobs SET chunks_done=? WHERE id=?').run(n, id);
export const finish = (id) => db.prepare(
  `UPDATE jobs SET state='done', outcome='finished', worker_pid=NULL, ended_at=? WHERE id=?`).run(Date.now(), id);
export const release = (id) => db.prepare(
  `UPDATE jobs SET state='queued', outcome='returned', worker=NULL, worker_pid=NULL,
   claimed_at=NULL, ended_at=? WHERE id=?`).run(Date.now(), id);

if (process.argv[1].endsWith('queue.mjs')) {
  db.exec(`DROP TABLE IF EXISTS jobs;
    CREATE TABLE jobs (id INTEGER PRIMARY KEY, name TEXT, chunks_total INTEGER,
      chunks_done INTEGER DEFAULT 0, state TEXT, attempts INTEGER DEFAULT 0, outcome TEXT,
      worker TEXT, worker_pid INTEGER, claimed_at INTEGER, ended_at INTEGER)`);
  const ins = db.prepare("INSERT INTO jobs (id,name,chunks_total,state) VALUES (?,?,5,'queued')");
  for (const [id, n] of [[1, 'export-invoices'], [2, 'rebuild-index'], [3, 'send-digest'],
    [4, 'resize-photos'], [5, 'sync-crm'], [6, 'archive-logs'], [7, 'mail-receipts'],
    [8, 'warm-cache'], [9, 'purge-uploads'], [10, 'tally-metrics'], [11, 'prune-sessions'],
    [12, 'ship-reports']]) ins.run(id, n);
  for (const r of db.prepare('SELECT id,name,chunks_total,state FROM jobs').all())
    console.log(`${r.id} ${r.name} ${r.chunks_total} chunks ${r.state}`);
}

• The worker. A job is five one second chunks, so a stop lands inside one. Save as worker.mjs.

// One worker. Claims the oldest queued job, runs it in 1 s chunks, answers a stop.
// ON_STOP=drain finishes the job in hand; ON_STOP=release puts it back at the next chunk.
import { createServer } from 'node:http';
import { claim, checkpoint, finish, release } from './queue.mjs';
const me = process.argv[2];
const admin = Number((process.argv.find((a) => a.startsWith('--admin=')) ?? '=0').split('=')[1]);
const SAVE = process.env.CHECKPOINT !== '0';      // CHECKPOINT=0 turns progress saving off
let onStop = process.env.ON_STOP ?? 'drain';
const t = () => new Date().toISOString().slice(11, 19);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
let stopping = null, current = null, chunk = 0;

function stop(via, mode) {
  if (stopping) return;
  if (mode) onStop = mode;
  stopping = via;
  console.log(`${t()} ${me} stop received via ${via}, mode=${onStop}`);
}
process.on('message', (m) => m?.stop && stop('ipc', m.mode));
process.on('SIGTERM', () => stop('SIGTERM'));      // registered, and never reached on Windows

let server;
if (admin) {
  server = createServer((req, res) => {
    if (req.method !== 'POST' || !req.url.startsWith('/stop')) return res.writeHead(404).end();
    stop('admin');
    res.writeHead(200, { 'content-type': 'application/json' });
    res.end(JSON.stringify({ worker: me, pid: process.pid, job: current, at_chunk: chunk, mode: onStop }));
  });
  server.listen(admin, '127.0.0.1', () =>
    console.log(`${t()} ${me} pid=${process.pid} stop endpoint on 127.0.0.1:${admin}`));
}

while (!stopping) {
  const job = claim(me);
  if (!job) { await sleep(300); continue; }
  current = job.id; chunk = job.chunks_done;
  console.log(`${t()} ${me} pid=${process.pid} claimed job ${job.id} ${job.name} at chunk ${chunk}/${job.chunks_total}`);
  while (chunk < job.chunks_total) {
    await sleep(1000);
    chunk += 1;
    if (SAVE) checkpoint(job.id, chunk);
    if (stopping && onStop === 'release') break;
  }
  if (chunk < job.chunks_total) {
    release(job.id);
    console.log(`${t()} ${me} returned job ${job.id} to the queue at chunk ${chunk}/${job.chunks_total}, chunks_done saved=${SAVE ? chunk : 0}`);
  } else {
    finish(job.id);
    console.log(`${t()} ${me} finished job ${job.id}, all ${job.chunks_total} chunks`);
  }
  current = null;
}
console.log(`${t()} ${me} no new job claimed, exiting 0`);
server?.close();
process.exit(0);

• The deploy. Stops the worker, waits out a grace period, starts the replacement. Save as deploy.mjs.

// One deploy: stop the running worker, wait for it, start its replacement.
// node deploy.mjs <ipc|signal> <drain|release> <graceSeconds> [--overlap]
import { fork } from 'node:child_process';
const [mech, mode, graceS] = process.argv.slice(2);
const GRACE = Number(graceS) * 1000;
const overlap = process.argv.includes('--overlap');
const t = () => new Date().toISOString().slice(11, 19);
const spawn = (n) => fork('worker.mjs', [n], { env: { ...process.env, ON_STOP: mode } });
const old = spawn('w1');
let next = null, stoppedAt = 0;

function startNext(why) {
  if (next) return;
  console.log(`${t()} deploy starting replacement w2 (${why}, ${((Date.now() - stoppedAt) / 1000).toFixed(1)}s after the stop)`);
  next = spawn('w2');
  next.on('exit', (c, s) => { console.log(`${t()} deploy w2 pid=${next.pid} exited code=${c} signal=${s}`); process.exit(0); });
  setTimeout(() => next.send({ stop: true, mode: 'drain' }), 3000);
}
setTimeout(() => {
  stoppedAt = Date.now();
  console.log(`${t()} deploy stopping w1 pid=${old.pid} via ${mech}, grace ${graceS}s`);
  if (mech === 'ipc') old.send({ stop: true }); else old.kill('SIGTERM');
  if (overlap) startNext('overlap');
  const force = setTimeout(() => {
    console.log(`${t()} deploy grace ${graceS}s expired, killing w1 pid=${old.pid}`);
    old.kill('SIGKILL');
  }, GRACE);
  old.on('exit', () => clearTimeout(force));
}, 3000);
old.on('exit', (c, s) => {
  console.log(`${t()} deploy w1 pid=${old.pid} exited code=${c} signal=${s} after ${((Date.now() - stoppedAt) / 1000).toFixed(1)}s`);
  startNext('w1 gone');
});
setTimeout(() => { old.kill('SIGKILL'); next?.kill('SIGKILL'); process.exit(1); }, 45000);

• The check. Save as jobs.mjs.

// The check. `node jobs.mjs` prints every job, its outcome and whether its holder exists.
import { db } from './queue.mjs';
const alive = (pid) => { if (!pid) return '-'; try { process.kill(pid, 0); return 'yes'; } catch { return 'NO'; } };
const p = (v, n) => String(v).padEnd(n);
console.log(p('id', 3) + p('name', 17) + p('state', 9) + p('outcome', 10) + p('try', 5) + p('done', 6) + p('worker', 8) + p('pid', 8) + 'alive');
for (const r of db.prepare('SELECT * FROM jobs ORDER BY id').all())
  console.log(p(r.id, 3) + p(r.name, 17) + p(r.state, 9) + p(r.outcome ?? '-', 10) + p(r.attempts, 5) +
    p(`${r.chunks_done}/${r.chunks_total}`, 6) + p(r.worker ?? '-', 8) + p(r.worker_pid ?? '-', 8) +
    alive(r.state === 'running' ? r.worker_pid : null));

Steps

1. 1

   Step 1.

   Create the table and seed twelve jobs.

   node queue.mjs
   

   1 export-invoices 5 chunks queued
   2 rebuild-index 5 chunks queued
   3 send-digest 5 chunks queued
   …
   12 ship-reports 5 chunks queued
   (node:8796) ExperimentalWarning: SQLite is an experimental feature and might change at any time

   Every stop below is sent three seconds in, so it lands inside a job.

2. 2

   Step 2.

   Deploy with a stop the worker receives and a generous grace period.

   node deploy.mjs ipc drain 30
   

   18:29:48 w1 pid=40488 claimed job 1 export-invoices at chunk 0/5
   18:29:51 deploy stopping w1 pid=40488 via ipc, grace 30s
   18:29:51 w1 stop received via ipc, mode=drain
   18:29:54 w1 finished job 1, all 5 chunks
   18:29:54 w1 no new job claimed, exiting 0
   18:29:54 deploy w1 pid=40488 exited code=0 signal=null after 2.1s
   18:29:54 deploy starting replacement w2 (w1 gone, 2.1s after the stop)
   18:29:54 w2 pid=36244 claimed job 2 rebuild-index at chunk 0/5

   The drain promise kept. Two more seconds, job 1 finished, nothing new claimed, exited code=0. The replacement then took the next job, not the same one.

3. 3

   Step 3.

   Run the same deploy with the other promise: put the job back.

   node deploy.mjs ipc release 30
   

   18:29:59 w1 pid=25616 claimed job 3 send-digest at chunk 0/5
   18:30:02 deploy stopping w1 pid=25616 via ipc, grace 30s
   18:30:02 w1 stop received via ipc, mode=release
   18:30:02 w1 returned job 3 to the queue at chunk 3/5, chunks_done saved=3
   18:30:02 deploy w1 pid=25616 exited code=0 signal=null after 0.1s
   18:30:02 deploy starting replacement w2 (w1 gone, 0.1s after the stop)
   18:30:02 w2 pid=26508 claimed job 3 send-digest at chunk 3/5
   18:30:04 w2 finished job 3, all 5 chunks

   The worker was gone in 0.1 seconds instead of 2.1, and the replacement resumed job 3 at chunk 3. Both numbers belong in the test.

4. 4

   Step 4.

   Send the stop as a signal instead of as a message.

   node deploy.mjs signal drain 30
   

   18:30:09 w1 pid=13236 claimed job 5 sync-crm at chunk 0/5
   18:30:12 deploy stopping w1 pid=13236 via signal, grace 30s
   18:30:12 deploy w1 pid=13236 exited code=null signal=SIGTERM after 0.0s
   18:30:12 deploy starting replacement w2 (w1 gone, 0.0s after the stop)
   18:30:12 w2 pid=2128 claimed job 6 archive-logs at chunk 0/5

   No stop received line appears. On Windows the SIGTERM handler in worker.mjs never runs, the process is gone in 0.0 seconds, and job 5 stays claimed. The grace bought nothing.

5. 5

   Step 5.

   Read the queue after the three deploys.

   node jobs.mjs
   

   id name             state    outcome   try  done  worker  pid     alive
   1  export-invoices  done     finished  1    5/5   w1      -       -
   3  send-digest      done     finished  2    5/5   w2      -       -
   4  resize-photos    done     finished  1    5/5   w2      -       -
   5  sync-crm         running  -         1    2/5   w1      13236   NO
   6  archive-logs     done     finished  1    5/5   w2      -       -
   7  mail-receipts    queued   -         0    0/5   -       -       -

   Three outcomes in one table. Job 1 finished on its first attempt. Job 3 finished on its second, the trace a release leaves. Job 5 still says running under pid 13236, which no longer exists.

6. 6

   Step 6.

   Cut the grace below the work left, and start the replacement at the stop.

   node deploy.mjs ipc drain 1 --overlap
   

   18:30:17 w1 pid=39692 claimed job 7 mail-receipts at chunk 0/5
   18:30:20 deploy stopping w1 pid=39692 via ipc, grace 1s
   18:30:20 deploy starting replacement w2 (overlap, 0.0s after the stop)
   18:30:20 w1 stop received via ipc, mode=drain
   18:30:21 w2 pid=35924 claimed job 8 warm-cache at chunk 0/5
   18:30:21 deploy grace 1s expired, killing w1 pid=39692
   18:30:21 deploy w1 pid=39692 exited code=null signal=SIGKILL after 1.0s

   The handler ran and job 7 was lost anyway: two seconds of work remained, one second of grace was allowed. The replacement was forked at 18:30:20 and w1 was alive until 18:30:21.

7. 7

   Step 7.

   Start a worker that is nobody's child, with its own stop endpoint.

   ON_STOP=release CHECKPOINT=0 node worker.mjs w9 --admin=8935
   

   18:30:43 w9 pid=15928 claimed job 9 purge-uploads at chunk 0/5
   18:30:43 w9 pid=15928 stop endpoint on 127.0.0.1:8935

   A worker started by a service manager has no IPC channel, so the message form used in steps 2 and 3 is unavailable. CHECKPOINT=0 turns off progress saving.

8. 8

   Step 8.

   From the first shell, three seconds into job 9, send the stop.

   curl -sS -X POST http://127.0.0.1:8935/stop
   

   {"worker":"w9","pid":15928,"job":9,"at_chunk":3,"mode":"release"}

   The reply names the job in hand and the chunk it had reached. The worker's own shell then printed returned job 9 to the queue at chunk 4/5, chunks_done saved=0 and exited 0.

9. 9

   Step 9.

   Read the row that job 9 went back to.

   node jobs.mjs
   

   id name             state    outcome   try  done  worker  pid     alive
   …
   7  mail-receipts    running  -         1    3/5   w1      39692   NO
   8  warm-cache       done     finished  1    5/5   w2      -       -
   9  purge-uploads    queued   returned  1    0/5   -       -       -
   10 tally-metrics    queued   -         0    0/5   -       -       -

   The shutdown was clean and the work is gone. Four chunks ran and the row reads 0/5, so the next claim starts from the beginning.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | stop received, then finished job N, then exited code=0 | The worker drained: it finished the job in hand and claimed nothing new | Record how long the drain took. That number is the grace period the deploy has to allow | | returned job N to the queue at chunk 3/5, row back to queued | The worker released the claim and the queue will redeliver | Check the attempt counter, and check that the job is safe to run a second time | | Exit code=null signal=SIGTERM with no stop received line above it | The handler never ran. On Windows this is the normal result | Deliver the stop by a channel the process can receive: an IPC message, or a request to its own port | | Row running, empty outcome, alive reads NO | Nobody holds the job and nothing will retry it | Find it with How to test a stuck background job | | grace 1s expired, killing w1 | The grace period was shorter than the work left in the job | Set the grace from the slowest job, not from the average one | | A worker claims at chunk 0 after chunks_done saved=0 | Progress is not checkpointed, so redelivery repeats the whole job | Save progress at a chunk boundary, or accept the repeat and make the job safe to repeat |

Common mistakes

Thresholds

What to check next

• How to test a stuck background job: the row step 4 left behind.
• How to check if a queue worker is running: the pid column, asked before the deploy.
• How to test job timeout and cancellation: for a job that outlives any grace period.
• How to test duplicate job execution: what a release costs when the job is not safe to repeat.
• How to check a background job is idempotent: the property that makes redelivery harmless.

FAQ

What is a graceful shutdown?

A stop the process receives and acts on. The worker stops claiming new work, finishes or returns the job it holds, then exits. Step 2 ends in exited code=0 with the row marked done.

How do I test graceful shutdown?

Stop a worker while a job is in flight, then read the job row rather than the exit code. Steps 2, 3 and 4 produce a finished job, a returned job and an abandoned one.

Should a worker finish the job or put it back?

Either, as long as the queue and the job agree. Finishing needs a grace period longer than the job. Returning needs the job to be safe to repeat from its last saved point, chunk zero in step 9.

Why did my SIGTERM handler not run?

On Windows it will not. Step 4 signalled a forked Node worker and the process was gone in 0.0 seconds with no handler output. Use an IPC message, or a stop endpoint as in step 8.

Verified