Webhook dead letter queue

Point the sender at an endpoint that always answers 500, let the attempts run out, and read dlq.jsonl. One event landed there, carrying the four refusals that put it there and the time it was abandoned. The event that recovered on its third attempt is absent from the file.

Why check this

A dead letter path is tested on the day it is built and then never again, because nothing exercises it in a healthy system. Run this after any change to the delivery worker, and on staging sign-off for any service that emits events. The failure it prevents is a sender that drops an abandoned event on the floor: the payment succeeded, the receiver was down for four minutes, and no record of the missed event exists anywhere.

The second failure is a record too thin to act on. A row that holds the payload and nothing else cannot tell you whether the endpoint refused the event, timed out, or was never reachable.

Prerequisites

• Node 22 or later, for fetch in the sender and node:http in the receiver.
• A free TCP port. Everything here uses 127.0.0.1:8931. Confirm with netstat -ano | grep 8931 first.
• No provider account, and none is used. The give-up rule below is the one implemented in sender.mjs. Every provider publishes its own, and most expose the abandoned events through their dashboard rather than a file.
• Save the receiver as receiver.mjs. /down answers 500 to everything, /fail/2 refuses twice then accepts.

// receiver.mjs - a webhook endpoint you can make fail, hang or lose its reply.
import { createServer } from 'node:http';

const port = Number(process.argv[2] ?? 8931);
const t0 = Date.now();
const perEvent = new Map();
let order = 0;

createServer((req, res) => {
  let raw = '';
  req.on('data', (c) => (raw += c));
  req.on('end', async () => {
    const ev = JSON.parse(raw || '{}');
    const n = (perEvent.get(ev.id) ?? 0) + 1;
    perEvent.set(ev.id, n);
    const arrival = ++order;
    const [, mode, arg] = req.url.split('/');
    const say = (status, note = '') =>
      console.log(
        `#${arrival} t=${((Date.now() - t0) / 1000).toFixed(3)}s id=${ev.id} seq=${ev.seq} attempt=${n} -> ${status} ${note}`
      );
    if (mode === 'down' || (mode === 'fail' && n <= Number(arg)) || (mode === 'flaky' && arrival <= Number(arg))) {
      say(500);
      return res.writeHead(500).end('not today');
    }
    if (mode === 'slow') await new Promise((r) => setTimeout(r, Number(arg)));
    if (mode === 'drop') {
      say('200-then-lost', '(handler committed, reply discarded)');
      return res.socket.destroy();
    }
    say(200, '(handler committed)');
    res.writeHead(200).end('ok');
  });
}).listen(port, () => console.log(`receiver listening on ${port}`));

• Save the sender as sender.mjs. When the attempts run out it appends one JSON line to dlq.jsonl holding the event, the reason and every attempt.

// sender.mjs - delivers one event with exponential backoff, then dead-letters it.
import { appendFileSync } from 'node:fs';

const [url, id, seq] = process.argv.slice(2);
const MAX = Number(process.env.ATTEMPTS ?? 4);
const BASE = Number(process.env.BASE_MS ?? 1000);
const TIMEOUT = Number(process.env.TIMEOUT_MS ?? 5000);

const event = { id, seq: Number(seq ?? 1), type: 'payment.succeeded', createdAt: new Date().toISOString() };
const t0 = Date.now();
const history = [];
let prev = null;

for (let attempt = 1; attempt <= MAX; attempt += 1) {
  const started = Date.now();
  let outcome;
  try {
    const r = await fetch(url, {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify(event),
      signal: AbortSignal.timeout(TIMEOUT),
    });
    outcome = `HTTP ${r.status}`;
    await r.text();
    if (r.ok) { report(attempt, started, outcome); history.push(rec(attempt, started, outcome)); done('delivered'); }
  } catch (e) {
    outcome = e.name === 'TimeoutError' ? `timeout at ${TIMEOUT} ms` : (e.cause?.code ?? e.name);
  }
  report(attempt, started, outcome);
  history.push(rec(attempt, started, outcome));
  if (attempt === MAX) break;
  const ideal = BASE * 2 ** (attempt - 1);
  const delay = Math.round(ideal / 2 + Math.random() * (ideal / 2));
  console.log(`         sleeping ${delay} ms before attempt ${attempt + 1}`);
  await new Promise((r) => setTimeout(r, delay));
}
appendFileSync('dlq.jsonl', JSON.stringify({ event, reason: `gave up after ${MAX} attempts`, lastOutcome: history.at(-1).outcome, failedAt: new Date().toISOString(), attempts: history }) + '\n');
done(`dead-lettered after ${MAX} attempts`);

function rec(attempt, started, outcome) {
  return { attempt, startedAt: new Date(started).toISOString(), tookMs: Date.now() - started, outcome };
}
function report(attempt, started, outcome) {
  const gap = prev === null ? '   -  ' : `${((started - prev) / 1000).toFixed(3)}s`;
  prev = started;
  console.log(`attempt ${attempt}  t=${((started - t0) / 1000).toFixed(3)}s  gap=${gap}  took=${String(Date.now() - started).padStart(5)} ms  ${outcome}`);
}
function done(msg) { console.log(`result: ${msg}`); process.exit(0); }

• Save the redelivery tool as replay.mjs. It sends every row in the file once.

import { readFileSync } from 'node:fs';
const url = process.argv[2];
for (const line of readFileSync('dlq.jsonl', 'utf8').trim().split('\n')) {
  const row = JSON.parse(line);
  const res = await fetch(url, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(row.event) });
  console.log(`replayed ${row.event.id} (dead-lettered ${row.failedAt}) -> HTTP ${res.status}`);
}

Steps

1. 1

   Step 1.

   Start the receiver with its console going to a file.

   node receiver.mjs 8931 > receiver.log 2>&1 &
   

   receiver listening on 8931

   Delete any dlq.jsonl left over from an earlier run first, or the counts below include it.

2. 2

   Step 2.

   Send an event to an endpoint that refuses everything, and let the four attempts run out.

   ATTEMPTS=4 BASE_MS=1000 node sender.mjs http://127.0.0.1:8931/down evt_4001 11
   

   attempt 1  t=0.000s  gap=   -    took=   53 ms  HTTP 500
          sleeping 512 ms before attempt 2
   attempt 2  t=0.577s  gap=0.577s  took=    6 ms  HTTP 500
          sleeping 1259 ms before attempt 3
   attempt 3  t=1.851s  gap=1.274s  took=    5 ms  HTTP 500
          sleeping 3356 ms before attempt 4
   attempt 4  t=5.217s  gap=3.366s  took=    3 ms  HTTP 500
   result: dead-lettered after 4 attempts

   The whole life of this event was 5.22 s. A production policy stretches the same four attempts over minutes or hours.

3. 3

   Step 3.

   Send a second event to an endpoint that recovers, so the file has something to be wrong about.

   ATTEMPTS=4 BASE_MS=1000 node sender.mjs http://127.0.0.1:8931/fail/2 evt_4002 12
   

   attempt 1  t=0.000s  gap=   -    took=   48 ms  HTTP 500
          sleeping 979 ms before attempt 2
   attempt 2  t=1.048s  gap=1.048s  took=   15 ms  HTTP 500
          sleeping 1798 ms before attempt 3
   attempt 3  t=2.868s  gap=1.820s  took=    4 ms  HTTP 200
   result: delivered

   Two refusals and a delivery. This event must not appear in the dead letter file.

4. 4

   Step 4.

   List what is in the file, one line per abandoned event.

   node -e "for (const l of require('fs').readFileSync('dlq.jsonl','utf8').trim().split('\n')) { const r = JSON.parse(l); console.log(r.event.id, '| seq', r.event.seq, '|', r.reason, '| last:', r.lastOutcome, '|', r.attempts.map(a => a.outcome).join(' -> ')); }"
   

   evt_4001 | seq 11 | gave up after 4 attempts | last: HTTP 500 | HTTP 500 -> HTTP 500 -> HTTP 500 -> HTTP 500

   One row for two events. The recovered event is absent, and the abandoned one carries its whole failure history rather than a single status.

5. 5

   Step 5.

   Read one record in full, which is what an operator needs before deciding to replay.

   node -e "console.log(JSON.stringify(JSON.parse(require('fs').readFileSync('dlq.jsonl','utf8').trim().split('\n')[0]), null, 2))"
   

   {
   "event": {
     "id": "evt_4001",
     "seq": 11,
     "type": "payment.succeeded",
     "createdAt": "2026-09-11T22:47:23.746Z"
   },
   "reason": "gave up after 4 attempts",
   "lastOutcome": "HTTP 500",
   "failedAt": "2026-09-11T22:47:28.967Z",
   "attempts": [
     {
       "attempt": 1,
       "startedAt": "2026-09-11T22:47:23.747Z",
       "tookMs": 53,
       "outcome": "HTTP 500"
     },
     {
       "attempt": 2,
       "startedAt": "2026-09-11T22:47:24.324Z",
       "tookMs": 6,
       "outcome": "HTTP 500"
     },
     {
       "attempt": 3,
       "startedAt": "2026-09-11T22:47:25.598Z",
       "tookMs": 5,
       "outcome": "HTTP 500"
     },
     {
       "attempt": 4,
       "startedAt": "2026-09-11T22:47:28.964Z",
       "tookMs": 3,
       "outcome": "HTTP 500"
     }
   ]
   }

   Four attempts in 5.2 s, all refused, and the timestamps are UTC while the console timeline above is relative. Correlating the two needs the receiver's own clock, not a wall clock read from a different zone.

6. 6

   Step 6.

   Replay the file against a healthy path and read what comes back.

   node replay.mjs http://127.0.0.1:8931/ok
   

   replayed evt_4001 (dead-lettered 2026-09-11T22:47:28.967Z) -> HTTP 200

   The replay says nothing about whether the receiver had seen this event before. Only the receiver knows that.

7. 7

   Step 7.

   Ask the receiver how many times it has now been handed the replayed event.

   tail -n 2 receiver.log
   

   #7 t=10.144s id=evt_4002 seq=12 attempt=3 -> 200 (handler committed)
   #8 t=18.309s id=evt_4001 seq=11 attempt=5 -> 200 (handler committed)

   attempt=5. The endpoint had already parsed this event four times before the replay, refusing it each time. A replay is a fifth delivery, not a first one.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | One row for two events | The give-up rule fires only when attempts run out | Confirm the recovered event is absent, every time. | | attempts lists four identical outcomes | The endpoint was refusing, not flapping | Fix the endpoint before replaying, or the replay adds four more. | | lastOutcome is a timeout rather than a status | The handler may have committed anyway | Read Webhook timeout before replaying. | | attempt=5 at the receiver after a replay | The receiver counts the replay as another delivery | Deduplicate on the event id. See Webhook idempotency. | | The file exists and nothing reads it | The queue is a log file, not a queue | Alert on a row appearing, with the reason in the alert. |

Common mistakes

What to check next

• How to test webhook retries: the schedule that decides when an event reaches this file.
• Webhook timeout: how an event gets dead-lettered after the receiver already committed it.
• Webhook idempotency: the key that makes the replay in step 6 safe to run twice.
• Webhook signature verification failed: why the re-serialised payload above no longer matches its signature.

FAQ

What belongs in a dead letter record?

The event, the reason it was abandoned, the outcome of every attempt with its timestamp, and the destination it was aimed at. The record in step 5 carries all four. A payload on its own leaves an operator guessing whether to replay or to fix the endpoint first.

How do I replay a dead-lettered webhook?

Post the stored event to the endpoint again, as replay.mjs does, after the endpoint is healthy. Replay is a delivery like any other, so the receiver needs a deduplication key. Record the replay against the original row rather than deleting it.

Does a dead-lettered event mean the receiver never processed it?

No. It means the sender never recorded a success. An endpoint that commits and then times out produces exactly this: a dead letter row on one side and a completed handler on the other.

How long should dead letters be kept?

Long enough to cover the outage plus the investigation, and no longer than the payload may be retained under your own data rules. Payment events carry personal data, so a file with no retention rule becomes a compliance problem of its own.

Verified