How to test webhook retries

Start a receiver that answers 500 three times, point a sender with exponential backoff at it, and read the wall-clock gap between attempts. Here the gaps were 0.929 s, 2.003 s and 3.238 s against a nominal 1 s, 2 s and 4 s schedule, and the fourth attempt returned HTTP 200.

Why check this

Retry code runs on the day the receiver is down, which is never the day it was written. Test it when the delivery code changes, when a queue or worker is replaced, and on staging sign-off for anything that emits events. The failure it prevents is a sender that retries at a fixed interval with no jitter: the receiver comes back, every pending event arrives in the same millisecond, and the endpoint that was already struggling falls over a second time.

The second failure is quieter. A sender with no ceiling on attempts keeps an event alive for hours, so an order that was refunded at 09:00 gets its payment.succeeded delivered at 11:00.

Prerequisites

• Node 22 or later. The sender uses fetch and AbortSignal.timeout, both built in since Node 18.
• A free TCP port. Everything here uses 127.0.0.1:8931. Confirm it is free with netstat -ano | grep 8931 before you start.
• No provider account, and none is used. The schedule measured below is the one implemented in sender.mjs. Every provider publishes its own and changes it without notice.
• Save the receiver as receiver.mjs. It fails, hangs or loses its reply depending on the path you post to.

// receiver.mjs - a webhook endpoint you can make fail, hang or lose its reply.
import { createServer } from 'node:http';

const port = Number(process.argv[2] ?? 8931);
const t0 = Date.now();
const perEvent = new Map();
let order = 0;

createServer((req, res) => {
  let raw = '';
  req.on('data', (c) => (raw += c));
  req.on('end', async () => {
    const ev = JSON.parse(raw || '{}');
    const n = (perEvent.get(ev.id) ?? 0) + 1;
    perEvent.set(ev.id, n);
    const arrival = ++order;
    const [, mode, arg] = req.url.split('/');
    const say = (status, note = '') =>
      console.log(
        `#${arrival} t=${((Date.now() - t0) / 1000).toFixed(3)}s id=${ev.id} seq=${ev.seq} attempt=${n} -> ${status} ${note}`
      );
    if (mode === 'down' || (mode === 'fail' && n <= Number(arg)) || (mode === 'flaky' && arrival <= Number(arg))) {
      say(500);
      return res.writeHead(500).end('not today');
    }
    if (mode === 'slow') await new Promise((r) => setTimeout(r, Number(arg)));
    if (mode === 'drop') {
      say('200-then-lost', '(handler committed, reply discarded)');
      return res.socket.destroy();
    }
    say(200, '(handler committed)');
    res.writeHead(200).end('ok');
  });
}).listen(port, () => console.log(`receiver listening on ${port}`));

• Save the sender as sender.mjs. It doubles the delay on every attempt and draws the actual sleep from the upper half of that window, which is the equal-jitter shape.

// sender.mjs - delivers one event with exponential backoff, then dead-letters it.
import { appendFileSync } from 'node:fs';

const [url, id, seq] = process.argv.slice(2);
const MAX = Number(process.env.ATTEMPTS ?? 4);
const BASE = Number(process.env.BASE_MS ?? 1000);
const TIMEOUT = Number(process.env.TIMEOUT_MS ?? 5000);

const event = { id, seq: Number(seq ?? 1), type: 'payment.succeeded', createdAt: new Date().toISOString() };
const t0 = Date.now();
const history = [];
let prev = null;

for (let attempt = 1; attempt <= MAX; attempt += 1) {
  const started = Date.now();
  let outcome;
  try {
    const r = await fetch(url, {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify(event),
      signal: AbortSignal.timeout(TIMEOUT),
    });
    outcome = `HTTP ${r.status}`;
    await r.text();
    if (r.ok) { report(attempt, started, outcome); history.push(rec(attempt, started, outcome)); done('delivered'); }
  } catch (e) {
    outcome = e.name === 'TimeoutError' ? `timeout at ${TIMEOUT} ms` : (e.cause?.code ?? e.name);
  }
  report(attempt, started, outcome);
  history.push(rec(attempt, started, outcome));
  if (attempt === MAX) break;
  const ideal = BASE * 2 ** (attempt - 1);
  const delay = Math.round(ideal / 2 + Math.random() * (ideal / 2));
  console.log(`         sleeping ${delay} ms before attempt ${attempt + 1}`);
  await new Promise((r) => setTimeout(r, delay));
}
appendFileSync('dlq.jsonl', JSON.stringify({ event, reason: `gave up after ${MAX} attempts`, lastOutcome: history.at(-1).outcome, failedAt: new Date().toISOString(), attempts: history }) + '\n');
done(`dead-lettered after ${MAX} attempts`);

function rec(attempt, started, outcome) {
  return { attempt, startedAt: new Date(started).toISOString(), tookMs: Date.now() - started, outcome };
}
function report(attempt, started, outcome) {
  const gap = prev === null ? '   -  ' : `${((started - prev) / 1000).toFixed(3)}s`;
  prev = started;
  console.log(`attempt ${attempt}  t=${((started - t0) / 1000).toFixed(3)}s  gap=${gap}  took=${String(Date.now() - started).padStart(5)} ms  ${outcome}`);
}
function done(msg) { console.log(`result: ${msg}`); process.exit(0); }

Steps

1. 1

   Step 1.

   Start the receiver in the background with its console going to a file. The file gets one line.

   node receiver.mjs 8931 > receiver.log 2>&1 &
   

   receiver listening on 8931

   The receiver's own clock starts here, so every t= it prints later is measured from this moment.

2. 2

   Step 2.

   Deliver one event to a path that refuses the first three attempts, and watch the gaps.

   ATTEMPTS=5 BASE_MS=1000 node sender.mjs http://127.0.0.1:8931/fail/3 evt_1001 1
   

   attempt 1  t=0.000s  gap=   -    took=   72 ms  HTTP 500
          sleeping 850 ms before attempt 2
   attempt 2  t=0.929s  gap=0.929s  took=    8 ms  HTTP 500
          sleeping 1991 ms before attempt 3
   attempt 3  t=2.932s  gap=2.003s  took=   18 ms  HTTP 500
          sleeping 3212 ms before attempt 4
   attempt 4  t=6.170s  gap=3.238s  took=    5 ms  HTTP 200
   result: delivered

   Four attempts, three refusals, one delivery, 6.17 s from first send to success. The gap column is the number to assert on, because it is what the receiver experiences.

3. 3

   Step 3.

   Run the same delivery under a second event id and compare the two schedules.

   ATTEMPTS=5 BASE_MS=1000 node sender.mjs http://127.0.0.1:8931/fail/3 evt_1002 2
   

   attempt 1  t=0.000s  gap=   -    took=   75 ms  HTTP 500
          sleeping 849 ms before attempt 2
   attempt 2  t=0.971s  gap=0.971s  took=    7 ms  HTTP 500
          sleeping 1859 ms before attempt 3
   attempt 3  t=2.842s  gap=1.871s  took=   11 ms  HTTP 500
          sleeping 2770 ms before attempt 4
   attempt 4  t=5.627s  gap=2.785s  took=   19 ms  HTTP 200
   result: delivered

   The nominal schedule is identical and the drawn sleeps are not: 850 and 849 ms, then 1991 and 1859 ms, then 3212 and 2770 ms. That spread is the jitter, and it is what stops a fleet of senders from arriving together.

4. 4

   Step 4.

   Read the same two deliveries from the receiver's side.

   cat receiver.log
   

   receiver listening on 8931
   #1 t=2.050s id=evt_1001 seq=1 attempt=1 -> 500 
   #2 t=2.924s id=evt_1001 seq=1 attempt=2 -> 500 
   #3 t=4.938s id=evt_1001 seq=1 attempt=3 -> 500 
   #4 t=8.164s id=evt_1001 seq=1 attempt=4 -> 200 (handler committed)
   #5 t=8.324s id=evt_1002 seq=2 attempt=1 -> 500 
   #6 t=9.238s id=evt_1002 seq=2 attempt=2 -> 500 
   #7 t=11.113s id=evt_1002 seq=2 attempt=3 -> 500 
   #8 t=13.906s id=evt_1002 seq=2 attempt=4 -> 200 (handler committed)

   Eight requests for two events. The handler ran four times per event and committed once, which is the reason a receiver needs a key it can deduplicate on.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | Gaps roughly double: 0.9 s, 2.0 s, 3.2 s | The delay is exponential | Confirm there is a ceiling, or attempt 12 is four hours out. | | Two runs draw identical sleeps | No jitter is applied | Add it. Without jitter every stalled sender resumes at once. | | A gap above its nominal ceiling, 2.003 s against 2.000 s | The failed request itself sits inside the gap | Assert on a band, never on equality. | | attempt=4 in the receiver log for one delivered event | The handler ran four times | Deduplicate on the event id. See Webhook idempotency. | | The sender logs a delivery the receiver never recorded | The reply was lost, not the event | Read Webhook timeout. |

Common mistakes

What to check next

• Webhook timeout: what happens when the receiver is slow rather than failing, and why both sides then disagree.
• Webhook dead letter queue: where an event goes when the attempts run out.
• Webhook idempotency: the receiver side of the four attempts counted above.
• How to test retry-after header: the header that overrides a client's own backoff.

FAQ

What is a reasonable webhook retry policy?

Exponential delay with jitter, a ceiling on the single delay, and a hard limit on attempts or elapsed time. The run above uses 1 s doubling, equal jitter and five attempts, which spans about 14 s. Pick the limit from how stale the event may be, not from a round number.

What is Stripe's webhook retry policy?

Stripe documents its own schedule and changes it. This page cannot measure it, because testing it would mean sending traffic through somebody else's account. Read the provider's current documentation for the schedule, and use a local sender to test the code that consumes it.

How do I test retry logic without a provider account?

Run both ends yourself, as above. A receiver that answers 500 on demand and a sender you control give you the attempt count, the gaps and the give-up point, none of which a sandbox account exposes as numbers.

Should the limit be a number of attempts or a time window?

A window is easier to reason about, because it answers how stale a delivered event can be. Convert it to attempts for the assertion. Five attempts on the schedule above run for about 14 s in the worst case and about 7 s in the best.

Verified