Webhook ordering

Send two events in order from a sender that retries, and log the arrival order at the receiver. Here seq=1 was refused once, seq=2 arrived 0.3 s later and committed first, and seq=1 committed 0.7 s after that. One retry is enough to reverse a pair.

Why check this

Run this before release on any handler whose behaviour depends on which event came first: a subscription that is created then cancelled, an order that is authorised then captured, a record created then updated. The failure it prevents is a handler that applies the older state last. A cancelled subscription goes back to active because customer.subscription.updated was retried and landed after the cancellation.

Nothing in HTTP orders two separate requests, and no retry policy can restore an order it never had. The test is not whether ordering holds. It is whether the handler survives without it.

Prerequisites

Node 22 or later, for fetch in the sender and node:http in the receiver.
A free TCP port. Everything here uses 127.0.0.1:8931. Confirm with netstat -ano | grep 8931 first.
No provider account, and none is used. The delivery behaviour below is the one implemented in sender.mjs. Providers publish their own and none of them promises order.
Save the receiver as receiver.mjs. /flaky/1 refuses the first request it ever sees, whichever event that turns out to be, and /drop commits the work then destroys the socket.

// receiver.mjs - a webhook endpoint you can make fail, hang or lose its reply.
import { createServer } from 'node:http';

const port = Number(process.argv[2] ?? 8931);
const t0 = Date.now();
const perEvent = new Map();
let order = 0;

createServer((req, res) => {
  let raw = '';
  req.on('data', (c) => (raw += c));
  req.on('end', async () => {
    const ev = JSON.parse(raw || '{}');
    const n = (perEvent.get(ev.id) ?? 0) + 1;
    perEvent.set(ev.id, n);
    const arrival = ++order;
    const [, mode, arg] = req.url.split('/');
    const say = (status, note = '') =>
      console.log(
        `#${arrival} t=${((Date.now() - t0) / 1000).toFixed(3)}s id=${ev.id} seq=${ev.seq} attempt=${n} -> ${status} ${note}`
      );
    if (mode === 'down' || (mode === 'fail' && n <= Number(arg)) || (mode === 'flaky' && arrival <= Number(arg))) {
      say(500);
      return res.writeHead(500).end('not today');
    }
    if (mode === 'slow') await new Promise((r) => setTimeout(r, Number(arg)));
    if (mode === 'drop') {
      say('200-then-lost', '(handler committed, reply discarded)');
      return res.socket.destroy();
    }
    say(200, '(handler committed)');
    res.writeHead(200).end('ok');
  });
}).listen(port, () => console.log(`receiver listening on ${port}`));

Save the sender as sender.mjs. One process delivers one event, which is how a real fleet behaves: the events do not queue behind each other.

// sender.mjs - delivers one event with exponential backoff, then dead-letters it.
import { appendFileSync } from 'node:fs';

const [url, id, seq] = process.argv.slice(2);
const MAX = Number(process.env.ATTEMPTS ?? 4);
const BASE = Number(process.env.BASE_MS ?? 1000);
const TIMEOUT = Number(process.env.TIMEOUT_MS ?? 5000);

const event = { id, seq: Number(seq ?? 1), type: 'payment.succeeded', createdAt: new Date().toISOString() };
const t0 = Date.now();
const history = [];
let prev = null;

for (let attempt = 1; attempt <= MAX; attempt += 1) {
  const started = Date.now();
  let outcome;
  try {
    const r = await fetch(url, {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify(event),
      signal: AbortSignal.timeout(TIMEOUT),
    });
    outcome = `HTTP ${r.status}`;
    await r.text();
    if (r.ok) { report(attempt, started, outcome); history.push(rec(attempt, started, outcome)); done('delivered'); }
  } catch (e) {
    outcome = e.name === 'TimeoutError' ? `timeout at ${TIMEOUT} ms` : (e.cause?.code ?? e.name);
  }
  report(attempt, started, outcome);
  history.push(rec(attempt, started, outcome));
  if (attempt === MAX) break;
  const ideal = BASE * 2 ** (attempt - 1);
  const delay = Math.round(ideal / 2 + Math.random() * (ideal / 2));
  console.log(`         sleeping ${delay} ms before attempt ${attempt + 1}`);
  await new Promise((r) => setTimeout(r, delay));
}
appendFileSync('dlq.jsonl', JSON.stringify({ event, reason: `gave up after ${MAX} attempts`, lastOutcome: history.at(-1).outcome, failedAt: new Date().toISOString(), attempts: history }) + '\n');
done(`dead-lettered after ${MAX} attempts`);

function rec(attempt, started, outcome) {
  return { attempt, startedAt: new Date(started).toISOString(), tookMs: Date.now() - started, outcome };
}
function report(attempt, started, outcome) {
  const gap = prev === null ? '   -  ' : `${((started - prev) / 1000).toFixed(3)}s`;
  prev = started;
  console.log(`attempt ${attempt}  t=${((started - t0) / 1000).toFixed(3)}s  gap=${gap}  took=${String(Date.now() - started).padStart(5)} ms  ${outcome}`);
}
function done(msg) { console.log(`result: ${msg}`); process.exit(0); }

Steps

1
Step 1.
Start the receiver with its console going to a file.
```
node receiver.mjs 8931 > receiver.log 2>&1 &
```
```
receiver listening on 8931
```
The arrival counter it prints as #n is the order the handler saw, which is the only order that matters.
2
Step 2.
Send seq=1 first and seq=2 a third of a second later, against a path that refuses the first request it sees.
```
ATTEMPTS=4 BASE_MS=1000 node sender.mjs http://127.0.0.1:8931/flaky/1 evt_A 1 > sender-a.log 2>&1 & sleep 0.3; ATTEMPTS=4 BASE_MS=1000 node sender.mjs http://127.0.0.1:8931/flaky/1 evt_B 2
```
```
attempt 1  t=0.000s  gap=   -    took=   50 ms  HTTP 200
result: delivered
```
That is evt_B, the second event, delivered on its first attempt. The shell also prints a completion notice for the background job. evt_A wrote to the log file instead.

Read what happened to the first event.

cat sender-a.log

attempt 1  t=0.000s  gap=   -    took=   74 ms  HTTP 500
       sleeping 980 ms before attempt 2
attempt 2  t=1.066s  gap=1.066s  took=    2 ms  HTTP 200
result: delivered

Both events are delivered and both senders report success. Neither one has any idea it overtook the other.

Read the order the handler actually ran in.

cat receiver.log

receiver listening on 8931
#1 t=2.056s id=evt_A seq=1 attempt=1 -> 500 
#2 t=2.356s id=evt_B seq=2 attempt=1 -> 200 (handler committed)
#3 t=3.061s id=evt_A seq=1 attempt=2 -> 200 (handler committed)

Commit order is seq=2 then seq=1. The events were emitted in order, sent in order, and applied in reverse, because one of them was refused once.

Send a third event to a path that commits the work and then loses the reply.

ATTEMPTS=3 BASE_MS=1000 node sender.mjs http://127.0.0.1:8931/drop evt_C 3

attempt 1  t=0.000s  gap=   -    took=   46 ms  UND_ERR_SOCKET
       sleeping 732 ms before attempt 2
attempt 2  t=0.785s  gap=0.785s  took=    4 ms  UND_ERR_SOCKET
       sleeping 1878 ms before attempt 3
attempt 3  t=2.671s  gap=1.886s  took=    5 ms  UND_ERR_SOCKET
result: dead-lettered after 3 attempts

Three attempts, no status code on any of them, and the sender files the event as undelivered. UND_ERR_SOCKET is what Node's fetch reports when the peer closes the connection, as opposed to TimeoutError when your own deadline fires.

Count how many times the handler ran for that same undelivered event.

tail -n 3 receiver.log

#4 t=15.192s id=evt_C seq=3 attempt=1 -> 200-then-lost (handler committed, reply discarded)
#5 t=15.940s id=evt_C seq=3 attempt=2 -> 200-then-lost (handler committed, reply discarded)
#6 t=17.827s id=evt_C seq=3 attempt=3 -> 200-then-lost (handler committed, reply discarded)

Three commits for an event the sender dead-lettered. The reply was lost, not the event, and that is why ordering and deduplication are one conversation rather than two.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | #2 is seq=2 and #3 is seq=1 | Arrival order is not emission order | Order by a field in the payload, never by arrival. | | The retried event lands last | Its backoff put it behind everything sent meanwhile | Compare a version or timestamp before writing, and drop the older one. | | Both senders report delivered | Neither side can detect the swap | Detect it at the handler, from the payload. | | Three commits, zero deliveries recorded | The reply was lost after the work was done | Deduplicate on the event id. See Webhook idempotency. | | UND_ERR_SOCKET rather than a status | The peer closed the connection | Distinguish it from your own timeout. See Webhook timeout. |

Common mistakes

Sign: The handler applies whatever arrives, assuming the sender ordered it.Cause: One refusal reversed a pair in step 4, with 0.3 s between the two events. Any retry policy produces this, and a busier sender produces it more often. Carry a sequence number or an updated-at value in the payload and refuse to apply an older one.

Sign: Ordering is tested by sending both events from the same script, one after the other.Cause: A loop that awaits each delivery cannot reverse anything, because the second event is not sent until the first succeeds. The two senders in step 2 run as separate processes, which is what a real fleet does. Test with concurrency or the test proves nothing.

Sign: A duplicate commit is investigated as a sender defect.Cause: In step 6 the handler committed three times for an event the sender recorded as never delivered. The receiver had already destroyed the socket after doing the work. A 200 that never arrives is indistinguishable from a 500, so the sender has to retry and the receiver has to expect it.

Sign: Ordering is fixed by reducing the receiver to one worker.Cause: Serial processing controls the order the handler runs in, not the order events arrive in. The reversal in step 4 happened before any worker was involved. It also caps throughput at one event at a time, and the retried event still arrives after the later one.

What to check next

Webhook idempotency: the key that makes the three commits in step 6 harmless.
How to test webhook retries: the backoff that put the first event behind the second.
Webhook timeout: the other way a delivery is recorded differently on each side.
How to test webhook delivery: the basic delivery path, before failure is added to it.

FAQ

Does Stripe deliver webhook events in order?

No provider guarantees it, and this page cannot measure any provider's behaviour without sending traffic through their account. Read the provider's documentation for what it promises, then assume order is absent and test the handler that way.

How do I handle events that arrive out of order?

Put a version in the payload: a sequence number, a revision, or the timestamp of the state change. On each event, compare it with what is stored and discard anything older. That turns a reversed pair into a no-op instead of a rollback.

Why did the retried event arrive last?

Its backoff delay ran while the later event was being delivered on a different connection. In step 4 the first event waited 980 ms for its second attempt, and the second event finished inside that window.

Can a single worker fix ordering?

It fixes concurrent processing, not arrival. A serial worker still receives seq=2 before seq=1 here, and it still applies them in that order. The fix is a version check in the payload.

Verified

Verified 2026-09-12by Maks Vernynode 22.23.2

Each output block is what the command above it printed on that date, on the host named in the step. Figures read from a live site move between runs. Compare the shape of the answer rather than the digits, and see the methodology for how a page is re-verified.

Payment gateway testing checklist15 checks
All payments and webhooks checkscluster

intermediate12 minpublished 2026-09-12updated 2026-09-12Maks Verny