Webhook idempotency

Deliver the same event id twice and count the rows the handler wrote: curl -X POST http://127.0.0.1:8471/hooks -H 'X-Delivery-Id: del_b2' -d '{"id":"evt_9f3"}', then curl http://127.0.0.1:8471/hooks. One row means the id was honoured. Then send both deliveries at the same instant and count again, because that is where a check-then-write handler writes two.

Why check this

Webhook delivery is at least once. A receiver that answers slowly, answers 500 or drops the connection gets the same event again, and a provider that retries in parallel can have two attempts in flight together. Run this when a receiver is added, when its handler changes, and in regression before any release that touches billing. The failure it prevents is one payment credited to the account twice.

Prerequisites

const http = require('node:http');

let mode = 'event';
const seen = new Set();
const ledger = [];
const readStore = () => new Promise((r) => setTimeout(r, 20));

http.createServer((req, res) => {
  const url = new URL(req.url, 'http://127.0.0.1:8471');
  const json = (code, body, extra = {}) => {
    res.writeHead(code, { 'content-type': 'application/json', ...extra });
    res.end(JSON.stringify(body));
  };
  if (req.method === 'GET') return json(200, { mode, rows: ledger.length, ledger });
  if (req.method === 'DELETE') {
    mode = url.searchParams.get('mode') || 'event';
    seen.clear();
    ledger.length = 0;
    return json(200, { mode, rows: 0 });
  }
  let raw = '';
  req.on('data', (c) => { raw += c; });
  req.on('end', async () => {
    const evt = JSON.parse(raw || '{}');
    const delivery = req.headers['x-delivery-id'];
    const key = mode === 'delivery' ? delivery : evt.id;
    const already = seen.has(key);
    if (mode === 'strict') seen.add(key);
    await readStore();
    if (already) return json(200, { duplicate: true, event: evt.id, delivery }, { 'idempotent-replay': 'true' });
    seen.add(key);
    ledger.push({ event: evt.id, delivery, amount: evt.amount });
    json(200, { processed: true, event: evt.id, delivery });
  });
}).listen(8471, '127.0.0.1', () => console.log('receiver on 8471'));
const mode = process.argv[2] || 'event';
const url = 'http://127.0.0.1:8471/hooks';
const body = JSON.stringify({ id: 'evt_9f3', type: 'payment_intent.succeeded', amount: 2500 });
const send = (delivery) =>
  fetch(url, { method: 'POST', headers: { 'content-type': 'application/json', 'x-delivery-id': delivery }, body })
    .then((r) => r.text());

(async () => {
  await fetch(`${url}?mode=${mode}`, { method: 'DELETE' });
  for (const line of await Promise.all([send('del_a1'), send('del_b2')])) console.log(line);
  console.log(await (await fetch(url)).text());
})();

Steps

  1. Step 1.

    Deliver the event once.

    curl -s -X POST http://127.0.0.1:8471/hooks -H 'content-type: application/json' -H 'X-Delivery-Id: del_a1' -d '{"id":"evt_9f3","type":"payment_intent.succeeded","amount":2500}'
    
    {"processed":true,"event":"evt_9f3","delivery":"del_a1"}
  2. Step 2.

    Deliver the same event id again with a different delivery id. This is what a retry looks like on the wire.

    curl -s -X POST http://127.0.0.1:8471/hooks -H 'content-type: application/json' -H 'X-Delivery-Id: del_b2' -d '{"id":"evt_9f3","type":"payment_intent.succeeded","amount":2500}'
    
    {"duplicate":true,"event":"evt_9f3","delivery":"del_b2"}
  3. Step 3.

    Count the rows. The count is the assertion, not the status of the second call.

    curl -s http://127.0.0.1:8471/hooks
    
    {"mode":"event","rows":1,"ledger":[{"event":"evt_9f3","delivery":"del_a1","amount":2500}]}

    One row from two deliveries. A suite that stops here reports a receiver that deduplicates.

  4. Step 4.

    Send the same two deliveries in one tick. The script resets the ledger first, then prints both replies and the count.

    node fire-twice.js event
    
    {"processed":true,"event":"evt_9f3","delivery":"del_a1"}
    {"processed":true,"event":"evt_9f3","delivery":"del_b2"}
    {"mode":"event","rows":2,"ledger":[{"event":"evt_9f3","delivery":"del_a1","amount":2500},{"event":"evt_9f3","delivery":"del_b2","amount":2500}]}

    Two rows, same event id, same handler that passed step 3. Both deliveries read the set before either wrote to it.

  5. Step 5.

    Send the pair again with the id reserved before the store round trip instead of after it.

    node fire-twice.js strict
    
    {"processed":true,"event":"evt_9f3","delivery":"del_a1"}
    {"duplicate":true,"event":"evt_9f3","delivery":"del_b2"}
    {"mode":"strict","rows":1,"ledger":[{"event":"evt_9f3","delivery":"del_a1","amount":2500}]}

    One line moved in the handler and the count is back to one.

  6. Step 6.

    Key the receiver on the delivery id from the header instead of the event id, then repeat the sequential redelivery from steps 1 and 2.

    curl -s -X DELETE 'http://127.0.0.1:8471/hooks?mode=delivery' && for d in del_a1 del_b2; do curl -s -X POST http://127.0.0.1:8471/hooks -H 'content-type: application/json' -H "X-Delivery-Id: $d" -d '{"id":"evt_9f3","type":"payment_intent.succeeded","amount":2500}'; echo; done
    
    {"mode":"delivery","rows":0}
    {"processed":true,"event":"evt_9f3","delivery":"del_a1"}
    {"processed":true,"event":"evt_9f3","delivery":"del_b2"}
  7. Step 7.

    Count again.

    curl -s http://127.0.0.1:8471/hooks
    
    {"mode":"delivery","rows":2,"ledger":[{"event":"evt_9f3","delivery":"del_a1","amount":2500},{"event":"evt_9f3","delivery":"del_b2","amount":2500}]}

    Two rows with no concurrency anywhere. A delivery id is new on every attempt, so keyed on it the receiver has nothing to compare and every retry is a new event.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | Sequential pair, one row | The event id is honoured in the easy case | Keep going. Step 4 is the case that decides. | | Concurrent pair, two rows | Check-then-write with a gap in the middle | Reserve the id before the work, or put a unique constraint on it. | | Concurrent pair, one row | The reservation happens before the store call | Repeat it five times. A race that fails one run in five still fails. | | Two rows from a sequential pair | The handler is keyed on the delivery id, not the event id | Read the event id out of the body and key on that. | | Second delivery returns 500 | A unique constraint fired and was not caught | The provider treats 500 as failure and retries again, so this grows the queue. | | Second delivery returns 200 with no marker | A replay that no test can distinguish from work | Add a response field or header that says the event was already seen. |

Common mistakes

Sign: The deduplication test passes in CI and duplicates still reach the ledger in production.Cause: The test sends the two deliveries in sequence. A handler that reads its store, awaits, then writes only fails when both attempts are inside that await together, which is exactly what a provider retrying in parallel produces. Steps 3 and 4 are the same handler.
Sign: The receiver deduplicates on the delivery id in the header.Cause: Every redelivery carries a fresh delivery id and the original event id. Keyed on the delivery id nothing ever matches, and step 7 writes two rows without any concurrency at all. The event id is the only field a retry repeats.
Sign: The processed id is recorded after the work commits.Cause: A crash in between leaves no record that the work happened, so the next delivery repeats it. Reserve the id first, do the work, then mark the row done. A reservation that is never completed also tells you which events to investigate.
Sign: Deduplication works on one instance and fails behind a load balancer.Cause: A Set lives in one process. Two receiver instances hold two of them, so one event delivered to each is processed twice with no race inside either. The store has to be the one both instances share.

What to check next

FAQ

What is webhook deduplication?

Recording the id of every event you have processed and refusing to act twice on the same id. The check is a count of what the handler wrote, not a comparison of the two HTTP responses. Two deliveries, one ledger row.

Why does Stripe send duplicate webhook events?

Delivery is at least once. A receiver that times out, answers a non-2xx status or drops the connection is sent the event again, and the retry carries the event id of the first attempt. Every provider with this model behaves the same way.

What should a webhook idempotency key be?

The provider's event id from the body. It is stable across every retry of one event, which is the property the check needs. The delivery id in the header changes on each attempt and identifies the HTTP call, not the event.

Is this the same as API idempotency?

The mechanism is the same and the key is not. On your own API the client chooses the key and sends it in a header. On a webhook the provider chose it and put it in the body, and you have no say in what it is.

How long should processed event ids be kept?

Longer than the provider's retry window, which is a documented number rather than a guess. Look it up, then replay an event just inside the window and just outside it, and confirm the behaviour changes only at the boundary.

Does answering 200 quickly remove the need for this?

No. It lowers the retry rate and leaves the race untouched. A network timeout after your handler committed still produces a retry, and that retry carries an event id you have already acted on.

Verified

Verified by Maks Vernycurl 8.21.0node 22.23.2

Each output block is what the command above it printed on that date, on the host named in the step. Figures read from a live site move between runs. Compare the shape of the answer rather than the digits, and see the methodology for how a page is re-verified.

intermediate14 minpublished updated Maks Verny