Why does the same payment webhook arrive twice?

A payment webhook can arrive twice. The receiver is the only place that can make the second arrival harmless, because delivery over a network is at-least-once: a sender that gets no answer inside its timeout cannot tell a lost request from a lost reply, so it asks again. A failed delivery comes back ten more times across the next sixteen hours; after that it sits in the dashboard until somebody replays it. Each delivery carries its own evt_… id in X-Webhook-Id. The invoice inside data carries a different one that never changes. Deduplicate on the first, book orders against the second, and a repeat costs one index lookup.

The delivery contract lives in the webhook documentation. What follows is why it has that shape, and what it asks of the code on your side.

Why can't a sender promise an event arrived once?

Because nothing on a network can. Two parties that talk only through a channel that drops messages cannot reach certainty about each other's state, which has been a proof since 1975 rather than a budget question. Jim Gray gave it the name it still travels under three years later, the Two Generals Paradox.

Watch what that does to one webhook. The sender opens the request, writes the body, waits. The 10-second attempt timeout expires with nothing on the socket. Did the receiver never see it? Or did it verify the signature, credit the order, and lose the reply on the way back? From the sending side those are the same silence.

Which leaves a choice, and only one half of it is honest about money. Send once, and a paid invoice sometimes never reaches the order system. Send again, and a handler sometimes runs twice on one payment, the one failure a receiver can engineer against.

Why does everyone ship at-least-once?

Nothing here is specific to crypto. Google documents at-least-once as the Pub/Sub default on every subscription type and tells subscribers to be idempotent. Stripe's webhook page says endpoints occasionally receive the same event more than once and that the answer is to log the ids you have processed.

What changes between products is the price of the duplicate. A repeated analytics ping bends a chart. A repeated invoice.paid ships a second parcel or mints a second licence key, and the order system has no idea, because each run looked correct on its own.

RFC 9110 is precise about the word itself: a method is idempotent when the intended effect of several identical requests is the same as the effect of one. It is a property of what the receiver does with an event, not of what the sender puts on the wire. A header can carry a key; only the receiver can make the effect the same twice.

What the outbox pattern does not fix

It does not fix duplicates, and that is not a flaw in the pattern. A state change and a message have to leave together, and no transaction spans a database and a network. Write the row first, and a crash before the send loses the message. Send first, and a crash before the write announces something that never happened. That is the dual write, and the transactional outbox is the standard answer: the message goes into the same database, in the same transaction as the fact it describes, and a relay carries it out to the wire.

Read the guarantee closely, because it is narrower than the reputation. The message exists if and only if the transaction committed. Richardson lists the other half under the pattern's issues: the relay can publish a message, die before recording that it published, and publish it again on restart.

The trade is worth naming. A message that vanished leaves no trace anywhere, while a message that repeats arrives carrying the id it had the first time, and an id is something a receiver can index. Whether a particular sender runs an outbox is invisible from outside. What reaches your endpoint is the property it produces.

Which id should the receiver deduplicate on?

Two identities travel in one delivery, and choosing the wrong one is the failure this design is shaped around.

X-Webhook-Id holds the same value as event_id in the body, an evt_…, and it names the delivery: one endpoint's copy of one transition. Every retry of that delivery carries it, and a manual replay keeps it.

The resource id inside data is an inv_…, wdr_… or pcd_…, and it names the thing the money happened to. Every endpoint subscribed to that event sees the same one, and so does every later status the resource moves through. The payload reference has the envelope in full.

Now the two ways to get it wrong. A merchant running two endpoints who keys the order table on event_id sees two different ids for one payment and books it twice. A merchant who dedupes deliveries on the invoice id blocks the second delivery for that invoice. That second delivery is invoice.paid landing after invoice.confirming, so the order is never fulfilled at all.

Stripe reached the same split from the other side: log event ids to catch repeats, and reach for the object id in data.object plus the event type when two separate event objects describe one thing.

Why is the row written before the work starts?

Writing the row first is a crash-recovery rule, and it earns its keep in one narrow window. A handler that fulfils the order and then records the event leaves a gap between the two. A process killed in that gap has shipped an order nothing in the system knows about, and the next delivery ships it again.

Turn the handler around. Verify the signature, insert the delivery, answer, and let a worker do the rest on a schedule you control.

create table webhook_delivery (
    event_id     text primary key,          -- evt_…, this delivery
    resource_id  text        not null,      -- inv_… / wdr_… / pcd_…, the payment
    event_type   text        not null,
    body         jsonb       not null,
    received_at  timestamptz not null default now(),
    processed_at timestamptz
);

insert into webhook_delivery (event_id, resource_id, event_type, body)
values ($1, $2, $3, $4)
on conflict (event_id) do nothing
returning event_id;

Zero rows back means this delivery is already on record: answer 2xx and stop, because there is nothing left to decide. One row back means the work is yours, and the 2xx you send is a receipt for the bytes rather than a claim that anything shipped.

That split is also what keeps a handler inside the 10-second attempt timeout. The third-party call that hangs and the licence mint that stalls on a bad day both belong behind a queue with retries of its own.

What makes a replay safe?

Replay is an operator action on events that failed, and it keeps the delivery's own id. A replay of something your table already holds is therefore caught at the door, at the price of one index lookup.

The interesting case is the other one. An endpoint that spent the outage rejecting deliveries has an empty table, so after the fix every replayed event is new to it. Meanwhile the business state behind those events may already be correct, because somebody in support read the dashboard and marked the order paid, or an overnight reconciliation job found it first.

A dedupe table cannot see any of that. It knows which deliveries it has read and nothing else. The guard for that case sits on the transition: an invoice already booked as paid does not book again, and the check belongs in the same transaction as the write it protects.

How long does an event keep coming back?

Eleven attempts, with the gaps growing from a minute at the start to eight hours at the end, which puts the last one roughly 16 hours after the first. The delivery schedule is published attempt by attempt.

Read that as two separate facts about your own downtime. A deploy passes unnoticed: the early rungs are minutes apart, so the event comes back before anyone opens a dashboard. A database outage that runs through a working day does not, and the events whose cycle expired inside it are marked failed.

Nothing happens to the endpoint. When the eleventh attempt fails, the event is the only thing marked failed; the endpoint is left alone, still active and still subscribed. There is no strike count and nothing to switch back on. A receiver that was down all afternoon comes back to a live endpoint and a list of failed events waiting for replay.

The dashboard shows delivery state, attempt count and next retry for the hundred most recent events, newest first, and it does not page past that. Treat it as a window on the last stretch of traffic. Your own delivery table is the archive.

What breaks while the webhook secret rotates?

For 24 hours after a rotation the signature header carries two v1 values instead of one, and a delivery is valid when it matches either. The window is what lets a receiver pick up the new secret on its own deploy schedule.

A receiver that reads v1 as a field rather than a list fails every delivery in that window. Each failure is one attempt on the ladder. Sixteen hours later those events are marked failed, and the first visible symptom is usually an order that never shipped.

So parse v1 as a list, compare each candidate with a timing-safe function, and accept on the first match. The signature page spells out what gets hashed and in what order. Every official Paymos SDK already behaves this way and the conformance suite pins a two-value header as a test vector, so an integration built on an SDK inherits it.

How do you prove a receiver is idempotent?

By replaying, not by reading the code. The property under test is a claim about the second run, and the second run is exactly what a unit test tends not to produce.

The dashboard's API Playground sends real HMAC-signed requests with the merchant's own credentials, in Sandbox only, and its webhook playground produces a genuine delivery. That is enough to run the whole check against a staging receiver:

1. Fire a delivery, let the handler finish, and note the order row it created.
2. Fire the same event again. It should reach your table, find the id, and return 2xx without touching the order.
3. Hold the receiver open past 70 seconds — the 10-second timeout plus the first minute of backoff — so the retry lands while the first run is still going. That collision is what the unique index is there for.
4. Replay a failed event after fixing the receiver, and confirm an order already booked stays booked once.
5. Compare your delivery table against the invoices the API reports as paid. A missing row is a subscription or firewall problem; a double fulfilment is a key problem.

Run those five before the receiver ships and the retry ladder turns into background noise. Attempt eleven gets handled the way attempt one did: the id is already on record, and nothing downstream moves.