Webhook reliability: retries, idempotency, and the design choices that survive partner outages

Key takeaways

• Webhooks are at-least-once by design, so your consumer will receive duplicates. Idempotency isn't optional; dedupe on the event ID before you process anything.
• Retries without jitter stampede. AWS found that adding randomized jitter to exponential backoff cut retry call volume by more than half (AWS, 2015).
• Providers disagree on retry behavior: Stripe retries for up to three days (Stripe), Svix runs a fixed eight-attempt schedule (Svix), and GitHub doesn't auto-retry at all (GitHub). Design for the weakest partner, not the best.
• Verify signatures with a timestamp tolerance and constant-time comparison, over the raw request body. A 5-minute tolerance is Stripe's default for blocking replays (Stripe).

TL;DR

Reliable webhooks come down to four moves. On the sending side: retry with exponential backoff plus jitter, and give up into a dead-letter queue instead of retrying forever. On the receiving side: treat every delivery as a possible duplicate and make processing idempotent, keyed on the event ID. Everywhere: sign payloads with HMAC over the raw body, check a timestamp to block replays, and log every attempt so that when a partner says "we sent it," you can prove otherwise in seconds. None of this is exotic. It's just rarely done all at once, which is why integrations quietly rot.

Why do webhooks fail in production?

Webhooks fail because they inherit every failure mode of the public internet plus a few of their own. The receiver is down, slow, or mid-deploy. The network drops the connection. The payload arrives twice. Or someone replays an old one. And the ground is shifting under all of it: average API uptime slipped from 99.66% to 99.46% between early 2024 and early 2025, roughly 60% more downtime year over year, across more than two billion monitoring checks (Uptrends, 2025). That's API endpoints generally, not webhooks specifically, but your integrations ride on exactly those endpoints.

Here's an honest caveat worth stating up front: there's very little rigorous public data on webhook failure rates specifically. Most numbers you'll find are vendor blog inventions with no methodology. So this guide leans on how the reliable providers actually behave, documented in their own words, rather than on invented percentages. That's the more useful signal anyway.

What retry strategy should a webhook sender use?

Use exponential backoff with jitter, and cap it with a retry budget. Plain exponential backoff has a nasty property: when many clients fail at the same instant, they all retry at the same intervals and stampede the recovering server. AWS demonstrated that adding jitter, randomizing each delay, spread the load out and "reduced our call count by more than half" in a hundred-client test (AWS, 2015). Their conclusion was blunt: jittered backoff "should be considered a standard approach for remote clients."

The other half is knowing when to stop. Retrying forever just turns one outage into two. Mature providers bound the effort and then escalate. The specifics vary more than you'd expect:

The lesson from that table isn't "copy Stripe." It's that as a consumer you cannot assume anything about retries. GitHub, for instance, does not redeliver failed deliveries automatically; you have to re-drive them yourself from the UI or API within three days (GitHub). Design for the partner with the stingiest guarantee, not the most generous one.

How do you make webhook processing idempotent?

Dedupe on the event's unique ID before doing any work, and store that ID durably. At-least-once delivery is the norm, which is a polite way of saying you will get the same event more than once. If receiving "payment succeeded" twice ships two orders, that's not a partner bug; it's a design gap on your side.

The pattern that survives is a receipts table. Every provider worth integrating sends a stable event identifier. Record it the moment you accept the delivery, inside the same transaction as the side effect, with a unique constraint. If the insert conflicts, you've seen this event; acknowledge with a 2xx and do nothing. This is the consumer-side mirror of what Stripe does on its API: send an Idempotency-Key and Stripe saves the first response and returns it verbatim on retries, "including 500 errors," with keys safely pruned after 24 hours (Stripe). The IETF has a draft Idempotency-Key header field aiming to standardize exactly this for making "non-idempotent HTTP methods such as POST or PATCH fault-tolerant" (IETF draft) — useful to know, though it's a proposal, not a ratified standard.

One subtlety: idempotency and ordering are different problems. Deduplication stops double-processing; it does not guarantee events arrive in order. If your logic depends on sequence, key on the event ID and compare a version or timestamp on the resource before applying a change.

How should you verify webhook signatures?

Compute an HMAC over the raw request body, compare it in constant time, and reject anything whose timestamp is too old. Three separate defenses, and skipping any one leaves a hole. Signing proves the payload came from your partner; the timestamp check stops an attacker from replaying a captured-but-valid request; constant-time comparison stops timing attacks on the signature check itself.

The industry has converged here, which makes it easy to get right. GitHub signs with HMAC-SHA256 in an X-Hub-Signature-256 header that "always starts with sha256=," and its docs explicitly require a constant-time comparison such as crypto.timingSafeEqual, never == (GitHub). Stripe signs the concatenation of the timestamp and the raw body, ships it in Stripe-Signature as t= and v1= values, and applies a default five-minute tolerance to block replays, warning against setting that tolerance to zero (Stripe). The Standard Webhooks specification generalizes the same idea into three portable headers, webhook-id, webhook-timestamp, and webhook-signature, signing id.timestamp.payload and supporting multiple signatures for key rotation (Standard Webhooks).

Two implementation traps catch almost everyone. First, you must verify against the raw bytes: if your framework parses JSON and re-serializes before you hash, the signature won't match, because key order and whitespace change. Second, verify before you parse or process. A signature check that runs after your JSON deserializer has already touched the payload is a check running too late.

When do you need a dead-letter queue?

You need a dead-letter queue the moment "retry" and "give up" become different outcomes, which is immediately. A dead-letter queue (DLQ) is where messages go when they can't be processed successfully after a bounded number of attempts. In Amazon SQS, you attach a redrive policy with a maxReceiveCount; once a message has been received that many times without success, SQS moves it to the DLQ instead of redelivering forever (AWS SQS). AWS also recommends the DLQ's retention period be longer than the source queue's, so a message that failed near the end of its life still lands somewhere you can inspect it.

The DLQ turns a silent, permanent loss into a visible, recoverable one. A poison event, one that will never process because of a bug or bad data, stops blocking the queue behind it and instead waits in the DLQ for a human to look, fix, and redrive. Without one, a single malformed payload can wedge an entire integration and you find out from an angry customer, not a dashboard.

What we got wrong: On one integration we ran retries with no jitter and no DLQ, because the partner "was reliable." When they had a brief outage, every one of our workers retried in lockstep the instant they came back, we hammered their recovering endpoint, and they rate-limited us into a second, longer outage. Worse, the events that failed during the window were just gone; there was no queue holding them. We rebuilt it with jittered backoff and a dead-letter queue in an afternoon. The next partner outage was a non-event: retries spread out, the stragglers landed in the DLQ, and we redrove 40 events with one command once the partner recovered.

What does an observable webhook pipeline look like?

It looks like a queryable log of every delivery attempt, with enough context to assign blame in seconds. The recurring failure mode of integrations isn't just lost events; it's the argument afterward. The partner says "we sent it," you say "we never got it," and nobody can prove anything. An event log ends that.

Record, per attempt: the event ID, the received timestamp, the signature verification result, the HTTP status you returned, the attempt number, and the outcome. When you can pull up "event evt_123, received at 14:02:11, signature valid, we returned 500 on attempts 1–3, succeeded on attempt 4," the blame game lasts one message. This is also how you catch slow rot: a rising rate of DLQ arrivals or signature failures is an early warning long before a customer notices. Providers model this too. Svix emits an operational message.attempt.exhausted event when retries run out (Svix), which is a pattern worth copying: make exhaustion a first-class, alertable event rather than a log line nobody reads.

A reference checklist that survives partner outages

Pull it together into something you can audit against. If you can tick all of these, an outage on either side degrades gracefully instead of losing data.

1. Sending: exponential backoff with jitter, a bounded retry budget, and a DLQ for exhausted messages.
2. Receiving: respond 2xx fast (many providers time out at 10–15 seconds), then process asynchronously off a queue.
3. Idempotency: a durable receipts table keyed on the event ID, written in the same transaction as the side effect.
4. Security: HMAC-SHA256 over the raw body, constant-time comparison, and a timestamp tolerance to block replays.
5. Ordering: if sequence matters, version your resources; don't rely on delivery order.
6. Observability: a per-attempt delivery log and an alert on DLQ growth and signature-failure spikes.

Notice how little of this is provider-specific. The providers vary wildly on retries and headers, but the consumer-side discipline, dedupe, verify, queue, log, is constant. Build that once and it protects every integration you add later.

Frequently asked questions

How many times should I retry a failed webhook?

Bound it by total time, not just attempt count, and back off with jitter. Real providers span a huge range: Svix uses eight attempts over roughly a day, while Stripe retries for up to three days (Stripe, Svix). After the budget is spent, move the message to a dead-letter queue rather than retrying indefinitely, so a permanent failure becomes visible instead of infinite.

Do I still need idempotency if the provider signs payloads?

Yes. Signatures prove who sent the payload and that it wasn't tampered with; they say nothing about whether you've already processed it. Because delivery is at-least-once, duplicates are normal, and Stripe explicitly persists and replays prior responses for repeated idempotency keys (Stripe). Dedupe on the event ID regardless of signing.

What stops someone from replaying a captured webhook?

A timestamp inside the signed payload, checked against a tolerance window. Stripe signs the timestamp together with the body and rejects deliveries outside a default five-minute tolerance, and warns against disabling that check (Stripe). The Standard Webhooks spec builds the same timestamp-based replay protection into its webhook-timestamp header (Standard Webhooks).

Why verify the signature over the raw body instead of parsed JSON?

Because parsing and re-serializing changes the bytes. Key ordering, whitespace, and number formatting can all shift, so the HMAC you compute over re-serialized JSON won't match the one the provider computed over the original. GitHub and Stripe both require the raw request body for exactly this reason (GitHub).

Build it once, reuse it everywhere

Webhook reliability has a reputation for being fiddly, but the surface area is small and stable. Senders retry with jitter and give up into a dead-letter queue. Receivers respond fast, dedupe on the event ID, and verify signatures over the raw body with a replay window. Everyone logs every attempt. The providers will keep disagreeing on the details, so the durable investment is the consumer-side machinery that doesn't care which partner is on the other end.

If you're wiring up a partner integration that has to survive real outages, or untangling one that already burned you, that reliability layer is exactly the kind of thing we help teams design once and reuse across every connector.