Webhooks Made Simple: How to Handle Retries, Duplicates, and Out‑of‑Order Events

TL;DR: Webhooks are not guaranteed to arrive once, on time, or in the correct order. This is normal behavior in real systems. Reliable integrations first verify incoming webhook requests, acknowledge them quickly, make event processing safe to repeat, and update data based on the current state of a resource rather than the order in which events arrive.

Why webhooks are harder than they look

Webhooks look simple on the surface. A provider sends an event to your endpoint, and your application reacts.

In real production systems, however, things are rarely this straightforward.

Networks fail. Servers restart. Deployments happen. Traffic spikes. Providers retry. Because of these realities, webhook delivery in production behaves very differently from demos or local testing.

If a system assumes that each webhook is sent only once, arrives immediately, or always arrives in the correct order, the integration will eventually break.

To understand why these failures are so common, the first step is to examine how webhooks actually behave once they leave the provider and enter real production environments.

How webhooks actually behave in production

Webhook providers prioritize reliability over clean delivery patterns. Most providers use at least‑once delivery, meaning they prefer sending the same event multiple times rather than risking permanent data loss.

Because of this design choice:

• The same webhook may arrive more than once
• Delivery may be delayed
• Events may arrive out of order
• Retries may occur minutes or even hours later

The key idea to understand is simple. Webhooks guarantee delivery, not uniqueness or order.

Once this behavior is recognized as normal, a natural question follows. If this is expected, why do webhook integrations still fail so often?

Why webhooks often go wrong

In most cases, webhook failures do not come from the provider. They come from assumptions made by the system consuming the webhook.

When webhook consumers are not designed for production realities, issues surface as:

• Duplicate emails or user notifications
• Repeated workflow transitions
• Double billing or provisioning
• Corrupted or inconsistent application state
• Customer support tickets reporting repeated actions

Although these problems differ in appearance, they share the same root cause. The webhook consumer assumes events are unique and arrive in order.

One specific behavior exposes this assumption more clearly than any other. Retries.

Why retries are normal and not a bug

Webhook providers retry deliveries when:

• The endpoint responds slowly
• The endpoint returns an error
• The service is temporarily unavailable
• A network disruption occurs

From the provider’s perspective, retrying is the safest option. From the application’s perspective, a retry is indistinguishable from a duplicate event.

This leads to an important conclusion. Retries are not something to eliminate. They are something to handle safely.

Handling retries safely requires webhook processing to behave the same way whether an event is received once or multiple times. This is where idempotency becomes essential.

What idempotency means in practice

Idempotency means that processing the same webhook multiple times results in the same final outcome as processing it once.

In simple terms, if the same event arrives again, nothing bad should happen.

An idempotent webhook consumer:

• Recognizes events it has already processed
• Avoids repeating side effects
• Keeps system state consistent

Without idempotency, even a single retry can create user‑visible failures.

However, idempotency alone does not solve every problem. Even when duplicates are handled safely, systems can still fail if events arrive in the wrong order. To understand why, it helps to look at how idempotency is usually implemented.

A simple way to implement idempotency

A common and production‑ready approach includes:

• Each webhook includes a unique event identifier.
• The identifier is stored when the event is processed.
• If the same identifier appears again, processing is skipped.

This creates exactly‑once business behavior on top of at least‑once delivery.

If no stable event identifier exists, deduplication is still possible, but it requires careful design and clear documentation.

Even with deduplication in place, systems can fail if older events overwrite newer state. This brings us to the next challenge: event ordering.

Why event order cannot be trusted

Even when webhook events are generated in sequence, they may not arrive in that same sequence.

This happens because:

• Retries may overlap with newer deliveries
• Different network routes introduce variable latency
• Systems process events concurrently at scale

In document workflows, this can result in a document completed event arriving before a document viewed event.

This behavior is normal in distributed systems.

Because of this, arrival order cannot be treated as the source of truth. Systems need another way to decide whether an incoming event should be applied.

How state‑based processing solves ordering problems

When arrival order cannot be trusted, reliable systems rely on the current state of the resource instead.

State‑based processing works by:

• Loading the current state of the resource
• Checking whether the incoming event represents a valid transition
• Applying the change only if it is still appropriate
• Ignoring stale or backward updates

For example:

• Once a document reaches a completed state, it should never move backward
• Late events that would downgrade state should be ignored

This approach keeps systems correct even when events arrive out of order, arrive more than once, or are processed concurrently.

With retries handled through idempotency and ordering handled through state checks, webhook delivery becomes predictable rather than dangerous.

How retries, idempotency, and ordering work together

Retries introduce duplicate deliveries. Duplicate deliveries require idempotent processing. Idempotency alone is insufficient if outdated events can overwrite valid state.

A production‑grade webhook consumer therefore:

• Expects retries
• Treats duplicate events safely
• Applies state changes conditionally
• Prevents irreversible actions from running more than once

Together, these patterns separate simple webhook demos from systems that survive real production traffic.

To see how this applies in practice, consider a common real‑world scenario.

How webhooks are used in eSignature workflows

In eSignature systems, webhooks often represent document lifecycle milestones such as:

• Document sent
• Document viewed
• Document signed
• Document completed

Because webhook delivery is inherently unpredictable, consumers of these events should always assume:

• Events may be retried
• Events may be duplicated
• Events may arrive out of order

Designing for these assumptions ensures:

• Document completion logic runs once
• Signed documents are stored once
• User notifications are not duplicated
• Document history remains consistent and auditable

These challenges become more severe when the same webhook infrastructure serves multiple customers.

Why multi‑tenant SaaS systems must be extra careful

In multi‑tenant SaaS platforms, a single webhook handler processes events for many tenants.

This amplifies the impact of mistakes:

• One duplicate event can affect multiple customers
• Weak isolation can lead to cross‑tenant data exposure
• Debugging becomes more complex

To reduce these risks, webhook processing must:

• Reliably identify the tenant for every event
• Strictly isolate tenant data
• Apply idempotency within tenant boundaries
• Maintain tenant‑specific audit trails

At this scale, webhook correctness becomes not only a reliability concern but also a security and trust requirement.

Common mistakes to avoid in production webhook handling

Many production issues appear repeatedly when teams:

• Process events before checking idempotency
• Perform heavy work before acknowledging the webhook
• Assume events arrive in the correct order
• Allow resource state to move backward
• Mix tenant data during processing

These patterns eventually lead to duplicated actions, corrupted workflows, and customer‑visible incidents, especially in regulated or compliance‑sensitive environments.

Why eSignature and compliance workflows depend on webhooks

Webhook‑driven workflows often trigger irreversible actions such as:

• Storing signed documents
• Updating compliance records
• Notifying stakeholders
• Activating billing or access rules

Errors in these workflows are expensive and highly visible.

Idempotent, state‑based processing ensures these actions occur once and only once, even when retries, concurrency, or system recovery occur.

Beyond correctness, production systems must also ensure that incoming webhook requests are authentic. This introduces the need for webhook security.

Securing webhook endpoints in production

So far, the focus has been on handling webhook behavior correctly after an event is received. In production environments, it is equally important to ensure that incoming webhook events are genuine.

Webhook endpoints are typically public and accessible over the internet. Without request verification, fake or tampered events can trigger unintended actions, even if retry handling and ordering logic are implemented correctly.

Many providers address this by signing webhook requests so the receiving application can verify authenticity.

As an example, BoldSign signs each webhook request using HMAC‑SHA256. The generated signature is included in the X‑BoldSign‑Signature header and can be validated by the receiving application using your BoldSign Webhook signing secret key.

When webhook signature verification is implemented:

• Only genuine events are processed
• Payload tampering is detected
• Forged or unauthorized requests are rejected

This verification step is especially important when webhooks trigger document completion workflows, user notifications, compliance updates, or billing‑related changes.

Webhook security complements idempotency and state‑based processing by ensuring that only trusted events reach business logic.

Real‑world situations where reliable webhooks matter

The table below shows simple, real‑world situations where reliable webhook handling is important.

Why correct webhook handling is important

When webhooks are handled correctly:

• Integrations survive retries and outages
• Workflows remain consistent under load
• Replay and recovery become safe
• Support investigations are simpler
• Systems scale without unexpected failures

This is especially important for systems that manage legal documents, compliance data, and user trust.

Final key takeaways

Webhooks use at least‑once delivery, which makes retries and duplicates inevitable. Delivery order cannot be trusted. Idempotency prevents repeated side effects. State‑based processing protects workflow correctness. In multi‑tenant SaaS systems, webhook correctness is also a security responsibility.

Reliable webhook consumers acknowledge requests quickly, tolerate retries, and remain stable under concurrency.

You can start exploring this capability today using a free sandbox account and reach out through the support portal if you need help designing or validating your webhook‑based document automation architecture.

FAQs