
I have been saying the words "we'll just fire an event" in design reviews for years. The words come out cleanly. The underlying picture in my head, if I am honest, has always been a little blurry. Is the event fired inside the database transaction, or after? Who consumes it? If the broker is down for thirty seconds, does anything actually break?

This week I stumbled on a [ByteByteGo infographic](https://blog.bytebytego.com/) that lined up five of these patterns side by side - Transactional Outbox, CQRS, CQRS with Event Sourcing, Saga, Competing Consumers - and I realized I could recite all of them and confidently describe maybe two.

So I did the thing that always works for me when I half-understand something. I sat down and drew each one until I could explain, in plain language, what it is, what it is for, and when I would actually reach for it. This article is that exercise, written out.

## Why These Patterns Exist At All

Message brokers - Kafka, RabbitMQ, SQS, NATS, Pulsar, pick your favorite - are plumbing. They move messages from producers to consumers without either side having to know about the other. That is the whole value proposition, and it is a big one.

But the moment you put a broker in the middle of a system, new questions appear that a single-database design never had to answer:

- What if my database commit succeeds but the broker publish fails?
- What if two services need the same data but want completely different shapes of it?
- What if a "transaction" has to span four services that each own their own database?
- What if one producer emits faster than a single consumer can process?

Each of the five patterns below is a specific answer to one of those questions. None of them are new. All of them show up in real architectures under different names. What helped me was seeing them as *answers to questions* rather than as architectural vocabulary to namedrop.

## 1. Transactional Outbox - Stop Losing Events Between the DB and the Broker

The first time I shipped a service that fired events, my code looked something like this:

```typescript
await db.orders.insert(newOrder);
await broker.publish("order.created", newOrder);
```

It works. Most of the time. The problem is the tiny window between those two lines. If the process crashes after the insert but before the publish, the row exists and the event does not. If the publish succeeds but the commit fails, the event exists for an order that was never created. There is no way to make both lines atomic, because the database and the broker do not share a transaction.

The fix is to stop trying to publish inside the request path at all. Write the event to an **outbox table** in the *same database transaction* as the business row. A separate relay process reads the outbox and publishes to the broker afterwards.

```mermaid caption="Transactional Outbox - the broker publish is moved out of the request path and into a relay that reads from a table written in the same transaction as the business row."
flowchart LR
  S[Order Service]
  subgraph DB[Application Database]
    T[(Order Table)]
    O[(Outbox Table)]
  end
  R[Relay / CDC]
  B[(Message Broker)]
  C[Consumers]
  S -- "single transaction" --> T
  S -- "single transaction" --> O
  R -- "poll or CDC" --> O
  R --> B
  B --> C
```

**Use case I actually hit.** Order service writes an order, publishes `order.created`, and the email service sends a confirmation. Before the outbox: the broker had a thirty-second blip during a rolling restart and we dropped roughly two hundred confirmation emails. After the outbox: the blip became latency instead of loss - the events piled up in the table and the relay drained them once the broker came back.

The pattern already has its own deep-dive in the [pattern library](/patterns/outbox) if you want the schema and the tradeoff between polling and CDC. The thing I want to flag here is cultural rather than technical: once a team has the outbox, `await broker.publish(...)` inside a request handler stops being acceptable. The guarantee the pattern gives you only holds if everybody uses it.

## 2. CQRS - Reads and Writes Shouldn't Share a Model

CQRS stands for Command Query Responsibility Segregation, which is a long way of saying: the thing you use to change data should not be the same thing you use to read data.

In a traditional app, one model and one database serve both. You write `Order`, you read `Order`, both through the same schema. That works until your reads get complicated enough that they start deforming your writes. You add denormalized columns to make a dashboard fast. You add indexes that slow down inserts. You cache. The write model becomes a compromise between two jobs.

CQRS separates them. Writes go through a **command** side that models business operations and lands them in a write-optimized store. Reads go through a **query** side that pulls from a store shaped for the reads your product actually does. The two sides stay synchronized asynchronously - usually via events flowing through the broker.

```mermaid caption="CQRS - the command side owns state changes, the query side owns read shapes, and events on the broker keep the two eventually consistent."
flowchart LR
  Client[Client]
  Cmd[Command Handler]
  Q[Query Handler]
  W[(Write DB)]
  R[(Read DB)]
  B[(Broker)]
  Client -- "write" --> Cmd
  Cmd --> W
  W -- "change events" --> B
  B --> R
  Client -- "read" --> Q
  Q --> R
```

**Use case I actually hit.** A product catalog where the write model was Postgres (authoritative, relational, transactional) and the read model was Elasticsearch (great at fuzzy product search, typo tolerance, faceted filters). Every catalog update was a write against Postgres. A stream of events replayed the updates into Elasticsearch. The write side never had to care about search. The read side never had to care about business rules.

The trap to name out loud: CQRS gives you **eventual consistency** between the two sides. A user who updates a product and immediately lists products may see the old record. For most domains that is fine. For domains where it is not fine - a seller who needs to confirm their price change took effect - you need to either read-after-write from the write side, or make the query-side update synchronous, which defeats most of the point.

## 3. CQRS With Event Sourcing - Store What Happened, Not Just What Is

Regular CQRS keeps the current state on the write side. Event Sourcing goes one step further: it throws away the idea of storing the current state at all. What you store is the sequence of events that produced the state. Current state is a function of replaying those events.

Instead of a row saying "order 42: status=shipped, total=$99," you have an append-only log of events: `OrderCreated`, `OrderPaid`, `OrderShipped`. The current state is whatever you get by folding those events in order. Your read models are projections over the same event log.

```mermaid caption="CQRS with Event Sourcing - the event log is the source of truth. Read models are projections that can be rebuilt from the log at any time."
flowchart LR
  Client[Client]
  Cmd[Command Handler]
  ES[(Event Store)]
  B[(Broker)]
  P1[Projection A]
  P2[Projection B]
  R1[(Read DB - Orders)]
  R2[(Read DB - Analytics)]
  Client -- "command" --> Cmd
  Cmd -- "append event" --> ES
  ES --> B
  B --> P1 --> R1
  B --> P2 --> R2
  Client -- "read" --> R1
```

**Use case where it clearly earns its keep.** A banking ledger. You do not want to store "current balance" as a number that someone can `UPDATE`. You want an append-only log of every debit and credit, and the balance is whatever that log folds to. When an auditor shows up and asks "why was this account at $412.73 on March 2nd?", you have an answer more precise than any state table can give you.

**When I would not reach for it.** A CRUD app. A settings screen. A user profile. The value of event sourcing scales with how much you care about the *history* of the data, not the current value. If nobody will ever ask "how did we get here?", storing events is a lot of operational weight for benefits you will never consume.

The other honest caveat: event sourcing changes how you think about schema migrations, projections, and deployment. You need to be able to rebuild read models from the log. Events become part of your versioned API to your future self. I have only seen it go well in teams that deliberately invested in the tooling for it.

## 4. Saga - Distributed Transactions Without a Distributed Transaction

Sagas answer a specific question: how do you keep a multi-step business operation consistent when each step runs in a different service with its own database?

Take a checkout: reserve inventory, charge the customer, arrange shipping, send a confirmation. In a monolith this is one database transaction. In a microservice architecture, each step is a different service with its own database and no shared transaction. If the charge succeeds but shipping fails, you have a customer who paid for something that will never arrive - and no automatic rollback.

A saga decomposes the business operation into a sequence of local transactions, each of which emits an event that triggers the next step. If any step fails, **compensating transactions** run in reverse to undo the completed steps. There is no global lock; there is a protocol.

```mermaid caption="Saga (choreography style) - each service completes its local transaction and emits an event. Failure at any step triggers compensation events running in reverse."
sequenceDiagram
  participant User
  participant Order as Order Service
  participant Payment as Payment Service
  participant Ship as Shipping Service
  User->>Order: Place order
  Order->>Order: Create order (local tx)
  Order-->>Payment: order.created
  Payment->>Payment: Charge card (local tx)
  Payment-->>Ship: payment.charged
  Ship->>Ship: Arrange shipment (local tx)
  Ship-->>Order: shipment.arranged
  Note over Order,Ship: Any failure emits a compensation event<br/>that walks the saga backwards.
```

There are two flavors of saga. **Choreography** (shown above) has each service listen for events and decide what to do next - no central coordinator. **Orchestration** has a dedicated saga orchestrator that issues commands to each service and owns the state machine. Choreography is simpler to deploy but harder to observe. Orchestration is easier to reason about but introduces a coupling point. My rule of thumb: fewer than four steps, choreography. Four or more, orchestration, because the debuggability is worth it.

**Use case I actually hit.** An e-commerce checkout with five services. We started choreographed. By step four we could not answer the question "where did saga X get stuck?" without stitching logs from five systems by hand. We moved to an orchestrator with a state table. The code got slightly more boring and the on-call pages got dramatically shorter.

The pattern library has a [longer saga entry](/patterns/saga) with both styles in code. The thing worth repeating here: sagas give you **eventual consistency**, not atomicity. For a moment, the order exists and the payment does not. Your product, your dashboards, and your customer service team need to understand that.

## 5. Competing Consumers - One Queue, Many Workers

This is the most mechanically simple pattern of the five, and the one I reach for most often. It is also the one that needs the least explanation, so I will keep this short.

Producers drop messages onto a queue. Multiple consumer instances pull from the same queue. Each message is delivered to exactly one consumer. Load balances itself - faster consumers pull more, slower ones pull less. Add more consumers to go faster. Remove some to save cost.

```mermaid caption="Competing Consumers - a single queue, many worker instances. The broker load-balances messages across whichever consumers are currently pulling."
flowchart LR
  P1[Producer 1] --> Q
  P2[Producer 2] --> Q
  P3[Producer 3] --> Q
  Q[[Message Queue]]
  Q --> C1[Consumer A]
  Q --> C2[Consumer B]
  Q --> C3[Consumer C]
```

**Use case I actually hit.** Image processing pipeline. Users upload photos, a queue holds "process this photo" jobs, a fleet of worker containers pulls from the queue. On a slow day, three workers. On a campaign spike, forty. The queue absorbs the bursts so the upload endpoint never blocks.

**Two things I wish I had internalized earlier.** First, **messages must be processable in any order** - the broker load-balances across workers, so ordering per-message is not preserved across the fleet. If you need per-entity ordering (all events for user 42 processed in order), use a partitioned queue keyed on entity id, where each partition is its own competing-consumers pool of one. Second, **consumers must be idempotent**. Queues almost universally give at-least-once delivery. A worker that crashes after processing but before acknowledging will see the same message again. If running the handler twice produces the wrong answer, your pattern is broken - not the broker.

## How I Decide Which One I Actually Need

Drawing them side by side made the decision tree obvious in a way that reading the names never did:

- **I need to atomically record a state change and notify others?** → **Outbox.** Almost always. It is the default, not the exotic choice.
- **My reads and writes have genuinely different shapes and my write model is suffering for it?** → **CQRS.** Accept eventual consistency on the read side.
- **The history of changes is a first-class product concern (audit, debugging, rebuildable projections)?** → **CQRS with Event Sourcing.** Pay the tooling cost deliberately.
- **A business operation spans multiple services and must either all succeed or all compensate?** → **Saga.** Orchestrated if it has more than three steps.
- **I have a stream of independent tasks and I want to scale processing horizontally?** → **Competing Consumers.** Bake in idempotency from day one.

Most real systems use more than one of these at the same time. The checkout flow I mentioned earlier uses four of the five: the outbox to publish events atomically with the write, a saga to coordinate the flow, competing consumers on each step to scale, and a small CQRS slice for the order-lookup read model the customer service team uses. None of them felt clever in isolation. Together they made the system boring, which is the highest compliment I can give a distributed design.

{/* subscribe-cta */}

## The Quiet Lesson

The thing that surprised me most, drawing these out, is how much of "distributed systems design" is not really about algorithms or consensus or exotic infrastructure. It is about naming the *kind of inconsistency* you are willing to tolerate - across a transaction boundary, across two databases, across five services - and then picking the pattern that turns that inconsistency into something your system recovers from on its own.

The broker is the plumbing. These patterns are the plumbing conventions that keep the water going the right direction when one of the pipes shakes.

If you only take one thing from this article: the next time you write `await broker.publish(...)` directly after a database write, pause. That line is a micro-decision about consistency, availability, and what you owe the next engineer who has to debug this at 3am. The patterns above are the menu of better answers.


Five Message Broker Patterns


The first time a senior engineer stopped me mid-dashboard and said "I don't care about the average, show me P99," I thought he was being fussy. The average looked great. The system was handling traffic. What else did he want?

What he wanted, I figured out later, was the shape of the distribution - not its center of gravity. Averages lie in a very specific way for distributed systems. P99 is how engineers force the lie to show itself.

This is my attempt to write down *why* P99 is the number that ends up carved on every serious latency dashboard I have ever seen, even though most engineers encounter it as an unexamined default.

## The Median Is A Comforting Fiction

Here is the thing the P50 will not tell you.

Imagine a service where 99 out of 100 requests take 50ms, and one request takes 10 seconds. The median is 50ms. The average is around 150ms. Both numbers tell you the service is "fast." Neither of them tells you that 1% of your users just watched a progress spinner for 10 seconds.

At a hundred requests per second, that is one angry user every second. At a thousand requests per second, it is ten. At the scale of an actual product, the users at the tail are not a rounding error - they are a population large enough to notice, complain, and leave.

```mermaid A healthy-looking distribution with a long right tail. The P50 lives on the left; P99 lives in the territory on the right that averages smooth over.
xychart-beta
    title "Request Latency Distribution"
    x-axis ["0-50ms", "50-100ms", "100-200ms", "200-500ms", "500ms-1s", "1s-3s", "3s-10s"]
    y-axis "Requests (log)" 1 --> 10000
    bar [6000, 2500, 900, 400, 150, 40, 10]
```

This is the shape almost every real-world latency distribution has: a fat body on the left, a long tail on the right. The tail is where the interesting stories live. The body is where metrics go to look fine.

## The Tail At Scale

The academic argument for P99 - and the one that reframed how I think about system performance - comes from a 2013 paper by Jeff Dean and Luiz Barroso called *The Tail at Scale*. The paper is short, cited thousands of times, and worth reading on its own. Here is the part that matters.

Modern systems are not monoliths. A single user request, say a search or a product page, fans out into dozens or hundreds of internal sub-requests. The gateway calls auth. Auth calls a user service. The user service calls a feature flag service. Everything calls caches. Everything calls databases. The page cannot render until *all* of those calls return.

```mermaid A single user request fans out across a dozen services, caches, and databases. The user only sees the slowest path.
flowchart TD
    U[User Request] --> G[API Gateway]
    G --> A[Auth Service]
    G --> P[Product Service]
    G --> R[Recommendations]
    G --> I[Inventory]
    A --> C1[(Session Cache)]
    A --> D1[(User DB)]
    P --> C2[(Product Cache)]
    P --> D2[(Catalog DB)]
    R --> M[ML Ranker]
    R --> D3[(User Events)]
    I --> D4[(Stock DB)]
    M --> F[Feature Store]
```

Now do the math. Suppose every service on that path has a healthy P99 of one second - meaning 99% of calls are faster than a second. You might reasonably say each service is performing well.

If the user-facing request depends on 100 independent sub-requests, the probability that *none* of them land in their slow tail is $0.99^{100}$, which is roughly 0.37. In other words, there is a **63% chance** that the user waits at least a second on a sub-request, even though every single service individually meets its SLO.

That number is the entire argument for P99.

```mermaid Even when every individual service meets its P99 SLO, the probability of a tail hit climbs steeply with fan-out depth. At 100 sub-requests the composite response is more likely than not to catch a slow one.
xychart-beta
    title "Probability of a Tail Hit vs Fan-Out Depth (P99 per service)"
    x-axis ["1 svc", "5 svcs", "10 svcs", "25 svcs", "50 svcs", "100 svcs"]
    y-axis "P(at least one slow sub-request) %" 0 --> 100
    bar [1, 5, 10, 22, 40, 63]
```

The individual P99s look fine. The *composition* of them is catastrophic. As soon as you fan out, rare events stop being rare — they become the dominant user experience. A tail that you could ignore at the component level becomes the median experience at the system level.

This is the phenomenon Dean and Barroso called "the tail at scale." It is the uncomfortable reason averages lie. Averages smooth; composition amplifies.

## The Money Follows The Tail

The other reason engineers fixate on P99 is that the people paying for the system care about it even when they do not know the acronym.

The numbers from industry have been cited so often they are almost folklore at this point, but the folklore happens to be true:

- Amazon's internal studies have reported that every additional 100ms of latency costs about **1% of sales**.
- Google has reported that a 400ms increase in search latency produces about a **0.59% drop in searches per user**.
- User research has consistently found that once page load times exceed a few seconds, abandonment climbs sharply.

Notice what these numbers are actually measuring. They are not measuring the average response time. They are measuring what happens when a user's request lands in the slow part of the distribution. The users who feel "the internet is slow today" are the users whose requests took 2 seconds instead of 200ms. That is a P99 problem, not a P50 problem.

When the business tells you "the site feels slow," they are telling you about the tail, even if they are looking at a graph of averages. If you chase the average down from 300ms to 250ms but leave the P99 at 4 seconds, your users will not notice the improvement. The users who cared were the ones in the tail, and they are still there.

## P99 Is An Early Warning

There is a second reason to watch P99 that has nothing to do with user experience: it is often the first place failures announce themselves.

A healthy system runs with spare capacity. When capacity is comfortable, a slow GC pause here or a head-of-line blocking moment there gets absorbed. The median does not move. The P99 twitches, briefly, and settles.

As the system creeps closer to its saturation point, the twitches get bigger and more frequent. Queues build up behind transient slowdowns. Threads that would have freed quickly now stay held. The *body* of the distribution still looks fine - most requests are completing - but the tail is getting longer. P99 starts drifting up while P50 stays flat.

```mermaid The path from a benign P99 twitch to a full outage. You see it on the P99 graph minutes or hours before it shows up on the P50.
flowchart LR
    A[Minor GC pause] --> B[Request slows]
    B --> C[Thread held longer]
    C --> D[Queue builds]
    D --> E[More requests slow]
    E --> F[More retries]
    F --> G[Queue saturates]
    G --> H[Cascading failure]
    style H fill:#3a1a1a,stroke:#7a2a2a
```

```mermaid P50 holds flat for most of the degradation window while P99 climbs steadily. The widening gap is the early warning signal. By the time P50 moves, the incident has already started.
xychart-beta
    title "P50 vs P99 as System Approaches Saturation"
    x-axis ["t-60m", "t-45m", "t-30m", "t-15m", "t-5m", "now"]
    y-axis "Latency (ms)" 0 --> 2000
    line [55, 52, 58, 60, 62, 65]
    line [130, 140, 280, 650, 1300, 1900]
```

By the time the median moves, you are past warning and into incident. By the time the average moves, you are in the post-mortem.

This is the operational case for watching P99 even when the business case is not obvious. The 99th percentile is the canary. The median is the corpse.

## Why Not P95? Why Not P999?

A fair question at this point is: if we are chasing the tail, why stop at P99? Why not P99.9, or P99.99?

The honest answer is: it depends on scale, and on what you can afford to care about.

At low traffic, P99 is already measuring something rare. At a thousand requests per second, P99 is ten requests a second in the slow bucket. P99.9 is one. P99.99 is one every ten seconds. You cannot meaningfully alert on a percentile that fires once per hour.

At high traffic - the Googles, Metas, Cloudflares of the world - the math flips. A thousand requests per second is a slow minute. Their user-facing systems operate at millions of requests per second, and P99.9 or P99.99 becomes the right level of resolution, because "one in a thousand" is still thousands of users per minute.

P95 exists for the opposite reason. At small scales, or for systems where 5% of slow requests is acceptable, P95 is a cheaper number to keep green. It is forgiving in a way P99 is not.

My rule of thumb is: the percentile you should target is the one where the count of "slow requests per unit time" is high enough to notice and low enough to fix. For most services I have worked on, that has landed on P99. For systems operating at web scale, it is usually P99.9. The principle is the same either way: *you are targeting the tail, not the center.*

```mermaid At 10,000 requests per second, P95 fires 500 times a second -- too noisy to act on. P99.99 fires once every ten seconds -- too rare to alert on. P99 sits in the actionable middle.
xychart-beta
    title "Slow Requests per Second at 10K rps by Percentile"
    x-axis ["P95", "P99", "P99.9", "P99.99"]
    y-axis "Slow req/sec" 0 --> 550
    bar [500, 100, 10, 1]
```

{/* subscribe-cta */}

## What Watching P99 Actually Changes

Once you take P99 seriously, several engineering practices stop looking optional.

**Timeouts become budgets, not upper bounds.** If the user-facing request budget is 1 second and you fan out to five sub-requests, each sub-request gets a fraction of that budget - not the full second. Sub-request timeouts that equal the full user budget are a way of guaranteeing that P99 at the leaf becomes P99 at the root.

**Hedged requests and request duplication become real tools.** The Dean and Barroso paper's most practical advice is to send a duplicate request if the first one has not returned by, say, P95. The second request gives you another independent chance to avoid the slow tail. The cost is a small amount of extra load; the benefit is that your user-visible P99 becomes close to your server-side P95.

**Load shedding starts to make sense.** If slow requests are the enemy, a service that rejects 1% of requests cleanly under overload is better than a service that queues them and serves all of them slowly. The first service has a controlled error rate and a healthy P99. The second has a beautiful success rate and a terrifying one.

**Observability shifts from averages to histograms.** An average cannot be decomposed. A histogram can. Once you store full latency histograms - not just means and maxes - you can compute any percentile you want, slice by endpoint, and compare tail behavior across deploys. Most reliability work I have done in the last few years started with "can we actually see the distribution?" and the answer was uncomfortably often "no."

## The Mental Model That Stuck

The shift that took me a long time to make is this: I used to think of a service as having a single "speed." The average, the typical response, the thing I would quote if someone asked. A service was "fast" or "slow."

A service does not have a single speed. It has a distribution. Every single request samples from that distribution independently. At the scale of one user, the sample is the experience. At the scale of many users fanning out across many services, the joint probability of every sample avoiding the tail collapses fast.

P99 is a shorthand for reasoning about the distribution's right edge - the place where the business loses money, where the operator loses sleep, and where the system shows you what is about to break before it actually breaks.

## The Short Version

Averages are a story about a system that does not exist. P50 is a story about the user who did not have a problem. P99 is a story about the users who did.

Once you fan out, the tail eats you. Once you are at scale, the tail is the product. Once you are in an incident, the tail told you first.

That is why every serious latency dashboard, at every company I have ever respected the engineering of, has P99 on it in large type. Not because it is a best practice. Because everything else, sooner or later, lies.

## References

- Dean, J., & Barroso, L. A. (2013). [The Tail at Scale](https://doi.org/10.1145/2408776.2408794). *Communications of the ACM*, 56(2), 74–80.
- Puzhavakath Narayanan, S., et al. (2016). Reducing latency through page-aware management of web objects by content delivery networks. *ACM SIGMETRICS Performance Evaluation Review*, 44(1), 89–100.
- Google Research. (2015). [User Preference and Search Engine Latency](https://research.google.com/pubs/archive/34439.pdf).
- Zhao, Z., et al. (2025). SLOpt: Serving real-time inference pipeline with strict latency constraint. *IEEE Transactions on Computers*.


Why Engineers Are Obsessed With P99


There's a failure mode that will ruin your week. Not because it's loud, it won't be. Your services stay up. Metrics look mostly reasonable. Alerts don't fire immediately. And yet your system is producing no useful output. Users are getting errors. Revenue is going to zero. And your dashboards are telling you everything is fine.

This is a **meta-stable failure**. It's the failure mode that keeps distributed systems engineers up at night precisely because it doesn't look like a failure.

This is first time i heard the term "**meta-stable failure**.", it spark curiosity in me and dig down deeper.

## Up, But Down

The formal definition, from a 2021 HotOS paper co-authored by engineers at AWS, Meta, and others: *"Metastable failures occur in open systems with an uncontrolled source of load where a trigger causes the system to enter a bad state that persists even when the trigger is removed."*

Read that last clause again. *Even when the trigger is removed.*

This is what separates meta-stable failures from ordinary outages. In a normal incident, you find the cause, fix it, and the system recovers. In a meta-stable failure, you fix the cause and the system stays broken. The failure has become self-sustaining. You're not fighting the original trigger anymore, you're fighting the system's own behavior.

## The Sustaining Loop

To understand why, you need to understand what's actually happening mechanically.

Imagine a backend service under normal load. A routine deployment causes a brief latency spike. Clients, following best practices, retry their failed requests. But those retries arrive while the service is already struggling, which increases load further. The higher load causes more failures. More failures cause more retries. More retries cause more load.

The original trigger, the deployment, is long gone. The latency spike from the deploy resolved in 30 seconds. But the feedback loop it kicked off is still running, hours later, at full intensity. Your service is extremely busy processing retries for requests it will never successfully complete. It has high throughput. It has near-zero *goodput*, useful work delivered to users.

The system is "up." In a meaningful sense, it's completely down.

## Why Good Engineering Made It Worse

Here's the uncomfortable part. Every mechanism in that loop was put there deliberately, by thoughtful engineers, for good reasons.

**Retries** exist because transient failures are real. Networks drop packets. A request that fails once often succeeds on the second attempt. Without retries, users see errors from hiccups that the system would have handled fine with one more chance.

**Timeouts** exist because hanging requests are worse than failed ones. A request that waits forever holds resources, threads, connections, memory, that block other requests. Timeouts free those resources so the system can keep serving other traffic.

**Backoff** is supposed to prevent retry storms, and in mild overload scenarios it works well. But in deep overload, even exponential backoff with jitter doesn't reduce enough load to allow recovery. The system is so far from its recovery threshold that even well-behaved clients can't get it there.

Each of these was the right decision in isolation. The meta-stable failure emerges from their interaction under specific conditions.

## The Trigger Is Not The Problem

This is the key insight that changes how you think about these failures, and the one that gets buried when teams do post-mortems.

A meta-stable failure has two distinct phases: a **trigger** that pushes the system into the bad state, and a **sustaining mechanism** that keeps it there. Engineers almost always focus their post-mortem on the trigger, the deployment, the traffic spike, the upstream timeout. They fix it and declare the incident closed.

But the trigger is almost incidental. A different trigger, a thundering herd, a cache invalidation, a brief network partition, would have caused exactly the same outcome. The system has a stability property that makes it vulnerable: under certain load conditions, a feedback loop can lock it into a degraded state it cannot escape from on its own.

Fix the trigger and you've prevented *this* incident. Fix the sustaining mechanism and you've changed the system's fundamental failure behavior.

## Four Places Sustaining Loops Hide

The HotOS paper identifies common sources. These aren't exotic, they're patterns you've almost certainly shipped.

**Retry logic without load shedding.** Retries increase load during overload. If your service doesn't have a mechanism to start rejecting work, actively, deliberately, with a 503, it has no way to reduce its own load. The retry traffic fills every available slot. The only escape is an operator manually cutting traffic.

**Caching with thundering herds.** A cache miss causes a backend query. Under normal load, this is fine, misses are occasional and the backend handles them. But a cache eviction or flush can cause every node to miss simultaneously, sending a spike to the backend that exceeds its capacity, causing timeouts, causing retries, filling the queue. The cache can't repopulate if the backend can't respond, so the thundering herd sustains itself.

**Slow error paths.** Your fast path is highly optimized. Your error path, the code that runs when something goes wrong, was written once and never touched again. If error handling is significantly slower than normal processing, failures consume more resources per request than successes. At high error rates, this means the system is burning capacity on work that produces no value. The more it fails, the less capacity it has to succeed.

**Autoscaling with lag.** Cloud autoscaling is not instantaneous. There's a window, sometimes several minutes, between when load exceeds capacity and when new instances are ready. During that window, if the load is sustained by a feedback loop, the existing instances may saturate before help arrives. New instances that join an already-overloaded cluster can get immediately hammered and fail to start up successfully at all.

## What Recovery Actually Looks Like

If you're in a meta-stable failure and you remove the trigger, the system won't recover. It's locked in the bad state. You need to do something that breaks the sustaining loop.

The most common effective interventions:

**Reduce load below the recovery threshold.** If retries are sustaining the loop, stop the retries. Cut client traffic at the load balancer. Drop the offered load far enough that the service can start clearing its backlog. This feels wrong, you're making the user-visible failure more severe, but it's the only way to let the system breathe.

**Restart the sustaining mechanism.** If cache state is part of the loop, a controlled warm-up sequence (where you rebuild cache before allowing full traffic) can break the cycle. If connection pool exhaustion is sustaining the failure, recycling connections on the server side can free the system.

**Emergency throttling.** A hard rate limit, not a soft one, that doesn't grow under retry pressure. Circuit breakers that stay open long enough for backend queues to drain, not just long enough to let one request through.

{/* subscribe-cta */}

## How to Build Systems That Don't Get Trapped

The sustaining loop is the target, not the trigger. That reframes what resilience engineering actually means for this failure mode.

**Give your services a way to shed load.** Every service that accepts open-loop traffic needs a pressure valve. This means returning 503 when load exceeds a threshold, not queueing indefinitely. Queues that grow without bound under load are a meta-stable failure waiting to happen. They accept work the service will never complete, holding capacity that could be freed.

**Treat your error paths with the same care as your fast paths.** Profile them. Load test them. If your error handling is two orders of magnitude slower than normal processing, it will become a resource sink at exactly the wrong time.

**Design for controlled recovery.** The hardest operational moment in a meta-stable failure is the transition back to normal. Load needs to ramp up slowly, too fast, and the recovering system re-enters the bad state. Traffic ramp-up procedures, caching warm-up sequences, and connection pre-warming aren't nice-to-haves; they're the mechanism that prevents re-entry.

**Think in feedback loops, not component failures.** Most distributed systems reliability work treats failure as: component X failed, what is our tolerance for X failing? Meta-stable failures are a different animal. They emerge from the *interaction* of functioning components. The question isn't "what happens when X fails" but "what happens to the system's dynamics when load pushes past a threshold." Control theory, not traditional reliability engineering, is the right mental model.

## The Honest Thing About Post-Mortems

When you've been through a meta-stable failure and you're writing the post-mortem, the trigger makes for a satisfying story. There was a deploy. There was a traffic spike. Here's what we did wrong, here's the fix, here's the test to prevent it next time.

But if you stop there, you've documented a specific trigger while leaving the underlying failure mode intact. The next time, different trigger, same catastrophic loop, same hours of degradation.

The meta-stable failure asks a harder question: not "what went wrong" but "what property of this system makes it capable of getting trapped like this?" That question doesn't have a clean action item. It has architectural answers that take quarters to implement. It requires reasoning about your system's dynamics under load, not just its behavior under component failure.

That's the harder work. And it's the only work that actually changes the failure mode.

reference video:
https://www.youtube.com/watch?v=u3GjIXP9N0s

Meta-Stable Failure: When Your System Is Up But Completely Down


I watched a ten-minute video this week that rearranged the way I think about retries. It was [Arpit Bhayani's](https://www.youtube.com/watch?v=8sTuCPh3s0s) walk-through of the thundering herd problem - one of those talks that is short, unhurried, and pointed enough to leave a mark. I had written retry logic dozens of times. I had read the AWS blog post about jitter. I thought I understood it. I did not.

The thing that got me was not any single technical fact. It was the small realization that I had been holding two incompatible ideas in my head at the same time - "retries make systems more reliable" and "retries cause outages" - without ever forcing them into the same room.

This article is me forcing them into the same room.

## The Setup That Made It Click

Arpit opens with a scene that almost every backend engineer has lived through, even if they did not realize it. A service has a brief hiccup. Maybe the deploy took a few seconds longer than expected. Maybe a database failed over. Maybe a downstream dependency got slow. The clients, behaving exactly as their engineers intended, notice the failures and retry.

That is where the trouble starts.

Each client is a single, well-behaved program. None of them are misbehaving. None of them are attacking your service. They are doing the correct thing - retrying a failed request so that the end user sees success instead of an error. And yet, in aggregate, they form a wave of traffic that hits your already-struggling service at exactly the moment it least wants to see it.

The image Arpit uses is the one the name implies: a herd of animals, all waking up at the same moment and stampeding in the same direction. Individually, each one is just running. Collectively, they trample whatever is in the way.

That framing was new to me. I had always thought of the thundering herd as something that happened *to* a system. Arpit's point is that it is something the system's own clients produce, following instructions the system's own engineers wrote.

## Retries Are Not Free

The first insight I took from the video is one I had technically known but never really internalized: every retry is a request. That sounds obvious. It is not.

A retry feels, from the client's perspective, like a second chance. The first attempt failed, so we try again, and if the second succeeds we are whole. No harm done. But the server does not see it that way. The server sees two requests, both of which it must process, both of which consume CPU and memory and a connection slot, even if the first one failed quickly.

When you multiply that by thousands of clients, the math changes. A service running at 80% capacity can handle its load. The same service, faced with every client sending twice as many requests as normal, is running at 160% of its capacity - except capacity does not work like that. It does not degrade linearly. Past some threshold, everything gets slower, which causes more timeouts, which causes more retries.

I had read about this. I had not understood it as *my* retry logic's fault.

## Exponential Backoff, and the Trap Inside It

The standard answer to naive retries is exponential backoff. Wait one second, then two, then four, then eight. The rationale is clean: each retry gives the server more time to recover, and a client that keeps failing will exponentially reduce the load it offers.

This is what I had implemented, many times. I thought of it as the solved version of the problem.

What Arpit's video pointed out - and the thing I want to make sure lands, because this is the piece I had genuinely missed - is that exponential backoff alone does not solve the thundering herd. It only moves it.

Here is why. Suppose a thousand clients all hit a failure at the same moment. They all wait one second. They all retry at the same moment. They all fail. They all wait two seconds. They all retry at the same moment. They all fail. The waves of traffic are farther apart now, but each wave is still synchronized. The herd is still thundering - it is just thundering at regular intervals instead of continuously.

If your service cannot handle a thousand simultaneous requests, it does not matter whether those requests arrive every second or every minute. Each wave will saturate it. Each saturation will produce more failures. Each failure will trigger the next synchronized retry.

The quiet, uncomfortable insight is that well-behaved clients obeying well-written retry logic can still collectively DDoS the service they are trying to talk to.

## Jitter Is the Missing Piece

This is where jitter comes in, and why it is more than a small implementation detail.

Jitter means adding a random offset to the backoff delay. Instead of every client waiting exactly four seconds, each client waits somewhere between, say, two and six seconds. The distribution spreads the herd out in time. Instead of a wave, you get a stream. Instead of a thousand simultaneous retries, you get roughly a thousand retries scattered across a window of several seconds - which is exactly what the service can actually handle.

The service gets breathing room. The retries eventually succeed. The system recovers. Nobody notices.

What was interesting to me is how small this change is in code. It is literally one extra line - adding a random value to the sleep before retry. The effect on system behavior is enormous. The behavior of the whole system depends on a function call that looks, to anyone reading the code, like a minor detail.

That asymmetry is, I think, the reason this problem keeps happening in real-world systems. The fix is trivial to implement. Recognizing that you need it - that backoff alone is not enough - is the hard part.

## The Framing Shift

The part of the video that I keep coming back to is not any of the specific techniques. It is a framing shift in how I think about retries at all.

Before, I saw retries as a reliability mechanism - something I added to a client to protect users from transient failures. The mental model was: my client is defending itself against an unreliable network.

After, I see retries as a *source of load* - something I am producing that arrives at a server that may not be ready to receive it. The mental model is now closer to: my client is offering additional work to a service that may be in no state to accept it.

Both framings are true. But they lead to very different code. The first framing optimizes for "retry until success." The second framing optimizes for "retry in a way that does not make the failure worse."

I do not think I would have made that shift by reading a definition of the thundering herd problem. I had read definitions. What the video did was put it in a sequence - naive retry, then backoff, then backoff-plus-jitter - where each step visibly failed to solve the problem until the last one. Following the failure made the reasoning stick in a way that stating the conclusion did not.

{/* subscribe-cta */}

## What I'm Changing

Watching that video, I made a short list of things I wanted to audit in my own work. Writing them down here partly because the list is useful, and partly because I suspect other engineers have some version of the same list waiting to be written.

**Every retry loop I own needs jitter, not just backoff.** This is the one-line change. I checked a handful of services after watching the video and found backoff-without-jitter in more places than I want to admit. None of those services had caused incidents yet. That is not the same as being safe.

**Client-side retry policy is a system-design decision, not a defensive local habit.** If every team independently picks a retry policy for the calls *they* make, the aggregate behavior is not anyone's responsibility - which means it is nobody's. I want the retry defaults to live in a shared client, with jitter baked in, so that individual teams cannot accidentally skip it.

**Servers need to be robust to thundering herds from their own clients.** Load shedding, server-side rate limiting, and fast rejection with 429 responses are not about hostile traffic. They are about surviving the helpful traffic produced by your own well-meaning client retry loops. I used to think of those mechanisms as paranoia. Now I think of them as the other half of the retry story.

**The absence of an incident is not evidence of safety.** Thundering herd failures are a threshold effect. The system is fine up to some load, and then it is not. The transition is sharp. You do not get a gentle warning. You get a clean graph and then a vertical line.

## The Short Version

If you retry without backoff, you make failures worse. If you back off without jitter, you delay the problem instead of solving it. If you back off with jitter, you give the service the one thing it needs to recover: uncorrelated traffic.

That is the entire story. It takes ten minutes to explain on video, about thirty seconds to implement in code, and - for a lot of us, me included - far too many years to actually internalize.

If you have not watched the video, [watch it](https://www.youtube.com/watch?v=8sTuCPh3s0s). If you have written retry logic recently, open that file and check for the jitter. It is almost always faster to fix now than to debug at 3 AM when a brief upstream hiccup turns into a multi-hour incident because every client in your fleet decided to retry on exactly the same second.


Thundering Herd: What a 10-Minute Video Taught Me About Retries


You've pasted it into a form before. Copied it out of an `Authorization` header, stared at three dots separating what looks like Base64 noise, and wondered what any of it actually means. JWT - JSON Web Token - is one of those technologies you absorb through osmosis rather than deliberate study. Then, one day, you get a cryptic `401 Unauthorized` in production and realize the osmosis wasn't enough.

This essay is the deliberate study version. We'll cover the structure, the claims that matter, the expiry trap engineers fall into, and the security property JWTs actually give you versus the ones you might be assuming.

## The Three-Segment Structure

A JWT looks like this:

```
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c
```

Three segments separated by dots. Each segment is Base64url-encoded - like regular Base64 but with `+` replaced by `-` and `/` by `_`, so the result is URL-safe. You can decode each segment independently without any secret or key.

**Segment one: the header.** Decode it and you get JSON like:

```json
{
  "alg": "HS256",
  "typ": "JWT"
}
```

`alg` names the signing algorithm. `HS256` means HMAC-SHA256, a symmetric algorithm where the same secret signs and verifies. `RS256` means RSA with SHA-256, an asymmetric algorithm where a private key signs and a public key verifies. This distinction matters more than most engineers realize - we'll get to it.

**Segment two: the payload.** Also JSON, containing *claims* - assertions about the subject:

```json
{
  "sub": "1234567890",
  "name": "John Doe",
  "iat": 1516239022,
  "exp": 1516242622
}
```

Claims are just key-value pairs. Some are *registered* (standardized by the RFC, like `sub`, `iat`, `exp`). Others are *public* (widely used by convention, like `email`, `roles`). The rest are *private* (application-specific, like `org_id`, `plan_tier`).

**Segment three: the signature.** The cryptographic proof that the header and payload haven't been tampered with. Computed over `base64url(header) + "." + base64url(payload)` using the algorithm named in the header. This is the only part you cannot decode without the secret or public key.

## The Claims That Actually Matter

Most JWTs carry a handful of registered claims. Know these by heart:

**`sub` (Subject):** Who the token is about - typically a user ID. Opaque to the client, meaningful to your backend.

**`iss` (Issuer):** Who issued the token. Useful when your system accepts tokens from multiple identity providers. Always validate this if you do.

**`aud` (Audience):** Who the token is intended for. If your API has an `aud` of `api.example.com`, reject tokens with a different audience even if the signature is valid. Many teams skip audience validation and regret it.

**`iat` (Issued At):** Unix timestamp (seconds since epoch) when the token was created. Useful for detecting tokens that are suspiciously old even if not yet expired.

**`exp` (Expiry):** The timestamp after which the token must be rejected. This is the claim responsible for the most `401` errors on a Monday morning.

**`nbf` (Not Before):** The timestamp before which the token must not be accepted. Less commonly used, but available.

## The Expiry Trap

`exp` is a Unix timestamp in *seconds*, not milliseconds. This traps developers who compare it against `Date.now()`, which returns milliseconds:

```typescript
// Wrong: Date.now() returns milliseconds
if (payload.exp < Date.now()) { /* always expires immediately */ }

// Correct: compare seconds to seconds
if (payload.exp < Math.floor(Date.now() / 1000)) { /* correct */ }
```

The second trap is clock skew. If your token-issuing server and your verifying server have clocks that differ by even a few seconds, a just-issued token can appear expired. Production systems address this with a small *leeway* - typically 30–60 seconds - added to `exp` before rejecting. Most JWT libraries support configuring leeway directly.

The third trap is timezone confusion. `exp` is always UTC. It has no timezone. Tools that display it in local time can make a token that expires at midnight UTC appear to expire at midnight local time, throwing off your debugging.

{/* subscribe-cta */}

## What the Signature Actually Guarantees

This is where engineers carry incorrect mental models the longest.

The signature guarantees **integrity**: if someone modifies the payload after issuance, the signature check fails. You'll know the token was tampered with.

The signature does *not* guarantee **confidentiality**: the payload is Base64-encoded, not encrypted. Anyone who holds the token can read every claim without the signing key. Never put sensitive data in a JWT payload - no passwords, no PII you'd rather keep private, no security-sensitive flags.

For confidentiality, you want JWE (JSON Web Encryption), a different standard that actually encrypts the payload. JWE is less common in practice precisely because most use cases only need integrity, not secrecy.

## Symmetric vs. Asymmetric: Why It Matters in Microservices

With `HS256` (symmetric), both the issuer and every verifier share the same secret. This works fine for a monolith. In a microservices architecture it forces every service that validates JWTs to hold the signing secret - which means every one of those services is now a high-value target. Compromise any one of them and an attacker can mint arbitrary tokens.

With `RS256` (asymmetric), the auth service holds the private key and signs tokens. Every other service holds only the public key, which it uses to verify. The public key is, as the name implies, public - you can distribute it freely. An attacker who compromises a downstream service gets nothing useful for forging tokens.

If you're building a new system with more than two services, start with `RS256`. The operational complexity is modest; the security posture improvement is significant.

## The Stateless Tradeoff

JWTs are often described as "stateless authentication." The auth server doesn't need to store session records - all the information needed to validate a request travels with the token. This is genuinely useful for horizontal scaling: any instance of your service can validate any token without a database lookup.

The cost is that you cannot revoke a JWT before it expires. Revocation requires statefulness. The common mitigations:

**Short expiry + refresh tokens.** Issue access tokens that expire in 15 minutes. Issue a long-lived refresh token separately and store it server-side. When the access token expires, the client exchanges the refresh token for a new access token. Revocation means invalidating the refresh token.

**Token blocklist.** Store revoked JTIs (`jti` - the JWT ID claim, a unique identifier for the token) in Redis. Check every incoming token against the blocklist. This adds a cache lookup per request but preserves the scalability benefit.

**Leaky bucket.** For most applications, short expiry is enough. If a user changes their password, revoke their refresh tokens; their access tokens will naturally expire within minutes. Perfect revocation is less important than it sounds for most use cases.

## Reading JWTs in the Wild

Knowing the structure lets you debug auth failures faster. When a request gets a `401`, before reaching for logs:

1. Copy the token from the request header.
2. Paste it into a decoder (like the one on this site - it runs entirely in your browser).
3. Check `exp` against the current UTC timestamp. Expired? That's your answer.
4. Check `iss` and `aud`. Does your service accept this issuer? Is the audience correct?
5. Check the claims your service cares about. Is the `sub` in your user table? Is the `role` claim present?

Most `401 Unauthorized` errors fall into one of these buckets. You don't need to verify the signature to diagnose expiry or claim issues - you just need to read the payload.

## A Note on Security

None of the above replaces proper security review. JWTs are a tool with a specific threat model: they protect against payload tampering in transit. They don't protect against token theft (use HTTPS, secure cookie flags, short expiry), implementation bugs (use a vetted library, not a handrolled one), or weak secrets (for HS256, your secret should be at least 256 bits of entropy - a random string from a CSPRNG, not a password).

The `alg: "none"` attack is worth knowing: early JWT implementations accepted tokens where the header set `alg` to `none`, bypassing signature verification entirely. Modern libraries reject this by default, but if you're ever auditing a custom implementation, look for it.

Understanding the format won't make you immune to auth bugs. But it will make you faster at diagnosing them and more deliberate about the tradeoffs your auth architecture is making.


JWT: What Every Engineer Should Know


There is a class of distributed systems bug that appears only under load, only in production, and always at the worst possible moment. It looks like a race condition. It shows up in your Stripe dashboard as a duplicate charge. Your on-call engineer escalates it at 3am, and by the time the postmortem is written, three engineers have burned a weekend and a customer has filed a chargeback.

The pattern that prevents it has a name. It is called idempotency. And despite being the foundational primitive for safe distributed operations, it is consistently the last thing teams think about when designing new endpoints.

This essay is about why idempotency matters, what it actually costs to implement correctly, and when the pattern is worth every bit of the investment.

## What Idempotency Actually Means

An operation is idempotent if applying it more than once produces the same result as applying it once. This sounds obvious until you try to design a payment API. Charging a card is not naturally idempotent - if you call `POST /charges` twice with the same payload, you get two charges. Networks time out. Clients retry. Servers crash mid-write. The gap between "the server received the request" and "the client received the acknowledgment" is where money disappears.

The classic solution is the idempotency key - a client-generated token, typically a UUID, that the server uses to deduplicate requests within a time window.

```typescript
// Client sends this header with every mutating request
const charge = await stripe.charges.create({
  amount: 2000,
  currency: "usd",
  source: "tok_visa",
}, {
  idempotencyKey: "order_abc123_attempt_1",
});
```

If the network drops and the client retries, Stripe detects the same key and returns the original response rather than creating a new charge. The client cannot distinguish a first execution from a replay - which is exactly the point.

But the simplicity of the interface hides considerable complexity on the server side.

## The Three Layers of Idempotency

Implementing idempotency correctly requires thinking at three distinct layers, and most teams only address one or two of them.

**Layer 1: The deduplication store.** You need somewhere to record that you've seen a key and what the result was. This is usually a fast key-value store - Redis, DynamoDB, or a dedicated column in your Postgres transactions table. The entry must be written atomically with the underlying operation, or you have a new race condition: two concurrent requests with the same key, both checking before either has written.

```typescript
async function createCharge(
  idempotencyKey: string,
  params: ChargeParams
): Promise<Charge> {
  // Atomic check-and-set using a database transaction
  return await db.transaction(async (tx) => {
    const existing = await tx.idempotencyKeys.findUnique({
      where: { key: idempotencyKey },
    });

    if (existing) {
      // Return the stored result - not a new execution
      return JSON.parse(existing.responseBody) as Charge;
    }

    // Execute the actual operation
    const charge = await processCharge(params);

    // Record the key and result atomically
    await tx.idempotencyKeys.create({
      data: {
        key: idempotencyKey,
        responseBody: JSON.stringify(charge),
        expiresAt: addDays(new Date(), 7),
      },
    });

    return charge;
  });
}
```

**Layer 2: Request fingerprinting.** Idempotency keys only work if you tie them to a specific request shape. If a client sends the same key with different parameters - say, the same `idempotencyKey` but a different `amount` - you have to decide: honor the key and ignore the parameter mismatch, or reject the request with a 422? Stripe rejects. That is the correct answer. A mismatched fingerprint means the client has a bug, and surfacing that error immediately is better than silently ignoring the mismatch.

**Layer 3: Side effects beyond your database.** Your charge endpoint probably sends an email confirmation, increments a counter in a metrics system, and triggers a webhook. None of those systems have your idempotency guarantee. You need to ensure that downstream side effects are either naturally idempotent (incrementing a monotonic counter with a version check) or wrapped in their own deduplication logic. Idempotency at the API layer does not automatically propagate to async workers.

<PullQuote attribution={{ name: "Werner Vogels", role: "CTO, Amazon" }}>
Everything fails, all the time. Design for it.
</PullQuote>

## When the Pattern Pays Off

Idempotency is not free. The deduplication store adds a round trip. The fingerprinting logic adds complexity. The key expiration policy adds a new class of edge case - what happens when a key expires and the client retries six days later?

So when is it worth the cost?

The answer is almost always "yes" for any operation that changes state and could be retried by an upstream caller. In practice, this means:

- Payment and billing operations
- Order creation
- Message sends (email, SMS, push notifications)
- Provisioning actions (create VM, allocate resource)
- Any operation that touches external systems you do not control

The operations where idempotency is less critical are read operations (which are already naturally idempotent) and internal operations that are naturally safe to repeat. But even there, building the habit of asking "what happens if this runs twice?" is worth the mental overhead.

## The Subtle Failure Mode Nobody Talks About

There is a failure mode that trips up even experienced engineers: partial completion.

Your charge endpoint writes to the database, then calls the payment processor, then sends the confirmation email. The payment processor call succeeds, but the database write fails on the way back (a transient network partition, a rollback triggered by a constraint violation). The customer's card has been charged. Your database does not know about it. When the client retries with the same idempotency key, your deduplication store has nothing - and you charge the card again.

The solution is to write the idempotency key entry *before* you start executing the operation, not after. Set its status to `pending`, execute the operation, then update the status to `completed` with the stored result. If the system crashes mid-execution, subsequent attempts will find the `pending` entry and can either wait (if the operation might still complete) or attempt recovery.

This is the pattern Stripe describes in their engineering blog. It is sometimes called the "idempotency receipt" pattern, and it is the detail that separates robust implementations from ones that fail under concurrent retries.

## What This Means for Your API Design

If you are designing a new service today, these are the decisions that matter:

**Use client-generated keys, not server-generated.** Servers cannot create idempotency keys because they cannot know whether a request is a retry. Only the client knows that. The client generates a UUID, stores it locally, and sends it with every retry attempt until it receives a definitive response.

**Be explicit about key scope.** Does your key apply across all operations or just a specific endpoint? Stripe scopes keys to the calling API key, which prevents one integration from colliding with another. Define your scope explicitly in your documentation.

**Choose a reasonable expiration window.** Seven days is a common choice. Long enough to cover any realistic retry window, short enough that your deduplication store does not grow unbounded. Paginate your key cleanup job.

**Communicate idempotency status to callers.** Return a header like `Idempotent-Replayed: true` when you return a cached response. This helps clients distinguish "the operation ran" from "the operation ran and I'm giving you the result from last time." It also helps your debugging story: a duplicate alert with `Idempotent-Replayed: true` is a client bug, not a server bug.

## The Real Cost of Not Implementing It

The cost of skipping idempotency is not just duplicate charges. It is the cumulative technical debt that accumulates when you build the wrong primitive into your foundation. Every endpoint you add without idempotency is an endpoint that needs to be retrofitted when you start handling retries. Every integration that assumes single delivery is an integration that will behave incorrectly under network partitions.

The engineers who understand this build retry logic correctly from day one. Their systems degrade gracefully. Their on-call rotations are quieter. Their postmortems are shorter.

Idempotency is not glamorous. It does not make it into architecture talks at conferences. But it is one of the clearest examples of a pattern that pays for itself many times over - in the operational incidents it prevents, the customer trust it preserves, and the engineering time it does not consume.

The pattern worth paying for is the one you never have to think about again.

---

*If this resonated, the follow-up is about exactly-once delivery - what it means, what it costs, and why distributed systems practitioners generally settle for at-least-once with idempotency instead.*


Essaystaggeddistributed-systems

Five Message Broker Patterns

Why Engineers Are Obsessed With P99

Meta-Stable Failure: When Your System Is Up But Completely Down

Thundering Herd: What a 10-Minute Video Taught Me About Retries

JWT: What Every Engineer Should Know

The Pattern Worth Paying For