
The first time a senior engineer stopped me mid-dashboard and said "I don't care about the average, show me P99," I thought he was being fussy. The average looked great. The system was handling traffic. What else did he want?

What he wanted, I figured out later, was the shape of the distribution - not its center of gravity. Averages lie in a very specific way for distributed systems. P99 is how engineers force the lie to show itself.

This is my attempt to write down *why* P99 is the number that ends up carved on every serious latency dashboard I have ever seen, even though most engineers encounter it as an unexamined default.

## The Median Is A Comforting Fiction

Here is the thing the P50 will not tell you.

Imagine a service where 99 out of 100 requests take 50ms, and one request takes 10 seconds. The median is 50ms. The average is around 150ms. Both numbers tell you the service is "fast." Neither of them tells you that 1% of your users just watched a progress spinner for 10 seconds.

At a hundred requests per second, that is one angry user every second. At a thousand requests per second, it is ten. At the scale of an actual product, the users at the tail are not a rounding error - they are a population large enough to notice, complain, and leave.

```mermaid A healthy-looking distribution with a long right tail. The P50 lives on the left; P99 lives in the territory on the right that averages smooth over.
xychart-beta
    title "Request Latency Distribution"
    x-axis ["0-50ms", "50-100ms", "100-200ms", "200-500ms", "500ms-1s", "1s-3s", "3s-10s"]
    y-axis "Requests (log)" 1 --> 10000
    bar [6000, 2500, 900, 400, 150, 40, 10]
```

This is the shape almost every real-world latency distribution has: a fat body on the left, a long tail on the right. The tail is where the interesting stories live. The body is where metrics go to look fine.

## The Tail At Scale

The academic argument for P99 - and the one that reframed how I think about system performance - comes from a 2013 paper by Jeff Dean and Luiz Barroso called *The Tail at Scale*. The paper is short, cited thousands of times, and worth reading on its own. Here is the part that matters.

Modern systems are not monoliths. A single user request, say a search or a product page, fans out into dozens or hundreds of internal sub-requests. The gateway calls auth. Auth calls a user service. The user service calls a feature flag service. Everything calls caches. Everything calls databases. The page cannot render until *all* of those calls return.

```mermaid A single user request fans out across a dozen services, caches, and databases. The user only sees the slowest path.
flowchart TD
    U[User Request] --> G[API Gateway]
    G --> A[Auth Service]
    G --> P[Product Service]
    G --> R[Recommendations]
    G --> I[Inventory]
    A --> C1[(Session Cache)]
    A --> D1[(User DB)]
    P --> C2[(Product Cache)]
    P --> D2[(Catalog DB)]
    R --> M[ML Ranker]
    R --> D3[(User Events)]
    I --> D4[(Stock DB)]
    M --> F[Feature Store]
```

Now do the math. Suppose every service on that path has a healthy P99 of one second - meaning 99% of calls are faster than a second. You might reasonably say each service is performing well.

If the user-facing request depends on 100 independent sub-requests, the probability that *none* of them land in their slow tail is $0.99^{100}$, which is roughly 0.37. In other words, there is a **63% chance** that the user waits at least a second on a sub-request, even though every single service individually meets its SLO.

That number is the entire argument for P99.

```mermaid Even when every individual service meets its P99 SLO, the probability of a tail hit climbs steeply with fan-out depth. At 100 sub-requests the composite response is more likely than not to catch a slow one.
xychart-beta
    title "Probability of a Tail Hit vs Fan-Out Depth (P99 per service)"
    x-axis ["1 svc", "5 svcs", "10 svcs", "25 svcs", "50 svcs", "100 svcs"]
    y-axis "P(at least one slow sub-request) %" 0 --> 100
    bar [1, 5, 10, 22, 40, 63]
```

The individual P99s look fine. The *composition* of them is catastrophic. As soon as you fan out, rare events stop being rare — they become the dominant user experience. A tail that you could ignore at the component level becomes the median experience at the system level.

This is the phenomenon Dean and Barroso called "the tail at scale." It is the uncomfortable reason averages lie. Averages smooth; composition amplifies.

## The Money Follows The Tail

The other reason engineers fixate on P99 is that the people paying for the system care about it even when they do not know the acronym.

The numbers from industry have been cited so often they are almost folklore at this point, but the folklore happens to be true:

- Amazon's internal studies have reported that every additional 100ms of latency costs about **1% of sales**.
- Google has reported that a 400ms increase in search latency produces about a **0.59% drop in searches per user**.
- User research has consistently found that once page load times exceed a few seconds, abandonment climbs sharply.

Notice what these numbers are actually measuring. They are not measuring the average response time. They are measuring what happens when a user's request lands in the slow part of the distribution. The users who feel "the internet is slow today" are the users whose requests took 2 seconds instead of 200ms. That is a P99 problem, not a P50 problem.

When the business tells you "the site feels slow," they are telling you about the tail, even if they are looking at a graph of averages. If you chase the average down from 300ms to 250ms but leave the P99 at 4 seconds, your users will not notice the improvement. The users who cared were the ones in the tail, and they are still there.

## P99 Is An Early Warning

There is a second reason to watch P99 that has nothing to do with user experience: it is often the first place failures announce themselves.

A healthy system runs with spare capacity. When capacity is comfortable, a slow GC pause here or a head-of-line blocking moment there gets absorbed. The median does not move. The P99 twitches, briefly, and settles.

As the system creeps closer to its saturation point, the twitches get bigger and more frequent. Queues build up behind transient slowdowns. Threads that would have freed quickly now stay held. The *body* of the distribution still looks fine - most requests are completing - but the tail is getting longer. P99 starts drifting up while P50 stays flat.

```mermaid The path from a benign P99 twitch to a full outage. You see it on the P99 graph minutes or hours before it shows up on the P50.
flowchart LR
    A[Minor GC pause] --> B[Request slows]
    B --> C[Thread held longer]
    C --> D[Queue builds]
    D --> E[More requests slow]
    E --> F[More retries]
    F --> G[Queue saturates]
    G --> H[Cascading failure]
    style H fill:#3a1a1a,stroke:#7a2a2a
```

```mermaid P50 holds flat for most of the degradation window while P99 climbs steadily. The widening gap is the early warning signal. By the time P50 moves, the incident has already started.
xychart-beta
    title "P50 vs P99 as System Approaches Saturation"
    x-axis ["t-60m", "t-45m", "t-30m", "t-15m", "t-5m", "now"]
    y-axis "Latency (ms)" 0 --> 2000
    line [55, 52, 58, 60, 62, 65]
    line [130, 140, 280, 650, 1300, 1900]
```

By the time the median moves, you are past warning and into incident. By the time the average moves, you are in the post-mortem.

This is the operational case for watching P99 even when the business case is not obvious. The 99th percentile is the canary. The median is the corpse.

## Why Not P95? Why Not P999?

A fair question at this point is: if we are chasing the tail, why stop at P99? Why not P99.9, or P99.99?

The honest answer is: it depends on scale, and on what you can afford to care about.

At low traffic, P99 is already measuring something rare. At a thousand requests per second, P99 is ten requests a second in the slow bucket. P99.9 is one. P99.99 is one every ten seconds. You cannot meaningfully alert on a percentile that fires once per hour.

At high traffic - the Googles, Metas, Cloudflares of the world - the math flips. A thousand requests per second is a slow minute. Their user-facing systems operate at millions of requests per second, and P99.9 or P99.99 becomes the right level of resolution, because "one in a thousand" is still thousands of users per minute.

P95 exists for the opposite reason. At small scales, or for systems where 5% of slow requests is acceptable, P95 is a cheaper number to keep green. It is forgiving in a way P99 is not.

My rule of thumb is: the percentile you should target is the one where the count of "slow requests per unit time" is high enough to notice and low enough to fix. For most services I have worked on, that has landed on P99. For systems operating at web scale, it is usually P99.9. The principle is the same either way: *you are targeting the tail, not the center.*

```mermaid At 10,000 requests per second, P95 fires 500 times a second -- too noisy to act on. P99.99 fires once every ten seconds -- too rare to alert on. P99 sits in the actionable middle.
xychart-beta
    title "Slow Requests per Second at 10K rps by Percentile"
    x-axis ["P95", "P99", "P99.9", "P99.99"]
    y-axis "Slow req/sec" 0 --> 550
    bar [500, 100, 10, 1]
```

{/* subscribe-cta */}

## What Watching P99 Actually Changes

Once you take P99 seriously, several engineering practices stop looking optional.

**Timeouts become budgets, not upper bounds.** If the user-facing request budget is 1 second and you fan out to five sub-requests, each sub-request gets a fraction of that budget - not the full second. Sub-request timeouts that equal the full user budget are a way of guaranteeing that P99 at the leaf becomes P99 at the root.

**Hedged requests and request duplication become real tools.** The Dean and Barroso paper's most practical advice is to send a duplicate request if the first one has not returned by, say, P95. The second request gives you another independent chance to avoid the slow tail. The cost is a small amount of extra load; the benefit is that your user-visible P99 becomes close to your server-side P95.

**Load shedding starts to make sense.** If slow requests are the enemy, a service that rejects 1% of requests cleanly under overload is better than a service that queues them and serves all of them slowly. The first service has a controlled error rate and a healthy P99. The second has a beautiful success rate and a terrifying one.

**Observability shifts from averages to histograms.** An average cannot be decomposed. A histogram can. Once you store full latency histograms - not just means and maxes - you can compute any percentile you want, slice by endpoint, and compare tail behavior across deploys. Most reliability work I have done in the last few years started with "can we actually see the distribution?" and the answer was uncomfortably often "no."

## The Mental Model That Stuck

The shift that took me a long time to make is this: I used to think of a service as having a single "speed." The average, the typical response, the thing I would quote if someone asked. A service was "fast" or "slow."

A service does not have a single speed. It has a distribution. Every single request samples from that distribution independently. At the scale of one user, the sample is the experience. At the scale of many users fanning out across many services, the joint probability of every sample avoiding the tail collapses fast.

P99 is a shorthand for reasoning about the distribution's right edge - the place where the business loses money, where the operator loses sleep, and where the system shows you what is about to break before it actually breaks.

## The Short Version

Averages are a story about a system that does not exist. P50 is a story about the user who did not have a problem. P99 is a story about the users who did.

Once you fan out, the tail eats you. Once you are at scale, the tail is the product. Once you are in an incident, the tail told you first.

That is why every serious latency dashboard, at every company I have ever respected the engineering of, has P99 on it in large type. Not because it is a best practice. Because everything else, sooner or later, lies.

## References

- Dean, J., & Barroso, L. A. (2013). [The Tail at Scale](https://doi.org/10.1145/2408776.2408794). *Communications of the ACM*, 56(2), 74–80.
- Puzhavakath Narayanan, S., et al. (2016). Reducing latency through page-aware management of web objects by content delivery networks. *ACM SIGMETRICS Performance Evaluation Review*, 44(1), 89–100.
- Google Research. (2015). [User Preference and Search Engine Latency](https://research.google.com/pubs/archive/34439.pdf).
- Zhao, Z., et al. (2025). SLOpt: Serving real-time inference pipeline with strict latency constraint. *IEEE Transactions on Computers*.


Why Engineers Are Obsessed With P99


There's a failure mode that will ruin your week. Not because it's loud, it won't be. Your services stay up. Metrics look mostly reasonable. Alerts don't fire immediately. And yet your system is producing no useful output. Users are getting errors. Revenue is going to zero. And your dashboards are telling you everything is fine.

This is a **meta-stable failure**. It's the failure mode that keeps distributed systems engineers up at night precisely because it doesn't look like a failure.

This is first time i heard the term "**meta-stable failure**.", it spark curiosity in me and dig down deeper.

## Up, But Down

The formal definition, from a 2021 HotOS paper co-authored by engineers at AWS, Meta, and others: *"Metastable failures occur in open systems with an uncontrolled source of load where a trigger causes the system to enter a bad state that persists even when the trigger is removed."*

Read that last clause again. *Even when the trigger is removed.*

This is what separates meta-stable failures from ordinary outages. In a normal incident, you find the cause, fix it, and the system recovers. In a meta-stable failure, you fix the cause and the system stays broken. The failure has become self-sustaining. You're not fighting the original trigger anymore, you're fighting the system's own behavior.

## The Sustaining Loop

To understand why, you need to understand what's actually happening mechanically.

Imagine a backend service under normal load. A routine deployment causes a brief latency spike. Clients, following best practices, retry their failed requests. But those retries arrive while the service is already struggling, which increases load further. The higher load causes more failures. More failures cause more retries. More retries cause more load.

The original trigger, the deployment, is long gone. The latency spike from the deploy resolved in 30 seconds. But the feedback loop it kicked off is still running, hours later, at full intensity. Your service is extremely busy processing retries for requests it will never successfully complete. It has high throughput. It has near-zero *goodput*, useful work delivered to users.

The system is "up." In a meaningful sense, it's completely down.

## Why Good Engineering Made It Worse

Here's the uncomfortable part. Every mechanism in that loop was put there deliberately, by thoughtful engineers, for good reasons.

**Retries** exist because transient failures are real. Networks drop packets. A request that fails once often succeeds on the second attempt. Without retries, users see errors from hiccups that the system would have handled fine with one more chance.

**Timeouts** exist because hanging requests are worse than failed ones. A request that waits forever holds resources, threads, connections, memory, that block other requests. Timeouts free those resources so the system can keep serving other traffic.

**Backoff** is supposed to prevent retry storms, and in mild overload scenarios it works well. But in deep overload, even exponential backoff with jitter doesn't reduce enough load to allow recovery. The system is so far from its recovery threshold that even well-behaved clients can't get it there.

Each of these was the right decision in isolation. The meta-stable failure emerges from their interaction under specific conditions.

## The Trigger Is Not The Problem

This is the key insight that changes how you think about these failures, and the one that gets buried when teams do post-mortems.

A meta-stable failure has two distinct phases: a **trigger** that pushes the system into the bad state, and a **sustaining mechanism** that keeps it there. Engineers almost always focus their post-mortem on the trigger, the deployment, the traffic spike, the upstream timeout. They fix it and declare the incident closed.

But the trigger is almost incidental. A different trigger, a thundering herd, a cache invalidation, a brief network partition, would have caused exactly the same outcome. The system has a stability property that makes it vulnerable: under certain load conditions, a feedback loop can lock it into a degraded state it cannot escape from on its own.

Fix the trigger and you've prevented *this* incident. Fix the sustaining mechanism and you've changed the system's fundamental failure behavior.

## Four Places Sustaining Loops Hide

The HotOS paper identifies common sources. These aren't exotic, they're patterns you've almost certainly shipped.

**Retry logic without load shedding.** Retries increase load during overload. If your service doesn't have a mechanism to start rejecting work, actively, deliberately, with a 503, it has no way to reduce its own load. The retry traffic fills every available slot. The only escape is an operator manually cutting traffic.

**Caching with thundering herds.** A cache miss causes a backend query. Under normal load, this is fine, misses are occasional and the backend handles them. But a cache eviction or flush can cause every node to miss simultaneously, sending a spike to the backend that exceeds its capacity, causing timeouts, causing retries, filling the queue. The cache can't repopulate if the backend can't respond, so the thundering herd sustains itself.

**Slow error paths.** Your fast path is highly optimized. Your error path, the code that runs when something goes wrong, was written once and never touched again. If error handling is significantly slower than normal processing, failures consume more resources per request than successes. At high error rates, this means the system is burning capacity on work that produces no value. The more it fails, the less capacity it has to succeed.

**Autoscaling with lag.** Cloud autoscaling is not instantaneous. There's a window, sometimes several minutes, between when load exceeds capacity and when new instances are ready. During that window, if the load is sustained by a feedback loop, the existing instances may saturate before help arrives. New instances that join an already-overloaded cluster can get immediately hammered and fail to start up successfully at all.

## What Recovery Actually Looks Like

If you're in a meta-stable failure and you remove the trigger, the system won't recover. It's locked in the bad state. You need to do something that breaks the sustaining loop.

The most common effective interventions:

**Reduce load below the recovery threshold.** If retries are sustaining the loop, stop the retries. Cut client traffic at the load balancer. Drop the offered load far enough that the service can start clearing its backlog. This feels wrong, you're making the user-visible failure more severe, but it's the only way to let the system breathe.

**Restart the sustaining mechanism.** If cache state is part of the loop, a controlled warm-up sequence (where you rebuild cache before allowing full traffic) can break the cycle. If connection pool exhaustion is sustaining the failure, recycling connections on the server side can free the system.

**Emergency throttling.** A hard rate limit, not a soft one, that doesn't grow under retry pressure. Circuit breakers that stay open long enough for backend queues to drain, not just long enough to let one request through.

{/* subscribe-cta */}

## How to Build Systems That Don't Get Trapped

The sustaining loop is the target, not the trigger. That reframes what resilience engineering actually means for this failure mode.

**Give your services a way to shed load.** Every service that accepts open-loop traffic needs a pressure valve. This means returning 503 when load exceeds a threshold, not queueing indefinitely. Queues that grow without bound under load are a meta-stable failure waiting to happen. They accept work the service will never complete, holding capacity that could be freed.

**Treat your error paths with the same care as your fast paths.** Profile them. Load test them. If your error handling is two orders of magnitude slower than normal processing, it will become a resource sink at exactly the wrong time.

**Design for controlled recovery.** The hardest operational moment in a meta-stable failure is the transition back to normal. Load needs to ramp up slowly, too fast, and the recovering system re-enters the bad state. Traffic ramp-up procedures, caching warm-up sequences, and connection pre-warming aren't nice-to-haves; they're the mechanism that prevents re-entry.

**Think in feedback loops, not component failures.** Most distributed systems reliability work treats failure as: component X failed, what is our tolerance for X failing? Meta-stable failures are a different animal. They emerge from the *interaction* of functioning components. The question isn't "what happens when X fails" but "what happens to the system's dynamics when load pushes past a threshold." Control theory, not traditional reliability engineering, is the right mental model.

## The Honest Thing About Post-Mortems

When you've been through a meta-stable failure and you're writing the post-mortem, the trigger makes for a satisfying story. There was a deploy. There was a traffic spike. Here's what we did wrong, here's the fix, here's the test to prevent it next time.

But if you stop there, you've documented a specific trigger while leaving the underlying failure mode intact. The next time, different trigger, same catastrophic loop, same hours of degradation.

The meta-stable failure asks a harder question: not "what went wrong" but "what property of this system makes it capable of getting trapped like this?" That question doesn't have a clean action item. It has architectural answers that take quarters to implement. It requires reasoning about your system's dynamics under load, not just its behavior under component failure.

That's the harder work. And it's the only work that actually changes the failure mode.

reference video:
https://www.youtube.com/watch?v=u3GjIXP9N0s

Meta-Stable Failure: When Your System Is Up But Completely Down


I watched a ten-minute video this week that rearranged the way I think about retries. It was [Arpit Bhayani's](https://www.youtube.com/watch?v=8sTuCPh3s0s) walk-through of the thundering herd problem - one of those talks that is short, unhurried, and pointed enough to leave a mark. I had written retry logic dozens of times. I had read the AWS blog post about jitter. I thought I understood it. I did not.

The thing that got me was not any single technical fact. It was the small realization that I had been holding two incompatible ideas in my head at the same time - "retries make systems more reliable" and "retries cause outages" - without ever forcing them into the same room.

This article is me forcing them into the same room.

## The Setup That Made It Click

Arpit opens with a scene that almost every backend engineer has lived through, even if they did not realize it. A service has a brief hiccup. Maybe the deploy took a few seconds longer than expected. Maybe a database failed over. Maybe a downstream dependency got slow. The clients, behaving exactly as their engineers intended, notice the failures and retry.

That is where the trouble starts.

Each client is a single, well-behaved program. None of them are misbehaving. None of them are attacking your service. They are doing the correct thing - retrying a failed request so that the end user sees success instead of an error. And yet, in aggregate, they form a wave of traffic that hits your already-struggling service at exactly the moment it least wants to see it.

The image Arpit uses is the one the name implies: a herd of animals, all waking up at the same moment and stampeding in the same direction. Individually, each one is just running. Collectively, they trample whatever is in the way.

That framing was new to me. I had always thought of the thundering herd as something that happened *to* a system. Arpit's point is that it is something the system's own clients produce, following instructions the system's own engineers wrote.

## Retries Are Not Free

The first insight I took from the video is one I had technically known but never really internalized: every retry is a request. That sounds obvious. It is not.

A retry feels, from the client's perspective, like a second chance. The first attempt failed, so we try again, and if the second succeeds we are whole. No harm done. But the server does not see it that way. The server sees two requests, both of which it must process, both of which consume CPU and memory and a connection slot, even if the first one failed quickly.

When you multiply that by thousands of clients, the math changes. A service running at 80% capacity can handle its load. The same service, faced with every client sending twice as many requests as normal, is running at 160% of its capacity - except capacity does not work like that. It does not degrade linearly. Past some threshold, everything gets slower, which causes more timeouts, which causes more retries.

I had read about this. I had not understood it as *my* retry logic's fault.

## Exponential Backoff, and the Trap Inside It

The standard answer to naive retries is exponential backoff. Wait one second, then two, then four, then eight. The rationale is clean: each retry gives the server more time to recover, and a client that keeps failing will exponentially reduce the load it offers.

This is what I had implemented, many times. I thought of it as the solved version of the problem.

What Arpit's video pointed out - and the thing I want to make sure lands, because this is the piece I had genuinely missed - is that exponential backoff alone does not solve the thundering herd. It only moves it.

Here is why. Suppose a thousand clients all hit a failure at the same moment. They all wait one second. They all retry at the same moment. They all fail. They all wait two seconds. They all retry at the same moment. They all fail. The waves of traffic are farther apart now, but each wave is still synchronized. The herd is still thundering - it is just thundering at regular intervals instead of continuously.

If your service cannot handle a thousand simultaneous requests, it does not matter whether those requests arrive every second or every minute. Each wave will saturate it. Each saturation will produce more failures. Each failure will trigger the next synchronized retry.

The quiet, uncomfortable insight is that well-behaved clients obeying well-written retry logic can still collectively DDoS the service they are trying to talk to.

## Jitter Is the Missing Piece

This is where jitter comes in, and why it is more than a small implementation detail.

Jitter means adding a random offset to the backoff delay. Instead of every client waiting exactly four seconds, each client waits somewhere between, say, two and six seconds. The distribution spreads the herd out in time. Instead of a wave, you get a stream. Instead of a thousand simultaneous retries, you get roughly a thousand retries scattered across a window of several seconds - which is exactly what the service can actually handle.

The service gets breathing room. The retries eventually succeed. The system recovers. Nobody notices.

What was interesting to me is how small this change is in code. It is literally one extra line - adding a random value to the sleep before retry. The effect on system behavior is enormous. The behavior of the whole system depends on a function call that looks, to anyone reading the code, like a minor detail.

That asymmetry is, I think, the reason this problem keeps happening in real-world systems. The fix is trivial to implement. Recognizing that you need it - that backoff alone is not enough - is the hard part.

## The Framing Shift

The part of the video that I keep coming back to is not any of the specific techniques. It is a framing shift in how I think about retries at all.

Before, I saw retries as a reliability mechanism - something I added to a client to protect users from transient failures. The mental model was: my client is defending itself against an unreliable network.

After, I see retries as a *source of load* - something I am producing that arrives at a server that may not be ready to receive it. The mental model is now closer to: my client is offering additional work to a service that may be in no state to accept it.

Both framings are true. But they lead to very different code. The first framing optimizes for "retry until success." The second framing optimizes for "retry in a way that does not make the failure worse."

I do not think I would have made that shift by reading a definition of the thundering herd problem. I had read definitions. What the video did was put it in a sequence - naive retry, then backoff, then backoff-plus-jitter - where each step visibly failed to solve the problem until the last one. Following the failure made the reasoning stick in a way that stating the conclusion did not.

{/* subscribe-cta */}

## What I'm Changing

Watching that video, I made a short list of things I wanted to audit in my own work. Writing them down here partly because the list is useful, and partly because I suspect other engineers have some version of the same list waiting to be written.

**Every retry loop I own needs jitter, not just backoff.** This is the one-line change. I checked a handful of services after watching the video and found backoff-without-jitter in more places than I want to admit. None of those services had caused incidents yet. That is not the same as being safe.

**Client-side retry policy is a system-design decision, not a defensive local habit.** If every team independently picks a retry policy for the calls *they* make, the aggregate behavior is not anyone's responsibility - which means it is nobody's. I want the retry defaults to live in a shared client, with jitter baked in, so that individual teams cannot accidentally skip it.

**Servers need to be robust to thundering herds from their own clients.** Load shedding, server-side rate limiting, and fast rejection with 429 responses are not about hostile traffic. They are about surviving the helpful traffic produced by your own well-meaning client retry loops. I used to think of those mechanisms as paranoia. Now I think of them as the other half of the retry story.

**The absence of an incident is not evidence of safety.** Thundering herd failures are a threshold effect. The system is fine up to some load, and then it is not. The transition is sharp. You do not get a gentle warning. You get a clean graph and then a vertical line.

## The Short Version

If you retry without backoff, you make failures worse. If you back off without jitter, you delay the problem instead of solving it. If you back off with jitter, you give the service the one thing it needs to recover: uncorrelated traffic.

That is the entire story. It takes ten minutes to explain on video, about thirty seconds to implement in code, and - for a lot of us, me included - far too many years to actually internalize.

If you have not watched the video, [watch it](https://www.youtube.com/watch?v=8sTuCPh3s0s). If you have written retry logic recently, open that file and check for the jitter. It is almost always faster to fix now than to debug at 3 AM when a brief upstream hiccup turns into a multi-hour incident because every client in your fleet decided to retry on exactly the same second.


Thundering Herd: What a 10-Minute Video Taught Me About Retries


There is a class of distributed systems bug that appears only under load, only in production, and always at the worst possible moment. It looks like a race condition. It shows up in your Stripe dashboard as a duplicate charge. Your on-call engineer escalates it at 3am, and by the time the postmortem is written, three engineers have burned a weekend and a customer has filed a chargeback.

The pattern that prevents it has a name. It is called idempotency. And despite being the foundational primitive for safe distributed operations, it is consistently the last thing teams think about when designing new endpoints.

This essay is about why idempotency matters, what it actually costs to implement correctly, and when the pattern is worth every bit of the investment.

## What Idempotency Actually Means

An operation is idempotent if applying it more than once produces the same result as applying it once. This sounds obvious until you try to design a payment API. Charging a card is not naturally idempotent - if you call `POST /charges` twice with the same payload, you get two charges. Networks time out. Clients retry. Servers crash mid-write. The gap between "the server received the request" and "the client received the acknowledgment" is where money disappears.

The classic solution is the idempotency key - a client-generated token, typically a UUID, that the server uses to deduplicate requests within a time window.

```typescript
// Client sends this header with every mutating request
const charge = await stripe.charges.create({
  amount: 2000,
  currency: "usd",
  source: "tok_visa",
}, {
  idempotencyKey: "order_abc123_attempt_1",
});
```

If the network drops and the client retries, Stripe detects the same key and returns the original response rather than creating a new charge. The client cannot distinguish a first execution from a replay - which is exactly the point.

But the simplicity of the interface hides considerable complexity on the server side.

## The Three Layers of Idempotency

Implementing idempotency correctly requires thinking at three distinct layers, and most teams only address one or two of them.

**Layer 1: The deduplication store.** You need somewhere to record that you've seen a key and what the result was. This is usually a fast key-value store - Redis, DynamoDB, or a dedicated column in your Postgres transactions table. The entry must be written atomically with the underlying operation, or you have a new race condition: two concurrent requests with the same key, both checking before either has written.

```typescript
async function createCharge(
  idempotencyKey: string,
  params: ChargeParams
): Promise<Charge> {
  // Atomic check-and-set using a database transaction
  return await db.transaction(async (tx) => {
    const existing = await tx.idempotencyKeys.findUnique({
      where: { key: idempotencyKey },
    });

    if (existing) {
      // Return the stored result - not a new execution
      return JSON.parse(existing.responseBody) as Charge;
    }

    // Execute the actual operation
    const charge = await processCharge(params);

    // Record the key and result atomically
    await tx.idempotencyKeys.create({
      data: {
        key: idempotencyKey,
        responseBody: JSON.stringify(charge),
        expiresAt: addDays(new Date(), 7),
      },
    });

    return charge;
  });
}
```

**Layer 2: Request fingerprinting.** Idempotency keys only work if you tie them to a specific request shape. If a client sends the same key with different parameters - say, the same `idempotencyKey` but a different `amount` - you have to decide: honor the key and ignore the parameter mismatch, or reject the request with a 422? Stripe rejects. That is the correct answer. A mismatched fingerprint means the client has a bug, and surfacing that error immediately is better than silently ignoring the mismatch.

**Layer 3: Side effects beyond your database.** Your charge endpoint probably sends an email confirmation, increments a counter in a metrics system, and triggers a webhook. None of those systems have your idempotency guarantee. You need to ensure that downstream side effects are either naturally idempotent (incrementing a monotonic counter with a version check) or wrapped in their own deduplication logic. Idempotency at the API layer does not automatically propagate to async workers.

<PullQuote attribution={{ name: "Werner Vogels", role: "CTO, Amazon" }}>
Everything fails, all the time. Design for it.
</PullQuote>

## When the Pattern Pays Off

Idempotency is not free. The deduplication store adds a round trip. The fingerprinting logic adds complexity. The key expiration policy adds a new class of edge case - what happens when a key expires and the client retries six days later?

So when is it worth the cost?

The answer is almost always "yes" for any operation that changes state and could be retried by an upstream caller. In practice, this means:

- Payment and billing operations
- Order creation
- Message sends (email, SMS, push notifications)
- Provisioning actions (create VM, allocate resource)
- Any operation that touches external systems you do not control

The operations where idempotency is less critical are read operations (which are already naturally idempotent) and internal operations that are naturally safe to repeat. But even there, building the habit of asking "what happens if this runs twice?" is worth the mental overhead.

## The Subtle Failure Mode Nobody Talks About

There is a failure mode that trips up even experienced engineers: partial completion.

Your charge endpoint writes to the database, then calls the payment processor, then sends the confirmation email. The payment processor call succeeds, but the database write fails on the way back (a transient network partition, a rollback triggered by a constraint violation). The customer's card has been charged. Your database does not know about it. When the client retries with the same idempotency key, your deduplication store has nothing - and you charge the card again.

The solution is to write the idempotency key entry *before* you start executing the operation, not after. Set its status to `pending`, execute the operation, then update the status to `completed` with the stored result. If the system crashes mid-execution, subsequent attempts will find the `pending` entry and can either wait (if the operation might still complete) or attempt recovery.

This is the pattern Stripe describes in their engineering blog. It is sometimes called the "idempotency receipt" pattern, and it is the detail that separates robust implementations from ones that fail under concurrent retries.

## What This Means for Your API Design

If you are designing a new service today, these are the decisions that matter:

**Use client-generated keys, not server-generated.** Servers cannot create idempotency keys because they cannot know whether a request is a retry. Only the client knows that. The client generates a UUID, stores it locally, and sends it with every retry attempt until it receives a definitive response.

**Be explicit about key scope.** Does your key apply across all operations or just a specific endpoint? Stripe scopes keys to the calling API key, which prevents one integration from colliding with another. Define your scope explicitly in your documentation.

**Choose a reasonable expiration window.** Seven days is a common choice. Long enough to cover any realistic retry window, short enough that your deduplication store does not grow unbounded. Paginate your key cleanup job.

**Communicate idempotency status to callers.** Return a header like `Idempotent-Replayed: true` when you return a cached response. This helps clients distinguish "the operation ran" from "the operation ran and I'm giving you the result from last time." It also helps your debugging story: a duplicate alert with `Idempotent-Replayed: true` is a client bug, not a server bug.

## The Real Cost of Not Implementing It

The cost of skipping idempotency is not just duplicate charges. It is the cumulative technical debt that accumulates when you build the wrong primitive into your foundation. Every endpoint you add without idempotency is an endpoint that needs to be retrofitted when you start handling retries. Every integration that assumes single delivery is an integration that will behave incorrectly under network partitions.

The engineers who understand this build retry logic correctly from day one. Their systems degrade gracefully. Their on-call rotations are quieter. Their postmortems are shorter.

Idempotency is not glamorous. It does not make it into architecture talks at conferences. But it is one of the clearest examples of a pattern that pays for itself many times over - in the operational incidents it prevents, the customer trust it preserves, and the engineering time it does not consume.

The pattern worth paying for is the one you never have to think about again.

---

*If this resonated, the follow-up is about exactly-once delivery - what it means, what it costs, and why distributed systems practitioners generally settle for at-least-once with idempotency instead.*


Essaystaggedreliability

Why Engineers Are Obsessed With P99

Meta-Stable Failure: When Your System Is Up But Completely Down

Thundering Herd: What a 10-Minute Video Taught Me About Retries

The Pattern Worth Paying For