
There's a failure mode that will ruin your week. Not because it's loud, it won't be. Your services stay up. Metrics look mostly reasonable. Alerts don't fire immediately. And yet your system is producing no useful output. Users are getting errors. Revenue is going to zero. And your dashboards are telling you everything is fine.

This is a **meta-stable failure**. It's the failure mode that keeps distributed systems engineers up at night precisely because it doesn't look like a failure.

This is first time i heard the term "**meta-stable failure**.", it spark curiosity in me and dig down deeper.

## Up, But Down

The formal definition, from a 2021 HotOS paper co-authored by engineers at AWS, Meta, and others: *"Metastable failures occur in open systems with an uncontrolled source of load where a trigger causes the system to enter a bad state that persists even when the trigger is removed."*

Read that last clause again. *Even when the trigger is removed.*

This is what separates meta-stable failures from ordinary outages. In a normal incident, you find the cause, fix it, and the system recovers. In a meta-stable failure, you fix the cause and the system stays broken. The failure has become self-sustaining. You're not fighting the original trigger anymore, you're fighting the system's own behavior.

## The Sustaining Loop

To understand why, you need to understand what's actually happening mechanically.

Imagine a backend service under normal load. A routine deployment causes a brief latency spike. Clients, following best practices, retry their failed requests. But those retries arrive while the service is already struggling, which increases load further. The higher load causes more failures. More failures cause more retries. More retries cause more load.

The original trigger, the deployment, is long gone. The latency spike from the deploy resolved in 30 seconds. But the feedback loop it kicked off is still running, hours later, at full intensity. Your service is extremely busy processing retries for requests it will never successfully complete. It has high throughput. It has near-zero *goodput*, useful work delivered to users.

The system is "up." In a meaningful sense, it's completely down.

## Why Good Engineering Made It Worse

Here's the uncomfortable part. Every mechanism in that loop was put there deliberately, by thoughtful engineers, for good reasons.

**Retries** exist because transient failures are real. Networks drop packets. A request that fails once often succeeds on the second attempt. Without retries, users see errors from hiccups that the system would have handled fine with one more chance.

**Timeouts** exist because hanging requests are worse than failed ones. A request that waits forever holds resources, threads, connections, memory, that block other requests. Timeouts free those resources so the system can keep serving other traffic.

**Backoff** is supposed to prevent retry storms, and in mild overload scenarios it works well. But in deep overload, even exponential backoff with jitter doesn't reduce enough load to allow recovery. The system is so far from its recovery threshold that even well-behaved clients can't get it there.

Each of these was the right decision in isolation. The meta-stable failure emerges from their interaction under specific conditions.

## The Trigger Is Not The Problem

This is the key insight that changes how you think about these failures, and the one that gets buried when teams do post-mortems.

A meta-stable failure has two distinct phases: a **trigger** that pushes the system into the bad state, and a **sustaining mechanism** that keeps it there. Engineers almost always focus their post-mortem on the trigger, the deployment, the traffic spike, the upstream timeout. They fix it and declare the incident closed.

But the trigger is almost incidental. A different trigger, a thundering herd, a cache invalidation, a brief network partition, would have caused exactly the same outcome. The system has a stability property that makes it vulnerable: under certain load conditions, a feedback loop can lock it into a degraded state it cannot escape from on its own.

Fix the trigger and you've prevented *this* incident. Fix the sustaining mechanism and you've changed the system's fundamental failure behavior.

## Four Places Sustaining Loops Hide

The HotOS paper identifies common sources. These aren't exotic, they're patterns you've almost certainly shipped.

**Retry logic without load shedding.** Retries increase load during overload. If your service doesn't have a mechanism to start rejecting work, actively, deliberately, with a 503, it has no way to reduce its own load. The retry traffic fills every available slot. The only escape is an operator manually cutting traffic.

**Caching with thundering herds.** A cache miss causes a backend query. Under normal load, this is fine, misses are occasional and the backend handles them. But a cache eviction or flush can cause every node to miss simultaneously, sending a spike to the backend that exceeds its capacity, causing timeouts, causing retries, filling the queue. The cache can't repopulate if the backend can't respond, so the thundering herd sustains itself.

**Slow error paths.** Your fast path is highly optimized. Your error path, the code that runs when something goes wrong, was written once and never touched again. If error handling is significantly slower than normal processing, failures consume more resources per request than successes. At high error rates, this means the system is burning capacity on work that produces no value. The more it fails, the less capacity it has to succeed.

**Autoscaling with lag.** Cloud autoscaling is not instantaneous. There's a window, sometimes several minutes, between when load exceeds capacity and when new instances are ready. During that window, if the load is sustained by a feedback loop, the existing instances may saturate before help arrives. New instances that join an already-overloaded cluster can get immediately hammered and fail to start up successfully at all.

## What Recovery Actually Looks Like

If you're in a meta-stable failure and you remove the trigger, the system won't recover. It's locked in the bad state. You need to do something that breaks the sustaining loop.

The most common effective interventions:

**Reduce load below the recovery threshold.** If retries are sustaining the loop, stop the retries. Cut client traffic at the load balancer. Drop the offered load far enough that the service can start clearing its backlog. This feels wrong, you're making the user-visible failure more severe, but it's the only way to let the system breathe.

**Restart the sustaining mechanism.** If cache state is part of the loop, a controlled warm-up sequence (where you rebuild cache before allowing full traffic) can break the cycle. If connection pool exhaustion is sustaining the failure, recycling connections on the server side can free the system.

**Emergency throttling.** A hard rate limit, not a soft one, that doesn't grow under retry pressure. Circuit breakers that stay open long enough for backend queues to drain, not just long enough to let one request through.

{/* subscribe-cta */}

## How to Build Systems That Don't Get Trapped

The sustaining loop is the target, not the trigger. That reframes what resilience engineering actually means for this failure mode.

**Give your services a way to shed load.** Every service that accepts open-loop traffic needs a pressure valve. This means returning 503 when load exceeds a threshold, not queueing indefinitely. Queues that grow without bound under load are a meta-stable failure waiting to happen. They accept work the service will never complete, holding capacity that could be freed.

**Treat your error paths with the same care as your fast paths.** Profile them. Load test them. If your error handling is two orders of magnitude slower than normal processing, it will become a resource sink at exactly the wrong time.

**Design for controlled recovery.** The hardest operational moment in a meta-stable failure is the transition back to normal. Load needs to ramp up slowly, too fast, and the recovering system re-enters the bad state. Traffic ramp-up procedures, caching warm-up sequences, and connection pre-warming aren't nice-to-haves; they're the mechanism that prevents re-entry.

**Think in feedback loops, not component failures.** Most distributed systems reliability work treats failure as: component X failed, what is our tolerance for X failing? Meta-stable failures are a different animal. They emerge from the *interaction* of functioning components. The question isn't "what happens when X fails" but "what happens to the system's dynamics when load pushes past a threshold." Control theory, not traditional reliability engineering, is the right mental model.

## The Honest Thing About Post-Mortems

When you've been through a meta-stable failure and you're writing the post-mortem, the trigger makes for a satisfying story. There was a deploy. There was a traffic spike. Here's what we did wrong, here's the fix, here's the test to prevent it next time.

But if you stop there, you've documented a specific trigger while leaving the underlying failure mode intact. The next time, different trigger, same catastrophic loop, same hours of degradation.

The meta-stable failure asks a harder question: not "what went wrong" but "what property of this system makes it capable of getting trapped like this?" That question doesn't have a clean action item. It has architectural answers that take quarters to implement. It requires reasoning about your system's dynamics under load, not just its behavior under component failure.

That's the harder work. And it's the only work that actually changes the failure mode.

reference video:
https://www.youtube.com/watch?v=u3GjIXP9N0s

Meta-Stable Failure: When Your System Is Up But Completely Down


I watched a ten-minute video this week that rearranged the way I think about retries. It was [Arpit Bhayani's](https://www.youtube.com/watch?v=8sTuCPh3s0s) walk-through of the thundering herd problem - one of those talks that is short, unhurried, and pointed enough to leave a mark. I had written retry logic dozens of times. I had read the AWS blog post about jitter. I thought I understood it. I did not.

The thing that got me was not any single technical fact. It was the small realization that I had been holding two incompatible ideas in my head at the same time - "retries make systems more reliable" and "retries cause outages" - without ever forcing them into the same room.

This article is me forcing them into the same room.

## The Setup That Made It Click

Arpit opens with a scene that almost every backend engineer has lived through, even if they did not realize it. A service has a brief hiccup. Maybe the deploy took a few seconds longer than expected. Maybe a database failed over. Maybe a downstream dependency got slow. The clients, behaving exactly as their engineers intended, notice the failures and retry.

That is where the trouble starts.

Each client is a single, well-behaved program. None of them are misbehaving. None of them are attacking your service. They are doing the correct thing - retrying a failed request so that the end user sees success instead of an error. And yet, in aggregate, they form a wave of traffic that hits your already-struggling service at exactly the moment it least wants to see it.

The image Arpit uses is the one the name implies: a herd of animals, all waking up at the same moment and stampeding in the same direction. Individually, each one is just running. Collectively, they trample whatever is in the way.

That framing was new to me. I had always thought of the thundering herd as something that happened *to* a system. Arpit's point is that it is something the system's own clients produce, following instructions the system's own engineers wrote.

## Retries Are Not Free

The first insight I took from the video is one I had technically known but never really internalized: every retry is a request. That sounds obvious. It is not.

A retry feels, from the client's perspective, like a second chance. The first attempt failed, so we try again, and if the second succeeds we are whole. No harm done. But the server does not see it that way. The server sees two requests, both of which it must process, both of which consume CPU and memory and a connection slot, even if the first one failed quickly.

When you multiply that by thousands of clients, the math changes. A service running at 80% capacity can handle its load. The same service, faced with every client sending twice as many requests as normal, is running at 160% of its capacity - except capacity does not work like that. It does not degrade linearly. Past some threshold, everything gets slower, which causes more timeouts, which causes more retries.

I had read about this. I had not understood it as *my* retry logic's fault.

## Exponential Backoff, and the Trap Inside It

The standard answer to naive retries is exponential backoff. Wait one second, then two, then four, then eight. The rationale is clean: each retry gives the server more time to recover, and a client that keeps failing will exponentially reduce the load it offers.

This is what I had implemented, many times. I thought of it as the solved version of the problem.

What Arpit's video pointed out - and the thing I want to make sure lands, because this is the piece I had genuinely missed - is that exponential backoff alone does not solve the thundering herd. It only moves it.

Here is why. Suppose a thousand clients all hit a failure at the same moment. They all wait one second. They all retry at the same moment. They all fail. They all wait two seconds. They all retry at the same moment. They all fail. The waves of traffic are farther apart now, but each wave is still synchronized. The herd is still thundering - it is just thundering at regular intervals instead of continuously.

If your service cannot handle a thousand simultaneous requests, it does not matter whether those requests arrive every second or every minute. Each wave will saturate it. Each saturation will produce more failures. Each failure will trigger the next synchronized retry.

The quiet, uncomfortable insight is that well-behaved clients obeying well-written retry logic can still collectively DDoS the service they are trying to talk to.

## Jitter Is the Missing Piece

This is where jitter comes in, and why it is more than a small implementation detail.

Jitter means adding a random offset to the backoff delay. Instead of every client waiting exactly four seconds, each client waits somewhere between, say, two and six seconds. The distribution spreads the herd out in time. Instead of a wave, you get a stream. Instead of a thousand simultaneous retries, you get roughly a thousand retries scattered across a window of several seconds - which is exactly what the service can actually handle.

The service gets breathing room. The retries eventually succeed. The system recovers. Nobody notices.

What was interesting to me is how small this change is in code. It is literally one extra line - adding a random value to the sleep before retry. The effect on system behavior is enormous. The behavior of the whole system depends on a function call that looks, to anyone reading the code, like a minor detail.

That asymmetry is, I think, the reason this problem keeps happening in real-world systems. The fix is trivial to implement. Recognizing that you need it - that backoff alone is not enough - is the hard part.

## The Framing Shift

The part of the video that I keep coming back to is not any of the specific techniques. It is a framing shift in how I think about retries at all.

Before, I saw retries as a reliability mechanism - something I added to a client to protect users from transient failures. The mental model was: my client is defending itself against an unreliable network.

After, I see retries as a *source of load* - something I am producing that arrives at a server that may not be ready to receive it. The mental model is now closer to: my client is offering additional work to a service that may be in no state to accept it.

Both framings are true. But they lead to very different code. The first framing optimizes for "retry until success." The second framing optimizes for "retry in a way that does not make the failure worse."

I do not think I would have made that shift by reading a definition of the thundering herd problem. I had read definitions. What the video did was put it in a sequence - naive retry, then backoff, then backoff-plus-jitter - where each step visibly failed to solve the problem until the last one. Following the failure made the reasoning stick in a way that stating the conclusion did not.

{/* subscribe-cta */}

## What I'm Changing

Watching that video, I made a short list of things I wanted to audit in my own work. Writing them down here partly because the list is useful, and partly because I suspect other engineers have some version of the same list waiting to be written.

**Every retry loop I own needs jitter, not just backoff.** This is the one-line change. I checked a handful of services after watching the video and found backoff-without-jitter in more places than I want to admit. None of those services had caused incidents yet. That is not the same as being safe.

**Client-side retry policy is a system-design decision, not a defensive local habit.** If every team independently picks a retry policy for the calls *they* make, the aggregate behavior is not anyone's responsibility - which means it is nobody's. I want the retry defaults to live in a shared client, with jitter baked in, so that individual teams cannot accidentally skip it.

**Servers need to be robust to thundering herds from their own clients.** Load shedding, server-side rate limiting, and fast rejection with 429 responses are not about hostile traffic. They are about surviving the helpful traffic produced by your own well-meaning client retry loops. I used to think of those mechanisms as paranoia. Now I think of them as the other half of the retry story.

**The absence of an incident is not evidence of safety.** Thundering herd failures are a threshold effect. The system is fine up to some load, and then it is not. The transition is sharp. You do not get a gentle warning. You get a clean graph and then a vertical line.

## The Short Version

If you retry without backoff, you make failures worse. If you back off without jitter, you delay the problem instead of solving it. If you back off with jitter, you give the service the one thing it needs to recover: uncorrelated traffic.

That is the entire story. It takes ten minutes to explain on video, about thirty seconds to implement in code, and - for a lot of us, me included - far too many years to actually internalize.

If you have not watched the video, [watch it](https://www.youtube.com/watch?v=8sTuCPh3s0s). If you have written retry logic recently, open that file and check for the jitter. It is almost always faster to fix now than to debug at 3 AM when a brief upstream hiccup turns into a multi-hour incident because every client in your fleet decided to retry on exactly the same second.


Essaystaggedresilience

Meta-Stable Failure: When Your System Is Up But Completely Down

Thundering Herd: What a 10-Minute Video Taught Me About Retries