
I have been saying the words "we'll just fire an event" in design reviews for years. The words come out cleanly. The underlying picture in my head, if I am honest, has always been a little blurry. Is the event fired inside the database transaction, or after? Who consumes it? If the broker is down for thirty seconds, does anything actually break?

This week I stumbled on a [ByteByteGo infographic](https://blog.bytebytego.com/) that lined up five of these patterns side by side - Transactional Outbox, CQRS, CQRS with Event Sourcing, Saga, Competing Consumers - and I realized I could recite all of them and confidently describe maybe two.

So I did the thing that always works for me when I half-understand something. I sat down and drew each one until I could explain, in plain language, what it is, what it is for, and when I would actually reach for it. This article is that exercise, written out.

## Why These Patterns Exist At All

Message brokers - Kafka, RabbitMQ, SQS, NATS, Pulsar, pick your favorite - are plumbing. They move messages from producers to consumers without either side having to know about the other. That is the whole value proposition, and it is a big one.

But the moment you put a broker in the middle of a system, new questions appear that a single-database design never had to answer:

- What if my database commit succeeds but the broker publish fails?
- What if two services need the same data but want completely different shapes of it?
- What if a "transaction" has to span four services that each own their own database?
- What if one producer emits faster than a single consumer can process?

Each of the five patterns below is a specific answer to one of those questions. None of them are new. All of them show up in real architectures under different names. What helped me was seeing them as *answers to questions* rather than as architectural vocabulary to namedrop.

## 1. Transactional Outbox - Stop Losing Events Between the DB and the Broker

The first time I shipped a service that fired events, my code looked something like this:

```typescript
await db.orders.insert(newOrder);
await broker.publish("order.created", newOrder);
```

It works. Most of the time. The problem is the tiny window between those two lines. If the process crashes after the insert but before the publish, the row exists and the event does not. If the publish succeeds but the commit fails, the event exists for an order that was never created. There is no way to make both lines atomic, because the database and the broker do not share a transaction.

The fix is to stop trying to publish inside the request path at all. Write the event to an **outbox table** in the *same database transaction* as the business row. A separate relay process reads the outbox and publishes to the broker afterwards.

```mermaid caption="Transactional Outbox - the broker publish is moved out of the request path and into a relay that reads from a table written in the same transaction as the business row."
flowchart LR
  S[Order Service]
  subgraph DB[Application Database]
    T[(Order Table)]
    O[(Outbox Table)]
  end
  R[Relay / CDC]
  B[(Message Broker)]
  C[Consumers]
  S -- "single transaction" --> T
  S -- "single transaction" --> O
  R -- "poll or CDC" --> O
  R --> B
  B --> C
```

**Use case I actually hit.** Order service writes an order, publishes `order.created`, and the email service sends a confirmation. Before the outbox: the broker had a thirty-second blip during a rolling restart and we dropped roughly two hundred confirmation emails. After the outbox: the blip became latency instead of loss - the events piled up in the table and the relay drained them once the broker came back.

The pattern already has its own deep-dive in the [pattern library](/patterns/outbox) if you want the schema and the tradeoff between polling and CDC. The thing I want to flag here is cultural rather than technical: once a team has the outbox, `await broker.publish(...)` inside a request handler stops being acceptable. The guarantee the pattern gives you only holds if everybody uses it.

## 2. CQRS - Reads and Writes Shouldn't Share a Model

CQRS stands for Command Query Responsibility Segregation, which is a long way of saying: the thing you use to change data should not be the same thing you use to read data.

In a traditional app, one model and one database serve both. You write `Order`, you read `Order`, both through the same schema. That works until your reads get complicated enough that they start deforming your writes. You add denormalized columns to make a dashboard fast. You add indexes that slow down inserts. You cache. The write model becomes a compromise between two jobs.

CQRS separates them. Writes go through a **command** side that models business operations and lands them in a write-optimized store. Reads go through a **query** side that pulls from a store shaped for the reads your product actually does. The two sides stay synchronized asynchronously - usually via events flowing through the broker.

```mermaid caption="CQRS - the command side owns state changes, the query side owns read shapes, and events on the broker keep the two eventually consistent."
flowchart LR
  Client[Client]
  Cmd[Command Handler]
  Q[Query Handler]
  W[(Write DB)]
  R[(Read DB)]
  B[(Broker)]
  Client -- "write" --> Cmd
  Cmd --> W
  W -- "change events" --> B
  B --> R
  Client -- "read" --> Q
  Q --> R
```

**Use case I actually hit.** A product catalog where the write model was Postgres (authoritative, relational, transactional) and the read model was Elasticsearch (great at fuzzy product search, typo tolerance, faceted filters). Every catalog update was a write against Postgres. A stream of events replayed the updates into Elasticsearch. The write side never had to care about search. The read side never had to care about business rules.

The trap to name out loud: CQRS gives you **eventual consistency** between the two sides. A user who updates a product and immediately lists products may see the old record. For most domains that is fine. For domains where it is not fine - a seller who needs to confirm their price change took effect - you need to either read-after-write from the write side, or make the query-side update synchronous, which defeats most of the point.

## 3. CQRS With Event Sourcing - Store What Happened, Not Just What Is

Regular CQRS keeps the current state on the write side. Event Sourcing goes one step further: it throws away the idea of storing the current state at all. What you store is the sequence of events that produced the state. Current state is a function of replaying those events.

Instead of a row saying "order 42: status=shipped, total=$99," you have an append-only log of events: `OrderCreated`, `OrderPaid`, `OrderShipped`. The current state is whatever you get by folding those events in order. Your read models are projections over the same event log.

```mermaid caption="CQRS with Event Sourcing - the event log is the source of truth. Read models are projections that can be rebuilt from the log at any time."
flowchart LR
  Client[Client]
  Cmd[Command Handler]
  ES[(Event Store)]
  B[(Broker)]
  P1[Projection A]
  P2[Projection B]
  R1[(Read DB - Orders)]
  R2[(Read DB - Analytics)]
  Client -- "command" --> Cmd
  Cmd -- "append event" --> ES
  ES --> B
  B --> P1 --> R1
  B --> P2 --> R2
  Client -- "read" --> R1
```

**Use case where it clearly earns its keep.** A banking ledger. You do not want to store "current balance" as a number that someone can `UPDATE`. You want an append-only log of every debit and credit, and the balance is whatever that log folds to. When an auditor shows up and asks "why was this account at $412.73 on March 2nd?", you have an answer more precise than any state table can give you.

**When I would not reach for it.** A CRUD app. A settings screen. A user profile. The value of event sourcing scales with how much you care about the *history* of the data, not the current value. If nobody will ever ask "how did we get here?", storing events is a lot of operational weight for benefits you will never consume.

The other honest caveat: event sourcing changes how you think about schema migrations, projections, and deployment. You need to be able to rebuild read models from the log. Events become part of your versioned API to your future self. I have only seen it go well in teams that deliberately invested in the tooling for it.

## 4. Saga - Distributed Transactions Without a Distributed Transaction

Sagas answer a specific question: how do you keep a multi-step business operation consistent when each step runs in a different service with its own database?

Take a checkout: reserve inventory, charge the customer, arrange shipping, send a confirmation. In a monolith this is one database transaction. In a microservice architecture, each step is a different service with its own database and no shared transaction. If the charge succeeds but shipping fails, you have a customer who paid for something that will never arrive - and no automatic rollback.

A saga decomposes the business operation into a sequence of local transactions, each of which emits an event that triggers the next step. If any step fails, **compensating transactions** run in reverse to undo the completed steps. There is no global lock; there is a protocol.

```mermaid caption="Saga (choreography style) - each service completes its local transaction and emits an event. Failure at any step triggers compensation events running in reverse."
sequenceDiagram
  participant User
  participant Order as Order Service
  participant Payment as Payment Service
  participant Ship as Shipping Service
  User->>Order: Place order
  Order->>Order: Create order (local tx)
  Order-->>Payment: order.created
  Payment->>Payment: Charge card (local tx)
  Payment-->>Ship: payment.charged
  Ship->>Ship: Arrange shipment (local tx)
  Ship-->>Order: shipment.arranged
  Note over Order,Ship: Any failure emits a compensation event<br/>that walks the saga backwards.
```

There are two flavors of saga. **Choreography** (shown above) has each service listen for events and decide what to do next - no central coordinator. **Orchestration** has a dedicated saga orchestrator that issues commands to each service and owns the state machine. Choreography is simpler to deploy but harder to observe. Orchestration is easier to reason about but introduces a coupling point. My rule of thumb: fewer than four steps, choreography. Four or more, orchestration, because the debuggability is worth it.

**Use case I actually hit.** An e-commerce checkout with five services. We started choreographed. By step four we could not answer the question "where did saga X get stuck?" without stitching logs from five systems by hand. We moved to an orchestrator with a state table. The code got slightly more boring and the on-call pages got dramatically shorter.

The pattern library has a [longer saga entry](/patterns/saga) with both styles in code. The thing worth repeating here: sagas give you **eventual consistency**, not atomicity. For a moment, the order exists and the payment does not. Your product, your dashboards, and your customer service team need to understand that.

## 5. Competing Consumers - One Queue, Many Workers

This is the most mechanically simple pattern of the five, and the one I reach for most often. It is also the one that needs the least explanation, so I will keep this short.

Producers drop messages onto a queue. Multiple consumer instances pull from the same queue. Each message is delivered to exactly one consumer. Load balances itself - faster consumers pull more, slower ones pull less. Add more consumers to go faster. Remove some to save cost.

```mermaid caption="Competing Consumers - a single queue, many worker instances. The broker load-balances messages across whichever consumers are currently pulling."
flowchart LR
  P1[Producer 1] --> Q
  P2[Producer 2] --> Q
  P3[Producer 3] --> Q
  Q[[Message Queue]]
  Q --> C1[Consumer A]
  Q --> C2[Consumer B]
  Q --> C3[Consumer C]
```

**Use case I actually hit.** Image processing pipeline. Users upload photos, a queue holds "process this photo" jobs, a fleet of worker containers pulls from the queue. On a slow day, three workers. On a campaign spike, forty. The queue absorbs the bursts so the upload endpoint never blocks.

**Two things I wish I had internalized earlier.** First, **messages must be processable in any order** - the broker load-balances across workers, so ordering per-message is not preserved across the fleet. If you need per-entity ordering (all events for user 42 processed in order), use a partitioned queue keyed on entity id, where each partition is its own competing-consumers pool of one. Second, **consumers must be idempotent**. Queues almost universally give at-least-once delivery. A worker that crashes after processing but before acknowledging will see the same message again. If running the handler twice produces the wrong answer, your pattern is broken - not the broker.

## How I Decide Which One I Actually Need

Drawing them side by side made the decision tree obvious in a way that reading the names never did:

- **I need to atomically record a state change and notify others?** → **Outbox.** Almost always. It is the default, not the exotic choice.
- **My reads and writes have genuinely different shapes and my write model is suffering for it?** → **CQRS.** Accept eventual consistency on the read side.
- **The history of changes is a first-class product concern (audit, debugging, rebuildable projections)?** → **CQRS with Event Sourcing.** Pay the tooling cost deliberately.
- **A business operation spans multiple services and must either all succeed or all compensate?** → **Saga.** Orchestrated if it has more than three steps.
- **I have a stream of independent tasks and I want to scale processing horizontally?** → **Competing Consumers.** Bake in idempotency from day one.

Most real systems use more than one of these at the same time. The checkout flow I mentioned earlier uses four of the five: the outbox to publish events atomically with the write, a saga to coordinate the flow, competing consumers on each step to scale, and a small CQRS slice for the order-lookup read model the customer service team uses. None of them felt clever in isolation. Together they made the system boring, which is the highest compliment I can give a distributed design.

{/* subscribe-cta */}

## The Quiet Lesson

The thing that surprised me most, drawing these out, is how much of "distributed systems design" is not really about algorithms or consensus or exotic infrastructure. It is about naming the *kind of inconsistency* you are willing to tolerate - across a transaction boundary, across two databases, across five services - and then picking the pattern that turns that inconsistency into something your system recovers from on its own.

The broker is the plumbing. These patterns are the plumbing conventions that keep the water going the right direction when one of the pipes shakes.

If you only take one thing from this article: the next time you write `await broker.publish(...)` directly after a database write, pause. That line is a micro-decision about consistency, availability, and what you owe the next engineer who has to debug this at 3am. The patterns above are the menu of better answers.


Five Message Broker Patterns


There is a class of distributed systems bug that appears only under load, only in production, and always at the worst possible moment. It looks like a race condition. It shows up in your Stripe dashboard as a duplicate charge. Your on-call engineer escalates it at 3am, and by the time the postmortem is written, three engineers have burned a weekend and a customer has filed a chargeback.

The pattern that prevents it has a name. It is called idempotency. And despite being the foundational primitive for safe distributed operations, it is consistently the last thing teams think about when designing new endpoints.

This essay is about why idempotency matters, what it actually costs to implement correctly, and when the pattern is worth every bit of the investment.

## What Idempotency Actually Means

An operation is idempotent if applying it more than once produces the same result as applying it once. This sounds obvious until you try to design a payment API. Charging a card is not naturally idempotent - if you call `POST /charges` twice with the same payload, you get two charges. Networks time out. Clients retry. Servers crash mid-write. The gap between "the server received the request" and "the client received the acknowledgment" is where money disappears.

The classic solution is the idempotency key - a client-generated token, typically a UUID, that the server uses to deduplicate requests within a time window.

```typescript
// Client sends this header with every mutating request
const charge = await stripe.charges.create({
  amount: 2000,
  currency: "usd",
  source: "tok_visa",
}, {
  idempotencyKey: "order_abc123_attempt_1",
});
```

If the network drops and the client retries, Stripe detects the same key and returns the original response rather than creating a new charge. The client cannot distinguish a first execution from a replay - which is exactly the point.

But the simplicity of the interface hides considerable complexity on the server side.

## The Three Layers of Idempotency

Implementing idempotency correctly requires thinking at three distinct layers, and most teams only address one or two of them.

**Layer 1: The deduplication store.** You need somewhere to record that you've seen a key and what the result was. This is usually a fast key-value store - Redis, DynamoDB, or a dedicated column in your Postgres transactions table. The entry must be written atomically with the underlying operation, or you have a new race condition: two concurrent requests with the same key, both checking before either has written.

```typescript
async function createCharge(
  idempotencyKey: string,
  params: ChargeParams
): Promise<Charge> {
  // Atomic check-and-set using a database transaction
  return await db.transaction(async (tx) => {
    const existing = await tx.idempotencyKeys.findUnique({
      where: { key: idempotencyKey },
    });

    if (existing) {
      // Return the stored result - not a new execution
      return JSON.parse(existing.responseBody) as Charge;
    }

    // Execute the actual operation
    const charge = await processCharge(params);

    // Record the key and result atomically
    await tx.idempotencyKeys.create({
      data: {
        key: idempotencyKey,
        responseBody: JSON.stringify(charge),
        expiresAt: addDays(new Date(), 7),
      },
    });

    return charge;
  });
}
```

**Layer 2: Request fingerprinting.** Idempotency keys only work if you tie them to a specific request shape. If a client sends the same key with different parameters - say, the same `idempotencyKey` but a different `amount` - you have to decide: honor the key and ignore the parameter mismatch, or reject the request with a 422? Stripe rejects. That is the correct answer. A mismatched fingerprint means the client has a bug, and surfacing that error immediately is better than silently ignoring the mismatch.

**Layer 3: Side effects beyond your database.** Your charge endpoint probably sends an email confirmation, increments a counter in a metrics system, and triggers a webhook. None of those systems have your idempotency guarantee. You need to ensure that downstream side effects are either naturally idempotent (incrementing a monotonic counter with a version check) or wrapped in their own deduplication logic. Idempotency at the API layer does not automatically propagate to async workers.

<PullQuote attribution={{ name: "Werner Vogels", role: "CTO, Amazon" }}>
Everything fails, all the time. Design for it.
</PullQuote>

## When the Pattern Pays Off

Idempotency is not free. The deduplication store adds a round trip. The fingerprinting logic adds complexity. The key expiration policy adds a new class of edge case - what happens when a key expires and the client retries six days later?

So when is it worth the cost?

The answer is almost always "yes" for any operation that changes state and could be retried by an upstream caller. In practice, this means:

- Payment and billing operations
- Order creation
- Message sends (email, SMS, push notifications)
- Provisioning actions (create VM, allocate resource)
- Any operation that touches external systems you do not control

The operations where idempotency is less critical are read operations (which are already naturally idempotent) and internal operations that are naturally safe to repeat. But even there, building the habit of asking "what happens if this runs twice?" is worth the mental overhead.

## The Subtle Failure Mode Nobody Talks About

There is a failure mode that trips up even experienced engineers: partial completion.

Your charge endpoint writes to the database, then calls the payment processor, then sends the confirmation email. The payment processor call succeeds, but the database write fails on the way back (a transient network partition, a rollback triggered by a constraint violation). The customer's card has been charged. Your database does not know about it. When the client retries with the same idempotency key, your deduplication store has nothing - and you charge the card again.

The solution is to write the idempotency key entry *before* you start executing the operation, not after. Set its status to `pending`, execute the operation, then update the status to `completed` with the stored result. If the system crashes mid-execution, subsequent attempts will find the `pending` entry and can either wait (if the operation might still complete) or attempt recovery.

This is the pattern Stripe describes in their engineering blog. It is sometimes called the "idempotency receipt" pattern, and it is the detail that separates robust implementations from ones that fail under concurrent retries.

## What This Means for Your API Design

If you are designing a new service today, these are the decisions that matter:

**Use client-generated keys, not server-generated.** Servers cannot create idempotency keys because they cannot know whether a request is a retry. Only the client knows that. The client generates a UUID, stores it locally, and sends it with every retry attempt until it receives a definitive response.

**Be explicit about key scope.** Does your key apply across all operations or just a specific endpoint? Stripe scopes keys to the calling API key, which prevents one integration from colliding with another. Define your scope explicitly in your documentation.

**Choose a reasonable expiration window.** Seven days is a common choice. Long enough to cover any realistic retry window, short enough that your deduplication store does not grow unbounded. Paginate your key cleanup job.

**Communicate idempotency status to callers.** Return a header like `Idempotent-Replayed: true` when you return a cached response. This helps clients distinguish "the operation ran" from "the operation ran and I'm giving you the result from last time." It also helps your debugging story: a duplicate alert with `Idempotent-Replayed: true` is a client bug, not a server bug.

## The Real Cost of Not Implementing It

The cost of skipping idempotency is not just duplicate charges. It is the cumulative technical debt that accumulates when you build the wrong primitive into your foundation. Every endpoint you add without idempotency is an endpoint that needs to be retrofitted when you start handling retries. Every integration that assumes single delivery is an integration that will behave incorrectly under network partitions.

The engineers who understand this build retry logic correctly from day one. Their systems degrade gracefully. Their on-call rotations are quieter. Their postmortems are shorter.

Idempotency is not glamorous. It does not make it into architecture talks at conferences. But it is one of the clearest examples of a pattern that pays for itself many times over - in the operational incidents it prevents, the customer trust it preserves, and the engineering time it does not consume.

The pattern worth paying for is the one you never have to think about again.

---

*If this resonated, the follow-up is about exactly-once delivery - what it means, what it costs, and why distributed systems practitioners generally settle for at-least-once with idempotency instead.*


Essaystaggedarchitecture

Five Message Broker Patterns

The Pattern Worth Paying For