Platform at Scale, Part 8 · You're only as fast as the slowest system you talk to.
The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.
It's a festive Saturday and one outlet is flooded. Five tables send orders within the same few seconds. Our platform is ready for that. By now, at peak, it takes more than 20,000 orders a minute across every outlet. The partner system the outlet uses to receive orders is not. It accepts the first order and rejects the other four.
The diners have already paid. The kitchen never sees four of those orders.
Why partners broke differently
By now our middleware connected to 50+ partners, global and local. Each had its own system, its own API and its own limits. Some handled orders in parallel without blinking. Some could only process one order at a time per outlet. Send a second while the first was still in flight, and they rejected it, timed out, or worse, accepted it twice.
We had spent years teaching our platform to handle bursts. Now we were sending those bursts straight into systems that couldn't absorb them. Our capacity stopped mattering. The slowest partner set the pace for its outlets.
Orders weren't the only thing we sent. Status updates went back to delivery partners too: accepted, ready, picked up. Part 7 left those for this part.
The design
Everything we sent to a partner followed one path, built so that nothing could be lost and no partner could receive more than it could take: the order service wrote the order and an outbox row in one transaction, a relay published to RabbitMQ when the broker was up, each partner had its own queue with a lane per outlet, and consumers sent to the partner's API only as many orders as that partner could handle, acknowledging each one only on success.
The outbox means a partner order is saved the moment the diner's order is. The lanes and limits mean each partner only ever gets what it can handle. Every partner also had its own credentials in Secret Manager, rotated on a schedule, and our outbound traffic left from a fixed NAT address that partners allowlisted (Part 14). A changing outbound address is a partner outage waiting for a Saturday.
What we built
- Save the send with the order. When an order was placed, the orders service wrote a "pending partner delivery" row in its outbox, in the same transaction as the order (Part 7). A relay moved those rows into RabbitMQ. If RabbitMQ was down, orders were still accepted and paid. The rows simply waited, and the relay caught up when the broker came back. The broker could never take ordering down with it.
- A queue per partner, a lane per outlet. One queue per partner had a hidden flaw: a single stuck outlet could block every other outlet on that partner. So within each partner's queue, messages were routed by outlet, and the in-flight limit applied per outlet. A one-at-a-time partner got one order in flight per outlet, not one for the whole partner. A slow outlet only held up itself. Later, a new partner integration got its queue, lanes, limits and dead-letter queue from a golden-path template instead of a copy (Part 13).
- Limits set per partner, starting at one. Every partner had its own settings: orders in flight per outlet, timeout, retry budget and backoff. A new partner started at one order in flight per outlet with a cautious timeout. We tested in their sandbox where one existed, watched their rejection rate in the first weeks, and only raised the limit with evidence.
- Remove from the queue only on success. An order left the queue only when the partner confirmed it. If they rejected it, it was retried with growing gaps, within that partner's budget.
- Ask before resending. A timeout doesn't mean the partner didn't receive the order. So after a timeout we didn't resend blindly. Where the partner's API allowed it, we first asked for the order's status using our order ID, and only resent if they had no record of it. It's Part 4's "unknown" payment state, applied to partners.
- Send our order ID every time. Partners that honoured it ignored our duplicates. Many didn't, which is why the next step existed.
- Reconcile with every partner. Every 15 minutes during service, a job compared the orders we sent with the orders each partner acknowledged. A missing order was resent. A duplicate on the partner's side was cancelled through their API where possible, or flagged to support to cancel with the outlet before the kitchen cooked it twice.
- A dead letter, an alert and a person. When an order used up its retry budget, it went to a dead-letter queue, the integrations on-call was paged, and the outlet was told straight away. Support could get the order to the kitchen while the diner was still at the table, not the next morning.
- Status updates on the same path. Updates to delivery partners used the same queues and the same outlet lanes, so each order's updates arrived in order. If an update arrived late, the partner always ended with the latest state.
Rejected partner orders on peak nights dropped from about 3% to 0.2%.
When the queue grows
Queue depth per outlet and per partner became one of our most watched numbers, because a growing queue was the earliest sign of trouble. Watching it wasn't enough. Each signal had a planned response:
| Signal | What it usually means | What happens |
|---|---|---|
| One outlet's lane growing | That outlet's connection to the partner is slow or stuck | Alert the integrations on-call. If it's stuck more than 5 minutes, tell the outlet and hand over to support |
| A whole partner's queue growing | The partner is having an outage | Page on-call and contact the partner. After 10 minutes, pause sending so retries aren't wasted, and warn the merchants affected |
| Queues growing across many partners | Our consumers or RabbitMQ are the problem | Page the platform on-call, scale up consumers, check the broker |
| Anything in the dead-letter queue | An order that couldn't be delivered | Page. Every one is handled before the diner notices |
- Signal
- One outlet's lane growing
- What it usually means
- That outlet's connection to the partner is slow or stuck
- What happens
- Alert the integrations on-call. If it's stuck more than 5 minutes, tell the outlet and hand over to support
- Signal
- A whole partner's queue growing
- What it usually means
- The partner is having an outage
- What happens
- Page on-call and contact the partner. After 10 minutes, pause sending so retries aren't wasted, and warn the merchants affected
- Signal
- Queues growing across many partners
- What it usually means
- Our consumers or RabbitMQ are the problem
- What happens
- Page the platform on-call, scale up consumers, check the broker
- Signal
- Anything in the dead-letter queue
- What it usually means
- An order that couldn't be delivered
- What happens
- Page. Every one is handled before the diner notices
Orders keep coming
A slow partner never stopped us taking orders. The diner had already paid, and the order was safe in the outbox, so refusing it would only have made things worse.
But the merchant needed to know. Their back office showed a clear warning when orders to a partner were delayed. And if a partner stayed backed up for more than 20 minutes, we suggested pausing that channel at the outlet until it recovered. Whether to pause was the merchant's call. They knew whether their staff could cope with the delay.
The trade-off we made
| Option | Why it was tempting | Why we didn't choose it |
|---|---|---|
| Keep sending directly, retry on failure | Nothing new to build | Retries during a burst hammer a partner that's already saying no |
| Ask every partner to support parallel orders | Fixes it at the source | Not ours to control, and many partners couldn't change on our timeline |
| Pub/Sub, which we already ran | No new system | For us, a fixed number of in-flight orders per partner and outlet was simpler to express with RabbitMQ's per-queue consumers and explicit acknowledgements |
| Cloud Tasks with rate limits | Managed, has rate controls | Limits orders per second, but our problem was orders at the same time, per outlet |
| Call partners directly when RabbitMQ is down | Keeps orders moving | Brings the bursts back exactly when things are already failing. The outbox waits instead |
| Outbox plus RabbitMQ: a queue per partner, a lane per outlet, limits per partner, reconciliation | Never loses an order, and matches each partner's real limit | Our choice, accepting a broker to operate |
- Option
- Keep sending directly, retry on failure
- Why it was tempting
- Nothing new to build
- Why we didn't choose it
- Retries during a burst hammer a partner that's already saying no
- Option
- Ask every partner to support parallel orders
- Why it was tempting
- Fixes it at the source
- Why we didn't choose it
- Not ours to control, and many partners couldn't change on our timeline
- Option
- Pub/Sub, which we already ran
- Why it was tempting
- No new system
- Why we didn't choose it
- For us, a fixed number of in-flight orders per partner and outlet was simpler to express with RabbitMQ's per-queue consumers and explicit acknowledgements
- Option
- Cloud Tasks with rate limits
- Why it was tempting
- Managed, has rate controls
- Why we didn't choose it
- Limits orders per second, but our problem was orders at the same time, per outlet
- Option
- Call partners directly when RabbitMQ is down
- Why it was tempting
- Keeps orders moving
- Why we didn't choose it
- Brings the bursts back exactly when things are already failing. The outbox waits instead
- Option
- Outbox plus RabbitMQ: a queue per partner, a lane per outlet, limits per partner, reconciliation
- Why it was tempting
- Never loses an order, and matches each partner's real limit
- Why we didn't choose it
- Our choice, accepting a broker to operate
Our capacity was never the bottleneck here. The partner's was. So we chose the tool that let us match each partner's limit exactly, even though it meant running one more system.
Where it hurt, and what it cost
Orders now waited in line. On a busy night, an order for a one-at-a-time partner could take 20 seconds longer to reach the kitchen at p95. We chose a slower order over a lost one.
RabbitMQ became something we had to run, upgrade and monitor, including its own peak capacity. Reconciliation with every partner took engineering time to build and support time to act on. Infrastructure and on-call time added about $1,500 a month.
What we measured
- Rejected partner orders on peak nights: from about 3% to 0.2%
- Extra time to reach the kitchen for one-at-a-time partners: 20 seconds at p95
- Queue depth per partner at peak
- Dead-letter orders per week, and time to resolve each: 8 minutes
- Duplicates on the partner side found by reconciliation: about 25 a week
- Monthly cost of the broker and reconciliation: about $1,500 a month
What this part leaves for later
| Gap left here | Where it's covered |
|---|---|
| RabbitMQ running in a single zone | Part 9: zones and regions |
| One merchant's promotion taking most of a partner's consumers | Part 11: noisy tenants |
| Changing partner integrations safely during a festive week | Part 12: safe deploys |
| New partner integrations from a template instead of a copy | Part 13: the platform |
| Partner credentials, signed inbound webhooks and a fixed outbound address | Part 14: the edge |
- Gap left here
- RabbitMQ running in a single zone
- Where it's covered
- Part 9: zones and regions
- Gap left here
- One merchant's promotion taking most of a partner's consumers
- Where it's covered
- Part 11: noisy tenants
- Gap left here
- Changing partner integrations safely during a festive week
- Where it's covered
- Part 12: safe deploys
- Gap left here
- New partner integrations from a template instead of a copy
- Where it's covered
- Part 13: the platform
- Gap left here
- Partner credentials, signed inbound webhooks and a fixed outbound address
- Where it's covered
- Part 14: the edge
What I took from it
Your platform is only as fast as the slowest system it has to talk to. When you can't change a partner, design around their limits: never lose the order, never assume they'll remove your duplicates, and make waiting visible before it turns into a lost order.
Next in the series: Part 9: The region that wasn't there