Back to all articles
Platform at ScalePart 08
IntegrationsRabbitMQPartner APIsSystem designReliability

Platform at Scale, Part 8: Partners That Take One Order at a Time

Janahan Thavendran9 min read
In this article

Platform at Scale, Part 8 · You're only as fast as the slowest system you talk to.

The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.

It's a festive Saturday and one outlet is flooded. Five tables send orders within the same few seconds. Our platform is ready for that. By now, at peak, it takes more than 20,000 orders a minute across every outlet. The partner system the outlet uses to receive orders is not. It accepts the first order and rejects the other four.

The diners have already paid. The kitchen never sees four of those orders.

Why partners broke differently

By now our middleware connected to 50+ partners, global and local. Each had its own system, its own API and its own limits. Some handled orders in parallel without blinking. Some could only process one order at a time per outlet. Send a second while the first was still in flight, and they rejected it, timed out, or worse, accepted it twice.

We had spent years teaching our platform to handle bursts. Now we were sending those bursts straight into systems that couldn't absorb them. Our capacity stopped mattering. The slowest partner set the pace for its outlets.

Orders weren't the only thing we sent. Status updates went back to delivery partners too: accepted, ready, picked up. Part 7 left those for this part.

The design

Everything we sent to a partner followed one path, built so that nothing could be lost and no partner could receive more than it could take: the order service wrote the order and an outbox row in one transaction, a relay published to RabbitMQ when the broker was up, each partner had its own queue with a lane per outlet, and consumers sent to the partner's API only as many orders as that partner could handle, acknowledging each one only on success.

The outbox means a partner order is saved the moment the diner's order is. The lanes and limits mean each partner only ever gets what it can handle. Every partner also had its own credentials in Secret Manager, rotated on a schedule, and our outbound traffic left from a fixed NAT address that partners allowlisted (Part 14). A changing outbound address is a partner outage waiting for a Saturday.

What we built

  1. Save the send with the order. When an order was placed, the orders service wrote a "pending partner delivery" row in its outbox, in the same transaction as the order (Part 7). A relay moved those rows into RabbitMQ. If RabbitMQ was down, orders were still accepted and paid. The rows simply waited, and the relay caught up when the broker came back. The broker could never take ordering down with it.
  2. A queue per partner, a lane per outlet. One queue per partner had a hidden flaw: a single stuck outlet could block every other outlet on that partner. So within each partner's queue, messages were routed by outlet, and the in-flight limit applied per outlet. A one-at-a-time partner got one order in flight per outlet, not one for the whole partner. A slow outlet only held up itself. Later, a new partner integration got its queue, lanes, limits and dead-letter queue from a golden-path template instead of a copy (Part 13).
  3. Limits set per partner, starting at one. Every partner had its own settings: orders in flight per outlet, timeout, retry budget and backoff. A new partner started at one order in flight per outlet with a cautious timeout. We tested in their sandbox where one existed, watched their rejection rate in the first weeks, and only raised the limit with evidence.
  4. Remove from the queue only on success. An order left the queue only when the partner confirmed it. If they rejected it, it was retried with growing gaps, within that partner's budget.
  5. Ask before resending. A timeout doesn't mean the partner didn't receive the order. So after a timeout we didn't resend blindly. Where the partner's API allowed it, we first asked for the order's status using our order ID, and only resent if they had no record of it. It's Part 4's "unknown" payment state, applied to partners.
  6. Send our order ID every time. Partners that honoured it ignored our duplicates. Many didn't, which is why the next step existed.
  7. Reconcile with every partner. Every 15 minutes during service, a job compared the orders we sent with the orders each partner acknowledged. A missing order was resent. A duplicate on the partner's side was cancelled through their API where possible, or flagged to support to cancel with the outlet before the kitchen cooked it twice.
  8. A dead letter, an alert and a person. When an order used up its retry budget, it went to a dead-letter queue, the integrations on-call was paged, and the outlet was told straight away. Support could get the order to the kitchen while the diner was still at the table, not the next morning.
  9. Status updates on the same path. Updates to delivery partners used the same queues and the same outlet lanes, so each order's updates arrived in order. If an update arrived late, the partner always ended with the latest state.

Rejected partner orders on peak nights dropped from about 3% to 0.2%.

When the queue grows

Queue depth per outlet and per partner became one of our most watched numbers, because a growing queue was the earliest sign of trouble. Watching it wasn't enough. Each signal had a planned response:

Signal
One outlet's lane growing
What it usually means
That outlet's connection to the partner is slow or stuck
What happens
Alert the integrations on-call. If it's stuck more than 5 minutes, tell the outlet and hand over to support
Signal
A whole partner's queue growing
What it usually means
The partner is having an outage
What happens
Page on-call and contact the partner. After 10 minutes, pause sending so retries aren't wasted, and warn the merchants affected
Signal
Queues growing across many partners
What it usually means
Our consumers or RabbitMQ are the problem
What happens
Page the platform on-call, scale up consumers, check the broker
Signal
Anything in the dead-letter queue
What it usually means
An order that couldn't be delivered
What happens
Page. Every one is handled before the diner notices

Orders keep coming

A slow partner never stopped us taking orders. The diner had already paid, and the order was safe in the outbox, so refusing it would only have made things worse.

But the merchant needed to know. Their back office showed a clear warning when orders to a partner were delayed. And if a partner stayed backed up for more than 20 minutes, we suggested pausing that channel at the outlet until it recovered. Whether to pause was the merchant's call. They knew whether their staff could cope with the delay.

The trade-off we made

Option
Keep sending directly, retry on failure
Why it was tempting
Nothing new to build
Why we didn't choose it
Retries during a burst hammer a partner that's already saying no
Option
Ask every partner to support parallel orders
Why it was tempting
Fixes it at the source
Why we didn't choose it
Not ours to control, and many partners couldn't change on our timeline
Option
Pub/Sub, which we already ran
Why it was tempting
No new system
Why we didn't choose it
For us, a fixed number of in-flight orders per partner and outlet was simpler to express with RabbitMQ's per-queue consumers and explicit acknowledgements
Option
Cloud Tasks with rate limits
Why it was tempting
Managed, has rate controls
Why we didn't choose it
Limits orders per second, but our problem was orders at the same time, per outlet
Option
Call partners directly when RabbitMQ is down
Why it was tempting
Keeps orders moving
Why we didn't choose it
Brings the bursts back exactly when things are already failing. The outbox waits instead
Option
Outbox plus RabbitMQ: a queue per partner, a lane per outlet, limits per partner, reconciliation
Why it was tempting
Never loses an order, and matches each partner's real limit
Why we didn't choose it
Our choice, accepting a broker to operate

Our capacity was never the bottleneck here. The partner's was. So we chose the tool that let us match each partner's limit exactly, even though it meant running one more system.

Where it hurt, and what it cost

Orders now waited in line. On a busy night, an order for a one-at-a-time partner could take 20 seconds longer to reach the kitchen at p95. We chose a slower order over a lost one.

RabbitMQ became something we had to run, upgrade and monitor, including its own peak capacity. Reconciliation with every partner took engineering time to build and support time to act on. Infrastructure and on-call time added about $1,500 a month.

What we measured

  • Rejected partner orders on peak nights: from about 3% to 0.2%
  • Extra time to reach the kitchen for one-at-a-time partners: 20 seconds at p95
  • Queue depth per partner at peak
  • Dead-letter orders per week, and time to resolve each: 8 minutes
  • Duplicates on the partner side found by reconciliation: about 25 a week
  • Monthly cost of the broker and reconciliation: about $1,500 a month

What this part leaves for later

Gap left here
RabbitMQ running in a single zone
Where it's covered
Part 9: zones and regions
Gap left here
One merchant's promotion taking most of a partner's consumers
Where it's covered
Part 11: noisy tenants
Gap left here
Changing partner integrations safely during a festive week
Where it's covered
Part 12: safe deploys
Gap left here
New partner integrations from a template instead of a copy
Where it's covered
Part 13: the platform
Gap left here
Partner credentials, signed inbound webhooks and a fixed outbound address
Where it's covered
Part 14: the edge

What I took from it

Your platform is only as fast as the slowest system it has to talk to. When you can't change a partner, design around their limits: never lose the order, never assume they'll remove your duplicates, and make waiting visible before it turns into a lost order.

Next in the series: Part 9: The region that wasn't there