Platform at Scale, Part 11 · Fairness at every shared layer, so one merchant's spike stays one merchant's problem.
The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.
At 7:45 on a Saturday, one merchant launches a promotion on a delivery app. Their order rate across all their outlets goes up 8 times in two minutes.
The platform was sized for the total peak, not for one merchant's spike. Part 8's outlet lanes stopped one stuck outlet from blocking the others, but nothing stopped one busy merchant from taking most of the consumers working that partner. Their burst also filled the orders service's connection pool. Within minutes, orders at a restaurant across town, on a different partner, were slowing down for reasons that had nothing to do with them.
The incident wasn't the promotion. Promotions are good for everyone. The incident was that one tenant's traffic competed on equal terms with everyone else's, and the first person to notice was a merchant whose orders were late through no fault of their own.
Why the platform couldn't tell tenants apart
Part 5 isolated services from each other: separate databases, capped pools, separate caches. But inside each service, every tenant still shared everything: the orders service's connection pool, the catalog cache and the consumers working each partner's queue. Fairness between services isn't fairness between tenants.
Part 6 gave us the "worst merchants right now" dashboard. We could see one merchant spiking. We had no way to stop that spike reaching everyone else.
What we fixed, in order
Stage 1: quotas per merchant, outlet and integration
Every merchant, outlet and partner integration got a rate limit, sized from their own historical peak plus a margin, not from the platform average.
What happened above the quota depended on who was asking:
- Real diner orders were never rejected. Above the quota, they queued briefly, and the diner saw "Order received". Turning away a paying diner because of our limits would have been the worst possible outcome.
- Integrations and bulk jobs like menu pushes, partner syncs and report exports got a "too many requests" response with a time to retry. Partners knew the limits before we enforced them. A quota is a contract, not a surprise. Quotas were enforced at the gateway, and every request carried a tenant claim that each service checked again (Part 14). Rate limits, tenant authorisation and abuse rules were three separate layers, not one.
Stage 2: a fair share of partner consumers
Part 8's lanes made sure one stuck outlet only held up itself. This stage added a cap on how much of a partner's consumer capacity one merchant could use at once: 30%. A merchant's promotion now drained at its own pace while every other merchant on that partner kept moving.
Stage 3: fair use of the database
Inside each service's capped connection pool from Part 5, one merchant could hold at most 20% of the connections at once. A merchant's burst waited for its own share instead of taking everyone's.
Reads that weren't on the order path, like history, exports and reports, also got a maximum execution time. A runaway query was stopped instead of holding a connection for minutes. A hot tenant now slowed themselves down before they slowed anyone else.
Stage 4: a shedding order, decided in advance
Part 7 established that events are not orders. We turned that into a platform rule: under overload, work is shed in a fixed order, and orders and payments are always last.
| Order | What we shed | What the diner or merchant sees |
|---|---|---|
| 1 | Analytics events (Part 7) | Nothing. Devices hold them and send them later |
| 2 | Recommendations | Menus without suggestions |
| 3 | History and dashboards | Data labelled "Updated X seconds ago" (Part 3) |
| 4 | Partner menu syncs | Menu changes reach partner apps a little later |
| Never | Orders and payments | Nothing. These are what the platform exists for |
- Order
- 1
- What we shed
- Analytics events (Part 7)
- What the diner or merchant sees
- Nothing. Devices hold them and send them later
- Order
- 2
- What we shed
- Recommendations
- What the diner or merchant sees
- Menus without suggestions
- Order
- 3
- What we shed
- History and dashboards
- What the diner or merchant sees
- Data labelled "Updated X seconds ago" (Part 3)
- Order
- 4
- What we shed
- Partner menu syncs
- What the diner or merchant sees
- Menu changes reach partner apps a little later
- Order
- Never
- What we shed
- Orders and payments
- What the diner or merchant sees
- Nothing. These are what the platform exists for
The order was set centrally and sent to every client, so devices and services degraded the same way, instead of each one timing out on its own.
Stage 5: alerts on each tenant's rate of change
Part 6's merchant views watched totals. We added alerts on how fast each merchant's traffic was changing. A merchant whose order rate tripled in two minutes triggered a heads-up to their account manager and the integrations team. Usually it was a promotion, and the merchant heard from us before they had a problem, not after.
The trade-off we made
| Option | Why it was tempting | Why we didn't choose it |
|---|---|---|
| Trust tenants, no limits | No code, no friction | One promotion becomes everyone's incident |
| Global rate limits | Simple to build | Punishes every tenant for one tenant's spike |
| Per-tenant quotas at the front door only | Cheap, stops the flood at the edge | Legitimate traffic that gets through still competes for every shared pool behind it |
| One database per tenant | Maximum fairness | Cost and operational load nobody wanted at our size |
| Quotas, a fair share of partner consumers, per-tenant connection limits and a shedding order | Turns a platform incident into one tenant's slower channel | Our choice, with tenant identity carried everywhere |
- Option
- Trust tenants, no limits
- Why it was tempting
- No code, no friction
- Why we didn't choose it
- One promotion becomes everyone's incident
- Option
- Global rate limits
- Why it was tempting
- Simple to build
- Why we didn't choose it
- Punishes every tenant for one tenant's spike
- Option
- Per-tenant quotas at the front door only
- Why it was tempting
- Cheap, stops the flood at the edge
- Why we didn't choose it
- Legitimate traffic that gets through still competes for every shared pool behind it
- Option
- One database per tenant
- Why it was tempting
- Maximum fairness
- Why we didn't choose it
- Cost and operational load nobody wanted at our size
- Option
- Quotas, a fair share of partner consumers, per-tenant connection limits and a shedding order
- Why it was tempting
- Turns a platform incident into one tenant's slower channel
- Why we didn't choose it
- Our choice, with tenant identity carried everywhere
We chose fairness at every shared layer over isolation at the database. Tenants still share MySQL, Redis and RabbitMQ. What changed is that none of them can take more than their share of any of them.
What we measured
- Incidents where one merchant's spike slowed other merchants: from 4 a quarter to zero
- Order placement time at p95 for other merchants during a spike: 2 seconds
- Quota hits per week, and how many were promotions versus runaway integrations
- Time from a spike starting to the merchant being contacted: 5 minutes
What it cost
Tenant identity had to travel everywhere: through the API, the queues, the connection pools and the logs. Later, the golden path generated the quota class and the identity plumbing, so a new service couldn't forget them (Part 13). Quota changes became a support process. Some merchants were briefly annoyed by a limit they'd never hit before.
In return, a promotion, a runaway integration or a partner burst became one merchant's slower channel, not a platform-wide incident.
What this part leaves for later
| Gap left here | Where it's covered |
|---|---|
| Full isolation for a merchant who needs it by contract | A different platform, outside this series |
| Per-tenant data residency and encryption | Ties to Part 9's second region and Part 10's archive |
| The tenant authorisation model and where it's enforced | Part 14: the edge |
- Gap left here
- Full isolation for a merchant who needs it by contract
- Where it's covered
- A different platform, outside this series
- Gap left here
- Per-tenant data residency and encryption
- Where it's covered
- Ties to Part 9's second region and Part 10's archive
- Gap left here
- The tenant authorisation model and where it's enforced
- Where it's covered
- Part 14: the edge
What I took from it
Averages hide disasters.
Fairness is a design decision at every shared layer, not a switch you turn on at the front door. Quotas, a fair share of every queue and pool, and a shedding order decided in advance turn a platform-wide incident into one merchant's problem, which is the right size for it.
Next in the series: Part 12: Deploying on a Saturday