Back to all articles
Platform at ScalePart 11
Multi-tenancyRate limitingSystem designReliabilityPlatform

Platform at Scale, Part 11: One Tenant Can't Sink the Ship

Janahan Thavendran7 min read
In this article

Platform at Scale, Part 11 · Fairness at every shared layer, so one merchant's spike stays one merchant's problem.

The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.

At 7:45 on a Saturday, one merchant launches a promotion on a delivery app. Their order rate across all their outlets goes up 8 times in two minutes.

The platform was sized for the total peak, not for one merchant's spike. Part 8's outlet lanes stopped one stuck outlet from blocking the others, but nothing stopped one busy merchant from taking most of the consumers working that partner. Their burst also filled the orders service's connection pool. Within minutes, orders at a restaurant across town, on a different partner, were slowing down for reasons that had nothing to do with them.

The incident wasn't the promotion. Promotions are good for everyone. The incident was that one tenant's traffic competed on equal terms with everyone else's, and the first person to notice was a merchant whose orders were late through no fault of their own.

Why the platform couldn't tell tenants apart

Part 5 isolated services from each other: separate databases, capped pools, separate caches. But inside each service, every tenant still shared everything: the orders service's connection pool, the catalog cache and the consumers working each partner's queue. Fairness between services isn't fairness between tenants.

Part 6 gave us the "worst merchants right now" dashboard. We could see one merchant spiking. We had no way to stop that spike reaching everyone else.

What we fixed, in order

Stage 1: quotas per merchant, outlet and integration

Every merchant, outlet and partner integration got a rate limit, sized from their own historical peak plus a margin, not from the platform average.

What happened above the quota depended on who was asking:

  • Real diner orders were never rejected. Above the quota, they queued briefly, and the diner saw "Order received". Turning away a paying diner because of our limits would have been the worst possible outcome.
  • Integrations and bulk jobs like menu pushes, partner syncs and report exports got a "too many requests" response with a time to retry. Partners knew the limits before we enforced them. A quota is a contract, not a surprise. Quotas were enforced at the gateway, and every request carried a tenant claim that each service checked again (Part 14). Rate limits, tenant authorisation and abuse rules were three separate layers, not one.

Stage 2: a fair share of partner consumers

Part 8's lanes made sure one stuck outlet only held up itself. This stage added a cap on how much of a partner's consumer capacity one merchant could use at once: 30%. A merchant's promotion now drained at its own pace while every other merchant on that partner kept moving.

Stage 3: fair use of the database

Inside each service's capped connection pool from Part 5, one merchant could hold at most 20% of the connections at once. A merchant's burst waited for its own share instead of taking everyone's.

Reads that weren't on the order path, like history, exports and reports, also got a maximum execution time. A runaway query was stopped instead of holding a connection for minutes. A hot tenant now slowed themselves down before they slowed anyone else.

Stage 4: a shedding order, decided in advance

Part 7 established that events are not orders. We turned that into a platform rule: under overload, work is shed in a fixed order, and orders and payments are always last.

Order
1
What we shed
Analytics events (Part 7)
What the diner or merchant sees
Nothing. Devices hold them and send them later
Order
2
What we shed
Recommendations
What the diner or merchant sees
Menus without suggestions
Order
3
What we shed
History and dashboards
What the diner or merchant sees
Data labelled "Updated X seconds ago" (Part 3)
Order
4
What we shed
Partner menu syncs
What the diner or merchant sees
Menu changes reach partner apps a little later
Order
Never
What we shed
Orders and payments
What the diner or merchant sees
Nothing. These are what the platform exists for

The order was set centrally and sent to every client, so devices and services degraded the same way, instead of each one timing out on its own.

Stage 5: alerts on each tenant's rate of change

Part 6's merchant views watched totals. We added alerts on how fast each merchant's traffic was changing. A merchant whose order rate tripled in two minutes triggered a heads-up to their account manager and the integrations team. Usually it was a promotion, and the merchant heard from us before they had a problem, not after.

The trade-off we made

Option
Trust tenants, no limits
Why it was tempting
No code, no friction
Why we didn't choose it
One promotion becomes everyone's incident
Option
Global rate limits
Why it was tempting
Simple to build
Why we didn't choose it
Punishes every tenant for one tenant's spike
Option
Per-tenant quotas at the front door only
Why it was tempting
Cheap, stops the flood at the edge
Why we didn't choose it
Legitimate traffic that gets through still competes for every shared pool behind it
Option
One database per tenant
Why it was tempting
Maximum fairness
Why we didn't choose it
Cost and operational load nobody wanted at our size
Option
Quotas, a fair share of partner consumers, per-tenant connection limits and a shedding order
Why it was tempting
Turns a platform incident into one tenant's slower channel
Why we didn't choose it
Our choice, with tenant identity carried everywhere

We chose fairness at every shared layer over isolation at the database. Tenants still share MySQL, Redis and RabbitMQ. What changed is that none of them can take more than their share of any of them.

What we measured

  • Incidents where one merchant's spike slowed other merchants: from 4 a quarter to zero
  • Order placement time at p95 for other merchants during a spike: 2 seconds
  • Quota hits per week, and how many were promotions versus runaway integrations
  • Time from a spike starting to the merchant being contacted: 5 minutes

What it cost

Tenant identity had to travel everywhere: through the API, the queues, the connection pools and the logs. Later, the golden path generated the quota class and the identity plumbing, so a new service couldn't forget them (Part 13). Quota changes became a support process. Some merchants were briefly annoyed by a limit they'd never hit before.

In return, a promotion, a runaway integration or a partner burst became one merchant's slower channel, not a platform-wide incident.

What this part leaves for later

Gap left here
Full isolation for a merchant who needs it by contract
Where it's covered
A different platform, outside this series
Gap left here
Per-tenant data residency and encryption
Where it's covered
Ties to Part 9's second region and Part 10's archive
Gap left here
The tenant authorisation model and where it's enforced
Where it's covered
Part 14: the edge

What I took from it

Averages hide disasters.

Fairness is a design decision at every shared layer, not a switch you turn on at the front door. Quotas, a fair share of every queue and pool, and a shedding order decided in advance turn a platform-wide incident into one merchant's problem, which is the right size for it.

Next in the series: Part 12: Deploying on a Saturday