Back to all articles
Platform at ScalePart 01
System designScalingGCPMySQLCapacity planning

Platform at Scale, Part 1: Saturday Night, Every Channel at Once

Janahan Thavendran15 min read
In this article

Platform at Scale, Part 1 · Fix in order of cost, and plan for the peak minute, not the average day.

The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.

It's 7:40 on a public holiday Saturday. Every table in one outlet is ordering. The restaurant's own app is taking pickup orders. A tablet by the counter keeps chiming with delivery orders arriving from partner apps through our middleware. And on every channel, placing an order is starting to take far too long.

The merchant doesn't see our CPU graph. They see an order that hangs after the diner presses "Place order", a delivery rider waiting for an order that never reached the kitchen, and a family walking out.

That was the night our platform had to work. Every decision in this series traces back to it.

Why Saturday decides everything

Our customers were restaurants. Our load came from their diners, and from the delivery platforms sending orders on their behalf.

The week had a shape. On Monday, most restaurants are close to empty. Tuesday to Thursday, orders build gradually. Friday picks up. Saturday is the peak, and Sunday stays high. Public holidays and festive seasons like Chinese New Year and Christmas push every outlet to full capacity, on every channel at once.

That shape was the business case. Friday to Sunday carried about 55% of a merchant's weekly orders. A slow Monday cost almost nothing. A slow Saturday cost merchants the revenue that paid their rent, and cost us their trust.

Delivery made it sharper. Delivery platforms track how reliably an outlet accepts orders. Every order we failed to pass through in time could count against the outlet, not against us. So a platform problem on Saturday became the merchant's ranking problem on Monday.

Peak per minute, not per day

The number that mattered wasn't orders per day. It was orders per minute at 7:30 on a Saturday, and how long checkout took at that moment.

Daily totals looked comfortable. About 300,000 orders a day across the platform was nothing one database couldn't handle. But orders didn't arrive evenly. They came in two waves, lunch and dinner, and the dinner wave on a festive Saturday was 6 times the average minute.

So we stopped planning around averages and planned around three numbers:

  • Peak orders per minute on the busiest Saturday and the busiest festive day.
  • p95 checkout time: how long the slowest 5% of diners waited after pressing "Pay".
  • Partner orders accepted within the partner's timeout: the number delivery platforms judged the outlet on.

What one order actually does

To see why the platform struggled, look at a single order. When a diner presses "Place order" on any channel, the monolith does roughly this:

  1. Load the outlet's menu, modifiers and prices to validate the cart.
  2. Check availability and the active menu period.
  3. Write the order and every order line to MySQL.
  4. Call the payment provider and wait.
  5. Write the payment result.
  6. Send the ticket to the kitchen display or printer.
  7. Update loyalty points.
  8. Send status updates back to the channel the order came from.

A delivery partner order adds its own work. The partner sends the order to our middleware. We map the partner's item IDs to our menu items, validate it, create the order, and answer before the partner's timeout, all while the outlet is in its own dinner rush.

At this stage, a Saturday peak meant about 1,250 orders a minute, and each order meant roughly 60 database queries. That put MySQL at about 1,250 queries a second at 7:30 on a Saturday. On Monday at the same time, it was barely awake.

What broke first

The symptoms showed up in the same order every bad weekend.

MySQL CPU climbed toward 100%. Queries that took milliseconds on Monday took seconds. App threads sat waiting on the database, so new requests queued behind them. Open connections climbed toward MySQL's limit, and once they hit it, new requests failed outright with connection errors.

Then it spread outward. Order placement hung on every channel. Checkout took 9 seconds at p95. Partner webhooks timed out, so partners either rejected the order or sent it again. A resent order is load we didn't need and a duplicate the kitchen doesn't want.

Monday always looked fine, which made the problem easy to postpone until the next weekend. We stopped doing that.

Where the lag came from

The slowdown wasn't one problem. It was a chain, and each link made the next one worse: orders spike on every channel at once, MySQL saturates, app threads wait, the connection pool runs out, checkout hangs, partners time out, and partners resend.

The chain had a loop in it. When partners resent orders that had timed out, they added load to the same database that caused the timeouts. A slow Saturday could turn itself into a failing one.

Two design choices made every link heavier.

Everything ran inside the diner's request. Validating the cart, writing the order and taking the payment had to finish before we answered. Sending the kitchen ticket, updating loyalty points and sending status back to partners did not. But all of it ran in the same request. The diner waited for work they didn't need to wait for, and every extra step held a thread and a database connection a little longer. Moving that work off the request path came later, in Parts 7 and 8.

Transactions stayed open too long. Parts of the order flow opened a database transaction and then called the payment provider inside it. While the provider took its time, the transaction kept its row locks and its connection. At peak, a slow provider meant other orders for the same outlet queued on those locks, and the connection pool drained faster. Keeping transactions short, with the payment call outside them, gave back a surprising amount of capacity.

How we found out: complaints or alerts

There are only two ways to learn that order placement is slow. Your users tell you, or your alerts do.

Early on, we heard it from users. A merchant called support. An outlet manager messaged their account manager. A delivery partner flagged rejected orders. By the time a complaint reached us, the slowdown had been running for 25 minutes. The diners it hit had already waited or left, and we were starting from a message that said "it's slow", with no detail on which outlet, which channel or since when.

Complaints are the worst monitoring system you can have. They arrive late, they're vague, and they only come from merchants upset enough to call. The quiet ones just leave at renewal.

So we made alerts the way we found out, every time. And we alerted on what diners and partners feel, not only on server health:

  • Order placement time at p95, per channel, above 3 seconds
  • Failed or timed-out orders above 2% in any five-minute window
  • Partner orders not acknowledged within the partner's timeout
  • MySQL connections above 80% of the limit, the early warning before errors start

The rule was simple: we should know before the merchant does. On a festive Saturday, an alert gives you minutes to act. A complaint gives you an apology to write.

That rule doesn't depend on size. With ten merchants or ten thousand, the first person to notice a problem should be your on-call engineer, not your customer. Part 6 covers how we built this out once the platform split into services.

How we fixed it, in order

We fixed it in stages. Each one bought time for the next, and the order was the strategy.

Stage 1: more RAM and CPU

The fastest fix was a bigger Cloud SQL instance and bigger app servers. RAM mattered most for MySQL. A larger InnoDB buffer pool kept more of the hot data, like today's orders and active menus, in memory instead of on disk.

It took an afternoon, needed no code change, and the next Saturday survived.

But it was a painkiller. Cloud SQL charges for its size every hour, so we paid for Saturday's capacity on every quiet Monday. It added about $1,800 a month. And there's always a biggest instance you can buy. We also put a billing alert on the instance so the cost was seen, not just felt, moved Cloud SQL's own maintenance window into our Monday window, and tested the high-availability failover instead of just switching it on.

Stage 2: indexes from real queries

Instead of guessing, we turned on MySQL's slow query log and ranked queries by total time spent, not by the slowest single query. A query that takes 40 milliseconds but runs on every order costs far more than a 3-second report that runs once a day.

The top of that list was almost all queries that ran on every order. EXPLAIN showed several of them scanning far more rows than they returned. We added composite indexes shaped to each exact query:

Query that ran on every order
An outlet's open orders for the kitchen screen
Index we added
(outlet_id, status, created_at)
Why those columns, in that order
Filter by outlet, then status, then read newest first
Query that ran on every order
Matching a partner's order to ours
Index we added
Unique (partner_id, partner_order_id)
Why those columns, in that order
Fast lookup, and a resent partner order can't create a second row
Query that ran on every order
Loading active items for an outlet's menu
Index we added
(outlet_id, menu_id, is_active)
Why those columns, in that order
Skips inactive items and other outlets' rows entirely

The second one did two jobs. It made the lookup fast, and the unique constraint stopped a resent partner order from becoming a duplicate in the kitchen.

Database CPU at peak dropped from 95% to 55% without another cent of hardware.

Indexes aren't free. Every index slows down every write to its table, and the orders table was our busiest writer. Each index also takes buffer pool memory. So we only added what the slow query log justified, and dropped indexes nothing used. Indexes were built with MySQL's online schema changes, not a locking ALTER. We reviewed index usage every quarter as traffic shifted, and alerted when a known slow query came back.

Building an index on a large table loads the database while it runs. That's where the weekly curve helped us. Monday morning became our window for index builds, migrations and risky deploys. Friday to Sunday, and every festive week, became a change freeze. Part 12 later made deploys safe enough to shrink that freeze, and Part 13 made them safe for every team, not just ours.

Stage 3: Redis for repeated reads

Even with indexes, the database spent its evening answering questions it had already answered. Menus, outlet settings and partner item mappings were read on every order and changed rarely. We cached them in Redis on Memorystore, and that one change took the biggest share of reads off MySQL. Memorystore sat on a private IP with AUTH and TLS, reachable only from the app's network, and only the platform ever wrote to it (Part 14).

Part 2 goes deep on this, because the hard part wasn't caching. It was knowing when to stop trusting the cache.

Stage 4: scale out, not up

Eventually the app server itself was the limit, and there was no bigger box worth buying. We ran several identical copies of the monolith behind Cloud Load Balancing, with a firewall layer in front. That firewall was Cloud Armor, and the load balancer handled TLS, health checks and connection draining, sending each request to any copy with no sticky sessions. The full edge and identity layer is Part 14.

That needed two changes first. The app had to stop keeping anything in its own memory between requests, so sessions moved to Redis and any copy could take any request. And we had to do the connection math. Every app copy opened its own pool of database connections. Ten copies with 20 connections each is 200 connections, and MySQL has a limit. Scaling out the app could take the database down. So every new copy came with a capped connection pool, and we watched total connections as closely as CPU.

Autoscaling on requests per second and CPU handled normal growth through the week. But it reacts in minutes, and a Saturday dinner rush builds in seconds. So for known peaks we scaled up on a schedule: Friday afternoon before the weekend, and days ahead of Chinese New Year and Christmas week.

How we sized for a peak we hadn't seen yet

Fixing last Saturday wasn't enough. We had to be ready for the next Chinese New Year before it arrived.

We kept the capacity model simple:

  1. Forecast the peak minute. Take the same festival last year, scale it by merchant growth since then, and add a safety margin of 30%.
  2. Turn orders into database load. Peak orders per minute times queries per order gave the queries per second MySQL had to survive.
  3. Do the connection math from the database down. App copies times pool size per copy had to stay well under MySQL's connection limit. We sized pools from the database's limit down, not from the app up.
  4. Test it before the real day. We replayed a recorded Saturday against a staging copy of the platform at 1x, 2x and 3x the traffic, and watched what broke first: order placement time, MySQL CPU, connections or partner response times.

The load test always found the next bottleneck before diners did. It also turned the scheduled scale-up from a guess into a plan: how many app copies, which database size, and from what time on which day.

The trade-off we made

Option
Keep buying bigger machines
Why it was tempting
No code change, done today
Why we didn't lead with it
Peak prices every Monday, and a hard ceiling
Option
Rebuild as microservices now
Why it was tempting
Looks like the right architecture
Why we didn't lead with it
Months of work, while a peak arrived every seven days
Option
Scale out the app first
Why it was tempting
Adds capacity fast
Why we didn't lead with it
The database was the bottleneck, and more app copies meant more connections hitting it
Option
Cheapest fix first: bigger box, then indexes, then cache, then scale out
Why it was tempting
Each step bought time for the next
Why we didn't lead with it
Our choice

With a small team and a peak every week, we chose the fastest fix over the most elegant one. The bigger machine was the wrong long-term answer and the right answer for that Saturday. It bought the weeks we needed to do indexes properly. Indexes were free capacity, so they came before any new system. Redis came before more servers because caching removes work, while more servers only spread it.

We didn't rebuild into microservices at this point. That came later, for a different reason, in Part 5.

What it cost the business

We stopped watching the total cloud bill and started watching infrastructure cost per order. A total bill always grows with the business. Cost per order tells you whether you're growing efficiently.

The two ends of the week pulled in opposite directions. On a normal Monday, cost per order was high, because we paid for capacity nobody used. On a festive Saturday, it was low, because every server was busy. Peak-month infrastructure spend was about 60% higher than in a normal month.

That made capacity a business decision, not only a technical one. Over-provision, and you quietly burn margin every Monday. Under-provision, and you fail the merchant on the Saturday night that pays for their week. We chose to pay for scheduled peak capacity and accept a higher Monday cost, because a failed Saturday was counted in merchants, not dollars. Cost per order was the running cost. The cost of building, in platform-team hours per new service, comes in Part 13.

What this part leaves for later

Part 1 is about keeping one monolith alive through Saturday. The deeper fixes each get their own part.

Gap left here
Reports and dashboards still reading from the primary database
Where it's covered
Part 3: read replicas
Gap left here
Duplicate orders and payments beyond one unique index
Where it's covered
Part 4: idempotency, locks and reconciliation
Gap left here
One failing feature taking every channel down
Where it's covered
Part 5: splitting the monolith
Gap left here
Following one order across services, and alerts after the split
Where it's covered
Part 6: logs and monitoring
Gap left here
Moving side work and partner calls into queues
Where it's covered
Parts 7 and 8
Gap left here
A zone failing, and restores nobody had tested
Where it's covered
Part 9: zones and regions
Gap left here
Tables growing until the indexes no longer fit in memory
Where it's covered
Part 10: data growth
Gap left here
One merchant's spike slowing everyone else
Where it's covered
Part 11: noisy tenants
Gap left here
The weekend change freeze as the only way to deploy safely
Where it's covered
Part 12: safe deploys
Gap left here
How every new service gets built, provisioned and deployed safely
Where it's covered
Part 13: the platform
Gap left here
Who authenticates callers, which web origins are allowed, and how services get credentials
Where it's covered
Part 14: the edge

What I took from it

Fix in order of cost: a bigger box to survive the weekend, indexes for free capacity, a cache to remove repeated work, then scale out.

And plan for the minute that matters. In an omnichannel platform, Saturday night is every channel's peak at once. Build for it, measure it per minute, and don't pay for it on Monday.

Next in the series: Part 2: The menu is the storefront