Platform at Scale, Part 5 · Contain failure, and split the resources, not just the code.
The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.
It's Saturday dinner. A loyalty points lookup starts to slow down, because one query on the points table has grown too heavy. Each slow lookup holds its database connection a little longer. Within minutes, order placement stops on every channel.
The diners weren't trying to check their points. They were trying to pay for dinner. But in a monolith, everything shares one process, one connection pool and one fate. It was Part 1's lag chain again, except this time the trigger was a feature nobody thought of as critical.
Why the monolith stopped being enough
By now the monolith carried many products: dine-in ordering on table tablets, the mobile app and website, the delivery partner middleware, payments, loyalty, recommendations and the merchant back office.
Three problems grew with it:
- Shared failure. A slow or broken part could use up resources every other part needed. The least important feature could take down the most important one.
- Shared deploys. Every change shipped with everything else. With weekends frozen (Part 1), every team queued for the same Monday window, and a bad change to loyalty was a bad change to ordering.
- Shared ownership. Many teams changed the same code. Every release needed more coordination, and moving fast meant moving carefully.
The question we split on
We didn't ask what a good microservice looks like. We asked one question: what must keep working when this part breaks?
The business answer came straight from Part 1. Saturday night is every channel at once. Losing one channel for a few minutes is survivable. Losing all of them is not. So the goal wasn't many services. It was making sure no single failure could stop every sale.
The shape: a small core, split by domain, with products on top
The core held what every product depends on and nobody can afford to get wrong. We didn't make it one big service, or a catalog bug could still take down payments one layer lower. We split it into four services by domain, and no further, because the team had to own and run every one of them.
| Layer | Service | What it owns | If it fails |
|---|---|---|---|
| Core | Orders | Orders, order lines, kitchen tickets | No new orders on any channel. The service we protected most |
| Core | Payments | Payment intents, states, refunds (Part 4) | Checkout stops. Orders already placed are safe |
| Core | Catalog | Menus, availability, outlet settings, partner item mappings (Part 2) | Menus can't change. Cached menus keep serving |
| Core | Merchant | Merchants, outlets, staff, devices | Back office and new device sign-ins stop. Ordering carries on |
| Product | Dine-in ordering | Table tablet ordering | That channel stops. Every other channel carries on |
| Product | App and web ordering | Mobile app and website orders | That channel stops |
| Product | Partner middleware | Delivery partner orders and menu sync | Partner orders wait or time out. Dine-in carries on |
| Product | Loyalty | Points and rewards | Orders go through. Points arrive later |
| Product | Recommendations | Dish suggestions | The menu shows without suggestions |
- Layer
- Core
- Service
- Orders
- What it owns
- Orders, order lines, kitchen tickets
- If it fails
- No new orders on any channel. The service we protected most
- Layer
- Core
- Service
- Payments
- What it owns
- Payment intents, states, refunds (Part 4)
- If it fails
- Checkout stops. Orders already placed are safe
- Layer
- Core
- Service
- Catalog
- What it owns
- Menus, availability, outlet settings, partner item mappings (Part 2)
- If it fails
- Menus can't change. Cached menus keep serving
- Layer
- Core
- Service
- Merchant
- What it owns
- Merchants, outlets, staff, devices
- If it fails
- Back office and new device sign-ins stop. Ordering carries on
- Layer
- Product
- Service
- Dine-in ordering
- What it owns
- Table tablet ordering
- If it fails
- That channel stops. Every other channel carries on
- Layer
- Product
- Service
- App and web ordering
- What it owns
- Mobile app and website orders
- If it fails
- That channel stops
- Layer
- Product
- Service
- Partner middleware
- What it owns
- Delivery partner orders and menu sync
- If it fails
- Partner orders wait or time out. Dine-in carries on
- Layer
- Product
- Service
- Loyalty
- What it owns
- Points and rewards
- If it fails
- Orders go through. Points arrive later
- Layer
- Product
- Service
- Recommendations
- What it owns
- Dish suggestions
- If it fails
- The menu shows without suggestions
The honest rule
One rule held it together: products depend on the core, and the core never depends on a product.
It's tempting to add "and a product failing never stops a sale". That's only true for some products. If the partner middleware is down, partner orders don't arrive, and that counts against the outlet. So we wrote the honest version: a product failing degrades its own channel, never every channel.
That split products into two kinds:
- Revenue channels: dine-in ordering, app and web ordering, and the partner middleware. A failure loses that channel's orders, so each got its own targets for peak nights: order placement success above 99.5%, p95 placement time under 3 seconds, and partner orders acknowledged within each partner's timeout.
- Enhancements: loyalty and recommendations. They can fail without stopping a single sale, so they were built to fail quietly.
Splitting the code wasn't enough. We had to split the resources
Part 1's lag chain was a connection pool running dry. If every service still shared one MySQL database and one pool, splitting the code would stop crashes from spreading, but not slowdowns. Loyalty's slow query could still drain the connections that orders needed.
So the split went all the way down:
- Databases. Orders and payments shared one Cloud SQL instance, because they're written together and must stay consistent. Catalog and merchant had their own. Each product had its own small database. No service read another service's tables. It asked through an API. Those APIs sat behind one gateway that authenticated the caller, enforced tenant quotas and versioned the contract, and calls between services inside the network carried mutual TLS (Part 14). A network call is not a trusted call.
- Connection pools. Every service had its own capped pool. On each database instance, the caps of every service using it added up to well under MySQL's limit: Part 1's connection math, done per instance.
- Redis. The menu cache from Part 2 belonged to the catalog service, on its own Memorystore instance. Sessions lived on another. A flood of keys in one could no longer push another service's data out of memory.
- Replicas. The replicas from Part 3 stayed with the orders and payments database, where the dashboards and reports needed them.
Calls between services
A function call became a network call, and network calls fail in ways function calls never did. Every call between services followed four rules:
- Timeouts everywhere. Every call had a timeout shorter than the caller's own deadline, so one slow service couldn't hold the caller hostage.
- Circuit breakers on enhancements. After repeated failures, a caller stopped calling for a short time and used its fallback: no points shown, no suggestions. The failing service got room to recover instead of a flood of retries.
- Retries only when safe. Part 4's idempotency keys now travelled with every request from a product to the core. A retried order or payment hit the same unique constraints and couldn't create a duplicate.
- The core never calls a product. When an order was placed, the orders service published an "order placed" event. Loyalty listened and added the points. If loyalty was down, the events waited and the points arrived later. Partner status updates worked the same way. Parts 7 and 8 cover the event and queue systems behind this.
How we moved: one piece at a time
We didn't rewrite the platform. We carved pieces out of the monolith while it kept running, in order of risk:
- Loyalty and recommendations first. Low risk, noisy, and the perfect place to prove the pattern.
- Partner middleware next. The biggest risk to outlets on peak nights.
- The core last: catalog, then merchant, then payments and orders together.
Each move followed the same steps. Copy the data and keep it in sync. Send a small share of traffic to the new service and compare its results with the monolith. Switch reads, then switch writes. Every cut-over happened in the Monday window, never in a festive week. The monolith shrank until it was gone. The split created the next problem: how a small team creates, deploys and owns a new service without reinventing everything. That became Part 13.
Where it hurt
Calls were slower than function calls, and failures got stranger: timeouts, partial failures, a call that succeeded on the other side but whose response never came back.
Data got harder. A question that used to be one SQL join became several API calls stitched together. Reports that needed data from many services couldn't join live databases anymore, so they moved to a reporting store fed by events.
Consistency changed too. An order and its loyalty points were no longer saved in one transaction. Points could arrive seconds or minutes later. We accepted that for anything that wasn't money, and kept orders and payments together for exactly this reason.
And when something went wrong, finding where became much harder. That became its own problem, in Part 6.
The trade-off we made
| Option | Why it was tempting | Why we didn't choose it |
|---|---|---|
| Keep the monolith, tidy it into modules | Cheapest, no new infrastructure | Modules still share one process and one pool, so one failure still stops everything |
| A full rewrite | A clean start | Freezes the product for months, with the riskiest launch at the end |
| Many small services, one per feature | Maximum independence | Far more services than our team could run, monitor and be on call for |
| Split the code, keep one shared database | Faster to migrate | Failures still spread through the shared connection pool |
| A core of four services, products on top, separate databases and pools, moved one piece at a time | Contains failure, sized to the team | Our choice |
- Option
- Keep the monolith, tidy it into modules
- Why it was tempting
- Cheapest, no new infrastructure
- Why we didn't choose it
- Modules still share one process and one pool, so one failure still stops everything
- Option
- A full rewrite
- Why it was tempting
- A clean start
- Why we didn't choose it
- Freezes the product for months, with the riskiest launch at the end
- Option
- Many small services, one per feature
- Why it was tempting
- Maximum independence
- Why we didn't choose it
- Far more services than our team could run, monitor and be on call for
- Option
- Split the code, keep one shared database
- Why it was tempting
- Faster to migrate
- Why we didn't choose it
- Failures still spread through the shared connection pool
- Option
- A core of four services, products on top, separate databases and pools, moved one piece at a time
- Why it was tempting
- Contains failure, sized to the team
- Why we didn't choose it
- Our choice
Team capacity decided the shape. With the engineers we had, a handful of well-owned services was safer than many thin ones nobody fully owned. Moving piece by piece meant we kept shipping features during the migration.
We gave up the simplicity of one codebase, one deploy and one transaction. In return, a broken product stopped meaning a broken platform.
What we measured
- Incidents where one product's failure affected other channels: from 6 a quarter to one
- Order placement success per channel on peak nights: 99.7%
- Deploys per team per week: from 1 to 8
- Time to recover from an incident: from 90 to 25 minutes
What it cost
More services meant more deploy pipelines, more databases, more infrastructure and more on-call surface. The migration took 14 months alongside normal feature work. The recurring cost was setting up every new service by hand, which Part 13 removed. Infrastructure cost rose by 25%, because services stopped sharing resources.
What we got back: a broken enhancement no longer stopped orders, a broken channel no longer stopped the others, and teams could ship on their own schedule.
What this part leaves for later
| Gap left here | Where it's covered |
|---|---|
| Following one order across services, and alerts per service | Part 6: logs and monitoring |
| Events between services: delivery, duplicates and retries | Part 7: events and Cloud Run |
| Partner calls and status updates through queues | Part 8: partner queues |
| Tenants still sharing everything inside each service | Part 11: noisy tenants |
| Changing a service's API without breaking its callers | Part 12: safe deploys |
| Creating and owning new services without reinventing the pipeline | Part 13: the platform |
| Identity between services, the API gateway, and why no service mesh | Part 14: the edge |
- Gap left here
- Following one order across services, and alerts per service
- Where it's covered
- Part 6: logs and monitoring
- Gap left here
- Events between services: delivery, duplicates and retries
- Where it's covered
- Part 7: events and Cloud Run
- Gap left here
- Partner calls and status updates through queues
- Where it's covered
- Part 8: partner queues
- Gap left here
- Tenants still sharing everything inside each service
- Where it's covered
- Part 11: noisy tenants
- Gap left here
- Changing a service's API without breaking its callers
- Where it's covered
- Part 12: safe deploys
- Gap left here
- Creating and owning new services without reinventing the pipeline
- Where it's covered
- Part 13: the platform
- Gap left here
- Identity between services, the API gateway, and why no service mesh
- Where it's covered
- Part 14: the edge
What I took from it
Microservices aren't the goal. Containing failure is.
Before splitting anything, ask what must keep working when this part breaks. Then split the resources, not just the code, because a shared connection pool will carry a failure straight past every service boundary you drew.
Next in the series: Part 6: You can't fix what you can't see