Back to all articles
Platform at ScalePart 05
System designMicroservicesArchitectureResiliencePlatform

Platform at Scale, Part 5: One Service Down, Everything Down

Janahan Thavendran10 min read
In this article

Platform at Scale, Part 5 · Contain failure, and split the resources, not just the code.

The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.

It's Saturday dinner. A loyalty points lookup starts to slow down, because one query on the points table has grown too heavy. Each slow lookup holds its database connection a little longer. Within minutes, order placement stops on every channel.

The diners weren't trying to check their points. They were trying to pay for dinner. But in a monolith, everything shares one process, one connection pool and one fate. It was Part 1's lag chain again, except this time the trigger was a feature nobody thought of as critical.

Why the monolith stopped being enough

By now the monolith carried many products: dine-in ordering on table tablets, the mobile app and website, the delivery partner middleware, payments, loyalty, recommendations and the merchant back office.

Three problems grew with it:

  • Shared failure. A slow or broken part could use up resources every other part needed. The least important feature could take down the most important one.
  • Shared deploys. Every change shipped with everything else. With weekends frozen (Part 1), every team queued for the same Monday window, and a bad change to loyalty was a bad change to ordering.
  • Shared ownership. Many teams changed the same code. Every release needed more coordination, and moving fast meant moving carefully.

The question we split on

We didn't ask what a good microservice looks like. We asked one question: what must keep working when this part breaks?

The business answer came straight from Part 1. Saturday night is every channel at once. Losing one channel for a few minutes is survivable. Losing all of them is not. So the goal wasn't many services. It was making sure no single failure could stop every sale.

The shape: a small core, split by domain, with products on top

The core held what every product depends on and nobody can afford to get wrong. We didn't make it one big service, or a catalog bug could still take down payments one layer lower. We split it into four services by domain, and no further, because the team had to own and run every one of them.

Layer
Core
Service
Orders
What it owns
Orders, order lines, kitchen tickets
If it fails
No new orders on any channel. The service we protected most
Layer
Core
Service
Payments
What it owns
Payment intents, states, refunds (Part 4)
If it fails
Checkout stops. Orders already placed are safe
Layer
Core
Service
Catalog
What it owns
Menus, availability, outlet settings, partner item mappings (Part 2)
If it fails
Menus can't change. Cached menus keep serving
Layer
Core
Service
Merchant
What it owns
Merchants, outlets, staff, devices
If it fails
Back office and new device sign-ins stop. Ordering carries on
Layer
Product
Service
Dine-in ordering
What it owns
Table tablet ordering
If it fails
That channel stops. Every other channel carries on
Layer
Product
Service
App and web ordering
What it owns
Mobile app and website orders
If it fails
That channel stops
Layer
Product
Service
Partner middleware
What it owns
Delivery partner orders and menu sync
If it fails
Partner orders wait or time out. Dine-in carries on
Layer
Product
Service
Loyalty
What it owns
Points and rewards
If it fails
Orders go through. Points arrive later
Layer
Product
Service
Recommendations
What it owns
Dish suggestions
If it fails
The menu shows without suggestions

The honest rule

One rule held it together: products depend on the core, and the core never depends on a product.

It's tempting to add "and a product failing never stops a sale". That's only true for some products. If the partner middleware is down, partner orders don't arrive, and that counts against the outlet. So we wrote the honest version: a product failing degrades its own channel, never every channel.

That split products into two kinds:

  • Revenue channels: dine-in ordering, app and web ordering, and the partner middleware. A failure loses that channel's orders, so each got its own targets for peak nights: order placement success above 99.5%, p95 placement time under 3 seconds, and partner orders acknowledged within each partner's timeout.
  • Enhancements: loyalty and recommendations. They can fail without stopping a single sale, so they were built to fail quietly.

Splitting the code wasn't enough. We had to split the resources

Part 1's lag chain was a connection pool running dry. If every service still shared one MySQL database and one pool, splitting the code would stop crashes from spreading, but not slowdowns. Loyalty's slow query could still drain the connections that orders needed.

So the split went all the way down:

  • Databases. Orders and payments shared one Cloud SQL instance, because they're written together and must stay consistent. Catalog and merchant had their own. Each product had its own small database. No service read another service's tables. It asked through an API. Those APIs sat behind one gateway that authenticated the caller, enforced tenant quotas and versioned the contract, and calls between services inside the network carried mutual TLS (Part 14). A network call is not a trusted call.
  • Connection pools. Every service had its own capped pool. On each database instance, the caps of every service using it added up to well under MySQL's limit: Part 1's connection math, done per instance.
  • Redis. The menu cache from Part 2 belonged to the catalog service, on its own Memorystore instance. Sessions lived on another. A flood of keys in one could no longer push another service's data out of memory.
  • Replicas. The replicas from Part 3 stayed with the orders and payments database, where the dashboards and reports needed them.

Calls between services

A function call became a network call, and network calls fail in ways function calls never did. Every call between services followed four rules:

  1. Timeouts everywhere. Every call had a timeout shorter than the caller's own deadline, so one slow service couldn't hold the caller hostage.
  2. Circuit breakers on enhancements. After repeated failures, a caller stopped calling for a short time and used its fallback: no points shown, no suggestions. The failing service got room to recover instead of a flood of retries.
  3. Retries only when safe. Part 4's idempotency keys now travelled with every request from a product to the core. A retried order or payment hit the same unique constraints and couldn't create a duplicate.
  4. The core never calls a product. When an order was placed, the orders service published an "order placed" event. Loyalty listened and added the points. If loyalty was down, the events waited and the points arrived later. Partner status updates worked the same way. Parts 7 and 8 cover the event and queue systems behind this.

How we moved: one piece at a time

We didn't rewrite the platform. We carved pieces out of the monolith while it kept running, in order of risk:

  1. Loyalty and recommendations first. Low risk, noisy, and the perfect place to prove the pattern.
  2. Partner middleware next. The biggest risk to outlets on peak nights.
  3. The core last: catalog, then merchant, then payments and orders together.

Each move followed the same steps. Copy the data and keep it in sync. Send a small share of traffic to the new service and compare its results with the monolith. Switch reads, then switch writes. Every cut-over happened in the Monday window, never in a festive week. The monolith shrank until it was gone. The split created the next problem: how a small team creates, deploys and owns a new service without reinventing everything. That became Part 13.

Where it hurt

Calls were slower than function calls, and failures got stranger: timeouts, partial failures, a call that succeeded on the other side but whose response never came back.

Data got harder. A question that used to be one SQL join became several API calls stitched together. Reports that needed data from many services couldn't join live databases anymore, so they moved to a reporting store fed by events.

Consistency changed too. An order and its loyalty points were no longer saved in one transaction. Points could arrive seconds or minutes later. We accepted that for anything that wasn't money, and kept orders and payments together for exactly this reason.

And when something went wrong, finding where became much harder. That became its own problem, in Part 6.

The trade-off we made

Option
Keep the monolith, tidy it into modules
Why it was tempting
Cheapest, no new infrastructure
Why we didn't choose it
Modules still share one process and one pool, so one failure still stops everything
Option
A full rewrite
Why it was tempting
A clean start
Why we didn't choose it
Freezes the product for months, with the riskiest launch at the end
Option
Many small services, one per feature
Why it was tempting
Maximum independence
Why we didn't choose it
Far more services than our team could run, monitor and be on call for
Option
Split the code, keep one shared database
Why it was tempting
Faster to migrate
Why we didn't choose it
Failures still spread through the shared connection pool
Option
A core of four services, products on top, separate databases and pools, moved one piece at a time
Why it was tempting
Contains failure, sized to the team
Why we didn't choose it
Our choice

Team capacity decided the shape. With the engineers we had, a handful of well-owned services was safer than many thin ones nobody fully owned. Moving piece by piece meant we kept shipping features during the migration.

We gave up the simplicity of one codebase, one deploy and one transaction. In return, a broken product stopped meaning a broken platform.

What we measured

  • Incidents where one product's failure affected other channels: from 6 a quarter to one
  • Order placement success per channel on peak nights: 99.7%
  • Deploys per team per week: from 1 to 8
  • Time to recover from an incident: from 90 to 25 minutes

What it cost

More services meant more deploy pipelines, more databases, more infrastructure and more on-call surface. The migration took 14 months alongside normal feature work. The recurring cost was setting up every new service by hand, which Part 13 removed. Infrastructure cost rose by 25%, because services stopped sharing resources.

What we got back: a broken enhancement no longer stopped orders, a broken channel no longer stopped the others, and teams could ship on their own schedule.

What this part leaves for later

Gap left here
Following one order across services, and alerts per service
Where it's covered
Part 6: logs and monitoring
Gap left here
Events between services: delivery, duplicates and retries
Where it's covered
Part 7: events and Cloud Run
Gap left here
Partner calls and status updates through queues
Where it's covered
Part 8: partner queues
Gap left here
Tenants still sharing everything inside each service
Where it's covered
Part 11: noisy tenants
Gap left here
Changing a service's API without breaking its callers
Where it's covered
Part 12: safe deploys
Gap left here
Creating and owning new services without reinventing the pipeline
Where it's covered
Part 13: the platform
Gap left here
Identity between services, the API gateway, and why no service mesh
Where it's covered
Part 14: the edge

What I took from it

Microservices aren't the goal. Containing failure is.

Before splitting anything, ask what must keep working when this part breaks. Then split the resources, not just the code, because a shared connection pool will carry a failure straight past every service boundary you drew.

Next in the series: Part 6: You can't fix what you can't see