Back to all articles
Platform at ScalePart 09
Disaster recoveryAvailabilityGCPSystem designReliability

Platform at Scale, Part 9: The Region That Wasn't There

Janahan Thavendran8 min read
In this article

Platform at Scale, Part 9 · Every stateful piece in more than one zone before a second region, and rehearse the restore.

The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.

It's 8:10 on a festive Saturday when one GCP zone has a bad night. Not the whole region. One zone.

Cloud SQL's primary was set up for high availability, so it failed over to another zone within a minute, and orders kept going. But Memorystore and the RabbitMQ cluster lived in a single zone, because that was cheaper and nobody had asked the question. The menu cache disappeared. Devices kept showing their last menu and the rebuild limits from Part 2 protected MySQL, so that part held. The partner queues didn't. They stopped draining, and partner orders piled up in the outbox from Part 8.

The database failover worked perfectly. The platform still had a bad night, because of the pieces around the database.

The lesson wasn't "buy a second region". It was that availability is a chain, and a single-zone cache or broker is a broken link, no matter how healthy the database is.

Why we hadn't planned for it

Parts 1 to 8 were about capacity: Saturday is the peak, plan for the minute, buy the cheapest fix first. Availability is a different question, and we had been answering it with "GCP handles it".

GCP handles the parts you configure for high availability. It doesn't handle the parts you left in one zone to save $1,200 a month. It doesn't restore your data if you've never tested a restore. And it doesn't tell you which dependency fails first when a zone goes dark.

So before buying a second region, we did the boring thing first, the same way we did indexes before replicas.

What we fixed, in order

Stage 1: every stateful piece in more than one zone

Our app copies already ran across zones behind the load balancer (Part 1). The stateful pieces were the gap.

Cloud SQL high availability was already on. We turned it on for both Memorystore instances from Part 5, the catalog cache and sessions. We moved RabbitMQ to a cluster across three zones, with its queues replicated across them. Part 8's design survived unchanged: a queue per partner, a lane per outlet. Only where the broker ran changed.

This was configuration, not architecture. It cost $1,200 more a month and removed the exact failure that caused the incident.

Stage 2: restore drills, not backups

A backup you've never restored is a hope.

Once a quarter, we restored the primary database into a separate staging project and timed it. We wrote down two numbers we actually achieved, not the ones we assumed: how long until we were back (recovery time), and how much recent data we would have lost (recovery point).

The first drill took 6 hours. The point was that we learned that on a Tuesday, not on a Saturday.

Stage 3: decide what "down" means to the business

Before spending on a second region, we asked which failure we were actually insuring against:

Failure
One zone fails
How often
It happens
What covers it
Every stateful piece across zones
Recovery we aimed for
Minutes
Failure
A whole GCP region fails
How often
Rare, but not zero
What covers it
A second region
Recovery we aimed for
4 hours, losing at most 5 minutes of data
Failure
A dependency fails, like a payment provider or partner API
How often
Common
What covers it
Timeouts, fallbacks and queues (Parts 4, 5 and 8)
Recovery we aimed for
Not a region problem

Only the second row justified a second region. The first and third were cheaper to fix without one.

Stage 4: a warm standby region, for the database only

We added a cross-region replica of the orders and payments database, and a runbook to promote it. Backups and archived data went to multi-region Cloud Storage.

We didn't go active-active. Writes stayed in one region. Splitting writes across regions for an ordering platform means two places can accept the same order or payment, which reopens every problem Part 4 solved. We didn't need that at our size.

We rehearsed promoting the standby in our drills. A standby you've never promoted is a diagram, not a plan. Failover itself was a DNS and global load balancer switch, and the standby could authenticate and decrypt on day one because keys, secrets, permissions and Cloud Armor rules were replicated with it (Part 14). Only named people could promote it, and every promotion was audited.

Stage 5: a planned degraded mode for every dependency

Each dependency got an explicit answer to "what happens when this fails?"

When this fails
Payment provider is slow
The platform does this
After its timeout, the payment moves to "unknown" and is resolved by asking the provider (Part 4). New checkouts get a clear message instead of waiting forever
When this fails
A partner API is down
The platform does this
Orders wait in the outbox and the partner queue (Part 8), and the merchant is warned
When this fails
The menu cache is down
The platform does this
Devices show their last menu, and rebuilds are limited (Part 2)
When this fails
RabbitMQ is down
The platform does this
Partner sends wait in the outbox (Part 8)
When this fails
Loyalty or recommendations are down
The platform does this
Circuit breakers skip them (Part 5)

A dependency outage became a degraded channel, never a platform outage.

The trade-off we made

Option
Single region, GCP will handle it
Why it was tempting
Cheapest, no work
Why we didn't lead with it
The incident proved it wasn't true for single-zone dependencies
Option
Multiple zones only
Why it was tempting
Cheap, covers the common case
Why we didn't lead with it
Doesn't cover a regional outage. Fine until it isn't
Option
Active-active across regions
Why it was tempting
Maximum availability
Why we didn't lead with it
Writes in two regions is a hard consistency problem we didn't need at our scale
Option
Warm standby region for the database
Why it was tempting
Covers a regional outage with limited data loss
Why we didn't lead with it
Costs a replica and a runbook we have to rehearse
Option
Every zone first, then a warm standby for the data, then drills
Why it was tempting
Cheapest first, sized to the failure we were insuring against
Why we didn't lead with it
Our choice

We didn't buy active-active. We bought multiple zones for every stateful piece, a rehearsed restore, and a warm standby for the database. That covered the failures that actually happened and the one that would hurt most, without pretending we had a multi-region platform.

What we measured

  • Recovery time and data loss in each restore drill: first 6 hours, latest 75 minutes
  • Time to promote the standby region in a drill: 40 minutes
  • Diner-visible impact of zone failures after Stage 1: under a minute of slower checkouts
  • Share of dependencies with a written, tested degraded mode: 90%

What it cost

Multi-zone Redis and RabbitMQ added about $1,200 a month. The cross-region replica added about $1,000 a month. Drills cost engineering days that shipped no features. In return, a zone failure became a blip, and a regional outage became a planned failover instead of a postmortem.

What this part leaves for later

Gap left here
Restores getting slower as the database grows
Where it's covered
Part 10: the table that grew forever
Gap left here
Writing in two regions at once, if the business ever needs it
Where it's covered
Outside this series. It interacts with Part 4's idempotency and Part 5's service boundaries
Gap left here
Where data is allowed to live
Where it's covered
A legal and security decision, not only an engineering one
Gap left here
Multiple zones and the warm standby as defaults for every new service
Where it's covered
Part 13: the platform
Gap left here
The global load balancer as the failover switch, and replicated keys and secrets
Where it's covered
Part 14: the edge

What I took from it

Availability is a chain of dependencies. The cheapest win is putting every stateful piece in more than one zone before you buy a second region.

Know your recovery time and data loss targets before you pay for them, and rehearse the restore you're counting on. A warm standby you've never promoted is a diagram, not a plan.

Next in the series: Part 10: The table that grew forever