Platform at Scale, Part 9 · Every stateful piece in more than one zone before a second region, and rehearse the restore.
The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.
It's 8:10 on a festive Saturday when one GCP zone has a bad night. Not the whole region. One zone.
Cloud SQL's primary was set up for high availability, so it failed over to another zone within a minute, and orders kept going. But Memorystore and the RabbitMQ cluster lived in a single zone, because that was cheaper and nobody had asked the question. The menu cache disappeared. Devices kept showing their last menu and the rebuild limits from Part 2 protected MySQL, so that part held. The partner queues didn't. They stopped draining, and partner orders piled up in the outbox from Part 8.
The database failover worked perfectly. The platform still had a bad night, because of the pieces around the database.
The lesson wasn't "buy a second region". It was that availability is a chain, and a single-zone cache or broker is a broken link, no matter how healthy the database is.
Why we hadn't planned for it
Parts 1 to 8 were about capacity: Saturday is the peak, plan for the minute, buy the cheapest fix first. Availability is a different question, and we had been answering it with "GCP handles it".
GCP handles the parts you configure for high availability. It doesn't handle the parts you left in one zone to save $1,200 a month. It doesn't restore your data if you've never tested a restore. And it doesn't tell you which dependency fails first when a zone goes dark.
So before buying a second region, we did the boring thing first, the same way we did indexes before replicas.
What we fixed, in order
Stage 1: every stateful piece in more than one zone
Our app copies already ran across zones behind the load balancer (Part 1). The stateful pieces were the gap.
Cloud SQL high availability was already on. We turned it on for both Memorystore instances from Part 5, the catalog cache and sessions. We moved RabbitMQ to a cluster across three zones, with its queues replicated across them. Part 8's design survived unchanged: a queue per partner, a lane per outlet. Only where the broker ran changed.
This was configuration, not architecture. It cost $1,200 more a month and removed the exact failure that caused the incident.
Stage 2: restore drills, not backups
A backup you've never restored is a hope.
Once a quarter, we restored the primary database into a separate staging project and timed it. We wrote down two numbers we actually achieved, not the ones we assumed: how long until we were back (recovery time), and how much recent data we would have lost (recovery point).
The first drill took 6 hours. The point was that we learned that on a Tuesday, not on a Saturday.
Stage 3: decide what "down" means to the business
Before spending on a second region, we asked which failure we were actually insuring against:
| Failure | How often | What covers it | Recovery we aimed for |
|---|---|---|---|
| One zone fails | It happens | Every stateful piece across zones | Minutes |
| A whole GCP region fails | Rare, but not zero | A second region | 4 hours, losing at most 5 minutes of data |
| A dependency fails, like a payment provider or partner API | Common | Timeouts, fallbacks and queues (Parts 4, 5 and 8) | Not a region problem |
- Failure
- One zone fails
- How often
- It happens
- What covers it
- Every stateful piece across zones
- Recovery we aimed for
- Minutes
- Failure
- A whole GCP region fails
- How often
- Rare, but not zero
- What covers it
- A second region
- Recovery we aimed for
- 4 hours, losing at most 5 minutes of data
- Failure
- A dependency fails, like a payment provider or partner API
- How often
- Common
- What covers it
- Timeouts, fallbacks and queues (Parts 4, 5 and 8)
- Recovery we aimed for
- Not a region problem
Only the second row justified a second region. The first and third were cheaper to fix without one.
Stage 4: a warm standby region, for the database only
We added a cross-region replica of the orders and payments database, and a runbook to promote it. Backups and archived data went to multi-region Cloud Storage.
We didn't go active-active. Writes stayed in one region. Splitting writes across regions for an ordering platform means two places can accept the same order or payment, which reopens every problem Part 4 solved. We didn't need that at our size.
We rehearsed promoting the standby in our drills. A standby you've never promoted is a diagram, not a plan. Failover itself was a DNS and global load balancer switch, and the standby could authenticate and decrypt on day one because keys, secrets, permissions and Cloud Armor rules were replicated with it (Part 14). Only named people could promote it, and every promotion was audited.
Stage 5: a planned degraded mode for every dependency
Each dependency got an explicit answer to "what happens when this fails?"
| When this fails | The platform does this |
|---|---|
| Payment provider is slow | After its timeout, the payment moves to "unknown" and is resolved by asking the provider (Part 4). New checkouts get a clear message instead of waiting forever |
| A partner API is down | Orders wait in the outbox and the partner queue (Part 8), and the merchant is warned |
| The menu cache is down | Devices show their last menu, and rebuilds are limited (Part 2) |
| RabbitMQ is down | Partner sends wait in the outbox (Part 8) |
| Loyalty or recommendations are down | Circuit breakers skip them (Part 5) |
- When this fails
- Payment provider is slow
- The platform does this
- After its timeout, the payment moves to "unknown" and is resolved by asking the provider (Part 4). New checkouts get a clear message instead of waiting forever
- When this fails
- A partner API is down
- The platform does this
- Orders wait in the outbox and the partner queue (Part 8), and the merchant is warned
- When this fails
- The menu cache is down
- The platform does this
- Devices show their last menu, and rebuilds are limited (Part 2)
- When this fails
- RabbitMQ is down
- The platform does this
- Partner sends wait in the outbox (Part 8)
- When this fails
- Loyalty or recommendations are down
- The platform does this
- Circuit breakers skip them (Part 5)
A dependency outage became a degraded channel, never a platform outage.
The trade-off we made
| Option | Why it was tempting | Why we didn't lead with it |
|---|---|---|
| Single region, GCP will handle it | Cheapest, no work | The incident proved it wasn't true for single-zone dependencies |
| Multiple zones only | Cheap, covers the common case | Doesn't cover a regional outage. Fine until it isn't |
| Active-active across regions | Maximum availability | Writes in two regions is a hard consistency problem we didn't need at our scale |
| Warm standby region for the database | Covers a regional outage with limited data loss | Costs a replica and a runbook we have to rehearse |
| Every zone first, then a warm standby for the data, then drills | Cheapest first, sized to the failure we were insuring against | Our choice |
- Option
- Single region, GCP will handle it
- Why it was tempting
- Cheapest, no work
- Why we didn't lead with it
- The incident proved it wasn't true for single-zone dependencies
- Option
- Multiple zones only
- Why it was tempting
- Cheap, covers the common case
- Why we didn't lead with it
- Doesn't cover a regional outage. Fine until it isn't
- Option
- Active-active across regions
- Why it was tempting
- Maximum availability
- Why we didn't lead with it
- Writes in two regions is a hard consistency problem we didn't need at our scale
- Option
- Warm standby region for the database
- Why it was tempting
- Covers a regional outage with limited data loss
- Why we didn't lead with it
- Costs a replica and a runbook we have to rehearse
- Option
- Every zone first, then a warm standby for the data, then drills
- Why it was tempting
- Cheapest first, sized to the failure we were insuring against
- Why we didn't lead with it
- Our choice
We didn't buy active-active. We bought multiple zones for every stateful piece, a rehearsed restore, and a warm standby for the database. That covered the failures that actually happened and the one that would hurt most, without pretending we had a multi-region platform.
What we measured
- Recovery time and data loss in each restore drill: first 6 hours, latest 75 minutes
- Time to promote the standby region in a drill: 40 minutes
- Diner-visible impact of zone failures after Stage 1: under a minute of slower checkouts
- Share of dependencies with a written, tested degraded mode: 90%
What it cost
Multi-zone Redis and RabbitMQ added about $1,200 a month. The cross-region replica added about $1,000 a month. Drills cost engineering days that shipped no features. In return, a zone failure became a blip, and a regional outage became a planned failover instead of a postmortem.
What this part leaves for later
| Gap left here | Where it's covered |
|---|---|
| Restores getting slower as the database grows | Part 10: the table that grew forever |
| Writing in two regions at once, if the business ever needs it | Outside this series. It interacts with Part 4's idempotency and Part 5's service boundaries |
| Where data is allowed to live | A legal and security decision, not only an engineering one |
| Multiple zones and the warm standby as defaults for every new service | Part 13: the platform |
| The global load balancer as the failover switch, and replicated keys and secrets | Part 14: the edge |
- Gap left here
- Restores getting slower as the database grows
- Where it's covered
- Part 10: the table that grew forever
- Gap left here
- Writing in two regions at once, if the business ever needs it
- Where it's covered
- Outside this series. It interacts with Part 4's idempotency and Part 5's service boundaries
- Gap left here
- Where data is allowed to live
- Where it's covered
- A legal and security decision, not only an engineering one
- Gap left here
- Multiple zones and the warm standby as defaults for every new service
- Where it's covered
- Part 13: the platform
- Gap left here
- The global load balancer as the failover switch, and replicated keys and secrets
- Where it's covered
- Part 14: the edge
What I took from it
Availability is a chain of dependencies. The cheapest win is putting every stateful piece in more than one zone before you buy a second region.
Know your recovery time and data loss targets before you pay for them, and rehearse the restore you're counting on. A warm standby you've never promoted is a diagram, not a plan.
Next in the series: Part 10: The table that grew forever