Back to all articles
Platform at ScalePart 12
DeploymentFeature flagsSRESystem designCI/CD

Platform at Scale, Part 12: Deploying on a Saturday

Janahan Thavendran7 min read
In this article

Platform at Scale, Part 12 · Deploys small, judged by what customers feel, and reversible by design.

The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.

It's the Thursday before a festive weekend, and Part 1's change freeze is in force. A merchant reports a checkout bug that drops the tip on one channel. The fix is small. Someone decides it's small enough for an exception and ships it on Thursday night, together with a database change the old app servers don't understand.

Checkout breaks for 35 minutes on the busiest night of the week. Rolling back takes 50 minutes, because nobody had ever rehearsed it, and the old code can't run against the new database.

The change freeze didn't fail. It was the only deployment strategy we had, and it couldn't handle a small fix under pressure.

Why a freeze is not a strategy

A change freeze avoids deploying. It doesn't make deploying safe. Under a freeze, the only deploys that happen are exceptions under pressure, which is the worst possible time to deploy without gradual rollouts, flags or a rehearsed rollback.

Part 1 used the freeze correctly, to protect peak weekends while the platform was fragile. This part is about making deploys safe enough that the freeze becomes a choice, not a crutch.

What we fixed, in order

Stage 1: feature flags

The tip fix didn't need a risky deploy. It needed a flag: ship the code switched off, turn it on for the affected channel at a chosen minute, and switch it off again in seconds if anything looks wrong.

Flags separated two questions that had always been tied together: when the code lands, and when the behaviour changes. Those carry very different risks.

Flags also create their own debt. Every flag had an owner and an expiry date, and we audited them every month. Only named roles could flip a production flag, and every flip was recorded.

Stage 2: gradual rollouts, judged by what diners feel

Part 6 defined order placement, payment and partner acknowledgement success the same way for every team. We used exactly those numbers to judge a release.

A new version took 1% of traffic for 15 minutes, then 10%, then 50%, then all of it. At every step, its order placement success, payment success and p95 checkout time were compared with the old version, per channel (Part 5). If they got worse, the rollout reversed itself automatically. The rollout watched what diners felt, not what the CPU did. Traffic was split at the gateway, by header, not only by load balancer weight (Part 14).

Stage 3: database changes in two halves

The change that broke the rollback was made in one step. We switched to a pattern called expand and contract:

  1. Add the new column, leaving the old one in place.
  2. Deploy code that writes to both.
  3. Fill in the new column for existing rows.
  4. Switch reads to the new column.
  5. In a later release, remove the old column.

At no point does a running version meet a database it doesn't understand, so the previous version can always run. That's what made rollback safe. Between services (Part 5), the same rule applied to APIs: add the new field, move every caller over, and only then remove the old one.

Stage 4: rollbacks we had already done

We rolled back on purpose, in staging, on a schedule, and timed it. Rollback became routine instead of improvisation. If a gradual rollout breached its numbers, rollback happened automatically. If a person had to do it, they had done it before.

Stage 5: error budgets decide when to stop

Part 6 already used error budgets to decide when to wake someone. Now the deploy pipeline read them too. If a service had used up its error budget, new deploys to it were blocked automatically, except changes that improved reliability, until the budget recovered. Not by memo, by pipeline.

The festive freeze didn't disappear. It shrank to 3 days around the peak, and only for big changes. Flags and small, gradually rolled-out changes carried on. The freeze became a measurement instead of a calendar. Error budgets governed our own deploys here. Making them govern every team's deploys, automatically, is Part 13.

The trade-off we made

Option
Keep the calendar freeze
Why it was tempting
Simple, no new tooling
Why we didn't choose it
The only deploys that happen are exceptions under pressure
Option
A full second copy of every service for each release (blue/green)
Why it was tempting
A clean switch and instant rollback
Why we didn't choose it
Doubles the running footprint for every deploy. Heavy for a small team
Option
Gradual rollouts on every deploy
Why it was tempting
Catches problems early
Why we didn't choose it
Needs outcome metrics and automatic rollback, which we already had from Part 6
Option
Feature flags everywhere
Why it was tempting
Separates deploying from changing behaviour
Why we didn't choose it
Flag debt is real. Flags need owners and expiry dates
Option
Flags, gradual rollouts, two-step database changes, rehearsed rollbacks and error budgets
Why it was tempting
Deploys are small and reversible on any day
Why we didn't choose it
Our choice, with more process

We didn't build a second copy of the whole platform for every release. We made small, reversible deploys the default, and let the error budget decide when to stop.

What we measured

  • Deploys per week: from 3 to 40
  • Share of deploys that caused an incident: from 12% to 2%
  • Time to roll back: from 50 minutes to 3 minutes
  • Deploy-caused incidents in festive weeks: one, caught at 1% and rolled back automatically
  • Flags past their expiry at each monthly audit: under 5

What it cost

The pipeline work took 10 weeks of engineering. Gradual rollouts made each deploy about 45 minutes slower. Flags added a new kind of debt that needed managing every month. The pipeline itself ran as a least-privilege deployer with short-lived credentials, never as a project owner, and every image was built from pinned base images and signed before it could deploy.

In return, the Thursday-night exception became a flag flip, and a rollback that took 50 minutes became a button that had been pressed before.

What this part leaves for later

Gap left here
Managing flags at scale: ownership, expiry and testing
Where it's covered
Part 13: policy as code
Gap left here
Safe deploys for every team, not just ours
Where it's covered
Part 13: golden paths
Gap left here
Schema changes across many services at once
Where it's covered
Part 13, and an open question for API governance
Gap left here
The pipeline's own identity, and gradual rollouts routed at the gateway
Where it's covered
Part 14: the edge

What I took from it

A change freeze is not a deployment strategy. It's a way to avoid deploying.

Make deploys small, judge them by what customers feel, make them reversible by design and let error budgets decide when to stop. Then you can deploy on a Saturday, and the freeze becomes a measurement instead of a calendar. Here, that was true for one team. Part 13 makes it true for every team, by making the safe path the default path.

Next in the series: Part 13: The pipeline was not a platform