Platform at Scale, Part 12 · Deploys small, judged by what customers feel, and reversible by design.
The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.
It's the Thursday before a festive weekend, and Part 1's change freeze is in force. A merchant reports a checkout bug that drops the tip on one channel. The fix is small. Someone decides it's small enough for an exception and ships it on Thursday night, together with a database change the old app servers don't understand.
Checkout breaks for 35 minutes on the busiest night of the week. Rolling back takes 50 minutes, because nobody had ever rehearsed it, and the old code can't run against the new database.
The change freeze didn't fail. It was the only deployment strategy we had, and it couldn't handle a small fix under pressure.
Why a freeze is not a strategy
A change freeze avoids deploying. It doesn't make deploying safe. Under a freeze, the only deploys that happen are exceptions under pressure, which is the worst possible time to deploy without gradual rollouts, flags or a rehearsed rollback.
Part 1 used the freeze correctly, to protect peak weekends while the platform was fragile. This part is about making deploys safe enough that the freeze becomes a choice, not a crutch.
What we fixed, in order
Stage 1: feature flags
The tip fix didn't need a risky deploy. It needed a flag: ship the code switched off, turn it on for the affected channel at a chosen minute, and switch it off again in seconds if anything looks wrong.
Flags separated two questions that had always been tied together: when the code lands, and when the behaviour changes. Those carry very different risks.
Flags also create their own debt. Every flag had an owner and an expiry date, and we audited them every month. Only named roles could flip a production flag, and every flip was recorded.
Stage 2: gradual rollouts, judged by what diners feel
Part 6 defined order placement, payment and partner acknowledgement success the same way for every team. We used exactly those numbers to judge a release.
A new version took 1% of traffic for 15 minutes, then 10%, then 50%, then all of it. At every step, its order placement success, payment success and p95 checkout time were compared with the old version, per channel (Part 5). If they got worse, the rollout reversed itself automatically. The rollout watched what diners felt, not what the CPU did. Traffic was split at the gateway, by header, not only by load balancer weight (Part 14).
Stage 3: database changes in two halves
The change that broke the rollback was made in one step. We switched to a pattern called expand and contract:
- Add the new column, leaving the old one in place.
- Deploy code that writes to both.
- Fill in the new column for existing rows.
- Switch reads to the new column.
- In a later release, remove the old column.
At no point does a running version meet a database it doesn't understand, so the previous version can always run. That's what made rollback safe. Between services (Part 5), the same rule applied to APIs: add the new field, move every caller over, and only then remove the old one.
Stage 4: rollbacks we had already done
We rolled back on purpose, in staging, on a schedule, and timed it. Rollback became routine instead of improvisation. If a gradual rollout breached its numbers, rollback happened automatically. If a person had to do it, they had done it before.
Stage 5: error budgets decide when to stop
Part 6 already used error budgets to decide when to wake someone. Now the deploy pipeline read them too. If a service had used up its error budget, new deploys to it were blocked automatically, except changes that improved reliability, until the budget recovered. Not by memo, by pipeline.
The festive freeze didn't disappear. It shrank to 3 days around the peak, and only for big changes. Flags and small, gradually rolled-out changes carried on. The freeze became a measurement instead of a calendar. Error budgets governed our own deploys here. Making them govern every team's deploys, automatically, is Part 13.
The trade-off we made
| Option | Why it was tempting | Why we didn't choose it |
|---|---|---|
| Keep the calendar freeze | Simple, no new tooling | The only deploys that happen are exceptions under pressure |
| A full second copy of every service for each release (blue/green) | A clean switch and instant rollback | Doubles the running footprint for every deploy. Heavy for a small team |
| Gradual rollouts on every deploy | Catches problems early | Needs outcome metrics and automatic rollback, which we already had from Part 6 |
| Feature flags everywhere | Separates deploying from changing behaviour | Flag debt is real. Flags need owners and expiry dates |
| Flags, gradual rollouts, two-step database changes, rehearsed rollbacks and error budgets | Deploys are small and reversible on any day | Our choice, with more process |
- Option
- Keep the calendar freeze
- Why it was tempting
- Simple, no new tooling
- Why we didn't choose it
- The only deploys that happen are exceptions under pressure
- Option
- A full second copy of every service for each release (blue/green)
- Why it was tempting
- A clean switch and instant rollback
- Why we didn't choose it
- Doubles the running footprint for every deploy. Heavy for a small team
- Option
- Gradual rollouts on every deploy
- Why it was tempting
- Catches problems early
- Why we didn't choose it
- Needs outcome metrics and automatic rollback, which we already had from Part 6
- Option
- Feature flags everywhere
- Why it was tempting
- Separates deploying from changing behaviour
- Why we didn't choose it
- Flag debt is real. Flags need owners and expiry dates
- Option
- Flags, gradual rollouts, two-step database changes, rehearsed rollbacks and error budgets
- Why it was tempting
- Deploys are small and reversible on any day
- Why we didn't choose it
- Our choice, with more process
We didn't build a second copy of the whole platform for every release. We made small, reversible deploys the default, and let the error budget decide when to stop.
What we measured
- Deploys per week: from 3 to 40
- Share of deploys that caused an incident: from 12% to 2%
- Time to roll back: from 50 minutes to 3 minutes
- Deploy-caused incidents in festive weeks: one, caught at 1% and rolled back automatically
- Flags past their expiry at each monthly audit: under 5
What it cost
The pipeline work took 10 weeks of engineering. Gradual rollouts made each deploy about 45 minutes slower. Flags added a new kind of debt that needed managing every month. The pipeline itself ran as a least-privilege deployer with short-lived credentials, never as a project owner, and every image was built from pinned base images and signed before it could deploy.
In return, the Thursday-night exception became a flag flip, and a rollback that took 50 minutes became a button that had been pressed before.
What this part leaves for later
| Gap left here | Where it's covered |
|---|---|
| Managing flags at scale: ownership, expiry and testing | Part 13: policy as code |
| Safe deploys for every team, not just ours | Part 13: golden paths |
| Schema changes across many services at once | Part 13, and an open question for API governance |
| The pipeline's own identity, and gradual rollouts routed at the gateway | Part 14: the edge |
- Gap left here
- Managing flags at scale: ownership, expiry and testing
- Where it's covered
- Part 13: policy as code
- Gap left here
- Safe deploys for every team, not just ours
- Where it's covered
- Part 13: golden paths
- Gap left here
- Schema changes across many services at once
- Where it's covered
- Part 13, and an open question for API governance
- Gap left here
- The pipeline's own identity, and gradual rollouts routed at the gateway
- Where it's covered
- Part 14: the edge
What I took from it
A change freeze is not a deployment strategy. It's a way to avoid deploying.
Make deploys small, judge them by what customers feel, make them reversible by design and let error budgets decide when to stop. Then you can deploy on a Saturday, and the freeze becomes a measurement instead of a calendar. Here, that was true for one team. Part 13 makes it true for every team, by making the safe path the default path.
Next in the series: Part 13: The pipeline was not a platform