Platform at Scale, Part 13 · Make the safe path the easy path, so every team inherits what the others learned.
The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.
A new team joins on a Monday. They have CI/CD, so by Wednesday they can build and deploy. By Friday they still aren't in production.
They spent the week writing Terraform by hand, copying another team's pipeline, filing a ticket for a staging database, guessing which alerts to set, and asking a senior engineer how the partner queues were configured. The CI/CD worked perfectly. The platform around it didn't exist.
That was the moment we stopped measuring the platform by how many pipelines we ran, and started measuring it by how long a new team took to ship safely without asking us how.
Why CI/CD alone stopped scaling
Part 12 made our deploys small, gated and reversible. It didn't make anyone else's deploys that way. Once Part 5 left us with a core and a set of product services, every new service repeated the same manual work:
- Every team built its own pipeline. Copy, paste and local edits meant ten pipelines that were 80% the same and secured ten different ways. A fix to the rollout logic in one never reached the others.
- Provisioning was a ticket. A database, a Redis namespace, a partner queue, a staging environment: each was a request to the platform team. We became a bottleneck and a ticket queue, not a product team.
- Guardrails were manual review. Because the pipeline enforced nothing, a person had to. That made the safe path the slow path, and the slow path is the one people route around.
- Knowledge lived in senior engineers. Which alerts to set, which connection pool cap to use, which migration pattern was safe. None of it was written into anything. Onboarding meant interrupting someone who knew.
- The bus factor was one. When one engineer held the deployment knowledge for a service, their holiday was a risk.
None of this shows up on a Saturday. It shows up as the cost of growth: every new service and every new team cost more platform-team hours than the last.
What we built, in order
We didn't buy or build a full internal developer platform on day one. We built four things, in order, on top of the pipeline from Part 12. Each one was useful on its own before the next one started.
Stage 1: infrastructure as code
Every resource, from Cloud SQL and Memorystore to RabbitMQ queues, Cloud Run services and Pub/Sub topics, was defined in Terraform and kept in version control. Changes were reviewed like code and applied by the pipeline, never by hand.
This made provisioning repeatable, and it was what made self-service safe later. A developer could ask for a database, but they got the approved database configuration, not a one-off.
Stage 2: a developer portal
We set up Backstage as a portal and a command line. Every service was in its catalog, with its owner, on-call rotation and dependencies, so the knowledge that lived in one engineer's head now lived on a page.
Each team saw their own service's health in one place: its targets and error budget (Parts 6 and 12), queue depth (Part 8) and replica lag (Part 3). A developer didn't need to know how their service was hosted. They needed to know their error budget and who was on call.
Stage 3: golden paths
A golden path isn't documentation. It's a working, self-service workflow for a common job: create a new service, add a partner integration, set up a staging environment. A developer picks a template and answers a few questions, like the service name, owner, channels and whether it touches payments. The platform then creates the repository, pipeline, infrastructure, alerts and dashboards.
The important part: using the template is how a service complies. A new service on the golden path got everything earlier parts had learned the hard way, without having to know it existed:
| What a new service gets by default | Where it was learned |
|---|---|
| A capped connection pool, sized from the database's limit down | Part 1 |
| Versioned cache keys and a live-version pointer, for services that cache | Part 2 |
| Reads routed to the primary by default, with replica reads opted into | Part 3 |
| Idempotency keys and the payment state machine, for services that touch payments | Part 4 |
| Timeouts and circuit breakers on calls to other services | Part 5 |
| Trace context, structured logs and outcome alerts | Part 6 |
| Event IDs, deduplication, backoff and warm-instance sizing for event consumers | Part 7 |
| A partner queue with outlet lanes, limits and a dead-letter queue, for partner integrations | Part 8 |
| Multi-zone settings for every stateful resource | Part 9 |
| Partitioned tables with a retention policy | Part 10 |
| A tenant quota class and tenant identity on every request | Part 11 |
| Feature flags, gradual rollouts gated on outcomes, and two-step database changes | Part 12 |
- What a new service gets by default
- A capped connection pool, sized from the database's limit down
- Where it was learned
- Part 1
- What a new service gets by default
- Versioned cache keys and a live-version pointer, for services that cache
- Where it was learned
- Part 2
- What a new service gets by default
- Reads routed to the primary by default, with replica reads opted into
- Where it was learned
- Part 3
- What a new service gets by default
- Idempotency keys and the payment state machine, for services that touch payments
- Where it was learned
- Part 4
- What a new service gets by default
- Timeouts and circuit breakers on calls to other services
- Where it was learned
- Part 5
- What a new service gets by default
- Trace context, structured logs and outcome alerts
- Where it was learned
- Part 6
- What a new service gets by default
- Event IDs, deduplication, backoff and warm-instance sizing for event consumers
- Where it was learned
- Part 7
- What a new service gets by default
- A partner queue with outlet lanes, limits and a dead-letter queue, for partner integrations
- Where it was learned
- Part 8
- What a new service gets by default
- Multi-zone settings for every stateful resource
- Where it was learned
- Part 9
- What a new service gets by default
- Partitioned tables with a retention policy
- Where it was learned
- Part 10
- What a new service gets by default
- A tenant quota class and tenant identity on every request
- Where it was learned
- Part 11
- What a new service gets by default
- Feature flags, gradual rollouts gated on outcomes, and two-step database changes
- Where it was learned
- Part 12
A developer who took the paved road got all of it. A developer who left the road had to explain why. Later the golden path also generated each service's edge: its gateway route, service account, secret bindings, CORS policy and WAF rules. Part 14 covers what those pieces are.
Stage 4: policy as code
To give teams freedom safely, the rules became code, checked automatically in the pipeline. Some of the rules we enforced:
- No public storage buckets.
- Every container has resource limits and a capped connection pool.
- Every feature flag has an owner and an expiry date (Part 12).
- Every service that touches payments uses Part 4's idempotency pattern.
- Every service declares its tenant quota class (Part 11).
- Every new table is born partitioned, or has an approved retention policy (Part 10).
- No long-lived key files, no project owner role in CI/CD, and one service account per service (Part 14).
A rule failure blocked the deploy with a message saying exactly what to fix. No ticket to the platform team.
The pipeline from Part 12 didn't disappear. It became the engine every golden path ran: the gradual rollout, the outcome gates, the automatic rollback and the two-step database check all became what every team got by default.
Where it hurt
The first golden path was slow to build and immediately wrong for the second team that used it. Golden paths are opinionated, and an opinion that fits the core doesn't fit a loyalty service. So we supported escape hatches: a team could leave the paved road, but the portal flagged it, and the platform team reviewed every exception monthly.
Rules that were too strict blocked legitimate work. The first version of "no public buckets" broke a partner integration that legitimately needed a signed public link. Every rule needed an exception process, and every exception needed an owner.
And the platform team changed shape. It stopped being a ticket queue and became a product team, with users (developers), a roadmap and its own targets. That's a different job, and not everyone on the team wanted it.
The trade-off we made
| Option | Why it was tempting | Why we didn't choose it |
|---|---|---|
| Keep CI/CD only, document the rest | No new work | Every team reinvents the pipeline, onboarding stays manual, and guardrails stay in people's heads |
| Build a full developer platform from scratch | Perfect fit for our stack | Months before any value, rebuilding what Backstage already does |
| Buy a managed developer platform | Fast to start | Ties us to a vendor that didn't understand our GCP, RabbitMQ and Cloud Run setup or Part 4's idempotency rules |
| Make one team's pipeline the template | Cheap, one pipeline to maintain | The template carries one team's assumptions, not the platform's requirements |
| Infrastructure as code, a portal, golden paths and policy as code, on the existing pipeline | Incremental, each piece useful alone, opinionated but escapable | Our choice, with more product work for the platform team |
- Option
- Keep CI/CD only, document the rest
- Why it was tempting
- No new work
- Why we didn't choose it
- Every team reinvents the pipeline, onboarding stays manual, and guardrails stay in people's heads
- Option
- Build a full developer platform from scratch
- Why it was tempting
- Perfect fit for our stack
- Why we didn't choose it
- Months before any value, rebuilding what Backstage already does
- Option
- Buy a managed developer platform
- Why it was tempting
- Fast to start
- Why we didn't choose it
- Ties us to a vendor that didn't understand our GCP, RabbitMQ and Cloud Run setup or Part 4's idempotency rules
- Option
- Make one team's pipeline the template
- Why it was tempting
- Cheap, one pipeline to maintain
- Why we didn't choose it
- The template carries one team's assumptions, not the platform's requirements
- Option
- Infrastructure as code, a portal, golden paths and policy as code, on the existing pipeline
- Why it was tempting
- Incremental, each piece useful alone, opinionated but escapable
- Why we didn't choose it
- Our choice, with more product work for the platform team
We chose to turn the platform into a product step by step, not in one big build. Each piece paid for itself before the next one started. Infrastructure as code made provisioning repeatable, the portal made it discoverable, golden paths made it fast and policy made it safe.
What we measured
- Time for a new team's first safe production deploy: from 3 weeks to 2 days
- Provisioning tickets to the platform team per month: from 60 to 10
- Share of services on a golden path, and the number of approved exceptions
- Deploys blocked by policy per week, and how many were false alarms
- Golden path runs that succeeded first time: 85%
What it cost
The platform team's roadmap stopped being "keep the lights on" and became a product roadmap. Golden paths needed maintenance every time the stack changed: a new Cloud Run feature, a new partner API shape, a new rule. The first investment shipped no customer features for 3 months.
The quieter cost was standardisation. Teams that wanted to do something differently now had to justify it. That's the point of a platform, but it's also a real cost to autonomy, and it has to be paid deliberately.
What this part leaves open
| Open question | Why it matters |
|---|---|
| Scorecards and cost per team | Once every service is in the portal, you can score each one on targets, cost per order and rule compliance |
| Multi-region as a golden path | Part 9's warm standby becomes a template option only once it's routine, not a runbook |
| Governing APIs between services | Part 5's core-to-product APIs need versioning and deprecation rules at the platform level, not per team |
| Targets for the platform itself | Golden path success rate and time to first deploy, or the platform becomes a new bottleneck nobody measures |
| The edge and identity as generated defaults | Part 14: the edge |
- Open question
- Scorecards and cost per team
- Why it matters
- Once every service is in the portal, you can score each one on targets, cost per order and rule compliance
- Open question
- Multi-region as a golden path
- Why it matters
- Part 9's warm standby becomes a template option only once it's routine, not a runbook
- Open question
- Governing APIs between services
- Why it matters
- Part 5's core-to-product APIs need versioning and deprecation rules at the platform level, not per team
- Open question
- Targets for the platform itself
- Why it matters
- Golden path success rate and time to first deploy, or the platform becomes a new bottleneck nobody measures
- Open question
- The edge and identity as generated defaults
- Why it matters
- Part 14: the edge
What I took from it
CI/CD automates a step. A platform makes the safe path the easy path. The measure of a platform isn't how many pipelines you run. It's how fast a new team ships to production safely without asking you how.
A golden path is a trade too: some autonomy, in exchange for speed and safety. Make it escapable, measure the escapes, and revisit the trade as the team grows.
The pipeline was one half of the platform. The other half is the boundary between us and everyone else, which is where Part 14 begins.
Next in the series: Part 14: The edge is part of the platform