Back to all articles
Platform at ScalePart 13
Platform engineeringDeveloper experienceGolden pathsPolicy as codeCI/CD

Platform at Scale, Part 13: The Pipeline Was Not a Platform

Janahan Thavendran10 min read
In this article

Platform at Scale, Part 13 · Make the safe path the easy path, so every team inherits what the others learned.

The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.

A new team joins on a Monday. They have CI/CD, so by Wednesday they can build and deploy. By Friday they still aren't in production.

They spent the week writing Terraform by hand, copying another team's pipeline, filing a ticket for a staging database, guessing which alerts to set, and asking a senior engineer how the partner queues were configured. The CI/CD worked perfectly. The platform around it didn't exist.

That was the moment we stopped measuring the platform by how many pipelines we ran, and started measuring it by how long a new team took to ship safely without asking us how.

Why CI/CD alone stopped scaling

Part 12 made our deploys small, gated and reversible. It didn't make anyone else's deploys that way. Once Part 5 left us with a core and a set of product services, every new service repeated the same manual work:

  • Every team built its own pipeline. Copy, paste and local edits meant ten pipelines that were 80% the same and secured ten different ways. A fix to the rollout logic in one never reached the others.
  • Provisioning was a ticket. A database, a Redis namespace, a partner queue, a staging environment: each was a request to the platform team. We became a bottleneck and a ticket queue, not a product team.
  • Guardrails were manual review. Because the pipeline enforced nothing, a person had to. That made the safe path the slow path, and the slow path is the one people route around.
  • Knowledge lived in senior engineers. Which alerts to set, which connection pool cap to use, which migration pattern was safe. None of it was written into anything. Onboarding meant interrupting someone who knew.
  • The bus factor was one. When one engineer held the deployment knowledge for a service, their holiday was a risk.

None of this shows up on a Saturday. It shows up as the cost of growth: every new service and every new team cost more platform-team hours than the last.

What we built, in order

We didn't buy or build a full internal developer platform on day one. We built four things, in order, on top of the pipeline from Part 12. Each one was useful on its own before the next one started.

Stage 1: infrastructure as code

Every resource, from Cloud SQL and Memorystore to RabbitMQ queues, Cloud Run services and Pub/Sub topics, was defined in Terraform and kept in version control. Changes were reviewed like code and applied by the pipeline, never by hand.

This made provisioning repeatable, and it was what made self-service safe later. A developer could ask for a database, but they got the approved database configuration, not a one-off.

Stage 2: a developer portal

We set up Backstage as a portal and a command line. Every service was in its catalog, with its owner, on-call rotation and dependencies, so the knowledge that lived in one engineer's head now lived on a page.

Each team saw their own service's health in one place: its targets and error budget (Parts 6 and 12), queue depth (Part 8) and replica lag (Part 3). A developer didn't need to know how their service was hosted. They needed to know their error budget and who was on call.

Stage 3: golden paths

A golden path isn't documentation. It's a working, self-service workflow for a common job: create a new service, add a partner integration, set up a staging environment. A developer picks a template and answers a few questions, like the service name, owner, channels and whether it touches payments. The platform then creates the repository, pipeline, infrastructure, alerts and dashboards.

The important part: using the template is how a service complies. A new service on the golden path got everything earlier parts had learned the hard way, without having to know it existed:

What a new service gets by default
A capped connection pool, sized from the database's limit down
Where it was learned
Part 1
What a new service gets by default
Versioned cache keys and a live-version pointer, for services that cache
Where it was learned
Part 2
What a new service gets by default
Reads routed to the primary by default, with replica reads opted into
Where it was learned
Part 3
What a new service gets by default
Idempotency keys and the payment state machine, for services that touch payments
Where it was learned
Part 4
What a new service gets by default
Timeouts and circuit breakers on calls to other services
Where it was learned
Part 5
What a new service gets by default
Trace context, structured logs and outcome alerts
Where it was learned
Part 6
What a new service gets by default
Event IDs, deduplication, backoff and warm-instance sizing for event consumers
Where it was learned
Part 7
What a new service gets by default
A partner queue with outlet lanes, limits and a dead-letter queue, for partner integrations
Where it was learned
Part 8
What a new service gets by default
Multi-zone settings for every stateful resource
Where it was learned
Part 9
What a new service gets by default
Partitioned tables with a retention policy
Where it was learned
Part 10
What a new service gets by default
A tenant quota class and tenant identity on every request
Where it was learned
Part 11
What a new service gets by default
Feature flags, gradual rollouts gated on outcomes, and two-step database changes
Where it was learned
Part 12

A developer who took the paved road got all of it. A developer who left the road had to explain why. Later the golden path also generated each service's edge: its gateway route, service account, secret bindings, CORS policy and WAF rules. Part 14 covers what those pieces are.

Stage 4: policy as code

To give teams freedom safely, the rules became code, checked automatically in the pipeline. Some of the rules we enforced:

  • No public storage buckets.
  • Every container has resource limits and a capped connection pool.
  • Every feature flag has an owner and an expiry date (Part 12).
  • Every service that touches payments uses Part 4's idempotency pattern.
  • Every service declares its tenant quota class (Part 11).
  • Every new table is born partitioned, or has an approved retention policy (Part 10).
  • No long-lived key files, no project owner role in CI/CD, and one service account per service (Part 14).

A rule failure blocked the deploy with a message saying exactly what to fix. No ticket to the platform team.

The pipeline from Part 12 didn't disappear. It became the engine every golden path ran: the gradual rollout, the outcome gates, the automatic rollback and the two-step database check all became what every team got by default.

Where it hurt

The first golden path was slow to build and immediately wrong for the second team that used it. Golden paths are opinionated, and an opinion that fits the core doesn't fit a loyalty service. So we supported escape hatches: a team could leave the paved road, but the portal flagged it, and the platform team reviewed every exception monthly.

Rules that were too strict blocked legitimate work. The first version of "no public buckets" broke a partner integration that legitimately needed a signed public link. Every rule needed an exception process, and every exception needed an owner.

And the platform team changed shape. It stopped being a ticket queue and became a product team, with users (developers), a roadmap and its own targets. That's a different job, and not everyone on the team wanted it.

The trade-off we made

Option
Keep CI/CD only, document the rest
Why it was tempting
No new work
Why we didn't choose it
Every team reinvents the pipeline, onboarding stays manual, and guardrails stay in people's heads
Option
Build a full developer platform from scratch
Why it was tempting
Perfect fit for our stack
Why we didn't choose it
Months before any value, rebuilding what Backstage already does
Option
Buy a managed developer platform
Why it was tempting
Fast to start
Why we didn't choose it
Ties us to a vendor that didn't understand our GCP, RabbitMQ and Cloud Run setup or Part 4's idempotency rules
Option
Make one team's pipeline the template
Why it was tempting
Cheap, one pipeline to maintain
Why we didn't choose it
The template carries one team's assumptions, not the platform's requirements
Option
Infrastructure as code, a portal, golden paths and policy as code, on the existing pipeline
Why it was tempting
Incremental, each piece useful alone, opinionated but escapable
Why we didn't choose it
Our choice, with more product work for the platform team

We chose to turn the platform into a product step by step, not in one big build. Each piece paid for itself before the next one started. Infrastructure as code made provisioning repeatable, the portal made it discoverable, golden paths made it fast and policy made it safe.

What we measured

  • Time for a new team's first safe production deploy: from 3 weeks to 2 days
  • Provisioning tickets to the platform team per month: from 60 to 10
  • Share of services on a golden path, and the number of approved exceptions
  • Deploys blocked by policy per week, and how many were false alarms
  • Golden path runs that succeeded first time: 85%

What it cost

The platform team's roadmap stopped being "keep the lights on" and became a product roadmap. Golden paths needed maintenance every time the stack changed: a new Cloud Run feature, a new partner API shape, a new rule. The first investment shipped no customer features for 3 months.

The quieter cost was standardisation. Teams that wanted to do something differently now had to justify it. That's the point of a platform, but it's also a real cost to autonomy, and it has to be paid deliberately.

What this part leaves open

Open question
Scorecards and cost per team
Why it matters
Once every service is in the portal, you can score each one on targets, cost per order and rule compliance
Open question
Multi-region as a golden path
Why it matters
Part 9's warm standby becomes a template option only once it's routine, not a runbook
Open question
Governing APIs between services
Why it matters
Part 5's core-to-product APIs need versioning and deprecation rules at the platform level, not per team
Open question
Targets for the platform itself
Why it matters
Golden path success rate and time to first deploy, or the platform becomes a new bottleneck nobody measures
Open question
The edge and identity as generated defaults
Why it matters
Part 14: the edge

What I took from it

CI/CD automates a step. A platform makes the safe path the easy path. The measure of a platform isn't how many pipelines you run. It's how fast a new team ships to production safely without asking you how.

A golden path is a trade too: some autonomy, in exchange for speed and safety. Make it escapable, measure the escapes, and revisit the trade as the team grows.

The pipeline was one half of the platform. The other half is the boundary between us and everyone else, which is where Part 14 begins.

Next in the series: Part 14: The edge is part of the platform