Platform at Scale, Part 14 · One front door, one identity per service, and a boundary the platform generates instead of each team.
The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.
A new partner integration goes live on a Thursday. On Saturday it silently drops every order. The partner's system was fine. Our middleware was fine. The partner had allowlisted our outbound IP address, but a new Cloud Run revision had come up behind a different NAT, so their firewall dropped us before our code ever ran.
The same evening, web checkout fails in one browser. The menu service was new, and it returned the right menu without the right CORS headers. The browser blocked it, the menu looked empty, and the diner left.
And while we were debugging both, someone noticed that a staging service account had Owner on the project. Nobody meant to give it that. It had been copied from a template, and the template was wrong.
Three failures in one night, with one thing in common: none of them were in the code we thought of as the platform. They were at the edge: routing, identity and the network between us and everyone else. We had a load balancer and "a firewall layer" in Part 1. We didn't have an edge layer.
Why the edge got scattered
Each earlier part pushed traffic toward a boundary we never designed in one place:
- Part 1 scaled out behind Cloud Load Balancing with a firewall in front. At first, that was the whole edge.
- Part 2 served menus to browsers, but nothing central said which web origins were allowed.
- Part 4 received payment notices, and each service that took a webhook wrote its own signature check.
- Part 5 split the monolith into services that call the core through APIs. At first, each service authenticated its own callers.
- Part 8 talked to 50+ partners, each allowlisting our IP, and nothing pinned that IP down.
- Part 11's quotas started inside each service, not at the door.
- Part 13's first golden paths created the pipeline and infrastructure, but not a route, a service account, a secret binding, a CORS policy or a WAF rule.
So every service configured its own edge. That meant ten slightly different answers to "who is calling", "from where" and "is this allowed". Consistency was a hope.
What we built, in order
Stage 1: the load balancer became the front door
We moved to a global layer-7 load balancer with TLS termination, managed certificates, health checks and connection draining. Every public hostname pointed at it, and nothing behind it had a public IP.
The load balancer also became where regional failover (Part 9) actually happens, because failover is a DNS and global load balancer switch, not a database trick. Graceful shutdown and a health endpoint became a requirement for every service, because draining does nothing if the service doesn't cooperate.
Stage 2: the API gateway became the policy point
Every request from a product, a partner or a browser passed through one gateway before it reached a service. The gateway took on five jobs that each service used to do its own way:
- Authentication: checked the caller's token or partner credential.
- Authorisation: checked the tenant claim. This merchant may act on this outlet. This partner may act on its own orders (Part 11).
- Quotas: per-tenant and per-partner limits, enforced at the door. Real diner orders were still queued above the quota, never rejected (Part 11).
- Versioning: routed
/v1and/v2to the right backend, so we could retire an old contract on a schedule (Part 5). - Gradual rollouts: sent a header-tagged share of traffic to a new version, which is what Part 12's rollouts needed at the edge, not only in load balancer weights.
The gateway also wrote the first access log line and created the trace, so Part 6's tracing started at the front door instead of somewhere in the middle.
Stage 3: CORS became a policy, not an afterthought
The browser enforces CORS, and the browser isn't a security boundary. We treated CORS as a compatibility contract with browsers: explicit allowed origins per environment, no wildcard with credentials, preflight requests answered at the gateway, and each service's policy generated for it. The menu service that broke on Saturday now gets its origins from the golden path, not from a developer's memory.
The important correction: CORS is not authorisation. A loose CORS policy doesn't let a stranger read another tenant's data if the gateway and the service both check the tenant claim. And a strict CORS policy doesn't protect an API that anyone can call from a script. We stopped treating CORS as security and handled authorisation properly behind it.
Stage 4: a WAF in front of everything
Cloud Armor sat in front of the load balancer with OWASP rules, bot management and rate-based rules. This is the layer before quotas. Quotas protect tenants from each other (Part 11). The WAF protects the platform from the internet.
Partner traffic got IP allowlisting and, where the partner supported it, mutual TLS. Inbound partner webhooks were signature-checked and allowlisted, the same way Part 4 checked payment notices.
Stage 5: one identity per service, and least privilege you can enforce
This was the biggest and least glamorous change:
- One service account per service, not one shared credential. The menu service couldn't read payment rows.
- Workload Identity, so Cloud Run services authenticated to GCP without long-lived key files.
- Secret Manager for every credential: database, Redis, partner keys and provider keys, rotated on a schedule.
- Internal-only ingress and invoker checks, so a core service couldn't be called from the internet even if someone found its address.
- Cloud SQL IAM authentication where the driver supported it, so the database credential wasn't a password at all.
- Separate identities for the pipeline. It deployed as a per-environment deployer with short-lived credentials, never as Owner. The staging Owner account was deleted, and policy as code (Part 13) blocked its return.
- Audit logs for permission changes, secret access and payment state changes, kept apart from application logs (Part 6).
Stage 6: the network stopped being implicit
Services ran on private subnets with no public IPs. Outbound traffic went through Cloud NAT on fixed IP addresses, because partners allowlist addresses, and an address that changes with a Cloud Run revision is a partner outage waiting for a Saturday. Private Google Access let services reach GCP without leaving the network, and VPC Service Controls limited what a compromised service could reach.
That fixed outbound address is the whole fix for the Thursday incident. Partners allowlist a stable address, not a revision.
What the golden path generates
None of this helps if a new service has to remember it. So Part 13's golden path generates the edge by default. A new service gets:
- A load balancer route and a health check
- A gateway route with its authentication and quota class
- Its own service account, with only the roles it needs
- Its secret bindings, with rotation
- Its CORS policy, from the environment
- Its WAF rule set and its place in the rate rules
- Its audit hook and access log format
A developer can't create a service without them, and policy as code blocks any deploy that tries to skip them. The safe path is the only easy path.
Where it hurt
The gateway became a shared dependency, which is a new way to fail. If the gateway is down, everything is down. So it got its own targets, its own gradual rollouts and its own capacity model from Part 1: sized for the peak minute, not the day, and kept stateless so it could scale like the app.
Per-service identity meant many more permissions to manage, and the first policy was too strict. A service that genuinely needed one column of the orders table was blocked until we wrote a narrower grant. That was the right kind of friction.
CORS broke twice during the migration: once when a staging origin leaked into a production policy, and once when a service answered preflight requests itself instead of leaving them to the gateway. Both were caught by gradual rollouts, which is the point of having them.
And fixed outbound addresses are a shared resource. One partner rate-limiting one address could affect every service behind it, so we used a small pool and assigned partners to addresses deliberately.
The trade-off we made
| Option | Why it was tempting | Why we didn't choose it |
|---|---|---|
| Let each service own its edge setup | No central dependency, teams move fast | Ten different answers to authentication, CORS and rate limits. Inconsistency is a security gap |
| Terminate TLS and route in each service | Simpler per service | No single place for the WAF, quotas or failover, and the public surface grows with every service |
| A service mesh for service-to-service identity | Strong identity, mutual TLS everywhere | More to operate than our team could run, for a problem mutual TLS plus gateway authentication already covered at our size |
| One shared service account | Easy to start | One leaked credential is the whole platform, with no least privilege and no audit trail |
| Long-lived key files for CI/CD | Familiar | Rotation is manual and a leak is permanent. Workload Identity removes the problem |
| Global load balancer, API gateway, Cloud Armor, one identity per service, fixed outbound addresses | One policy point, least privilege, stable addresses for partners | Our choice, with a shared dependency and more permissions to manage |
- Option
- Let each service own its edge setup
- Why it was tempting
- No central dependency, teams move fast
- Why we didn't choose it
- Ten different answers to authentication, CORS and rate limits. Inconsistency is a security gap
- Option
- Terminate TLS and route in each service
- Why it was tempting
- Simpler per service
- Why we didn't choose it
- No single place for the WAF, quotas or failover, and the public surface grows with every service
- Option
- A service mesh for service-to-service identity
- Why it was tempting
- Strong identity, mutual TLS everywhere
- Why we didn't choose it
- More to operate than our team could run, for a problem mutual TLS plus gateway authentication already covered at our size
- Option
- One shared service account
- Why it was tempting
- Easy to start
- Why we didn't choose it
- One leaked credential is the whole platform, with no least privilege and no audit trail
- Option
- Long-lived key files for CI/CD
- Why it was tempting
- Familiar
- Why we didn't choose it
- Rotation is manual and a leak is permanent. Workload Identity removes the problem
- Option
- Global load balancer, API gateway, Cloud Armor, one identity per service, fixed outbound addresses
- Why it was tempting
- One policy point, least privilege, stable addresses for partners
- Why we didn't choose it
- Our choice, with a shared dependency and more permissions to manage
We didn't adopt a service mesh. At our size, the gateway handled identity for traffic coming in, and mutual TLS plus per-service accounts handled traffic between services. A mesh is the right answer at a larger scale. For us it was the wrong cost, and we wrote down when to revisit it: when configuring mutual TLS per service becomes a maintenance burden of its own.
What we measured
- Publicly reachable endpoints: from 30 to two, the load balancer and the gateway
- Services with their own service account: 100%
- Long-lived key files in use: from 25 to zero
- Partner deliveries failed by allowlisting: from 3 a month to zero
- CORS failures reaching production: from 4 a quarter to zero
What it cost
The gateway, WAF and global load balancer added about $2,000 a month, and one more system with its own targets and on-call rotation. Per-service identity took 120 hours of permission work during the migration, and a permanent policy review load. Fixed outbound addresses added NAT cost and a list of which partner uses which address that we had to keep current.
The bigger cost was centralisation. Every new route, origin or partner credential now went through one place, and that place could become a bottleneck if it ran as a ticket queue. We ran it as a product, with golden paths (Part 13), so most changes never became tickets.
In return, the Thursday partner outage couldn't happen again, the browser failure became a generated policy, the staging Owner account became a policy violation, and the public attack surface shrank from every service to the load balancer and the gateway.
What this part leaves open
| Open question | Why it matters |
|---|---|
| A service mesh and mutual TLS at larger scale | The trigger is written down. It's a later chapter, not this one |
| Contract testing across core and products | Versioning at the gateway is a start, not a full regime |
| An active-active edge across regions | Part 9's failover uses the global load balancer. Regional gateways are a different design |
| Payment fraud scoring | The WAF handles floods and bots. Fraud is a product problem, not an edge one |
| Device identity for tablets | Certificates and rotation for devices sit closer to Part 2's device logic |
- Open question
- A service mesh and mutual TLS at larger scale
- Why it matters
- The trigger is written down. It's a later chapter, not this one
- Open question
- Contract testing across core and products
- Why it matters
- Versioning at the gateway is a start, not a full regime
- Open question
- An active-active edge across regions
- Why it matters
- Part 9's failover uses the global load balancer. Regional gateways are a different design
- Open question
- Payment fraud scoring
- Why it matters
- The WAF handles floods and bots. Fraud is a product problem, not an edge one
- Open question
- Device identity for tablets
- Why it matters
- Certificates and rotation for devices sit closer to Part 2's device logic
What I took from it
The edge is part of the platform, not a detail of each service. One front door, one policy point, one identity per service, one stable outbound address, and a golden path that generates all of them.
If every service sets up its own edge, you get ten answers and one incident. If the platform generates it, the safe path is the default path. It's the same lesson as Part 13, applied to the boundary instead of the pipeline.
The whole series, one line each
The series has three halves. Parts 1 to 8 are about surviving the Saturday. Parts 9 to 12 are about surviving the year: the zone, the table, the tenant and the deploy. Parts 13 and 14 are about the organisation: making the safe path the easy path for every team, in the pipeline and at the boundary.
- Saturday night: fix in order of cost, and plan for the peak minute, not the average day.
- The menu: caching is easy. Invalidation is the design.
- Read replicas: be a little wrong on purpose, screen by screen, and make the safe choice the default.
- Double payments: build "exactly once" yourself, then reconcile in case you got it wrong.
- Splitting the monolith: contain failure, and split the resources, not just the code.
- Monitoring: one business ID per order, one trace per request, and hear about problems before your customers do.
- Events: keep them off the order path, and give each kind exactly the guarantee it needs.
- Partners: you're only as fast as the slowest system you talk to.
- Zones and regions: every stateful piece in more than one zone before a second region, and rehearse the restore.
- Data growth: a spike you can plan for, a slope you have to manage. Archive and partition before you shard.
- Noisy tenants: fairness at every shared layer, so one merchant's spike stays one merchant's problem.
- Deploys: small, judged by what customers feel, and reversible by design.
- The platform: make the safe path the easy path, so every team inherits what the others learned.
- The edge: one front door, one identity per service, and a boundary the platform generates instead of each team.
Every fix in this series was a trade we chose, sized to the traffic, the team and the partners we had at that moment.