Back to all articles
Platform at ScalePart 14
Platform engineeringAPI gatewaySecurityIAMGCP

Platform at Scale, Part 14: The Edge Is Part of the Platform

Janahan Thavendran13 min read
In this article

Platform at Scale, Part 14 · One front door, one identity per service, and a boundary the platform generates instead of each team.

The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.

A new partner integration goes live on a Thursday. On Saturday it silently drops every order. The partner's system was fine. Our middleware was fine. The partner had allowlisted our outbound IP address, but a new Cloud Run revision had come up behind a different NAT, so their firewall dropped us before our code ever ran.

The same evening, web checkout fails in one browser. The menu service was new, and it returned the right menu without the right CORS headers. The browser blocked it, the menu looked empty, and the diner left.

And while we were debugging both, someone noticed that a staging service account had Owner on the project. Nobody meant to give it that. It had been copied from a template, and the template was wrong.

Three failures in one night, with one thing in common: none of them were in the code we thought of as the platform. They were at the edge: routing, identity and the network between us and everyone else. We had a load balancer and "a firewall layer" in Part 1. We didn't have an edge layer.

Why the edge got scattered

Each earlier part pushed traffic toward a boundary we never designed in one place:

  • Part 1 scaled out behind Cloud Load Balancing with a firewall in front. At first, that was the whole edge.
  • Part 2 served menus to browsers, but nothing central said which web origins were allowed.
  • Part 4 received payment notices, and each service that took a webhook wrote its own signature check.
  • Part 5 split the monolith into services that call the core through APIs. At first, each service authenticated its own callers.
  • Part 8 talked to 50+ partners, each allowlisting our IP, and nothing pinned that IP down.
  • Part 11's quotas started inside each service, not at the door.
  • Part 13's first golden paths created the pipeline and infrastructure, but not a route, a service account, a secret binding, a CORS policy or a WAF rule.

So every service configured its own edge. That meant ten slightly different answers to "who is calling", "from where" and "is this allowed". Consistency was a hope.

What we built, in order

Stage 1: the load balancer became the front door

We moved to a global layer-7 load balancer with TLS termination, managed certificates, health checks and connection draining. Every public hostname pointed at it, and nothing behind it had a public IP.

The load balancer also became where regional failover (Part 9) actually happens, because failover is a DNS and global load balancer switch, not a database trick. Graceful shutdown and a health endpoint became a requirement for every service, because draining does nothing if the service doesn't cooperate.

Stage 2: the API gateway became the policy point

Every request from a product, a partner or a browser passed through one gateway before it reached a service. The gateway took on five jobs that each service used to do its own way:

  1. Authentication: checked the caller's token or partner credential.
  2. Authorisation: checked the tenant claim. This merchant may act on this outlet. This partner may act on its own orders (Part 11).
  3. Quotas: per-tenant and per-partner limits, enforced at the door. Real diner orders were still queued above the quota, never rejected (Part 11).
  4. Versioning: routed /v1 and /v2 to the right backend, so we could retire an old contract on a schedule (Part 5).
  5. Gradual rollouts: sent a header-tagged share of traffic to a new version, which is what Part 12's rollouts needed at the edge, not only in load balancer weights.

The gateway also wrote the first access log line and created the trace, so Part 6's tracing started at the front door instead of somewhere in the middle.

Stage 3: CORS became a policy, not an afterthought

The browser enforces CORS, and the browser isn't a security boundary. We treated CORS as a compatibility contract with browsers: explicit allowed origins per environment, no wildcard with credentials, preflight requests answered at the gateway, and each service's policy generated for it. The menu service that broke on Saturday now gets its origins from the golden path, not from a developer's memory.

The important correction: CORS is not authorisation. A loose CORS policy doesn't let a stranger read another tenant's data if the gateway and the service both check the tenant claim. And a strict CORS policy doesn't protect an API that anyone can call from a script. We stopped treating CORS as security and handled authorisation properly behind it.

Stage 4: a WAF in front of everything

Cloud Armor sat in front of the load balancer with OWASP rules, bot management and rate-based rules. This is the layer before quotas. Quotas protect tenants from each other (Part 11). The WAF protects the platform from the internet.

Partner traffic got IP allowlisting and, where the partner supported it, mutual TLS. Inbound partner webhooks were signature-checked and allowlisted, the same way Part 4 checked payment notices.

Stage 5: one identity per service, and least privilege you can enforce

This was the biggest and least glamorous change:

  • One service account per service, not one shared credential. The menu service couldn't read payment rows.
  • Workload Identity, so Cloud Run services authenticated to GCP without long-lived key files.
  • Secret Manager for every credential: database, Redis, partner keys and provider keys, rotated on a schedule.
  • Internal-only ingress and invoker checks, so a core service couldn't be called from the internet even if someone found its address.
  • Cloud SQL IAM authentication where the driver supported it, so the database credential wasn't a password at all.
  • Separate identities for the pipeline. It deployed as a per-environment deployer with short-lived credentials, never as Owner. The staging Owner account was deleted, and policy as code (Part 13) blocked its return.
  • Audit logs for permission changes, secret access and payment state changes, kept apart from application logs (Part 6).

Stage 6: the network stopped being implicit

Services ran on private subnets with no public IPs. Outbound traffic went through Cloud NAT on fixed IP addresses, because partners allowlist addresses, and an address that changes with a Cloud Run revision is a partner outage waiting for a Saturday. Private Google Access let services reach GCP without leaving the network, and VPC Service Controls limited what a compromised service could reach.

That fixed outbound address is the whole fix for the Thursday incident. Partners allowlist a stable address, not a revision.

What the golden path generates

None of this helps if a new service has to remember it. So Part 13's golden path generates the edge by default. A new service gets:

  • A load balancer route and a health check
  • A gateway route with its authentication and quota class
  • Its own service account, with only the roles it needs
  • Its secret bindings, with rotation
  • Its CORS policy, from the environment
  • Its WAF rule set and its place in the rate rules
  • Its audit hook and access log format

A developer can't create a service without them, and policy as code blocks any deploy that tries to skip them. The safe path is the only easy path.

Where it hurt

The gateway became a shared dependency, which is a new way to fail. If the gateway is down, everything is down. So it got its own targets, its own gradual rollouts and its own capacity model from Part 1: sized for the peak minute, not the day, and kept stateless so it could scale like the app.

Per-service identity meant many more permissions to manage, and the first policy was too strict. A service that genuinely needed one column of the orders table was blocked until we wrote a narrower grant. That was the right kind of friction.

CORS broke twice during the migration: once when a staging origin leaked into a production policy, and once when a service answered preflight requests itself instead of leaving them to the gateway. Both were caught by gradual rollouts, which is the point of having them.

And fixed outbound addresses are a shared resource. One partner rate-limiting one address could affect every service behind it, so we used a small pool and assigned partners to addresses deliberately.

The trade-off we made

Option
Let each service own its edge setup
Why it was tempting
No central dependency, teams move fast
Why we didn't choose it
Ten different answers to authentication, CORS and rate limits. Inconsistency is a security gap
Option
Terminate TLS and route in each service
Why it was tempting
Simpler per service
Why we didn't choose it
No single place for the WAF, quotas or failover, and the public surface grows with every service
Option
A service mesh for service-to-service identity
Why it was tempting
Strong identity, mutual TLS everywhere
Why we didn't choose it
More to operate than our team could run, for a problem mutual TLS plus gateway authentication already covered at our size
Option
One shared service account
Why it was tempting
Easy to start
Why we didn't choose it
One leaked credential is the whole platform, with no least privilege and no audit trail
Option
Long-lived key files for CI/CD
Why it was tempting
Familiar
Why we didn't choose it
Rotation is manual and a leak is permanent. Workload Identity removes the problem
Option
Global load balancer, API gateway, Cloud Armor, one identity per service, fixed outbound addresses
Why it was tempting
One policy point, least privilege, stable addresses for partners
Why we didn't choose it
Our choice, with a shared dependency and more permissions to manage

We didn't adopt a service mesh. At our size, the gateway handled identity for traffic coming in, and mutual TLS plus per-service accounts handled traffic between services. A mesh is the right answer at a larger scale. For us it was the wrong cost, and we wrote down when to revisit it: when configuring mutual TLS per service becomes a maintenance burden of its own.

What we measured

  • Publicly reachable endpoints: from 30 to two, the load balancer and the gateway
  • Services with their own service account: 100%
  • Long-lived key files in use: from 25 to zero
  • Partner deliveries failed by allowlisting: from 3 a month to zero
  • CORS failures reaching production: from 4 a quarter to zero

What it cost

The gateway, WAF and global load balancer added about $2,000 a month, and one more system with its own targets and on-call rotation. Per-service identity took 120 hours of permission work during the migration, and a permanent policy review load. Fixed outbound addresses added NAT cost and a list of which partner uses which address that we had to keep current.

The bigger cost was centralisation. Every new route, origin or partner credential now went through one place, and that place could become a bottleneck if it ran as a ticket queue. We ran it as a product, with golden paths (Part 13), so most changes never became tickets.

In return, the Thursday partner outage couldn't happen again, the browser failure became a generated policy, the staging Owner account became a policy violation, and the public attack surface shrank from every service to the load balancer and the gateway.

What this part leaves open

Open question
A service mesh and mutual TLS at larger scale
Why it matters
The trigger is written down. It's a later chapter, not this one
Open question
Contract testing across core and products
Why it matters
Versioning at the gateway is a start, not a full regime
Open question
An active-active edge across regions
Why it matters
Part 9's failover uses the global load balancer. Regional gateways are a different design
Open question
Payment fraud scoring
Why it matters
The WAF handles floods and bots. Fraud is a product problem, not an edge one
Open question
Device identity for tablets
Why it matters
Certificates and rotation for devices sit closer to Part 2's device logic

What I took from it

The edge is part of the platform, not a detail of each service. One front door, one policy point, one identity per service, one stable outbound address, and a golden path that generates all of them.

If every service sets up its own edge, you get ten answers and one incident. If the platform generates it, the safe path is the default path. It's the same lesson as Part 13, applied to the boundary instead of the pipeline.

The whole series, one line each

The series has three halves. Parts 1 to 8 are about surviving the Saturday. Parts 9 to 12 are about surviving the year: the zone, the table, the tenant and the deploy. Parts 13 and 14 are about the organisation: making the safe path the easy path for every team, in the pipeline and at the boundary.

  1. Saturday night: fix in order of cost, and plan for the peak minute, not the average day.
  2. The menu: caching is easy. Invalidation is the design.
  3. Read replicas: be a little wrong on purpose, screen by screen, and make the safe choice the default.
  4. Double payments: build "exactly once" yourself, then reconcile in case you got it wrong.
  5. Splitting the monolith: contain failure, and split the resources, not just the code.
  6. Monitoring: one business ID per order, one trace per request, and hear about problems before your customers do.
  7. Events: keep them off the order path, and give each kind exactly the guarantee it needs.
  8. Partners: you're only as fast as the slowest system you talk to.
  9. Zones and regions: every stateful piece in more than one zone before a second region, and rehearse the restore.
  10. Data growth: a spike you can plan for, a slope you have to manage. Archive and partition before you shard.
  11. Noisy tenants: fairness at every shared layer, so one merchant's spike stays one merchant's problem.
  12. Deploys: small, judged by what customers feel, and reversible by design.
  13. The platform: make the safe path the easy path, so every team inherits what the others learned.
  14. The edge: one front door, one identity per service, and a boundary the platform generates instead of each team.

Every fix in this series was a trade we chose, sized to the traffic, the team and the partners we had at that moment.