Back to all articles
Platform at ScalePart 06
ObservabilityOpenTelemetrySRESystem designGCP

Platform at Scale, Part 6: You Can't Fix What You Can't See

Janahan Thavendran10 min read
In this article

Platform at Scale, Part 6 · One business ID per order, one trace per request, and hear about problems before your customers do.

The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.

A merchant calls on a busy Saturday. An order from table 12 was paid for but never reached the kitchen. Where did it go?

Our alerts were quiet. Platform-wide, order success looked healthy. This one order had fallen into a gap between services, and the numbers we watched were too broad to notice it.

In the monolith, we would have opened one log and searched. Now the order had passed through dine-in ordering, the orders service, payments, the provider's notice and the kitchen ticket. Five services, five sets of logs and nothing tying them together. Answering one simple question took 3 hours.

Why splitting made us blind

The problem wasn't clocks. GCP keeps servers in sync well enough. The problem was that nothing connected the pieces. A log line in payments and a log line in the kitchen service could be the same diner's dinner, and nothing said so.

Our dashboards had the same gap. They showed CPU and memory per service. A service could look perfectly healthy while orders quietly failed somewhere between two of them.

Three IDs, three jobs

We'd been saying "one ID per request". That turned out to be the wrong idea, because an order isn't one request. It's a placement, a payment, perhaps a retry, a provider notice, a kitchen ticket, partner updates and sometimes a refund, spread over minutes or days. So we used three IDs, each with one job:

ID
Order ID
What it identifies
The business event: one order
How long it lives
The whole order lifecycle, across every request and service
The question it answers
What happened to this order?
ID
Trace ID
What it identifies
One request's path through the services
How long it lives
One request
The question it answers
Where did this request spend its time?
ID
Span ID
What it identifies
One step inside that request, like a MySQL query or a provider call
How long it lives
One step
The question it answers
Which step was slow, or failed?

Every log line carried the order ID once it existed, and the trace ID always. Search the order ID and you saw every request that touched that order. Open any of those traces and you saw exactly where its time went.

What we set up on GCP

  1. A standard trace header. We used OpenTelemetry in every service and the W3C traceparent header, not a homemade one, so every service, including the partner middleware, passed trace context along the same way. Traces went to Cloud Trace.
  2. Spans where time is actually spent. Each service recorded a span on entry, and on every MySQL query, Redis call, payment provider call and partner call. A slow checkout no longer meant "search the logs". It meant opening the trace and seeing that 1.8 seconds of 2 were spent waiting for the provider.
  3. One place and one shape for logs. Every service sent structured logs to Cloud Logging with the same fields: service, trace ID, order ID, merchant ID, outlet ID, channel and event. The gateway created the trace and wrote the first access log line, so tracing started at the front door (Part 14).
  4. A search window that covers support. Order and payment logs stayed searchable for 90 days, long enough to cover merchant questions and payment disputes.

Logs became evidence, never the record. The truth about a payment still lived in MySQL and reconciliation (Parts 3 and 4). Logs explained what happened. They didn't decide it. And logs are not an audit trail: changes to menus, prices, payment states, permissions and secrets went to a separate, tamper-evident audit trail with its own retention (Part 14).

The Saturday question that took 3 hours now took 10 minutes.

One definition of success

Before the split, "order success rate" meant whatever the query behind the dashboard said. After it, two teams could report two different numbers for the same night. So we wrote the definitions down once:

Measure
Order placement, per channel
Counts as success
Order confirmed
Counts as failure
Timeouts, server errors, failures in our dependencies
Not counted
Diner-side outcomes, like an invalid cart or the diner abandoning
Measure
Payment
Counts as success
Payment reaches a final state with no platform error
Counts as failure
Timeouts, errors, payments stuck in "unknown" (Part 4)
Not counted
Card declined by the bank
Measure
Partner acknowledgement
Counts as success
Partner order acknowledged within that partner's timeout
Counts as failure
Acknowledged late, or not at all
Not counted
Orders the partner cancelled before sending

Every team, dashboard and alert used these definitions. When a merchant asked "how did Saturday go?", there was one answer.

Alerts: what to wake someone for

Each revenue channel from Part 5 got a target for peak nights, like 99.5% of orders placed successfully. Alerts fired on how fast we were burning through that target's error budget. A fast burn over the last hour paged someone. A slow burn over six hours opened a ticket for the next working day. A short blip that recovered on its own didn't wake anyone.

Every alert had three things:

  • A severity. Page, if diners or partners were affected right now. Ticket, if it could wait for the next working day. Dashboard only, for everything else.
  • An owner. The team that owned the service from Part 5, not a shared inbox.
  • A runbook. What to check first, and how to degrade safely: which fallback to switch on, and which channel to protect.

We kept the early warnings from earlier parts alongside the outcome alerts: MySQL connections near their cap (Part 1), cache rebuild spikes at menu switches (Part 2), replica lag (Part 3) and partner queue depth (Part 8). Outcome alerts tell you diners are affected. Early warnings fire before they are, and buy you minutes.

Alerts per week dropped from 120 to 15, and almost every page now needed someone to act. The same outcome measures later became the default deploy gates in every golden-path pipeline (Part 13).

Per-merchant views without exploding the bill

In a multi-tenant platform, an average can hide a disaster. One merchant's orders can fail completely while the platform average looks fine. That's exactly what happened at table 12.

But adding merchant and outlet IDs as labels on every metric creates tens of thousands of separate series, which is slow to query and expensive to keep. So we layered it:

  • Targets and alerts were measured per service and per channel.
  • Per merchant came from the logs: a "worst merchants right now" dashboard showing the merchants with the most failures, and drill-downs from there.
  • Per outlet was only queried during an investigation.

These views were internal. Merchant-facing reports only ever showed a merchant their own data.

Sampling that never drops what matters

Logs and traces cost money, and they cost most in the busiest weeks. Chinese New Year produced our biggest logging bill. So we sampled, with firm rules about what was never sampled:

  • Never sampled: errors, payment events, order state changes and partner acknowledgements.
  • Traces: sampled after the request finished, not before. Every failed or slow trace was kept, plus 5% of healthy ones.
  • Debug logs: sampled, and kept for only 7 days.

When monitoring is the thing in trouble

Watching the system must never slow it down. Logs were written asynchronously through a local buffer. If Cloud Logging was slow or unreachable, the buffer filled and debug logs were dropped first, while order, payment and error logs were kept. The trace exporter dropped spans rather than block a request.

A monitoring outage became a blind spot, never an outage.

Payments and privacy in logs

Payments added hard rules:

  • An allowlist, not good intentions. In payment code, the logging library only wrote approved fields. Anything else was dropped by the library, not by a developer remembering to.
  • No card data, ever. No card numbers, no security codes. Only the provider's tokens and our own payment reference.
  • Controlled access. Who could search payment logs was restricted, and every search was audited.
  • Shorter retention for anything near payments: 90 days.

The trade-off we made

Option
Self-hosted logging and dashboards
Why it was tempting
No per-gigabyte bill
Why we didn't choose it
Another platform to run and keep alive during the same peaks it was meant to watch
Option
A third-party monitoring tool
Why it was tempting
Strong features out of the box
Why we didn't choose it
Another contract and another place to send data, when we were already on GCP
Option
A request ID in the logs, no tracing
Why it was tempting
Simple to add
Why we didn't choose it
You can search, but you can't see where the time went
Option
Keep every log line and every trace
Why it was tempting
Never miss anything
Why we didn't choose it
The bill grows fastest in exactly the busiest weeks
Option
Cloud Logging, Monitoring and Trace, with OpenTelemetry, tail-sampled traces and money paths never sampled
Why it was tempting
Already wired into GCP, nothing new to operate, and portable instrumentation
Why we didn't choose it
Our choice

We chose the tools our team could start using that week over the ones with the most features. OpenTelemetry kept the instrumentation portable, so choosing GCP's backends didn't lock our code to them.

We accepted paying more in peak months. In return, monitoring didn't become one more system we had to keep alive on a Saturday night.

What we measured

  • Time to trace one order end to end: from 3 hours to 10 minutes
  • Incidents found by our alerts versus reported by a merchant first: from 40% to 95%
  • Alerts per week, and the share of pages that needed action
  • Log and trace cost in a peak month versus a normal month
  • Tracing overhead on checkout: 5 ms at p95

The second number mattered most. It's the Part 1 rule, measured: we should hear about a problem before the merchant does.

What it cost

Cloud Logging, Monitoring and Trace added about $2,500 a month, rising sharply in peak weeks. Instrumenting every service took real engineering time that didn't ship a single feature. It paid back the first time an incident took minutes instead of an afternoon.

What this part leaves for later

Gap left here
Carrying trace context through Pub/Sub messages and background workers
Where it's covered
Part 7: events and Cloud Run
Gap left here
Tracing partner calls through queues, and queue depth alerts
Where it's covered
Part 8: partner queues
Gap left here
Alerts on how fast one merchant's traffic is changing
Where it's covered
Part 11: noisy tenants
Gap left here
Error budgets that block deploys, not just page people
Where it's covered
Part 12: safe deploys
Gap left here
Outcome measures as the default deploy gate for every new service
Where it's covered
Part 13: the platform
Gap left here
Access logs at the front door, audit trails, and alerts on failed logins and WAF blocks
Where it's covered
Part 14: the edge

What I took from it

In a distributed system, every order needs one business ID and every request needs one trace.

Watch what your customers feel, keep the warnings that come before it, and never let the watching slow down the thing you're watching.

Next in the series: Part 7: Click events and the Cloud Run bill