Platform at Scale, Part 6 · One business ID per order, one trace per request, and hear about problems before your customers do.
The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.
A merchant calls on a busy Saturday. An order from table 12 was paid for but never reached the kitchen. Where did it go?
Our alerts were quiet. Platform-wide, order success looked healthy. This one order had fallen into a gap between services, and the numbers we watched were too broad to notice it.
In the monolith, we would have opened one log and searched. Now the order had passed through dine-in ordering, the orders service, payments, the provider's notice and the kitchen ticket. Five services, five sets of logs and nothing tying them together. Answering one simple question took 3 hours.
Why splitting made us blind
The problem wasn't clocks. GCP keeps servers in sync well enough. The problem was that nothing connected the pieces. A log line in payments and a log line in the kitchen service could be the same diner's dinner, and nothing said so.
Our dashboards had the same gap. They showed CPU and memory per service. A service could look perfectly healthy while orders quietly failed somewhere between two of them.
Three IDs, three jobs
We'd been saying "one ID per request". That turned out to be the wrong idea, because an order isn't one request. It's a placement, a payment, perhaps a retry, a provider notice, a kitchen ticket, partner updates and sometimes a refund, spread over minutes or days. So we used three IDs, each with one job:
| ID | What it identifies | How long it lives | The question it answers |
|---|---|---|---|
| Order ID | The business event: one order | The whole order lifecycle, across every request and service | What happened to this order? |
| Trace ID | One request's path through the services | One request | Where did this request spend its time? |
| Span ID | One step inside that request, like a MySQL query or a provider call | One step | Which step was slow, or failed? |
- ID
- Order ID
- What it identifies
- The business event: one order
- How long it lives
- The whole order lifecycle, across every request and service
- The question it answers
- What happened to this order?
- ID
- Trace ID
- What it identifies
- One request's path through the services
- How long it lives
- One request
- The question it answers
- Where did this request spend its time?
- ID
- Span ID
- What it identifies
- One step inside that request, like a MySQL query or a provider call
- How long it lives
- One step
- The question it answers
- Which step was slow, or failed?
Every log line carried the order ID once it existed, and the trace ID always. Search the order ID and you saw every request that touched that order. Open any of those traces and you saw exactly where its time went.
What we set up on GCP
- A standard trace header. We used OpenTelemetry in every service and the W3C
traceparentheader, not a homemade one, so every service, including the partner middleware, passed trace context along the same way. Traces went to Cloud Trace. - Spans where time is actually spent. Each service recorded a span on entry, and on every MySQL query, Redis call, payment provider call and partner call. A slow checkout no longer meant "search the logs". It meant opening the trace and seeing that 1.8 seconds of 2 were spent waiting for the provider.
- One place and one shape for logs. Every service sent structured logs to Cloud Logging with the same fields: service, trace ID, order ID, merchant ID, outlet ID, channel and event. The gateway created the trace and wrote the first access log line, so tracing started at the front door (Part 14).
- A search window that covers support. Order and payment logs stayed searchable for 90 days, long enough to cover merchant questions and payment disputes.
Logs became evidence, never the record. The truth about a payment still lived in MySQL and reconciliation (Parts 3 and 4). Logs explained what happened. They didn't decide it. And logs are not an audit trail: changes to menus, prices, payment states, permissions and secrets went to a separate, tamper-evident audit trail with its own retention (Part 14).
The Saturday question that took 3 hours now took 10 minutes.
One definition of success
Before the split, "order success rate" meant whatever the query behind the dashboard said. After it, two teams could report two different numbers for the same night. So we wrote the definitions down once:
| Measure | Counts as success | Counts as failure | Not counted |
|---|---|---|---|
| Order placement, per channel | Order confirmed | Timeouts, server errors, failures in our dependencies | Diner-side outcomes, like an invalid cart or the diner abandoning |
| Payment | Payment reaches a final state with no platform error | Timeouts, errors, payments stuck in "unknown" (Part 4) | Card declined by the bank |
| Partner acknowledgement | Partner order acknowledged within that partner's timeout | Acknowledged late, or not at all | Orders the partner cancelled before sending |
- Measure
- Order placement, per channel
- Counts as success
- Order confirmed
- Counts as failure
- Timeouts, server errors, failures in our dependencies
- Not counted
- Diner-side outcomes, like an invalid cart or the diner abandoning
- Measure
- Payment
- Counts as success
- Payment reaches a final state with no platform error
- Counts as failure
- Timeouts, errors, payments stuck in "unknown" (Part 4)
- Not counted
- Card declined by the bank
- Measure
- Partner acknowledgement
- Counts as success
- Partner order acknowledged within that partner's timeout
- Counts as failure
- Acknowledged late, or not at all
- Not counted
- Orders the partner cancelled before sending
Every team, dashboard and alert used these definitions. When a merchant asked "how did Saturday go?", there was one answer.
Alerts: what to wake someone for
Each revenue channel from Part 5 got a target for peak nights, like 99.5% of orders placed successfully. Alerts fired on how fast we were burning through that target's error budget. A fast burn over the last hour paged someone. A slow burn over six hours opened a ticket for the next working day. A short blip that recovered on its own didn't wake anyone.
Every alert had three things:
- A severity. Page, if diners or partners were affected right now. Ticket, if it could wait for the next working day. Dashboard only, for everything else.
- An owner. The team that owned the service from Part 5, not a shared inbox.
- A runbook. What to check first, and how to degrade safely: which fallback to switch on, and which channel to protect.
We kept the early warnings from earlier parts alongside the outcome alerts: MySQL connections near their cap (Part 1), cache rebuild spikes at menu switches (Part 2), replica lag (Part 3) and partner queue depth (Part 8). Outcome alerts tell you diners are affected. Early warnings fire before they are, and buy you minutes.
Alerts per week dropped from 120 to 15, and almost every page now needed someone to act. The same outcome measures later became the default deploy gates in every golden-path pipeline (Part 13).
Per-merchant views without exploding the bill
In a multi-tenant platform, an average can hide a disaster. One merchant's orders can fail completely while the platform average looks fine. That's exactly what happened at table 12.
But adding merchant and outlet IDs as labels on every metric creates tens of thousands of separate series, which is slow to query and expensive to keep. So we layered it:
- Targets and alerts were measured per service and per channel.
- Per merchant came from the logs: a "worst merchants right now" dashboard showing the merchants with the most failures, and drill-downs from there.
- Per outlet was only queried during an investigation.
These views were internal. Merchant-facing reports only ever showed a merchant their own data.
Sampling that never drops what matters
Logs and traces cost money, and they cost most in the busiest weeks. Chinese New Year produced our biggest logging bill. So we sampled, with firm rules about what was never sampled:
- Never sampled: errors, payment events, order state changes and partner acknowledgements.
- Traces: sampled after the request finished, not before. Every failed or slow trace was kept, plus 5% of healthy ones.
- Debug logs: sampled, and kept for only 7 days.
When monitoring is the thing in trouble
Watching the system must never slow it down. Logs were written asynchronously through a local buffer. If Cloud Logging was slow or unreachable, the buffer filled and debug logs were dropped first, while order, payment and error logs were kept. The trace exporter dropped spans rather than block a request.
A monitoring outage became a blind spot, never an outage.
Payments and privacy in logs
Payments added hard rules:
- An allowlist, not good intentions. In payment code, the logging library only wrote approved fields. Anything else was dropped by the library, not by a developer remembering to.
- No card data, ever. No card numbers, no security codes. Only the provider's tokens and our own payment reference.
- Controlled access. Who could search payment logs was restricted, and every search was audited.
- Shorter retention for anything near payments: 90 days.
The trade-off we made
| Option | Why it was tempting | Why we didn't choose it |
|---|---|---|
| Self-hosted logging and dashboards | No per-gigabyte bill | Another platform to run and keep alive during the same peaks it was meant to watch |
| A third-party monitoring tool | Strong features out of the box | Another contract and another place to send data, when we were already on GCP |
| A request ID in the logs, no tracing | Simple to add | You can search, but you can't see where the time went |
| Keep every log line and every trace | Never miss anything | The bill grows fastest in exactly the busiest weeks |
| Cloud Logging, Monitoring and Trace, with OpenTelemetry, tail-sampled traces and money paths never sampled | Already wired into GCP, nothing new to operate, and portable instrumentation | Our choice |
- Option
- Self-hosted logging and dashboards
- Why it was tempting
- No per-gigabyte bill
- Why we didn't choose it
- Another platform to run and keep alive during the same peaks it was meant to watch
- Option
- A third-party monitoring tool
- Why it was tempting
- Strong features out of the box
- Why we didn't choose it
- Another contract and another place to send data, when we were already on GCP
- Option
- A request ID in the logs, no tracing
- Why it was tempting
- Simple to add
- Why we didn't choose it
- You can search, but you can't see where the time went
- Option
- Keep every log line and every trace
- Why it was tempting
- Never miss anything
- Why we didn't choose it
- The bill grows fastest in exactly the busiest weeks
- Option
- Cloud Logging, Monitoring and Trace, with OpenTelemetry, tail-sampled traces and money paths never sampled
- Why it was tempting
- Already wired into GCP, nothing new to operate, and portable instrumentation
- Why we didn't choose it
- Our choice
We chose the tools our team could start using that week over the ones with the most features. OpenTelemetry kept the instrumentation portable, so choosing GCP's backends didn't lock our code to them.
We accepted paying more in peak months. In return, monitoring didn't become one more system we had to keep alive on a Saturday night.
What we measured
- Time to trace one order end to end: from 3 hours to 10 minutes
- Incidents found by our alerts versus reported by a merchant first: from 40% to 95%
- Alerts per week, and the share of pages that needed action
- Log and trace cost in a peak month versus a normal month
- Tracing overhead on checkout: 5 ms at p95
The second number mattered most. It's the Part 1 rule, measured: we should hear about a problem before the merchant does.
What it cost
Cloud Logging, Monitoring and Trace added about $2,500 a month, rising sharply in peak weeks. Instrumenting every service took real engineering time that didn't ship a single feature. It paid back the first time an incident took minutes instead of an afternoon.
What this part leaves for later
| Gap left here | Where it's covered |
|---|---|
| Carrying trace context through Pub/Sub messages and background workers | Part 7: events and Cloud Run |
| Tracing partner calls through queues, and queue depth alerts | Part 8: partner queues |
| Alerts on how fast one merchant's traffic is changing | Part 11: noisy tenants |
| Error budgets that block deploys, not just page people | Part 12: safe deploys |
| Outcome measures as the default deploy gate for every new service | Part 13: the platform |
| Access logs at the front door, audit trails, and alerts on failed logins and WAF blocks | Part 14: the edge |
- Gap left here
- Carrying trace context through Pub/Sub messages and background workers
- Where it's covered
- Part 7: events and Cloud Run
- Gap left here
- Tracing partner calls through queues, and queue depth alerts
- Where it's covered
- Part 8: partner queues
- Gap left here
- Alerts on how fast one merchant's traffic is changing
- Where it's covered
- Part 11: noisy tenants
- Gap left here
- Error budgets that block deploys, not just page people
- Where it's covered
- Part 12: safe deploys
- Gap left here
- Outcome measures as the default deploy gate for every new service
- Where it's covered
- Part 13: the platform
- Gap left here
- Access logs at the front door, audit trails, and alerts on failed logins and WAF blocks
- Where it's covered
- Part 14: the edge
What I took from it
In a distributed system, every order needs one business ID and every request needs one trace.
Watch what your customers feel, keep the warnings that come before it, and never let the watching slow down the thing you're watching.
Next in the series: Part 7: Click events and the Cloud Run bill