Back to all articles
Platform at ScalePart 07
Event-drivenPub/SubCloud RunSystem designGCP

Platform at Scale, Part 7: Click Events and the Cloud Run Bill

Janahan Thavendran10 min read
In this article

Platform at Scale, Part 7 · Keep events off the order path, and give each kind exactly the guarantee it needs.

The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.

Every time a diner taps a dish, adds it to the cart or removes it, we record an event. One click on its own is worthless. Millions of them are what our recommendation models learned from: which dishes people look at, which they add and which they drop.

Those events were supposed to be background data. On a busy night they started fighting the orders for the same servers.

Why it broke

The first version was the simple one. The app sent click events to the same backend that took orders, and the backend wrote them as they arrived.

A diner placing one order might generate 15 click events before paying. So at peak, events outnumbered orders many times over. Writing them in the request path held threads and database connections, the same lag chain as Part 1, with the least important data as the trigger.

And click events weren't the only side work in the request. Part 1 showed that kitchen tickets, loyalty points and partner updates also ran while the diner waited. This part moves click events and the "order placed" event that drives kitchen tickets and loyalty. Partner calls get their own part, Part 8.

Not all events are equal

Before building anything, we sorted events by what happens when one goes wrong:

Kind
Analytics events
Examples
Dish viewed, added, removed
If one arrives late
Fine, even by minutes
If one arrives twice
Skews a model or a report
What we built
Delivered at least once, duplicates removed before storage
Kind
Business events
Examples
Order placed, driving the kitchen ticket and loyalty points
If one arrives late
Kitchen: seconds matter. Points: minutes are fine
If one arrives twice
A second ticket or double points. Not acceptable
What we built
Never lost, applied exactly once by each consumer, and reconciled

It's the same principle as Part 4, at different stakes. A duplicate payment charges a diner. Duplicate points give away money slowly. A duplicate click quietly teaches a model the wrong thing.

Analytics events: off the order path

We moved click events to Pub/Sub. The app published an event and moved on, without waiting for anything to be stored. A separate consumer picked events up and wrote them to our analytics store. If analytics fell behind, events waited in Pub/Sub. Diners never noticed.

The event ID, and where duplicates die

The device created each event's ID at the moment the event happened, and kept the event with its ID in a small local queue until Pub/Sub accepted it. A retry resent the same event with the same ID. It never created a new one. Every event also carried a schema version, and device and diner identifiers were hashed before publishing.

Pub/Sub delivers every message at least once, so duplicates still arrived, just like the provider notices in Part 4. The consumer dropped IDs it had already seen before grouping events into a batch, so one duplicate could never make a whole batch fail. The analytics store also refused a second event with the same ID, as a second guard. Models and reports both read clean data.

Order

Pub/Sub doesn't promise order unless you ask for it. For analytics we didn't. Our models learned from totals per dish and per session, not from the exact sequence of taps. Every event still carried its device time and a sequence number within the session, so the order could be rebuilt when an analysis needed it.

Then the consumers broke instead

Our first consumers were Cloud Functions, handling one event per instance. That looked tidy, and at peak it fell apart:

  • Too many instances. A burst of events meant a burst of new instances.
  • Cold starts. New instances took time to start, right when the backlog was growing fastest.
  • Too many connections. Every new instance opened its own connection to the store downstream. A spike of instances meant a spike of connections, and writes started failing.
  • Retry storms. Failed events were retried immediately, which created more load, which caused more failures.
  • Duplicates. Every retry was another chance to count the same click twice.

What we changed

  1. Moved consumers to Cloud Run. One instance handled many events at once, so the instance count stayed sane, and each instance had a capped connection pool (Part 1's connection math again). Every publisher and subscriber had its own Pub/Sub service account (Part 14).
  2. Removed duplicates, then batched. Events were written in groups, cutting connections and cost, after duplicates were dropped.
  3. Retried with backoff. Each retry waited longer than the last, up to 5 attempts, with a limit on how many events each instance worked on at once. A failing store got breathing room instead of a stampede.
  4. Added a dead-letter topic. Events that used up their retries went there. Its depth was an alert owned by the data team (Part 6). They inspected it every day, fixed the cause and replayed the events, so the dead-letter topic was a safety net, not a graveyard.
  5. Warmed up before known peaks, sized like Part 1. Forecast peak events per minute, divide by what one instance handles per minute, add a margin. That was the minimum number of running instances for festive weekends, and only for those weekends. Later this sizing became a golden-path setting, so a new event consumer didn't have to rediscover it (Part 13).

Business events: never lost, never applied twice

Part 5 said the orders service publishes an "order placed" event, and loyalty listens. Doing that safely needed more care than click events did.

The outbox: save the order and the event together

The naive version saves the order, then publishes the event. If the publish fails, the event is lost: no kitchen ticket, no points. Publish first and roll back the order, and you get a ghost event for an order that doesn't exist.

So the orders service wrote the order and an "order placed" row into an outbox table in the same MySQL transaction. Either both were saved or neither was. A small relay read unsent rows from the outbox, published them to Pub/Sub and marked them sent. If publishing failed, the row stayed and was tried again. The relay could occasionally publish a row twice, which is why every consumer had to handle duplicates.

Every message also carried the trace context from Part 6 in its attributes. A kitchen ticket's trace started at the diner's tap.

Kitchen tickets: fast, in order, once

The diner's confirmation no longer waited for the kitchen. The kitchen service received "order placed" events with the outlet ID as the ordering key, so an outlet's tickets arrived in the order the orders were placed. Each ticket was keyed on the order ID, so a second delivery didn't print a second ticket. Our target was a ticket on the kitchen screen within 3 seconds of confirmation.

Loyalty: late is fine, wrong is not

The loyalty service recorded "points awarded for order X" with a unique constraint on the order ID. A second delivery of the same event did nothing. Every day, a reconciliation checked that each completed order had exactly one points entry and fixed any gap. Points could arrive a few minutes late, but never twice, and never missing.

The trade-off we made

Option
Keep writing events in the order request
Why it was tempting
Simple, nothing new
Why we didn't choose it
The least important data slowed down the most important action
Option
Run our own Kafka
Why it was tempting
Powerful, ordered streams
Why we didn't choose it
Brokers to operate and tune, more than our team and our volume justified
Option
A full streaming pipeline (Dataflow)
Why it was tempting
Built for heavy event processing
Why we didn't choose it
More cost and specialist skill than we needed at our volume then
Option
Publish to Pub/Sub straight after saving the order
Why it was tempting
One less table
Why we didn't choose it
A failure between the two steps loses an event or creates a ghost one
Option
Pub/Sub with Cloud Run consumers: deduplicated analytics, plus an outbox and exactly-once consumers for business events
Why it was tempting
Managed, scales on its own, and the right guarantee for each kind of event
Why we didn't choose it
Our choice

We started with Cloud Functions because it was the fastest to ship. When it broke under peak load, we moved to Cloud Run rather than to something heavier. It fixed the actual problems, the instance count, cold starts and connections, without adding a platform to run.

The bigger choice was refusing to give every event the same guarantee. Analytics could be minutes late and lightly duplicated before cleanup. Business events could be a little late, but never lost and never applied twice. Paying for the strongest guarantee everywhere would have been expensive. Paying for the weakest everywhere would have printed double tickets.

What we measured

  • Click events per order: 15
  • Consumer instances at peak: from about 900 on Cloud Functions to 40 on Cloud Run
  • Analytics event delay at p95: 3 minutes
  • Duplicate rate arriving at the consumer: 1.5%, and zero in the store
  • Kitchen ticket time at p95: 2 seconds from confirmation
  • Loyalty gaps found by reconciliation: 5 a day
  • Monthly event pipeline cost: from about $6,500 to about $2,800

What it cost

Pub/Sub, Cloud Run and analytics storage added about $2,800 a month. Warm instances cost money on quiet days too, so we only kept them for known peaks. The outbox and its relay were one more moving part on the order path. And analytics data was no longer instant: 3 minutes late at p95, which was fine for training models and building reports, and would never have been fine for anything a diner waits on.

What this part leaves for later

Gap left here
Sending orders to delivery partners, and partner status updates
Where it's covered
Part 8: partner queues
Gap left here
Partners that can't handle orders arriving at the same time
Where it's covered
Part 8: partner queues
Gap left here
Shedding analytics events first when the platform is overloaded
Where it's covered
Part 11: noisy tenants
Gap left here
Analytics data as the base for a warehouse of old orders
Where it's covered
Part 10: data growth

What I took from it

Events are not orders, and not all events are equal.

Keep every one of them off the order path. Expect every one to arrive twice. Decide for each kind whether late, duplicated or lost is acceptable, and build exactly that guarantee. Then size your consumers for the rush, not for Monday.

Next in the series: Part 8: Partners that take one order at a time