Platform at Scale, Part 2 · Caching is easy. Invalidation is the design.
The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.
A diner opens the menu, on the table tablet, in the app or on the website. Before they tap anything, the screen has to show them the restaurant: categories, dishes, photos, prices, add-ons. That's the menu, and on a busy Saturday it was requested more than anything else on the platform.
It was also the thing that changed at the worst moments. A dish runs out mid-service. Breakfast becomes lunch at 11:00. A merchant adjusts a price before dinner.
Caching the menu was easy. Knowing the exact moment the cached menu stopped being true was the hard part.
Why the menu matters to the business
For a merchant, the menu is the storefront. A menu that loads slowly is a diner who gives up, or waits for a server to take the order instead. A wrong menu is worse. If it shows a dish the kitchen has run out of, the diner orders it, the kitchen rejects it, someone apologises and someone refunds. On delivery, a rejected order can also count against the outlet with the delivery platform.
So the menu had two jobs that pull against each other: be fast, and be right.
What one menu actually is
"The menu" sounds like one thing. It isn't. To show one outlet's menu on one channel, the platform pulled together:
- Categories and their display order
- Items, photos and descriptions
- Modifier groups and options, like size, spice level or add-ons, each with its own prices and rules
- Prices, which could differ by channel. Dine-in and delivery prices were often not the same
- Outlet-level overrides on a group-wide menu
- Menu periods, like breakfast, lunch and dinner
- Availability: what's sold out right now
Building that from MySQL took about 40 queries per menu. On Monday nobody noticed. On Saturday, every table tablet, phone, website visit and partner sync asked for it, and the answer barely changed between requests. Menu reads were roughly 45% of all database reads at peak.
The design
We put Redis in front of MySQL, run as Memorystore on GCP. A device asks for a menu. We check Redis. If the built menu is there, it goes straight back as one ready-made document. If not, we build it from MySQL, store it in Redis on the way out, and return it. Menu photos, the heaviest part of any menu, never went through Redis at all. They came from a CDN with signed links, so our servers never served the same photo twice.
Menus weren't the only thing in Redis. Outlet settings and partner item mappings, the other lookups from Part 1, followed the same pattern: built from MySQL, stored under a versioned key, and checked against MySQL wherever an order or a payment depended on them. This part follows the menu because it changed most often and hurt most when it was wrong.
The detail that mattered most was the key:
menu:{merchant}:{outlet}:{channel}:{period}:v{version}
Every part earned its place. Merchant and outlet, so one tenant's menu could never collide with another's in the cache. Channel, because dine-in and delivery prices differed. Period, because breakfast and lunch are different menus. Version, because of how we handled change.
A key keeps data apart. It doesn't decide who may read it. So before any cache lookup, the API checked that the device or session asking was allowed to read that outlet. A request for an outlet it didn't belong to was refused before Redis was touched. Later the gateway made this check once for every service (Part 14). Key design is not authorisation.
Bump the version, don't hunt the keys
Each menu had a small pointer saying which version was live: one key per merchant, outlet, channel and period, holding a single number. A request read the pointer first, then the menu stored under that version. Both were tiny reads. The pointer was also stored in MySQL, so if it ever went missing from Redis we could restore it.
An ordinary edit built the new version, then moved the pointer to it in one step. The next request found the new version and the old one was never read again.
A scheduled change worked differently. A festive price list starting at 17:00 was built ahead of time, but the pointer stayed where it was. At 17:00, the pointer moved. Nobody saw the new prices before their start time, and nobody saw the old ones after.
That beat deleting keys. Deleting means finding every key a change affects, across every channel and period, and missing one means a stale menu. Bumping a version changes one number, and every stale copy becomes unreachable at once.
Old versions weren't deleted. They expired 10 minutes after the pointer moved: long enough for any request already reading them to finish, short enough that dead menus didn't pile up in memory.
Split what rarely changes from what changes all the time
A sold-out flag is tiny. Rebuilding a whole menu because one dish ran out was wasteful, and during a busy service merchants marked items sold out constantly.
So we split the menu in two:
- Structure: categories, items, photos, modifiers and prices. Cached for the whole menu period, changed only by a merchant edit.
- Availability: a small set of sold-out item IDs per outlet. Kept separately, updated the moment a merchant tapped "sold out", and pushed to devices.
The device combined the two. A dish running out changed a few bytes, not the whole menu.
If the availability set was ever missing from Redis, the server rebuilt it from MySQL, where every sold-out tap was also recorded. It never treated a missing set as "everything is available". Missing meant "ask the database", never "assume yes", because a wrong "available" turns straight into a rejected order.
What triggered a change
| Trigger | How often | What we did | How fast it had to show |
|---|---|---|---|
| Merchant edits the menu | A few times a week | Bump the version | Next request |
| Item sold out, or back in stock | Many times a service | Update the availability set and push to devices | Seconds |
| Menu period changes, like breakfast to lunch | Fixed times every day | A key per period, with the next period built early | At the exact minute |
| Scheduled price change | Occasional | Build the version early, move the pointer at the start time | At the exact minute |
| Delivery partner menu sync | On every edit | Push the change to each partner through the middleware | Minutes, depending on the partner |
- Trigger
- Merchant edits the menu
- How often
- A few times a week
- What we did
- Bump the version
- How fast it had to show
- Next request
- Trigger
- Item sold out, or back in stock
- How often
- Many times a service
- What we did
- Update the availability set and push to devices
- How fast it had to show
- Seconds
- Trigger
- Menu period changes, like breakfast to lunch
- How often
- Fixed times every day
- What we did
- A key per period, with the next period built early
- How fast it had to show
- At the exact minute
- Trigger
- Scheduled price change
- How often
- Occasional
- What we did
- Build the version early, move the pointer at the start time
- How fast it had to show
- At the exact minute
- Trigger
- Delivery partner menu sync
- How often
- On every edit
- What we did
- Push the change to each partner through the middleware
- How fast it had to show
- Minutes, depending on the partner
The last row was outside our control. Our cache could be perfectly fresh while a delivery app still showed the old menu, because the partner hadn't processed our update yet. So a partner order for a sold-out dish still had to be caught at our end.
The 11:00 problem
A diner starts ordering at 10:55. They add a breakfast set and a coffee to the cart, change their mind about a side, and press "Place order" at 11:01.
At 11:00 the outlet switched from the breakfast menu to the lunch menu. The breakfast set no longer exists. Either the order fails with an error the diner doesn't understand, or worse, it goes through for a dish the kitchen stopped making a minute ago.
Menu periods change the truth on a clock. Sold-out items change it with a tap. Both can happen while a diner is halfway through an order. Here's how we handled it:
- Every cached menu carried its end time. It expired at that exact moment, not on a fixed timer.
- Build the next menu before it's needed. Many outlets switch at the same times, like 11:00 and 17:00. If every cache expired together and rebuilt from MySQL at once, we'd create our own spike. So the next period's menu was built a few minutes early and waited under its own key. If a key was missing anyway, the first request took a short lock in Redis, a key that can only be set if it doesn't already exist and expires after a few seconds, and rebuilt the menu. Every other request for that menu waited briefly and read the result instead of hitting MySQL. If the builder died, the lock expired and the next request took over.
- Tell the device when its menu ends. Devices and apps got the end time with the menu and refreshed on their own at the switch.
- Warn before the switch. A few minutes before the cut-off, the screen showed "Breakfast ends at 11:00", so diners could finish in time or move to the next menu.
- Check the cart against the live menu at checkout. The server checked every cart item, price and period against MySQL at the moment of ordering, never against what the device had cached.
- Make the gap clear, not broken. If items had expired, the diner saw which ones and why. The new menu loaded, the rest of the cart stayed, and similar items from the new menu were suggested in their place.
- Let the merchant set a grace window. Some kitchens can still make breakfast at 11:01. Some can't. Merchants chose a short grace period for carts started before the cut-off, or none at all.
The grace window was a product decision, not a caching one. A longer window is kinder to the diner and harder on the kitchen. We made it the merchant's call, because they know their kitchen.
When Redis has a bad day
A cache you depend on is a new way to fail. If Memorystore became unreachable at peak, every menu request would fall through to MySQL at once, which is exactly the load we cached to avoid.
We protected against that in two places. Devices kept the last menu they received and kept showing it, so a cache outage didn't blank every ordering screen. And server-side rebuilds were limited, so a cache problem made menus slower instead of taking the database down with it. Because checkout always validated against MySQL, a stale menu could slow a diner down but never charge them wrongly.
What the diner saw mattered as much as what the server did. While the device could reach the server, ordering stayed open on its last menu with no warning, because checkout still checked every item against MySQL. If something had changed, the diner got the same clear message as after a menu switch: what changed, the updated menu, and the rest of the cart kept. If the device couldn't reach the server at all, it paused new orders and said so, rather than accept an order it couldn't confirm.
How much memory it took
The cache held one menu per outlet, per channel, per period, plus old versions waiting to expire. That came to about 6 GB on Memorystore. Least-used keys were evicted first, so menus for outlets closed at night quietly made room for the ones about to open.
The trade-off we made
| Option | Why it was tempting | Why we didn't choose it |
|---|---|---|
| A short fixed expiry on every menu, like 60 seconds | Simple, no change-handling code | Either stale for up to a minute, so sold-out dishes stay orderable, or so short the cache barely helps at peak |
| Delete the affected keys on every change | Exact | Easy to miss one key across channels and periods |
| A cache inside each app server | No new infrastructure | Every server holds a different copy, and a change has to reach all of them |
| Read replicas for menu reads | Spreads the load | Replicas move the work, caching removes it. Every menu would still be built thousands of times a night, just on another machine |
| Versioned keys in Memorystore, availability split out, checked at checkout | Fast reads, one number to change, correct where money moves | Our choice |
- Option
- A short fixed expiry on every menu, like 60 seconds
- Why it was tempting
- Simple, no change-handling code
- Why we didn't choose it
- Either stale for up to a minute, so sold-out dishes stay orderable, or so short the cache barely helps at peak
- Option
- Delete the affected keys on every change
- Why it was tempting
- Exact
- Why we didn't choose it
- Easy to miss one key across channels and periods
- Option
- A cache inside each app server
- Why it was tempting
- No new infrastructure
- Why we didn't choose it
- Every server holds a different copy, and a change has to reach all of them
- Option
- Read replicas for menu reads
- Why it was tempting
- Spreads the load
- Why we didn't choose it
- Replicas move the work, caching removes it. Every menu would still be built thousands of times a night, just on another machine
- Option
- Versioned keys in Memorystore, availability split out, checked at checkout
- Why it was tempting
- Fast reads, one number to change, correct where money moves
- Why we didn't choose it
- Our choice
The deciding point was where the truth had to live. Browsing could tolerate a menu that was seconds old. Checkout could not tolerate anything but the truth. So we let the cache serve the browsing and made MySQL the judge at checkout. That way, a mistake in invalidation cost a few seconds of an outdated screen, never a wrong order or a wrong charge.
What we measured
- Menu cache hit rate: 97%
- Menu load time at p95: from 2.5 seconds to 300 ms
- MySQL reads per second at Saturday peak: from 9,000 to 3,500
- Orders rejected because an item was unavailable: from about 400 to 60 a week, before and after the availability split
- Refunds caused by menu mismatches: from about 120 to under 20 a month
We also watched two numbers the hit rate hides: how long a rebuild took on a miss, and how many rebuilds ran at once. The average hit rate looked great all day. The real risk was a dip at exactly 11:00 and 17:00, when many outlets switched menus together.
What it cost
Memorystore added about $700 a month. The bigger cost was engineering discipline: every path that changed a menu had to bump the version or update availability, and every new feature had to respect that. Once we had golden paths (Part 13), new services got the key shape and the version pointer generated for them, instead of remembered. Devices needed their own logic for end times, warnings and offline menus.
Against that, we avoided two more upgrades of the Cloud SQL instance, and the database could spend its Saturday on orders instead of rebuilding the same menu thousands of times.
What this part leaves for later
| Gap left here | Where it's covered |
|---|---|
| Partner menu updates arriving out of order or retried | Part 8: partner queues, with a version on every update |
| A partner order for a stale item becoming a duplicate or a double charge | Part 4: idempotency and reconciliation |
| Watching cache health alongside everything else after the split | Part 6: logs and monitoring |
| Memorystore running in a single zone | Part 9: zones and regions |
| One merchant's menu pushes crowding out everyone else's | Part 11: noisy tenants |
| Every new service reusing this cache pattern without copying it | Part 13: the platform |
| Which web origins may call the menu API, and who can edit a menu | Part 14: the edge |
- Gap left here
- Partner menu updates arriving out of order or retried
- Where it's covered
- Part 8: partner queues, with a version on every update
- Gap left here
- A partner order for a stale item becoming a duplicate or a double charge
- Where it's covered
- Part 4: idempotency and reconciliation
- Gap left here
- Watching cache health alongside everything else after the split
- Where it's covered
- Part 6: logs and monitoring
- Gap left here
- Memorystore running in a single zone
- Where it's covered
- Part 9: zones and regions
- Gap left here
- One merchant's menu pushes crowding out everyone else's
- Where it's covered
- Part 11: noisy tenants
- Gap left here
- Every new service reusing this cache pattern without copying it
- Where it's covered
- Part 13: the platform
- Gap left here
- Which web origins may call the menu API, and who can edit a menu
- Where it's covered
- Part 14: the edge
What I took from it
Caching is easy. Invalidation is the design.
Cache what people browse, split what changes often from what rarely changes, and check the truth at the moment money changes hands.
Next in the series: Part 3: When the replica is a few seconds behind