Back to all articles
Platform at ScalePart 04
PaymentsIdempotencyReconciliationSystem designMySQL

Platform at Scale, Part 4: Two Taps, One Charge

Janahan Thavendran11 min read
In this article

Platform at Scale, Part 4 · Build "exactly once" yourself, then reconcile in case you got it wrong.

The figures in this series are illustrative. They're rounded and changed to protect confidential data, and they show the shape of each problem rather than the company's actual numbers.

A diner taps "Pay". The spinner keeps turning. They tap again. Two minutes later their bank app shows two charges for one dinner.

They don't call the bank. They call the restaurant, and the restaurant calls us. That was the most expensive kind of bug we had, because it cost money and trust in the same moment.

Why it happened

On a quiet day, payments looked solid. Under festive load, the gaps showed. Responses slowed down, and every slow response invited a retry.

A double charge needs only two attempts at the same payment, and a busy platform offers plenty of routes to that:

  • A diner taps twice because the first tap seems to do nothing.
  • The device times out waiting and retries, while the first attempt quietly succeeds.
  • The payment provider sends its "payment succeeded" notice twice, and we process both.
  • Two devices at the same table, a tablet and a phone, both pay the same bill.

Orders had their own races. Two people at one table add items to the same bill in the same second, and one addition overwrites the other. Or the last portion of a limited dish gets sold twice.

None of it showed up in testing with one user. It only appeared when many people acted at once.

Part 1 had already fixed two things: a unique index stopped duplicate partner orders, and the payment call moved out of the open database transaction. This part had to go further without undoing that second lesson. Nothing could hold a lock while waiting for a payment provider.

Fix 1: one payment intent, one key

When a diner starts paying, the device creates one idempotency key for that intent: this bill, this amount, this attempt to pay. It saves the key on the device and keeps it until it receives a final result. If the app restarts or the page reloads mid-payment, the retry uses the same key, not a new one.

On the server, the key is stored with what it was for: the bill and the amount. A retry with the same key but a different amount is rejected. Keys are kept for 72 hours, longer than any realistic retry.

The server also stores the full response against the key, errors included. If the first attempt returned an error but the charge actually went through, the retry gets the true resolved state, never a fresh attempt. Later, the golden path gave every new service that touched payments this key lifecycle and state machine by default (Part 13).

Fix 2: every payment has a state, and every state has one answer

A key alone doesn't answer the hard question: what if the retry arrives while the first attempt is still talking to the provider? So every payment moved through explicit states, written to MySQL before each next step. Every state change was also written to an audit trail recording what changed it and when (Part 14).

A payment starts as created, moves to sent to provider, and ends as succeeded, failed or unknown. "Unknown" means the provider timed out, and it's resolved by asking the provider, never by charging again.

State
Created, or sent to provider
What a retry with the same key gets
"Processing". The device polls the payment status, which reads from the primary
State
Succeeded
What a retry with the same key gets
The same success response as the first attempt
State
Failed
What a retry with the same key gets
The same failure. To try again, the diner starts a new payment intent with a new key
State
Unknown, after a provider timeout
What a retry with the same key gets
Nothing is charged again. We ask the provider what happened first

Fix 3: our call to the provider is idempotent too

The nastiest case is a crash in the middle. The provider charges the card, and our server dies before writing "succeeded". A naive retry charges again.

So our outbound call had to be as safe as our inbound one. Where a provider supported it, we sent our payment reference as their idempotency key, so a repeated call returned the original charge instead of making a new one. Where they didn't, a recovery job looked for payments stuck in "sent" or "unknown" for more than 60 seconds, asked the provider for the outcome by our reference, and wrote the real result. Where a provider separated authorising a card from capturing the money, each step had its own key, so a retry could never capture twice. Provider API keys lived in Secret Manager with scheduled rotation, and the payments service ran under its own service account, with no access to menus or analytics.

Fix 4: one payment per bill at a time

A per-payment key doesn't stop two devices at one table. Each device creates its own key, so both payments look new.

That needed a bill-level rule: only one payment in progress per bill. MySQL doesn't support conditional unique indexes, so we used a pattern it does support. The payments table had an extra column that held the bill ID while a payment was in progress and was empty otherwise, with a unique index on it. MySQL allows many empty values in a unique index but only one copy of each bill ID. A second device trying to pay the same bill was refused by the database itself, and saw "A payment for this bill is already in progress" with its live status.

Fix 5: short locks, released before the provider call

The bill still needed protecting while its total was checked and the payment intent was written. The order of steps was the whole design:

  1. Start a transaction and lock the bill row.
  2. Check the total and the items against the live menu (Part 2).
  3. Write the payment intent as "created".
  4. Commit, which releases the lock.
  5. Call the provider, with no transaction open and no locks held.
  6. Write the result in a new, short transaction.

Locks were always taken in the same order: bill, then stock, then payment. A consistent order prevents most deadlocks. When MySQL still picked a deadlock victim, we retried that request once.

This gave us serialisation per bill, not per platform. Only requests touching the same bill waited for each other. Every other table carried on.

Fix 6: design the race out where you can

Locks are a last resort. Where we could, we changed the data model so there was nothing to overwrite.

  • Shared bills: each item became its own order line row. Adding an item is an insert, not an update to a shared record, and the total is calculated from the lines. Two people adding items at once both land. Row locks were kept only for genuinely shared values like a discount, a tip or a service charge.
  • Limited stock: one atomic statement, with no lock held: UPDATE dish_stock SET remaining = remaining - 1 WHERE dish_id = ? AND remaining > 0. If it changes one row, the dish is yours. If it changes none, it's sold out, and the diner is told before paying.

Fix 7: provider notices move forward, never back

Before anything else, every provider notice was signature-checked and IP-allowlisted (Part 14). Removing duplicates is worthless if anyone can post a fake "payment succeeded". Genuine notices can still arrive twice, late, or out of order. A success can arrive before the "created" notice, or a late failure after a success.

So the notice handler matched on the provider's transaction ID and only moved a payment forward. Final states like "succeeded" or "refunded" never went back to an earlier state. A late "failed" after "succeeded" was logged for reconciliation, not applied. The handler always read and wrote the primary, never a replica that might be seconds behind (Part 3).

Fix 8: an honest button

The pay button disabled after the first tap and showed the real state from the server: processing, succeeded or failed. Half the double taps disappeared once diners could see something was happening.

Fix 9: reconcile in case we got it wrong

Prevention is never perfect, so reconciliation was the safety net, not an afterthought.

Every day, our payment records, the provider's reports and the bank settlement were matched three ways. In festive weeks we ran it every 4 hours, because volume and risk were both higher. The job ignored payments younger than 30 minutes, so a slow payment wasn't mistaken for a missing one. It read from the primary and the provider's own records, never from a replica.

Refunds were idempotent too. Each refund carried its own key built from the payment ID and the reason, so a reconciliation job that ran twice could never refund twice. Every exception went to a named owner in payments operations, with a target to refund a duplicate charge within 24 hours, before the diner had to ask.

Duplicate charges dropped from about 300 a month to under 10.

The trade-off we made

Option
Rely only on the provider's duplicate checks
Why it was tempting
No work for us
Why we didn't choose it
Not consistent across providers, and no help with our own races. We still passed our key to providers that supported it
Option
Locks in Redis
Why it was tempting
Fast
Why we didn't choose it
Adds a second system that must be up for payments to be correct
Option
Push every checkout through one global queue
Why it was tempting
Removes races completely
Why we didn't choose it
Slows every table to protect the few that collide. A row lock already serialises per bill
Option
Hold the bill lock through the provider call
Why it was tempting
Simplest to write
Why we didn't choose it
Exactly the mistake Part 1 fixed: locks and connections held while a provider takes its time
Option
MySQL constraints, short row locks, a state machine, idempotent provider calls and reconciliation
Why it was tempting
Correctness enforced where the money is recorded
Why we didn't choose it
Our choice

We accepted a slightly slower checkout and more code on the payment path in exchange for never charging twice. And we accepted that prevention would never be perfect. Reconciliation was the net, so we didn't over-engineer the first line of defence for every edge case.

What we measured

  • Duplicate charges per month: from about 300 to under 10
  • Payments stuck in "unknown" longer than 15 minutes
  • Reconciliation exceptions per day, and time to resolve them
  • Time from a duplicate charge to its refund
  • Checkout time added by the key check, lock and state writes: 40 ms at p95

What it cost

Every payment now does more work: a key check, a short lock, several state writes and sometimes a provider lookup. That added 40 ms to checkout at p95. Reconciliation needed its own jobs, reports and a person who owned the exceptions.

Before the fixes, the cost was refunds, support hours and merchants questioning whether they could trust the platform on their busiest night: about $12,000 in refunds and about 600 support tickets in the worst month.

What this part leaves for later

Gap left here
Keys and locks once payments and orders live in different services
Where it's covered
Part 5: splitting the monolith
Gap left here
Following one payment across services
Where it's covered
Part 6: logs and monitoring
Gap left here
Moving side work and partner calls into queues
Where it's covered
Parts 7 and 8
Gap left here
The payment state machine as a default for every new service
Where it's covered
Part 13: the platform

What I took from it

Networks retry, people double-tap and providers resend. "Exactly once" is never something you get for free in payments.

You build it: one key per intent, a state for every step, no locks while you wait, and calls to the provider that are safe to repeat. Then you reconcile, in case you still got it wrong.

Next in the series: Part 5: One service down, everything down