APIs as a Product, Part 6 of 10 · Syncing catalog, orders and payments with 50+ partners.
Our busiest days were never decided by our merchants. They were decided by their customers. Weekends, Christmas week and Chinese New Year brought the traffic; Mondays were quiet.
Some of our partners' systems could not accept two orders at the same time. On a quiet day that never mattered, because orders arrived a few seconds apart. At peak, they arrived together, and the second one was rejected. The diner had already paid. The kitchen never saw the order.
That is what middleware looks like when it fails. Nothing crashes. A small mismatch between two systems quietly turns into a refund, an angry merchant and a support ticket that takes a day to trace.
What the middleware actually did
On the platform I worked on, the middleware kept four things in sync with more than 50 global and local partners: catalog, orders, loyalty and payments. Menus flowed out to partners. Orders and payments flowed back in. Loyalty points had to agree on both sides. Each partner had its own model of a menu, its own idea of what "accepted" meant, and its own limits.
The fix for partners that couldn't take concurrent orders wasn't to ask them to rebuild their systems. We put orders to partners onto queues, using RabbitMQ, so we could send them at a pace each partner could handle. A partner's limit became a setting on our side, not an outage on theirs.
What fails at the edges
Most failures lived in the space between "we sent it" and "they confirmed it".
Timeouts were the worst. If a partner took too long to respond, we didn't know whether the order had been accepted or not. Retrying could create a duplicate. Not retrying could lose it. The answer was to make every order and payment safe to send twice, so a retry could never create a second order or a second charge. Double payments taught the same lesson from the other side: checking state before acting, and refusing to act twice on the same order, mattered more than any amount of speed.
Catalog changes were quieter but just as costly. We cached menus until they changed, which kept load down at peak. But it meant a missed update could leave a partner showing a dish the kitchen had stopped making. A sync that is fast but occasionally wrong is worse than one that is a little slower and always right.
Every domain had its own version of this sync lag. A price change that reached a partner hours late meant the partner showed the old price. An order status that lagged meant a customer saw "preparing" when the food was ready. A loyalty balance that lagged meant the wrong points. None of these were errors. They showed up as a refund, a wrong charge or a complaint, never on an error dashboard. And the fix was always the sync itself, not a new API version.
Nobody notices middleware when it works, and that is the job. Partners don't judge you on the thousands of orders that arrived. They judge you on the one that didn't.
Related reading: the full engineering story of the partner queues, idempotent payments and menu caching is in Platform at Scale Part 8, Part 4 and Part 2.