APIs as a Product, Part 8 of 10 · No number without a partner's name next to it.
Picture a dashboard that says the API is healthy. The overall error rate is well inside what was agreed. Latency is normal. Nobody has been paged.
Then a merchant calls to ask why orders from one partner have dried up. Split the same numbers by partner and the picture changes: that partner is failing on a large share of its calls. Small enough that the overall average barely moves, big enough that, for the merchants who rely on that channel, the API is broken.
Averages are comfortable. They are also where your worst partner hides.
Every number gets a partner's name
That is why our weekly review started with a view per partner, not platform-wide numbers. We used Datadog for the monitoring, but the tool mattered less than the rule: no number without a partner's name next to it.
| Number | What it caught |
|---|---|
| Error rate, split into our errors and theirs | Whether a problem was ours to fix or a partner's integration to help with |
| Latency, both directions | Slow responses from us, and partners slow to confirm orders, which led to timeouts and retries |
| Sync lag | Time from a change in the core to confirmation that each consumer had received it |
| Version lag | Traffic each consumer still sent to old versions |
| Webhook delivery | Partners whose endpoints were down or rejecting our events |
| Adoption | Partners stuck in sandbox, live partners whose traffic was falling, features nobody used |
- Number
- Error rate, split into our errors and theirs
- What it caught
- Whether a problem was ours to fix or a partner's integration to help with
- Number
- Latency, both directions
- What it caught
- Slow responses from us, and partners slow to confirm orders, which led to timeouts and retries
- Number
- Sync lag
- What it caught
- Time from a change in the core to confirmation that each consumer had received it
- Number
- Version lag
- What it caught
- Traffic each consumer still sent to old versions
- Number
- Webhook delivery
- What it caught
- Partners whose endpoints were down or rejecting our events
- Number
- Adoption
- What it caught
- Partners stuck in sandbox, live partners whose traffic was falling, features nobody used
The split between our errors and theirs mattered most. A spike in server errors was our problem. A spike in bad requests from one partner usually meant they had shipped a new app version. Both needed action, but from different people.
Sync lag was the number I watched most closely, because nobody complained about it until it was too late. A menu change that took hours to reach a partner meant customers could order something the kitchen had stopped making. That never showed up as an error. It showed up as a refund.
Adoption is a health number too
The quietest warning sign was a partner whose traffic slowly declined. No errors, no complaints, just fewer calls each week. Sometimes that was seasonal. Sometimes it meant the partner had moved their users to another provider and hadn't told us yet. We only caught it because we were looking at each partner's trend, not the total.
The same went for partners who got a sandbox key and never went live. Each one was a sales conversation and an integration effort that had stalled somewhere. Looking at where they stopped told us more about our docs and onboarding than any survey.
None of these numbers were hard to collect. The hard part was looking at them every week, and treating a partner whose numbers were drifting as a customer to call, not a line on a chart.
A healthy platform with an unhealthy partner is not a healthy platform. It is a platform that hasn't noticed yet.
Related reading: The Shared Platform, Part 5: Measuring a Platform Nobody Sees and Platform at Scale, Part 6: You Can't Fix What You Can't See.