Measuring AI ROI Honestly

Part 9, the final part of my series on AI in production. Earlier parts covered getting answers right, making agents safe, choosing what to build, the trade-offs, and the teams behind it. This one is about the question every leader eventually asks: was it worth it?

Every AI project eventually produces the same slide. A chart with a line going up, a launch date marked on it, and a headline: "Since we launched the AI assistant, conversion is up."

The room is pleased. Nobody asks the only question that matters: compared to what?

When I co-owned a payments P&L across many markets, I learned to distrust any number that went up after a launch until I knew what would have happened without it. Markets move. Seasons change. Other teams ship things in the same quarter. A metric rising after your launch tells you that something changed, not that your launch changed it.

AI projects are especially exposed to this, because they usually launch with high expectations, high spend and a lot of other change happening at the same time. If you can't separate the AI's effect from everything else, you can't defend the budget when someone finally asks. Worse, you can't tell which parts are working and which you should switch off.

Why before-and-after numbers mislead

Comparing the months before launch with the months after is the most common way to report AI value, and the least reliable. These are the usual reasons:

None of these require bad intent. They happen naturally when a team that wants the project to succeed picks the comparison after seeing the results. The fix is to decide how you'll measure before launch, and to compare like with like at the same time.

Holdouts: the simplest honest answer

A holdout is a group that doesn't get the AI, chosen at random, measured over the same period as everyone else. Seasonality, other launches and market conditions hit both groups equally, so the difference between them is the AI's effect.

A few design choices decide whether a holdout tells you anything:

• Split at the right level. If reps share accounts, split by account or by team so the groups don't influence each other. A rep who sees a colleague's AI-written follow-up will copy it, and your holdout stops being one.
• Make it big enough, for long enough. Small groups and short windows produce noise that looks like signal. Agree the size and duration up front, and run it for at least one full sales or billing cycle.
• Keep it small enough to be fair. Most teams hold out somewhere between a tenth and a fifth of the population. Explain to the holdout group why it matters, and give them the tool when the measurement ends.
• Pick the outcome before you start. Win rate, time to close, retention, cost per ticket. One primary measure, chosen before launch, with a few guardrails that must not get worse.

The objection I hear most is that holding out a useful tool costs money. It does, a little, for a short time. Not knowing whether a large AI budget is working costs far more.

When a holdout isn't possible

Sometimes you can't keep a group without the tool: the team is too small, leadership wants everyone on it, or the AI is built into a product every customer uses. You can still do better than before-and-after.

• Roll out in waves. Launch to one team or region first, then the next a few weeks later. Each wave acts as a temporary comparison for the ones still waiting. This is often easier to agree than a permanent holdout, because everyone gets the tool eventually.
• Switch it off for a short window. Turning a feature off for a random slice of sessions or days, briefly, tells you what it was adding.
• Compare matched groups. Find teams, accounts or regions that looked similar before launch and compare how they diverge after. It's weaker than a random split, because the groups may differ in ways you can't see, so treat the result as an estimate.
• Measure inside the workflow. If the assistant answers questions on calls, compare deals where it answered the key question with similar deals where it didn't. This can't prove cause on its own, but it shows where value is likely to come from.

Whichever method you use, write down the comparison before launch. A method chosen after you've seen the numbers is just a more sophisticated before-and-after.

Count incremental value, and the full cost

The value of an AI feature is the difference it makes, not everything that happens while it's running. A deal closed by a rep who used the assistant isn't an AI deal; it's only partly one, and only if it would have closed later, smaller or not at all without it.

It helps to trace value from what the AI does to what the business earns:

Most reports stop at the first two levels. Time saved only becomes value if it's used for something, so the behaviour level is the bridge to real outcomes. If reps save an hour a day and spend it the same way as before, the ROI is zero.

Then count the full cost, not just the model bill:

• Model, infrastructure and tooling.
• The people building and maintaining it, including keeping evals and data current.
• Reviewer and approver time.
• The cost of mistakes: wrong answers that reached customers, rework, escalations.

Incremental value minus full cost is the honest ROI. It's usually smaller than the launch slide suggested, and much easier to defend.

Agree the outcome with the people who own it

The most common reason AI ROI turns into an argument is that each stakeholder was measuring something different. Sales leadership expected more closed deals. Finance expected lower cost per deal. The AI team reported time saved. All three can be true at once, and the project can still be judged a failure.

Before building, sit down with the people who own the outcome and agree four things in writing:

• The one outcome that defines success, owned by a business leader, not the AI team. For a sales assistant, that's usually win rate or time to close, owned by the Head of Sales.
• The guardrails that must not get worse: customer satisfaction, error rate, compliance incidents.
• How it will be measured: the holdout or comparison, the period, and who calculates the result.
• What happens at each result: expand if it clears the bar, adjust if it's close, stop if it doesn't.

This changes the AI team's role. Instead of defending a project, it reports against a measure the business chose. When the result comes in, the conversation is about the decision agreed months earlier, not about whose number is right.

Report it so people can trust it

A credible ROI report fits on one page and answers five questions:

1. What did we compare? The method, the groups and the period, decided before launch.
2. What changed? The difference in the primary outcome, shown as a range rather than a single precise number, because small samples are uncertain.
3. What did it cost? The full cost, including people and mistakes.
4. What didn't work? The parts with no measurable effect, and what you'll stop doing.
5. What's the decision? Expand, adjust or stop, and what you'll measure next.

The fourth question is the one most reports leave out, and the one that builds the most trust. A team that reports where AI didn't help is believed when it reports where it did.

The number you can defend

AI budgets are getting bigger, and the questions about them are getting sharper. Sooner or later a CFO will look past the line going up and ask what would have happened without it.

The teams that answer well aren't the ones with the most impressive chart. They're the ones that decided how to measure before they launched, kept a fair comparison, and counted every cost. Their numbers are usually smaller, and they're the only ones that survive the question.

A modest number you can defend is worth more than a big one you can't.

That's the end of this series. Thank you for reading along. If you're moving AI from pilot to production and want to compare notes, my messages are open.