Quality, Cost and Speed

Part 6 of my series on AI in production. Part 5 was about choosing which problems to solve. This part is about the trade-offs that decide whether a good solution survives contact with real volume.

Most AI pilots are built with the most capable model available. That's a sensible way to find out whether a problem can be solved at all. Then the pilot succeeds, and two new problems appear.

The first is cost. A few cents per request is nothing during a pilot with twenty users. Multiply it by every customer conversation, every day, and it becomes a line item that finance wants to discuss.

The second is speed. An answer that takes eight seconds feels fine in a demo. During a live customer call, eight seconds is long enough for the conversation to move on, and the answer arrives too late to be useful.

Quality, cost and latency pull against each other. The most capable models are usually the slowest and most expensive; the fastest and cheapest make more mistakes. There's no setting that maximises all three. Deciding which one to give up, for each part of the product, is a product decision, not just an engineering one.

Start from the job's budget

Before comparing models, set two budgets for each task: how long the user can wait, and how much each completed task can cost relative to the value it creates. Different parts of the same product have very different budgets.

The cost budget works the same way. A pre-call brief for a large enterprise renewal can justify an expensive model. Tagging thousands of support tickets a day cannot. Write down the value per task, even roughly, and the acceptable cost follows from it.

Once the budgets exist, the trade-off stops being an argument about which model is best. It becomes a question with an answer: which option meets this task's quality bar inside its time and cost budget?

Quality isn't only the model

When answers are wrong, the first instinct is to reach for a bigger model. Often that's the most expensive fix for a problem that lives somewhere else.

In retrieval-based systems, many wrong answers come from the wrong documents being found, or the right ones being outdated. A larger model reading the wrong document still gives a wrong answer, just more fluently. Better chunking, filtering and source hygiene usually improve quality more, at a fraction of the cost.

The same goes for context. Stuffing everything into a long prompt makes every request slower and more expensive, and can make answers worse as the important part gets buried. Sending only what the task needs is cheaper, faster and often more accurate.

So the goal isn't the best model. It's the smallest, fastest model that clears the quality bar from your evals, with the rest of the system doing its share. You can only know where that is if you have an eval set to test against. This is where part 2 pays for itself: without evals, every model choice is a guess.

Ways to get more of all three

You can't escape the trade-off, but you can move the curve. These techniques let a product be fast, cheap and good enough at the same time.

Route by difficulty. Most requests are routine. Send them to a small, fast model, and escalate to a larger one only when the request is complex or the small model is unsure.

Cache what repeats. Sales teams ask the same product and security questions again and again. Answers to common, stable questions can be reused instantly, as long as the cache is cleared when the source documents change.

Do the work before it's needed. A pre-call brief doesn't have to be generated when the rep opens it. Build it in the background an hour before the meeting, with a slower, more thorough model, and it appears instantly.

Batch what isn't urgent. Summaries, tagging and weekly reports can run in bulk at lower cost, often off-peak, because nobody is waiting.

Stream and show progress. When an answer takes a few seconds, showing it as it's written makes the wait feel much shorter, and lets the user stop reading once they have what they need.

Specialise later. Once you have a large set of approved, high-quality examples, a smaller model tuned on them can match a larger general one on that narrow task, faster and cheaper. It's a late-stage optimisation, not a starting point.

Decide with data, and decide again

With budgets and an eval set in place, choosing a model becomes a short experiment. Run each candidate setup against the same eval set and record three numbers side by side:

• Pass rate on the criteria that matter for this task, with zero failures on the non-negotiables.
• Response time for the slowest requests, not the average, because the slow ones are the ones users remember.
• Cost per completed task at expected volume, including retries and escalations.

Pick the cheapest, fastest option that meets the quality bar and fits the budgets. It will often be a different answer for different tasks in the same product, and that's fine.

Then plan to decide again. Models get better, cheaper and faster every few months, and providers change their prices. A choice that was right at launch may be expensive or slow a year later. Teams with a good eval set can test a new model in a day and switch with confidence. Teams without one stay on whatever they launched with, because switching feels too risky.

This is also an organisational choice. Keep model access behind one internal interface, so swapping a model is a configuration change rather than a rewrite across every feature.

The best model is the one that fits the job

It's easy to treat model choice as a technical contest, where the newest and most capable model wins. In production, the question is different: what does this task need, how fast, and at what cost?

A live-call suggestion that's slightly less eloquent but arrives in time beats a perfect one that arrives late. A brief built overnight by a slow, thorough model beats a fast, shallow one. Getting these trade-offs right, task by task, is a large part of what separates an AI feature that scales from one that stays a pilot.

Next in the series: a case study. How we built an assistant that helps account managers and sales reps before, during and after customer calls, and what it did to deals.