Back to all articles
Cutting LLM Costs Without Losing TrustPart 03
Model routingLLM costsSales assistantsApplied AI

Cutting LLM Costs, Part 3: Not Every Question Deserves Your Most Expensive Model

Janahan Thavendran16 min read

TL;DR

  • Model routing sends each request to the cheapest model that can answer it well, so a CRM lookup no longer pays frontier prices.
  • A small classifier suits a fast sales assistant; a cascade with a risk gate suits wealth advice, where answers can be checked and mistakes cost more.
  • In the worked examples routing cuts monthly cost by 68–80%, but only while shadow samples, a regret rate and regular retraining keep it honest.
In this article

Part 3 of 12 in the series Cutting LLM Costs Without Losing Trust. Part 1 covered semantic caching and Part 2 covered prompt caching. This part covers model routing.

A rep types a question into the sales assistant: "When does Acme renew?"

The answer sits in one CRM field. The team's one model, Claude Sonnet 5.5, reads the account context, finds the date and writes one line. Then the same model, at the same price, takes the next question: "Acme wants 30% off for a three-year deal. What's our best counter?"

One of those questions needs the most capable model you can buy. The other needs a lookup.

Pull a week of logs and the pattern repeats. Most questions are lookups, rewrites and short summaries. Maybe one in ten needs real reasoning. Every one of them goes to the same model, and for 200 reps that comes to about $4,750 a month.

Parts 1 and 2 cut the cost of repeated work: the same answer, the same playbook. Neither helps with a new question that is simply easy. Model routing goes after that cost.

What model routing does

Model routing sends each request to the cheapest model that can answer it well. A small model takes the lookups, a mid-tier model the summaries and drafts, and the frontier model the questions that need reasoning.

The difference from Parts 1 and 2 matters. Caching skips a call or makes it cheaper, and the same model still writes the answer. Routing changes which model writes it. That is why it can save the most when most traffic is easy, and why it can do the most damage to trust: the rep reads whatever the cheaper model wrote.

Four ways to decide

Approach
One model for everything
How it decides
It doesn't. Every request goes to the same capable model
Where it breaks
Pays big-model prices to read a date out of the CRM
Approach
Static rules
How it decides
By feature: call summaries to small, open chat to mid
Where it breaks
A 10-minute check-in and a 90-minute pricing negotiation are both "summaries"
Approach
Small classifier
How it decides
Predicts a tier for each request before any model is called
Where it breaks
It can't see the answer, so a misjudged request gets a weak answer with nothing to catch it
Approach
Cascade
How it decides
Tries the cheap model, checks the answer, escalates if the check fails
Where it breaks
Escalated requests pay for every attempt and take longer

The last two are the ones worth building. The classifier makes one guess up front. The cascade spends a cheap check to earn the right to stay cheap. A useful rule: use a classifier when you can't check an answer cheaply, and a cascade when you can.

A classifier for the sales assistant

A sales assistant suits a classifier. Requests are short and come fast, and a rep on a live call won't wait for a second attempt. Checking an answer is also hard. "Is this the right discount argument for this account?" has no quick test.

So the assistant gets three tiers, with a classifier in front.

One guess up front, one model call
A rep asks a question
on a call or between calls
Small classifier
picks a tier in milliseconds
Easy · about 55% of requests
Small model · 55%
lookups, CRM reads, rewrites
Moderate · about 35%
Mid model · 35%
call summaries, follow-ups
Needs reasoning · about 10%
Frontier model · 10%
pricing and multi-account reasoning

Low confidence rounds up one tier. Anything that commits a price or a discount goes to the frontier model, whatever the classifier predicts.

model routing · a classifier in front of a sales assistant
Tier
Small
What it handles
Lookups, CRM field reads, reformatting, short rewrites
Example request
"When does Acme renew?" "Turn these notes into bullets."
Tier
Mid
What it handles
Call summaries, follow-up emails, objections answered from the playbook
Example request
"Summarise today's call with Globex and draft the follow-up."
Tier
Frontier
What it handles
Reasoning across accounts, calls or pricing options
Example request
"Acme wants 30% off for a three-year deal. What's our best counter?"

The classifier is small on purpose. A fine-tuned BERT-sized model, or embeddings with a simple logistic regression, runs in milliseconds on a CPU. It reads the request plus a few cheap signals: which feature it came from, how many documents retrieval pulled back, how long the conversation is, whether a price appears. It returns a tier and a confidence score.

When confidence is low, round up a tier. A wasted mid-tier call costs cents. A weak answer on a live deal costs more.

Rounding up only works if the confidence score means something, and most small classifiers are overconfident. Calibrate the scores on a held-out set, then set thresholds by cost. A wrong "small" on a pricing question costs more than a wrong "mid" on a summary, so the bar for sending pricing questions down a tier is higher.

The classifier also sees only the request, not what retrieval finds. "Can Acme keep its old rate?" looks like a lookup until the contract shows a custom clause. That is why a price and discount rule sits in front of the classifier, as the next section shows, and why the assistant can re-route after retrieval when the context carries a risk flag.

Train it on your own logs, not a public benchmark. Take 2,000 to 5,000 real requests, run each through all three tiers, and label the cheapest tier whose answer was good enough. An LLM judge does most of the labelling, and reps spot-check a sample. RouteLLM is built on the same idea. Its routers did well on general chat but were close to random on exam-style questions until a small set of in-domain examples was added. Your sales questions are out of domain for any router you didn't train.

Judge labels need checking. A judge can favour longer answers or its own model family, and if it is the frontier model you are labelling against, it is partly grading itself. Keep a held-out set of a few hundred requests labelled by people. Have two reviewers label a sample separately to see whether they agree, and trust the judge only where it matches them.

In this example the classifier sends about 55% of requests to small, 35% to mid and 10% to frontier.

When a wrong answer costs more: a cascade

Now take the same idea to an assistant for 40 wealth advisors, about 50,000 requests a month. It preps client meetings, writes CRM notes, explains statements and drafts rebalancing rationale.

Two things change. Many outputs can be checked cheaply: a portfolio summary either matches the holdings data or it doesn't. And a wrong answer is no longer a lost minute. It can be a complaint or a suitability breach. So the design is a cascade, with a rule in front of it.

Cheap model first, checked before anyone sees it
Advisor request
meeting prep, notes, statements
Risk gate
product, suitability, tax or unreviewed
Risky → frontier model, advisor signs off
Frontier + advisor
signs off before it goes out
Everything else → cascade, escalating on each failed check
Small model drafts
everything the gate lets through
Verifier check
numbers, citations, disclosures
Mid model retries
checked again
Frontier model
after a second failure
Answer
back to the advisor

Each failed check moves the request up one tier. Escalated requests pay for every attempt, so watch the escalation rate.

model routing · a cascade with a risk gate for wealth advisors
  1. Risk gate. Anything that recommends a product, touches suitability or tax, or goes to a client unreviewed skips the cascade. It goes straight to the frontier model, and an advisor signs it off. Here that is 20% of requests.
  2. Small model drafts. Everything else starts on the small model.
  3. Verifier checks. Do the numbers match the holdings? Is every figure cited? Are the required disclosures there? A small judge scores the rest, for a fraction of a cent.
  4. Escalate on failure. A failed check goes to the mid tier, and a second failure goes to the frontier.

If 65% of small-model drafts pass and 85% of mid-tier retries pass, the final split is about 52% small, 24% mid and 24% frontier.

The verifier is a model too, and it can be wrong. Measure its false-pass rate: on a labelled set, how often does it pass an answer a reviewer would fail? Each false pass is a weak answer that skipped escalation, so a lenient verifier makes the cascade look cheaper than it is. Re-test it whenever the small model or the prompts change.

The gate does the real work, because difficulty and risk are different things. "Should I move my pension into this fund?" is easy to phrase, and it is exactly the question a cheap model should never answer alone. The same rule fits a sales assistant: anything that commits a price or a discount to a customer goes to the frontier model, whatever the classifier thinks.

What it saves

Prices are November 2026 list prices for one model family, per million input / output tokens: Claude Haiku 5.5 at $0.10 / $0.50 and Sonnet 5.5 at $2 / $10 for the small and mid tiers, and Opus 5.5 at $4 / $20 for the frontier. The gap between small and mid is twenty times; between mid and frontier, two. So moving work down saves far more than keeping work off the top. Prices change often, so check current rates before you plan.

Sales assistant. The team this series follows: 200 reps × 60 requests a day × 22 working days = 264,000 requests a month, at about 7,000 input and 400 output tokens each.

Approach
One model for everything
Model mix
100% Sonnet 5.5
Cost per request
$0.0180
Monthly cost
$4,752
Saving
—
Approach
Static rules by feature
Model mix
40% Haiku, 60% Sonnet
Cost per request
$0.0112
Monthly cost
$2,946
Saving
38%
Approach
Small classifier
Model mix
55% Haiku, 45% Sonnet
Cost per request
$0.0086
Monthly cost
$2,669
Saving
44%
Approach
Small classifier, hardest tenth to Opus
Model mix
55% Haiku, 35% Sonnet, 10% Opus 5.5
Cost per request
$0.0104
Monthly cost
$3,144
Saving
34%

Both classifier lines include $400 a month to host the classifier. Static rules save about two-fifths, but only by moving whole features. The classifier saves more because it moves single requests, and it can do something rules can't: send the hardest tenth up to Opus. That upgrade costs about $475 a month here, and it's the main reason to build a classifier rather than rules.

Wealth advisor assistant. 50,000 requests a month, at about 5,000 input and 600 output tokens each.

Approach
Always frontier
Final model mix
100% Opus 5.5
Cost per request
$0.032
Monthly cost
$1,600
Saving
—
Approach
Cascade, no risk gate
Final model mix
65% small, 30% mid, 5% frontier
Cost per request
$0.0088
Monthly cost
$442
Saving
72%
Approach
Cascade with risk gate
Final model mix
52% small, 24% mid, 24% frontier
Cost per request
$0.0135
Monthly cost
$674
Saving
58%

The cascade lines cost more than their final mix alone, because a cascade pays for attempts that never reach the advisor. Here is one request without the risk gate, assuming a verifier check is a Haiku call costing about $0.0006:

Cost item
Final answers at the final mix
Per request
$0.0069
Cost item
Drafts that failed and escalated
Per request
$0.0011
Cost item
Verifier checks on small and mid answers
Per request
$0.0008
Cost item
Total
Per request
$0.0088

With the gate, 20% of requests go straight to the frontier at $0.032, and the other 80% follow this path. That gives $0.0135. The risk gate gives up about $230 a month in savings. That is probably the cheapest compliance control the firm will buy.

The tables still leave costs out. Shadow sampling 2% of sales traffic on Opus adds about 5,300 calls, roughly $190 a month. Labelling the training set, judge calls and engineering time come on top. Caching from Parts 1 and 2 pulls the other way and makes every tier cheaper. Count all of it before you promise finance a number.

These numbers sit inside what the research reports. RouteLLM cut costs by over 85% on MT Bench, 45% on MMLU and 35% on GSM8K against using only GPT-4, while keeping 95% of its quality. FrugalGPT, a learned cascade, matched GPT-4 at up to 98% lower cost on its benchmarks. The spread is the useful part: savings depend on how much of your traffic is easy.

Where it bites in a sales tool

Cheap answers look fine. A weak answer from a small model is fluent and confident. Nobody files a ticket. Quality slips, and the only sign is reps quietly using the assistant less.

Traffic drifts. Launch a new product line and reps ask questions the classifier has never seen. It routes them on surface features, often to the small tier.

Models change underneath you. A new small model may handle half of what needed the mid tier. A price cut may make the frontier cheap enough that routing no longer pays. Thresholds tuned last quarter are wrong this quarter.

Escalations are slow on live calls. A verifier has to read the whole draft before it can pass it, so the cheap answer can't stream to the rep while it is checked. A rough budget, to replace with your own measurements:

Step
Classifier
Rough time
under 20 ms
Step
Small model answer
Rough time
about 1 s
Step
Verifier check
Rough time
about 0.5 s
Step
Mid model answer
Rough time
about 2 s
Step
Frontier model answer
Rough time
about 4 s
Step
Worst cascade path: small, check, mid, check, frontier
Rough time
about 8 s

Eight seconds is fine for meeting prep at 7am and painful mid-call. Keep live-call questions on the classifier path, and watch the slowest 1% and 5% of responses, not the average.

The cheapest model is no model

Before routing a request to the smallest model, ask whether it needs a model at all. Some jobs in a sales assistant are rules wearing an AI costume: tagging a call by the product names it mentions, filling a CRM field from a dropdown the rep already picked, sending a follow-up email that is the same template every time, looking up a contract's renewal date. Code does those for nothing, the same way every time, and never makes up an answer.

Give the router a tier zero: requests that go to a rule, a lookup or a template. Keep the models for the parts that need judgement.

What else the router has to handle

Data policy. Tiers often mean different providers, regions and retention terms. A cheaper model can sit with a provider that keeps prompts longer or runs in another region. Route on data class as well as difficulty: client financial data and personal details only go to models that meet the same policy, whatever their tier.

Tool calls. A small model that writes a fine paragraph can still break a structured output or a CRM write. Validate every tool call against its schema and retry on a stronger model when it fails, or start every write action at the mid tier. Text quality is not the only test.

Tier flips mid-conversation. Routing each turn on its own can move a conversation between models, and the rep sees the tone and depth change from one answer to the next. Route per conversation, or let a conversation move up a tier but not back down.

Outages and rate limits. Every tier needs a fallback: the same tier from another provider, or the next tier up with a cost alert. Decide in advance whether a frontier outage means answering on the mid tier with a warning, or queueing the request.

How to see it working

Start with the deal, before launch. For example: we accept up to 2% regret on small and mid answers, and none on anything behind a risk gate, to save around 80%. If regret goes above the ceiling, raise the thresholds and give back some of the saving. Without a stated ceiling, every regret number looks fine.

Every request already logs which model answered it. Add a small shadow sample: re-run 1–2% of small and mid answers on the frontier model and let a judge compare them. The share where the cheap answer was clearly worse is your regret rate.

Then watch these numbers each week:

What to watch
Share of requests per tier
What it tells you
Whether the router is moving traffic
Warning sign
Frontier share creeping back up
What to watch
Regret rate
What it tells you
Whether cheap answers are good enough
Warning sign
Above the ceiling you agreed
What to watch
Verifier false-pass rate
What it tells you
Whether the cascade's check can be trusted
Warning sign
Rising after a model or prompt change
What to watch
Escalation rate
What it tells you
Whether the small model and verifier still fit
Warning sign
Climbing week on week
What to watch
Tool-call failures by tier
What it tells you
Whether small models can handle writes
Warning sign
Any rise on the small tier
What to watch
Cost per conversation
What it tells you
What finance actually sees
Warning sign
No fall after launch
What to watch
Slowest 1% and 5% of live-call responses
What it tells you
Whether reps feel the routing
Warning sign
Rising after a cascade change

Watch each number by feature, rep team, account segment and product line, not only in total. A healthy overall regret rate can hide a new product where most questions go to the small model and fail.

Keep a fixed set of a few hundred real requests per tier, and re-score it whenever a model, prompt or threshold changes. Retrain the classifier monthly, or after any launch that changes what reps ask. The shadow samples give you the new labels.

It pairs with caching

The three levers so far work at different points. The semantic cache decides whether a model runs at all. The router decides which model runs. The prompt cache makes that run cheaper.

One catch: a prompt cache belongs to one model. Each tier keeps its own cached playbook, so a tier with little traffic may find its cache has expired. Check the cached share per tier, not just overall.

The semantic cache needs the same care. Part 1's cache can serve a small model's answer to a question that should have gone to the frontier. Store the tier with each cached answer, and serve it only to a request routed to that tier or lower.

The lever that touches trust the most

Prompt caching trades almost no trust, because the same model still writes every answer. Routing trades the most, because a cheaper model writes some of them.

In sales, a weak answer costs a rep thirty seconds and a retry. In wealth advice, the same weak answer on the wrong question costs a complaint, or a regulator's letter. The router is where you write that difference down.

So don't start by picking a router. Start with a week of logs and one question for each type of request: what happens if the cheap model gets this wrong?

Model routing looks like an infrastructure decision. It is really a product decision about which mistakes your users can afford.

Next in the series, Part 4: trimming retrieved context, because two good chunks beat ten average ones.