Part 3 of 12 in the series Cutting LLM Costs Without Losing Trust. Part 1 covered semantic caching and Part 2 covered prompt caching. This part covers model routing.
A rep types a question into the sales assistant: "When does Acme renew?"
The answer sits in one CRM field. The team's one model, Claude Sonnet 5.5, reads the account context, finds the date and writes one line. Then the same model, at the same price, takes the next question: "Acme wants 30% off for a three-year deal. What's our best counter?"
One of those questions needs the most capable model you can buy. The other needs a lookup.
Pull a week of logs and the pattern repeats. Most questions are lookups, rewrites and short summaries. Maybe one in ten needs real reasoning. Every one of them goes to the same model, and for 200 reps that comes to about $4,750 a month.
Parts 1 and 2 cut the cost of repeated work: the same answer, the same playbook. Neither helps with a new question that is simply easy. Model routing goes after that cost.
What model routing does
Model routing sends each request to the cheapest model that can answer it well. A small model takes the lookups, a mid-tier model the summaries and drafts, and the frontier model the questions that need reasoning.
The difference from Parts 1 and 2 matters. Caching skips a call or makes it cheaper, and the same model still writes the answer. Routing changes which model writes it. That is why it can save the most when most traffic is easy, and why it can do the most damage to trust: the rep reads whatever the cheaper model wrote.
Four ways to decide
| Approach | How it decides | Where it breaks |
|---|---|---|
| One model for everything | It doesn't. Every request goes to the same capable model | Pays big-model prices to read a date out of the CRM |
| Static rules | By feature: call summaries to small, open chat to mid | A 10-minute check-in and a 90-minute pricing negotiation are both "summaries" |
| Small classifier | Predicts a tier for each request before any model is called | It can't see the answer, so a misjudged request gets a weak answer with nothing to catch it |
| Cascade | Tries the cheap model, checks the answer, escalates if the check fails | Escalated requests pay for every attempt and take longer |
- Approach
- One model for everything
- How it decides
- It doesn't. Every request goes to the same capable model
- Where it breaks
- Pays big-model prices to read a date out of the CRM
- Approach
- Static rules
- How it decides
- By feature: call summaries to small, open chat to mid
- Where it breaks
- A 10-minute check-in and a 90-minute pricing negotiation are both "summaries"
- Approach
- Small classifier
- How it decides
- Predicts a tier for each request before any model is called
- Where it breaks
- It can't see the answer, so a misjudged request gets a weak answer with nothing to catch it
- Approach
- Cascade
- How it decides
- Tries the cheap model, checks the answer, escalates if the check fails
- Where it breaks
- Escalated requests pay for every attempt and take longer
The last two are the ones worth building. The classifier makes one guess up front. The cascade spends a cheap check to earn the right to stay cheap. A useful rule: use a classifier when you can't check an answer cheaply, and a cascade when you can.
A classifier for the sales assistant
A sales assistant suits a classifier. Requests are short and come fast, and a rep on a live call won't wait for a second attempt. Checking an answer is also hard. "Is this the right discount argument for this account?" has no quick test.
So the assistant gets three tiers, with a classifier in front.
Low confidence rounds up one tier. Anything that commits a price or a discount goes to the frontier model, whatever the classifier predicts.
| Tier | What it handles | Example request |
|---|---|---|
| Small | Lookups, CRM field reads, reformatting, short rewrites | "When does Acme renew?" "Turn these notes into bullets." |
| Mid | Call summaries, follow-up emails, objections answered from the playbook | "Summarise today's call with Globex and draft the follow-up." |
| Frontier | Reasoning across accounts, calls or pricing options | "Acme wants 30% off for a three-year deal. What's our best counter?" |
- Tier
- Small
- What it handles
- Lookups, CRM field reads, reformatting, short rewrites
- Example request
- "When does Acme renew?" "Turn these notes into bullets."
- Tier
- Mid
- What it handles
- Call summaries, follow-up emails, objections answered from the playbook
- Example request
- "Summarise today's call with Globex and draft the follow-up."
- Tier
- Frontier
- What it handles
- Reasoning across accounts, calls or pricing options
- Example request
- "Acme wants 30% off for a three-year deal. What's our best counter?"
The classifier is small on purpose. A fine-tuned BERT-sized model, or embeddings with a simple logistic regression, runs in milliseconds on a CPU. It reads the request plus a few cheap signals: which feature it came from, how many documents retrieval pulled back, how long the conversation is, whether a price appears. It returns a tier and a confidence score.
When confidence is low, round up a tier. A wasted mid-tier call costs cents. A weak answer on a live deal costs more.
Rounding up only works if the confidence score means something, and most small classifiers are overconfident. Calibrate the scores on a held-out set, then set thresholds by cost. A wrong "small" on a pricing question costs more than a wrong "mid" on a summary, so the bar for sending pricing questions down a tier is higher.
The classifier also sees only the request, not what retrieval finds. "Can Acme keep its old rate?" looks like a lookup until the contract shows a custom clause. That is why a price and discount rule sits in front of the classifier, as the next section shows, and why the assistant can re-route after retrieval when the context carries a risk flag.
Train it on your own logs, not a public benchmark. Take 2,000 to 5,000 real requests, run each through all three tiers, and label the cheapest tier whose answer was good enough. An LLM judge does most of the labelling, and reps spot-check a sample. RouteLLM is built on the same idea. Its routers did well on general chat but were close to random on exam-style questions until a small set of in-domain examples was added. Your sales questions are out of domain for any router you didn't train.
Judge labels need checking. A judge can favour longer answers or its own model family, and if it is the frontier model you are labelling against, it is partly grading itself. Keep a held-out set of a few hundred requests labelled by people. Have two reviewers label a sample separately to see whether they agree, and trust the judge only where it matches them.
In this example the classifier sends about 55% of requests to small, 35% to mid and 10% to frontier.
When a wrong answer costs more: a cascade
Now take the same idea to an assistant for 40 wealth advisors, about 50,000 requests a month. It preps client meetings, writes CRM notes, explains statements and drafts rebalancing rationale.
Two things change. Many outputs can be checked cheaply: a portfolio summary either matches the holdings data or it doesn't. And a wrong answer is no longer a lost minute. It can be a complaint or a suitability breach. So the design is a cascade, with a rule in front of it.
Each failed check moves the request up one tier. Escalated requests pay for every attempt, so watch the escalation rate.
- Risk gate. Anything that recommends a product, touches suitability or tax, or goes to a client unreviewed skips the cascade. It goes straight to the frontier model, and an advisor signs it off. Here that is 20% of requests.
- Small model drafts. Everything else starts on the small model.
- Verifier checks. Do the numbers match the holdings? Is every figure cited? Are the required disclosures there? A small judge scores the rest, for a fraction of a cent.
- Escalate on failure. A failed check goes to the mid tier, and a second failure goes to the frontier.
If 65% of small-model drafts pass and 85% of mid-tier retries pass, the final split is about 52% small, 24% mid and 24% frontier.
The verifier is a model too, and it can be wrong. Measure its false-pass rate: on a labelled set, how often does it pass an answer a reviewer would fail? Each false pass is a weak answer that skipped escalation, so a lenient verifier makes the cascade look cheaper than it is. Re-test it whenever the small model or the prompts change.
The gate does the real work, because difficulty and risk are different things. "Should I move my pension into this fund?" is easy to phrase, and it is exactly the question a cheap model should never answer alone. The same rule fits a sales assistant: anything that commits a price or a discount to a customer goes to the frontier model, whatever the classifier thinks.
What it saves
Prices are November 2026 list prices for one model family, per million input / output tokens: Claude Haiku 5.5 at $0.10 / $0.50 and Sonnet 5.5 at $2 / $10 for the small and mid tiers, and Opus 5.5 at $4 / $20 for the frontier. The gap between small and mid is twenty times; between mid and frontier, two. So moving work down saves far more than keeping work off the top. Prices change often, so check current rates before you plan.
Sales assistant. The team this series follows: 200 reps × 60 requests a day × 22 working days = 264,000 requests a month, at about 7,000 input and 400 output tokens each.
| Approach | Model mix | Cost per request | Monthly cost | Saving |
|---|---|---|---|---|
| One model for everything | 100% Sonnet 5.5 | $0.0180 | $4,752 | — |
| Static rules by feature | 40% Haiku, 60% Sonnet | $0.0112 | $2,946 | 38% |
| Small classifier | 55% Haiku, 45% Sonnet | $0.0086 | $2,669 | 44% |
| Small classifier, hardest tenth to Opus | 55% Haiku, 35% Sonnet, 10% Opus 5.5 | $0.0104 | $3,144 | 34% |
- Approach
- One model for everything
- Model mix
- 100% Sonnet 5.5
- Cost per request
- $0.0180
- Monthly cost
- $4,752
- Saving
- —
- Approach
- Static rules by feature
- Model mix
- 40% Haiku, 60% Sonnet
- Cost per request
- $0.0112
- Monthly cost
- $2,946
- Saving
- 38%
- Approach
- Small classifier
- Model mix
- 55% Haiku, 45% Sonnet
- Cost per request
- $0.0086
- Monthly cost
- $2,669
- Saving
- 44%
- Approach
- Small classifier, hardest tenth to Opus
- Model mix
- 55% Haiku, 35% Sonnet, 10% Opus 5.5
- Cost per request
- $0.0104
- Monthly cost
- $3,144
- Saving
- 34%
Both classifier lines include $400 a month to host the classifier. Static rules save about two-fifths, but only by moving whole features. The classifier saves more because it moves single requests, and it can do something rules can't: send the hardest tenth up to Opus. That upgrade costs about $475 a month here, and it's the main reason to build a classifier rather than rules.
Wealth advisor assistant. 50,000 requests a month, at about 5,000 input and 600 output tokens each.
| Approach | Final model mix | Cost per request | Monthly cost | Saving |
|---|---|---|---|---|
| Always frontier | 100% Opus 5.5 | $0.032 | $1,600 | — |
| Cascade, no risk gate | 65% small, 30% mid, 5% frontier | $0.0088 | $442 | 72% |
| Cascade with risk gate | 52% small, 24% mid, 24% frontier | $0.0135 | $674 | 58% |
- Approach
- Always frontier
- Final model mix
- 100% Opus 5.5
- Cost per request
- $0.032
- Monthly cost
- $1,600
- Saving
- —
- Approach
- Cascade, no risk gate
- Final model mix
- 65% small, 30% mid, 5% frontier
- Cost per request
- $0.0088
- Monthly cost
- $442
- Saving
- 72%
- Approach
- Cascade with risk gate
- Final model mix
- 52% small, 24% mid, 24% frontier
- Cost per request
- $0.0135
- Monthly cost
- $674
- Saving
- 58%
The cascade lines cost more than their final mix alone, because a cascade pays for attempts that never reach the advisor. Here is one request without the risk gate, assuming a verifier check is a Haiku call costing about $0.0006:
| Cost item | Per request |
|---|---|
| Final answers at the final mix | $0.0069 |
| Drafts that failed and escalated | $0.0011 |
| Verifier checks on small and mid answers | $0.0008 |
| Total | $0.0088 |
- Cost item
- Final answers at the final mix
- Per request
- $0.0069
- Cost item
- Drafts that failed and escalated
- Per request
- $0.0011
- Cost item
- Verifier checks on small and mid answers
- Per request
- $0.0008
- Cost item
- Total
- Per request
- $0.0088
With the gate, 20% of requests go straight to the frontier at $0.032, and the other 80% follow this path. That gives $0.0135. The risk gate gives up about $230 a month in savings. That is probably the cheapest compliance control the firm will buy.
The tables still leave costs out. Shadow sampling 2% of sales traffic on Opus adds about 5,300 calls, roughly $190 a month. Labelling the training set, judge calls and engineering time come on top. Caching from Parts 1 and 2 pulls the other way and makes every tier cheaper. Count all of it before you promise finance a number.
These numbers sit inside what the research reports. RouteLLM cut costs by over 85% on MT Bench, 45% on MMLU and 35% on GSM8K against using only GPT-4, while keeping 95% of its quality. FrugalGPT, a learned cascade, matched GPT-4 at up to 98% lower cost on its benchmarks. The spread is the useful part: savings depend on how much of your traffic is easy.
Where it bites in a sales tool
Cheap answers look fine. A weak answer from a small model is fluent and confident. Nobody files a ticket. Quality slips, and the only sign is reps quietly using the assistant less.
Traffic drifts. Launch a new product line and reps ask questions the classifier has never seen. It routes them on surface features, often to the small tier.
Models change underneath you. A new small model may handle half of what needed the mid tier. A price cut may make the frontier cheap enough that routing no longer pays. Thresholds tuned last quarter are wrong this quarter.
Escalations are slow on live calls. A verifier has to read the whole draft before it can pass it, so the cheap answer can't stream to the rep while it is checked. A rough budget, to replace with your own measurements:
| Step | Rough time |
|---|---|
| Classifier | under 20 ms |
| Small model answer | about 1 s |
| Verifier check | about 0.5 s |
| Mid model answer | about 2 s |
| Frontier model answer | about 4 s |
| Worst cascade path: small, check, mid, check, frontier | about 8 s |
- Step
- Classifier
- Rough time
- under 20 ms
- Step
- Small model answer
- Rough time
- about 1 s
- Step
- Verifier check
- Rough time
- about 0.5 s
- Step
- Mid model answer
- Rough time
- about 2 s
- Step
- Frontier model answer
- Rough time
- about 4 s
- Step
- Worst cascade path: small, check, mid, check, frontier
- Rough time
- about 8 s
Eight seconds is fine for meeting prep at 7am and painful mid-call. Keep live-call questions on the classifier path, and watch the slowest 1% and 5% of responses, not the average.
The cheapest model is no model
Before routing a request to the smallest model, ask whether it needs a model at all. Some jobs in a sales assistant are rules wearing an AI costume: tagging a call by the product names it mentions, filling a CRM field from a dropdown the rep already picked, sending a follow-up email that is the same template every time, looking up a contract's renewal date. Code does those for nothing, the same way every time, and never makes up an answer.
Give the router a tier zero: requests that go to a rule, a lookup or a template. Keep the models for the parts that need judgement.
What else the router has to handle
Data policy. Tiers often mean different providers, regions and retention terms. A cheaper model can sit with a provider that keeps prompts longer or runs in another region. Route on data class as well as difficulty: client financial data and personal details only go to models that meet the same policy, whatever their tier.
Tool calls. A small model that writes a fine paragraph can still break a structured output or a CRM write. Validate every tool call against its schema and retry on a stronger model when it fails, or start every write action at the mid tier. Text quality is not the only test.
Tier flips mid-conversation. Routing each turn on its own can move a conversation between models, and the rep sees the tone and depth change from one answer to the next. Route per conversation, or let a conversation move up a tier but not back down.
Outages and rate limits. Every tier needs a fallback: the same tier from another provider, or the next tier up with a cost alert. Decide in advance whether a frontier outage means answering on the mid tier with a warning, or queueing the request.
How to see it working
Start with the deal, before launch. For example: we accept up to 2% regret on small and mid answers, and none on anything behind a risk gate, to save around 80%. If regret goes above the ceiling, raise the thresholds and give back some of the saving. Without a stated ceiling, every regret number looks fine.
Every request already logs which model answered it. Add a small shadow sample: re-run 1–2% of small and mid answers on the frontier model and let a judge compare them. The share where the cheap answer was clearly worse is your regret rate.
Then watch these numbers each week:
| What to watch | What it tells you | Warning sign |
|---|---|---|
| Share of requests per tier | Whether the router is moving traffic | Frontier share creeping back up |
| Regret rate | Whether cheap answers are good enough | Above the ceiling you agreed |
| Verifier false-pass rate | Whether the cascade's check can be trusted | Rising after a model or prompt change |
| Escalation rate | Whether the small model and verifier still fit | Climbing week on week |
| Tool-call failures by tier | Whether small models can handle writes | Any rise on the small tier |
| Cost per conversation | What finance actually sees | No fall after launch |
| Slowest 1% and 5% of live-call responses | Whether reps feel the routing | Rising after a cascade change |
- What to watch
- Share of requests per tier
- What it tells you
- Whether the router is moving traffic
- Warning sign
- Frontier share creeping back up
- What to watch
- Regret rate
- What it tells you
- Whether cheap answers are good enough
- Warning sign
- Above the ceiling you agreed
- What to watch
- Verifier false-pass rate
- What it tells you
- Whether the cascade's check can be trusted
- Warning sign
- Rising after a model or prompt change
- What to watch
- Escalation rate
- What it tells you
- Whether the small model and verifier still fit
- Warning sign
- Climbing week on week
- What to watch
- Tool-call failures by tier
- What it tells you
- Whether small models can handle writes
- Warning sign
- Any rise on the small tier
- What to watch
- Cost per conversation
- What it tells you
- What finance actually sees
- Warning sign
- No fall after launch
- What to watch
- Slowest 1% and 5% of live-call responses
- What it tells you
- Whether reps feel the routing
- Warning sign
- Rising after a cascade change
Watch each number by feature, rep team, account segment and product line, not only in total. A healthy overall regret rate can hide a new product where most questions go to the small model and fail.
Keep a fixed set of a few hundred real requests per tier, and re-score it whenever a model, prompt or threshold changes. Retrain the classifier monthly, or after any launch that changes what reps ask. The shadow samples give you the new labels.
It pairs with caching
The three levers so far work at different points. The semantic cache decides whether a model runs at all. The router decides which model runs. The prompt cache makes that run cheaper.
One catch: a prompt cache belongs to one model. Each tier keeps its own cached playbook, so a tier with little traffic may find its cache has expired. Check the cached share per tier, not just overall.
The semantic cache needs the same care. Part 1's cache can serve a small model's answer to a question that should have gone to the frontier. Store the tier with each cached answer, and serve it only to a request routed to that tier or lower.
The lever that touches trust the most
Prompt caching trades almost no trust, because the same model still writes every answer. Routing trades the most, because a cheaper model writes some of them.
In sales, a weak answer costs a rep thirty seconds and a retry. In wealth advice, the same weak answer on the wrong question costs a complaint, or a regulator's letter. The router is where you write that difference down.
So don't start by picking a router. Start with a week of logs and one question for each type of request: what happens if the cheap model gets this wrong?
Model routing looks like an infrastructure decision. It is really a product decision about which mistakes your users can afford.
Next in the series, Part 4: trimming retrieved context, because two good chunks beat ten average ones.