Back to all articles
Cutting LLM Costs Without Losing TrustPart 04
RAGRetrievalLLM costsSales assistantsApplied AI

Cutting LLM Costs, Part 4: Trimming Retrieved Context

Janahan Thavendran12 min read

TL;DR

  • Retrieved chunks change with every question, so caching can't cut them; sending only the text the answer needs can cut their cost by about 70%.
  • Most of the cut happens before the reranker: one live version per page, metadata filters built from the question and the account, and parent sections that keep each rule with its exceptions.
  • Entitlement questions go to a plan and contract table, and a golden set tagged by account checks that the right chunk survives the cut.
In this article

Part 4 of 12 in the series Cutting LLM Costs Without Losing Trust. Part 1 covered semantic caching, Part 2 prompt caching and Part 3 model routing. This part covers trimming retrieved context.

A rep on a call types: "Does the Enterprise plan include SSO with Okta?"

Retrieval does its job and sends back ten chunks. One is the current security page, and it has the answer. The rest are near misses: two case studies that mention Okta, a release note, the integrations list, and three older versions of the pricing page. One of those says SSO was a paid add-on.

The model reads all ten, about 5,000 tokens, to write one sentence. Some days it writes the right sentence. Some days it hedges, because the old pricing page says otherwise.

The answer was in one chunk. The bill was for ten.

Caching from Parts 1 and 2 doesn't touch this. Retrieved chunks change with every question, so they sit in the part of the prompt that can't be cached. Routing from Part 3 makes each token cheaper, but still sends all ten chunks. Trimming goes after the tokens themselves.

What trimming retrieved context does

Trimming means sending the model only the retrieved text it needs to answer. It looks like a selection problem: pick the best few chunks. In practice it is a system. A clean index, metadata filters, selection, then assembly that keeps each fact whole.

The difference from the earlier parts matters. Caching and routing cut cost without changing what the model reads. Trimming changes what it reads. That makes it the one lever in the series that can make answers cheaper and better at the same time. It is also the one that fails by omission: the model can't flag a fact it never saw.

Where the tokens go

Open one product question in the logs and split the prompt into its parts. The numbers below are this example's, not a benchmark, but the shape is common.

Part of the prompt
Instructions, tools, playbook
Tokens
about 4,000
How often it changes
Rarely
Which lever helps
Prompt caching (Part 2)
Part of the prompt
Retrieved chunks
Tokens
about 5,000
How often it changes
Every question
Which lever helps
Trimming (this part)
Part of the prompt
Account context from the CRM
Tokens
about 600
How often it changes
Every call
Which lever helps
Trimming: send only the fields the question needs
Part of the prompt
Conversation so far
Tokens
about 380
How often it changes
Every turn
Which lever helps
Summarise older turns
Part of the prompt
The rep's question
Tokens
about 20
How often it changes
Every turn
Which lever helps
Nothing to cut

The stable top is already cheap after Part 2. The biggest block that is paid in full on every question is the retrieved text. That is where this part looks.

Trim the system, not just the list

A better reranker doesn't fix the Okta answer. A reranker only chooses among the chunks retrieval hands it, and by then the old pricing page is already in the running. Most of the cut has to happen earlier: in what the index holds, how each chunk is tagged, and what the query is allowed to see.

Most of the cut happens before the reranker
Rep’s question + account context
what is asked, for which customer
Everything else → four layers
1 · Clean the index
one live version per page, tagged with plan, region, dates
2 · Filter before you search
plan, region, status = current, this account’s contract
3 · Retrieve, rerank, cut
30–50 candidates in, only those above the score out
4 · Assemble whole facts
parent section sent, exceptions kept with their rule
Entitlement questions → decided by the table
Entitlement lookup
plan × contract table, the source billing uses
The model answers
and cites its source

Old versions are filtered out, not out-scored. The reranker only chooses among what is left.

trimming retrieved context · four layers and an entitlement lookup

Questions about what a customer is entitled to skip document search. The documents explain; the table decides.

1. Clean the index. Keep one live version of each page. When pricing changes, last year's page moves to an archive index. It stays searchable for "what did we sell in 2024?", but never mixes into live answers. Every chunk carries metadata: product, plan, region, effective dates, status, owner and last review date.

2. Filter before you search. Build filters from the question and the account, not just the words. Plan is Enterprise, status is current, effective today, region matches the customer's. Now the 2024 pricing page is never a candidate. It is filtered out, not out-scored, and that is the difference between usually right and reliably right.

3. Retrieve, rerank and cut. Search the filtered set for 30 to 50 candidates, rerank them against the question, and keep those above a score, up to a cap. Allow the answer to be zero chunks.

4. Assemble whole facts. Search small chunks, send their parent. A small chunk like "SSO with Okta is included on Enterprise" matches the question precisely. The model then gets the section it came from, so "except legacy contracts" arrives with the rule. When chunking, never split a rule from its "except", "unless" or footnote, and tag each exception with the rule it changes so the two always travel together.

Small chunks also lose context on their own. Anthropic's contextual retrieval adds a short line to each chunk saying where it comes from, such as "From the Enterprise security page, 2026". In their tests, that plus keyword search and reranking cut failed retrievals by 67%.

Some answers aren't documents. "Does Acme get SSO?" depends on Acme's contract, not just the plan page. Exceptions like legacy contracts belong in a structured entitlements table, ideally the same source billing uses. The assistant reads the account's plan and contract type from the CRM, looks them up, and gets one answer for this customer: "Acme is on a legacy contract, so SSO is a paid add-on." The documents then explain why. No reranker can make that call, because the fact that decides it was never in a document.

What it saves

Take the same 200 reps from Part 3: 264,000 requests a month. Say 40% of them are product questions that go through retrieval, about 105,600 a month. Filtering, reranking and cutting bring the retrieved text down from about ten chunks to about three, parent sections included, from 5,000 tokens to 1,500. Prices are Part 3's: Sonnet 5.5 at $2 per million input tokens, or about $1.16 averaged across the routed mix of 55% Haiku, 35% Sonnet and 10% Opus.

Setup
10 chunks, all on Sonnet 5.5
Retrieved tokens per question
5,000
Monthly cost of retrieved text
$1,056
Setup
3 chunks, all on Sonnet 5.5
Retrieved tokens per question
1,500
Monthly cost of retrieved text
$317
Setup
10 chunks, routed as in Part 3
Retrieved tokens per question
5,000
Monthly cost of retrieved text
$610
Setup
3 chunks, routed as in Part 3
Retrieved tokens per question
1,500
Monthly cost of retrieved text
$183

Trimming cuts the cost of retrieved text by 70% either way. In dollars, it matters most where tokens are expensive. Before routing, it saves about $740 a month. After routing, about $430, less about $20 for a reranker at Cohere Rerank 5 Fast's per-token price (Part 1). At this team's size the dollars are modest; the quality case below is the stronger one. The hard questions that Part 3 sends to the frontier model also tend to pull the most context, so that is where to trim first.

The quality case is stronger than the cost case. In Lost in the Middle, models used information best when it sat at the start or end of the context, and got worse as the context grew. Chroma's context rot study tested 18 models and found answers grew less reliable as input got longer. A single distracting passage was enough to hurt. Three old pricing pages are three distractors, and a version filter removes them for free.

There is a limit in the other direction. In Anthropic's tests, passing 20 chunks beat passing 10 or 5. The right number depends on your questions and your chunks. Two good chunks beat ten average ones, but the word that matters is good. Find the number with your own golden set, not a rule of thumb.

Where it bites in a sales tool

Exceptions live in other systems. Parent sections keep an exception that sits on the same page. They can't help when it lives in a contract, a deal desk note or an approval email. If an exception decides the answer, move it into the entitlements table or tag it onto the rule it changes. Otherwise a cut will drop it sooner or later.

New launches fall below the line. A reranker scores a question about a product launched last week against pages it has barely seen. The right chunk scores low and gets cut. Recall drops most exactly when reps ask the most questions.

Metadata goes stale. Filters trust the tags. A page still marked current after a price change, or a draft tagged as live, passes every filter and wins. Give every page an owner and a review date, and make archiving the old version part of the release checklist, not a cleanup job.

Padding hides a miss. When nothing clears the score, a system built to always send three chunks sends three weak ones, and the model guesses. Let the assistant say "I can't find that in the knowledge base" and link the owner. A rep forgives a gap more easily than a wrong answer.

How to see it working

Before you cut anything, build a golden set: a few hundred real rep questions, each tagged with the chunk that holds its answer and the account it was asked about, since the right answer can differ by contract. It answers the one question that matters here: does the right chunk survive the cut?

Then watch these numbers each week:

What to watch
Retrieved tokens per question
What it tells you
Whether the cut is holding
Warning sign
Creeping back up
What to watch
Live pages past their review date
What it tells you
Whether filters can trust the tags
Warning sign
Growing, or any page with no owner
What to watch
Chunks kept after reranking
What it tells you
Whether the score threshold fits
Warning sign
Most questions hit the cap, or keep none
What to watch
Survival rate on the golden set
What it tells you
Whether the right chunk, with its exceptions, makes it through
Warning sign
Falling, or lower for new products
What to watch
Entitlement answers that match billing
What it tells you
Whether the lookup and the documents agree
Warning sign
Any mismatch
What to watch
Answer accuracy on the golden set
What it tells you
Whether answers got better or worse
Warning sign
Falling after a threshold change
What to watch
"Can't find it" rate
What it tells you
How often nothing clears the bar
Warning sign
Rising after a launch or content change
What to watch
Cost per conversation
What it tells you
What finance actually sees
Warning sign
No fall after the change

Re-run the golden set whenever the knowledge base, the chunking or the threshold changes, and add a fresh batch of questions after every launch.

The parts you wrote yourself

Retrieved chunks get the attention, but two other blocks in that table grow quietly, and the retrieval stack has a bill of its own.

Instructions and tool definitions. The 4,000-token top is cheap once cached, but cached isn't free: on Claude Sonnet 5.5 a cached read is $0.10 per million tokens, and every prompt edit means a fresh cache write at $2.50 per million. Tool definitions are often the heaviest part. Every tool's JSON schema, with its long descriptions and examples, rides along on every call whether the tool is used or not. Read them like a chunk list. Remove tools the feature never calls, cut descriptions to what the model needs to choose correctly, and load rarely used tools only for the tasks that need them. Test on the golden set, because the same failure applies: cut the line that told the model when not to call a tool, and it starts calling it.

The conversation so far. During a live call, every turn adds a question and an answer to the next request. By the twentieth question, the history can be bigger than the retrieved chunks. Summarise older turns into a few lines (what the prospect asked about, what the rep promised) and keep only the last few turns word for word. Two cautions. A summary can drop the condition that mattered, so keep commitments and numbers verbatim. And rewriting the history breaks any cache that covers it: keep the summary after the stable cached prefix, and refresh it every few turns rather than on every one.

The retrieval stack itself. Trimming cuts what reaches the model, not what it cost to find. Embeddings are cheap, at $0.02 per million tokens for OpenAI's text-embedding-3-small, so indexing the knowledge base costs little. The cost sits elsewhere: re-embedding everything when you change embedding models, the vector store's storage, replicas and monthly minimum, and a reranker call on every question. For a low-volume assistant, the vector store's minimum alone can be bigger than the model bill.

It pairs with caching and routing

Each lever so far covers a different part of the request. The semantic cache skips the model when the question has been answered before. The prompt cache makes the stable top of the prompt cheap. Routing picks the model. Trimming shrinks the part in between that changes with every question.

Order matters when you add them. Clean the knowledge base and trim before you train a router. A router's labels come from comparing answers, and answers built on ten noisy chunks make the small model look worse than it is.

The lever that can raise quality

Most levers in this series trade a little trust for money. Trimming can buy both, because a model that reads less noise often answers better.

But it fails quietly, and in one direction. A cache that misses costs money. A router that misjudges gives a weaker answer. A cut that drops the right chunk gives a confident wrong answer, with nothing in the prompt to warn the model.

Trimming looks like choosing the best chunks. Most of it is deciding what is allowed to be a chunk at all: one live version, tagged, filtered to this customer, with its exceptions attached. Two good chunks beat ten average ones only when the system makes sure they are the right two.

Next in the series, Part 5: controlling output length, because reps need three lines, not three paragraphs.