Cutting LLM Costs, Part 1: Your Sales Assistant Is Paying for the Same Answer Twice

Part 1 of 9 in the series Cutting LLM Costs Without Losing Trust. Each part takes one cost lever for an AI sales assistant in production, shows where it saves money, and where it can quietly break the rep's trust in the tool. This one covers semantic caching.

A rep is twenty minutes into a discovery call. The prospect asks whether the product works with their Salesforce setup. The rep types the question into the sales assistant and waits.

One second. Two. Three. The prospect fills the silence with another question, and the moment to answer with confidence has passed.

That same morning, another rep asked the assistant "Do we integrate with Salesforce?" Yesterday someone asked "Can we sync opportunities into SFDC?" Last week it was "Is there a native Salesforce connector?" Each time, the assistant pulled the same docs, sent them to the model, paid for the tokens, and wrote almost the same answer from scratch.

Why your existing cache doesn't catch it

Most teams already have a cache. It's usually Redis or Memcached, and it matches on the exact text of the request.

"Do we integrate with Salesforce?" and "Is there a native Salesforce connector?" are different strings, so each one is a miss. Full retrieval, full model call, full wait.

Sales tools feel this more than most. A few dozen product questions come up on nearly every call: integrations, SSO, SOC 2, data residency, implementation timelines. Every rep asks them, and every rep asks them in their own words.

The gap is bigger than most teams expect. In one case reported by VentureBeat, exact matching caught only 18% of repeated questions. After switching to semantic caching, the hit rate reached 67% and the LLM bill fell 73%.

Match the meaning, not the words

Semantic caching swaps string matching for meaning matching. Teams in production now run it in layers, cheapest check first.

The first check is an exact match on the cleaned-up question. It costs almost nothing and catches the true repeats.

On a miss, the question becomes an embedding, a vector that captures what it's about, and is compared with questions already answered. Most 2026 production setups put the bar between 0.92 and 0.97 cosine similarity. Below about 0.90, wrong answers start slipping through.

A close match still isn't proof. So the better setups add a verification step: a small reranking model reads both questions side by side and confirms the stored answer fits the new one. It adds around 10 milliseconds and lifts the precision of cache hits from about 85% to 96–98%.

Only when all three checks fail does the question reach the model. The new answer is then written back, so the next rep who asks it differently gets the fast path.

What it looks like on a sales floor

Here's how a few real-world rep questions would resolve against the cache. Scores are illustrative; the exact numbers depend on your embedding model.

The first three are exactly what you want. The last two are why this needs a product owner, not just an engineer: one clears the similarity bar and is still wrong, the other is a real question the cache can't even read.

Where it bites in a sales tool

Those last two questions sit close together in meaning and far apart in answer. One is policy. The other is about a single account, and the answer lives in the CRM, not the product docs.

A cache that only looks at similarity will serve the policy answer to the account question. The rep will then say it out loud, to a customer, with confidence.

Scope it. Cache only what is true for every customer: capabilities, integrations, security posture, standard terms. Anything that drew on account context, like deal history, pricing given or open tickets, skips the cache or is keyed to that account. A simple rule works: if the answer used account data, don't write it to the cache.

Filter before you search. Region, role and language change answers too. A rep selling in the Gulf and one selling in Germany may quote different prices for the same plan, and a partner rep shouldn't see what an account executive sees. These become mandatory filters on the lookup, applied before similarity search, not checks bolted on after.

Never cache actions. "Log this call in Salesforce" or "draft a follow-up to Acme" must reach the agent every time. The same goes for live data like deal stage or stock levels. As assistants become agents that take actions, this bypass rule matters more than the threshold.

Make follow-ups stand alone. "And HubSpot?" means nothing without the question before it. Rewrite short follow-ups into a full question using the conversation before you embed them, or skip the cache for them.

Version everything. Sales collateral changes every quarter, often the week before a launch. A time-based expiry is only a backstop. Tag each cached answer with the version of the source docs, the prompt and the embedding model, so when one changes you flush only the answers it touched, not the whole cache.

Set the bar by risk. Pricing, legal and compliance answers get the strictest threshold plus verification. General how-it-works answers can be looser. One global number either leaks mistakes or saves almost nothing.

Pair it with prompt caching

Semantic caching skips the model when a question has been answered before. Prompt caching makes the model cheaper when it hasn't.

Most model providers now let you cache a fixed prompt prefix: the system instructions, the sales playbook, the product overview that rides along with every request. The model reuses its work on that prefix instead of reading it again, which cuts cost and time to first token.

The two work together. Prompt caching handles the large, stable context every call shares. Semantic caching handles the many ways reps ask about it. A sales assistant needs both: one for the fresh question about a new deal, the other for the hundredth question about SSO.

You may not need to build either from scratch. Gateways and managed services like LiteLLM, Kong and Redis LangCache now offer semantic caching as a feature. Check what your stack already has before writing your own.

Prompt caching gets its own deep dive in Part 2: what belongs in the cached prefix, and how to see the savings on your bill.

How to know it's paying off

For the PM, the question isn't whether the cache works. It's whether it saves money, makes reps faster, and keeps answers right.

Start with cost. Tokens avoided are cache hits times the average tokens per model call, and in a RAG assistant that average is high, because every prompt carries retrieved context. Subtract what you spend on embeddings and the vector store to get net savings. Then watch inference cost per active rep fall as the cache warms up.

Then speed. Track p50 and p95 latency, and look hardest at the questions asked during live calls. Generation takes 1.2 seconds or more; a cache hit comes back in 30 to 50 milliseconds. That's the gap between a rep reading the answer while the prospect is still talking and a rep filling silence.

For hit rate, structured domains like support and FAQs tend to land between 30% and 60%. Measure it only on the questions you've chosen to cache, since account questions are left out by design.

Finally, quality. Each week, have someone who knows the product review 1 to 2% of cache hits and check the answer matched what the rep meant. Track thumbs-down on cached answers apart from generated ones, so a bad cache entry can't hide in the average.

Add one number most dashboards miss: false hits avoided. These are matches that cleared the similarity bar but failed verification. It tells you how often the cache almost served a wrong answer, and whether your threshold is drifting as products and pricing change.

Before launch, build a test set from real rep questions, not synthetic paraphrases. Include near-misses that differ only by account, date, region or plan. If the cache can't tell "our standard discount" from "Acme's discount", it isn't ready for a live call.

It's a trust decision, not a cost one

It's tempting to treat semantic caching as a cost project. In a sales tool, it's really about trust.

A rep will forgive a slow answer. They won't forgive one that was confidently wrong in front of a customer, and after the first time, they stop opening the tool.

So the decision that matters isn't the vector database or the threshold. It's drawing a clear line between what's true for every customer and what's true for this one, and only caching the first.

The series

Each part uses the same sales assistant, so the savings stack on top of each other.

1. Semantic caching: stop paying for the same answer twice (this post)
2. Prompt caching: stop re-reading the same playbook on every call
3. Model routing: use the big model only when the question needs it
4. Trimming retrieved context: two good chunks beat ten average ones
5. Controlling output length: reps need three lines, not three paragraphs
6. Batch processing: post-call work doesn't need to be instant
7. Budgets for agent loops: cap the steps before an agent burns your margin
8. Fine-tuning a small model: turn your best answers into a cheaper model
9. Cost per outcome: measure cost per deal influenced, not per token

Next up, Part 2: prompt caching.

Sources

• Why your LLM bill is exploding (VentureBeat, via Pulse)
• The similarity_threshold you adjust on Tuesday
• reflex-cache: tiered exact, semantic and verification cache
• Prompt vs semantic caching (Redis)
• Top LLM gateways that support semantic caching in 2026
• Semantic caching: metadata, tenant boundaries and versioning
• semcache: a correctness-aware semantic cache
• Semantic caching for LLMs: evaluation and isolation