Back to all articles
Cutting LLM Costs Without Losing Trust
Semantic cachingLLM costsObservabilityApplied AI

Semantic Caching in Practice: From Three Ways of Asking to a Number You Can Track

Janahan Thavendran6 min read

TL;DR

  • A semantic cache matches the meaning of a question, so three rephrased Salesforce questions from reps cost one model call instead of three.
  • The similarity threshold decides what counts as the same question; set it by risk, keep account answers out, and give every entry a time-to-live.
  • Watch hit rate, similarity scores, cost saved, latency and memory, because a cache fails quietly.
In this article

A rep is twenty minutes into a discovery call. That morning, another rep asked the sales assistant, "Do we integrate with Salesforce?" Yesterday someone asked, "Can we sync opportunities into SFDC?" Now this rep types, "Is there a native Salesforce connector?"

To a person, that is one question. To a normal cache, it is three, because the strings are different. Each one goes to the model, pays for a full response, and makes the rep wait while the prospect is still on the line.

What changes with a semantic cache

A normal cache asks, "Have I seen this exact text before?" A semantic cache asks, "Have I seen this meaning before?"

To answer that, it turns each prompt into a vector embedding, a list of numbers that captures what the question is about. It then compares the new vector with the ones already stored and produces a similarity score.

  1. A rep's question arrives and is converted into an embedding.
  2. The cache compares it with stored questions and finds the closest one.
  3. If the score clears the threshold, that is a cache hit. The saved answer is returned straight away and the model is never called.
  4. If it doesn't, that is a cache miss. The question goes to the model, and both the question and the new answer are stored for next time.

On a hit, the answer comes back in milliseconds instead of the seconds a full generation takes, and you pay for nothing but the lookup. That is the difference between a rep reading the answer while the prospect is still talking and a rep filling the silence. For example, a rephrased question that scores about 97% against a stored one is served from the cache.

The threshold is the real decision

The similarity threshold decides what counts as "the same question." Set it too high and the cache catches only near-identical wording, so you save very little. Set it too low and it returns an answer to a question that only sounds similar.

Demo setups often use a threshold around 0.87. Most production setups sit between 0.92 and 0.97, and below about 0.90 wrong answers start slipping through. A bar as low as 0.5 would match questions that merely share a topic.

Sales questions show why that matters. "What's our standard discount on annual plans?" and "What discount did we give Acme on their annual plan?" sit very close in meaning and far apart in answer. One is policy. The other lives in the CRM. A cache that only looks at similarity will serve the policy answer to the account question, and the rep will say it out loud to a customer.

So set the bar by risk. A question about integrations can tolerate a looser match than a question about pricing, security or contract terms. And cache only what is true for every customer: if an answer used account data, it doesn't go in the cache.

Make the cache expire on purpose

A cached answer is only right while the facts behind it are right, and sales material changes every quarter. Say the discount a rep can offer without approval moves from 10% to 15%. A cache that remembers 10% will keep telling reps the old number until something tells it to stop.

The simplest protection is a time-to-live on every entry, so old answers expire without anyone remembering to clear them. For content that changes with a release or a price update, go further: tag each cached answer with the version of the source documents behind it, and clear only the entries a change touches.

Then watch it

A semantic cache fails quietly. A wrong match doesn't throw an error, and a low hit rate doesn't break anything. It just costs money, or occasionally a rep's trust in the tool. So the cache needs a small set of numbers on a dashboard from the first day.

What to watch
Hit rate over time
What it tells you
Whether the cache is paying for itself, measured only on the questions you chose to cache
Warning sign
Flat and low, or a sudden drop after a content update
What to watch
Similarity score distribution
What it tells you
How close real rep questions land to the threshold
Warning sign
A pile of hits just above the bar: the threshold is probably too loose
What to watch
Near-misses caught
What it tells you
Matches that cleared the bar but failed a check, such as an account-specific question
Warning sign
None ever caught, or a rising count after a pricing change
What to watch
Estimated cost saved
What it tells you
The money the cache keeps out of your model bill
Warning sign
Saving less than the cache costs to run
What to watch
Latency on live-call questions
What it tells you
Whether reps feel the difference mid-call
Warning sign
Hits that are not much faster than misses
What to watch
Entry count and memory use
What it tells you
Whether the cache is growing faster than it is being used
Warning sign
Steady growth with a falling hit rate

You can collect these in whatever observability stack you already run. OpenTelemetry and Prometheus metrics are a common way to do it. Open-source tools built around Valkey and Redis, BetterDB for example, can also show hit rate, similarity scores, cost saved and memory use for a semantic cache without much setup.

What matters is not the tool. It is that someone looks at the numbers each week and has permission to change the threshold when they say it is wrong.

What to take away

A semantic cache is a bet that two questions mean the same thing. On a sales floor, the savings come from being right about that bet often enough. The risk comes from the times you are not, because a rep repeats the answer to a customer.

So the work isn't only in building the cache. It is in setting the bar by risk, keeping account answers out, letting old answers expire, and watching the numbers that show whether the bet is paying off. A cache you cannot see into is a cache you cannot trust.