Back to all articles
Cutting LLM Costs Without Losing TrustPart 02
Prompt cachingLLM costsRAGSales assistantsApplied AI

Cutting LLM Costs, Part 2: Your Sales Assistant Is Re-Reading the Same Playbook on Every Call

Janahan Thavendran7 min read

TL;DR

  • Prompt caching reuses the model's work on the start of the prompt, so the playbook is not re-read on every request, while every answer is still freshly generated.
  • It only matches from the front: stable instructions, tools and playbook go first, and the date, rep and question go last.
  • Caches expire, so check reuse before the prefix lapses, and track the cached share of input tokens to see the savings.
In this article

Part 2 of 12 in the series Cutting LLM Costs Without Losing Trust. Part 1 covered semantic caching. This part covers prompt caching.

Picture the weekly cost review for a sales assistant. Someone asks why a short question costs so much. So you open one request and look at what the model was actually sent.

The rep had typed eight words: "Does the Enterprise plan support SSO with Okta?"

Those eight words sat at the bottom of a prompt that also carried the instructions, the sales playbook, the product overview and every tool definition. Thousands of words, sent again on every call, for every rep.

The question is new every time. The playbook is not.

Semantic caching from Part 1 can't help here, because it only works when a question has been asked before. Prompt caching goes after the part that never changes.

The fix: remember what never changes

Prompt caching lets the provider reuse its work on the start of your prompt. When the next request begins with the same content, that part is billed at a lower rate and the first words come back sooner.

The model still runs on every request. It reads the rep's real question and the current account data, and writes a fresh answer. Nothing stale is served. Output tokens cost the same as before, so the saving is on input and on time to first word, not on the whole bill.

One rule: stable first, changing last

A prompt cache matches from the first character forward. The first difference ends the match, so the order of the prompt is the whole design.

Order
1. Cached
What goes there
Instructions, tool definitions, sales playbook
How often it changes
Rarely
Order
2. Cached per call
What goes there
Account context for this call
How often it changes
Once per call
Order
3. Grows each turn
What goes there
Conversation so far
How often it changes
Every turn
Order
4. Fresh
What goes there
The rep's current question
How often it changes
Every turn

A changing block placed before the conversation means the conversation can never be reused. In a long call, keep anything that is fixed for the whole call ahead of the history. If your account data refreshes on every turn, move it after the history.

Anthropic lets you set several cache breakpoints, so each stable block can be reused on its own. A real prompt often has three: the instructions and tools, the playbook, and the conversation so far. When the playbook changes, the instructions and tools can still be read from the cache.

What quietly breaks the match

The usual cause is something small near the top. The full list is longer than most teams expect:

  • The rep's name, region, locale or today's date
  • A request ID or timestamp
  • JSON written with keys in a different order, or extra whitespace
  • A tool list that is reordered, or tool descriptions built on the fly
  • A feature flag that changes wording in the prompt

Build the prompt the same way every time, byte for byte. Anything that has to vary goes after the cached part.

Where it breaks in production

It expires. On Anthropic's default setting a cached prefix lives about five minutes without use. Writing it costs about a quarter more than normal input, and reading it costs a tenth. So one reuse inside the window already pays. With dozens of reps online, that happens all day. For a small team in a quiet time zone, check before you assume.

It has a minimum size. Each provider sets a minimum prefix length, and it can differ by model. Below it, caching silently does nothing and no error appears. Check the minimum for the model you use.

Changes reset it. Updating the playbook, changing a tool schema or switching model all start a new prefix. A cache is also per model. So when you later route simple questions to a smaller model, each model builds its own cache, and the quieter one may expire between calls. If the playbook inside the cached part is out of date, every answer built on it is out of date too, only faster. Version your prompts, bump the version on each release, and expect a short spike of writes after every deploy. Time playbook updates for quiet hours.

It keeps data for a while. The provider holds the cached prompt for its lifetime. For a sales assistant, ask what exactly sits in the prefix, how long it is kept and how it is isolated between customers. Run it past security before launch. Keep account-specific and personal data out of the cached part where you can.

What it is worth

Take the team this series follows: 264,000 requests a month. The average request is 7,000 input tokens, and 4,000 of them are the stable instructions, tools and playbook. Prices are Claude Sonnet 5.5's list prices: $2 per million input tokens, $0.10 for a cache read (a twentieth of the normal price) and $2.50 for a cache write (1.25 times). Assume 95 in 100 requests find the cache.

No caching
Input cost per request
$0.0140
Input cost per month
$3,696
With caching
Input cost per request
$0.0069
Input cost per month
$1,816

That is about half the input bill, roughly $1,880 a month. The stable part is 57% of the input, so the saving can't go much past that: the rest changes with every request, which is where Part 4 comes in. Output cost is unchanged. Rerun it with your own prompt size, traffic and hit rate, because the saving swings with all three.

What to show finance

Check that it works before launch. Send the same long request twice and confirm the second one reports cached tokens. On Anthropic those are the cache read and cache creation input token counts. OpenAI and Gemini report a cached token count in their usage details.

Then log them and watch three numbers each week, by model and by feature. Alert when the hit rate falls after a deploy.

Number
Cached share of input tokens
Warning sign
Low, or falling after a prompt change
Number
Cache writes against reads
Warning sign
Many writes, few reads: the prefix keeps changing
Number
Cost per conversation
Warning sign
No fall after switching it on

Providers differ in how you switch it on. Anthropic lets you mark the cached parts. OpenAI caches automatically for prompts of 1,024 tokens or more with an exact prefix match. Gemini offers implicit and explicit caching. These rules are as of October 2026 and change quickly, so check the current docs before you plan around them.

The quiet failure

Most cost levers in this series trade a little trust for money. This one trades almost none, because the model still reads the real question and writes a new answer.

The risk is quieter. A prefix that stops matching throws no error. The bill just stops falling and nothing looks broken. So keep the front of the prompt boring, and watch the cached share like any other number.

Next in the series, Part 3: model routing, so the big model is used only when the question needs it.