Part 2 of 12 in the series Cutting LLM Costs Without Losing Trust. Part 1 covered semantic caching. This part covers prompt caching.
Picture the weekly cost review for a sales assistant. Someone asks why a short question costs so much. So you open one request and look at what the model was actually sent.
The rep had typed eight words: "Does the Enterprise plan support SSO with Okta?"
Those eight words sat at the bottom of a prompt that also carried the instructions, the sales playbook, the product overview and every tool definition. Thousands of words, sent again on every call, for every rep.
The question is new every time. The playbook is not.
Semantic caching from Part 1 can't help here, because it only works when a question has been asked before. Prompt caching goes after the part that never changes.
The fix: remember what never changes
Prompt caching lets the provider reuse its work on the start of your prompt. When the next request begins with the same content, that part is billed at a lower rate and the first words come back sooner.
The model still runs on every request. It reads the rep's real question and the current account data, and writes a fresh answer. Nothing stale is served. Output tokens cost the same as before, so the saving is on input and on time to first word, not on the whole bill.
One rule: stable first, changing last
A prompt cache matches from the first character forward. The first difference ends the match, so the order of the prompt is the whole design.
| Order | What goes there | How often it changes |
|---|---|---|
| 1. Cached | Instructions, tool definitions, sales playbook | Rarely |
| 2. Cached per call | Account context for this call | Once per call |
| 3. Grows each turn | Conversation so far | Every turn |
| 4. Fresh | The rep's current question | Every turn |
- Order
- 1. Cached
- What goes there
- Instructions, tool definitions, sales playbook
- How often it changes
- Rarely
- Order
- 2. Cached per call
- What goes there
- Account context for this call
- How often it changes
- Once per call
- Order
- 3. Grows each turn
- What goes there
- Conversation so far
- How often it changes
- Every turn
- Order
- 4. Fresh
- What goes there
- The rep's current question
- How often it changes
- Every turn
A changing block placed before the conversation means the conversation can never be reused. In a long call, keep anything that is fixed for the whole call ahead of the history. If your account data refreshes on every turn, move it after the history.
Anthropic lets you set several cache breakpoints, so each stable block can be reused on its own. A real prompt often has three: the instructions and tools, the playbook, and the conversation so far. When the playbook changes, the instructions and tools can still be read from the cache.
What quietly breaks the match
The usual cause is something small near the top. The full list is longer than most teams expect:
- The rep's name, region, locale or today's date
- A request ID or timestamp
- JSON written with keys in a different order, or extra whitespace
- A tool list that is reordered, or tool descriptions built on the fly
- A feature flag that changes wording in the prompt
Build the prompt the same way every time, byte for byte. Anything that has to vary goes after the cached part.
Where it breaks in production
It expires. On Anthropic's default setting a cached prefix lives about five minutes without use. Writing it costs about a quarter more than normal input, and reading it costs a tenth. So one reuse inside the window already pays. With dozens of reps online, that happens all day. For a small team in a quiet time zone, check before you assume.
It has a minimum size. Each provider sets a minimum prefix length, and it can differ by model. Below it, caching silently does nothing and no error appears. Check the minimum for the model you use.
Changes reset it. Updating the playbook, changing a tool schema or switching model all start a new prefix. A cache is also per model. So when you later route simple questions to a smaller model, each model builds its own cache, and the quieter one may expire between calls. If the playbook inside the cached part is out of date, every answer built on it is out of date too, only faster. Version your prompts, bump the version on each release, and expect a short spike of writes after every deploy. Time playbook updates for quiet hours.
It keeps data for a while. The provider holds the cached prompt for its lifetime. For a sales assistant, ask what exactly sits in the prefix, how long it is kept and how it is isolated between customers. Run it past security before launch. Keep account-specific and personal data out of the cached part where you can.
What it is worth
Take the team this series follows: 264,000 requests a month. The average request is 7,000 input tokens, and 4,000 of them are the stable instructions, tools and playbook. Prices are Claude Sonnet 5.5's list prices: $2 per million input tokens, $0.10 for a cache read (a twentieth of the normal price) and $2.50 for a cache write (1.25 times). Assume 95 in 100 requests find the cache.
| Input cost per request | Input cost per month | |
|---|---|---|
| No caching | $0.0140 | $3,696 |
| With caching | $0.0069 | $1,816 |
- No caching
- Input cost per request
- $0.0140
- Input cost per month
- $3,696
- With caching
- Input cost per request
- $0.0069
- Input cost per month
- $1,816
That is about half the input bill, roughly $1,880 a month. The stable part is 57% of the input, so the saving can't go much past that: the rest changes with every request, which is where Part 4 comes in. Output cost is unchanged. Rerun it with your own prompt size, traffic and hit rate, because the saving swings with all three.
What to show finance
Check that it works before launch. Send the same long request twice and confirm the second one reports cached tokens. On Anthropic those are the cache read and cache creation input token counts. OpenAI and Gemini report a cached token count in their usage details.
Then log them and watch three numbers each week, by model and by feature. Alert when the hit rate falls after a deploy.
| Number | Warning sign |
|---|---|
| Cached share of input tokens | Low, or falling after a prompt change |
| Cache writes against reads | Many writes, few reads: the prefix keeps changing |
| Cost per conversation | No fall after switching it on |
- Number
- Cached share of input tokens
- Warning sign
- Low, or falling after a prompt change
- Number
- Cache writes against reads
- Warning sign
- Many writes, few reads: the prefix keeps changing
- Number
- Cost per conversation
- Warning sign
- No fall after switching it on
Providers differ in how you switch it on. Anthropic lets you mark the cached parts. OpenAI caches automatically for prompts of 1,024 tokens or more with an exact prefix match. Gemini offers implicit and explicit caching. These rules are as of October 2026 and change quickly, so check the current docs before you plan around them.
The quiet failure
Most cost levers in this series trade a little trust for money. This one trades almost none, because the model still reads the real question and writes a new answer.
The risk is quieter. A prefix that stops matching throws no error. The bill just stops falling and nothing looks broken. So keep the front of the prompt boring, and watch the cached share like any other number.
Next in the series, Part 3: model routing, so the big model is used only when the question needs it.