Part 5 of 12 in the series Cutting LLM Costs Without Losing Trust. Part 1 covered semantic caching, Part 2 prompt caching, Part 3 model routing and Part 4 trimming retrieved context. This part covers the other side of the prompt: what the model writes back.
A rep is on a call with a security lead. The question comes fast: "Do you support SSO with Azure AD?"
The rep types it in. The assistant answers in four paragraphs. A heading, a summary, the supported identity providers, a note on SCIM, a caveat about plan tiers, and a closing line offering to help further.
The answer is yes. It's in the second sentence of the third paragraph.
The rep scrolls, loses the thread, and says "let me confirm that and get back to you." The prospect hears a rep who doesn't know the product.
After the call it happens again. The assistant writes a 900-word summary for the CRM. Nobody reads it. The manager skims the first line, and the next steps the rep needed are buried at the bottom.
The assistant got both answers right. Reps still couldn't use them.
The earlier parts cut what goes into the model. This one cuts what comes out, and it turns out that's where the money, the wait and the rep's patience all go.
Why the words coming out cost more than the words going in
Output tokens are priced higher than input. On Claude Sonnet 5.5, input is $2 per million tokens and output is $10. Five times more for every word the model writes.
Put the Azure AD answer into numbers. Take the series' average request: 7,000 input tokens, with 400 tokens written back:
| Tokens | Cost per call | |
|---|---|---|
| Input | 7,000 | $0.014 |
| Output, as written | 400 | $0.004 |
| Output, cut to what the rep needed | 80 | $0.0008 |
- Input
- Tokens
- 7,000
- Cost per call
- $0.014
- Output, as written
- Tokens
- 400
- Cost per call
- $0.004
- Output, cut to what the rep needed
- Tokens
- 80
- Cost per call
- $0.0008
Cutting the answer from 400 tokens to 80 saves $0.0032 a call. To save the same amount on the input side, you'd have to remove 1,600 input tokens, more than Part 4's trimming takes off an average request.
The bigger cost is time. The model reads the whole prompt in one pass, but it writes the answer one token at a time. So the length of the answer sets how long the rep waits for it to finish. Anthropic's latency guide says it plainly: the fewer tokens the model has to process and generate, the faster the response. OpenAI's guide makes the same point from the other side: when an answer starts quickly but takes too long to finish, look at what the model is generating.
Streaming helps the rep see the first words sooner. It doesn't make the answer shorter, and a streamed wall of text is still a wall of text on a live call. Stopping does: if the rep can close an answer once they have what they need, most APIs stop generating when the stream is closed, and you pay only for what was written.
One more line on the bill hides here. Reasoning models think before they answer, and that thinking is billed as output too. OpenAI notes that its output cap covers both reasoning and visible output. A short visible answer can still be expensive if the model thought hard to get there, which is why Part 3's advice on reasoning effort belongs next to this one.
Two moments, two different answers
"Be concise" isn't a spec. The rep needs different things at different times, and the hard part is deciding what those things are before the model writes a word.
During the call, the rep reads in a glance. They're listening and talking at the same time. Anything they can't take in within a few seconds is noise. The answer has to lead with the verdict, then give one fact the rep can say out loud, then point to the source in case the prospect pushes.
For the Azure AD question, that looks like:
Yes. SAML SSO with Azure AD is included on Business and Enterprise.
Say: "Yes, it's on your plan, and setup is in our admin console."
Source: Security overview, SSO section
Three lines. The rep reads the first word and keeps talking.
The "Say" line is a prompt for the rep, not a script. Reps who read it word for word sound like they're reading, and prospects notice. Write it the way a good rep would say it, and tell reps it's there to borrow from, not to recite.
The format also only works if the screen shows it that way. In a narrow side panel or on a phone, three lines of plain text wrap into six, and the verdict gets lost in the middle. The UI has to do its part: the verdict in bold on its own line, the say-line set apart, and the source as a collapsed link the rep can tap if the prospect pushes. Test the format on the screen reps actually use during calls, not on a wide desktop window.
After the call, three people read the same notes for different reasons. The rep wants their next steps. The manager wants to know if the deal moved. The CRM wants fields it can report on. One long summary serves none of them. A fixed set of fields serves all three:
| Field | What goes in it | Who reads it |
|---|---|---|
| Outcome | One line: did the deal move, and how | Manager |
| Next steps | Each action with an owner and a date | Rep |
| Objections | What the prospect pushed back on, in their words | Rep and manager |
| Open questions | What the rep promised to come back on | Rep |
| Risks | Anything that could stall the deal | Manager |
- Field
- Outcome
- What goes in it
- One line: did the deal move, and how
- Who reads it
- Manager
- Field
- Next steps
- What goes in it
- Each action with an owner and a date
- Who reads it
- Rep
- Field
- Objections
- What goes in it
- What the prospect pushed back on, in their words
- Who reads it
- Rep and manager
- Field
- Open questions
- What goes in it
- What the rep promised to come back on
- Who reads it
- Rep
- Field
- Risks
- What goes in it
- Anything that could stall the deal
- Who reads it
- Manager
Each field gets a line or two. Empty fields stay empty instead of being padded.
The budgets for each moment are product decisions, not engineering settings. Set them with the reps who use the tool, then test them: three lines on a live call might be one too many for your team.
How to make it say less
Telling the model to "be brief" gets you a shorter version of the same rambling answer. What works is giving it a shape to fill.
Write an answer contract. Put the shape in the system prompt: verdict first, one fact the rep can repeat, then the source. Count in sentences or lines, not words. Anthropic's guide notes that models count tokens, not words, so sentence or paragraph limits work better than word counts.
Show the length you want. Two or three example answers of the right size teach length faster than any instruction. If your examples run to four paragraphs, so will the answers.
Ban the padding by name. Most extra length is the same few habits: restating the question, a heading on a one-line answer, "Great question", a closing offer to help further, and a summary of what was just said. List them as things not to write.
Use structured output after the call. Ask for the fields from the table above as a schema, each with its own limit, so the model can't spend 300 tokens on the outcome and none on next steps. OpenAI's advice applies to any provider: request only the fields your application uses, and compute anything your code already knows, like dates and counts, in code.
Schema enforcement isn't a guarantee, though. Anthropic's structured outputs don't enforce length limits like maxLength, so the per-field limits live in the field descriptions and your code has to check them. A refusal, or an answer cut off at max_tokens, can still come back as JSON that doesn't match the schema. Check why the response stopped before you parse it, validate every field, and remember that each retry is a full second call, input included. A note that fails validation one time in fifty quietly adds 2% to its cost.
Turn down verbosity and effort where the API offers it. Some models now take a verbosity setting, and OpenAI suggests lower reasoning effort for routine extraction and classification. Filling five CRM fields from a transcript is routine.
Stop where the answer ends. A stop sequence ends generation the moment the model writes a marker you choose. Put a marker after the verdict line in the answer contract, stop there for the live answer, and fetch the rest only when the rep asks. Unlike a token cap, it stops where the contract says, not wherever the count runs out.
Keep max_tokens as a safety net, not a steering wheel. A hard cap stops a runaway answer, but it cuts wherever it lands. Anthropic warns that hitting the cap can stop the response mid-sentence or mid-word. Set it well above the length your contract asks for, and treat any answer that hits it as a bug to investigate, not a short answer to show.
Let the rep ask for more, but price it. The short answer doesn't have to be the only answer. Show the three lines, with a "more detail" option that fetches the fuller version only when someone wants it.
That second call isn't free. It's a new request that re-sends the prompt and writes the long answer. Using the numbers above, a fresh expansion costs about $0.018: 7,000 input tokens and 400 output. The short answer saved $0.0032. So the option pays for itself only if fewer than about 18% of answers get expanded. If the expansion reuses the same session's warm prompt cache, it costs about $0.0104 and the break-even rises to around 31%.
Two design choices keep it cheap. Send the expansion with the same cached prompt prefix while the cache is still warm, and pass the short answer in so the model only writes what's missing, not the whole answer again. And watch the click rate: it's both a cost number and a sign of whether the short answer was too short.
Where short goes wrong
Shorter is only cheaper if it stays right. Six ways it doesn't:
The condition gets cut. "Yes" is wrong if the real answer is "yes, on Enterprise only." The prospect hears yes, signs the cheaper plan and finds out at setup. Any condition that changes the answer, like plan, region, add-on or date, belongs in the verdict line, not in the detail nobody reads. Write that rule into the answer contract.
This is where Part 4 and this part collide. Trimming retrieved context can drop the chunk that holds the condition, like the plan table that says "Enterprise only" or the note about legacy contracts. Then the model never sees it, and no answer contract can bring it back. A short answer makes the gap invisible: a long one might have hedged, a three-line verdict just says yes. So test the two levers together. Your eval set needs questions whose correct answer depends on a condition, run against the trimmed context and the short contract at the same time.
Short sounds certain. A long answer hedges in its own wording. A three-line answer reads as fact. So "I'm not sure, check with the solutions engineer" has to be an allowed short answer, and a better one than a confident guess. Security, legal and pricing answers should keep their required wording even when it adds a line.
A cap gets mistaken for brevity. If max_tokens is doing the shortening, some answers are being cut off, not written short. The rep sees half a sentence. Check how often answers stop at the cap, and fix the prompt when they do.
Broken structure costs twice. The post-call fields come back as JSON. When a response breaks the schema, or stops at the token cap halfway through a field, you pay for the broken response and then for the retry. A small failure rate adds little on its own. The expensive version is a retry loop with no limit. Use the provider's structured output mode where it exists, check the stop reason on every response, retry once on a stronger model, and send whatever still fails to a person instead of retrying again. Track the failure rate per field: a sudden rise usually means a prompt or schema change broke something.
Three lines in English isn't three lines in Arabic. Tokenizers are built mostly on English text, so the same sentence costs more tokens in other languages. One audit found Arabic needs about 1.44 times the tokens of English on GPT-4o's tokenizer, and that every non-English language it tested pays a premium. A budget tuned on English answers will run over for reps selling in Arabic, German or Japanese, and so will the bill. Set and measure budgets per language, not once.
Your evals reward rambling. If you score answers with an LLM judge, it likely prefers the longer one. A study of preference labelling found GPT-4 picked longer answers even when quality was similar, and the bias was stronger when the length gap was large. Left alone, your eval will mark the three-line answer down and push the prompt back toward four paragraphs. Three changes fix most of it:
- Compare answers in pairs instead of scoring each one. Ask the judge which of two answers better serves a rep on a live call, and say in the instructions that extra length is a cost, not a bonus.
- Control for length. The team behind AlpacaEval adjusts each judgement to what it would be if both answers were the same length. That lifted its agreement with human rankings from 0.94 to 0.98. You can do a simpler version: only compare answers within the same length band.
- Calibrate the judge against reps. Keep a fixed set of answers that reps have rated. Run the judge on it whenever the prompt or the judge changes, and stop trusting the judge if it drifts from the reps.
After the call, there's a quieter failure. A short summary can drop the objection nobody wanted to hear. That's why objections and risks get their own fields instead of competing for space in a paragraph.
What it saves, at real 2026 prices
The same team as the earlier parts, 264,000 live requests and 20,000 calls a month, priced at Claude Sonnet 5.5 list prices: $2 per million input tokens, $10 per million output, $0.10 for cached input and $2.50 to write to the cache. Volumes and token counts are assumptions to swap for your own logs.
| Line | Live answers | Live answers, with Part 2's prompt cache | Post-call notes |
|---|---|---|---|
| Requests per month | 264,000 | 264,000 | 20,000 calls |
| Input per request | 7,000 tokens | 7,000 tokens, 4,000 of them cached | 8,000 tokens (transcript) |
| Input cost per month | $3,696 | $1,816 | $320 |
| Output before | 400 tokens: $1,056 | 400 tokens: $1,056 | 900 tokens: $180 |
| Output after | 80 tokens: $211 | 80 tokens: $211 | 250 tokens: $50 |
| Monthly saving | $845 (18% of the bill) | $845 (29% of the bill) | $130 (26% of the bill) |
- Line
- Requests per month
- Live answers
- 264,000
- Live answers, with Part 2's prompt cache
- 264,000
- Post-call notes
- 20,000 calls
- Line
- Input per request
- Live answers
- 7,000 tokens
- Live answers, with Part 2's prompt cache
- 7,000 tokens, 4,000 of them cached
- Post-call notes
- 8,000 tokens (transcript)
- Line
- Input cost per month
- Live answers
- $3,696
- Live answers, with Part 2's prompt cache
- $1,816
- Post-call notes
- $320
- Line
- Output before
- Live answers
- 400 tokens: $1,056
- Live answers, with Part 2's prompt cache
- 400 tokens: $1,056
- Post-call notes
- 900 tokens: $180
- Line
- Output after
- Live answers
- 80 tokens: $211
- Live answers, with Part 2's prompt cache
- 80 tokens: $211
- Post-call notes
- 250 tokens: $50
- Line
- Monthly saving
- Live answers
- $845 (18% of the bill)
- Live answers, with Part 2's prompt cache
- $845 (29% of the bill)
- Post-call notes
- $130 (26% of the bill)
Three things stand out.
The live answers carry the money. Post-call notes are a small line on the bill, so their win is the rep's and manager's reading time, not dollars.
The earlier parts make this one matter more. Once prompt caching makes the stable input cheap, output is a bigger share of what's left: 37% of the bill in the middle column, against 22% without the cache. So the same 320 fewer tokens save 29% of the bill instead of 18%.
The time saving follows the same shape. Because the model writes one token at a time, an 80-token answer finishes in roughly a fifth of the writing time of a 400-token one. On a live call, that's the difference between the rep reading the answer and the rep stalling.
The table assumes nobody clicks "more detail." They will. If 20% of live answers get expanded with a fresh call, the $845 saving turns into a loss of about $105 a month. If those expansions reuse the warm prompt cache, about $300 of the saving survives. The short answer is still the right one for a rep on a call, but this is why the expansion design and the click rate matter as much as the short answer itself.
How to know it's working
Shorter answers are easy to ship and easy to overdo. Watch both sides.
Is it shorter?
- Output tokens per answer, split by moment (live answer, brief, post-call note), at p50 and p95. The p95 shows the answers still rambling.
- Time until the full answer is on screen, at p95, for questions asked during calls.
- Share of answers that stopped at max_tokens. This should be close to zero. If it isn't, the cap is doing the prompt's job.
Is it still right?
- Accuracy on your eval set, with one extra check: did the answer keep every condition that changes it, like plan, region or date?
- How often reps open "more detail". A little is healthy. If they expand most answers, you cut too far.
- How often reps ask a follow-up straight after, like "which plans?" That's a sign the short answer dropped something.
- Thumbs-down on short answers compared with the old long ones.
After the call:
- How much reps edit the notes before saving them. Heavy edits mean the fields don't match what they need.
- How often next steps have an owner and a date. A note without them didn't do its job, whatever its length.
Give every feature an output budget, and an owner. Numbers on a dashboard drift unless someone is paged when they move. Set a p95 output budget for each feature, alert when it's breached, and name who fixes it. Prompts change, models change, and answer length creeps back up with each release. An example to start from, with budgets taken from the contracts above:
| Feature | p95 output budget | Alert when | Owner |
|---|---|---|---|
| Live answer | 120 tokens | Over budget for a day, or more than 1% of answers hit max_tokens | PM for the assistant |
| Post-call note | 350 tokens | Over budget for a day, or any field fails validation more than 2% of the time | PM for the assistant, with sales ops |
| "More detail" expansion | 450 tokens | Click rate above 25% for a week | PM for the assistant |
- Feature
- Live answer
- p95 output budget
- 120 tokens
- Alert when
- Over budget for a day, or more than 1% of answers hit max_tokens
- Owner
- PM for the assistant
- Feature
- Post-call note
- p95 output budget
- 350 tokens
- Alert when
- Over budget for a day, or any field fails validation more than 2% of the time
- Owner
- PM for the assistant, with sales ops
- Feature
- "More detail" expansion
- p95 output budget
- 450 tokens
- Alert when
- Click rate above 25% for a week
- Owner
- PM for the assistant
Run the budget check in CI as well as in production. A prompt change that pushes the p95 from 80 tokens to 200 should fail the build, the same way a change that drops accuracy does.
Brevity is a product decision
It's easy to treat answer length as a prompt tweak. Add "be concise", ship it, move on.
But the hard part was never making the model write less. It was deciding what the rep needs in the three seconds they have, and what the manager needs from the notes, before the model writes anything. That's a product call, made with the people who use the tool, not a setting.
Get it right and the savings follow on their own. The rep hears "yes" in the first word, says it with confidence, and the prospect never notices the assistant was there.
The cheapest token is the one nobody had to read.
The series
Each part uses the same sales assistant, so the savings stack on top of each other.
- Semantic caching: stop paying for the same answer twice
- Prompt caching: stop re-reading the same playbook on every call
- Model routing: use the big model only when the question needs it
- Trimming retrieved context: two good chunks beat ten average ones
- Controlling output length: your sales assistant talks too much (this post)
- Batch processing: not everything needs an answer right now
- Budgets for agent loops: your agent doesn't know when to stop
- Small models and fine-tuning: you probably don't need to fine-tune
- Prompt and context compression: your prompt is longer than the question
- Retrieval and tool-result caching: cache the lookup, not just the answer
- Commercial and governance levers: the biggest discount isn't in the code
- Cost per outcome: stop counting tokens, count deals
Next up, Part 6: batch processing, and which post-call work can wait.
Sources
- Pricing (Anthropic, checked 15 November 2026)
- Reducing latency (Anthropic docs)
- Structured outputs (Anthropic docs)
- Latency optimization (OpenAI docs)
- Verbosity Bias in Preference Labeling by Large Language Models (Saito et al., 2023)
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (Dubois et al., 2024)
- Measuring the Tokenization Premium: A Cost Audit for Underserved Language Communities (arXiv, 2026)