Most LLM bills are not dominated by clever prompts. They are dominated by the same long text being sent again and again: the system prompt, the tool definitions, the reference document, the conversation history. Two provider features attack exactly that waste. Prompt caching discounts the repeated part of your input. Batch APIs discount work that does not need an immediate answer. Used together, they can change the economics of a product without touching model quality.
This guide explains how each one works, how to structure prompts so the cache actually hits, a worked cost example, and the mistakes that quietly turn the savings into nothing.
How prompt caching works
When a model reads your prompt, it builds a for those tokens: the internal keys and values that attention needs. That work is the expensive part of reading a long input. If your next request starts with exactly the same prefix, the provider can reuse the stored work instead of redoing it, and it charges a reduced price for the cached part. Only the new tokens at the end are billed at the normal rate.
The rules differ by provider, so check the current documentation before you commit, but the pattern is consistent:
- The prefix must be identical, character for character. One changed word near the top breaks the match for everything after it.
- There is usually a minimum prefix length, often around a thousand tokens. Short prompts are not eligible.
- Cache entries expire after a few minutes unless traffic keeps them alive. Steady traffic keeps the cache warm, sporadic traffic does not.
- Some providers cache automatically. Others need you to mark the cacheable section, and may charge a premium to write it into the cache.
- Only input tokens are discounted. Output tokens are billed as usual.
For a provider where caching is built into the pricing, see .
Structure prompts for the cache
Because matching is prefix-based, the order of your prompt decides your savings. Put everything stable first and everything that changes last:
- System instructions
- Tool definitions and output schemas
- Few-shot examples or fixed reference documents
- Conversation history, oldest first
- The new user message
The most common mistake is putting something dynamic at the top: a timestamp, a user name, a request ID. It changes on every call, so the cache never matches. Move it to the end, or into the user message.
A second mistake is reordering tool definitions or JSON keys between requests. The text looks equivalent to a human, but it is a different prefix to the cache.
A worked cost example
Use hypothetical prices: $3 per million uncached input tokens, and cached reads at 10% of that, $0.30 per million. Your app sends a 3,000-token fixed prefix plus 200 tokens of new content, on 100,000 requests per month, with a high cache hit rate.
| Tokens per month | Cost | |
|---|---|---|
| No caching | 3,200 x 100,000 = 320M | $960 |
| Cached prefix | 300M at $0.30 per million | $90 |
| New content | 20M at $3 per million | $60 |
| With caching | $150 |
That saves $810, about 84% of the input bill. The result depends on two things: the share of your prompt that is a stable prefix, and how often requests arrive close enough together to hit a warm cache. A short prompt that rarely repeats gains almost nothing.
Does the cache write premium matter?
Where a provider charges extra to write the cache, say 1.25 times the normal input price, the first request is slightly more expensive. Is it worth it? Compare one write followed by one cache read at 0.1 times the price against two uncached requests at 1.0 each:
- With a write and one read: 1.25 + 0.10 = 1.35
- Without caching: 1.00 + 1.00 = 2.00
The write costs 0.25 extra, and each later read saves 0.9. Caching therefore pays for itself after a single cache hit. If you expect even one repeat within the cache lifetime, the premium is worth it. If a prefix is used once and never again, do not mark it for caching.
Caching in common architectures
Chat applications resend the full history on every turn, so each new turn shares a long prefix with the last one. Caching is a strong fit and removes most of the cost growth that long conversations cause.
Retrieval pipelines are trickier. The retrieved chunks change with every query, so the part after your instructions is rarely identical. Cache the instructions and any fixed documents, and accept that retrieved text is paid at the normal rate. covers how to size that part.
Agents reuse a long system prompt and a long tool list across many steps, which is ideal for caching. See for where the rest of the cost comes from.
Batch APIs
Batch endpoints accept a file of requests and return results asynchronously, often within a day. In return, major providers typically offer a meaningful discount on both input and output tokens, commonly around half the normal price. Latency is the price you pay.
Good batch candidates:
- Classifying or tagging a backlog of documents
- Summarizing or extracting data from large collections
- Running overnight
- Generating content in bulk where nobody is waiting
Bad candidates are anything a user is actively waiting on. Also check whether a provider lets the batch discount and the caching discount stack, because the rules vary and the combined saving may be smaller than you expect.
Run your own numbers
- Measure your fixed prefix and a typical new message with the .
- Enter prefix size, new tokens, answer size and monthly volume into the .
- Use the to compare models, then apply each provider's cached and batch rates from its own pricing page.
Remember that and caching multiply together: a model that is half the price and has a good hit rate is far cheaper than either change alone.
Mistakes that erase the savings
Checklist
- Static content first, dynamic content last
- Prefix is long enough to qualify for caching
- No timestamps or per-user text above the cached section
- Tool definitions and schemas kept in a stable order
- Non-urgent jobs moved to the batch endpoint
- Cache hit rate monitored in the provider's usage data
- Costs re-checked against current provider pricing