Prompt caching lets an LLM provider store the processed internal state of a repeated prompt prefix, so later requests that start with the same text skip recomputing it. Because input tokens dominate cost in many production workloads, caching those tokens can reduce the input portion of your bill by roughly 50–90% depending on the provider's pricing and how much of your prompt is shared across calls. The saving is real but conditional: it applies only to the cached prefix, only while the cache entry lives, and only when requests actually reuse it.
What Is Prompt Caching and Why It Matters
Every LLM request is billed in tokens — the small chunks of text (roughly 3–4 characters in English) that models read and generate. Providers charge separately for input tokens (what you send) and output tokens (what the model writes). In workloads like document question-answering, coding assistants, or agents with long system instructions, the same large block of text is sent on every call, and the provider reprocesses it from scratch each time. Prompt caching changes that: the provider keeps the computed representation of that block, and subsequent requests that begin with the identical text are billed at a discounted rate for the cached portion.
Why it matters comes down to arithmetic. If 8,000 of your 10,000 input tokens are a fixed system prompt plus reference documents, caching that prefix removes most of the input cost on every follow-up call. Providers typically price cached input tokens at a fraction of normal input tokens — commonly quoted discounts fall in the 50–90% range, though exact rates vary by provider and model and change over time, so treat any specific figure as an estimate to verify in current pricing docs. There is usually also a small premium for the first request that writes the cache, and a minimum prefix length below which caching is not offered.
The trade-off is latency and freshness as well as money. Cached prefixes are typically faster to process, which shortens time-to-first-token. But a cache entry is tied to an exact prefix match: change one character early in the prompt and the cache misses. Entries also expire after a short idle period — often minutes, sometimes longer with extended-retention options — so caching rewards workloads with bursts of similar requests rather than one-off calls.
How Prompt Caching Works Under the Hood
To understand caching, you need the KV-cache. A transformer model processes input through attention layers, and for each token it computes key and value vectors ("K" and "V") that later tokens attend to. These are stored in the KV-cache so the model does not recompute them for every new token. Prompt caching extends this idea across requests: the provider retains the KV state for a prompt prefix and, when a new request arrives with the same prefix, loads that state instead of running the prefix through the model again. Only the new suffix — your actual question — is processed from scratch.
Matching is exact and prefix-based. The provider hashes or fingerprints the token sequence from the start of the prompt up to a marked boundary, and a cache hit requires the tokens to match exactly, including whitespace and formatting. This is why the practical rules are: put stable content first (system instructions, tool definitions, long documents), variable content last (the user's question), and mark the cache boundary explicitly where the provider's API supports it. Some APIs cache automatically up to a detected prefix; others require a cache-control marker on a content block.
Underneath, this is a caching layer with ordinary cache semantics: entries live in fast storage tied to a specific model version, are evicted on a time-to-live or under memory pressure, and are usually scoped to your account or organization so other customers cannot read your cached state. A miss costs you the normal input price plus, on providers that charge it, a cache-write premium; a hit costs the discounted cached-input rate. Because the cached state is identical to what a fresh computation would produce, correctness is unaffected — you are skipping recomputation, not approximating it.
Setting Up Prompt Caching: Best Practices
Prompt caching is only offered by some providers, and the details differ, so treat the following as a general pattern rather than a single recipe. Most implementations share the same idea: you mark part of your request as cacheable, and the provider stores the computed state for that prefix.
Put stable content first, variable content last. Caching works on prefixes, so anything that changes between calls must sit at the end. A practical request layout looks like this:
[system instructions] -> [long reference docs] -> [few-shot examples] -> [user question]
|------------------ cacheable prefix ------------------| |-- variable --|
If you put a timestamp or a request ID near the top, you invalidate the cache on every call. Move such fields into the variable tail.
Explicitly mark the cache breakpoint where supported. Some APIs let you flag where the cacheable prefix ends; others cache automatically above a minimum length. Check your provider's docs, because the minimum token threshold and the cache lifetime (often minutes to hours, with a surcharge for writes and a discount for reads) vary.
Design for reuse. A cache entry is only useful if many requests share the same prefix. Long system prompts, tool definitions, policy documents, or codebase context that apply to a whole session are good candidates. One-off prompts are not.
Verify hit rates. Log cache-read versus cache-write tokens from the response metadata. If reads stay near zero, your prefix is probably churning. A simple diagnostic: hash the prefix and confirm it is identical across consecutive calls.
Checklist before shipping:
- Stable content ordered before variable content
- No timestamps, UUIDs, or user-specific text in the cached prefix
- Prefix length above the provider's minimum
- Cache metadata logged and monitored
- Fallback behavior tested when the cache misses
Measuring Savings with the LLM API Cost Calculator
Savings depend on three quantities: how many input tokens are cached, how often the cache is read versus written, and the provider's price multipliers for cache writes and reads. Since these multipliers differ by provider, use placeholders and plug in your own values.
Assume a workload of 1,000 requests per day. Each request has a 20,000-token cacheable prefix and a 500-token variable tail. Suppose the base input price is $3 per million tokens, a cache write costs 1.25x the base price, and a cache read costs 0.1x. Assume the prefix is written once per day and read on the other 999 requests. Treat these as illustrative assumptions, not published rates.
Without caching, daily input cost:
1,000 requests * 20,500 tokens = 20,500,000 tokens
20.5M / 1M * $3 = $61.50
With caching, the prefix cost splits into one write and 999 reads:
Write: 20,000 / 1M * $3 * 1.25 = $0.075
Reads: 999 * 20,000 = 19,980,000 tokens
19.98M / 1M * $3 * 0.1 = $5.994
Tail: 1,000 * 500 = 500,000 tokens
0.5M / 1M * $3 = $1.50
Total: 0.075 + 5.994 + 1.50 = $7.569
That is roughly $7.57 versus $61.50, about an 88% reduction — consistent with the "up to 90%" claim when the cached prefix dominates and reads are frequent. Change the multipliers to your provider's real values:
savings = 1 - (cache_write_cost + cache_read_cost + uncached_cost) / baseline_cost
The lever that matters most is read frequency: the more requests share a prefix before the cache expires, the closer you get to the read price.
Frequently Asked Questions
What exactly gets cached in prompt caching?
Prompt caching stores the model's internal processing state for a prefix of your input, typically the system prompt plus any long, static context such as documents, instructions, or few-shot examples. What is saved is not the raw text but the intermediate computations the model produces while reading that prefix, so a later request that reuses the identical prefix can skip recomputing it. The cache is keyed on exact token content, so even a small edit near the start of the prefix invalidates the cached portion from that point onward. Anything after the cached prefix (the changing user question, for instance) is still processed fresh each time.
Does prompt caching change the model's output quality?
No. Prompt caching is a performance and cost optimization, not a model change: the cached state is the same computation the model would have performed anyway, so outputs are equivalent to running the full prompt without caching. The main practical risk is subtle: if you cache a prefix and then change something inside it without realizing, the model sees a different prompt than you intended, which can shift results. Treat the cached block as a versioned artifact and keep it byte-for-byte stable to avoid surprises.
Which workloads benefit most from prompt caching?
The biggest wins come from workloads with a large, repeated prefix and high request volume: long system prompts, retrieval-augmented generation with a fixed document set, agent loops that resend the same instructions and tool definitions, and chatbots with extensive persona or policy text. Caching pays off when the reusable prefix is long relative to the variable part and when many requests arrive within the cache's lifetime. One-off requests or prompts where nearly everything changes each call see little or no benefit.
How does prompt caching relate to the KV-cache?
They are closely related: a KV-cache (key-value cache) stores the attention keys and values computed for tokens already processed, which is exactly the kind of state prompt caching preserves across requests. In a single generation, the KV-cache avoids reprocessing earlier tokens as the model emits new ones; prompt caching extends that idea across separate API calls by persisting the prefix's KV state server-side. Think of prompt caching as a durable, cross-request KV-cache for a fixed prefix.
Can I combine prompt caching with batching and streaming?
Yes, both work alongside caching because they operate at different layers. Batching groups independent requests into one call to improve throughput, and each request can still hit the same cached prefix, while streaming controls how the response is delivered token by token and does not affect prefix reuse. The one caveat is ordering and timing: cache hits depend on requests arriving while the cached entry is still alive, so very large or slow batches may see fewer hits than steady, frequent traffic. Test your specific pattern rather than assuming the savings stack linearly.