A context window is the maximum number of tokens a model can consider at once—your prompt plus its response—and every one of those tokens occupies memory and compute while the model works. The "1M tokens" figure describes capacity, not a free allowance: processing a long context costs real GPU memory and time, and providers price or limit it accordingly. Caching helps with repeated prefixes, but it does not make the underlying memory footprint disappear.
What a Context Window Actually Measures
A context window is measured in tokens, the atomic units a model reads and writes. A token is not a word or a character; it is a chunk of text produced by a tokenizer, often a word fragment, a whole short word, punctuation, or part of a number. English text commonly runs somewhere around three to four characters per token, so a rough estimate is that 1,000 tokens corresponds to roughly 700–800 English words—treat that as an estimate, since it varies by tokenizer and language. Code, JSON, and non-Latin scripts often tokenize less efficiently, so the same character count can consume more tokens.
The window is a hard ceiling on the combined length of input and output. If the model's window is 1M tokens and you send a 900,000-token prompt, you have only about 100,000 tokens left for the response (minus any reserved overhead). Exceeding the limit means the request is rejected or must be truncated, and truncation silently drops information.
Two clarifications matter for cost. First, "context window" is a capacity limit, not a billing unit by itself—providers bill per token processed, and long inputs are billed as input tokens regardless of whether they fit. Second, a large advertised window does not guarantee the model uses all of it equally well; attention over very long inputs is harder, and quality can degrade toward the edges. That is a quality concern, separate from the memory and cost story that follows.
From Tokens to KV-Cache: Where the Memory Goes
To predict the next token, a transformer model runs every token through its layers and, crucially, stores intermediate results for attention. Attention lets each token weigh every other token; to avoid recomputing those weights for tokens already seen, the model keeps a cache of keys and values per layer—the KV-cache. This cache is what makes generation fast, and it is also the dominant memory cost of long contexts.
The size of the KV-cache scales roughly linearly with sequence length and with the model's architecture. A simplified estimate looks like this:
KV_cache_bytes ≈ 2 × n_layers × n_kv_heads × head_dim × seq_len × bytes_per_value
Here 2 accounts for both keys and values, n_layers is the number of transformer layers, n_kv_heads is the number of key/value attention heads, head_dim is the dimension per head, and bytes_per_value is the storage per number (commonly 2 bytes for 16-bit precision). Many modern models use grouped-query attention, where n_kv_heads is much smaller than the number of query heads, which shrinks the cache substantially—but the linear growth with seq_len remains.
The practical consequence: doubling the context roughly doubles KV-cache memory per request. On a GPU with tens of gigabytes, a handful of concurrent long-context requests can exhaust memory, which caps how many requests a provider can serve at once. That memory pressure, not the text length alone, is a core reason long contexts are expensive to host.
Why 1M Tokens Is Not Free: Memory, Latency, and Price
A context window is a hard ceiling on how many tokens the model can see at once: your prompt, the conversation so far, and the generated answer all count against it. "1M tokens" describes capacity, not a free allowance. Three costs scale with how much of that window you actually fill.
Memory. Every token in context needs a key/value entry (KV-cache) stored for each layer and attention head. That memory grows linearly with context length and must be resident while the model generates. Fill more of the window and you need more accelerator memory, which is the scarcest resource in serving.
Latency. Attention work grows with context length: roughly linear in sequence length for the memory traffic, and the prefill step must process every input token before the first output token appears. Long prompts mean a longer wait before generation starts, and slower per-token generation afterward.
Price. Providers bill per token, and long-context requests consume more compute and memory per request, so per-token prices are often higher above certain thresholds. Caching input tokens reduces repeated prefill cost but does not make storage or attention free.
Worked example (estimates from stated assumptions). Assume a KV-cache of about 0.5 MB per token (a reasoned order-of-magnitude figure for a mid-sized model; real values depend on layers, heads, and precision), a prompt of 500,000 tokens, and a generated answer of 1,000 tokens.
KV-cache memory = 500,000 tokens x 0.5 MB/token = 250,000 MB = 250 GB
That exceeds a single high-end GPU's memory, so the request needs multiple devices or offloading — real infrastructure cost even if input tokens are cached. Now assume $1 per million input tokens and $4 per million output tokens:
input cost = 500,000 / 1,000,000 x $1 = $0.50
output cost = 1,000 / 1,000,000 x $4 = $0.004
total = $0.504
Cheap per call, but multiply by volume, and note the 250 GB memory footprint is what makes the per-token price what it is.
Estimating Your Own Cost With the LLM API Cost Calculator
Use a calculator to turn a prompt into a number before you send it. The formula:
total_cost = (input_tokens / 1e6 x input_price)
+ (output_tokens / 1e6 x output_price)
+ (cached_tokens / 1e6 x cache_price)
memory_gb = context_tokens x kv_bytes_per_token / 1e9
Checklist before running a long-context request:
- Count tokens for the full prompt, not just your question (system prompt, history, documents).
- Set a realistic expected output length; output tokens usually cost more.
- Confirm whether cached input tokens are billed at a discount and whether the discount applies on reuse.
- Estimate KV-cache memory with your model's actual per-token footprint, not a guess.
- Test with a trimmed prompt first to see latency and quality before scaling to the full window.
- Compare splitting the task across several smaller calls against one large-context call.
Frequently Asked Questions
What is an LLM context window?
A context window is the maximum number of tokens a model can hold in working memory at once, covering your prompt, any retrieved documents, and the response it is generating. A token is a chunk of text roughly three-quarters of a word in English, so a 1M-token window holds on the order of 750,000 words. Everything inside the window is reprocessed or attended to on every step, which is why the size of the window drives cost and latency, not just convenience.
Why does a larger context window cost more even if tokens are cached?
Caching (often called prompt caching or a KV-cache) avoids recomputing the key/value states for unchanged tokens, which cuts compute. It does not eliminate the memory needed to store those cached states, and every generated token must still attend over the entire cached context. So you keep paying for memory footprint and for the attention work at each decode step, and providers typically charge a reduced but nonzero rate for cached input.
How does the KV-cache scale with context length?
The cache stores a key and value vector per layer per attention head per token, so its size grows linearly with context length and with model size. A rough estimate is:
cache_bytes ≈ 2 × layers × heads × head_dim × seq_len × bytes_per_value
For a mid-sized model with, say, 32 layers, 32 heads of dimension 128, and 2 bytes per value, that is about 0.5 MB per token, so 1M tokens lands in the hundreds of gigabytes range. Treat that as an order-of-magnitude estimate, not a spec.
Does a 1M-token context window mean I can send 1M tokens for free?
No. The window is a capacity limit, not an allowance. You are billed for input tokens (usually at a lower rate for cached portions), for output tokens, and indirectly for the memory and compute the long context consumes; latency also rises because each new token attends over more of the context. Filling the window is the most expensive way to use it, not a free tier.
How can I estimate the cost of long-context prompts?
Break the prompt into uncached input, cached input, and output tokens, then multiply each by its per-token price and sum. Account for repeated calls: if you resend a large document across many turns, the cached portion dominates, so model that explicitly. Run a few representative prompts through an LLM API cost calculator to get concrete numbers before committing to a long-context design.