Your prompt feels short, yet the model returns a long answer, and the token count on your bill is larger than you expected. That happens because input tokens and output tokens are counted separately: the prompt is read once, but every output token is generated in a separate forward pass, and each one is added to the context the model must attend to. To predict cost, estimate the output length in tokens and multiply it by the provider's per-token output rate.
Why Output Tokens Differ From Prompt Tokens
A token is the unit a language model actually processes: a chunk of text, often a word piece such as "token", "ization", or a punctuation mark, produced by a tokenizer (the component that splits text into these chunks using a fixed vocabulary). Prompt tokens are what you send; output tokens, sometimes called completion tokens, are what the model writes back. They are counted separately because they are produced by different mechanisms: your prompt arrives all at once, while the output is manufactured step by step.
The asymmetry in volume has a simple cause. A short instruction like "summarize this article in detail" may be a dozen tokens, but the summary it requests could legitimately run to hundreds. The model has no fixed notion of length beyond what the prompt implies and what its training taught it about typical responses, so a vague request often yields a longer answer than you pictured. Anything the model emits counts — including formatting characters, whitespace patterns, and the special markers some chat formats use to signal the end of a turn.
There is also a structural reason the output side is expensive. During generation, the model must condition on the entire conversation so far, prompt plus everything it has already produced. So the work per output token grows with context length, and providers typically price output tokens higher than input tokens to reflect that. The practical takeaway: the prompt's size tells you little about the response's size.
How the Decoder Generates Tokens One at a Time
Most modern LLMs are decoder-only transformers. "Decoder" means the network is trained to predict the next token given everything before it; "transformer" is the attention-based architecture that lets each position look back at earlier positions. Generation is autoregressive: the model produces one token, appends it to the sequence, and runs again with the extended sequence. Repeat until a stop condition is met.
Inside each step, the model computes a probability distribution over its whole vocabulary — the fixed set of tokens it knows, often tens of thousands of entries — and picks one token from it. The selection rule is called the decoding strategy: greedy decoding always takes the most likely token; sampling draws randomly according to the probabilities, with a temperature parameter scaling how flat or peaked that distribution is. Whatever is chosen becomes input for the next step, which is why a single odd choice can steer the rest of the response.
Two constraints bound the loop. First, a maximum output length (often called max_tokens): generation halts when it is reached, even mid-sentence. Second, a stop condition: a designated end-of-sequence token, or a stop string the API watches for. This is why your bill can exceed expectations even when an answer looks finished — the model kept emitting tokens, possibly including invisible ones, until one of these conditions triggered.
The cost of this loop is the key to estimation. The model does not draft the whole answer and then count it; it commits to each token in sequence, and each commitment is billed. If you want N output tokens, expect roughly N sequential forward passes, each slightly more expensive than the last because the context keeps growing.
generation loop:
context = prompt_tokens
while not done:
p = model(context) # distribution over vocabulary
t = select(p, strategy) # greedy or sampled token
context = context + [t] # append, so next step sees it
done = (t == end_token) or len(output) >= max_tokens
## Estimating Output Token Count Before a Request
You can't know the exact output length before generation finishes, but you can bound it. Start with the **max output tokens** parameter (often called `max_tokens` or `max_completion_tokens`): the hard cap the API enforces on generated tokens. If you set it to 500, the model cannot emit more than 500 output tokens, so your worst-case cost is computable in advance. Set this deliberately, not by default.
Second, estimate the *likely* length. Output length correlates with the task shape: a classification label is a handful of tokens; a one-paragraph summary is roughly the length of its source paragraph scaled down; a structured JSON response grows with the number of fields. A practical rule of thumb: prose averages about 0.75 words per token in English, so a 300-word answer is roughly 400 tokens. Treat that as an estimate, not a guarantee.
Third, remember that the **prompt tokens** (your input, plus any system message and conversation history) are also countable before you send anything. Tokenize the exact string you plan to submit on the same model family you'll call, because tokenizers differ between models. Chat templates add a few structural tokens per message, so add a small buffer (a few tokens per message) to your count.
Worked example, with assumed numbers (not real prices):
- Prompt: 800 tokens (measured by tokenizing the input, plus a small template buffer).
- `max_tokens`: 400.
- Assumed input price: $0.50 per 1M tokens; assumed output price: $1.50 per 1M tokens.
Worst case:
input cost = 800 / 1,000,000 * 0.50 = $0.0004 output cost = 400 / 1,000,000 * 1.50 = $0.0006 total = $0.0010 per request
At 1,000 such requests: `1,000 * 0.0010 = $1.00`. If the model typically stops at 150 output tokens instead of the 400 cap, the realistic output cost is `150 / 1,000,000 * 1.50 = $0.000225`, so the total drops to about `$0.000625` per request, roughly $0.63 per 1,000 requests. The gap between worst case and typical case is why the cap and the estimate serve different purposes.
Checklist before sending:
- [ ] Tokenize the exact prompt string on the target model family.
- [ ] Add a buffer for chat template tokens.
- [ ] Set `max_tokens` to the smallest value that still fits a complete answer.
- [ ] Compute worst-case cost from prompt tokens + `max_tokens`.
- [ ] Multiply by expected request volume.
## Calculating Total API Cost With the Token Counter
Total cost is a sum of two independently priced quantities:
total_cost = (prompt_tokens / 1_000_000) * input_price_per_million + (completion_tokens / 1_000_000) * output_price_per_million
Output tokens usually cost more per token than input tokens, so a long answer can dominate the bill even when the prompt is short. A **token counter** — any function that tokenizes text and returns the count — lets you plug real numbers into that formula before the request. Use it twice: once on your prompt, and once on a representative sample output you wrote yourself (or an earlier response) to calibrate your output estimate. Then compute both a worst-case figure (using `max_tokens`) and a typical figure (using the sample), and budget against the worst case.
Two refinements worth knowing: **cached input** (some APIs charge less for repeated prompt prefixes) changes only the input side of the formula, and **reasoning tokens**, where a model bills internal thinking as output, inflate `completion_tokens` beyond the visible answer, so check the usage object returned by the API and compare it to your estimate.
## Frequently Asked Questions
### Why does my LLM generate more tokens than the prompt input?
A token is a chunk of text the model reads or writes, often a word fragment rather than a whole word. The output count differs from the input count because they measure two separate things: your prompt is encoded once, while the response grows token by token through repeated decoding steps, and each step can add a token to a sequence that also gets re-processed. Verbose phrasing, formatting, and the model restating parts of the prompt all inflate output length.
### How does the decoder generate output tokens one at a time?
A decoder-only model predicts the next token from the tokens so far, appends it to the sequence, then repeats. Each pass produces a probability distribution over the vocabulary, and a sampling rule picks one token. Generation stops when an end-of-sequence token appears or a length limit is hit. This loop is why a 20-word answer costs 20-plus forward passes.
### How can I estimate output token count before sending a request?
You cannot know the exact output in advance, but you can bound it. Run a tokenizer on your prompt for the input count, then estimate output as roughly the number of words or characters you expect the answer to contain divided by characters or words per token; a reasonable working range for English is about 0.7 to 1 token per word. Cap the estimate with `max_tokens` and treat it as a ceiling:
estimated_output_tokens ≈ expected_words × 1.3
### How do input and output tokens affect total API cost?
Providers usually price input and output tokens separately, and output tokens typically cost more per token than input. Total cost is the sum across both:
total_cost = (input_tokens × input_price) + (output_tokens × output_price)
Because output price per token is often higher, a long response can dominate the bill even when the prompt is short. Check your provider's current rates rather than assuming a fixed number.
### Can the token-counter help me estimate LLM API cost before a request?
Yes, if you have a token counter that uses the same tokenizer family as your model. Feed it your prompt to get the input count, add your bounded output estimate, and multiply each by the per-token prices. The result is an estimate, not a guarantee, since the actual output length depends on sampling and stopping conditions.