Your prompt costs more tokens than expected because tokenizers split text into subword pieces that don't match how you see the text: a space attached to the front of a word, a newline, a comma, or a three-digit number can each become its own token. Small formatting choices (extra spaces, indentation, punctuation style) change the number of these pieces, and API cost scales directly with the number of tokens sent. The walkthrough below shows where the extra tokens come from and how to spot them before you call the API.
Why Token Counts Diverge From What You See
A token is the basic unit a language model reads and bills for. Text is first run through a tokenizer, a program that splits strings into tokens drawn from a fixed vocabulary (the list of known pieces, often tens of thousands of entries). A token is often a whole common word like "cost", but it can also be a fragment like "ing", a single punctuation mark, or a space glued to a word. Token count is simply how many of these pieces your input produces — not characters, not words.
The mismatch starts with unit conversion. People count words or characters; tokenizers count vocabulary matches. One English word is often one token, but not reliably: a frequent word may be one token, while a rarer or longer word can split into two or more fragments, each billed. Meanwhile a character can be cheaper than a token (several characters inside one piece) or more expensive (special characters that have no multi-character representation). So a prompt of 300 visible words might land anywhere from roughly 350 to 500 tokens depending on vocabulary and formatting — that range is an estimate from typical English subword behavior, not a fixed rule.
Second, tokenizers treat context as part of the piece. A leading space is part of the token before a word, not a free separator. That means "the cat" and "the cat" (two spaces) don't just differ visually; the double space is a separate token in most tokenizers, and the word that follows may tokenize differently too. Newlines, carriage returns, and tabs are tokens of their own as well. Windows-style line endings (\r\n) can cost more than Unix-style (\n).
Third, byte-level tokenizers (common in modern LLMs) work from raw bytes, so anything that isn't a normal printable ASCII character — accented letters, emoji, CJK characters — may need multiple tokens per character. The counterintuitive result: two prompts that look identical to you can differ by dozens of tokens because invisible characters, spacing conventions, or non-ASCII text are handled differently. "More tokens than expected" almost always means "more pieces than the visible text implied," and the fix is to measure, not eyeball.
Whitespace, Newlines, and Repeated Spaces: The Hidden Cost
Whitespace is where most surprises hide because it is invisible, easy to accumulate, and fully billed. Copying text from a PDF or web page often brings double spaces after periods, trailing spaces at line ends, and non-breaking spaces (a space character that prevents line wrapping but still tokenizes). Each of these is a token or part of one. Indentation is the biggest offender in code and structured prompts: four spaces per level, repeated across lines, add up quickly, and a tab may tokenize differently from four spaces.
Consider a compact instruction versus the same text with cosmetic formatting:
Before: Summarize this in 3 bullets.
After: Summarize this in 3 bullets.
Before token estimate: ~7 tokens
After token estimate: ~12 tokens (each extra space run and the doubled space count as pieces)
The estimates are illustrative: the exact counts depend on the tokenizer, but the direction is reliable — each extra space between words is billed, and long runs of spaces may be split into several tokens. Trailing spaces and blank lines cost too. A prompt with a blank line between every paragraph pays for a newline token per gap; three consecutive newlines are not one token but several.
The practical before/after pattern:
Before (copied from a doc):
"Step 1: Open the file.\n\n\nStep 2: Read it. \n"
After (normalized):
"Step 1: Open the file.\nStep 2: Read it."
Normalizing removes doubled spaces, collapses triple newlines to single ones, and strips trailing whitespace. For code blocks and JSON, replace leading space-indentation with a consistent scheme and avoid mixing tabs and spaces. Measure with the tokenizer's own tooling (many providers expose a token-counting endpoint or library) rather than trusting character counts — a whitespace-only diff of 200 characters can be tens of tokens, and the exact number is worth checking, not guessing.
Punctuation, Numbers, and Code: When Characters Multiply
Tokenizers split text into tokens — the atomic units a model reads and bills for. The token count rarely equals the character count, and for dense text it can exceed it. A tokenizer may split a single visible character into several tokens, especially when that character sits outside the patterns it learned most often.
Punctuation. Common marks like periods and commas usually attach to the preceding word and cost nothing extra. Rare or combined marks — ;, …, —, ?!, curly quotes, emoji — often become their own tokens or split a word apart. A heavily punctuated sentence ("Wait—really?! Then…") can tokenize into more pieces than the same words written plainly. The fix is not to strip meaning: use straight quotes, one terminal mark per sentence, and avoid decorative symbols in instructions.
Numbers. Tokenizers are trained mostly on prose, so long digit strings are broken into chunks. A short year like 2024 is often one token, but 1234567890 may split into several. Prices with decimals, version strings (v2.13.4), and UUIDs fragment further because each . and - interrupts the digit pattern. If you pass IDs or figures, expect cost to scale with digit count, not with the single "number" you see.
Code. Code is symbol-dense: brackets, operators, indentation, and camelCase identifiers all create boundaries. Minified code is worst — no spaces, so long symbols split unpredictably — while well-spaced code with short names tokenizes more cleanly. Reformatting barely changes the API request (the model still parses it), but keeping sample code short and un-minified controls size.
Worked example. Assume, for illustration only, a tokenizer where an average English word is about 1.3 tokens (an estimate, not a benchmark). A 100-word prompt costs about 100 * 1.3 = 130 tokens. Naive minification of a code block — removing 40 spaces that each become one token — adds roughly 40 tokens, about 40 / 130 = 31% more. If API price were proportional to tokens, that segment's cost rises ~31% for zero semantic gain.
Checklist before sending a prompt:
- Count tokens with the provider's tokenizer, not words.
- Collapse repeated spaces; drop trailing whitespace and blank lines.
- Use straight quotes and one terminal punctuation mark.
- Shorten long digit strings, IDs, and decimal-heavy text.
- Keep code samples short and normally spaced, not minified.
Multilingual Text and Before/After Formatting Fixes That Cut Cost
English is heavily represented in most tokenizer training data, so English words compress into few tokens. Other scripts — Arabic, Chinese, Japanese, Korean, Hindi — and accented Latin text often split into more tokens per character, sometimes roughly one token per character or worse. The same meaning therefore costs more to send in a non-English language. This is a representation effect, not a quality verdict.
Before:
Q U E S T I O N : W h a t i s A I ? ? ? ?
Repeated spaces, spaced-out letters, and redundant ? inflate tokens.
After:
Question: What is AI?
Collapsing the spacing preserves meaning and removes the inflation.
Rule of thumb (estimate): if a script runs near one token per character, a 500-character passage is about 500 tokens; English prose of similar content might be roughly 60–70% of that (stated as an estimate). Translate or summarize non-English context when the task allows, and remove decorative spacing everywhere.
Reduction formula:
saved_tokens = token_count(before) - token_count(after)
saved_cost = saved_tokens / 1,000,000 * price_per_million_tokens
Measure both versions with the provider's tokenizer before deciding a rewrite is done.
Frequently Asked Questions
Why does my prompt use more tokens than the visible word count suggests?
A token is a chunk of text that the model's tokenizer treats as one unit: sometimes a whole short word, sometimes a word fragment, sometimes a single character or even a byte. Word count and token count diverge because many ordinary words split into several pieces ("unbelievable" might become un + bel + iev + able), and because every space, newline, and punctuation mark is itself encoded as at least one token. So a 100-word prompt can easily run to well over 100 tokens, and formatting-heavy text can push the ratio considerably higher.
Do extra spaces and newlines really increase token count and API cost?
Yes. Each newline typically costs a token of its own, and runs of spaces are usually encoded in small groups rather than compressed away, so indentation and trailing whitespace are billed like any other text. Three blank lines between paragraphs, a two-space indent on every line, or dozens of trailing spaces after a line all add tokens without adding meaning. Cost scales with token count, so trimming redundant blank lines and collapsing repeated spaces in instructions — while keeping the structure the model needs — directly lowers what you pay per call.
Why do numbers and code snippets tokenize into so many tokens?
Tokenizers are trained mostly on natural language, so digits and symbols are often handled in small pieces: a long number may split into fragments or even single digits, and each one counts. Code compounds this because it is dense with punctuation, brackets, operators, and indentation, and each of those characters or short sequences consumes tokens. As a rough estimate, a one-line expression like result = compute(a, b); can cost on the order of ten to fifteen tokens, several times its visible "word" count.
Is multilingual text more expensive to send than English?
Often, yes — though it depends on the language and the tokenizer. Tokenizers tend to give common English words efficient single-token or few-token encodings, while less-represented languages, non-Latin scripts, and characters outside the tokenizer's frequent set frequently split into multiple tokens per character or per byte. As a rough estimate, the same sentence in a lower-resource language can cost two to four times the tokens of its English equivalent. This is not a hard rule, so measure rather than assume.
How can I measure and reduce my prompt token count before calling the API?
Measure with the tokenizer that matches your model: tokenizer libraries (and many providers' token-counting utilities) let you count tokens for a candidate string offline, before any API call. Then reduce deliberately: collapse repeated spaces and excess blank lines, drop redundant boilerplate and duplicated instructions, shorten long numeric IDs, and send only the code or document sections actually needed. Re-count after each change; small formatting edits often produce measurable savings, which is the whole point of testing before/after versions.