Some numbers get split into more tokens than others because tokenizers like BPE (Byte Pair Encoding) are trained to compress whatever appears most often in their training text, and long or rare digit strings simply don't appear often enough to earn a single token. The number 42 is common enough to be one token in many tokenizers; 123456 usually isn't, so it gets broken into pieces the tokenizer does know, such as 123 + 456. The general rule: the more common and shorter a number, the fewer tokens it costs; the rarer, longer, or more unusually punctuated it is, the more it fragments.
What BPE Tokenization Actually Does to a String
BPE starts from raw bytes or characters and repeatedly merges the most frequent adjacent pair into a new unit. Train it on enough English text and merges like t + h → th, then th + e → the appear, so common words and word fragments become single tokens. The tokenizer stores a fixed vocabulary (the complete list of tokens it knows) plus the merge rules that built it. At encoding time, a new string is chopped using those learned merges — nothing is invented on the fly.
This is why tokenization is data-driven rather than semantic. A token has no obligation to correspond to a word, a digit, or a meaningful chunk; it corresponds to a sequence that was frequent in training. Most production tokenizers also include a fallback: any byte not covered by a merge can still be represented, so nothing is truly unencodable. That guarantees coverage but not efficiency — rare byte sequences fall back to one token per byte, or close to it.
Numbers are a special case. Digit strings are combinatorially huge: there are only so many times the appears, but the space of six-digit numbers is a million wide, and most specific values appear rarely outside datasets like prices, IDs, or timestamps. So tokenizers tend to learn short digit chunks — 1, 12, 123, 2024, 000 — and common date fragments, while longer or unusual combinations get assembled from those pieces. A useful mental model: tokenization is compression, and compression favors repetition.
A Worked Example: Splitting Multi-Digit Numbers and Dates
Take 123456. A tokenizer that has 123 and 456 in its vocabulary (both common in numeric text) can encode it as two tokens. If it also learned 1234, it might produce 1234 + 56 — two tokens, different split. The exact split depends on the merge rules and how the encoder resolves competing candidates, which is why different models disagree. A plausible encoding looks like:
"123456" -> ["123", "456"] # 2 tokens
"42" -> ["42"] # 1 token (very common)
"987654" -> ["987", "654"] # 2 tokens (if those chunks exist)
"1234567" -> ["123", "456", "7"] # 3 tokens (odd length, no clean pair)
Dates behave similarly but with more structure. 2024-06-01 contains a year, hyphens, and zero-padded month and day. 2024 is frequent enough to be a single token in many tokenizers; -06 and -01 may each be one token if hyphenated date fragments were common in training, or may split into - + 06. A reasonable estimate is 4–6 tokens for a full ISO date, but that is an estimate from the structure, not a guarantee — always verify against the specific tokenizer.
The lesson from the example: token count tracks how the string decomposes into known chunks, not how many characters or "ideas" it has. Two numbers of equal length can cost different amounts. 100000 may be cheap because 000 is a common chunk; 837291 may cost more because its digit pairs are rarer. When you estimate, think in chunks you'd expect a tokenizer to have seen often: years, round numbers, common prices, repeated digits.
Punctuation, Whitespace, and Why "1,234.56" Costs More
Byte-pair encoding (BPE) builds its vocabulary by repeatedly merging the most frequent adjacent pair of characters — or, in practice, the most frequent byte pair — in a training corpus. The result is a ranked list of subword units. When you tokenize new text, the model walks through it and applies the longest merge it knows at each position. Anything not covered by a learned merge stays as individual characters or bytes.
This is why separator characters matter. A string like 1234 has a chance of existing as one learned unit if it appeared often enough during training. But 1,234.56 contains commas and a period, and those characters pull the surrounding digits apart. The tokenizer cannot merge ,2 into the 1 token unless that exact sequence was learned. Instead you typically get fragments: 1 + , + 234 + . + 56, or similar. Each fragment is a separate token.
Whitespace compounds this. Many tokenizers encode a leading space as part of the token that follows it, so 2024 and 2024 are different tokens entirely. A space before a number can cost you an extra token if the spaced version is rarer.
Worked example, using a simplified tokenizer to make the arithmetic visible. Assume our toy vocabulary contains these learned units: 123, 456, 20, 24, ,, ., -, and single digits. Tokenizing:
"123456" -> ["123"]["456"] = 2 tokens
"1,234.56" -> ["1"][","]["234"]["."]["56"] = 5 tokens
"2024-06-01"-> ["20"]["24"]["-"]["0"]["6"]["-"]["0"]["1"] = 8 tokens
" 2024" -> [" "]["20"]["24"] = 3 tokens
Five short characters like 1,234 cost more than six characters like 123456, because the punctuation breaks every potential merge. That is the counterintuitive core of the answer.
A Rule-of-Thought for Estimating Token Count Before You Send
You do not need the exact tokenizer to get a useful bound. Use this reasoning chain:
- Word count first. Count whitespace-separated words. A common rough estimate for English is 1.3 tokens per word; treat that as a rule of thumb, not a measurement.
- Add digit cost. Long integer runs split at learned boundaries. Assume roughly one token per 2–4 digits for arbitrary numbers; common numbers like years may be cheaper.
- Add one token per punctuation mark or separator (commas, periods, hyphens, slashes) inside numbers and dates.
- Add one token for a leading space on tokens that rarely appear with a space.
- Round up. Estimation is asymmetric: overestimating costs you nothing, underestimating can truncate a prompt.
A compact checklist before sending:
[ ] Count words, multiply by 1.3
[ ] For each number: digits / 3, rounded up
[ ] +1 per comma, period, hyphen, slash in numbers/dates
[ ] +1 per rare symbol (@, #, unusual punctuation)
[ ] Sum, then round up ~10% as a safety margin
Example: a 50-word sentence with three dates like 2024-06-01 and two decimal prices. Words: 50 × 1.3 = 65 estimated tokens. Dates: 3 × 4 digits ≈ 3 × 2 = 6 tokens, plus 2 hyphens each = 6 more. Prices: 2 × ~3 digit tokens + separators ≈ 8. Total ≈ 85, round to ~94 with margin. The estimate is not exact, but it is close enough to decide whether you are near a context limit.
Frequently Asked Questions
Why does the number 123456 sometimes become several tokens instead of one?
Byte Pair Encoding (BPE) builds its vocabulary by merging the most frequent character pairs it sees during training, so a digit string only becomes a single token if that exact string appeared often enough in the training data to earn a slot. A common number like 2024 or a short chunk like 100 may survive as one token, but 123456 is rare in real text and gets broken into pieces the tokenizer already knows, such as "123" plus "456". The split points come from the learned merge table, not from any rule about digits, so the same length of number can cost wildly different token counts depending on which substrings happen to be common.
Do dates like 2024-06-01 always tokenize the same way across models?
No. Each model ships with its own tokenizer trained on its own corpus, so the vocabulary and merge order differ, and the same date string can split differently from one model to the next. A tokenizer that saw many ISO-style dates might keep "2024" and "-06" as units, while another might split the hyphens off separately or treat "06-01" as one piece. The practical consequence is that token counts are model-specific: an estimate from one tokenizer is only a rough guide for another, and you should measure with the tokenizer of the model you actually call.
Does punctuation really add to my token count and API cost?
Yes. Punctuation characters are tokens in their own right unless the tokenizer learned a merged piece that includes them, and separators inside numbers are a clear example: "1,234.56" typically costs more tokens than "1234.56" because the comma and period each tend to stand alone or attach awkwardly. Whitespace matters too, since a leading space often changes how the following word is tokenized. Because API pricing is generally per token, every extra punctuation mark and space you send is billed, so trimming redundant formatting is a small but real saving.
How can I estimate token count before sending text to an LLM?
The reliable method is to run your text through the same tokenizer library the model uses, which is exact but requires the tokenizer files. For a quick mental estimate, assume roughly one token per four characters of ordinary English prose, then add a little for digit strings, punctuation-heavy text, code, or non-Latin scripts, which tend to run denser. A useful refinement is to count characters in numbers and punctuation separately and assume each cluster of one to three characters may become its own token, so "1,234.56" (eight characters) plausibly lands around four to six tokens. Treat all of these as estimates and confirm with the real tokenizer when cost or context limits are tight.
Why do rarer or longer numbers split into more tokens than common ones?
BPE only creates a token for a substring that recurs often enough during training to justify a merge, so frequent numbers like years, round figures, and common constants get compact representations while unusual digit sequences stay fragmented. Longer numbers are also less likely to appear verbatim in training text, which means the tokenizer falls back on shorter substrings it already knows. The result is that token count tracks familiarity and frequency rather than raw length, and an obscure seven-digit number can cost more tokens than a familiar one of the same size.