Counting Tokens in Code and JSON Before an LLM Call

Token counting is the practice of measuring how many units of text an LLM (large language model) will process before you send a request, because billing and context limits are both measured in tokens rather than characters or words. Code and JSON tokenize differently from prose because they are dense with punctuation, brackets, and predictable structural patterns that tokenizers split into many small pieces, plus repeated whitespace like indentation. You can get a solid estimate before sending anything by counting tokens on the text you control, then adding a fixed allowance for chat wrappers and formatting — the worked examples below show how.

Why Code and JSON Tokenize Differently from Prose

A tokenizer is the component that converts raw text into tokens — the integer IDs a model actually reads. Modern tokenizers use subword algorithms: common words like "the" or "function" usually become a single token, while rarer or novel strings get split into fragments. Prose is friendly to this scheme because it is mostly dictionary words separated by single spaces. Code and JSON are not: they are full of characters that rarely appear inside ordinary words — {, }, [, ], ", :, ,, (, ), =, ; — and tokenizers generally cannot merge those symbols with surrounding words. A single line like {"user_id": 42} can easily become six to ten tokens even though it reads as one tiny object.

Whitespace matters too. Prose has one space between words; formatted code and pretty-printed JSON have runs of spaces or tabs for indentation, and newlines. Many tokenizers encode a leading space as part of the following token, which works fine for prose but means indentation depth changes how lines split. Deeply nested JSON is a worst case: each level adds indentation characters plus its own brackets and quotes, so the structural markup can rival or exceed the tokens spent on actual data.

Finally, numbers and identifiers tokenize inefficiently. Long digit strings (IDs, timestamps, hashes) and camelCase or snake_case names have no dictionary entry, so they fragment. The practical consequence: the same information costs noticeably more tokens as compact code or JSON than as a plain-English sentence — often roughly 1.5 to 3 times more, as a reasoned estimate from these splitting behaviors rather than a measured benchmark.

Worked Example: Counting Tokens in a Code Snippet

Take this small function:

def add_tax(price, rate):
    return round(price * (1 + rate), 2)

Count it piece by piece. The keyword def is one token; the leading space plus add_tax likely splits into a space-plus-add token and a _tax token, so two; (price, rate): breaks into a parenthesis, price, a comma, rate, a closing parenthesis, and a colon — about six. The second line adds four spaces of indentation (one or two tokens depending on the tokenizer), return (one), round (one), (price (one or two), * and (1 and + and rate) (roughly four), , (one), 2) (one or two). Add a newline token at the end of each line. Total: roughly 20–26 tokens. The same logic in prose — "add tax to a price at a given rate and round to two decimals" — is about 15 tokens. The code is only slightly longer in characters but costs more per character because every symbol is its own piece.

Two lessons follow. First, minifying whitespace saves tokens but only modestly — indentation here is maybe two or three tokens out of twenty-plus — so do not sacrifice readability for a rounding error. Second, the expensive parts are the punctuation skeleton and the identifiers, not the blank space. If you must shrink a snippet, shorten variable names and strip comments before touching indentation. Always count with the exact tokenizer family your target model uses, since counts differ by a few tokens between tokenizer versions; treat any hand count as an estimate within about ±10%.

Worked Example: JSON Payloads, Chat Overhead, and Formatting Tokens

Consider the same fact twice. As prose: "Order 1042 shipped to Berlin on March 3." That is roughly 10–12 tokens. As JSON:

{"order_id": 1042, "destination": "Berlin", "ship_date": "2024-03-03"}

Count the structural tokens: { (1), "order_id" (2–3 with the quote marks), : (1), 1042 (1–2), , (1), "destination" (2–3), : (1), "Berlin" (2–3), , (1), "ship_date" (2–3), : (1), "2024-03-03" (the date string often splits into three or four pieces plus quotes, so 4–6), } (1). Total: roughly 20–28 tokens — about double the prose version, consistent with the estimate above. Pretty-printing with newlines and indentation adds a few more.

Chat overhead is separate from your content. Chat-format APIs wrap each message in structural markers — role labels such as "user" and "assistant", delimiter tokens between messages, and a priming suffix that tells the model to start replying. These wrappers are not visible in your text but are counted. As a working estimate, budget roughly 3–5 tokens per message plus about 3 tokens for the reply priming; a single-turn request therefore carries on the order of 6–10 tokens of overhead, and a ten-message history adds tens of tokens before any content is counted. Exact wrapper sizes vary by model family, so verify once with a minimal request and reuse that constant.

Pre-Flight Checklist to Estimate Request Cost

Before sending, run this sequence. (1) Identify the tokenizer family for your target model and use its official counting tool or library — do not count characters and divide by four; that shortcut is fine for English prose but undercounts code and JSON substantially. (2) Count each component separately: system prompt, conversation history, the new user message, and any pasted code or JSON. (3) Add the chat overhead constant you measured. (4) Add expected output tokens — set a max-output limit and budget for it, since output is typically priced higher than input. (5) Compute cost with the current published per-token prices for your model, and multiply by expected daily request volume. Keep a margin of roughly 10–20% for tokenizer drift and formatting surprises. If the estimate is close to the context limit, trim history or minify payloads first; if it is close to a budget line, shorten the system prompt, which is paid on every single request.

FAQ

Why does a JSON payload use more tokens than the same data in plain prose? Because JSON spends tokens on structure — braces, quotes, colons, commas — and on keys repeated in every object, and none of those pieces merge with neighboring words in a tokenizer. The data values are a minority of the token count; the markup is the rest. Prose carries the same facts with ordinary words and single spaces, which tokenizers compress well.

How much token overhead do chat message wrappers add per request? As an estimate, plan for about 3–5 tokens per message plus roughly 3 tokens of reply priming. That means a single-turn request costs on the order of 6–10 extra tokens, and longer histories scale linearly. Measure it once on your target model by comparing a minimal request against your own count of the visible text.

Can I estimate LLM API cost before sending the request? Yes, and you should. Count tokens for every component with the model's official tokenizer, add the measured chat overhead, budget the output at your max-output setting, and multiply by current per-token prices. The result is an estimate accurate to within a small margin, which is enough for budgeting and for deciding whether to trim a payload.

Does indentation and whitespace in code affect token counts? Yes, but modestly. Each line's leading whitespace becomes one or two tokens depending on depth and tokenizer, and newlines count too. In a typical snippet, indentation is a small single-digit percentage of the total; punctuation and identifier splitting cost far more. Minify only when you need the margin, and prefer shortening names and removing comments first.

Why do numbers and punctuation-heavy strings tokenize inefficiently? Tokenizers are trained mostly on natural text, so frequent words get whole-token entries while digit runs, hashes, dates, and symbol clusters do not. Those strings fall back to character-level or short fragments, so a 10-digit ID or an ISO date can cost several tokens apiece. Expect identifiers, timestamps, and dense punctuation to be the most expensive characters in any payload.

Worked Example: JSON Payloads, Chat Overhead, and Formatting Tokens

A chat request is not just your content. Most APIs wrap it in a message structure, and the wrapper itself costs tokens. Take a minimal request that sends one user message containing a small JSON object:

{"messages":[{"role":"user","content":"{\"order_id\":1042,\"status\":\"shipped\"}"}]}

Model the tokenizer the way subword tokenizers (algorithms like BPE, byte-pair encoding, that split text into frequent chunks) usually behave: common words like order, status, user are one token each, while punctuation and structural characters ({, }, ":, \",) each tend to become their own token or merge awkwardly with neighbors.

Rough estimate for the visible message content {"order_id":1042,"status":"shipped"} (about 40 characters):

• Braces, quotes, colons, commas: ~10 structural characters → ~10 tokens
• order_id splits into order + _id → ~2–3 tokens
• 1042 → ~1–2 tokens (short numbers often merge; long ones don't)
• status → 1 token; shipped → 1–2 tokens

That's ~16–19 tokens for content most people would call "two fields." Now add the chat wrapper. Every message typically carries role markers and separators (e.g., <|user|>, newline, end-of-turn token). Assume ~4 tokens of wrapper per message plus a fixed ~3-token priming cost per request. Total estimate:

content tokens        ~17
message wrapper       ~4
request priming       ~3
--------------------------
estimated total       ~24 tokens

The payload's "real data" (an order ID and a status string) is a fraction of that. If you send 10 such messages in one request, multiply content and wrappers (~210) but pay priming once (~213). The lesson: overhead is roughly constant per request, so batching many small items into one request amortizes it.

Pre-Flight Checklist to Estimate Request Cost

1. Count raw content first. Use a tokenizer library for your target model, or a rough rule: ~4 characters per token for English prose, but closer to 2–3 for code and JSON because of punctuation density.
2. Add wrapper overhead per message. Assume ~3–5 tokens per message for role tags and separators; calibration against one real response beats guessing.
3. Add per-request priming. A small fixed cost (~2–5 tokens) for system framing; negligible for large requests, meaningful for tiny ones.
4. Count formatting separately. Indentation, newlines, and repeated quotes each consume tokens; minified JSON is usually cheaper than pretty-printed.
5. Multiply by expected output length. Output tokens are typically priced higher than input; estimate output as input_tokens * expected_ratio, with expected_ratio around 0.2–1.0 depending on the task.
6. Compute cost with a formula:

cost = (input_tokens / 1e6) * input_price
     + (output_tokens / 1e6) * output_price

7. Sanity-check with one real call. Send the actual payload once, read the reported usage numbers, and use the ratio between your estimate and the real count to correct future estimates.

Frequently Asked Questions

Why does a JSON payload use more tokens than the same data in plain prose?

Tokenizers split text into chunks that match patterns they saw during training, and JSON introduces punctuation and structure that rarely appears in natural language: braces, brackets, colons, commas, and quoted keys. Each of these characters often becomes its own token or fragments a nearby token, so the same information costs more tokens when wrapped in JSON syntax. A plain sentence like user Alice age 30 may tokenize compactly, while {"user": "Alice", "age": 30} adds structural tokens around every value. The shorter the values, the worse the ratio, because overhead is roughly fixed per key-value pair.

How much token overhead do chat message wrappers add per request?

Chat APIs don't send your raw text; they wrap each message in a structured format with role labels, separators, and control tokens, and they prepend a hidden system-level preamble. Exact counts depend on the model family and are not always published, but a reasonable estimate is a small fixed cost per message (roughly 3–5 tokens) plus a few tokens per request for the reply priming. With many short messages, this overhead can add up; with a few long messages, it is negligible. Check your provider's documentation for the specific model rather than assuming a universal number.

Can I estimate LLM API cost before sending the request?

Yes. Cost is input tokens times the input price plus expected output tokens times the output price, and both token counts can be computed ahead of time with an offline tokenizer library:

estimated_cost = (input_tokens * input_price + expected_output_tokens * output_price) / 1,000,000

Count your serialized payload (including wrappers) with the tokenizer matching your model, estimate output length from your prompt's instructions, and compare against your budget. Allow a margin of roughly 10–20% for estimation error and retries; treat any price you plug in as something you verify from the provider's current pricing page.

Does indentation and whitespace in code affect token counts?

Yes, and inconsistently. Tokenizers merge common patterns like four spaces, a newline plus indentation, or return, so uniform formatting is often cheaper than it looks, but deep or irregular indentation creates fragments that tokenize separately. Tabs versus spaces matter too, since a tab may split differently than a run of spaces. For token estimation, always count the exact string you will send: minified or reformatted code can change the count by a noticeable margin, so measure rather than assume.

Why do numbers and punctuation-heavy strings tokenize inefficiently?

Tokenizers learn frequent substrings from training text, so they have compact tokens for common words but not for arbitrary digit sequences, UUIDs, hashes, or dense punctuation. A long number may be split into groups of digits or single digits, and mixed symbols like {}[]:," fragment into individual tokens. This is why logs, base64 blobs, and minified JSON with numbers or punctuation, which are rare in prose, tokenize poorly. If precision matters, count those strings directly with the tokenizer instead of estimating from character length.