AI Token Counter

A language model never sees letters or words. It sees tokens: short chunks of text that were learned from data and then indexed in a fixed vocabulary. Some tokens are whole words (token), some are fragments (ing, ization), some are a single space plus a letter, and rare words are spelled out one piece at a time. Roughly, a token is to a model what a syllable is to a reader: the smallest thing it can pronounce, not the smallest thing that carries meaning.

• Tokens are not characters and not words. English averages around 4 to 5 characters per token, and that ratio changes with every language and every kind of text.
• Whitespace belongs to tokens. A leading space is usually part of the token that follows it, which is why " Hello" and "Hello" are different units.
• Punctuation is tokenised as well. The string Hello, world! is 4 tokens with the GPT-4o vocabulary — not one, and not thirteen.
• The vocabulary is fixed before you type anything. Your prompt is mapped onto it, never the other way around.

This tool runs a real byte-pair encoding (BPE) tokenizer in your browser with the gpt-tokenizer library. For OpenAI models the count is exact, because the library ships the same rank files the API uses: o200k_base for GPT-4o, o1 and o3-mini, and cl100k_base for GPT-4 Turbo and GPT-3.5 Turbo. For providers that do not publish their tokenizer — Anthropic, Google, Mistral, DeepSeek, Meta — the widget counts with the closest OpenAI encoder and labels the result approximate. On ordinary prose those estimates land within roughly a tenth of the true count, but treat them as a planning aid, never as an invoice.

Tokenizers are trained on a text mix still dominated by English, so the vocabulary is packed with English fragments and much thinner for other writing systems. A sentence that means exactly the same thing can therefore cost two to three times as many tokens in another language — and with it two to three times as much money. The table below is the same sentence in eight languages, counted with both OpenAI vocabularies, so the difference is visible rather than asserted.

Two lessons fall out of that table. First, the newer o200k_base vocabulary is markedly better outside English — the same Hindi sentence drops from 49 tokens to 20 — which is a concrete reason to move a multilingual product onto a GPT-4o-class model. Second, scripts without spaces behave differently again: the Chinese sentence is a single whitespace-delimited “word”, yet it is 14 tokens, because each character or character pair is its own unit.

Four figures sit next to the text you paste. Tokens is the one that controls billing and the context window. Characters is what most APIs and database columns limit. Words is what a human counts. Characters per token is the health check: for English prose, values between roughly 4 and 5.5 are normal, while a value near 2 tells you the text is code, non-Latin script or unusually punctuation-heavy — and that the prompt is more expensive than its length suggests.

A worked example: a 50-word English paragraph of 281 characters counts as 56 tokens, which is 5.0 characters per token and 1.12 tokens per word. Scale that to a 300-word support reply and you are near 340 input tokens. Add a 400-token answer and one request against a model priced at $0.15 per million input tokens costs well under a hundredth of a cent. Multiply by traffic and the number that looked negligible becomes a line item — which is exactly what the companion cost calculator does.

1. 1Paste the text you would really send, including system instructions and any retrieved context.
2. 2Compare the token count with the model’s context window, leaving room for the completion.
3. 3Check the tokens-per-word ratio in each of your languages to see where the budget actually goes.
4. 4Feed the token count into the cost calculator to turn it into money per month.

• Assuming four characters per token everywhere. That shortcut is English-specific. On Hindi it can be wrong by more than a factor of two.
• Counting words instead of tokens. A 1,000-word English document is usually around 1,100–1,300 tokens, and every embedded code sample or table pushes it higher.
• Forgetting the completion. The context window holds prompt and answer. A 100,000-token prompt on a 128,000-token model leaves very little room to reply.
• Ignoring message overhead. Chat APIs add a few special tokens per message for role and turn boundaries, so a conversation of thirty short messages costs more than the thirty messages counted separately.
• Trimming the wrong end of a prompt. Halving a long document saves tokens but often destroys the answer. Shrinking repeated few-shot examples usually saves the same tokens and changes nothing.

Tokenisation is the first step of the lifecycle, not an accounting detail. The count you see here decides four things downstream: whether the prompt fits the context window at all, what the request costs, how much GPU memory the KV cache needs while the model generates, and how faithfully a long retrieval pipeline behaves. Stage 5 of the course builds a byte-pair encoder from scratch so you can watch merges happen, Stage 11 shows the arithmetic that turns a context length into VRAM, and the companion cost calculator converts the number on this page into a monthly bill.