How to Calculate Tokens Per Minute for Multi-Turn LLM Chats

To calculate tokens per minute (TPM) for a multi-turn chat, count the tokens in the full request each turn — every message's content plus per-message overhead for role and formatting — then multiply by turns per minute and by how many times the history is resent. Because most APIs are stateless and the client resends the entire conversation each turn, a conversation that grows linearly in content produces token usage that grows roughly quadratically over the session.

Why TPM Budgeting Breaks Down in Multi-Turn Chats

Most chat APIs are stateless: the server keeps no memory of your last request, so the client must send the whole conversation — system prompt, every prior user and assistant message, and the new user message — on each call. Token counters that work fine for single prompts quietly undercount chat traffic because they ignore two things: the per-message wrapper (role labels, separators, and other formatting the model consumes) and the fact that the same history is billed again on every turn. A rate-limit budget built from "prompt tokens × requests per minute" will underestimate true TPM and start throwing 429 (rate-limit) errors mid-conversation, often at the worst moment — deep into a long thread.

The compounding is the bigger trap. If each turn adds roughly a fixed amount of content, turn t sends about t turns' worth of tokens. Summed across a session, total tokens scale with the square of the turn count, so average cost per turn rises as the chat goes on. A one-minute window that looks safe at turn 2 can exceed the limit by turn 10. The sections below give a formula that captures this and a worked example, plus a checklist for converting TPM into a safe requests-per-minute (RPM) budget.

The TPM Formula: Message Overhead, Roles, and Conversation Growth

Define the pieces first. Tokens are the subword units a model counts as input; a rough English estimate is about 1 token per 0.75 words, though this varies by tokenizer and text. Message overhead is the fixed token cost per message for its wrapper — role tag, separators, and format markers — which is small (order of a few tokens per message) but multiplies across history. Conversation growth means the history length increases each turn. A simple model:

per-turn input tokens  T(t) = S + t * (M_avg + O)

where S = system prompt tokens (sent every turn), t = current turn number (history messages included), M_avg = average content tokens per message, and O = per-message overhead tokens. Output tokens are separate: R(t) = average response tokens per turn. Total tokens for a window of N turns:

total = sum_{t=1..N} [ T(t) + R(t) ]
      = N*S + (M_avg + O) * N*(N+1)/2 + N*R_avg

The N*(N+1)/2 term is the quadratic growth; if you need TPM over a one-minute window, divide total by the window length in minutes. For budgeting, compute the peak turn — the largest T(t) + R(t) you expect — and size your RPM as RPM_safe = TPM_limit / peak_per_turn_tokens, with headroom for variance in output length. Treat tokenizer-dependent values (per-message overhead, tokens-per-word ratio) as estimates: measure once with the API's token-counting endpoint or a tokenizer library, then apply a safety margin, commonly 10–20%.

Worked Example: Counting Tokens Across Five Turns

Use round, clearly stated assumptions so the arithmetic is checkable: system prompt S = 100 tokens; each user message M_u = 40; each assistant message M_a = 80; per-message overhead O = 5; response length R = 80 tokens. Turn t input = S + sum of history messages + overheads + new user message. Compute per turn:

• Turn 1: 100 + (40+5) = 145 input, +80 output = 225
• Turn 2: 100 + (40+5) + (80+5) + (40+5) = 275 input, +80 = 355
• Turn 3: 100 + 45 + 85 + 45 + 85 + 45 = 405 input, +80 = 485
• Turn 4: 100 + 45+85+45+85+45+85+45 = 535 input, +80 = 615
• Turn 5: 100 + 45+85+45+85+45+85+45+85+45 = 665 input, +80 = 745

Total across five turns = 225+355+485+615+745 = 2,425 tokens. Note turn 5 costs about 3.3× turn 1 despite identical new content, and the sum exceeds 5 × 225 = 1,125 by more than double — that gap is the compounding. If those five turns happen in one minute, TPM is about 2,425; if spread over five minutes, about 485 TPM average, but the peak minute still matters for rate limits.

A Checklist for Budgeting Rate Limits with the Token Counter

1. Measure, don't guess. Run one real request per turn shape through the API's token counter; record input and output tokens separately.
2. Add per-message overhead explicitly for the system prompt and every history message.
3. Model the peak turn, not the average. Budget against turn N_max you expect in a minute.
4. Apply the quadratic term for long sessions: sum N*(N+1)/2 when estimating a window.
5. Convert to RPM with margin: RPM_safe = TPM_limit / peak_per_turn_tokens × (1 − margin).
6. Cap output length with a max-tokens setting; unbounded responses are the most common TPM surprise.
7. Trim or summarize history to break the growth curve, and re-measure after changing the strategy.
8. Recheck after prompt or model changes, since tokenizers and overhead conventions differ.

Worked Example: Counting Tokens Across Five Turns

Assume a chat-format API that adds 4 tokens of wrapper overhead per message (role marker, separators, and formatting), and an approximate tokenizer rate of 1 token per 4 characters of English text. These are stated assumptions for illustration — check them against your provider's tokenizer, since wrapper overhead varies by model family.

Setup: a system prompt of 40 tokens, and each user turn is about 20 tokens, each assistant reply about 100 tokens. The API bills all input tokens sent per request plus the output tokens generated.

Per-request totals (input = system + full history + new user message; output = assistant reply):

Turn 1: input  = 40 + 4 + (20 + 4)                     = 72
        output = 100 + 4                               = 104
        billed = 176
Turn 2: input  = 40 + 4 + 24 + 104 + 24                = 196
        output = 104
        billed = 300
Turn 3: input  = 40 + 4 + 24 + 104 + 24 + 104 + 24     = 324
        output = 104
        billed = 428
Turn 4: input  = 324 + 104 + 24                        = 452
        output = 104
        billed = 556
Turn 5: input  = 452 + 104 + 24                        = 580
        output = 104
        billed = 684

Total billed across 5 turns = 176 + 300 + 428 + 556 + 684 = 2,144

Two things to notice. First, the same 40-token system prompt and the same 4-token wrappers are re-billed on every turn — overhead is small per message but repeats. Second, input grows linearly per turn while cumulative billed tokens grow roughly quadratically: the conversation is resent in full each time. Over 10 turns with these numbers, cumulative billed tokens would exceed 3,400 — an estimate, but the shape holds regardless of exact values. Budget for the sum, not the last request.

A Checklist for Budgeting Rate Limits with the Token Counter

Use this before sizing a TPM (tokens per minute) limit against expected traffic:

1. Pin down your tokenizer and wrapper overhead. Measure them with the provider's count-tokens endpoint or offline tokenizer on a real sample; don't guess from character counts.
2. Compute per-turn billed tokens, not per-message. Formula:

billed_turn = system + sum(history_messages + wrappers) + new_message + wrappers
              + max_output_tokens

3. Model conversation growth. If average history length after N turns is H and average new message is M, input per request ≈ system + H + M + (N+2) * wrapper. Multiply by expected requests per minute.
4. Convert TPM to a safe request budget:

requests_per_minute_safe = TPM_limit / (avg_input_tokens + max_output_tokens) * safety_factor

Use a safety factor of roughly 0.7–0.8 to absorb bursts and variance — an estimate, not a guarantee. 5. Cap history explicitly. Truncate, summarize, or window older turns so growth stays bounded rather than quadratic. 6. Set a max-output cap and count it against TPM even if the model usually emits less. 7. Re-measure after launch. Log actual input/output tokens per request and update the average; prompt and usage patterns drift. 8. Watch for 429 responses and treat them as a signal your safety factor or history cap is too loose.

Frequently Asked Questions

What counts toward tokens per minute in a chat completion request?

Tokens per minute (TPM) is the rate at which an LLM API accepts and generates tokens in a rolling 60-second window. Every request counts the tokens you send (the prompt, including all message content and structural overhead) plus the tokens the model returns (the completion). Prompt tokens and completion tokens are usually pooled against the same TPM limit, so a long conversation history consumes budget even before the model writes a single word. If your provider offers separate input/output limits, check which one your requests actually hit.

How do chat message wrappers and role tokens add overhead per turn?

Chat APIs don't send raw text; each message is serialized into a structured format with a role label (system, user, assistant) and delimiter tokens that mark where each message begins and ends. Those wrapper tokens — typically a small fixed number per message, plus a few extra tokens to prime the reply — are billed like any other token. The overhead per message is small (roughly 3–5 tokens per message is a reasonable estimate for common chat formats, though it varies by model and tokenizer), but it multiplies: a five-turn chat with two messages per turn adds overhead on every single message, every time the history is resent.

Why does resending full conversation history make TPM grow quadratically?

Chat completion APIs are stateless by default: to give the model context, you resend the entire conversation on each turn. If each turn adds roughly N tokens of new content, then turn 1 sends N, turn 2 sends 2N, turn 3 sends 3N, and so on — the total sent after T turns is N × (1 + 2 + … + T) = N × T × (T + 1) / 2, which grows with the square of the turn count. In TPM terms, the per-request cost rises linearly, but the cumulative token spend across a conversation rises quadratically. Budget for the later turns, not the first one.

How do I convert tokens per minute into a safe requests-per-minute budget?

Divide your TPM limit by the average total tokens per request (prompt plus completion), then apply a safety margin:

requests_per_minute ≈ (TPM_limit × safety_factor) / avg_tokens_per_request

Use a safety factor around 0.7–0.8 to absorb variance in completion length and bursty traffic, and compute avg_tokens_per_request from your longest typical conversation state, not the first turn. If requests vary widely in size, budget with the 90th-percentile request size instead of the mean.

Does the KV-cache reduce tokens billed per multi-turn request?

No. The KV-cache (key-value cache) is a performance optimization: it lets the model skip recomputing attention for tokens it has already processed, which reduces latency and compute cost. Billing, however, is based on the tokens in the request payload, so a cached prefix is still counted and billed as prompt tokens unless your provider explicitly offers a discounted "cached input" rate. Even with such a discount, the tokens still count toward TPM rate limits — caching changes the price, not the token count.