AI agent cost is dominated by the tokens an agent sends and receives across many model calls, not by a single prompt: a "simple" task can chain planning, tool use, retrieval, and reflection into dozens of calls, each re-sending growing context. The bill is therefore a function of token volume (input and output), how often you call the model, how much context you carry forward, and the price tier of the model you use. Everything else—embeddings, caching, retries—modulates those four levers.
What Drives AI Agent Cost
An AI agent is a program that uses a large language model (LLM) in a loop: it reads a goal, decides on an action (call a tool, search a document store, ask itself a question), observes the result, and repeats until done. Each iteration is a separate API call, and API pricing is metered in tokens—the chunks of text (roughly word fragments) a model reads and writes. Input tokens are what you send; output tokens are what the model generates. Providers typically charge different rates per million tokens for each direction, and output is usually several times more expensive than input.
The first cost driver is therefore call count. A single-turn chatbot answers in one call; an agent may make five to fifty, depending on how many steps it takes to finish. Multiply that by context size and you get the dominant term: a long-running agent re-sends its entire conversation history on every call, so cost grows faster than linearly with the number of steps.
The second driver is model tier. Frontier models cost more per token than small or mid-tier models, often by an order of magnitude. Agents that use a large model for every trivial step (parsing a tool result, routing a request) pay premium rates for work a cheaper model could do.
Third is tool and retrieval overhead: search queries, embedding calls, code execution, and third-party APIs each carry their own price or compute cost, and their outputs get appended to context, feeding back into token growth. Fourth are the unglamorous multipliers—retries on failed calls, verbose system prompts, and always-on instructions—that quietly inflate every request.
Tokens, Embeddings, and Context Growth
Tokens are produced by a tokenizer, which splits text into subword units. As a working estimate, English text runs roughly 1.3 to 1.5 tokens per word, so a 1,000-word document is about 1,300–1,500 tokens; code and non-English text tend to tokenize less efficiently. Pricing is quoted per million tokens, and your cost for one call is simply input tokens times the input rate plus output tokens times the output rate. For an agent, sum this over every call in a task.
Context growth is where agents diverge from chatbots. The context window is the total text (instructions, history, tool outputs, retrieved documents) the model can attend to in one call. Because the model is stateless—it remembers nothing between calls—the agent must resend the relevant history each time. If step 1 has 2,000 tokens of context and each step adds 500, step 10 sends roughly 6,500 input tokens. Over a 10-step task that is tens of thousands of input tokens for one user request, and the growth is superlinear as tasks lengthen. Trimming history, summarizing intermediate results, and capping tool output are the main defenses.
Embeddings are numeric vectors representing text meaning, used for semantic search and retrieval-augmented generation (RAG)—fetching relevant documents to insert into context. Embedding calls are usually cheap per token compared with generation, but they are not free, and you pay them at ingestion time (indexing your corpus) and at query time. The bigger cost is indirect: every retrieved chunk inserted into context is input tokens on every subsequent call in that loop. Retrieval is cheap; carrying retrieved text forward is not.
A concrete estimate: assume a mid-tier model at a few dollars per million input tokens and several times that for output, a 10-step task, average context of 5,000 tokens, and 300 output tokens per step. That is about 50,000 input and 3,000 output tokens per task—pennies per task at mid-tier rates, but dollars if you use a frontier model or the agent runs 30 steps. Treat these as illustrative assumptions, not quoted prices; plug in your provider's actual rates.
Inference, KV-Cache, and Repeated Calls
Inference is the act of running the model to produce output. During generation, the model computes attention over all previous tokens; a KV-cache (key-value cache) stores the intermediate attention states for tokens already processed so they don't need recomputation on each new token. This is primarily a latency and compute optimization within a single request: it makes long outputs faster and cheaper to serve, and providers may pass some of that efficiency through in pricing or caching discounts.
For agents, the practical consequence is prompt caching. Some providers let you mark a stable prefix—system prompt, tool definitions, long instructions—so repeated calls reuse cached computation and bill those tokens at a reduced rate. Agents are ideal candidates because they resend identical prefixes dozens of times. The savings apply only to the cached portion; new history and outputs are billed normally. Design your prompts so the stable part comes first and variable content last.
Repeated calls also create compounding failure costs. An agent that retries a malformed tool call, loops on a failed step, or re-plans unnecessarily pays for tokens it never needed. Guardrails—step limits, output validation, deterministic routing for simple subtasks—cut cost more reliably than prompt tweaks. And because statelessness forces resending, batch related subtasks into one call where quality allows, rather than many tiny calls that each carry the full prefix.
Estimating and Reducing Agent Costs
Estimate before you build by modeling the task: count expected steps, estimate tokens per step (instructions plus average history plus tool output), and multiply by your model's per-token rates, adding output tokens at their higher rate. Run a handful of real tasks, log actual token counts from API responses (most providers return usage metadata), and extrapolate to your expected volume. Include retrieval, embedding, and tool costs, then add a buffer—commonly 30–50%—for retries and variance, since agent step counts are heavy-tailed.
To reduce cost, work the levers in order of impact: route trivial steps to a small model and reserve frontier models for hard reasoning; cap and summarize history instead of resending everything; use prompt caching for stable prefixes; limit tool output length and retrieve fewer, better chunks; set hard step and token budgets per task; and cache or memoize repeated subqueries. Measure cost per completed task, not per call—an agent that costs more per call but finishes in fewer steps is often cheaper overall.
Inference, KV-Cache, and Repeated Calls
Inference is the act of running the model to produce output. For an agent, that's rarely one call — it's a loop of planning, tool use, and re-prompting, so you pay inference cost many times per task.
The key mechanic is the KV-cache (key-value cache): during generation, the model stores internal attention states for every token it has already processed so it doesn't recompute them. A cache hit means the provider reuses that stored work; a cache miss means it recomputes the prompt from scratch. Most providers price cache reads far below fresh input tokens, so hitting the cache saves money — but only if the prefix (the start of your prompt) stays identical between calls. Change the system prompt, reorder tool definitions, or inject a timestamp at the top, and you invalidate the cache.
Agents also re-send context: each step typically includes the growing transcript of prior steps. With N steps and roughly constant step size, you pay for the transcript repeatedly — cost grows faster than linearly with step count. Parallel or retried calls multiply this.
A simple way to see it:
cost per call ≈ input_tokens × in_price
+ cached_tokens × cache_price
+ output_tokens × out_price
task cost ≈ Σ (cost per call) over all steps, retries, and sub-agents
Estimating and Reducing Agent Costs
Estimate before deploying by instrumenting a few representative tasks: count input, cached, and output tokens per call, multiply by your provider's prices, then scale by expected steps and volume. Treat step count as a distribution, not a constant — retries and tool failures add calls.
Worked example (assumed prices, not real quotes): input $2 per million tokens, cached input $0.5 per million, output $8 per million. A task runs 5 steps. Each step sends 4,000 input tokens (3,000 cached, 1,000 fresh) and produces 300 output tokens.
Per step:
fresh input: 1,000 × $2/1e6 = $0.0020
cached input: 3,000 × $0.5/1e6 = $0.0015
output: 300 × $8/1e6 = $0.0024
step total = $0.0059
task total: 5 × $0.0059 = $0.0295
At 10,000 tasks/month that's about $295 — and doubling step count roughly doubles it.
Reduction checklist:
- Keep the prompt prefix stable so the KV-cache hits.
- Trim and summarize history instead of resending full transcripts.
- Cap max steps and add early-stop conditions.
- Route simple steps to cheaper models; reserve large models for planning.
- Batch or parallelize only where it doesn't multiply retries.
- Cache embeddings and retrieval results; don't re-embed unchanged documents.
- Log tokens per step to find the expensive ones.
Frequently Asked Questions
What makes up the cost of running an AI agent?
An agent's cost is the sum of every model call it makes, not just one prompt and one reply. Each step bills input tokens (the text sent to the model), output tokens (what it generates), and often retrieved context fetched through embeddings and a vector search. Multiply that by the number of reasoning steps, tool calls, and retries in a loop, and the per-task total can be many times a single chat message. Add any fixed costs like hosting a vector database or running a smaller model for classification, and you have the full picture.
How does KV-cache affect inference cost?
The KV-cache (key-value cache) stores the internal attention states for tokens the model has already processed, so a repeated prefix does not need to be recomputed. When a provider supports prompt caching, reused input is billed at a lower rate than fresh input, which cuts the cost of long system prompts or conversation histories that recur across calls. The catch is that the cache usually expires after a short idle window, so an agent that pauses between steps may lose the benefit and pay full price again.
Do embeddings and retrieval add to agent cost?
Yes, but usually less than generation. Embedding a document means one cheap model call per chunk, a one-time cost if you store the vectors; retrieval itself is a similarity search in a vector database, billed by hosting or query volume rather than per token. The bigger hidden cost is what retrieval injects into the prompt: every retrieved chunk becomes input tokens on that call, so fetching ten long passages can inflate input cost more than the embedding step ever did. Keep retrieved context tight and deduplicated.
How can I estimate LLM API costs before deployment?
Estimate from token counts rather than guesses: run representative tasks, log input and output tokens per call, and count calls per task. Then multiply by your provider's per-token rates (as an estimate, input often costs a fraction of output, commonly cited around one-quarter to one-fifth, though this varies widely by model). Add a buffer for retries and cache misses, and build the estimate as cost_per_task = calls * (input_tokens * input_rate + output_tokens * output_rate). Test on a small batch first, since agent loops amplify any per-call error.
Which curriculum stage best explains agent cost?
There is no universal curriculum, so treat this as guidance rather than a fixed rule. Cost is best explained once you understand tokenization, context windows, and the inference loop, which typically sit in an intermediate stage after basic prompting. The clearest mental model comes when caching and retrieval are introduced, because that is where repeated calls and context growth become visible as line items.