Chunking is the single decision that most directly sets both the retrieval quality and the embedding bill of a RAG (retrieval-augmented generation) pipeline, because every chunk becomes one vector you pay to compute and one candidate the retriever can return. Smaller chunks sharpen retrieval precision but multiply the number of embeddings, while larger chunks cut the vector count but blur meaning and waste context window space. The practical goal is the smallest chunk size that still carries a self-contained idea, with overlap and metadata tuned to your documents rather than copied from a default.
Why Chunking Drives RAG Quality and Cost
A chunk is the unit of text you send to an embedding model, which converts it into a vector — a fixed-length list of numbers representing meaning — stored in a vector database for similarity search. At query time, the same model embeds the question, and the system retrieves the chunks whose vectors are closest. This means chunk boundaries decide what the retriever can possibly find: if an answer is split across two chunks and neither half makes sense alone, no amount of model quality will recover it.
Cost scales linearly with chunk count, not document count. Embedding cost is roughly:
cost = (total_tokens / 1,000,000) * price_per_million_tokens
total_tokens ≈ (document_tokens + overlap_tokens) * duplication_factor
Halving chunk size roughly doubles the number of vectors and therefore the embedding spend, and it also increases storage and index memory. Quality moves in the opposite direction up to a point: smaller chunks tend to produce more focused vectors and cleaner matches, until chunks get so short that they lose the context needed to be meaningful. That tension — precision versus bill — is the whole game.
There is a second, quieter cost: retrieval noise. Overly large chunks dilute their own vector with multiple topics, so a chunk about three subjects matches a query about one of them only weakly. You then compensate by retrieving more chunks (a larger top-k), which inflates the tokens you later send to the generation model. Chunking decisions thus propagate into inference cost, not just indexing cost.
Fixed-Size, Recursive, and Semantic Chunking Compared
Fixed-size chunking splits text every N tokens or characters, optionally with overlap. It is the simplest and fastest approach, trivially parallelizable, and completely blind to sentence and section structure — it will cut mid-sentence and mid-table. It works acceptably on homogeneous prose and poorly on structured documents like contracts, manuals, or code.
Recursive chunking tries a hierarchy of separators in order — paragraph breaks first, then sentences, then words — splitting only as aggressively as needed to hit the size limit. In practice this is the sensible default: it respects natural boundaries most of the time, keeps chunk sizes near the target, and costs almost nothing computationally. Most production pipelines that don't have a specific reason to do otherwise land here.
Semantic chunking embeds sentences or small passages, then cuts where consecutive embeddings diverge sharply, on the theory that a topic shift marks a natural boundary. It can produce notably coherent chunks on heterogeneous text, but it requires an extra embedding pass over small units before you even build the index — so you pay embedding cost twice, once for boundary detection and once for the final chunks — and it adds latency and tuning burden. It is worth the complexity when your documents mix topics within pages; it is overkill for clean, well-sectioned content.
Overlap, Metadata, and Context Window Tradeoffs
Overlap means repeating the last N tokens (the word-piece units models read and bill) of one chunk at the start of the next, so a sentence split across a boundary still appears intact somewhere. Overlap buys recall at a direct cost: every overlapped token is embedded twice. If you use 200-token overlap on 800-token chunks, roughly 25% of your embeddings are duplicates. That repeats in vector storage too, and duplicate near-identical vectors can crowd out diverse results in top-k retrieval.
Metadata is cheap to store and free to embed if you keep it out of the embedded text, but it costs tokens if you prepend it. A 40-token header (source, section, date) on a 500-token chunk is an 8% embedding increase. Use it when retrieval needs filtering or the header genuinely disambiguates meaning; otherwise attach metadata as a separate field in your vector store and filter on it.
The context window is the maximum tokens a model accepts in one call. Chunks must fit it alongside any header you prepend. Smaller chunks give sharper retrieval but strip surrounding context the generator needs; larger chunks preserve context but dilute the embedding's topical focus. There is no universal answer — tune against your own queries.
Estimating Embedding Cost with the LLM API Cost Calculator
Embedding cost is proportional to total tokens sent, so chunking decisions set your bill before you write a single query. The formula:
total_tokens = (corpus_tokens / (chunk_size - overlap)) * chunk_size
embedding_cost = total_tokens / 1e6 * price_per_million_tokens
Worked example, assuming 10,000,000 corpus tokens, 800-token chunks, 200-token overlap, and an illustrative price of $0.02 per million tokens:
effective_step = 800 - 200 = 600 tokens of new text per chunk
chunks = 10,000,000 / 600 ≈ 16,667
embedded_tokens = 16,667 * 800 ≈ 13,333,600
cost = 13.33 * $0.02 ≈ $0.27
Drop the overlap to zero and you embed 10M tokens instead ($0.20); push overlap to 400 and it climbs to 20M ($0.40). Same corpus, roughly 2x cost swing from overlap alone. Substitute your provider's real rate and rerun before indexing.
Checklist before you index:
- Count corpus tokens with the same tokenizer the embedding model uses.
- Fix chunk size and overlap; compute the formula above.
- Add header tokens if you prepend metadata.
- Multiply by re-indexing frequency (every corpus change re-embeds).
- Sanity-check chunk count against your vector store's limits.
Frequently Asked Questions
What is the best chunk size for RAG?
There is no universal best size; the right chunk size depends on your documents and the questions users ask. A practical starting point is a few hundred tokens per chunk, which is large enough to hold a complete idea but small enough that its embedding stays focused on one topic. Test a couple of sizes on real queries and measure retrieval quality rather than picking a number in advance, since the best size for dense technical text often differs from the best size for conversational content.
How does chunk overlap affect embedding cost?
Overlap means repeating some text at the boundaries of adjacent chunks so no sentence gets split without context. Because overlapped text appears in more than one chunk, it is embedded more than once, so cost rises roughly in proportion to the overlap fraction. For example, a 20 percent overlap means you embed roughly 20 percent more tokens than the raw corpus contains, so estimate cost from the total chunked token count, not the original document length.
Should I use semantic chunking or fixed-size chunking?
Fixed-size chunking splits text at a set token count and is simple, fast, and predictable in cost. Semantic chunking uses embeddings to detect topic shifts and cut at natural boundaries, which can improve retrieval on messy or long-form text but adds preprocessing cost and complexity. Start with fixed-size or recursive chunking, and move to semantic chunking only if retrieval quality on your evaluation queries shows it is worth the extra effort.
How do I estimate embedding costs before indexing a corpus?
Count the total tokens in your corpus after chunking, including any overlap, then multiply by the embedding model's per-token price. Use a cost calculator to convert chunk counts and average chunk length into a total, and add a margin for re-indexing when documents change. Because embedding prices are usually quoted per million tokens, the formula is:
cost = (total_chunked_tokens / 1_000_000) * price_per_million_tokens
Does chunking affect retrieval accuracy in vector search?
Yes. Chunks that are too large dilute their embedding across several topics, so similarity search matches them weakly; chunks that are too small lose the context needed to answer. Good chunking keeps each chunk topically coherent with enough surrounding text to be useful, and overlap helps preserve meaning at boundaries. Retrieval accuracy is therefore a direct function of how well chunk boundaries align with the units of meaning in your documents.