Best Data Preprocessing Tools for LLM Pipelines: A Stage-by-Stage Guide

Ask ten engineers which tool is best for preparing LLM data and you'll get ten answers, and most of them will be right for a different problem. One person is cleaning a few thousand support tickets for a fine-tune. Another is ingesting a mountain of PDFs for a retrieval system. A third is filtering terabytes of web text for pretraining. They all say "preprocessing," but they need different tools.

So instead of one ranked list, this guide goes through the pipeline stage by stage. For each stage you'll see what the job is, what tools people reach for, and how to choose based on how much data you have. It's also honest about the part most lists skip: the tool matters less than the order you do things in, and whether you look at your data.

First, which pipeline are you building?

Two situations get lumped together, and they shouldn't be.

Training and fine-tuning data. You're producing text or conversation examples to train a model. Quality, duplicates, and leakage into your evaluation set are the big concerns. Volume ranges from a few hundred examples to trillions of tokens.

Retrieval (RAG) ingestion. You're turning documents into searchable chunks. Parsing quality and chunking dominate. Duplicates matter less than getting tables, headings and scanned pages out of files intact.

Many of the stages overlap, and some tools serve both. But if you know which one you're building, half of the tool list disappears. If you're not sure whether you need to train at all, our guide to is a better starting point than any tool comparison.

The stages, in the order they usually happen

1. Extract the text from its source
2. Clean and normalize it
3. Identify the language
4. Filter low-quality documents
5. Remove duplicates
6. Remove or mask sensitive data
7. Format, split and tokenize
8. Validate by actually reading samples

Order matters. For example, normalizing text before deduplication makes near-duplicates easier to spot, because small encoding differences no longer hide them. NVIDIA's own write-up of its curation pipeline makes the same point about cleaning helping dedup recall.

Stage 1: Extraction

This is where raw files and web pages become text. It's also where many pipelines quietly lose quality: a PDF parsed badly gives you scrambled columns and missing tables, and nothing downstream can repair that.

• Web pages: trafilatura is a popular choice for pulling the main content out of HTML while dropping menus and ads.
• PDFs, Word and slides: Docling and Unstructured both parse mixed document formats and try to keep structure such as headings, lists and tables.
• Scanned documents: you'll need OCR, which is slower and more error-prone. Spot-check the output.

How to choose: run two tools on twenty representative files and compare the output side by side. Ten minutes of reading beats any benchmark, because your documents are the ones that matter.

Stage 2: Cleaning and normalization

Here you fix encoding damage, strange whitespace, stray HTML, and boilerplate such as cookie banners and footers.

• ftfy repairs broken Unicode (the "mojibake" you get when text was decoded wrongly).
• Regex and small scripts handle most of the rest. Don't underestimate them.
• Pandas or Polars are enough for tables of text up to a few million rows. Polars is generally faster on large files.

Keep an eye on being too aggressive. Stripping all punctuation, or flattening everything to lowercase, can damage code, formulas and anything where formatting carries meaning.

Stage 3: Language identification

If your data is multilingual, or supposed to be monolingual, you need to know what you actually have. fastText's language identification models are the usual lightweight option. They're fast, and good enough to filter or route documents.

A caveat that matters for this site's readers: short texts and mixed-language text are hard to classify, and a wrong label can throw out good data. Use a confidence threshold rather than a hard yes or no. And remember that non-English text breaks into more tokens, which affects both cost and context. We explain that in .

Stage 4: Quality filtering

Not every document deserves to be in your dataset. Quality filters drop pages that are mostly boilerplate, very short, full of symbols, or made of repeated lines.

• Heuristic filters are rules: length limits, ratio of alphabetic characters, repeated-line fraction, average word length. They're transparent and cheap.
• Model-based filters use a small classifier to score quality against a reference set. They catch more, but you're trusting a model's idea of "good."

Frameworks such as Hugging Face's datatrove and NVIDIA's NeMo Curator ship ready-made filters, so you don't write everything yourself. Start with heuristics, read what they remove, and only add a model-based filter if the heuristics clearly miss things.

Stage 5: Deduplication

Duplicates waste training compute and can cause a model to memorize text. There are three levels:

• Exact: hash each document and drop identical ones. Cheap and safe.
• Near-duplicate (fuzzy): MinHash with locality-sensitive hashing finds documents that are mostly the same. This is the workhorse for web-scale data. Our lesson walks through how it works.
• Semantic: embeddings find documents that mean the same thing in different words. It's the most expensive and the most debatable. NeMo Curator offers semantic deduplication among its modules.

For tooling, datasketch is a small Python library for MinHash and LSH, good for experiments and mid-size jobs. datatrove and NeMo Curator handle it at much larger scale, and NVIDIA describes GPU-accelerated deduplication as a way to cut the time and cost compared with CPU-only runs.

Here's what a tiny near-duplicate filter looks like with datasketch:

from datasketch import MinHash, MinHashLSH

def shingles(text, n=5):
    words = text.lower().split()
    return {" ".join(words[i:i + n]) for i in range(max(1, len(words) - n + 1))}

def minhash(text, num_perm=128):
    m = MinHash(num_perm=num_perm)
    for s in shingles(text):
        m.update(s.encode("utf-8"))
    return m

lsh = MinHashLSH(threshold=0.8, num_perm=128)
kept = []
for i, doc in enumerate(docs):
    m = minhash(doc)
    if not lsh.query(m):          # no near-duplicate seen so far
        lsh.insert(str(i), m)
        kept.append(doc)

It's not production code, but it's enough to learn what a threshold of 0.8 feels like on your own data. Try 0.7 and 0.9 and read what changes.

Stage 6: Sensitive data

Names, emails, phone numbers and keys sometimes hide in text. Before training or indexing, find them.

• Microsoft Presidio is a common open-source choice for detecting and masking personal data.
• NeMo Curator includes PII redaction among its modules.
• Regex rules are still worth adding for secrets like API keys.

No detector is perfect. Treat it as risk reduction, not a guarantee, and decide up front what your policy is for data you can't clear. Do this before chunking and indexing, not after, because it's much harder to find sensitive text once it's scattered across thousands of chunks.

Stage 7: Formatting, splitting and tokenizing

For fine-tuning, you turn cleaned text into the exact structure the model expects, usually chat-style messages with the right template. A mismatch here causes subtle quality problems that look like "the model is bad." Then you count tokens to estimate cost and fit lengths. The and our guide to help here. If you're wondering how much data to prepare, see .

For RAG, you chunk. Chunk size and overlap change both retrieval quality and embedding cost, which we cover in .

One rule for both: split train and evaluation data before you deduplicate across them, or at least deduplicate your training set against your evaluation set. Otherwise test examples leak into training, and your scores look better than reality. That's one reason .

Stage 8: Validation (the stage people skip)

Open a random sample of 50 documents from your final dataset and read them. It sounds too simple to matter, and it catches more problems than any tool: leftover navigation text, truncated files, a language you didn't expect, labels that don't match.

If you have a labeling interface such as Argilla or Label Studio, use it to review samples with a team. If you don't, a spreadsheet works fine.

Picking tools by scale

Don't start at the bottom row. Many teams build a distributed pipeline for data that fits in memory on a laptop. Start small, prove the steps work, and scale only when the job is genuinely slow.

Mistakes that cost the most

• Filtering too hard. Aggressive filters can throw away the unusual, valuable examples. Review what gets removed, not just what's kept.
• Deduplicating after the split. See above: leakage.
• Parsing documents badly and never checking. Everything downstream inherits the damage.
• Running PII removal late. Do it before chunking.
• No record of what you did. Keep your configuration and the order of steps. When results change, you'll need to know why.
• Treating the pipeline as one-off. Data keeps arriving. A pipeline you can rerun is worth more than a perfect one-time cleanup.

Quick answers

Which single tool should I start with? For a small project, none: a Python script and a parser. For anything bigger, try datatrove or NeMo Curator on a sample and see which fits your stack.

Do I need a GPU for preprocessing? Usually not. GPUs help with very large deduplication and with model-based filters or embeddings. Most cleaning is CPU work.

How much should I deduplicate? Enough to remove obvious repeats without deleting legitimate similar documents. Pick a threshold, read examples at the boundary, and adjust.

Is preprocessing different for RAG? Yes. Parsing quality and chunking matter most, and the goal is findable passages rather than a training corpus.

Where does this fit in the bigger picture? Data preparation is the first stage of the whole lifecycle. Our shows how it connects to training and inference, and the covers what comes after.

The short version

Preprocessing is a sequence, not a tool: extract, clean, identify language, filter, deduplicate, protect sensitive data, format, and then read your samples. Pick the lightest tools that work at your scale, check the output with your own eyes, and keep the steps repeatable. If the data is clean, almost any model you train on it will behave better.