Why do small models keep getting better? A compact model released today often matches what a model many times larger could do a couple of years ago. Better data and architecture explain part of it. Another big part is a technique called distillation: using a large, capable model to teach a smaller one.

If you have ever wondered how "distilled" models are made, whether you can do it yourself, or why a distilled model sometimes behaves so much like the one it learned from, this guide walks through the idea, the main variants and a practical recipe.

The core idea: a teacher and a student

Distillation involves two models:

The teacher: a large, expensive, high-quality model.
The student: a smaller, cheaper model you want to be good at the same job.

Instead of training the student only on raw data, you train it to imitate the teacher's behavior. The teacher has already figured out useful patterns, and the student gets a shortcut to them. The result is a model that is faster and cheaper to run, with much of the teacher's quality on the target task.

Why imitating a teacher beats learning from labels alone

The classic paper on this topic, by Hinton and colleagues in 2015, made a simple observation. When a classifier looks at a photo of a cat, a correct label says only "cat." A trained teacher says more: perhaps 90 percent cat, 8 percent dog, and a tiny fraction for "car." Those small probabilities are informative, because they reveal that cats look more like dogs than like cars. The authors called this hidden structure "dark knowledge."

Training a student on the teacher's full probability distribution, called soft targets, passes along that structure. Each example carries far more information than a single hard label, so the student learns faster and generalizes better than it would from labels alone. A temperature setting is often used to soften the teacher's distribution, so the small probabilities are not drowned out by the top choice. (If temperature is new to you, see .)

Two flavors of distillation for language models

For language models, two approaches are common, and they differ in what you need from the teacher.

	Logit distillation	Data (sequence-level) distillation
What the student learns from	The teacher's probability for every possible next token	Text the teacher generated
What you need	Access to the teacher's internal outputs	Only the ability to call the teacher
Signal richness	High	Lower, but still useful
Typical user	Labs that own both models	Anyone with API or model access

Logit distillation trains the student to match the teacher's whole next-token distribution at each position. It is the richest signal, but it requires access to the teacher's probabilities, and the two models generally need to share a vocabulary.

Data distillation is simpler and far more widespread. You send prompts to the teacher, collect its answers, and fine-tune the student on those prompt-and-answer pairs using ordinary supervised fine-tuning. Nothing special is needed from the teacher beyond its text output. This is what most teams mean when they talk about distilling a model from an API.

Reasoning distillation

A recent and influential use of data distillation involves reasoning. Large reasoning models produce long step-by-step chains of thought before answering. Smaller models can be trained on those traces, and the student learns to imitate the reasoning style, often improving a lot on math and coding tasks compared with the same small model before training. DeepSeek, for example, released a family of smaller models built on Qwen and Llama bases that were fine-tuned on reasoning outputs from its larger model. For background on why such models spend so many tokens, see .

Well-known examples

DistilBERT was one of the first widely used distilled language models. It was reported to be about 40 percent smaller and 60 percent faster than BERT, while keeping roughly 97 percent of its language understanding performance on a standard benchmark suite.
Gemma 2 is reported by Google to have used distillation from a larger model as part of training its smaller variants, which is a sign that the technique is now part of mainstream pretraining recipes, not only a niche post-processing step.
Distilled reasoning models from the open-weight community show the data-distillation route in action.

What distillation can do, and what it cannot

Distillation is powerful, but it has clear limits.

It can: shrink cost and latency for a defined task; transfer style, format and reasoning patterns; and give a small model a strong boost compared with training on raw labels.

It cannot:

Exceed the teacher. The student learns what the teacher shows it. If the teacher is wrong or biased, the student inherits those errors, and sometimes amplifies them.
Transfer everything. A small model has less capacity. It may copy the teacher's answers on the tasks you trained on, yet fall apart on questions outside that distribution. Narrow distillation gives narrow competence.
Replace broad knowledge. Facts and rare skills that need many parameters to store do not fit in a small student just because you wish they would.
Fix hallucinations. A student can reproduce the teacher's confident style without the teacher's reliability. See .

People mix these terms up all the time, so here is a clean comparison.

Technique	What it does	Changes model size?
Distillation	Trains a smaller model to imitate a bigger one	Yes, you pick a smaller student
Fine-tuning	Adapts one model to your data or behavior	No
Quantization	Stores the same model with fewer bits	No parameters removed, smaller files
Pruning	Removes weights or structures from a model	Yes, but the model is derived from itself

Distillation and fine-tuning often work together, since data distillation is simply supervised fine-tuning on teacher outputs. Quantization can then be applied on top of the student for an additional saving; see .

A practical recipe for a narrow task

Suppose you run a support-ticket classifier or an extraction pipeline on a large model, and the bill is climbing. Here is a straightforward path.

Define the task precisely. Write the exact input format and the exact output you expect. Narrow, well-defined tasks distill best.
Collect real inputs. Use real examples from your traffic, not only synthetic ones, so the student sees the distribution it will face. Remove private data first.
Generate teacher outputs. Run the inputs through the strongest model you can afford, with a carefully tested prompt.
Filter aggressively. Remove broken, inconsistent or low-quality outputs. Use programmatic checks and spot-check by hand. A smaller clean dataset beats a larger noisy one. How much you need depends on the task; see .
Train the student. Fine-tune a small base model, typically with LoRA or QLoRA to keep hardware needs modest. Stage 9 of the course, , explains the method.
Evaluate on held-out real data. Compare the student against the teacher and against your current setup, on examples neither model has trained on. Track quality, cost and latency together.
Iterate where it fails. Look at the mistakes, add examples that target them, and retrain.

A cost sketch with illustrative numbers

Costs are easy to hand-wave, so here is a small hypothetical. Suppose the large model costs 10 dollars per million tokens, and you generate 20,000 training examples of about 1,000 tokens each. That is 20 million tokens, or about 200 dollars of teacher usage.

Now suppose your production traffic is 50 million tokens per month. At 10 dollars per million, the large model costs 500 dollars a month. A tuned small model that costs, say, 0.50 dollars per million tokens would cost about 25 dollars a month, saving roughly 475 dollars. In this made-up case, the data generation cost is recovered in under a month, before counting training compute and your engineering time. Use the and the to plug in real prices. The savings are large only if volume is high and the task is narrow enough for the student to match the teacher.

Terms of service and licensing

Before distilling from a hosted model, read its terms. Some providers prohibit using their outputs to train competing models, and open-weight models come with their own licenses, some of which have conditions on derivative works and on using outputs for training. This is not a technicality: it can decide whether your product can ship. When in doubt, ask the provider or choose a teacher whose license clearly allows it.

Common pitfalls

Training on easy examples only. The student looks great on your test set and fails on real edge cases. Include hard and rare inputs.
No held-out set. Without one, you cannot tell whether the student learned the task or memorized the examples.
Copying the teacher's mistakes. Filter outputs, and where possible check them against ground truth.
A student that is too small. If quality plateaus no matter how much data you add, the model may lack capacity. Try the next size up.
Forgetting the teacher's prompt. The teacher's results depend on its prompt. Keep it versioned, because it defines the behavior you are distilling.

Key takeaways

Distillation trains a small student to imitate a large teacher, passing on more information than raw labels carry.
For LLMs, most teams use data distillation: fine-tune a small model on teacher-generated outputs.
The student cannot beat the teacher and tends to be good only where it was trained.
It pays off for high-volume, well-defined tasks, and rarely for open-ended general chat.
Check licenses, filter your data and evaluate on real held-out examples.

Done well, distillation is one of the most effective ways to turn an expensive capability into a cheap, fast and dependable one.