AI 101 — What is knowledge distillation?

Knowledge distillation is a way of training a small "student" AI model to imitate a big "teacher" model — so the student picks up much of the teacher's skill while being far smaller, faster, and cheaper to run.

If you have read tech news lately, you have seen this word doing heavy lifting without ever getting explained. A ByteDance founder reportedly told staff to stop using distillation even though rivals rely on it. Anthropic has previously accused several Chinese labs of using thousands of fake accounts to harvest Claude outputs for exactly this purpose. OpenAI, meanwhile, sells distillation openly as a feature of its developer platform. One technique sits behind all three stories, and understanding it explains a quiet fault line in the AI industry: who gets to learn from whom.

Why it matters right now

Big models are brilliant and expensive. They can take entire data centers to serve, and every query costs real money. Small models are cheap and fast, but trained on their own they lag well behind. Distillation is the bridge: it lets a small model inherit a disproportionate share of a big model's ability.

The most famous example remains DistilBERT, a 2019 project at Hugging Face that shrank the then-standard BERT language model by about 40 percent while keeping roughly 97 percent of its performance on language benchmarks, and running about 60 percent faster. That trade — nearly all the smarts for a fraction of the cost — is why distillation spread everywhere. It is how phone apps carry capable assistants, how startups offer chatbots at prices that would be impossible with a frontier model, and why today's best small open-weight models feel impossibly good for their size.

The mental model

A model at work produces more than an answer — it produces probabilities. Ask a vision model what is in a photo and it does not just say "dog"; internally it might be 72 percent sure it is a labrador, 21 percent sure it is a golden retriever, and 4 percent sure it is a wolf. Those probability spreads are richer teaching material than bare answers, because they show how the teacher thinks: which mistakes are near-misses and which are absurd.

In classic distillation, the student trains on those spreads instead of raw labels, absorbing the teacher's judgment, including its sense of what is almost-right. Modern practice also feeds the student huge amounts of text generated by the teacher — questions, answers, step-by-step reasoning — which is how many small models today learn to reason in ways that look suspiciously like their giant parents. The student never receives the teacher's actual internal numbers; it learns from the teacher's behavior alone. That distinction matters legally: copying outputs is a very different act from copying code or weights, and it is exactly the gray zone making lawyers busy.

The kitchen analogy

Think of a master chef and an apprentice. The apprentice could spend twenty years cooking alone, tasting failures, slowly building judgment. Or the apprentice could stand beside the chef for a year, watching every decision — when the pan is hot enough, why the sauce was saved with a knob of butter — and absorb decades of accumulated judgment quickly. The apprentice will never be the master on day one, but they will cook far better than someone with the same amount of solo practice.

Distillation is that apprenticeship for models. The teacher has already paid the enormous cost of learning from trillions of words; the student pays only the much smaller cost of learning from the teacher's answers.

Common misconceptions

"A distilled model is just the big model, zipped." No. The student is a separate network trained from scratch to behave similarly. It contains nothing of the teacher's internals, and on rare or very hard problems the gap shows. Distillation trades peak capability for efficiency.

"It is the same thing as fine-tuning." They overlap but differ: fine-tuning takes an existing model and adapts it to your data or task, while distillation moves capability from a bigger model into a smaller one. Labs often combine them — distill first for general skill, then fine-tune for the job.

"Only small labs distill." Frontier labs do it constantly, for their own product tiers and as a commercial offering — OpenAI lets developers generate training data from its larger models to teach smaller ones through its API. It is standard industry plumbing, not a hack.

"Distillation is inherently shady." It depends entirely on permission. Distilling your own models, or models whose terms allow it, is ordinary engineering. What Anthropic complained about is unauthorized distillation — harvesting a competitor's outputs against its rules, allegedly at scale. The technique is neutral; the terms of service decide.

Where to learn more

The founding idea is a short 2015 paper by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean that introduced training a smaller network to match a larger one's softened outputs. For modern practice, OpenAI's distillation documentation and Hugging Face's guides on smaller models are the friendliest starting points. And because distilled students often learn to produce long, visible reasoning, pair this with our explainers on What are reasoning tokens? and What is chain of thought? to see what that inherited thinking actually looks like in practice.

Related reading: What is fine-tuning? covers the other half of the modern training playbook, and What are reasoning tokens? explains the visible thinking that distilled students increasingly imitate.

If you were choosing an AI provider tomorrow, would a distilled bargain model sway you — or insist on the frontier original? Tell us in the comments.

Sources: Distilling the Knowledge in a Neural Network (arXiv) · DistilBERT paper (arXiv) · OpenAI — Model Distillation in the API · ChinaTalk — How to Buy Cheap Claude Tokens in China