Model Distillation
Also known as: knowledge distillation, student-teacher learning, model compression, distilled model
Model distillation (or knowledge distillation) is a technique for creating a smaller, more efficient AI model (the 'student') that is trained to approximate the behavior of a larger, more capable model (the 'teacher').
Instead of training the student model directly on raw data, it is trained on the outputs of the teacher model -- learning to replicate the teacher's predictions and reasoning patterns. The result is a smaller model that captures much of the teacher's capability but runs faster and at lower cost, making it practical for on-device deployment, low-latency applications, or cost-constrained environments.
Model Distillation, explained at three levels
At home
Model distillation is like a master chef training an apprentice. The master chef (the big AI model) has years of experience and can cook incredible meals. The apprentice (the small AI model) watches the master, learns from every decision the master makes, and gradually learns to cook nearly as well -- even though the apprentice has far less experience. The result is a chef who is almost as good but can work in a much smaller kitchen with much less equipment.
At work
Distillation is used when a large, accurate model needs to be deployed in environments where its size or compute requirements are impractical. A language model with 70 billion parameters might be too large to run on a phone or too expensive to run at scale for an API. A distilled version with 7 billion parameters, trained to approximate the larger model's outputs, can run on a phone and costs a tenth as much per inference -- with only a modest accuracy trade-off for many tasks.
At home
The distillation training process uses 'soft targets': instead of training the student model to predict hard class labels (cat/dog: 1 or 0), it trains the model to match the teacher's probability distributions (cat: 0.8, dog: 0.15, rabbit: 0.05). These soft targets contain more information than hard labels -- they tell the student not just the right answer but how confident the teacher is and what the 'near misses' look like. This richer signal leads to better learning efficiency than training on raw labels alone.
At work
Distillation for large language models is more complex than for classification models. LLM distillation typically involves generating a large corpus of examples using the teacher model (including chain-of-thought reasoning) and then training the student model on this generated data. This 'synthetic data distillation' approach allows the student to learn reasoning patterns from the teacher without requiring the teacher's architecture -- allowing distillation across different model families. DeepSeek-R1 (2025) demonstrated this approach by distilling reasoning capabilities into smaller open-source models.
At home
Distillation and quantization are complementary compression techniques often applied together. Distillation creates a smaller model with fewer parameters. Quantization reduces the numerical precision of a model's existing parameters. Applying both -- creating a smaller model and then quantizing it -- produces a very compact model suitable for edge deployment. The combined approach is standard for creating on-device AI models for smartphones and embedded devices.
At work
From a practitioner standpoint, the quality trade-off from distillation is task-dependent. Distilled models typically perform nearly as well as teacher models on common tasks that are well-represented in the distillation training data, but may underperform significantly on rare, complex, or out-of-distribution tasks. Evaluating a distilled model requires testing on your specific task distribution, not just on general benchmarks. Do not assume a distilled model will match the teacher's performance on your specialized use case without domain-specific evaluation.
Real-world examples
At home
Priya's company wanted to add a text classification feature to their mobile app. They could not use the large cloud API model they had tested (too expensive at scale). Their ML engineer used knowledge distillation to train a 150MB model on the outputs of the large model, creating a compact version that ran on-device and achieved 94% of the large model's accuracy on their classification task.
At work
Microsoft used knowledge distillation to create the Phi series of small language models, training them on high-quality synthetic data generated by larger models. The resulting models achieved performance competitive with much larger models on coding and reasoning benchmarks, making them practical for on-device AI applications and low-cost API deployments.
How it compares
| Feature | Model Distillation | Quantization | Pruning |
|---|---|---|---|
| What it does | Trains a new smaller model to approximate a larger model's outputs | Reduces numerical precision of existing model weights | Removes less-important weights from an existing model |
| Output | A new, smaller model (fewer parameters) | The same model with lower precision weights (smaller file size) | The same model with fewer weights (smaller file size) |
| Quality impact | Some accuracy loss; depends on teacher-student size ratio | Small accuracy loss for 8-bit; larger loss for 4-bit | Small accuracy loss if applied carefully; larger if aggressive |
| Training required | Yes -- student must be trained from scratch or fine-tuned | No -- applied post-training to existing weights | No -- applied post-training; may require fine-tuning after |
| Primary use | Creating small models for on-device or low-cost deployment | Reducing memory footprint and inference latency | Reducing model size for deployment on constrained hardware |
Distillation creates a new smaller model; quantization and pruning modify an existing model. All three reduce inference cost. They are complementary and often applied in combination for maximum compression.
For analysts
In BA artefacts, model distillation appears as a technical approach for meeting performance, cost, or on-device deployment requirements. When a requirement specifies that an AI feature must run on-device (for privacy or latency), or that the inference cost per request must be below a threshold that a large model cannot meet, distillation is a likely implementation approach. Document the required accuracy on the specific task (not a general benchmark) as the acceptance criterion, so the engineering team can evaluate whether the distilled model meets requirements. Include a re-evaluation trigger: 'The distilled model must be re-evaluated against the teacher model's outputs whenever the teacher model is updated, to ensure the accuracy gap has not increased.'
History & origin
Knowledge distillation was introduced in a 2015 paper by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean ('Distilling the Knowledge in a Neural Network'), which showed that small models could match much larger models when trained on the 'soft' probabilistic outputs of the large model rather than hard class labels. The technique became a standard tool in ML engineering for model compression. The DistilBERT model (2019) demonstrated that a model 40% smaller than BERT could achieve 97% of BERT's performance on benchmark tasks. Distillation is now used extensively for creating efficient versions of large language models for on-device deployment.