Model efficiency / Report summary

Model Distillation and the Diffusion of AI Capability

Smaller models can serve defined workloads economically when evaluation, routing and escalation preserve the application's required quality.

June 2026 · 13 pages · 2.2 MB · Tolaga Research

Download full report (PDF) ↓

Executive summary

Distillation transfers selected behaviors from a larger teacher into a smaller student model. The report treats it as a way to separate the creation of advanced capability from its economical deployment. Frontier models remain useful for difficult work and as sources of training examples, while smaller models can handle sufficiently well-defined tasks.

The central argument is that savings depend on the complete inference system. Selecting a small model is only the beginning: workload classification, representative evaluation, monitoring and escalation determine whether requests can actually be served by that model without unacceptable quality loss.

Key findings

  • The illustrative comparison uses a 70B model with TP8 and a 7B student with TP1; the serving footprints differ as well as model sizes.
  • Modeled 7B latency is approximately 0.58–0.69 times the 70B baseline across the workloads shown.
  • At 75% of requests handled by the smaller model, the illustrative cost reduction is approximately 3.0–3.2 times, depending on routing approach.
  • Capability transfer is uneven across tasks; strength on selected benchmarks does not demonstrate general equivalence to the teacher.

How tiered inference changes economics

The report compares upfront routing with a small-model-first cascade. Upfront routing classifies a request before inference and sends it to the appropriate tier. A cascade runs the smaller model first and escalates unresolved cases. Escalated cascade requests incur work on both models, so the two approaches have different cost and latency consequences.

In the all-small-model scenario, the modeled cost reduction is approximately 12.3 times. That is an illustrative endpoint, not the expected saving for every deployment. As more requests require the larger model, savings shrink. The report gives approximately 5.5–5.8 times at a 90% small-model share and 1.7–1.8 times at 50%.

Capability and deployment implications

The discussion distinguishes distillation from MoE. MoE scales capacity within a model by activating selected experts; distillation trains a separate student. A large MoE teacher can therefore support the creation of a smaller dense student. The techniques serve different roles and can coexist.

Public benchmark examples show why task boundaries matter. Distilled models may retain substantial mathematical performance while losing more capability in scientific reasoning or coding. Matching a teacher's style is also different from matching its reasoning reliability. Production evaluation must include ambiguous inputs and exceptions, not only familiar benchmark-like tasks.

The practical opportunity is to build reliable model tiers. Quality checks and fallback should preserve savings rather than automatically sending every small-model response through a full frontier-model review. Routing accuracy becomes an economic capability in its own right.

Assumptions and limitations

The cost and latency comparison is illustrative and does not establish that the 7B model can safely handle the assumed request shares. The deployment footprints differ. Published benchmark evidence and modeled infrastructure economics are separate forms of evidence, and neither replaces workload-specific validation.

Source: Model Distillation and the Diffusion of AI Capability, PDF pages 2–12. This summary describes the published report; consult the PDF for figures and references.