Distillation (AI)

One-line definition: Using a stronger “teacher” model’s outputs (or synthetic data generated by it) to train or fine-tune a “student” model more cheaply than training from scratch — a practice Kyle’s sources treat as a decisive, contested factor in the Chinese open-model lead.

How sources describe it

• “Pre-training creates a capable base model. Post-training turns it into a useful coding, reasoning, tool-using, and agentic system. A stronger teacher converts part of that expensive discovery process into a cheaper learning problem.” America’s Open-Model Paradox
• Western labs have a lawful path to distill from Chinese open-weight models but not from closed Western frontier models (GPT/Claude outputs are off-limits by terms of service) — creating an indirect flow where Western capability allegedly leaks into Chinese open weights and then lawfully re-enters Western post-training. America’s Open-Model Paradox
• “Distillation does not explain China’s entire open-model lead… But distillation compresses the costly final gap between a strong base and a near-frontier system.” America’s Open-Model Paradox (quoted and endorsed in Who’s Afraid of Chinese Models?)
• Ben Thompson frames the asymmetry as a policy failure: U.S. open-weight makers must respect frontier labs’ terms of service (no distillation), so they end up “distilling the distillation” through a Chinese detour — and argues the U.S. should pass a law protecting the fair-use case for training on data and barring anti-distillation terms of service, at least domestically. Who’s Afraid of Chinese Models?

Where it shows up

• America’s Open-Model Paradox — the central argument: distillation from Chinese open weights gives Chinese labs a “recurring structural advantage,” and the piece proposes a legal domestic path for capability transfer.
• Who’s Afraid of Chinese Models? — Ben Thompson engages the same argument, questions why distillation is treated as illegitimate, and proposes U.S. policy change.

Related concepts

• Open Source — distillation is one of the mechanisms by which open-weight models propagate frontier capability.
• China — the source of the open-weight “teacher” models discussed.