Distillation

Building a top model from scratch costs a fortune. Distillation is the shortcut: you take an existing, expensive model, ask it huge numbers of questions, and train a smaller new model on its answers. The small model never sees the original training data or the original weights, only the outputs, but it picks up a surprising amount of the behaviour. The result is a model that is much cheaper to build and to run, at some cost in quality.

Done inside one company, this is standard practice and how most of the small, fast models you use are made. Done across companies, it gets contentious fast, because it means one lab can absorb a rival’s expensive work through the public interface. Most providers now ban it in their terms of service, several have accused competitors of doing it anyway, and labs that avoid it tend to say so loudly.