Pretraining

Pretraining is where a language model actually learns language. It reads its way through an enormous pile of text, over and over, adjusting its internal numbers so it gets better at predicting what comes next. Nobody is telling it to be helpful yet, and it has no idea what a chat is. It is just absorbing patterns: grammar, facts, code, the shape of an argument. This phase eats most of the compute budget and typically runs for three to six months on a big cluster.

Everything that makes a model pleasant to talk to comes afterwards, in fine-tuning and related steps that are far cheaper. That split explains a lot of industry news. When you read that a model is “in pretraining”, it means a release is still months away. And when a lab wants to improve behaviour rather than raw knowledge, it usually does not start over, it adjusts the later stages.