dense model
A model where every parameter is used for every request, as opposed to a sparse model that activates only a fraction of itself at a time.
In a dense model, asking a short question and asking a hard one both run the whole network. That makes the maths predictable: a 32 billion parameter dense model needs roughly 32 billion parameters’ worth of memory and compute every time, which is why the parameter count tells you fairly directly what hardware you need.
The alternative is a sparse design such as a mixture of experts, where a router wakes up only the relevant slice. A model listed as 375B with 23B active is sparse, and it costs closer to a 23B model to run while carrying the knowledge of a much larger one. Dense models are simpler to fine-tune and deploy, which is why labs still ship both.