Quantization

A model is, underneath, a very large pile of numbers. By default those numbers are stored quite precisely, which is accurate but heavy: a model that needs 120 gigabytes at full precision will not fit on a normal computer. Quantization rounds those numbers to a coarser scale, so each one takes up less space. The same model might then run in 40 gigabytes, or even 12, depending on how far you go.

The trade-off is quality, and it is usually smaller than people expect. Mild quantization often costs almost nothing measurable, while aggressive quantization starts to show up as sloppier answers. This is the single technique that makes running open models on your own machine practical, which is why community quantized versions of big models appear on Hugging Face within days of a release.