dense model
A model that uses all of its parameters for every word it produces, unlike a mixture of experts.
In a dense model, every one of its parameters, the adjustable numbers learned during training, is involved in generating each word. The alternative design is mixture of experts, where only a small slice of the model wakes up per word, which makes it much faster for its size.
The tradeoff is simple to remember. Dense models tend to punch above their parameter count on quality; mixture of experts models punch above their count on speed and fit into less memory. If you are choosing a model to run on your own machine, a smaller dense model often gives better answers than a nominally larger sparse one, and takes longer to do it.
Not referenced in any article yet.