Mixture of experts

In a mixture-of-experts model, the network is split into many sub-networks and a router picks only a few of them for any given input. That is why these models are quoted with two numbers, for example 284 billion total parameters and 13 billion active: the first is how big the model is, the second is how much of it actually runs each time.

The practical effect is that you get the knowledge of a very large model at closer to the running cost of a small one. Almost every recent open model, from DeepSeek to Thinking Machines’ Inkling, is built this way. The trade-off used to be memory, since you still had to load the whole thing even though you only use a slice of it. Runtimes such as Swiftlet now chip away at that by streaming the unused experts from disk on demand.