← Glossary Term
multi-token prediction
A speed trick where a model guesses several words ahead at once instead of one at a time.
Language models normally produce one token, roughly three quarters of a word, then look at what they wrote and produce the next. Multi-token prediction lets the model propose a short run of tokens in one go, then verify them together. When the guesses hold up, you get several words for close to the price of one.
For anyone running a model at home this is one of the biggest levers on speed, often the difference between an assistant that feels sluggish and one that feels responsive on the same graphics card. It is closely related to speculative decoding, which uses a small helper model to do the guessing.
Mentioned in