speculative decoding
A speed trick where a small fast model drafts ahead and a bigger model checks its work.
Language models normally write one token at a time, and each one has to wait for the last. Speculative decoding breaks that queue. A small, fast model guesses the next several tokens, then the big model checks the whole batch in one pass. Where the guesses were right, they are kept. Where they were wrong, the big model corrects course.
Because checking is much cheaper than generating, this makes responses noticeably faster without changing the output, which is the appealing part: you get the big model’s answers, just sooner. It is one of the quiet engineering wins behind the falling prices of AI services. You will never see it as a feature, only as a bill that got smaller and a cursor that blinks less.