A 125 Billion Parameter Model on One Ordinary Graphics Card
Tim Dettmers previewed his lab's open-source week: an inference framework running Qwen 3.8 Flash Next on a single 24 GB GPU, DeepSeek V4.1 on a MacBook, and a compaction technique he says halves agent costs.
Tim Dettmers, the researcher behind the widely used bitsandbytes quantization library, published a preview on Monday of what his university lab is releasing this week: two open-source projects and four papers, deliberately shipped together. The numbers in it are the kind that change what is possible on hardware people already own. His inference framework runs Qwen 3.8 Flash Next, a 125 billion parameter model, on a single 24 GB graphics card, the sort found in an ordinary desktop. With an AMD Strix machine, an Nvidia DGX Spark or a MacBook with 128 GB of memory, he says you can run DeepSeek V4.1 at 550 billion parameters. On Mac, a Qwen 3.6 35B model runs at 450 tokens per second at 1.5 bits per weight.
That last figure deserves unpacking. Every parameter in a model is a number, normally stored using 16 bits of memory. Quantization squeezes those numbers into fewer bits, and at 1.5 bits a model needs roughly a tenth of the memory it would otherwise. The trade is precision, which is why doing it without wrecking output quality is hard and why Dettmers’ name is attached to much of this work. The other piece is CliffCompaction, an automatic compaction technique his lab has used internally for months. Compaction is how an agent keeps working after its conversation would have overflowed: it summarises what has happened so it can carry on. He says sessions run for millions of tokens, some of his past a hundred million, and that it cuts cost by about half. One partner company measured a 45 percent reduction in its total AI budget.
What is behind this
Dettmers frames the whole release as an argument that AI research does not belong only to whoever owns the most GPUs, and that small university labs have an advantage precisely because they must work on problems that are cheap to attack. It is a nice thought, and the more interesting technical claim underneath it is that the ceiling on local AI has not been the models. Open weights good enough to be useful have been downloadable for a while. The obstacle has been that a 125 billion parameter model would not fit in a normal machine, so most people ran small models and concluded local AI was disappointing.
The fair caveat: these are preview claims from a blog post, the code lands over the coming days, and nobody outside the lab has reproduced any of it. Quantization results in particular tend to look better on the benchmarks chosen to demonstrate them than on your own work.
What this means for you: If running models on your own machine has ever appealed, for privacy, for cost, for working offline, this is the week to look again, because the size limits you remember may no longer apply. For everyone else the relevant number is the cost one. If a compaction technique can halve what an agent spends, and the method is public, it will not stay a research curiosity for long. Cheaper agents are the mechanism by which all of this reaches ordinary products.
Sources
Source: https://timdettmers.com/2026/09/21/dlab-open-source-week/
A Mac Flaw Lets Any App Redirect What You Dictate to Meta's Assistant
Security researcher Patrick Wardle disclosed a zero-day in Meta's Muse for Mac: an undocumented setting any local process can flip, sending dictated prompts to an attacker's server and exposing the account token.
