190 tokens per second: a 35-billion-parameter model on a laptop

A 35-billion-parameter language model shouldn’t fit in 16 gigabytes of video memory, let alone generate at nearly two hundred tokens per second. And yet it happens, provided you accept two architectural constraints and treat the key-value cache as a design variable. Here is what we measured, with the complete numbers and the parameters that mattered.

There is one number in this story worth more than any adjective: 190.7 tokens per second at generation. That is how fast a 35-billion-parameter model wrote text on a laptop, with the fan spinning and the rest of the operating system still answering clicks. It is not the record of a cluster costing hundreds of thousands of euros. It is a laptop graphics card, 16 gigabytes of memory, a 157-watt power brick.

How we got there is worth telling, because the path was not linear and the two discoveries that count came from the data, not from intuition.

The problem, stated honestly

The model is called Qwen3.6-35B-A3B, quantized to UD-IQ3_XXS: 14,069,266,720 bytes, a 14-gigabyte file that weighs about half of the available memory. The GPU is an RTX 5080 Laptop, 16,303 MiB of video memory, a thermal ceiling of roughly 157 watts. The engine is llama.cpp b11201 with native CUDA support for the Blackwell architecture (SM120).

At a glance, the math doesn’t add up. Fourteen gigabytes of weights, plus the cache the context needs, plus the activations: on a 16-gigabyte card it should fit tightly, or not at all.

The first thing to understand is why the model exists in this shape.

Why a 35-billion model occupies 14 gigs

Qwen3.6-35B-A3B is not a dense model. It is a hybrid MoE with an SSM component: 41 blocks, 256 experts, and a parameter called full_attention_interval set to 4. That means full attention — the kind that builds the key-value cache — is applied to only one block in four. The other three are handled by a recurrent state component that does not accumulate memory as the context grows.

Block 0   ██████  full attention  -> builds KV cache
Block 1   ░░░░░░  SSM component   -> no KV cache
Block 2   ░░░░░░  SSM component   -> no KV cache
Block 3   ░░░░░░  SSM component   -> no KV cache
Block 4   ██████  full attention  -> builds KV cache
...

The “35B” in the name is the total parameter count; the “A3B” says roughly 3 billion are active per token. That is why it runs fast: each token moves few weights, not all 35 billion. Aggressive quantization (3.4 bits per weight) brings the file down to 14 gigs. The hybrid architecture is what makes the whole thing manageable, and the data shows it: memory does not explode as context grows the way it would in a classic transformer.

Before touching anything, we set an explicit priority, because on a laptop the constraints step on each other’s toes:

correctness > stability > memory > speed > context.

Invert this order and you get benchmarks that lie.

First discovery: internal speculative decoding (MTP)

The model ships with an integrated MTP head (Multi-Token Prediction, a “next-n” predictive layer already inside the file, no separate draft model needed). The idea is trivial to state: instead of generating one token at a time, the model ventures a few ahead and then verifies them in a single pass. Correct guesses get “discounted”, wrong ones are discarded.

The parameter that controls it, --spec-draft-n-max, sets how many tokens to venture. We rotated it keeping everything else fixed, 8K context, temperature 0, a 512-token prompt:

The winner is not the one that guesses the most. n-max 2 beats the baseline by 34 percent. n-max 3 and 4 do worse, because the acceptance rate drops: the more tokens you venture ahead, the more often you are wrong, and the cost of discarding eats the gain. It is a counterintuitive result and therefore precious: generosity pays up to a point, then it starts costing.

The acceptance rate, read from the server’s internal counters (draft_n_accepted over draft_n), oscillates between 64 and 86 percent depending on the task: very high on boilerplate, deterministic code, lower on free-form text. This percentage determines the real gain, and explains why the number changes from conversation to conversation.

Second discovery: context does not degrade speed

In classic transformers, lengthening the context slows generation: the cache grows, attention reads more memory, throughput drops. Not here. With MTP n-max 2 active:

From 8 to 64 thousand context tokens, generation speed stays practically nailed at 190 tok/s, and memory barely rises. That is the signature of the hybrid architecture: with full attention on only one block in four, the cache grows a quarter of what a homogeneous model would need, and Flash Attention does the rest.

So far, so good. Then there is the wall.

The 128K wall, and the variable that tears it down

Asking 128,000 tokens of context of 16 gigabytes is asking a lot. With the key cache in the standard 16-bit format (f16) the result is disastrous:

With a 16-bit cache the memory saturates, the system spills the cold parts into system RAM, and generation collapses to 17.5 tok/s: eleven times slower. The model is technically alive, but effectively unusable.

The solution lives in one parameter: -ctk q8_0 -ctv q8_0, i.e. quantizing the key and value caches to 8 bits instead of 16. Half the memory per cache element. At 128K the quantized cache fits in 16 gigabytes, the overflow disappears, and generation climbs back to 182.9 tok/s.

This is the design variable that superficial benchmarks ignore. On a 16-gigabyte card, cache quantization is not an optimization flourish: it decides whether long contexts exist at all. We therefore set q8_0 as the default on every profile, including the daily-driver one at 128K.

Beyond benchmarks: five real tasks

Throughput numbers say little about quality. We put the server through five realistic programming tasks and measured time to first token (TTFT), speed and MTP acceptance on every run, read from the server’s native counters:

Reading the data: time to first token between 100 and 300 milliseconds is below the perception threshold, and makes the experience in an editor comparable to a cloud service. Generation holds between 168 and 195 tok/s through the API as well, so MTP speculation works with the server running, not only in offline tests. MTP acceptance is confirmed higher on structured tasks (the xUnit test at 86%) and lower on free-form review (the refactor at 64%). Outputs came out relevant and correct in all five cases.

On stability: 32 requests out of 32 completed, zero errors or timeouts, GPU in thermal equilibrium between 67 and 75 degrees, power between 135 and 147 watts, memory stable. No throughput degradation over time, no leaks. The peak temperature observed across the whole campaign was 73 degrees, with no throttling.

During inference the laptop stayed usable: twelve of the CPU’s twenty-four logical threads (a Core Ultra 9 275HX) dedicated to the model, twelve left free for the system. VS Code, the browser and video calls all answered.

From benchmark to workstation

A model that runs fast but has to be launched by hand from a terminal is a toy. A model that starts by itself at boot and shows up in the editor’s dropdown is a tool. The second half of the work was all about that.

The final architecture:

The supervisor does three things: starts the server if it is not running, restarts it if it dies (with a fifteen-second delay), and respects an explicit manual stop without stubbornly bringing it back up. The watchdog runs detached from the terminal, so it survives closing the window it was launched from.

Two technical choices were worth more than the rest:

• Loopback only. The server listens on 127.0.0.1:30000, never on the network interface. A local model must not be reachable from other devices on the LAN.
• A single entry point. qwen36 start|stop|restart|status|logs as the one command, with duplicate control: never two instances fighting over the same GPU.

The VS Code integration goes through a provider with an OpenAI-compatible endpoint. The model appeared in the dropdown next to the cloud ones, with tool calling, streaming and reasoning mode verified with real requests, not declared on paper. The served context, read from the /props endpoint, is 131,072 tokens, with video memory at around 15.9 gigs out of 16 in steady state.

What is still on the table

Transparency about what doesn’t check out yet:

• The 128K context consumes nearly all video memory (15.9 of 16 gigs). Heavy multitasking while the model generates at full context can saturate the GPU. For full days you need the smaller profiles (8K, 16K, 64K), which trade context for headroom.
• The model is a “thinking” model: it produces internal reasoning first. Read by a client that knows how to handle it, fine; read by one that doesn’t, it returns empty content if the tokens run out inside the reasoning. An explicit parameter is needed for a direct answer.
• The full laptop reboot still has to be validated at the first scheduled opportunity: the supervisor has been tested under crash conditions, but the cold boot cycle deserves definitive confirmation.

What we take away

Three lessons that reach beyond this machine.

One. Hybrid MoE models with recurrent state are making large models accessible on small hardware, and the numbers from this test prove it: 35 billion parameters, 16 gigs of memory, nearly two hundred tokens per second. The bottleneck has moved from the weights to the cache.

Two. Speculative decoding is a real gain but not a free one, and it has an optimum: guessing two tokens ahead pays, guessing four doesn’t. Parameters should be measured, not maxed out.

Three. Key-cache quantization is the hidden variable of long contexts on small video cards. It is the literal difference between 17 and 183 tokens per second.

None of these results require exotic hardware. They require a laptop, an open-source engine, and the discipline of letting the numbers decide instead of intuition.

All reported values come from logs and CSVs measured during the test campaign. Full configuration, startup scripts and rollback procedures available in the repository.