190 tokens per second: a 35-billion-parameter model on a laptop A 35-billion-parameter language model shouldn’t fit in 16 gigabytes of video memory, let alone generate at nearly two hundred tokens per second. And yet it happens, provided you accept two architectural constraints and treat the key-value cache as a design variable. Here is what we measured, with the complete numbers and the parameters that mattered. ...
Today a small win, the kind worth double: the Scar under the corporate VPN, and from there connected to the GX10 with VS Code Remote as if I were on the sofa. For months those two things couldn’t coexist. ...
I told you the architecture of the Telegram gateway that mirrors VS Code conversations onto my GX10. This morning was the first shift where the setup did real work — and it produced exactly the kind of morning a blog post never plans for: a .NET solution that wouldn’t build, a private npm feed answering 401, and a personal access token pasted into the chat. Full story, minus the token. ...
Until recently I could talk to my agent from exactly two places: the GX10’s terminal, or VS Code sitting in front of one of the two machines. An agent that only answers when you’re at your desk is an agent with office hours. ...
My GX10 has been working hard: it runs models locally around the clock, and neither of the previous posts ever said how much. It was time to stop answering “I guess it’s fine” and open a dashboard. Here’s how I built it — and the numbers from its first proper week of work. ...
Seven dense days, in which the “blog” project went from being a static site with a makeshift Flask panel to a small ecosystem: a well-written CMS, images generated by the local model, and a direct line between the two machines at home that now also records what I do. I retrace everything in order, with links to the posts already written in detail. ...
Until today, every article on this blog had an aesthetic problem: no cover image. PaperMod shows covers on the home page, in lists, and at the top of each post — without an image, the layout loses half of its impact. Drawing them by hand every time, though, was the quickest way to stop publishing anything. ...
Over the past few days I have spent my time optimizing the communication between the two main machines of my setup: my gaming laptop, the Strix Scar 16, and the home server GX10. The goal was clear: create a smooth workflow that lets me leverage the server’s compute power to run heavy local models while using the laptop’s interface for everyday interaction. ...
16 + 128 = 144 GB: does “four-hands” inference between laptop and GX10 make sense? The idea I have two machines that do LLM inference: the laptop, with a GPU that has 16 GB of VRAM: fast, hungry for bandwidth (ballpark: 700–1000 GB/s depending on the chip), but with little space; the GX10 (Grace Blackwell GB10 SoC), with 128 GB of unified memory: lots of space, but LPDDR5x bandwidth of about 273 GB/s. Today it runs llama.cpp serving Qwen locally. The temptation is obvious: what if we merged the two memories into a single ~140 GB “super GPU”? The GX10 fits 100+ GB quantized models; the laptop doesn’t. But is the sum of capacities really worth the sum of performances? ...
This blog is the second thing I built in order to write the first thing on it. The site is generated by Hugo with the PaperMod theme — fast, static, zero runtime dependencies. But publishing an article from the terminal is a ritual that discourages: open the file, fix up the front matter, hugo server for the preview, docker compose up -d --build, pray. ...