16 + 128 = 144 GB: does “four-hands” inference between laptop and GX10 make sense?

The idea

I have two machines that do LLM inference:

• the laptop, with a GPU that has 16 GB of VRAM: fast, hungry for bandwidth (ballpark: 700–1000 GB/s depending on the chip), but with little space;
• the GX10 (Grace Blackwell GB10 SoC), with 128 GB of unified memory: lots of space, but LPDDR5x bandwidth of about 273 GB/s. Today it runs llama.cpp serving Qwen locally.

The temptation is obvious: what if we merged the two memories into a single ~140 GB “super GPU”? The GX10 fits 100+ GB quantized models; the laptop doesn’t. But is the sum of capacities really worth the sum of performances?

The brutal law: decoding is bandwidth, not space

Generating tokens is a memory-bound operation: at every token the model reads almost all of its quantized weight from RAM. Maximum speed is approximately:

$$\text{token/s} \approx \frac{\text{memory bandwidth}}{\text{model weight}}$$

Hence the first counter-intuitive verdict: for many models the laptop alone is faster than the GX10 alone. A 32 GB Qwen in Q4 on the GX10 runs at ~273/32 ≈ 8-9 t/s; the same model (or a 14 GB one) on the laptop’s GPU can exceed 30-40 t/s. Extra capacity is not free: you pay for it in bandwidth.

What happens when you join them

There are two classic schemes:

1. Pipeline parallel (split by layer): the first N layers on the laptop, the rest on the GX10. Every token crosses both machines in sequence. The time per token is the sum of the two halves’ times (plus network latency at every pass).
2. Tensor parallel (split by tensor): every layer is divided in half; the two machines work in parallel on different portions. Here the effective bandwidth approaches the sum of the bandwidths (800 + 273 ≈ 1000 GB/s in theory) — but it requires a data exchange (all-reduce) at every layer, so it is brutally sensitive to network latency and bandwidth.

The realistically good case is the split by layer with weights distributed in proportion: ~16 GB of layers land on the laptop (processed at ~800 GB/s), the rest on the GX10. For a 70 GB model in Q4:

• GX10 alone: 273/70 → ~4 t/s
• GX10 + laptop: the GX10 half takes ~0.13 s, the laptop half ~0.02 s → ~7 effective t/s

A real gain, but not a miracle — and only if the network is not the bottleneck: you need a dedicated wired link (2.5 GbE at minimum, better 10 GbE or Thunderbolt/USB4 direct). Over Wi-Fi or a shared network, the milliseconds of RTT multiplied by dozens of layers drag everything back under 5 t/s.

The real golden case: MoE models

The picture changes for Mixture-of-Experts models (Qwen3-235B-A22B and similar): enormous total weights, but few active parameters per token. There:

• the expert layers (most of the weights, read sparsely) sit comfortably on the GX10’s 128 GB;
• attention, the router and the hottest experts can live on the laptop’s GPU, read at high bandwidth.

The effective bytes-per-token is a fraction of the model, so the laptop’s bandwidth counts far more than the 16:128 ratio would suggest. It is the scenario where a heterogeneous cluster of two boxes does things neither one can do alone: a 235B Q4 in Q4_K_M (~130 GB) fits neither machine, but it fits the sum.

How you actually do it (today)

• llama.cpp RPC: the most mature. On the GX10 you launch an rpc-server, on the laptop you start llama-server with -r <ip-gx10>:50052 --n-gpu-layers 99 and tune the split with --split-mode (layer/tensor) and --tensor-split. No mental model change: the client just sees one bigger “GPU”.
• exo and similar: a more “turnkey” approach to tensor parallel over the network, great for experimenting, less so for daily production.
• Parallel pipeline via API (one model per machine): not combined inference of the same model, but the boring, winning strategy for many workflows (see below).

The three honest strategies

Verdict

The 144 GB hybrid cluster is not an accelerator, it is a capacity multiplier: it makes nothing faster than the local GPU would on models that fit, but it makes runnable models that otherwise would not run on either machine — MoEs above all. With a good network and llama.cpp RPC it is a weekend of work; with a dense 70B model the gain is real but limited (~1.5-2x versus the GX10 alone). With the right MoE, it is the only option on the table.

And then there is the hidden value: while the GX10 does the capacity work, the laptop stays free for the prefill, for a second model in parallel, or simply for not overheating. Two boxes, two jobs.