My GX10 has been working hard: it runs models locally around the clock, and neither of the previous posts ever said how much. It was time to stop answering “I guess it’s fine” and open a dashboard. Here’s how I built it — and the numbers from its first proper week of work.
The architecture in ten lines
Everything runs in Docker Compose, four containers:
node-exporter :9100 → CPU, RAM, disk, network (host)
prometheus :9090 → scraping, retention, rules
grafana :3001 → dashboards
backend (flask):5000 → my own application metrics
Plus one thing living outside Docker, which is the most interesting part:
llama.cpp native metrics. The inference server is a systemd service on the
host (that story is in the installation post),
started with --metrics. Prometheus inside its container reaches it through
the bridge network gateway:
# llama.cpp native metrics: the server runs on the host (systemd, --metrics).
# 172.18.0.1 is the gateway of the compose bridge — the host, as seen from
# inside the Prometheus container.
- job_name: 'llama_server'
static_configs:
- targets: ['172.18.0.1:30000']
labels:
service: 'llm-inference'
model: 'qwen3.8-flash-next'
Today there are three active targets (system, backend, llama.cpp) exposing 187 metrics in total. Zero extra agents to install by hand beyond node-exporter.
The rules that make the graphs readable
The difference from twenty years ago isn’t Prometheus: it’s recording
rules. llama.cpp exposes raw counters (tokens_predicted_total,
prompt_tokens_cached_total…); I pre-compute the derivatives the dashboards
need, so every refresh doesn’t re-derive the world:
- record: llm:decode_tokens_per_second
expr: sum by (model) (rate(llamacpp:tokens_predicted_total[5m]))
/ clamp_min(sum by (model) (rate(llamacpp:tokens_predicted_seconds_total[5m])), 0.001)
- record: llm:prompt_cache_hit_ratio
expr: sum by (model) (rate(llamacpp:prompt_tokens_cached_total[15m]))
/ clamp_min(sum by (model) (rate(llamacpp:prompt_tokens_cached_total[15m])
+ rate(llamacpp:prompt_tokens_total[15m])), 0.001)
The clamp_min is the divide-by-zero trick: when the model is idle the
denominator is 0 and the graph fills with holes; with the clamp the series
drops to zero cleanly.
Only two alert rules for now: request p95 above 5 seconds for 5 minutes (warning), and a model that hasn’t processed a token in 30 minutes (info — it tells me whether the systemd service is alive but idle, or wedged).
The default dashboard is provisioned, not mouse-built: the JSON lives in
the repo (monitoring/grafana/dashboards/) and
GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH points at it as the home page. If
Grafana dies, I restart it and the home is already there — same philosophy as
the CMS that draws its own covers:
configuration is code, clicking is an accident.
Here’s what the “LLM Models” dashboard looks like today: four stat cards on top (tokens generated, prompt tokens, instant decode speed, cache hit ratio) and the historical series across the whole week:
The GX10’s week, in numbers
Updated on September 22: Prometheus has been collecting since the 16th, so the 7-day window is now full, real history — no more four-and-a-half days extrapolated.
Inference (Qwen3.8 Flash Next, quantized, with MTP speculative decoding):
| Tokens generated | 1.8 million |
| Prompt tokens processed | ~150 million |
| …of which served from cache | 142 million (95%) |
| Decode speed | 26 tok/s median under load, p95 ~40, peaks ~51 tok/s |
| Prefill | median ~20,000 tok/s, peaks ~52,000 tok/s |
| MTP acceptance | 33–72%, median ~51% |
The interesting row is the cache one: of ~150 million prompt tokens, 142 were served from the prompt cache. Which means the actual prefill work was just 5%. That’s why long agent conversations stay smooth — the context rebuilds almost for free. 26 tok/s of median decode with spikes to 51 is the MTP draft’s signature: when acceptance is good it flies, when it drops you fall back to the model’s bare speed.
Hardware (node-exporter):
| CPU (20 cores) average | 5.4% |
| CPU p95 / peak (10-min average) | 16% / 66% |
| Unified memory average | 80% of 128 GB |
| Memory peak | 94% |
| Disk read / written | 3.4 TB / 360 GB |
Two things surprised me. First: the CPU is idle almost all the time (5.4% average, p95 at 16%) — the peaks are model updates and compiles, not inference: the GB10 does that and the ARM cores sit and watch. Second: 3.4 TB read in seven days, close to 500 GB a day. It looks absurd until you remember that every model load mmap-s 15–20 GB from the NVMe, and between model switches, service restarts and cooling caches the drive grinds away silently. The 360 GB written is the reassuring mirror image: it almost never writes, so the disk isn’t being eaten alive.
Memory at 94% peak is not an alarm — on 128 GB of unified memory that is the model living in working memory together with its KV cache, which is exactly where I want it.
What the chart says day by day
Tokens generated per calendar day, across the full week:
| Day | Tokens generated |
|---|---|
| Wed 16 | 52,000 |
| Thu 17 | 293,000 |
| Fri 18 | 409,000 |
| Sat 19 | 313,000 |
| Sun 20 | 378,000 |
| Mon 21 | 247,000 |
| Tue 22 (mid-day) | 141,000 |
Friday is the CMS-and-AI-covers day, and it shows: it’s exactly the day I wrote the covers post while having the model write most of it. And the graph doesn’t lie about Sundays either: 378,000 tokens, more than Saturday — agents don’t know what a weekend is, unlike the humans watching them.
Next steps
- Blackbox exporter to watch the published site (http://gx10-8532/) from a client’s point of view, not from inside the container.
- A real alert on KV cache pressure (today I only watch tokens, not the squeeze on the cache).
- Maybe pushover/telegram for notifications instead of the current silence — the “model idle for 30 minutes” alert was born and still has no channel.
If you’re holding the classic doubt — “can 128 GB of unified memory do serious inference or is it a toy?” — the data says: 26 tok/s median while the rest of the machine watches, 1.8 million tokens generated in a week, and the time series tells it better than any spec sheet.