Benchmark log: Qwen3.8-Flash-Next NVFP4 on one DGX Spark (vs llama.cpp, three nights) — and GLM-5.3-Flash MLX 4-bit stream concurrency on an M3 Ultra — numbers, method, and two questions #754

Two honest benchmark logs from production boxes, offered in the spirit of #22/#296/#167 ("independent run").
Method and caveats first; the numbers are worthless without them. Nothing here is a bug report — the
questions are at the end.

Method (both benches)

Same in-house harness for every engine: temperature 0, SSE streaming, stream_options.include_usage,
a warm-up request first, metrics taken from usage counters and the first-token timestamp.
Per engine, three cells: (1) a long prompt (~20 k tokens of our own application code, which we do not
publish, plus a five-line summary instruction) — prefill tok/s = prompt tokens / time-to-first-token;
(2) a short prompt asking for a small Python function, 500-token cap — decode tok/s on one request;
(3) the short cell run twice concurrently — cumulative tok/s plus each stream's rate.

Caveats to read every table with: single run per cell per night; the boxes carry production traffic
around the bench; the llama.cpp build we measure against is our long-running production service and its
version was not recorded in the log (my fault — next run will pin it); nights are not comparable to each
other, only within a night.

Bench A — Qwen3.8-Flash-Next NVFP4 on one DGX Spark (GB10, 128 GB)

Checkpoint local-inference-lab/Qwen3.8-Flash-Next-NVFP4 (106 GB) on local NVMe (copied from our NAS).
TensorFold Python 0.6.3 CUDA engine:
tensorfold serve <dir> --name qwen3.8-flash-next --backend cuda --parallel 4 --context 131072 --no-update-check.
Baseline: the production llama.cpp service on the same box.

Three nights to get two of TensorFold data:

• Night 1 (04→05/10): both variants fail at import — ModuleNotFoundError: No module named 'torch'
  in the venv. Fixed next morning (PyTorch CUDA build installed). llama.cpp not measured this night.

• Night 2 (05→06/10): both variants refused by the startup check (word for word, same in both logs):

  [tensorfold] loading qwen3.8-flash-next: Qwen3.8 Flash Next (qwen4_exp) on CUDA
  [tensorfold] CUDA rank 0 startup estimate 77.70 GiB within 83.97 GiB; native 262144, allocated prompt/reply window 131072, cache slots 131073
  tensorfold: --ple-on-ssd reads the MLX checkpoint's n-gram shards from disk; an NVFP4 checkpoint's tables stay memory-mapped, so drop --ple-on-ssd
  

  (our script carried --ple-on-ssd from the MLX recipes; the refusal message was excellent and we dropped
  the flag.) The MTP run's memory planning got to 81.09 GiB within 84.36 GiB — the box should fit it.

• Night 3 (06→07/10): no-drafts variant (--no-drafts) starts and serves. The MTP variant failed
  again, and this time the failure log did not make it back to our archive — the only artifact we kept is
  the timestamped "did not start" line. So the MTP row below is genuinely missing, not slow.

Read it carefully: night 3's llama.cpp row is our long-lived production service having a bad night (its
owner's complaint was already 2× slower than its night-2 baseline), which flatters TensorFold. The honest
claims are: (a) NVFP4 prefill on the CUDA engine is ~6–8× llama.cpp's on this box (1 488 vs 185–233
tok/s at 20 k context — the number that made us switch Qwen to TensorFold in production); (b) decode is
comfortably at or above llama.cpp's good-night figures single-stream, and 2-stream aggregate matches
llama.cpp's good night in one run; (c) we could not get the MTP variant to start on any of the three
nights.

Question 1: what is the right invocation for self-drafting MTP on NVFP4 on a single 128 GB Spark
(--parallel 4 --context 131072)? The night-2 refusal suggests NVFP4 keeps PLE tables mapped, so we
dropped --ple-on-ssd; after that, plain serving worked first try. If there is a known-good flag set (or
a reason MTP-on-NVFP4 is not expected to start at 0.6.3), we'd rerun the bench with it and post the
missing row — the box is idle at night and the harness is scripted.

Bench B — GLM-5.3-Flash MLX 4-bit on a Mac Studio (M3 Ultra, 512 GB): what five slots really buy

Checkpoint TensorFold/GLM-5.3-Flash-MLX-4bit-MTP (169 GB on internal SSD, 43 shards) on the Python
0.6.6 MLX backend:
tensorfold serve <dir> --backend mlx --context 262144 --parallel 5 --reasoning-effort xhigh --drafter none --prompt-cache-gib 16 --checkpoint-slots 8 --max-tokens 131072 --metrics-open
(the MTP head was off for these numbers — production runs drafted-none; reload after a restart is
54–79 s). Two decision models share the box (~38 GB); memory never left active 190–205 GB of a
355.4 GiB budget.

Concurrency bench: 1, 3 and 5 simultaneous streaming requests with distinct ~1 500-token prompts, 384
output tokens each, thinking disabled, server memory sampled every 2 s (ambient production traffic was
present at levels 1 and 3; level 5 got a quiet window).

A single stream on an otherwise quiet box decodes at 65–75 tok/s (/health, repeatedly); once
several streams co-decode each settles at 12–17 tok/s and the aggregate saturates at ~60–85 tok/s
regardless of stream count — the M3 Ultra's bandwidth is the ceiling, not the server. So the five slots
buy queueing (nothing blocks, nothing refused, waiting absorbs bursts) and not per-turn speed.

Memory math agrees with the code we serve: per token per stream the 11 sparse-attention layers cost
11 × (2×(512+128+128) + 2×128/4) = 17,600 bytes (~17.2 KiB) (the 34 linear-attention layers hold a fixed
4.2 MB each) → ~4.8 GB per stream at the full 262 144-token window, ~24 GB for five saturated slots,
~48 GB for ten — a --parallel 10 worst case of ~250–260 GB total against the 355.4 GiB budget. Observed
delta between 1 and 5 long streams: +1 to +2 GB (contexts stayed short).

Question 2: is the aggregate plateau consistent with what you measure for glm5_next on M3 Ultra
(one-rank MLX, five resident streams)? If a per-stream or aggregate number here looks off versus your own
M3 Ultra runs, we'd like to know which knob to doubt — batched gathers across streams (#368) is exactly
what this plateau makes interesting to us.

Sources (ours, available on request): harness sources, per-request JSON with chunk arrival times,
server /health + /metrics samples, free -h / vm_stat captures. Checkpoints, revisions, mlx 0.32.3
and both TensorFold versions are pinned in our logs; we'll add exact build hashes and the llama.cpp
version to any follow-up.