[voidstackloop]

Overview

This PR adds hybrid MoE expert execution to --cpu-moe and --moe-cache-mib:

1. --moe-cache-profile <file.strp>: Pre-populates the GPU expert cache using offline-profiled routing distributions so the hottest experts start resident in VRAM.
2. --moe-cache-hybrid: Keeps profiled hot experts on the GPU while the CPU computes the remaining misses directly from host RAM in parallel. Misses do not wait for PCIe uploads during generation.
3. --moe-cache-audit <file.jsonl>: Provides a structured JSONL audit trail recording cache hit rates, resident expert IDs, and transfer overhead per generation.

When to use it

The hybrid mode is for MoE models that do not fit in VRAM. It helps most when the GPU can hold only a small part of the experts: the CPU then computes many experts every token, and the hybrid runs that work in parallel with the GPU instead of after it. The further the model is beyond the VRAM, the more it gains; when most of the model fits, upstream's default offload (-fit) is faster.

Situation	Machine, model, cache	Best upstream	PR hybrid + adaptive
Model much larger than VRAM	RTX 4060 Laptop 8 GB, Qwen3.5-35B-A3B Q6_K (27 GB), 4 GB cache	22.5 t/s (-fit)	37.7 t/s (1.7x)
Cache holds a minority of the experts	L4, Qwen3.5-35B-A3B IQ4_XS (18.7 GB), experts kept in host RAM, cache capped at 5000 MiB	33.5 t/s (cache, same cap)	45.9 t/s (1.4x)
Cache holds most of the experts the generation uses	L4 24 GB, Qwen3.5-35B-A3B Q6_K (27 GB), 16 GB cache, 1024 tokens	51.6 t/s (-fit), 50.5 (cache)	50.1 t/s
	L4 24 GB, Qwen3-Coder-Next UD-IQ4_XS (36 GB), 16 GB cache, 1024 tokens	46.6 t/s (cache)	42.6 t/s
Model fits, or nearly fits, in VRAM	L4 24 GB, Flint UD-Q4_K_XL (24 GB), 5800 MiB cache	59.7 t/s (-fit)	29.9 t/s*
	L4 24 GB, GLM-4.7-Flash Q8_0 (30 GB), 16 GB cache	34.0 t/s (-fit)	24.9 t/s*

256 generated tokens unless noted. The laptop row is from the laptop tables below; the L4 rows are from Google Colab (12 vCPUs). * measured before the faster cache fill described below.

Not every case is covered yet: on a 12 GB RTX 4070 SUPER a tester in this thread measured the reverse for Qwen3.6-35B-A3B IQ4_XS (upstream cache 101 t/s, hybrid 85, before the faster fill), which has not been re-measured.

Additional information

• Tested against all 56 supported MoE architectures in test-llama-archs: all pass with 0 failures.
• End-to-end benchmark on RTX 4060 Laptop GPU: llama-completion achieves 0 bytes uploaded over PCIe during generation with 100% profile cache hits.
• Implements CUDA mailbox kernel synchronization in ggml/src/ggml-cuda/mailbox.cu for coordinating asynchronous CPU/GPU token routing without stalls.
• Behavior change for existing --moe-cache-mib users (without the hybrid flags): the expert biases of the cached host layers (e.g. gpt-oss) are now copied to the GPU, and their size counts against the --moe-cache-mib budget, so the same value holds slightly fewer expert slots than on master. Models without expert biases are unaffected.

Benchmarks (vs. upstream llama.cpp)

Generation speed with the MoE experts in host RAM and attention on the GPU.

Hardware: RTX 4060 Laptop (8 GB), 16 threads (8 used), 88 GB RAM. CUDA 13.4.
Setup: llama-completion, -c 2048 -n 256 --temp 0 -s 1, same 63-token prompt for every run, --moe-cache-mib 4096 where a cache is used.
Builds: upstream = ggml-org/llama.cpp master 1b29621; PR = this branch merged with that same master.
No expert profile (--moe-cache-profile) was used. --moe-cache-adaptive starts with an empty cache and fills it while generating.

Qwen3.5-35B-A3B Q6_K (mean of 2 rounds)

Build	Flags	Generation t/s
upstream	(none; -fit chooses the offload)	22.5
upstream	-ngl 99 --cpu-moe	19.6
upstream	-ngl 99 --cpu-moe --moe-cache-mib 4096	21.1
PR	-ngl 99 --cpu-moe --moe-cache-mib 4096	21.1
PR	... --moe-cache-hybrid --moe-cache-adaptive	37.7

Qwen3-Coder-Next Q6_K (1 round)

Build	Flags	Generation t/s
upstream	(none; -fit chooses the offload)	18.0
upstream	-ngl 99 --cpu-moe	17.3
upstream	-ngl 99 --cpu-moe --moe-cache-mib 4096	15.4
PR	-ngl 99 --cpu-moe --moe-cache-mib 4096	15.4
PR	... --moe-cache-hybrid --moe-cache-adaptive	30.9

Hybrid-adaptive is about 1.7x the best upstream configuration on both models.

Notes

• Without the hybrid flags the PR matches upstream: same speed, and the generated text is identical character for character (checked on both models).
• Run-to-run noise on the non-hybrid configs was up to ~5 t/s between rounds; the two hybrid rounds on the 35B model agreed within 3%.
• Prompt processing is not changed by the hybrid mode (~30-50 t/s in every config).
• Hybrid output drifts from the other configs after a few dozen tokens, as upstream's own configs do from each other when experts run through different CPU/GPU kernels. Each config is deterministic. test-llama-archs checks the hybrid configs against the CPU (NMSE <= 1e-4): 396/396 pass on CUDA (RTX 4060 Laptop and L4).

Qwen3.5-35B-A3B IQ4_XS on a second machine (2 rounds)

Hardware: NVIDIA L4 (24 GB) on Google Colab, 12 vCPUs, 53 GB RAM.
Builds: upstream = ggml-org/llama.cpp master d454a20; PR = 3d37f74 (this branch merged with d454a20).
Setup: same prompt and flags as above, --moe-cache-mib 5000 to approximate a 12 GB card.

Build	Flags	Round 1 t/s	Round 2 t/s
upstream	(none; -fit chooses the offload)	93.5	93.3
upstream	-ngl 99 --cpu-moe	31.4	31.4
upstream	-ngl 99 --cpu-moe --moe-cache-mib 5000	33.2	33.2
PR	-ngl 99 --cpu-moe --moe-cache-mib 5000	33.3	33.2
PR	... --moe-cache-hybrid --moe-cache-adaptive	41.3	41.6

• This 18.7 GB model fits entirely in the L4's 24 GB, so upstream's -fit row keeps every expert on the GPU and is the fastest; it is listed for completeness only. The hybrid mode is for models that do not fit in VRAM. Against the configs that keep the experts in host RAM, hybrid-adaptive is about 1.25x here (1.7x on the laptop above, where the CPU share is larger).

Faster fill of the adaptive cache (1967019)

The adaptive cache used to fill its free slots at 16 experts per generated token, so a large cache took hundreds of tokens to fill and the hybrid ran behind upstream's cache until then. It now uploads the experts that missed right after the token that missed them (up to 512 per token, only while the cache has free slots). The swaps once the cache is full are unchanged.

L4 24 GB, same prompt as above; upstream = master 84746c5, before = 4f025fa, after = 1967019. Generation t/s:

Model, cache	Tokens	upstream cache	PR hybrid before	PR hybrid after
Qwen3-Coder-Next UD-IQ4_XS (36 GB), 16 GB	256	38.5	21.1	29.1
	1024	46.6	32.7	42.6
Qwen3.5-35B-A3B Q6_K (27 GB), 16 GB	256	37.8	32.4†	34.3‡
	1024	50.5	47.0	50.1‡
Qwen3.5-35B-A3B IQ4_XS, 5000 MiB	256	33.5	41.4†	45.9

† on the earlier head 3d37f74. ‡ with the change one small edit before 1967019 (the periodic swap round now also runs while the cache fills).

• Generation t/s leaves out prompt processing, during which upstream's cache does its own loading (1.5-1.8 s here). In total time the hybrid matches upstream's cache on Qwen3.5-35B-A3B Q6_K: 21.4 s against 21.8 s for 1024 tokens.
• On Qwen3-Coder-Next (512 experts per layer) the hybrid still takes about 2 s longer at the start, while it uploads the experts after the prompt; over tokens 257-1024 both run at about 50 t/s.
• RTX 4060 Laptop, 4 GB cache, old and new code alternated: 40-42 vs 42-43 t/s at 256 tokens, 46-47 at 1024 for both.
• Accuracy is unchanged: L4 IQ4_XS, 20 chunks, hybrid + adaptive mean KLD 0.0040, max 0.13; test-llama-archs 402/402 on the L4.

Accuracy (KL divergence)

llama-perplexity --kl-divergence, -ub 1 so every token takes the hybrid path, 20 chunks of 512 tokens (the repo's docs). The base is the same build with -ngl 99 --cpu-moe and no cache.

Machine and model	Config	Mean KLD	Max KLD	99.9% KLD	Same top token
RTX 4060 Laptop, Q6_K, 2048 MiB cache	cache	0.00387	0.107	0.077	97.12%
	hybrid + adaptive	0.00380	0.113	0.086	97.22%
L4, IQ4_XS, 5000 MiB cache	cache	0.00411	0.166	0.070	97.20%
	hybrid + adaptive	0.00388	0.241	0.074	97.20%

Both configs differ from the base by the same small amount, which comes from running experts on GPU kernels instead of CPU kernels; the hybrid mode adds no divergence of its own on these runs.

Known limitation

• The hybrid path is only used for 1-token batches. MTP and other speculative decoding verify several tokens per batch, so they currently do not benefit from it, and with the hybrid flags they can run slower than with the plain cache.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - Assisted in formatting CLI argument parsing and test fixtures; all kernel logic and cache architectures have been reviewed, tested, and validated locally across test-llama-archs.

[ggml-gh-bot]

Hi @voidstackloop, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple backend changes in one PR: When adding support for a new model or feature, focus on CPU support only in the initial PR. Add support for other backends like CUDA in follow-up PRs. If you have a good reason to modify multiple backends in one PR, please explain it.

• Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[pwilkin]

/bot review

[ggml-gh-bot]

Automated code review

Code review: hybrid MoE expert cache (PR 30594)

I read pr.diff in full and explored the surrounding code (scheduler split logic, mmvq.cu/mmid.cu id handling, ggml_custom_4d, the MoE cache impl, the test harness). The mailbox hand-off protocol itself is well-constructed (epoch-stamped flags, payload-then-flag ordering with __threadfence_system, explicit set_tensor_backend pinning, safe fallback to the scheduler's copies when any precondition fails), and the profile parser validates its inputs correctly. But the change as a whole has scope and maintainability problems, plus a few concrete issues.

Blocking (scope / process)

(point 1) This is a very large, multi-area PR with no prior issue or discussion: it adds a new public ggml header (ggml/include/ggml-mailbox.h), a new public tensor flag (GGML_TENSOR_FLAG_NO_SYNC in ggml.h, consumed by the shared scheduler in ggml-backend.cpp), new CUDA kernels, CPU mul_mat_id behavior changes, 3 new llama_context_params fields and 5 new llama.h functions, 4 new CLI flags, a JSONL audit subsystem, server metrics, and a large test expansion - all in one PR. Per CONTRIBUTING.md this needed an issue/RFC first, and it should be split: (a) ggml mailbox infra + scheduler flag, (b) the hybrid/adaptive cache in llama, (c) audit/telemetry, (d) the CPU mul_mat_id chunking tweak. The prior bot checks (large PR, multiple backends) apply here.

(point 2) ggml-mailbox.h and GGML_TENSOR_FLAG_NO_SYNC are new public, installed ggml API surface written for one llama.cpp feature - the header's own comments describe llama.cpp's MoE cache. Other ggml consumers get this API and its maintenance burden permanently. Consider keeping it private (e.g. under ggml/src/) or, at minimum, open a maintainer discussion on whether the mailbox belongs in ggml at all vs. e.g. a backend-internal mechanism. Note the scheduler semantics of NO_SYNC are defined only implicitly (see point 6).

(point 3) The "Strata STRP" profile format references an external repo (extended-strata-moe's tools/moe_plan.py) that is not part of llama.cpp - there is no in-repo way to generate a profile, and the format is documented nowhere except the reader. A profile format the project parses but cannot produce is a maintenance and ownership question; the description should state who owns the format.

Correctness / robustness

(point 4) src/llama-moe-cache.cpp:869 - mailbox_split spin-waits 20 s and then GGML_ABORTs the whole process. In llama-server a transient GPU stall, a driver hiccup, or a GPU split that failed with a recoverable error (OOM) while the CPU split is already spinning will kill the server rather than surfacing the error. Consider a recoverable failure path (e.g. zero the cold rows, set an error flag, let the decode fail cleanly).

(point 5) ggml/src/ggml-cpu/ggml-cpu.c:1709 - static int mmid_rows is lazily initialized with getenv from inside the compute function, which runs concurrently on all threads: two threads can write the same non-atomic static simultaneously (UB, benign in practice but still a data race). Move the env read to a once-at-first-use atomic/thread-safe init, or read it at planning time. Also note this chunking (default 32 rows) now applies to every one-token-per-expert CPU mul_mat_id, not just the hybrid path - a global behavior change for all CPU MoE users based on measurements from a single CPU. That change deserves its own PR with broader perf data.

(point 6) The NO_SYNC mechanism (ggml-backend.cpp split loop) is safe today only because the hybrid CPU split reads exclusively the mailbox and host weights. Nothing in the flag or the scheduler enforces that: any future user of GGML_TENSOR_FLAG_NO_SYNC whose split reads an allocator-assigned tensor of the previous split gets a silent race. Since this is public API, the contract ("a NO_SYNC split must not read any tensor produced by the previous split except flag-ordered shared memory") needs to be spelled out in ggml.h on the flag itself, not just in a comment at the use site.

(point 7) src/llama-moe-cache.cpp:1269 - in the hybrid branch of copy(), every hit is counted as hits_profile (s->hits_profile += n_hit), including hits on experts that adaptive swaps put in later (from_profile[s] is correctly cleared there, but this path ignores it). The exported metric moe_cache_hits_profile_total / the audit field hits_profile will over-report profile hits in adaptive runs.

(point 8) src/llama-moe-cache.cpp do_rebalance() - reused victim slots are not re-linked in the SLRU lists (move() is never called), so a swapped-in expert inherits the victim's list position. Currently harmless because the hybrid never runs LRU operations, but it is a latent trap if the hybrid and the LRU paths ever interact (e.g. hybrid disabled per layer for a layer held whole while another layer in the same group still uses plan()). Worth a one-line comment or an explicit invariant that do_rebalance assumes no LRU consumers.

Will slow the review

(point 9) Leftover development tooling shipped in the diff: GGML_SCHED_PROFILE (global mutable profiling state + stderr print at every sched free, ggml-backend.cpp:1819), GGML_CPU_OP_PROFILE (atexit print, ggml-cpu.c:3146), and the LLAMA_MOE_CACHE_NO_MAILBOX escape hatch. These are debug aids used to produce the PR's benchmarks, not the feature - strip them or move them to a separate PR.

(point 10) Comments referencing "ADR-002" (src/llama-graph.cpp:2562, 2597) point at a document that does not exist in this repo and mean nothing to a reviewer; replace with the actual numbers.

(point 11) The new llama.h surface (3 llama_context_params fields, 5 functions, 2 stats structs) is all marked [EXPERIMENTAL] but lives in the stable header. Given src/llama-ext.h exists for experimental surface, the maintainers will likely ask why the cache stats/audit APIs are not routed there; be ready to justify or move them.

(point 12) tests/test-llama-archs.cpp adds 8 extra device configs per MoE arch on every GPU backend (and cb_eval is now set for all cached configs, which changes the eval path). This is a substantial CI-time increase on top of an already long binary - consider gating the newest configs behind the existing flag pattern (--all-host-experts style) or trimming to the ones that exercise distinct code paths (full-profile, hybrid-half, hybrid-1-token, adaptive).

(point 13) For plain (non-hybrid) --moe-cache-mib users this PR silently changes behavior twice: expert biases are now copied to the device (extra VRAM), and bias_bytes is charged against the cache budget (alloc_size(...) + g.bias_bytes <= budget), so the same --moe-cache-mib value now yields slightly fewer expert slots than before. Mention this in the PR description or release notes.

Nits

(point 14) ggml/src/ggml-cuda/ggml-cuda.cu:17 - #include "ggml-cuda/mailbox.cuh" is inserted between conv-transpose-1d.cuh and conv2d.cuh, breaking the alphabetical include order.

(point 15) ggml_backend_cuda_mailbox_buffer_type_name (ggml-cuda.cu) has GGML_UNUSED(buft) after the return - dead code.

(point 16) src/llama-graph.cpp non-mailbox hybrid path: x = cur->ne[1] == k ? cur : ggml_repeat_4d(...) - cur was just reshaped to [n_embd, 1, n_tokens], so cur->ne[1] == k is only true for k == 1; the condition looks like it guards a case that cannot happen. Clarify or drop it.

(point 17) llama_moe_cache_read_profile never validates or uses hdr[3] ("slots") of the STRP header - either check it or note in the comment that it is informational.

Overall: the engineering inside the mailbox and hybrid split logic is careful and the security-sensitive parsing is properly bounded, but this needs to be broken up and discussed with maintainers before it can be reviewed line-by-line; points 4, 5, 6 and 7 should be fixed in whatever form moves forward.

This review was generated automatically by pi coding agent using zai-org/GLM-5.3. It may contain mistakes. Maintainers make the final call.

[pwilkin]

@am17an @ggerganov this is a big PR, so I think it needs some rescoping or other suggestions.

[kroaton]

I tested this out and wrote some notes, maybe they'll be of some use:

• Hardware: RTX 4070 SUPER (12 GB) and a Ryzen 9 5950X.
• Models: Qwen3.6-35B-A3B (IQ4_XS, with MTP) and Flint Preview (Q4_K_XL, no MTP). Experts sit in host RAM with a 5.5–6 GB GPU cache.

Qwen3.6-35B-A3B: slower

• With MTP draft 2 (our normal setup) decode fell from about 125 t/s to 60 t/s. The parallel CPU+GPU path only handles 1-token batches. MTP verifies 3 tokens at a time, so it takes the slow path, where the GPU waits for the CPU at every layer.
• Without MTP: 85 t/s vs master's 101. The self-filling cache only reached 74.5% hits vs 88.5% for master's cache, and it was still warming when the run ended.
• Either way, MTP alone beats anything the PR can do on this model.

Flint Preview: faster, but I noticed perplexity is worse

• Speed: 61 t/s mean vs 57–58 for master, and 68 t/s (+19%) on the last rep, still warming. Flint has no MTP and misses a third of its cache lookups, so it's a great candidate for this technique.
• Quality: perplexity +5% (13.57 → 13.74, a 1.050 ± 0.015 ratio over the same text). Measured with ubatch 1 so every token goes through the hybrid path.
• KL divergence against master averaged 0.158, and the top token differed about 10% of the time.
• Most tokens match to rounding, but about 1 in 100 comes out badly wrong (KL divergence above 4). That looks like occasional wrong expert data rather than precision drift. No idea what the cause is for that issue.

I didn't do a perplexity test with Qwen as it didn't make sense to use this PR.