[am17an]

Overview

Cont #29887, adding multi-GPU support. Benchmarks on 2x 4090s

Performance

2x RTX 4090 (layer split), Qwen3.8-Flash-Next Q4_0 (93.7 GiB, 65 GiB of experts), EPYC 7742 (16 threads, one NUMA node)
SPEED-Bench qualitative, 1 sample per category (11 prompts), --osl 512, temperature 0, concurrency 1

config	decode t/s	vs master	prompt t/s	vs master	cache hit rate (decode)
master --fit	34.78	1.00x	440.7	1.00x	-
--fit --moe-cache-mib 6544	60.38	1.74x	402.0	0.91x	85.9%
-cmoe --moe-cache-mib 15000	64.39	1.85x	321.5	0.73x	95.8%

--moe-cache-mib is per GPU. With --fit, CUDA0 holds 15 full layers, so only CUDA1 has host experts and a cache. With -cmoe, CUDA0 caches 25 layers and CUDA1 caches 23.

Additional information

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES

[pwilkin]

CI is failing on build:

/__w/llama.cpp/llama.cpp/src/llama-moe-cache.cpp: In constructor 'llama_moe_cache::impl::impl(const llama_model&, const std::vector<ggml_backend*>&, const std::vector<ggml_backend_buffer_type*>&, size_t)':
/__w/llama.cpp/llama.cpp/src/llama-moe-cache.cpp:225:34: error: missing initializer for member 'llama_moe_cache::impl::device::ctx' [-Werror=missing-field-initializers]
  225 |                 devices.push_back({ backends[i], bufts[i] });
      |                 ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~
/__w/llama.cpp/llama.cpp/src/llama-moe-cache.cpp:225:34: error: missing initializer for member 'llama_moe_cache::impl::device::buf' [-Werror=missing-field-initializers]

[vbooka1]

93.7 GiB, 65 GiB of experts

pardon my ignorance, how do I calculate the weight of experts only?

[am17an]

@vbooka1 if you run with --moe-cache-mib there should be log line with the amount of experts added.

llama.cpp/src/llama-moe-cache.cpp

Line 366 in 2de17c2

LLAMA_LOG_INFO("%s: %10s MoE cache size = %8.2f MiB for %.2f MiB of host experts\n", __func__,

[Green-Sky]

per device

Would it not be better to have a comma seperated list to support heterogene setups?

[ggerganov]

Or maybe the best would be to specify the cache size "per layer"? I.e. a single number that has the meaning of the "moe cache size for a layer". This way, each device will allocate moe cache based on how many layers it works on.

[am17an]

Per layer leaves out an optimisation opportunity to share slots between layers which have the same expert types, some layers can use more experts than others.

[ggerganov]

some layers can use more experts than others.

Is this something we know for sure? I would expect expert usage to be rather uniform across the different layers.

[am17an]

Yes it increases the hit rate by 5% (it's already implemented in the implementation)

[ggerganov]

So do we keep the CLI arg like this, or do we do it a list of sizes for each device? The latter would be more complicated in terms of UX.

[am17an]

We should support it per device, one single value can mean it's that value per device. Comma separated values can be per device values. What do you think?

[ggerganov]

There is also the option to have a single global cache size (i.e. not per-device) and use the existing --tensor-split to infer how to split the cache among the devices. Would be inline with the rest "per-device" splitting logic.

[am17an]

Yeah that's also good, more intuitive I guess

[Yozam-87]

Please consider supporting per-device / comma-separated values (like --moe-cache-mib 512,4096)!
On asymmetric multi-GPU setups (e.g. a modern fast GPU for Flash Attention with -ts 1,0 and an older secondary GPU with lots of free VRAM), coupling the cache to --tensor-split makes the MoE cache unusable on the secondary card. With -ts 1,0, the secondary card gets 0 MB cache, even though it has gigabytes of free VRAM that could easily cache CPU experts without taking on attention layers.

[vbooka1]

Does this patch apply to Qwen-Next family only? GLM5.3 Flash both PP and TG became slower, PP about 10% slower and TG became 200% slower, 33 -> 16 t/s

0.33.942.823 I load_tensors:        CUDA0 model buffer size = 91667.78 MiB
0.33.942.824 I load_tensors:        CUDA1 model buffer size = 81986.62 MiB
0.33.942.825 I load_tensors:    CUDA_Host model buffer size = 104650.31 MiB
...
0.55.262.083 I impl:      CUDA1 MoE cache size =  9990.75 MiB for 99648.00 MiB of host experts
...
5.29.027.169 I llama_moe_cache: ubatch <= 8: hits = 50785, misses = 65842, hit rate = 43.54%, uploaded = 1423833.25 MiB
5.29.027.175 I llama_moe_cache: ubatch  > 8: hits = 120, misses = 1619, hit rate = 6.90%, uploaded = 35010.88 MiB

[sasa7812]

@Yozam-87 the way it is made now, from what I see, if you push the cache to the second GPU, the layers cached on that GPU will also get attention. If this is not what you want, you probably should be good with ts 1,0.

[BriNoB]

@Yozam-87 the way it is made now, from what I see, if you push the cache to the second GPU, the layers cached on that GPU will also get attention. If this is not what you want, you probably should be good with ts 1,0.

On my setup model doesn't even load with ts 1,0, if I would be able to have second value - I would be able to use cache. I have 16gb and 8gb GPUs - qwen3.8-flash-next

[mediaogre]

Thanks for #30112. It addresses the per-device sizing request from the #29887 thread. I haven’t enabled the cache yet. Everything below comes from reading the merged code against my layout before spending a restart on it: 3× RTX 3090, Qwen3.8-Flash-Next with about 22 expert layers on the CPU (--n-cpu-moe plus -ot pins), and MTP speculative decoding with verify batches of up to 4 tokens.

1. Miss cost on a host-offload rig. A miss is an upload over PCIe, while the path it replaces is the CPU computing the expert from system RAM. For a 2.1 MiB expert I estimate about 0.17 ms for the upload (Gen4 x8) against about 0.04 ms for the DRAM read, so a miss costs roughly 4× the existing path and the breakeven hit rate comes out near 79%. With the 1–2 GiB per card I can spare, I’d expect to land well below that, so on this layout the cache looks net-negative.

2. Request: an option to compute misses on the CPU, where the weights already live, instead of uploading them, with admission by frequency or hysteresis rather than on every miss. On a host-offload rig, any hit rate above zero would then be a gain.

3. Small documentation ask. When the budget can’t hold n_expert_used slots, the code warns and sets n_slots to zero, so the cache turns itself off. That’s the right behavior, and it would help if the --moe-cache-mib help text said so, since the alternative reading is that an undersized value is unsafe.

The log_stats() counters look like the right way to measure this. If a CPU-on-miss mode lands, I’m happy to run it on this rig and report per-device hit rates and decode speed at depth.

[sasa7812]

1. Miss cost on a host-offload rig. A miss is an upload over PCIe, while the path it replaces is the CPU computing the expert from system RAM. For a 2.1 MiB expert I estimate about 0.17 ms for the upload (Gen4 x8) against about 0.04 ms for the DRAM read, so a miss costs roughly 4× the existing path and the breakeven hit rate comes out near 79%. With the 1–2 GiB per card I can spare, I’d expect to land well below that, so on this layout the cache looks net-negative.

@mediaogre
In theory yes, in practice, however, you want this missed expert to end up on VRAM cache (otherwise you can just -ot whatever fits in the VRAM), you need to start CPU execution and VRAM cache loading in parallel, so both DMA to VRAM and CPU will be competing on same addresses of RAM, then transferring back the result of CPU computation pretty much breaks even with what it is now. But really depends on hardware, on some system you'll get slightly better speed, on some slightly worse.

#30594 (very unlikely to land) is probably what you up for, but as I said, I don't expect drastic speedup.

Also, look here #30637 , author tested the approach, it didn't behave well enough.

You have different problem to tackle with cache if I got your intent correctly from "With the 1–2 GiB per card I can spare", you probably don't want to spread you cache even across 3 GPUs as expert layers currently get assigned to GPU and you'll get cache fragmentation ditching your hit rate.

[eapache]

As the author of #30637, I will note: a hybrid approach (allowing certain experts to execute directly on CPU instead of ever going to cache) was one of the implementations I tested that performed about 5% better in some systems but ~2% worse on other systems. I didn’t do a deep investigation but I think the ratio between RAM speed and PCI speed needs to be big enough to make that approach worthwhile, which is of course not guaranteed (e.g. if you have a consumer DDR5/PCIe5 system and the memory controller drops your RAM at all below JEDEC speeds because of a sync issue then your RAM and PCI speeds become almost equal). My PR was the best I found that didn’t regress on any of the setups I tested.

Edit: there might be room to allow the user to specify a custom approach, or to choose an approach dynamically based on the system specs (akin to —fit) but that would of course add a lot of complexity to the code.

[BriNoB]

As the author of #30637, I will note: a hybrid approach (allowing certain experts to execute directly on CPU instead of ever going to cache) was one of the implementations I tested that performed about 5% better in some systems but ~2% worse on other systems. I didn’t do a deep investigation but I think the ratio between RAM speed and PCI speed needs to be big enough to make that approach worthwhile, which is of course not guaranteed (e.g. if you have a consumer DDR5/PCIe5 system and the memory controller drops your RAM at all below JEDEC speeds because of a sync issue then your RAM and PCI speeds become almost equal). My PR was the best I found that didn’t regress on any of the setups I tested.

Edit: there might be room to allow the user to specify a custom approach, or to choose an approach dynamically based on the system specs (akin to —fit) but that would of course add a lot of complexity to the code.

On my system - whenever I have cache on second GPU (PCIe 3 x4 link) - my generation drops, so it would just make sense to instead execute those experts on CPU, I have ddr4 ram. I'd like to contribute, let me know what angle is useful.

[eapache]

On my system - whenever I have cache on second GPU (PCIe 3 x4 link) - my generation drops

In my asymmetric multi-GPU system I have found a similar thing, and found it best to force full layers to that GPU instead. For example I have found the following to work best on my 4500+3060 system for Qwen3.8-Flash-Next:

ngl=49                                                                           
tensor-split=46,3                                                                
n-cpu-moe=46                                                                     
moe-cache-mib=16384  

This puts three full layers on the 3060 (which fills it ~completely), the non-expert parts of the remaining layers on the 4500, and then moe-cache-mib has to place the full cache on the 4500 which is fast enough to handle it. I get 40-45 t/s this way.

I'd like to contribute, let me know what angle is useful.

(I am just a contributor, not a maintainer, so take all this with some caution).

As a rule of thumb, it seems to be fairly easy to find changes that will speed up one specific hardware configuration, but much harder to find changes that will speed up all hardware configurations (or at least, speed up some configs without causing regressions on other configs). Since you have an asymmetric multi-GPU system this is pretty useful for testing since you can have llama.cpp running against just one GPU, just the other GPU, or both GPUs together, and get three very different performance configurations.

There are lots and lots of ideas floating about (#30059 has a big list, for example). Probably the most valuable thing to do here is to pick one or two ideas, and test them in a bunch of configs to see if any of them provide a useful improvement without any regressions. And of course paying attention to how complex it is... I imagine the llama.cpp folks won't accept a PR for 0.5% performance improvement if it takes 10k lines of gnarly code.

[Green-Sky]

There are lots and lots of ideas floating about (#30059 has a big list, for example). Probably the most valuable thing to do here is to pick one or two ideas, and test them in a bunch of configs to see if any of them provide a useful improvement without any regressions. And of course paying attention to how complex it is... I imagine the llama.cpp folks won't accept a PR for 0.5% performance improvement if it takes 10k lines of gnarly code.

Said pr is actually pretty small. ( <1k ) I dont have a multi-gpu setup, but that pr has given me consistent 50% uplift, so I am kinda stuck on that pr for now.

[eapache]

that pr has given me consistent 50% uplift

Compared with the latest master of llama.cpp including #30637? That is surprising to me, they are within about ~5% in my tests, so I would love to see what is special about your system or config that #30059 performs so much better.

[Green-Sky]

that pr has given me consistent 50% uplift

Compared with the latest master of llama.cpp including #30637? That is surprising to me, they are within about ~5% in my tests, so I would love to see what is special about your system or config that #30059 performs so much better.

Compared to before your 50/50 pr. Also i might be misremembering and it was 30%.

Will do some speed benches when I find some time.

My setup has a very "weak" cpu and plenty of pcie. Im running a rtx 2070.