Repository navigation
MoE hot experts: batch the hot experts of a layer into one device call #35
Description
Problem
With hot experts on RNGD (#27), Qwen3-30B-A3B training runs about 615 s/step against about 195 s/step host-only: each hot expert is called separately with its own upload and download.
Evidence
#27 verification (10 steps, 7582 s including about 1100 s of first-time compiles and 333 s setup).
Scope
Group the routed tokens of all hot experts of a layer into one device graph call (padded buckets); keep the CPU path for cold experts.
Done when
Hot-expert training is faster than host-only on the same run, with loss matching #27's CPU check.
Owner
gspark-etri
Activity
Record: batched hot experts on RNGD vs per-expert vs host-only (#35)
STATUS: #35 result
MoE hot experts: one batched device call per layer (#38)
gspark-etri commentedon Oct 21, 2026
PR #38 merged. Hot experts on RNGD now run as one batched device call per layer: 13.0 s/step against 42.4 s with per-expert calls, at parity with host-only (13.5 s, S=128). Staying open for the 'faster than host-only' part. Next levers per the record: more rows per call (longer S), E=64 hot fraction, and dropping HostMLP.backward's recomputed forward.
Record: profile and keep_graph A/B, hot experts beat host-only (#35)
gspark-etri commentedon Oct 21, 2026
The profile and A/B results are in #44 (record:
experiments/2026-10-21-moe-batched-hot-experts.md, follow-up section).HostMLP.backwardrecomputed the MoE block forward, about a third of the host MoE time. With the forward graph kept, hot experts + keep_graph run at 5.19 s/step, against host-only + keep_graph at 8.00 s andmainhost-only at 9.21 s. Setup: Qwen3-30B-A3B, S=128, hot fraction 0.25, RNGD, 6 round-robin rounds at host load 53. The paired median is −2.66 s (−33%), negative in 6 of 6 rounds.At host load 120–150 (#38) the two paths were at parity, so the gain depends on how busy the shared host is. E=64 and S=256/512 were not tried.