Repository navigation
EXL3: share decoded weights across prefill rows - #729
Conversation
…3 experts Checked ash hart PRs open+closed (read-only). Direct hits on our problems, all OPEN, branch BHCC2025/TensorFold exl3-cuda-experts: - ashhart#741 EXL3 routed experts behind experts.Grouped: gate+up+SwiGLU ONE kernel, down on pair rows, each expert's rows rotated by its own suh INLINE (register-resident rot for 1-2 pair chunks - the problem we had deemed impossible tile-incrementally). Rides on grouped.zig's Plan (in our tree). 297cu+124zig lines, with an exactness test harness (tf-cuda-test exl3-experts) - 367,872 values within FP64 bounds, alone==grouped byte-equal. - ashhart#740 EXL3 projections behind qlinear (the stacked base). - ashhart#718 branch-free trellis window extraction (perf). - ashhart#729 share decoded weights across prefill rows (perf). - ashhart#728 MiMo-V2.6 CUDA EXL3 two-GPU (another full CUDA EXL3 family). - ashhart#765 GLM CUDA multi-rank doorbells; ashhart#770/777/768 nemotron CUDA single-launch shared rounds + batched MTP drafting (patterns for our later concurrency work). PLAN: vendored exl3_experts.cu/.zig + exl3.cuh into our tree (this commit, unwired). Next: build wiring (fatbin + mod), port moeBlock to Grouped.decode with the router->plan plumbing from grouped.zig, run their exl3-experts test on our checkpoint, parity, bench. Our own grouped kernels (stage-1 parity) stay as fallback.
|
Thank you, @mrv777, the guarded 128-thread prefill path shares decoded weights while retaining the older path for unsupported work. We checked 8eaf366. exl3_projection.zig:154-161 rejects an unsuitable optional pipeline, and :328-337 selects it only for plain, non-indexed, non-routed windows of at least 32 rows. This is ready as the Metal part of the shared EXL3 composition, with your contribution and @vcruz305's original work credited. We will retain row/output, guard, gather and cache controls and qualify the final source before family integration. Please keep the synthetic timing workload identified separately from model throughput; the shared CUDA selection proceeds through #837/#741/#806 on equal tensors and bits. Posted by Ash's AI Assistant Synthena |
|
Thank you, @mrv777, for sharing decoded EXL3 tiles across prefill rows and retaining the row-count checks. We merged #729 for 1.1.0 in f12fcc5. The Metal projection now uses the same checkpoint scan and parts as CUDA, with packed sign planes retained in c56cb1a. The reported M3 Ultra operator run passes 21/21 checks; 32, 33 and 128-row comparisons retain bit equality with the eight-row kernel on M3 and M5 Ultra. Metal family adoption remains separate work. Apple M3/M5 Ultra: 32/33/128-row comparisons had zero mismatches; whole-model Metal EXL3 adoption remains separate. Posted by Ash's AI Assistant Synthena |
…lamaV3's notice One line for the CUDA module (ashhart#837, ashhart#806, ashhart#741, ashhart#740) and one for Metal's shared prefill tiles (ashhart#729). The third-party notice names the native scan, the CUDA kernels and their Zig side beside the decoders it already lists, under the same Turboderp attribution.
2 participants
Decode each weight tile once for four eight-row SIMDgroups in larger plain prefill windows. Preserve each row’s compensated arithmetic, scratch layout and output bits.
Use the 128-thread kernel only when its pipeline supports that capacity. Smaller windows, filtered/indexed work and unavailable pipelines retain the existing path. Keep the benchmark’s arithmetic reference independent of the new kernel.
M1 Max validation on #718 (
4dbeca49), 32-core GPU/32 GB, macOS 26.7, Zig 0.17.0:0efbd447); the benchmark compiles.Depends on #718, included as a separate commit; #681 has merged. This follow-up commit contains the prefill change. Lean check: 534 inherited findings, zero added. Timings use resident synthetic mul1/3-bit BF16 projections with packed-sign scales. Other chips and model throughput are unmeasured. Raw logs stay local.