Repository navigation
native: the CUDA lane backend forwards every entry a family's backend gives - #742
Conversation
… gives The native CUDA host wraps a family's backend so each call makes the device context current on the lane thread, and the wrapper lists the entries one by one. It had no prefill_step, so the lane core saw a backend that cannot step: a served prompt ran whole, and a decode beside a long prompt waited for all of it. The wrapper moves to cuda_bound.zig and forwards every optional entry the family gives (prefill_step, prepare_features and features were missing) and its prompt_rows; a test walks the vtable's fields, so an entry added later fails it until forwarded.
|
Thank you, @Bizuayeu. A wrapper that lists the entries one by one was a trap waiting for the first CUDA family that steps its prompt, and the test that walks the vtable's fields closes it for good. Your commit landed in 1.0.9 as One conflict was ours to resolve. #733 moved the What we ran:
A small follow-up if you feel like it: the field-walking test checks that each forwarded entry is present exactly when the family gives it, so a pointer aimed at the wrong inner entry would still pass. A test that calls each forwarded entry once through the fake would catch that. |
|
Thank you for landing it. The follow-up is #800: it hands |
What this changes
The native CUDA host wraps a family's lane backend so that every call makes the device context current on the lane
thread. The wrapper lists the entries one by one, and it has no
prefill_step: the lane core sees a backend thatcannot step, so a served prompt runs whole and the other streams wait for all of it. This moves the wrapper to
zig/src/native/cuda_bound.zigand forwards every optional entry the family gives (prefill_step,prepare_featuresandfeatureswere missing) and itsprompt_rows, with a test that walks the vtable's fields soan entry added later fails until it is forwarded.
cuda.ziggoes from 600 to 517 lines.No CUDA family on main implements
prefill_stepyet, so nothing on main changes behaviour; the GLM-5.3-Flash port in#548 does, and found this.
Receipt
0efbd447plus this commit; one GB10 (DGX Spark, sm_121, CUDA 13.3) for the host tests;two GB10s at TP=2 for the served run below.
lanes.Backend, and one place to coordinate them #548), one decode streamingbeside a fresh 32k-token prompt at
--parallel 2, over HTTP, two runs each; the longest wait between its tokens:27.9 s before, 1.7 s with this change (about one 2,048-row chunk), with 20 to 21 tokens during the prompt instead
of 1 to 2. The prompt itself took 29.1 s instead of 27.8 s, the decode's rounds running between its chunks.
zig build test-cuda-host: 361 passed, 0 failed. The new test "the lane core sees a family's prefill_stepthrough the lane backend" fails on main without the change (
TestUnexpectedResultat itsexpect, 359 of 360passed) and passes with it.
zig build test: 326 passed, 5 skipped, 0 failed.zig fmt --check zig/src/native: clean.lean_check: 539, the same as main; none in the two files.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.