Skip to content

native: the CUDA lane backend forwards every entry a family's backend gives - #742

Merged
ashhart merged 1 commit into
ashhart:mainfrom
Bizuayeu:pr/cuda-backend-forwards
Oct 13, 2026
Merged

ashhart merged 1 commit into
ashhart:mainfrom
Bizuayeu:pr/cuda-backend-forwards

Conversation

@Bizuayeu

Copy link
Copy Markdown
Contributor

What this changes

The native CUDA host wraps a family's lane backend so that every call makes the device context current on the lane
thread. The wrapper lists the entries one by one, and it has no prefill_step: the lane core sees a backend that
cannot step, so a served prompt runs whole and the other streams wait for all of it. This moves the wrapper to
zig/src/native/cuda_bound.zig and forwards every optional entry the family gives (prefill_step,
prepare_features and features were missing) and its prompt_rows, with a test that walks the vtable's fields so
an entry added later fails until it is forwarded. cuda.zig goes from 600 to 517 lines.

No CUDA family on main implements prefill_step yet, so nothing on main changes behaviour; the GLM-5.3-Flash port in
#548 does, and found this.

Receipt

  • Environment: TensorFold 0efbd447 plus this commit; one GB10 (DGX Spark, sm_121, CUDA 13.3) for the host tests;
    two GB10s at TP=2 for the served run below.
  • Exactness: the wrapper only forwards; no kernel or sum changes.
  • Decode speed before and after: not applicable on main. On the GLM-5.3-Flash family (Zig CUDA ports: shared layers below lanes.Backend, and one place to coordinate them #548), one decode streaming
    beside a fresh 32k-token prompt at --parallel 2, over HTTP, two runs each; the longest wait between its tokens:
    27.9 s before, 1.7 s with this change (about one 2,048-row chunk), with 20 to 21 tokens during the prompt instead
    of 1 to 2. The prompt itself took 29.1 s instead of 27.8 s, the decode's rounds running between its chunks.
  • Prompt speed before and after: not applicable (no kernel change).
  • Tests run:
    • zig build test-cuda-host: 361 passed, 0 failed. The new test "the lane core sees a family's prefill_step
      through the lane backend" fails on main without the change (TestUnexpectedResult at its expect, 359 of 360
      passed) and passes with it.
    • zig build test: 326 passed, 5 skipped, 0 failed.
    • zig fmt --check zig/src/native: clean. lean_check: 539, the same as main; none in the two files.
  • Not run, and why: Metal (the change is in the CUDA host only).

Checklist

  • Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
  • Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
  • No precision traded for speed. If the bits change, the description says which sums change and why.
  • Prompt processing is no slower than the last release.
  • The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
  • New tests fail before the change, pass after it, and skip cleanly without their dependency.
  • Comments and docstrings are one line. No measurements or history in the source.
  • No personal data, machine names, internal hosts or local paths. No AI attribution lines.

… gives

The native CUDA host wraps a family's backend so each call makes the device context current on the lane thread,
and the wrapper lists the entries one by one. It had no prefill_step, so the lane core saw a backend that cannot
step: a served prompt ran whole, and a decode beside a long prompt waited for all of it. The wrapper moves to
cuda_bound.zig and forwards every optional entry the family gives (prefill_step, prepare_features and features were
missing) and its prompt_rows; a test walks the vtable's fields, so an entry added later fails it until forwarded.
@ashhart
ashhart merged commit 6713d22 into ashhart:main Oct 13, 2026
2 checks passed
jschmied pushed a commit to jschmied/TensorFold that referenced this pull request Oct 13, 2026
@ashhart

ashhart commented Oct 13, 2026

Copy link
Copy Markdown
Owner

Thank you, @Bizuayeu. A wrapper that lists the entries one by one was a trap waiting for the first CUDA family that steps its prompt, and the test that walks the vtable's fields closes it for good. Your commit landed in 1.0.9 as 6e7d798ad, under your name, and the PR's head is merged into the history with no change to the tree (6713d22a7), so GitHub shows this PR as merged.

One conflict was ours to resolve. #733 moved the --learn refusal and its test from cuda.zig into cuda_refusal.zig, so cuda.zig's test block now keeps both _ = refusals; and your _ = lane_backend;, plus your new prefill_step test. The landed commit's added and removed lines are exactly your PR's.

What we ran:

  • Host tests. zig build test-cuda-host passes on macOS and on Linux (aarch64) with your two new tests, and so does the whole zig build test on Linux.
  • Served, GB10. Nemotron 3.5 Lightning MLX 4-bit gave the same reply drafted and with "draft": false in all 17 release cells, with the same token SHAs as before your change, both with the captured Triton set and with our own kernels. That is what we expected: no CUDA family in this release sets prefill_step, prepare_features or features, so the newly forwarded entries stay null until one does.

A small follow-up if you feel like it: the field-walking test checks that each forwarded entry is present exactly when the family gives it, so a pointer aimed at the wrong inner entry would still pass. A test that calls each forwarded entry once through the fake would catch that.

@Bizuayeu

Copy link
Copy Markdown
Contributor Author

Thank you for landing it. The follow-up is #800: it hands cuda_bound.vtable a backend that records which of its
entries ran, calls each forwarded entry once and checks that the entry of the same name ran, once. With
unspeculate's forwarder pointed at the inner prefill, it fails while the field-walking test still passes.

@ashhart ashhart mentioned this pull request Oct 14, 2026
@Bizuayeu
Bizuayeu deleted the pr/cuda-backend-forwards branch October 14, 2026 11:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants