Skip to content

native: a CUDA family's prompt cache, weight share and open errors come from its own declarations - #762

Open
Bizuayeu wants to merge 1 commit into
ashhart:mainfrom
Bizuayeu:pr/native-family-hooks
Open

Bizuayeu wants to merge 1 commit into
ashhart:mainfrom
Bizuayeu:pr/native-family-hooks

Conversation

@Bizuayeu

@Bizuayeu Bizuayeu commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

The CUDA host wired Nemotron's prompt-cache copies, rules and budget in by name. They now come from the family's
snap_* declarations through cuda_snaps.zig: a family without snap_save gets no prompt cache, snap_ready sets
aside its state buffers, and snap_grid makes the store keep states only on the family's chunk starts (for a family
whose rows depend on where a prompt's chunks begin). A family that declares warm_quiet gets 1.0.9's warm turns
(.warm = .waits). Nemotron declares warm_quiet and no grid and keeps 1.0.9's rules: any row, the head one
token on, warm turns, the same budget.

Three more hooks for a family split over hosts: weightShare (the weights this rank holds, for the memory check),
explains_open (the family's own text for an open error) and the learner only for a family with slide. The
GLM-5.3-Flash family (#548) uses all of them.

Receipt

  • Environment: TensorFold 1.0.9 (c36b536d) plus this change; one GB10 (DGX Spark, sm_121, CUDA 13.3).
  • Exactness: Nemotron's open, prompt cache and --slide are unchanged (the same calls and rules, now through the
    family's declarations).
  • Decode and prompt speed: not applicable.
  • Tests run:
    • zig build test-cuda-host: 430 passed, 0 failed; the new "a family with snap_grid keeps on its chunk starts, any
      other at any row as Nemotron does" needs this change.
    • zig build test: 620 passed, 5 skipped, 0 failed. zig fmt --check: clean. lean_check: 538, as main.
  • Not run, and why: a served Nemotron run on CUDA (no Nemotron checkpoint on these machines); Metal (CUDA only). weightShare, explains_open and the slide guard are not covered by a host test: they are branches of openWith, which needs a GPU and a real checkpoint to reach. The GLM-5.3-Flash family's runs at TP=2 and TP=3 (Zig CUDA ports: shared layers below lanes.Backend, and one place to coordinate them #548) go through all three.

Checklist

  • Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
  • Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
  • No precision traded for speed. If the bits change, the description says which sums change and why.
  • Prompt processing is no slower than the last release.
  • The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
  • New tests fail before the change, pass after it, and skip cleanly without their dependency.
  • Comments and docstrings are one line. No measurements or history in the source.
  • No personal data, machine names, internal hosts or local paths. No AI attribution lines.

…me from its own declarations

The CUDA host wired Nemotron's prompt-cache copies, rules and budget in by name. They now come from the family's
`snap_*` declarations through `cuda_snaps.zig`: a family without `snap_save` gets no prompt cache, `snap_ready` sets
aside its state buffers, and `snap_grid` makes the store keep states only on the family's chunk starts (a family
whose rows depend on where a prompt's chunks begin), and `warm_quiet` turns on the quiet-turn reply prefill (warm
turns and the store's `.waits`). Nemotron declares `warm_quiet` and no grid and keeps 1.0.9's rules: any row, the
head one token on, warm turns, the same budget.

Three more hooks for a family split over hosts: `weightShare` (the weights this rank holds, for the memory check),
`explains_open` (the family's own text for an open error) and the learner only for a family with `slide`.
Nemotron's open, cache, warm turns and --slide are unchanged; `cuda_snaps.zig`'s test covers the rules each way.
@Bizuayeu
Bizuayeu force-pushed the pr/native-family-hooks branch from bf10491 to 185bc33 Compare October 13, 2026 19:23
@Bizuayeu

Copy link
Copy Markdown
Contributor Author

Rebased onto 1.0.9. Its Nemotron store gained warm turns (.warm = .waits, warm_turns), so cuda_snaps.cacheRules now sets .waits for a family that declares warm_quiet (Nemotron does) and warm_turns follows the same declaration; Nemotron's rules, budget and warm turns are 1.0.9's, and a family without warm_quiet keeps none. The test covers the three cases. Host tests 430/430, zig build test 620 passed, 5 skipped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant