Skip to content

native cuda: refuse --drafter, --speed-up and --prompt-cache-over-cap instead of ignoring them - #733

Closed
jschmied wants to merge 1 commit into
ashhart:mainfrom
jschmied:cuda-refuse-flags
Closed

jschmied wants to merge 1 commit into
ashhart:mainfrom
jschmied:cuda-refuse-flags

Conversation

@jschmied

Copy link
Copy Markdown
Contributor

What this changes

The native CUDA server accepted three flags it does nothing with, and started as if they had worked:

  • --drafter DIR: no CUDA family loads a separate drafter. A server started with --drafter /tmp came up and drafted with Nemotron's MTP head.
  • --speed-up FILE: the two-Mac link. No CUDA family reads it.
  • --prompt-cache-over-cap: on CUDA the prompt cache is clamped to the memory the weights and streams leave, so --prompt-cache-gib 200 --prompt-cache-over-cap started with an 87.1 GiB cache and said nothing.

Each is now refused at startup with one line, the way CUDA already refuses --learn and Metal refuses --drafter for families that don't load one:

tensorfold: --drafter loads a separate drafter, and this CUDA family drafts with its own MTP head: serve without --drafter
tensorfold: --speed-up links two Macs: serve without --speed-up on CUDA
tensorfold: --prompt-cache-over-cap lifts a Mac's 70% cap: on CUDA the prompt cache takes at most the memory the weights and streams leave, so serve without it

--drafter is refused per family. A family that loads a drafter declares pub const takes_drafter = true, so the 27B in #608 opts in with one line and Nemotron keeps refusing. The refusals move to native/cuda_refusal.zig, --learn's with them, which keeps cuda.zig under the 600-line limit.

Receipt

  • Environment: TensorFold 0efbd447 (1.0.8) plus this commit; one DGX Spark (GB10, sm_121, driver 580.173.02, CUDA 13); TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit.
  • Behaviour: on 0efbd447, serve with --drafter /tmp, --speed-up FILE, or --prompt-cache-gib 200 --prompt-cache-over-cap started and answered. With this commit each exits before loading, with the line above. --prompt-cache-gib 4 alone still starts.
  • Exactness, decode and prompt speed: unchanged (startup checks only; nothing after the model opens changes).
  • Tests run: zig build test and zig build test-cuda-host (25/25 steps), all passing. New tests: "flags with nothing to do on CUDA are refused, not ignored" and "--drafter opens only a family that takes one". Removing the --speed-up refusal fails them. lean_check.py: 539 findings, as on main.
  • Not run, and why: Metal (unchanged). The parity survey also suggests Metal Nemotron and Qwen3.5-2B ignore --speed-up, but I haven't run that, so this PR leaves Metal alone.

Checklist

  • Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
  • Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (Nothing after startup changes.)
  • No precision traded for speed. If the bits change, the description says which sums change and why. (No bits change.)
  • Prompt processing is no slower than the last release.
  • The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
  • New tests fail before the change, pass after it, and skip cleanly without their dependency.
  • Comments and docstrings are one line. No measurements or history in the source.
  • No personal data, machine names, internal hosts or local paths. No AI attribution lines.

…mily without takes_drafter, --drafter at startup instead of ignoring them; the refusals in cuda_refusal.zig
jschmied pushed a commit to jschmied/TensorFold that referenced this pull request Oct 13, 2026
@ashhart

ashhart commented Oct 13, 2026

Copy link
Copy Markdown
Owner

Thank you, @jschmied. A flag the server takes and then ignores is worse than one it refuses, and the CUDA refusals in one file, each with a test, make the next one easy to add. Your commit landed in 1.0.9 as 9cc1adea3: the same patch, author date and noreply address as this PR's head, picked onto our Mac-side --speed-up refusal (1edf4a2e2). Because the PR's own head commit isn't in main, GitHub won't mark this PR merged, so I'm closing it with this link.

One change of ours follows yours (dd37df3bc). family() admitted --drafter whenever a family declared takes_drafter at all, so a family that declared takes_drafter = false would have taken one. It now admits only a true declaration, and a test with a declining family failed before the fix and passes after it.

What we checked. On Linux (aarch64) the CUDA host tests and the whole zig build test pass, a real server refuses --learn, --prompt-cache-over-cap and --drafter before it loads anything, and a CUDA build's command line rejects --speed-up. On a Mac, Nemotron and Qwen3.5-2B now refuse --speed-up with the wording the 27B uses, and Flash Next, GLM-5.3-Flash and GLM-5.3 still take it. The 1.0.9 notes list these refusals under behaviour changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants