Repository navigation
Conversation
…mily without takes_drafter, --drafter at startup instead of ignoring them; the refusals in cuda_refusal.zig
|
Thank you, @jschmied. A flag the server takes and then ignores is worse than one it refuses, and the CUDA refusals in one file, each with a test, make the next one easy to add. Your commit landed in 1.0.9 as One change of ours follows yours ( What we checked. On Linux (aarch64) the CUDA host tests and the whole |
What this changes
The native CUDA server accepted three flags it does nothing with, and started as if they had worked:
--drafter DIR: no CUDA family loads a separate drafter. A server started with--drafter /tmpcame up and drafted with Nemotron's MTP head.--speed-up FILE: the two-Mac link. No CUDA family reads it.--prompt-cache-over-cap: on CUDA the prompt cache is clamped to the memory the weights and streams leave, so--prompt-cache-gib 200 --prompt-cache-over-capstarted with an 87.1 GiB cache and said nothing.Each is now refused at startup with one line, the way CUDA already refuses
--learnand Metal refuses--drafterfor families that don't load one:--drafteris refused per family. A family that loads a drafter declarespub const takes_drafter = true, so the 27B in #608 opts in with one line and Nemotron keeps refusing. The refusals move tonative/cuda_refusal.zig,--learn's with them, which keepscuda.zigunder the 600-line limit.Receipt
0efbd447(1.0.8) plus this commit; one DGX Spark (GB10, sm_121, driver 580.173.02, CUDA 13);TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit.0efbd447,servewith--drafter /tmp,--speed-up FILE, or--prompt-cache-gib 200 --prompt-cache-over-capstarted and answered. With this commit each exits before loading, with the line above.--prompt-cache-gib 4alone still starts.zig build testandzig build test-cuda-host(25/25 steps), all passing. New tests: "flags with nothing to do on CUDA are refused, not ignored" and "--drafter opens only a family that takes one". Removing the--speed-uprefusal fails them.lean_check.py: 539 findings, as onmain.--speed-up, but I haven't run that, so this PR leaves Metal alone.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (Nothing after startup changes.)