[rootsbymenda]

What this changes

The release fatbins are built with nvcc 13.0 (docs/kernel-contract.md), and on an R550 driver the first cuModuleLoadData of any of them ends the process with SIGILL inside libcuda (#1724). Driver.openPath now reads cuDriverGetVersion right after cuInit, before any image loads, and refuses a driver below 13000 with error.DriverTooOld and a message naming the CUDA version it runs and the driver needed:

error: this NVIDIA driver runs CUDA 12.4, and TensorFold's GPU kernels need CUDA 13.0: install NVIDIA driver R580 or newer

README and RUNBOOK state the floor in the Linux install text, before the download step.

The floor comes from NVIDIA's CUDA 13 release notes: Table 2 maps CUDA 13.0 to driver branch R580, and Table 3 gives >= 580 as the minimum for 13.x under minor version compatibility ("Existing CUDA 13.x applications run on drivers >=580"). So 13000 also admits images built from source with a later 13.x nvcc. An R580 driver reports exactly 13000; the GB10 below, on 580.178.04, does.

Your reply doesn't say what to do when the driver reports no version. This refuses it too, both a failed cuDriverGetVersion and a reported 0, because such a driver can't be shown to load the images, and the message still names the driver to install. It's one line to admit it instead if you prefer.

Every CUDA path opens the driver through openPath before module.zig loads an image, so the check sits in one place.

Fixes #1724

Stacked on: nothing (based on main fbbc20d0f).

Receipt

• Environment, with the TensorFold commit, MLX or PyTorch version, machine and GPU, checkpoint and revision: head 132310eea on main fbbc20d0f; Zig 0.17.0; kernels built with nvcc 13.0.88, the release compiler. x86_64: Ubuntu 24.04.5, RTX 5090 (sm_120), driver 595.99.02 (cuDriverGetVersion 13020). aarch64: DGX Spark, GB10 (sm_121), driver 580.178.04 (13000, exactly the floor). No checkpoint: the check runs before any model or image loads.
• Exactness, from tools/bench_concurrent.py --alone --serial or token_sha against "draft": false: n/a. No kernel, sum or token path changes; the check runs once when the driver opens.
• Decode speed before and after, from tools/bench_openai.py: n/a, for the same reason (one cuDriverGetVersion call at startup).
• Prompt speed before and after: n/a, because the change can't reach prompt processing.
• Host test (driver.zig, run by test-cuda-host and test): a fake cuDriverGetVersion drives Driver.tooOld. 12.4 and 12.9 are refused with the message; 13.0 and 13.2 are admitted; a reported 0 and a failed read are refused. Control: the same test against main's openPath, which has no floor, fails at the 12.4 case (expected the refusal, found admitted).
• Stand-in R550: a libcuda.so.1 shim that forwards to the real driver, reports a chosen cuDriverGetVersion, and raises SIGILL in cuModuleLoadData as R550 does. Results:

  tf-cuda-test smoke, reports 12.4, image loads trap   main: two copy checks pass, then SIGILL in cuModuleLoadData, exit 132
                                                       head: the refusal above, error.DriverTooOld, exit 1, no image loaded
  tf-cuda-test smoke, reports 12.9 / 0, loads trap     head: refused with the message (12.9 / "reports no CUDA version"), exit 1
  tf-cuda-test smoke, reports 13.0                     head: all 7 smoke checks pass, exit 0
  tensorfold-native capabilities --json, reports 12.4  head: the refusal on stderr, "chip": null, exit 0
  

• No false refusal on real drivers (the guarded binaries):

  RTX 5090, 595.99.02    tf-cuda-test info: driver CUDA 13020, sm_120, kernels embedded; smoke: 7/7 pass
                         tensorfold-native capabilities --json: "chip": "nvidia-sm120"
                         tf-cuda-test kernel-contract: a partial run, one 14-minute window (840 s, then the timeout):
                           246 results, all PASS, and 32 controls red as planted; 0 failures
  GB10, 580.178.04       tf-cuda-test info: driver CUDA 13000, sm_121, kernels embedded; smoke: 7/7 pass
                         tensorfold-native capabilities --json: "chip": "nvidia-sm121"
                         zig build kernel-contract -Dnvcc=... -Dsm=121, the full contract, 87 min, exit 1 from the limits alone:
                           SUMMARY 509 kernels: 509 pass gates 1-4, 0 fail, 6 known findings; 342 controls: 342 red as planted,
                           0 missed; 181 limit problems
  

  On the 5090 the contract is CPU-bound here (float64 references), so the full run didn't fit our GPU booking. The partial run still shows the guarded runner opening the real driver and loading every image it reached. The GB10 run is the complete one.
  • The 6 known findings are the ones registered on main.
  • The 181 limit problems are main's, in launches this change doesn't touch (it changes no kernel, record or contract file):
    • 168 launches with no compiled usage recorded: qwen4_hc 48, qwen4_ple 40, qwen4_projection 20, qwen4_gdn 20, qwen35 15, qwen35_kv8 15, nvfp4_experts_tile 10;
    • 13 live launches whose usage differs from the records: qwen35_kv8 6, qwen35 4, nvfp4_experts_tile 2, qwen35_keep 1.
  • Flash Next TP=2: exact output-column splits, 21% faster plain decode #1784, on the same main with nvcc 13.0 on a GB10, reports the same families.
• Tests run, with pass, fail and skip counts (x86_64 Linux unless noted):

  zig fmt --check build.zig zig                           exit 0
  git diff --check fbbc20d0f..HEAD                        exit 0
  python3 tools/zig/lean_check.py                         base 424 problems, head 424, new 0 (the same 424)
  zig build test --test-timeout 10m, empty cache          head: 5939/5967 passed, 27 skipped, 1 failed
                                                          main: 5938/5966 passed, 27 skipped, 1 failed (the same test)
  zig build test-cuda-host                                head: 537/538 passed, 1 failed (the same test)
  zig build test-kernel-contract                          exit 0, 138/138
  zig build test-golden                                   exit 0, 5/5 steps
  zig build test-server-security test-pull-security       exit 0, 28/28
  zig build fuzz-cortheon                                 exit 0, 19/19
  zig build                                               exit 0, 29/29 steps
  zig build -Dnvcc=... -Dsm=120 / zig build native        exit 0, 169/169 and 117/117 steps
  zig build -Dnvcc=... -Dsm=121 / zig build native (aarch64)  exit 0, 169/169 and 117/117 steps
  

  The one failure is inherited from main: exl3_load.test.a scan that fails an allocation frees every group prefix it held returns error.NondeterministicMemoryUsage from checkAllAllocationFailures on main fbbc20d0f as well.
• atc overlap --base fbbc20d0f: README.md is shared with open PRs; none of them conflicts with this change (checked with git merge-tree: lfm2_moe: serve LFM2.5-24B-A2B on CUDA (plain serve, shared kernels) #1492, lfm2_moe CUDA: attention pages from the shared KV pool, and the prompt cache (stacked on #1492) #1503 and TensorFold Web: a small local web app, and tensorfold web to open it #1781 conflict with main the same way with or without it).
• Not run, and why: an R550 driver itself, since we have neither that driver nor an sm_86 card; the crash is reproduced only through the stand-in above. Metal and HIP, which this CUDA-only change doesn't touch. A model serve, since no checkpoint path changes and the check runs before one loads.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. (Startup check only.)
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (Not measured; no token path changes.)
• No precision traded for speed. If the bits change, the description says which sums change and why. (No bits change.)
• A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt. (No kernel changes; the GB10 SUMMARY line is above.)
• Prompt processing is no slower than the last release. (Not reached by the change.)
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (The host test needs no driver.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[ashhart]

Thank you @rootsbymenda. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.