Repository navigation
Conversation
…ecodeF32 and promptF32) A tensor-parallel family sums its ranks' partial rows before rounding, so it needs a projection's fp32 sums as the kernel holds them. qmmf.cu and prompt16.cu already take F32 as a template argument; this instantiates FP4's fp32 tiles (and reduce_kernel<true> for the split path), loads them beside the bf16 ones, and adds Lane.matmulOut, Prompt.matmulOut and qlinear's decodeF32 and promptF32. Other formats refuse fp32 out with EpilogueNotBuilt, as the experts do. matmul, decode and prompt keep their signatures and bf16 bits.
…n block The GLM-5.3-Flash CUDA port reads nvidia/GLM-5.3-Flash-NVFP4 as stored. admit() checks config.json's quantization_config (top level, else text_config's) for quant_method modelopt, quant_algo NVFP4 and 4-bit float weights in groups of 16, and refuses anything else with a reason the caller logs (the Why of nemotron/config.zig), so a test reads the reason instead of failing on an error log. MIXED_PRECISION is refused for now. Which linears are NVFP4 is left to the tensors, not to the block's ignore list: the pinned revision 423acf37 stores the MTP layer's experts in BF16 without listing layer 45 there (da920bb0 added it). The fixture is that revision's config.json.
nvfp4.zig is the format the GLM-5.3-Flash CUDA port's W4A16 linears read: e2m1 codes two to a byte (input 2j in byte j's low nibble) [n, k/2], an e4m3 scale per 16 inputs [n, k/16], and weight_scale_2 as an fp32 scalar that multiplies. dims() checks the four tensors fit together; Linear.read() maps them from a Checkpoint and marks input_scale as used, since the W4A4 math's static input scale is not read by W4A16 but a leftover tensor would refuse the checkpoint; row() gives a row as e2m1 x e4m3 x global in f32, the reference the GPU kernels will be compared with. stored() says a module is NVFP4 by its weight_scale tensor. The e2m1 and e4m3 values come from core.quant's nvfp4 and fp8 (ashhart#730), as f32. The fixture is the whole header of nvidia/GLM-5.3-Flash-NVFP4 (147,661 names, dtypes and shapes, gzip'd): every NVFP4 linear reads, 42 MoE layers x 288 experts x 3 plus the three dense MLP layers' 3, all under .mlp., and the MTP layer's experts stay BF16.
…e with them A GLM-5.3-Flash rank reads about half of the 183 GB checkpoint through the mapping, and its column cuts touch whole tensors. On GB10's unified memory the page cache holds memory the rank's device allocations need (our Python serving engine drops it with posix_fadvise for the same reason). safetensors.File.release drops a file's pages from the mapping (madvise) and the page cache (fadvise DONTNEED); Checkpoint.release does every file. The mapping stays valid and refaults on a later read, which a test checks. The GLM-5.3-Flash CUDA family (ashhart#548) calls it after each layer it loads.
…me from its own declarations The CUDA host wired Nemotron's prompt-cache copies, rules and budget in by name. They now come from the family's `snap_*` declarations through `cuda_snaps.zig`: a family without `snap_save` gets no prompt cache, `snap_ready` sets aside its state buffers, and `snap_grid` makes the store keep states only on the family's chunk starts (a family whose rows depend on where a prompt's chunks begin), and `warm_quiet` turns on the quiet-turn reply prefill (warm turns and the store's `.waits`). Nemotron declares `warm_quiet` and no grid and keeps 1.0.9's rules: any row, the head one token on, warm turns, the same budget. Three more hooks for a family split over hosts: `weightShare` (the weights this rank holds, for the memory check), `explains_open` (the family's own text for an open error) and the learner only for a family with `slide`. Nemotron's open, cache, warm turns and --slide are unchanged; `cuda_snaps.zig`'s test covers the rules each way.
…skip), as the decode form does A family that computes one expert itself (GLM-5.3-Flash stacks its BF16 shared expert as the last expert id) plans every pair together and has the grouped kernels skip that id. The decode form already takes skip; Grouped.prompt refused it with OptionNotBuilt for NVFP4. The prompt entries now take skip and return before staging any rows of that expert's items. Nemotron passes -1, so its bits do not change. quant-ref now checks it: with the shared expert skipped, its pairs' prompt rows keep what was there and every other pair's row equals the run without skip, for fp8 and nvfp4.
packGateUp packed each matrix into a scratch half and copied the halves into their interleaved places, two allocations and a copy per expert. A loader packs every routed expert of every layer this way (GLM at TP=2: 45 x 256 experts, 2.4 MB a half), and the fresh pages and the copies put about 20 s on the load. packOne's loop now takes the matrix's place among the stacked parts, so packGateUp writes gate and up straight where they go, and it needs no allocator. The words are the same: a host test checks packGateUp against packOne's blocks interleaved.
… over hosts The family ashhart#593 tracks, on the shared NVFP4 layer (`qlinear`, `experts.Grouped`) and hand-written `.cu` for the rest: KDA and MLA/DSA attention, the hyper-connection glue, MoE routing, the MTP draft head and sampling. Rank 0 serves through `tensorfold-native serve` and the other ranks follow its steps; each rank holds its share of every axis (uneven where a width doesn't divide). A prompt piece with at least as many rows as ranks is split over them (`TF_GLM_PREFILL_REDUCE=gather` all-gathers it instead): each rank sends the others their rows' partials and receives its own over NCCL send and receive (`nccl.zig` binds `ncclSend` and `ncclRecv`), and the hyper-connection step adds them in rank order. Drafts and concurrency go through the lanes core: `prefill_step` takes a prompt a chunk at a time between the other streams' rounds, and `hidden_rows` lets several streams' verify windows share one forward. The prompt cache keeps states on the family's 2,048-row chunk starts (`snap_grid`). Checks in `tf-cuda-test`: `glm-ref` holds every projection whose reduction order is the kernels' own against float64 on synthetic inputs; the lanes checks (cover, concurrent, shared) run at TP over hosts and, with `glm-synth`'s two-layer checkpoint, on one GPU.
… EXL3, 64 otherwise) An EXL3 matrix splits only at 128 columns, so at TP=4 the head's 154,880 rows go 303, 303, 302 and 302 blocks of 128 instead of 605 units of 64 each. `Config.head_unit` is 64 by default and 128 when `quantization_config.quant_method` is `exl3`; `Share.of` cuts the vocabulary in it. NVFP4 keeps 64, so its shares at TP=2 and TP=3 are unchanged. The patch BHCC2025 posted on ashhart#593, with its test of the TP=4 and TP=2 shares.
…e width (TP=3's last rank, 640) At TP=3 the routed and shared experts' 2,048 rows split 704, 704 and 640. The block was gcd(512, w), as glue.py picks it, and only 512 and 64 had kernels, so the last rank's 640 (gcd 128) stopped the forward with NoSwigluBlock. The values are element-wise and each 64-column sum is one group whichever block runs it, so the 64-column kernel serves any width that 512 doesn't divide; TP=2's widths keep the 512-column kernel.
…it reads the config The family read config.json with config.parse alone, so a quant_algo other than NVFP4 was never refused on the serve path. Engine.init now runs modelopt.admit on the same text first: a refusal logs admit's reason with the file's path and returns its error (NotModelOpt or UnsupportedQuant), which explain words for the server. glm-synth's two-layer checkpoint writes NVFP4 linears and now says so in its config.json, with the block nvidia/GLM-5.3-Flash-NVFP4 carries, so the lanes checks on it are admitted the same way.
…tes rank 0 admits The server admitted GLM-5.3-Flash's streams and its prompt cache by rank 0's room alone: a follower never returns from open, so a rank with less free memory than rank 0 (another job, the page cache) could run short after the server said it was up. Now the native host passes the family a reading of its budget (Options.room: free memory less the reserve, under TENSORFOLD_CUDA_MEMORY_LIMIT_GB, past what the load took), every rank reads it once its weights are in, and the ranks all-gather the values before a follower starts following (cuda_ranks.gather, which agree now uses too). Rank 0 returns the smallest of the others (Loaded.peer_room) and the host admits by it when it is smaller than its own, naming that rank in a refusal. Rank 0's per-stream and per-state bytes are the largest of the ranks, since the cut gives a remainder to the first ranks, so the room it admits by holds on every rank. Before the load, a rank's weights were taken as the checkpoint over the world size; TP=3's uneven cut puts more than that on rank 0 (22 of 64 KDA heads, 704 of 2,048 expert rows). weightShare now reads config.json and takes the checkpoint's bytes times the rank's largest share of an axis (Share.part), an upper bound but for small replicated tensors. Nemotron declares neither field and takes the host's path as before.
…s, and a hub snapshot's revision only against another
8 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #804 (at
6380c8da): until #804 lands this also shows its commits, and this change is the last one,bce5c609. It is the third follow-up promised on #804 (comment).What this changes
Ranks compare the checkpoint by what it holds, not by its folder's name, which refused identical copies kept under
other names. The fingerprint now carries the sha256 of
model.safetensors.index.json(if there is one) and of eachshard's name and header, in name order;
config.jsonis already hashed on its own, and the tensors' bytes are notread. A hub snapshot's revision (the folder under
snapshots/) is still compared, but only when both ranks read a hubsnapshot. The fingerprint's layout goes to 5.
Receipt
6380c8da(TensorFold 1.0.10 and glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804) plus this change and the split start's probe(glm: a split start checks that every pair of ranks can exchange, before the weights load #899) on one branch; three DGX Spark (GB10, sm_121, CUDA 13.3) over RoCE; nvidia/GLM-5.3-Flash-NVFP4 at
revision
423acf37. The tests ran atc62a6977;bce5c609only shortens a doc comment to one line.pass, and at TP=2 the server's replies equal glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804's head by hash, rounds and accepted drafts.
zig build test-cuda-host512 passed (one more than glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804: a checkpoint hashed in threefolders, the same headers agreeing under another name and another tensor name not),
zig build test519 passed and5 skipped,
zig fmt --checkclean,glm-refand the thin and synthetic lanes checks; at TP=3 the concurrent (41PASS) and shared checks, at TP=2 the concurrent, shared and cover checks; 0 failed.
lean_check: 523, as glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804.snapshot; the host test covers the hash).
Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (the lanes checks above).