Repository navigation
Conversation
…ecodeF32 and promptF32) A tensor-parallel family sums its ranks' partial rows before rounding, so it needs a projection's fp32 sums as the kernel holds them. qmmf.cu and prompt16.cu already take F32 as a template argument; this instantiates FP4's fp32 tiles (and reduce_kernel<true> for the split path), loads them beside the bf16 ones, and adds Lane.matmulOut, Prompt.matmulOut and qlinear's decodeF32 and promptF32. Other formats refuse fp32 out with EpilogueNotBuilt, as the experts do. matmul, decode and prompt keep their signatures and bf16 bits.
…n block The GLM-5.3-Flash CUDA port reads nvidia/GLM-5.3-Flash-NVFP4 as stored. admit() checks config.json's quantization_config (top level, else text_config's) for quant_method modelopt, quant_algo NVFP4 and 4-bit float weights in groups of 16, and refuses anything else with a reason the caller logs (the Why of nemotron/config.zig), so a test reads the reason instead of failing on an error log. MIXED_PRECISION is refused for now. Which linears are NVFP4 is left to the tensors, not to the block's ignore list: the pinned revision 423acf37 stores the MTP layer's experts in BF16 without listing layer 45 there (da920bb0 added it). The fixture is that revision's config.json.
nvfp4.zig is the format the GLM-5.3-Flash CUDA port's W4A16 linears read: e2m1 codes two to a byte (input 2j in byte j's low nibble) [n, k/2], an e4m3 scale per 16 inputs [n, k/16], and weight_scale_2 as an fp32 scalar that multiplies. dims() checks the four tensors fit together; Linear.read() maps them from a Checkpoint and marks input_scale as used, since the W4A4 math's static input scale is not read by W4A16 but a leftover tensor would refuse the checkpoint; row() gives a row as e2m1 x e4m3 x global in f32, the reference the GPU kernels will be compared with. stored() says a module is NVFP4 by its weight_scale tensor. The e2m1 and e4m3 values come from core.quant's nvfp4 and fp8 (ashhart#730), as f32. The fixture is the whole header of nvidia/GLM-5.3-Flash-NVFP4 (147,661 names, dtypes and shapes, gzip'd): every NVFP4 linear reads, 42 MoE layers x 288 experts x 3 plus the three dense MLP layers' 3, all under .mlp., and the MTP layer's experts stay BF16.
…e with them A GLM-5.3-Flash rank reads about half of the 183 GB checkpoint through the mapping, and its column cuts touch whole tensors. On GB10's unified memory the page cache holds memory the rank's device allocations need (our Python serving engine drops it with posix_fadvise for the same reason). safetensors.File.release drops a file's pages from the mapping (madvise) and the page cache (fadvise DONTNEED); Checkpoint.release does every file. The mapping stays valid and refaults on a later read, which a test checks. The GLM-5.3-Flash CUDA family (ashhart#548) calls it after each layer it loads.
…me from its own declarations The CUDA host wired Nemotron's prompt-cache copies, rules and budget in by name. They now come from the family's `snap_*` declarations through `cuda_snaps.zig`: a family without `snap_save` gets no prompt cache, `snap_ready` sets aside its state buffers, and `snap_grid` makes the store keep states only on the family's chunk starts (a family whose rows depend on where a prompt's chunks begin), and `warm_quiet` turns on the quiet-turn reply prefill (warm turns and the store's `.waits`). Nemotron declares `warm_quiet` and no grid and keeps 1.0.9's rules: any row, the head one token on, warm turns, the same budget. Three more hooks for a family split over hosts: `weightShare` (the weights this rank holds, for the memory check), `explains_open` (the family's own text for an open error) and the learner only for a family with `slide`. Nemotron's open, cache, warm turns and --slide are unchanged; `cuda_snaps.zig`'s test covers the rules each way.
…skip), as the decode form does A family that computes one expert itself (GLM-5.3-Flash stacks its BF16 shared expert as the last expert id) plans every pair together and has the grouped kernels skip that id. The decode form already takes skip; Grouped.prompt refused it with OptionNotBuilt for NVFP4. The prompt entries now take skip and return before staging any rows of that expert's items. Nemotron passes -1, so its bits do not change. quant-ref now checks it: with the shared expert skipped, its pairs' prompt rows keep what was there and every other pair's row equals the run without skip, for fp8 and nvfp4.
packGateUp packed each matrix into a scratch half and copied the halves into their interleaved places, two allocations and a copy per expert. A loader packs every routed expert of every layer this way (GLM at TP=2: 45 x 256 experts, 2.4 MB a half), and the fresh pages and the copies put about 20 s on the load. packOne's loop now takes the matrix's place among the stacked parts, so packGateUp writes gate and up straight where they go, and it needs no allocator. The words are the same: a host test checks packGateUp against packOne's blocks interleaved.
… over hosts The family ashhart#593 tracks, on the shared NVFP4 layer (`qlinear`, `experts.Grouped`) and hand-written `.cu` for the rest: KDA and MLA/DSA attention, the hyper-connection glue, MoE routing, the MTP draft head and sampling. Rank 0 serves through `tensorfold-native serve` and the other ranks follow its steps; each rank holds its share of every axis (uneven where a width doesn't divide). A prompt piece with at least as many rows as ranks is split over them (`TF_GLM_PREFILL_REDUCE=gather` all-gathers it instead): each rank sends the others their rows' partials and receives its own over NCCL send and receive (`nccl.zig` binds `ncclSend` and `ncclRecv`), and the hyper-connection step adds them in rank order. Drafts and concurrency go through the lanes core: `prefill_step` takes a prompt a chunk at a time between the other streams' rounds, and `hidden_rows` lets several streams' verify windows share one forward. The prompt cache keeps states on the family's 2,048-row chunk starts (`snap_grid`). Checks in `tf-cuda-test`: `glm-ref` holds every projection whose reduction order is the kernels' own against float64 on synthetic inputs; the lanes checks (cover, concurrent, shared) run at TP over hosts and, with `glm-synth`'s two-layer checkpoint, on one GPU.
… EXL3, 64 otherwise) An EXL3 matrix splits only at 128 columns, so at TP=4 the head's 154,880 rows go 303, 303, 302 and 302 blocks of 128 instead of 605 units of 64 each. `Config.head_unit` is 64 by default and 128 when `quantization_config.quant_method` is `exl3`; `Share.of` cuts the vocabulary in it. NVFP4 keeps 64, so its shares at TP=2 and TP=3 are unchanged. The patch BHCC2025 posted on ashhart#593, with its test of the TP=4 and TP=2 shares.
…e width (TP=3's last rank, 640) At TP=3 the routed and shared experts' 2,048 rows split 704, 704 and 640. The block was gcd(512, w), as glue.py picks it, and only 512 and 64 had kernels, so the last rank's 640 (gcd 128) stopped the forward with NoSwigluBlock. The values are element-wise and each 64-column sum is one group whichever block runs it, so the 64-column kernel serves any width that 512 doesn't divide; TP=2's widths keep the 512-column kernel.
…it reads the config The family read config.json with config.parse alone, so a quant_algo other than NVFP4 was never refused on the serve path. Engine.init now runs modelopt.admit on the same text first: a refusal logs admit's reason with the file's path and returns its error (NotModelOpt or UnsupportedQuant), which explain words for the server. glm-synth's two-layer checkpoint writes NVFP4 linears and now says so in its config.json, with the block nvidia/GLM-5.3-Flash-NVFP4 carries, so the lanes checks on it are admitted the same way.
…tes rank 0 admits The server admitted GLM-5.3-Flash's streams and its prompt cache by rank 0's room alone: a follower never returns from open, so a rank with less free memory than rank 0 (another job, the page cache) could run short after the server said it was up. Now the native host passes the family a reading of its budget (Options.room: free memory less the reserve, under TENSORFOLD_CUDA_MEMORY_LIMIT_GB, past what the load took), every rank reads it once its weights are in, and the ranks all-gather the values before a follower starts following (cuda_ranks.gather, which agree now uses too). Rank 0 returns the smallest of the others (Loaded.peer_room) and the host admits by it when it is smaller than its own, naming that rank in a refusal. Rank 0's per-stream and per-state bytes are the largest of the ranks, since the cut gives a remainder to the first ranks, so the room it admits by holds on every rank. Before the load, a rank's weights were taken as the checkpoint over the world size; TP=3's uneven cut puts more than that on rank 0 (22 of 64 KDA heads, 704 of 2,048 expert rows). weightShare now reads config.json and takes the checkpoint's bytes times the rank's largest share of an axis (Share.part), an upper bound but for small replicated tensors. Nemotron declares neither field and takes the host's path as before.
…layer shared SwiGLU limit, GLM's bytes unchanged tf_glm_topk takes n_group and topk_group: with more than one group it keeps the topk_group groups of the best top-2 biased sums (lower group on ties) before the top-k. GLM's one group skips that path. swiglu's limit 0 now means unclamped (as the experts' 0), and the MoE layer carries the shared expert's limit apart from the routed one's. Tests: glm-ref adds group routing vs float64 and the swiglu limit; glm-route-old compares GLM's select and swiglu with 6380c8d's fatbin; glm-route-ling checks captured Ling layer-4 routing against HF and exllamav3.
…opt-in 16-row WY chunked prompt path - glm-kda-ling: glm_kda's fused chain, 16-row windows, wide prompt and chunked prompt on Ling-3.1 layer 0 (q/k/v convs as one q|k|v conv): rel L2 3.6e-3 vs fla, within float64's bound; replays == chain bits. - glm_kda_chunk.cu: new entry tf_glm_kda_chunk over prep's saves (kda.chunked, kda.chunkReplay). Opt-in only: GLM stays on the row path; glm_kda.cu and its fatbin are unchanged. Not the row path's bits; cuts at multiples of 16 == uncut, a prompt resumed elsewhere differs from a fresh one (measured in glm-kda-chunk). - glm-kda-chunk: float64 bounds end to end and from the saves, saves equal the wide chain's, resumed vs fresh. - glm-kda-speed: 64 heads x 128, 2048-row prompt row path vs chunked, decode row, 16-row window.
…e heads, head-wise gate chunks_wide<HB, RW>: q and cache rows LW + RW wide, scores over all of them, values from the 512 latent; tf_glm_chunks_r16 is the 576-wide 16-head form (RW 0 builds GLM's kernels to the same SASS). absorb_q/expand_v take the head width as a template (GLM 256, Ling 128; the 128 absorb reads q_nope at a q stride, writes qa at a stride). New tf_glm_rope (GPT-J pairs, fp32 angles, sincosf) and tf_glm_head_gate (fp32 sigmoid). Host: absorbQStrided, expandV at 128, rope, ropeFreqs, headGate, attentionRoped. Checks: ling-mla (float64 bounds; decode rows, a verify window and a cut prompt across key 1536 equal the uncut call), ling-mla-real (layer 7 vs HF and float64; 400 + 32 decode rows, an 8-row window and a cut at 300 equal the prompt), glm-mla-same (GLM's MLA launches against a 6380c8d glm_glue fatbin, byte for byte).
…SwiGLU limits as GLM's kernels take them config.zig parses config.json (BailingMoeV3ForCausalLM, bailing_hybrid) into a typed Config: widths, the MLA/KDA schedule ((i+1) % layer_group_size, tail MLA), dense layers, KDA safe gate, MLA head-wise gate and GPT-J rope, sigmoid routing with 8 groups keeping 4, per-layer expert and shared SwiGLU limits, one MLA MTP layer, FP8 128x128 blocks and modules_to_not_convert. It refuses what the engine does not run, each with a one-sentence reason. names.zig enumerates every tensor the config implies by role, with FP8 scale companions; against the checkpoint's index (fixtures/names.txt, folded, count and sha256 checked) it finds all 214,025 names, none unaccounted: 210,925 backbone and 3,100 of the MTP layer 72. cuda.zig gives select()'s Groups and each layer's limits for moe.Layer. Host tests: zig build test-ling (also under test-cuda-host).
…Opt NVFP4) at TP=1, synthetic checkpoints in each config.zig: Quant gains nvfp4. quantization_config fp8 (128x128) or modelopt NVFP4 (groups of 16), or hf_quant_config.json's block (parseWith); ignore / exclude_modules / modules_to_not_convert as names, dotted tails or * globs; format(module) gives each linear's format; exl3 refused naming ashhart#832. names.zig emits each format's side tensors and moduleName() gives the module a role reads. weights.zig / weights_proj.zig: one Layer per layer with a qlinear.Weight per projection (bf16, fp8g, nvfp4 through modelopt.Packer and fp8.packScales), routed experts as experts.Layer (Fp8Experts / Nvfp4Experts packers; bf16 as lane weights, no grouped bf16 kernel), the shared expert as projections, norms, fp32 q|k|v convs, A_log, dt_bias, bf16 router (a quantized one refused) and bias, the MTP layer's tensors. A file tensor whose dtype or shape disagrees with the config is refused. Tests: test-ling (21, with TF_LING_INDEX on the real index); ling-synth writes 4-layer + MTP checkpoints with the real names (bf16 282, FP8 517, NVFP4 978 tensors, NVFP4 mixing a bf16 dense layer); ling-weights: every projection decode and prompt vs float64 from the file, worst 0.456 of the bound, experts' gate-up and down, 37 small tensors byte-equal in each format; ling-weights-real: layers 0, 4, 7 with 4 experts from the FP8 release vs torch (oracle/ling_weights.py), 35 products, worst 0.414 of the bound, rel L2 <= 1.7e-3. Mutations caught: gate/up swapped, conv order, NVFP4 global dropped, exceptions ignored, expert down scales misplaced.
…thout the checkpoint, memory per format
# Conflicts: # zig/tests/cuda/main.zig
…dense and MoE MLPs, head, windows and commits Every projection runs as the qlinear.Weight and every routed-expert layer as the experts.Layer the checkpoint loaded (bf16, FP8 block, NVFP4); MLA's absorbed kv_b is derived through kv_b's own format on identity rows. ling_glue.cu rounds the residual, head gate and expert sum where the HF bf16 model does. Decode windows run KDA's three-kernel chain as GLM's forward does; commit(keep) replays the KDA states of a partial window. ling-forward-synth: synthetic bf16/FP8/NVFP4 logits vs a float64 reference routed on the same picks (2.1-2.3e-2, as far as HF's bf16 rounding points are), cut == uncut, windows == decode rows, a partial commit, batched == alone, fused == three-kernel chain, bit for bit. ling-layer-real: real FP8 layers 0, 4, 7, 71 vs HF 3.0-3.7e-3 without routing flips. GLM files untouched.
Ling's reference (inclusionAI's SwigluStepAndMul, SGLang's BailingMoeV3MLP) clamps silu's output; GLM's glue and the grouped expert kernels clamp the gate before silu. Ling now launches its own instances: tf_ling_swiglu_512/64 in glm_glue (glue.swigluWith) for the shared, dense and bf16 routed experts, and epilogue 4 of the FP8 and NVFP4 grouped kernels, decode and prompt, chosen by experts.Layer.after (set by the Ling loader, refused by affine4). Every pre-existing kernel keeps its SASS and resource usage (cuobjdump over all 40 fatbins vs a33dde0; 37 fatbins byte-identical, the other 3 only add the new instances). ling-clamp-old runs the kept FP8 (12) and NVFP4 (13) instance calls against a33dde0's fatbins byte for byte; glm-route-old and glm-mla-same still equal 6380c8d. At L = 10 both orders give the same bf16 bytes (silu is rounded to bf16 first and bf16(silu(g)) = 10 for g >= 10), so the real layers 65 and 71 and the synthetic forward measure the same digits; ling-clamp checks the order against float64 at L = 10 and L = 2 (where 16,255 bf16 gates differ) and catches 5 of 5 mutants. The float64 oracle clamps after silu too.
… and checks its bytes against the host packers; ling-layer-real can dump its rows
…hase; ling-layer-bytes writes every path's bytes for cmp
…P column copies -50 %), the same bytes
…ver hooks and kept prompt states Engine: a sequence per slot, prompt chunks cut at any row (marks included), verify windows of up to 16 rows, shared windows of up to 8 streams, greedy (tf_glm_top1) and sampled (sample.cu) draws, kept states as arena copies. Lanes: GLM's cuda_lanes.zig without the ranks. native.zig registers bailing_hybrid with the CUDA server. ling-lanes checks chunked == whole, drafted == plain, resumed == fresh (marks off multiples of 16), concurrent == solo, in tokens and state bytes, and the served logits against ling_forward.py's float64 on the same picks.
…ed on the GPU (Packer.fp8Experts, fp4Experts), the file's pages dropped per batch
…h_proj, MLA + MoE on its own cache, chains of 3 mtp.Head runs the MTP layer over a view of the weights (final_layernorm as its norm, the shared head); each sequence keeps a second State for its MLA cache. Prompt passes feed every row whose next token is known, Engine.draft absorbs a verify's kept rows and chains drafts on the layer's own final-norm rows, kept states copy the MTP cache. ling-lanes --mtp: MTP-drafted == plain (greedy, sampled, resumed, concurrent), chained drafts by hand, MTP logits vs ling_forward.py --mtp float64; ling-mtp-real: the real layer vs HF's BailingMoeV3MTPLayer (oracle/ling_mtp.py).
… no copies (4 KDA, 2 per MLP), the same bytes
…e expert kernels (same bits, -7 to -27 % expert time up to 768 rows)
…ne columns (Ling's 64), same picks and weights; ling-route-old
…dle GPU, the expert kernels and plan tiles side by side
Contributor
|
@jschmied, we ran GLM's checks on this branch at c48be49 on our three GB10 hosts, with the GLM-5.3 NVFP4 checkpoint, and they all pass:
So the Ling commits leave GLM's checks and replies as they were. We have no Ling weights here, so this covers GLM only. |
…_spin), so other contexts' time slices miss the least
Contributor
Author
|
Thank you, @Bizuayeu, that's the check we couldn't run. We'll push three more pieces to this branch shortly: serving at TP=1 (engine, lane backend, kept prompt states), the MTP drafter ( A rerun once it's up would be welcome, if you have the time. |
# Conflicts: # zig/tests/cuda/main.zig
Contributor
Author
|
Pushed |
Owner
|
Thank you, @jschmied, for the GLM TP=2/3 checks and for identifying the upcoming serving, MTP and packer changes separately. The routing comparison needs care: At c48be49, the zero-limit GLM behavior and the 16-row helper's arbitrary-resume behavior remain open. Please send the pushed serving/MTP head and its delta from c48be49 so we can recheck those paths and the unchanged-shared-kernel claim. Posted by Ash's AI Assistant Synthena |
… to unclamped At 6380c8d GLM's glue swiglu (like its Metal glm_rows_glu) clamps at the limit it is given, 0 included. 51c0157 made the glue map 0 to +inf for every caller; that mapping now applies to Ling's kernels only. - glm-route-old: GLM's swiglu equals 6380c8d's kernels at limits 10 and 0 (0 was not compared before). - glm-ref: the limit-0 check runs on tf_ling_swiglu_*, the path Ling's unclamped layers take.
…ig is 6380c8d's again Nothing served called it: GLM and Ling run the row path, which resumes at any row. Removed with it: glm_kda_chunk.cu, glm-kda-chunk, glm-kda-speed (wide vs chunked timing), and the chunked path in glm-kda-ling (chain, 16-row windows and the wide prompt on Ling's layer 0 stay checked against fla and float64).
Contributor
Author
|
Thank you. Both open items are closed at
Delta from
Checked at |
Owner
|
Thank you, @jschmied, for restoring GLM's zero limit behavior and removing the 16 row WY path. We checked e29e0c2. Please send a focused revision covering these cases, with geometry rejection and allocation failure regressions, so we can recheck it. Posted by Ash's AI Assistant Synthena |
Contributor
|
@jschmied, we ran GLM's checks again at e29e0c2 on our three GB10 hosts, with the GLM-5.3 NVFP4 checkpoint, and they all pass:
We had also run ac58df6 the same way, with the same results. As before, this covers GLM only. |
…with a named error parse ran KDA head widths, conv tap counts and the MLA widths straight into kernels built for one shape. It now refuses, before any allocation or launch: head_dim other than 128 and short_conv_kernel_size other than 4 (glm_kda.cu's DK and TAPS), key/value heads other than the query heads (KdaGeometry); kv_lora_rank other than 512, qk_nope/v_head_dim other than 128, qk_rope_head_dim other than 64, NeoX rope, a q_lora_rank without an rmsnorm instance (MlaGeometry); hidden_size other than 512/1536/4096, MLP and expert widths outside whole 128-column blocks (Width); tie_word_embeddings (TiedHead). - config_test: 26 refused values on the tiny config, each with its reason; the tiny config with the kernels' own values and the real config accepted. cuda.zig's test holds the fixed widths to glm.kda, glm.latent and fp8.block.
…oup and every group kept checked vs float64 HF's group_limited_topk scores a group by its top two (topk(2) raises on a group of one), and with one pick it skips the renormalization glue.select applies. Both now stop at config load (Routing), naming HF's rule; n_group 1, topk_group == n_group and groups of two stay accepted. - config_test: the refusals on the tiny config, and the four accepted edge cases. - glm-ref: glue.select against the float64 rule on five more shapes: one group, 8 of 8 groups kept, groups of two (64 experts in 32, 16 in 8) and the tiny config's 16 in 4; picks equal, weights within 2^-16.
… driver to fail every call in turn A failed name copy or named.append used to free the device buffer already in buffers (Store.deinit freed it again) and lose the copied name. alloc now reserves both lists and copies the name first, then allocates and appends without failing: on any error the store holds nothing new. - fake_driver.zig: the driver's entry points over host memory (kernels no-ops), counting allocations and handles, with call n failing on request; host tests and GPU-free checks build on it. - store_test: every host allocation failing in turn (checkAllAllocationFailures) and every device call failing in turn leave no leak, no buffer held and no buffer freed twice. The old alloc fails both.
…s.zig read the entry's unset slot) opening() did getOrPut and then freeSlot(), which walks every entry, the new one included, before its value was set. In a release build a reused bucket keeps its old slot, so a free slot could count as taken and the stream got a higher one. Lanes.claim picks the slot first, then puts the entry. - slotCheck: claims over buckets that still hold slot 0 from cleared entries (the same keys hash back into them), as a host test and as `ling-slots` in the release build, where the old order fails (SlotTaken).
…ns, not at the first MLA layer The capacity check sat in block_mla, after the KDA layers before it had run: a prompt chunk past the capacity had already committed those layers' states and conv windows (and a window had written its saves) when CacheFull came back. Forward.rowsOf now refuses every segment without room first, so a refused request changes nothing. The MTP layer, one MLA block, already refused before its cache write. - ling-lanes (engine checks): at 1012 of 1024, a 30-row prompt chunk, a 16-row window and (with --mtp) MTP rows past the end each come back CacheFull with the sequence's arenas byte for byte as they were; the next window equals a twin's (tokens, logits, states) and a new request on that slot equals one on a fresh slot. With the old check the arenas differ and the next commit fails (KdaLayersOutOfStep).
… sync fails; every init failure checked The one gap in init's errdefer chain: sequence 0 (its arena, MTP cache and hold row) and the slot list leaked when bind(0) or the closing synchronize failed. Both are freed now. - ling-init-faults DIR [--mtp]: native.open (Engine.init, sequence 0, the lane backend) on the fake driver, once cleanly, then with each driver call and each host allocation of the open failing in turn; each failed open must leave no device or pinned allocation, no module/stream handle, no host leak and nothing freed twice. bf16, FP8 and NVFP4, with and without --mtp: 714 to 1,832 calls and 251 to 426 allocations a clean open, 8,591 failure points, none left anything. Dropping this fix, or any of five errdefers in engine, graph, state, weights and mtp, fails it.
…s names, dtypes and shapes, from headers alone The loader's shapes now come from one table, names.dims (each role's [n, k], value count or conv shape); weights.zig reads every tensor through it instead of computing shapes inline (the same shapes, ling-weights unchanged). - ling-ckpt-check DIR [INDEX]: each shard's header only, every tensor against the weight map at the loader's dtype and shape (FP8's e4m3 weights and scale_inv blocks, ModelOpt NVFP4's codes and scales, bf16 exceptions, small tensors and convs), nothing missing, nothing extra, the bytes against the index's total_size. On the FP8 release (54 shards over sshfs, 3 s): 214,025 tensors, 3,100 of them the MTP layer's, 573,753,281,152 bytes, no mismatch. The three synthetic formats pass too; a wrong dims entry or scale shape fails it.
…ng-3.1-flash@a6943f4b by sha256 ling_hf.py pins the revision (a6943f4bf04f8041ad983d2b58dd43b5ba86ae4a: config.json as of ashhart#2; the modeling, configuration and index files are unchanged since the upload) and the sha256 of modeling_bailing_moe_v3.py, configuration_bailing_moe_v3.py, config.json and model.safetensors.index.json. ling_mtp.py checks the modeling package it imports and the checkpoint's config and index; ling_layer.py takes the modeling folder the chain was captured with and checks it; ling_weights.py checks the index. Any other copy stops the oracle naming both hashes. - test-ling holds fixtures/config.json to the pinned sha256. - The local copies match HF at that revision byte for byte; ling_layer.py rebuilds the layer-4 fixture identically.
Contributor
Author
|
@ashhart, thank you. @Bizuayeu, thanks for rerunning the GLM checks at e29e0c2. A focused revision on top of 1. Geometry rejected at config load (
Regression ( 2. Regression: 3. Capacity refusal before KDA commits ( Regression ( 4. Singleton routing groups ( Regression: the refusals and the four accepted edge cases in 5. Engine init cleanup ( Regression: 6. Undefined slot read in Regression: 7. Full checkpoint and HF source
Re-run at
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Ling-3.1-flash (#850) on CUDA at TP=1, built on #804's GLM kernels as agreed there and on #850. It's a draft stacked on #804: its first 12 commits are #804's (head
6380c8da), and the review is the 9 commits after it. It does not serve yet. Lanes hooks, serving, MTP and TP are open;docs/recipes/ling-3.1.mdlists them as items others can take, along with how to test without the 572 GB checkpoint.51c01573);f8e594f6);b55aa13d; it gives up resumed == fresh at rows that aren't a multiple of 16, so nothing uses it by default);min(silu(g), L) * clip(u, -L, L), as new kernel instances (5e716da5).zig/src/families/ling/, format-agnostic:5bc9618a);qlinearandexperts. EXL3 is refused and points to EXL3 checkpoints on the native engine (tracker) #832 (0907774b);ling-synthfor small synthetic checkpoints;commit(keep)(a33dde0c);0069c7d4).Receipt
Environment: glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804
6380c8daplus this branch (c48be499), one GB10 (sm121), CUDA 13.0, Zig 0.17.0. Checkpoint:inclusionAI/Ling-3.1-flash, block FP8, its real layers read one at a time.Exactness, bit for bit, in bf16, FP8 and NVFP4 (
ling-forward-synth):On real layers, a prompt cut at 300 equals the uncut prompt and a 16-row window equals 16 decode rows. Served
"draft": false, resumed and concurrent comparisons need the lanes hooks, which aren't here yet.Against the reference:
Decode and prompt speed: not measured. The graph is untuned and nothing serves Ling yet.
GLM unchanged:
glm-route-oldandglm-mla-samerun glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804's fatbins next to the new ones, byte for byte (14.3 MB of MLA outputs);cuobjdump, all fatbins);ling-clamp-oldchecks the kept expert instances' bytes;glm-ref,glm-shared-check,glm-lanes-coverandglm-lanes-concurrentpass on the synthetic GLM checkpoint fromglm-synth.Tests run:
zig build testpasses;test-ling21/21; the CUDA commands named above pass. Every part has deliberate mutations, all caught: routing 6, KDA 5, MLA 6, config 8, loader 5, graph 6, clamp 5.zig fmtis clean, andlean_check.pyreports 527, as on glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804.Not run, and why:
Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (Kernel and graph level only, above; served checks wait for the hooks.)Open question for @ashhart (from #850): group-limited top-k is built into #804's
tf_glm_topk, since Ling's MoE runs through #804's routing. You placed it "beside #798". Should it move into #798's shared routing instead?