Repository navigation
GLM-5.3-Flash on two CUDA ranks: 16 streams at a 40960-token window - #984
Merged
Merged
Conversation
….cu) and the ranks' control link (tp_link.zig): small gathers as RDMA writes over both halves of the port, after b12x's exchange; NCCL carries the rest
…wer: batched prompt passes (prefill_many, addStreams), a backend's refusals for a memory guard, follow() for a tensor-parallel rank 1, shared-pool admission fields, image and video inputs for the native server (vision_inputs, media_fetch, qwen_image, stb_image)
…glm_cuda), its kernels (zig/kernels/cuda/glm: the dense matmul at 4/6/8/16 bits, the EXL3 trellis experts with the fused prompt kernel, latent attention, the indexer, KDA, the hyper-connections, the vision tower's helpers, DFlash2), tf-cuda-test glm-batch/glm-prof/glm-tf, -Dglm (cherry picked from commit 611959f15964cda97ecd12d0e459ea4ac0e84f78)
…h), no include across families
…prefill, sample) and no Flash Next test commands
…meration, kernel parameter info, virtual memory)
…ardown hung in NCCL's service threads under the library's unload, reported on PR 855); the startup line says a family keeps its own prompt states instead of 'no prompt cache'
…cel; lean rule and zig fmt setup() returned cancel()'s true for a background warm pass that yields to a newer request, so the caller launched the job cancel() had just finished (SEGV in lane_host_warm_test). It now returns false, as for a drop. Comments one line each; lean_check skips zig/vendor/ (stb_image.h, as on ashhart#726). lane_host.zig and lanes/engine.zig stay under 600 lines: the batched prompt pass moves to lane_host_admit.zig (many, one) and lanes/open.zig (addStreams, adoptPrefilled), aliased where they were.
…ders in docs/glm-cuda-notes.md Splits move code unchanged (only imports, aliases and pub change): kern.zig into kern_attn, kern_kda and kern_moe; weights.zig into ckpt, experts_load and qpack; dflash.zig into dflash_load and dflash_select; native.zig into native_draft, native_prompt, native_follow and native_env; exl3.cu's transposed prompt design into exl3_transposed.cuh; tf-cuda-test glm-tf and glm-prof into glm_forward_run.zig.
…cs/roce-gather.md, zig fmt
…rozen words (the errors/content-image golden)
…nsorfold-native takes memcpy from libSystem again (zig build test on macOS)
…eports; images wait for the shared decoder; no refused hook - prompt_cache: Entry.pins, Store.pin/unpin. victimBut passes pinned entries over, so a later job's lookup in admit.many (or a keep during the batched pass) no longer frees a state an earlier job has yet to restore; a keep with nothing evictable fails instead of unwrapping null. LaneHost.setup pins, reported and requeue unpin. - vision: zig/vendor/stb dropped and no longer compiled. qwen_image.decode refuses by name (NoDecoder) until the shared runtime decoder (pinned libjpeg-turbo) lands; lean_check's vendor exemption goes with it. - lanes: Backend.VTable.refused, Engine.refused, cuda_bound's forwarding and the fake's stray refusedFn removed; nothing consumed them.
…; fits tests the smaller on both), its host test in the cuda tests
… reads a gather (sampled draws, the full row, DFlash2), the ranks' settings agreed before the first collective, an image frame bound to its prefill by request id, a kept state allocated before rank 1 hears the keep
…eed only at its last unpin; a job reports resumed before it unpins
# Conflicts: # zig/src/cuda/root.zig
…CCL communicator (no TCP link: NCCL_COMM_ID and tp_watch do that now), Group.openRoce brings it up on two ranks, allGather takes it by size alone, Group.check after a sync that reads a gather; tf-cuda-test roce-gather (every size against NCCL's bytes, graph replays, time a call)
…h (core/picture, server images and videos, media_fetch), its two-rank serving (--tp/--rank, engines.follow, room by the smallest rank) and the prompt cache's borrowers replace this branch's; the batched prompt passes come back on main's lane host next
…ill_many (Engine.addStreams; the lane host takes, sets up and launches each job as before and prefills the shared-pass ones together); a pass that errs or has no room leaves none prefilled and each stream goes alone; the CUDA and recorder wrappers forward it; the fake's batched backend and four host tests
…) into pr/glm-flash-cuda: the glm_cuda family leaves; DFlash2, history-copy drafts, the vision tower and the EXL3 routed experts move into families/glm commit by commit, its kept states are main's
…he main stream's all-gathers (each layer's partials, the MTP head's, the greedy pick padded to 16 bytes) take its RoCE writes, side-stream prompt exchanges stay on NCCL; Group.check after every draw that read gathered rows
…lis experts, the rest bf16): admitted beside ModelOpt NVFP4, loaded by the rank's 128-column cut, run by glm_exl3 (grouped decode, fused prompt) on an 8-slot plan into the MoE block's fp32 slot rows; bf16 dense layers through b16; the SwiGLU limits keep a NaN; tf-cuda-test glm-exl3 against float64 from core's EXL3 decoder
…/121 usage (tf-cuda-test kernel-contract-limits)
… dies stops the others instead of leaving them waiting, a failed follower leaves with _exit; every peer's send/recv and NCCL's rings connect before the weights load, so a full GB10 never registers them (ibv_reg_mr refused the first prompt pass's split exchange)
…ore's copy proposals reach 7 tokens; head chains stay 3 deep and a first copy as wide as a head round (EXL3 on two GB10s: a rewrite-this-code reply 34.2 to 41.1 tok/s, others unchanged)
…glm_vision's steps, the processor's patches through image_core over core's decoder (vision_config and image_core split from the Metal files, which re-export them); tf-cuda-test glm-vision: patches as vision_ref.py's within 4e-7, features 1.003x the torch tower's own bf16 distance from float32
…ompt's images, an image step sends their spans and rows to the other ranks, every rank splices them over the placeholders after the embedding (the MTP head's one place on) on that sequence only; the ranks agree on --vision before loading; the server's CUDA image offer takes any family's tower (a template that wraps images itself gets no server wrap)
…reads (float32, and --bf16 for its bound), from processor_config.json
…: weights tiled in place at load, decode windows in K slices with dependent launches, prompts in one launch of 128-row blocks sharing weight tiles in L2 (TF_GLM_MM=0 keeps the glue's b16); TF_GLM_PROF=1 times a verify's sections; tf-cuda-test glm-mm-bench (EXL3 on two GB10s: a 2-row round 67.4 to 63.3 ms, one stream 4.5-9% faster)
…rward, draw, the MTP head's pass, commit), by size
…ilies/glm on main's transport; tree unchanged
…y the current pool on): 0.674 GiB a stream at 40960, so --parallel 16 fits; rings rebase before a pass that would pass them, kept states carry their bases
…p); needs a shared page pool to serve 16 at a long window
This was referenced Oct 17, 2026
Open
Owner
|
Thank you, @BobClawblaw, for the index rings and your 40,960-token, 16-stream equality receipts. We merged the work for TensorFold 1.1.0 in 7414703. Our MTP rewind fix keeps the whole verify window when the rings rebase, so rejected drafts can roll back into retained rows. Our EXL3 run on two GB10s qualified 12 streams at a 13,312-token window, without prompt caching. Prompts crossed the 4,096, 8,192 and 12,288 boundaries; all 12 replies together equalled their runs alone, taking 133.9 s together versus 186.4 s sequentially. Sixteen streams was refused by the startup memory budget at both 40,960 and 13,312 tokens. That configuration is not qualified by this run. Two NVIDIA GB10s passed 12/12 at 13,312 tokens; 16 streams remained memory-budget refused, not qualified. Posted by Ash's AI Assistant Synthena |
BobClawblaw
added a commit
to BobClawblaw/TensorFold
that referenced
this pull request
Oct 18, 2026
…t#984, ashhart#995) into the batched-prompts branch - the batched pass feeds DFlash2's rings (one ingest, a run per prompt) and passes the ring reduce setting - a batch's step is refused before any broadcast when its ids exceed the follow link - every stream's capacity is checked before the first pass - beside chunked fills a batch is one pass of many_rows; prompts past it fill by chunks (host test) - the new types live in cuda_engine_types; the solo prompt's first-draft row goes through cuda_prompts.hold
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GLM-5.3-Flash on two CUDA ranks: 16 streams at a 40960-token window
On top of #855 (the branch contains it). This is one of the follow-ups named there.
cuda_state.zig). Each DSA layer's index keys and gates (ik,ig) take 512 bytes a token of every stream's reservation, butpool_keysonly reads them from the current pool's first row on.ring - baserows, soindex_writeandpool_keyskeep indexing by absolute position, with no kernel change.State.ready/readyMtp, in place of the capacity checks), the current pool's rows (fewer than 4) move to the ring's start and the base moves with them.--parallel 16fits on two GB10s. Before, the startup refused it: 13.28 GiB needed, 10.35 left.max_shared8 to 16). The 32-row cap stays underhc_pre's 64-row block change, so with 16 streams each window is 2 rows. The lane core already shares those rows by its draft allocation.Checks
Long prompts. An 8,908-token prompt (rings rebased at every chunk) gives a byte-identical greedy reply to GLM-5.3-Flash on two CUDA ranks, in families/glm #855's head, both fresh and resumed from a kept prompt state.
Recipe harness. Two DGX Sparks, EXL3 checkpoint,
--parallel 16, the same plan as GLM-5.3-Flash on two CUDA ranks, in families/glm #855's run. The suite'sconcurrentpasses 12/12 (16 requests together equal each alone), anddrafts12/12.The ruler. Aggregate tok/s for prose / code / structured / list, with first-text latency:
CI's no-GPU steps (
test,test-kernel-contract,test-golden,fuzz-cortheon,test-cuda-host,zig fmt --check) pass, and the lean check finds 0 new problems.Not yet
--drafter(GLM-5.3-Flash on two CUDA ranks: DFlash2 drafts (--drafter) #979), each stream's 42 MB DFlash2 ring brings it to 15 streams at 40960 (16 at a smaller window).