Repository navigation
glm: a split start checks that every pair of ranks can exchange, before the weights load - #899
Conversation
…ghts load, refusing a pair that cannot reach each other
ea9667d to
bb19e68
Compare
|
Rebased onto main at |
…peer before the weights load, and refuses a pair that cannot
…ing start on a neighbours-only fabric loads Settings.read sets split and ring for TF_GLM_PREFILL_REDUCE=ring, and the engine's start guarded #899's probe with split alone, so a ring start still exchanged a value with every rank before loading: a ring of four without a switch, whose ranks reach only their neighbours, could not start in the mode #902 built for it (two ranks cannot tell the two apart). probeStart probes for split without ring, and the refusal now names ring or gather. Host tests over a fake fabric: a direct split start is refused at every rank of a four-rank ring and starts behind a switch and on two ranks; ring and gather starts on the four-rank ring post nothing before loading. Co-authored-by: Bizuayeu <30091157+Bizuayeu@users.noreply.github.com>
…ollow-ups named as ours After review of the fourteen lines that overstated: #710's fresh-quantization gain is one measured case; #902's stop and group cleanup hold where the watch lines are open and through the shared helper; #887 says its chunks change the attention bits below 8,192, on M5 too, with no larger error against float64; #899 records the ring start and the wider refusal; #802 keeps the author's B70 receipts apart from our host-checked lifetime fixes; #836's equalities are the GB10's and its Windows build the author's; #888 says a failed restore's pinned memory is kept, never freed; #734's tree rounds are checked against the host's, not float64; the Living Weights lines (#876, #877, #878, #891) say which runs their numbers come from, the ten-fact costs, and the M3 Ultra's four-fact result; #946 and #910 read plainer.
|
Thank you, @Bizuayeu, for making an unreachable split fabric fail at startup with a useful remedy. We merged the probe for 1.1.0 in 8e64cfa. The integrated guard now probes every peer only for direct split; neighbour-only ring mode bypasses it. The refusal names Host fabric controls cover the refusal and ring start. A successful two-rank run alone cannot establish the four-rank neighbour-only case. Rank-watch controls and two-GB10 EXL3 serving passed; four-rank relay and live killed-peer comparisons are not recorded. Posted by Ash's AI Assistant Synthena |
A live killed-peer comparison on 1.1.0 (
In all twelve the surviving rank logged Not covered: four ranks. Our three GB10s are a switchless ring whose per-host /32 addresses |
… every peer before the weights load, and refuses a pair that cannot
@Bizuayeu rebased ashhart#899 onto 1.0.12 (b5b797e). The commit carries the same change as the head landed here (fa7aad4), its context moved by 1.0.12's multi-line Fingerprint.of call, so the tree stays as it was.
…shhart#865), the 27B's K splits (ashhart#887), drafter releases (ashhart#893, ashhart#894), GLM's split check (ashhart#899) and Intel Arc (ashhart#802)
…ing start on a neighbours-only fabric loads Settings.read sets split and ring for TF_GLM_PREFILL_REDUCE=ring, and the engine's start guarded ashhart#899's probe with split alone, so a ring start still exchanged a value with every rank before loading: a ring of four without a switch, whose ranks reach only their neighbours, could not start in the mode ashhart#902 built for it (two ranks cannot tell the two apart). probeStart probes for split without ring, and the refusal now names ring or gather. Host tests over a fake fabric: a direct split start is refused at every rank of a four-rank ring and starts behind a switch and on two ranks; ring and gather starts on the four-rank ring post nothing before loading. Co-authored-by: Bizuayeu <30091157+Bizuayeu@users.noreply.github.com>
…ollow-ups named as ours After review of the fourteen lines that overstated: ashhart#710's fresh-quantization gain is one measured case; ashhart#902's stop and group cleanup hold where the watch lines are open and through the shared helper; ashhart#887 says its chunks change the attention bits below 8,192, on M5 too, with no larger error against float64; ashhart#899 records the ring start and the wider refusal; ashhart#802 keeps the author's B70 receipts apart from our host-checked lifetime fixes; ashhart#836's equalities are the GB10's and its Windows build the author's; ashhart#888 says a failed restore's pinned memory is kept, never freed; ashhart#734's tree rounds are checked against the host's, not float64; the Living Weights lines (ashhart#876, ashhart#877, ashhart#878, ashhart#891) say which runs their numbers come from, the ten-fact costs, and the M3 Ultra's four-fact result; ashhart#946 and ashhart#910 read plainer.
2 participants
The first follow-up promised on #804 (comment),
after @SvangenStudios's TP=4 run on a ring of four (comment),
as one commit on main:
bb19e68e.What this changes
TF_GLM_PREFILL_REDUCE=split, the default, sends each rank's partial rows straight to the rank that glues them andthe glued rows straight back, so every pair of ranks must connect. On a ring of four whose ranks 0 and 2 cannot reach
each other, it failed in the first prompt chunk with
ncclGroupEnd: unhandled system error. With split,Engine.initnow exchanges one value each way with every peer before the weights load, peers in rank order (every pair's turn then
comes in the same order on both of its ranks, so no pair waits on a later one), and a pair that cannot connect
refuses at start:
REFUSE TF_GLM_PREFILL_REDUCE=split: rank 0 cannot exchange with rank 2; ... needs TF_GLM_PREFILL_REDUCE=gather.With #902 also in, ring (split whose exchanges go only to neighbours) should skip this probe; whichever of the two
lands second makes the probe run for split only and names ring in its refusal.
Receipt
46a08c18(TensorFold 1.0.12) plus this change and the checkpoint check (core: nemotron's checkpoint identity moves to core, and GLM's ranks compare the checkpoint by it #900) on onebranch; three DGX Spark (GB10, sm_121, CUDA 13.3) over RoCE; nvidia/GLM-5.3-Flash-NVFP4 at revision
423acf37.checks pass, and at TP=2 four of the server's replies, compared by hash, rounds and accepted drafts, equal main's.
zig build test-cuda-host853 passed,zig build test621 passed and 7 skipped,zig fmt --checkclean,glm-refand the thin and synthetic lanes checks; at TP=3 the concurrent (41 PASS) andshared checks, at TP=2 the concurrent, shared and cover checks; 0 failed.
lean_check: 516, as main.check it on the ring of four.
Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (the lanes checks above).