Repository navigation
syncer: pull a learner that finishes its catch-up into rounds launched without it - #338
Merged
Merged
Conversation
…quorum timeout Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…al to serial Independent reductions (tensors, learners, HeLoCo's three norms) run concurrently; element-wise passes (accumulation, rescale, Nesterov and Delayed Nesterov steps, finiteness checks) are chunked. No reduction is split, so every thread count gives the exact bits of the serial path. --merge-threads / YETO_SYNCER_MERGE_THREADS selects the count (default: cores capped at 32); 1 runs the historical code unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e full roster Releasing the slot of a refused stale-base push let an exact or streaming fold close one push short and stop the syncer with 'fold incomplete'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…und) With quorum 1 a round whose only delta from the fast island arrived as a late carry (its anchor spent) waited for the stalled island, and with --pipeline 2 so did the next round. The new flag counts expected carried members as responders for quorum and the all-answered check. Off keeps the historical RL async rule; the flag joins the semantic profile hash only when on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
build_layout recomputed every bin's numel for each placed tensor, which is quadratic in the tensor count: the unfused 120B MoE shape list (41,643 tensors, 32 fragments) took 107 s per learner process. Track the totals instead; the placement (first smallest bin wins) is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… in parallel Every INIT_PARAMS payload is hashed and decoded on the connection's read loop. On a Kunpeng 920 the vendored sha2 runs its portable code (its aarch64 backend needs the asm feature and another crate), about 0.25 GB/s, and the f32 decode plus the finiteness check are one more single-threaded pass, so a 120B model's 483 GB of f32 INIT took about 40 minutes to admit. - sha256.rs: SHA-256 with the ARMv8 SHA-2 instructions when the CPU has them (runtime detected), else sha2; byte-identical digests (padding boundaries, split updates and known vectors tested against sha2). Used for INIT digests and the downlink digest. - decode_tensor splits large f32 / bf16 payloads over the merge threads (disjoint chunks, same per-value conversion, same error for a non-finite value), and INIT computes its digest while it decodes. Measured on the production syncer host with the same 120B INIT stream: 0.21 -> 0.5 GB/s; the digest (now ~1.6 GB/s, one sequential pass) is the remaining bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
append_tensor_bytes sliced every value by the chunk's byte range and appended it on its own, about 0.4 GB/s on a Kunpeng 920. Every f32 broadcast of a 120B fragment (15 GB) goes through it, one 4 MiB chunk at a time on the forwarder, so the syncer's downlink to one island ran at ~0.42 GB/s on a 50 Gb/s link: the initial cut alone took ~20 minutes. Only the values cut by the range ends now go byte by byte; the bytes are unchanged (tested against the value-by-value encoding, NaN, -0.0, subnormal and every cut position). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cheduler A --checkpoint-every checkpoint (other than per-round ones) was written synchronously in publish_round: the scheduler merged and broadcast nothing until the whole state was on disk. For the 120B state (966 GB of f32 params and momentum) that is a 13.7 minute stall per checkpoint on an NVMe that sustains ~1.2 GB/s of buffered writes (measured on the production syncer host). With --background-checkpoint the cut is taken as the existing copy-free CheckpointSnapshot (later commits copy only a fragment the writer has not reached) and saved on its own thread while rounds go on. At most one such write runs: the next periodic checkpoint, and every synchronous save (terminal, budget cutoff, shutdown), waits for it, so two writers never share <path>.tmp and the newest cut lands last. A failed write surfaces from that wait; the previous file stays intact. The flag needs a checkpoint path and --checkpoint-every >= 2 outside policy sweeps (per-round checkpoints still persist before broadcast). Off by default; the files are byte-identical to the synchronous path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oad copy A reassembled INIT_PARAMS was copied out of its reassembly buffer, then hashed and decoded on the connection's read loop, so that stream stopped reading for each fragment's ~10 s digest (one sequential pass) and the learner's round-robin chunks backed up behind it. A 120B f32 INIT (483 GB) arrived at ~0.5 GB/s on a 50 Gb/s link. The reassembled frame now keeps its buffer (no copy), and an INIT_PARAMS is validated, hashed and decoded on the blocking pool while the read loop goes on; up to four per connection are in flight (each holds one fragment's payload and values) before the loop waits for the oldest. A failed decode is reported to the learner and ends the connection as before; the coordinator applies INITs in any order, from the current generation only. Direct (unchunked) INIT frames keep the inline path. Measured on the production syncer host with a synthetic 120B learner: the same INIT stream now takes 322 s (1.5 GB/s, bound by the Python client's send loop) instead of 16.9 min. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_send_large_parts filled each 4 MiB CHUNK envelope by allocating a zeroed bytearray and assigning slices into it. On a Kunpeng 920 that loop sends at most ~2 GB/s and measured 1.5 GB/s for a 120B INIT (483 GB, 322 s), the bound once the syncer keeps up. The envelope is now b"".join of its header and the part slices: one copy, ~5.9 GB/s in the same micro-benchmark. Bytes and validation are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When --total-steps is a multiple of --checkpoint-every, the periodic checkpoint of the last step and the final checkpoint wrote the same cut back to back. For the 120B state that is two ~14 minute writes of 966 GB at the end of a run (measured on the production syncer host: 10:44 -> 10:58 -> 11:12). The periodic write now skips the last step; the final checkpoint, which is always written when a checkpoint path is set, persists it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A late push waiting for its fragment's next round held a decoded f32 outer gradient (about 4P bytes per slow island at 120B). Queue the original compressed payload instead and decode it when the push is attached to the next round. The decode is deterministic, so merged results are bit-identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Drop the wire copy of a push once it joins a round. The compressed bytes exist so a late push can wait in the late-carry queue without its decoded gradient; a push accepted by its own round, or attached to a later one, kept that copy until the round was committed (up to 2P more bytes per bf16 island, on top of the gradient the streaming fold had just released). - Release the slot of a stale-base push under a streaming fold as well. Only the exact fold fixes its order and weights for the full roster; a streaming fold normalises over what it folded, and async carry (where stale bases happen) needs one, so the earlier guard left those rounds waiting for the quorum timeout again. - Decode PUSH payloads on the calling thread, as before the parallel INIT decode: a large push no longer spawns a thread pool per connection that competes with the merge threads. INIT_PARAMS asks for threads explicitly. - Assert the SHA-2 feature on the hardware hasher path in debug builds. - Document the worst-case memory of --background-checkpoint. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop a leftover `+ 0` and give the speed check headroom on a slow CI runner; the rescanning greedy it guards against took minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d without it A round's expected set is frozen at launch to the members not held by a catch-up or background resend. After a --resume, learners reconnect one by one and each spends a long time in its catch-up, so a round can be launched to the first learner alone. If that learner's H-gate keeps it from answering, the learners that finish their catch-up later were only pulled when the quorum timeout relaunched the round, and in-order commits stalled the whole fleet for that timeout. When a catch-up or recovery replay tail runs, the member now also joins every still-gathering round that has no slot, push or carried delta of its logical learner: it becomes expected in the current attempt and gets that attempt's pull. Attempt, quorum size, base version and the quorum, soft and grace deadlines are unchanged, so a push that arrives inside an armed grace window still counts. Completed rounds, rounds whose base the member was not caught up to, and exact folds that already hold pushes are skipped; an empty exact fold is rebuilt for the new roster. Fixed rosters are unchanged. A reconnect generation is still handled only by the rebind, so no round is pulled twice. The real-syncer protocol test now waits for that pull and expects the merged value with both answers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A capacity quorum is a share of the expected capacity, and its rule assumes the expected set only shrinks while a round gathers. A learner joining after its catch-up would raise the bar of a quorum already reached, and with an armed grace window turn a closing round back into one that waits for the soft deadline or the quorum timeout. Skip such rounds; count quorums keep the join. The design notes described the rendezvous as fully frozen; state the catch-up exception. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #335 (stacked on
pr/syncer-capacity).Summary
Pull a learner that finishes its catch-up into the rounds still gathering that were launched without it, instead of waiting for the quorum timeout to relaunch them.
Motivation
A round's expected set is frozen at launch to the members not held by a catch-up. After a syncer
--resume, learners reconnect one by one and each spends a long time in its catch-up, so a round can launch to the first learner alone. If that learner's H gate keeps it from answering, the others were only pulled when the quorum timeout relaunched the round, and in-order commits stalled the whole fleet for that timeout.Changes
docs/DESIGN.mdstates the exception to the frozen rendezvous.Testing
cargo testinsyncer/: 328 passed (new: a capacity-quorum round is not joined).test_agentic_suspend_harbor::test_rollout_processes_share_a_port_range...,test_dispense_syncer::test_islands_of_different_speeds_share_one_sequence,test_rl_local_init_real_syncer::test_digest_mismatch_stops_the_run_with_a_clear_error[island]); all pass when rerun alone.tests/test_delta_protocol.pynow expects the mid-round joiner to be pulled into the open round.scripts/precommit_check.py: clean.Run locally on macOS arm64, Python 3.11 with the CI install (
torch==2.8.0CPU,-r pyproject.toml --extra test),PYTHONPATH=. python -m pytest ... --timeout=600. Optional stacks that are not installed (Miles, harbor, lark-oapi, matplotlib) skip as on CI.Notes
--carry-answers-round), it is a small follow-up.🤖 Generated with Claude Code