docs: #350 is measured, and the cap coming off does not unfreeze the board - #411
Merged
Merged
Conversation
…board v0.21.7 headlines "the bridge height cap is gone" (#398). True, but on its own it does not do what a reader would infer, and #350 appeared nowhere in the release notes or the CHANGELOG. MEASURED ON SERVER 1, 2026-09-18/19, not projected. The bridge reaches ~44 GiB RSS at h=798,257 (108.06M UTXOs), meets its MemoryMax=44G cgroup ceiling, throttles at ~99% memory pressure for two to three hours, and is OOM-killed. Two kills in dmesg: 20:26:49 and 01:08:48. It resumes from its checkpoint and does advance each cycle (798,257 -> 800,257 on the third), but the tip is 967,626 and the working set grows with the UTXO set. #398 removes HAZYNC_BRIDGE_TO only; EMIT_FROM=967500 still gates emission at a height the bridge cannot currently reach. ⚠ systemd reports active (running) for the whole frozen window, so nothing alerts. The 21:41 -> 01:08 freeze went unnoticed. Judge it by wchan=mem_cgroup_handle_over_high and flat utime+stime ticks, never by unit state. ⚠ AND THE SCOPE IS STATED, not left to inference: the provers are NOT waiting on this. Bundles exist contiguously to 418,268 against a frontier of 93,333, so ~322,000 blocks of witnesses already sit ahead of the fleet. #350 gates tip-following and the 418,269-967,499 gap; it does not gate proving throughput. Three files: the release notes gain the caveat next to the claim it qualifies, the CHANGELOG names #350 inline so it is not invisible, and STATUS.md's open-issue row stops calling a production crash loop a projection.
defenwycke
added a commit
that referenced
this pull request
Sep 19, 2026
…ed nowhere (#413) * docs: the bridge was OOM-killed 22 times, not twice — and it progressed nowhere Correcting two statements I put on main in #411, both wrong for the same reason. ⛔ THE COUNT. I wrote "two kills in dmesg, 20:26:49 and 01:08:48". dmesg is a RING BUFFER; those were simply the last two it still held. The journal says 22, between 20:27:08 and 06:35:29, and `systemctl show -p NRestarts` agrees. Each one fired an ntfy alert via OnFailure=hazync-alert@%n.service, so the operator was paged all night while the docs said it had happened twice. Count service failures from the journal, never from dmesg. ⛔ THE PROGRESS CLAIM. I wrote that it "does make forward progress each cycle (798,257 -> 800,257 on the third)". It does not. 800,257 was reached at 01:24 and is still the highest checkpoint ever; the last three resumes were all FROM 800,257. Once hazync-bridge-backfill grew 12.2 -> 21.5 GiB the bridge could no longer reach a 2,000-block checkpoint before hitting the cap, so every ~12-minute cycle reloads ~29 GiB of state, walks a few hundred blocks and dies. Also corrects the STATUS.md row, which named 798,257 as the ceiling; it is 800,257. ⚠ Unchanged, and still true: the provers are not waiting on this. The board gained ~900 blocks during the thrash. The fix is the cap raise in the draft PR, gated on the backfill finishing. * docs: alerts fire on the kill, not during the throttle — and fix the stale height Two fixes to the paragraph I just corrected. ⛔ IT CONTRADICTED ITSELF. The paragraph now says 22 kills fired an alert each, then ended "systemd reports active (running) throughout the frozen window, so nothing alerts". Both are true of DIFFERENT phases and the text ran them together. Each OOM kill alerts via OnFailure=hazync-alert@%n.service; the throttle that precedes it does not, which is why the 21:41->01:08 freeze passed unnoticed while the later thrash paged all night. Also: "~44 GiB RSS at h=798,257 (108.06M UTXOs)" was the first ceiling encounter. It reached 800,257 at 01:24 and has sat there since, at 112.1M coins. * docs: say plainly that the bridge is stopped, not still crash-looping The #350 paragraph describes the bridge in the present tense as OOM-looping and progressing nowhere. That was true when written; it is not true now. It was stopped at 06:54 on 2026-09-19 after 23 kills and ~36 alerts, deliberately, with its 800,257 checkpoint intact, and stays down until the memory cap is raised. Shipping the paragraph unqualified would tell readers a component is actively crash-looping when we turned it off on purpose. ⚠ AND IT SAYS WHY NOBODY IS AFFECTED, rather than leaving it implied: the bridge produces witnesses above h=418,268 and the frontier is below 94,000, so ~322,000 blocks of bundles already sit ahead of the fleet. The board kept advancing THROUGH the thrash and has kept advancing since. Neither half of the fix ships here, because neither is a release artifact: CKPT=200 (#414) and the 48G/52G cap raise (#412) are systemd drop-ins on server 1. The cap raise waits on the backfill walk releasing its ~21.5 GiB.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v0.21.7 headlines "the bridge height cap is gone" (#398). True — but on its own it does not do what a reader would infer, and #350 appeared nowhere in the release notes or the CHANGELOG.
Measured on server 1, 2026-09-18/19 — not projected
The bridge reaches ~44 GiB RSS at h=798,257 (108.06M UTXOs), meets its
MemoryMax=44Gcgroup ceiling, throttles at ~99% memory pressure for two to three hours, and is OOM-killed. Two kills indmesg: 20:26:49 and 01:08:48. It resumes from its checkpoint and does advance each cycle (798,257 → 800,257 on the third), but the tip is 967,626 and the working set grows with the UTXO set.#398 removes
HAZYNC_BRIDGE_TOonly.EMIT_FROM=967500still gates emission — at a height the bridge cannot currently reach.⚠ Nothing alerts on this
systemdreportsactive (running)for the entire frozen window; the 21:41 → 01:08 freeze went unnoticed. Judge it bywchan=mem_cgroup_handle_over_highand flatutime+stimeticks, never by unit state.⚠ The scope is stated, not left to inference
The provers are not waiting on this. Bundles exist contiguously to 418,268 against a frontier of 93,333 — roughly 322,000 blocks of witnesses already ahead of the fleet. #350 gates tip-following and the 418,269–967,499 gap; it does not gate proving throughput.
Three files
RELEASE_NOTES_v0.21.7.mdCHANGELOG.mddocs/STATUS.mdDocs only. No code, and #410's trial doc is deliberately not touched.