verifier cutover: move the size pins, and make a stale site page someone - #417
Merged
Merged
Conversation
check-deployed-verifier.sh compares the README's `raw` figure against the DEPLOYED module, not against the release. So these pins must move in the SAME cutover as the site: update them first and the check goes red, update the box first and it also goes red. They are one change, not two. Measured, not copied: gzip -c < hazync-verify.wasm = 295,130 bytes (old 295,077). THREAT_MODEL.md carried "both measured 200, 1,064,517 bytes, on 2026-09-14" — the size is now v0.21.7's and the claim is dated to the cutover rather than left to imply it is still current.
… not ship
⛔ THE HOLE. On 2026-09-19 the site served v0.21.6's module for hours after v0.21.7
published and nothing noticed. release.sh classes `deployed-verifier` as IGNORABLE_JOB
because that job tests the LIVE SITE and is red BY CONSTRUCTION between a release and
its cutover — so a green release is compatible with a stale site. Afterwards, nothing
looked again.
⚠ AND THE CHECK ALREADY EXISTED. scripts/check-deployed-verifier.sh was written, correct,
and exited 1 on exactly this condition. It was installed on no box and wired to no timer.
A check that runs nowhere cannot fail, which is the same shape as a probe whose silence
reads as success.
Two parts:
1. hazync-check-deployed-verifier.{service,timer} — hourly at :37, through hazync-run-check
so a failure reaches ntfy via OnFailure=hazync-alert@%n.service. CHECK_FAILS_BEFORE_ALERT=2
because a release and its cutover are minutes apart and one transient fetch is not a stale
deployment.
⛔ The unit deliberately does NOT set ALLOW_SKIP=1. The script honours it and exits 0 when
tooling is missing — right for a laptop, exactly wrong here, and it would rebuild the very
hole this closes.
2. release.sh now NAMES the cutover in its closing block, next to the other things it cannot
do itself, and points at the one command that proves it landed. The script already knew
`deployed-verifier` was ignorable; it never said what the operator owed as a result.
Verified before committing: the check FAILED against the stale site ("release asset is
1065400 bytes, the deployment is 1064517") and PASSES after the cutover. It can fail, and
it can pass.
defenwycke
changed the title
⛔ I PROPOSED MACHINERY FOR A SOLVED PROBLEM. adversarial.yml's deployed-verifier job already runs on a schedule — `if: github.event_name == 'schedule' || workflow_dispatch` — and last night's scheduled run has it completed/success. A systemd timer would duplicate it on boxes that cannot even run it: the script needs `node` to INSTANTIATE the wasm and call method_id(), and neither the coordinator nor the web box has node. Installing a JS runtime on the box that holds the ledger, to re-run a check GitHub already runs, is a worse trade than the gap it closes. ⚠ AND THE GAP IS WORSE THAN A CRON LINE SUGGESTS. The workflow says '0 3 * * *', but GitHub defers scheduled runs under load: the last five started 07:33, 07:37, 07:59, 07:56 and 08:02 UTC. v0.21.7 published at 08:46 — 45 minutes AFTER that morning's run had already passed. So the stale site would not have been noticed until the next morning's run, roughly 23 hours later. Do not size this from the cron expression. ⚠ THE REAL FIX IS PROMPTING, NOT MONITORING. Nothing told the operator the cutover was owed; release.sh finished green and silent. The closing note added in the previous commit addresses that directly, and is what stays. What stays: the size pins, and release.sh naming the cutover in its closing block.
defenwycke
force-pushed
the
docs/verifier-cutover-v0217
branch
from
September 19, 2026 10:11
22130aa to
8fdc54f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two halves of the same incident.
1. The cutover's repo-side size pins
check-deployed-verifier.shcompares the README'srawfigure against the deployed module, not against the release — so these move in the same cutover as the site. Update them first and the check goes red; update the box first and it also goes red.Gzipped measured, not derived:
gzip -c < hazync-verify.wasm.docs/THREAT_MODEL.mdcarried "both measured 200, 1,064,517 bytes, on 2026-09-14" — now the v0.21.7 size, dated to the cutover rather than left implying it's current.2.
release.shnow says the cutover is owedhazync.orgserved v0.21.6's verifier for ~80 minutes after v0.21.7 published, and nothing prompted anyone.release.shclassesdeployed-verifierasIGNORABLE_JOBbecause that job tests the live site and is red by construction between a release and its cutover. The script knew the job was ignorable; it never said what the operator owed as a result. It finished green and silent.So it now names the cutover in its closing block, beside the other things it can't do itself, and points at the one command that proves it landed.
⚠ What I removed, and why
I first added
hazync-check-deployed-verifier.{service,timer}. That was wrong —adversarial.yml'sdeployed-verifierjob already runs on a schedule, and last night's scheduled run has itcompleted/success. A systemd timer would duplicate it on boxes that cannot run it: the script needsnodeto instantiate the wasm and callmethod_id(), and neither the coordinator nor the web box has node. Installing a JS runtime on the box holding the ledger, to re-run a check GitHub already runs, is a worse trade than the gap it closes.⚠ Do not size the gap from the cron expression. The workflow says
0 3 * * *, but GitHub defers scheduled runs under load — the last five started 07:33, 07:37, 07:59, 07:56, 08:02 UTC. v0.21.7 published at 08:46, 45 minutes after that morning's run had already passed, so a stale site would have gone unnoticed until the next morning.That makes this a prompting failure, not a monitoring one — which is what the
release.shnote fixes.Verified both ways
Before the cutover the check failed:
release asset is 1065400 bytes, the deployment is 1064517 — the site is serving something the release did not ship.After, it passes — deployed size = release = README,
method_idcanonical, live spine verifies through block 79,706, bit-flipped proof rejected. It can fail, and it can pass.