Script sweep: every gate in one compiled language, and no gate invoked by nothing

[avrabe]

[fathom (gale) — script sweep: every gate in one compiled language]

A sweep of every script in the repo, to decide what is actually used and move
verdict-bearing logic off shell. Inventory is measured, not estimated.

Inventory

80 tracked .sh/.py — 60 shell, 20 Python.

bucket	count
invoked by a workflow	24 (12 sh / 12 py)
referenced only by another script	18
referenced by neither ("dark")	38

The dark set is not rot: only 4 are untouched >90d. The other 34 are recent
silicon/bench operator tools. But 5 of them are gate-like — they assert
properties and nothing runs them:

11  silicon/wasm-testbed/arch/riscv/run_rv_funccheck.sh
 6  benches/gust/optimization/wsc-facts-phase1.sh
 5  silicon/wasm-testbed/resurvey_struct_return_decides.sh
 3  silicon/wasm-testbed/arm_harness.py
 3  benches/gust/run-fused.sh

This is the shape emit-wcet.sh already had: a real gate, written, invoked by
nothing. It was catalogued in #324 and dismissed as "builders, and less alarming".

Why this is worth doing

Hazard classes in the 12 CI-invoked shell scripts: 23 || true, 20
2>/dev/null, 12 verdicts read through a pipe, 11 grep -c/wc -l as a
verdict. Not all are defects — some are correct and commented. But each is a
place where a failure becomes a passing verdict, and reviewing them one at a time
is the only way to tell which.

Re-reading a single script the sweep started on produced four real defects (fixed
in abd76eb):

1. cmp A B && echo "byte-identical" — in an && list a failing left-hand
   command is exempt from set -e. A drifted object printed its diff to stderr
   and the script ran on to print ALL GATES GREEN. Verified directly: cmp a b && echo on differing files exits 0.
2. Fixing (1) exposed the comparison asserted the wrong thing — it compared
   current-synth output against a synth#757 reproduction frozen at 0.45/0.46.
   The file's header already stated the real claim (--emit-wcet does not
   perturb the object) and nothing checked it.
3. It resolved synth from a path on one laptop and no runner, and the job never
   ran varve install.
4. It overwrote a committed sidecar every run, so checking the evidence rewrote it.

And in a Python gate: check-object-freshness.py claimed to census every
committed object while globbing */*-cm3.o — one directory deep, one suffix. 24
committed, 22 seen. Language is not the whole story; enumeration is.

Target

An xtask crate. There is no Rust workspace or xtask today, so it has to be
built first. Every gate becomes cargo xtask check <name>, conforming to the
CLI baseline (--version/-V, --help exit 0, unknown flag exit 2,
--format json).

Scope decided: one language. The 7 Python gates port too, not just the
shell — a repo with gates in three languages cannot state a single rule about
how a verdict is produced.

The safety mechanism

The failure mode of this migration is porting a gate into one that is green
because it cannot distinguish — the exact defect class the sweep exists to
remove. So no gate is retired on review alone:

1. Port the gate to xtask, including its negative control (--self-test).
2. Run old and new in the same job and assert the verdicts agree, on the
   real inputs, for at least one merge cycle.
3. Only then delete the original.

A port whose negative control does not fire is not done.

Phases

• 0 — xtask skeleton, CI harness, one gate end-to-end as the pattern.
• 1 — the 7 Python gates, one PR each, shadow-then-retire.
• 2 — verdict-bearing shell (check-wit-resolve.sh, emit-wcet.sh's embedded
  Python, the build scripts' assertions).
• 3 — the 5 dark gate-like scripts: decide gate-or-tool, wire the gates.
• 4 — label the remaining operator tools so "dark" stops being ambiguous;
  delete the 4 stale after confirming nothing needs them.

Phases land against the 2-PR ceiling (~60-75 jobs each), so this is paced, not
a single sweep.

Non-goals

• Rewriting operator tools. Shell is fine for loud orchestration a human runs.
• build-cross-arch.sh stays shell: it is used only as a negative control, where
  its failure is the assertion.

[avrabe]

[fathom (gale) — Phase 0 done, Phase 1 started]

Branch feat/xtask-gate-migration (no PR yet — the queue is at the 2-PR ceiling
with #367 and #369).

Phase 0. tools/xtask, standalone with an empty [workspace] — the idiom
crates/gale-kiln already uses. Not a root workspace: the root is a plain
package and every bench crate pins its own target and rustflags, so one would
force all of them into a members/exclude list. A cargo alias keeps cargo xtask
without that.

What compiling actually buys, which turned out to be the whole point: a gate that
cannot run is not a gate that passed. Verdict has four outcomes with
distinct exit codes — 0 pass, 1 the property failed, 2 usage, 3 could not answer.
All four are exercised.

Ported so far (2 of 7+):

gate	what the port changed
wit-resolve	had no negative control at all; the added matched pair immediately caught that the planted "good" world was malformed (unversioned import), which would have made the clean half a false negative
data-overlap	three corrections, below

data-overlap is the one worth reading. The Python reports exit 0, "no data
segments — nothing to check" on a module that plainly has one:

$ wasm-tools print passive.wasm | grep data
  (data (;0;) "AAAA")
$ python3 check-data-overlap.py passive.wasm
  no data segments — nothing to check     # exit 0

A passive segment has no (i32.const OFF), so the regex skips it. If
wasm-tools print ever changes shape, that happens to every segment at once, on
any module, and the gate stays green. The port classifies each occurrence
Active / Passive / Unknown; Unknown refuses.

It also mis-counted \\41 (a backslash byte, then '4', then '1' = 3 bytes) as 2,
because it substituted the hex-escape pattern first and scanned from the second
backslash. A short payload length shrinks a segment's computed extent — which is
precisely how a real overlap gets reported disjoint.

Both are in the shadow period: old and new run together and their verdicts must
agree before the old is deleted. The agreement step also refuses the case where
both agree the property is broken, so "they agree" cannot become a way to pass a
red.

Remaining Phase 1: check-cross-arch (295), check-driver-components (322),
check-graph-env (165), check-object-freshness (283, in flight on #369),
check-wcet-evt (477), check-required-contexts (210). The last two need a YAML
parser, so that is where a dependency gets added deliberately rather than by
hand-rolling a parser — hand-rolled YAML is the same defect class in a new place.

[avrabe]

[fathom (gale) — the sweep's rule applies to workflow YAML too, and there it was worse]

The inventory counted scripts. It did not count the shell embedded in run:
blocks, and that is where the largest instance of this defect class was sitting.

#367 failed on llvm-lto-test (msgq) — a Zephyr kernel test, on a PR touching
only rivet YAML. Root cause, from the job's own step conclusions:

success  #7  Initialize west workspace      <- west sdk install FAILED 3x here
failure  #8  Setup SDK paths
skipped  #9  Build and run test (LLVM+LTO)

The install step passed while installing nothing; the failure resurfaced one step
later wearing a different name. The idiom:

for attempt in 1 2 3; do CMD && break || sleep 5; done

All attempts fail → last command run is sleep → loop exits 0. bash -e (what
Actions uses) does not help: a failing command in an &&/|| list is exempt, and
the list's status is the final sleep.

28 of them, 10 workflows — every west update and west sdk install in the
repo. Fixed in #370 with a guard (tools/check-retry-loops.py, negative-controlled
on the real tree) so it cannot come back by copy-paste.

Two things this changes about the plan above:

1. The inventory undercounts. 80 tracked .sh/.py files was the wrong
   denominator; shell inside run: blocks is unversioned, unreviewed as code,
   and — as here — carries verdicts. Phase 4 should sweep workflow run: blocks
   as a first-class surface, not as an afterthought.

2. The hazard scan was tuned for the wrong spellings. It counted || true
   and 2>/dev/null. && break || is the same defect and scored zero, because
   nobody wrote the word "true". The general shape is a trailing branch chosen
   to succeed, and the guard now matches that rather than one spelling.