[stephrobert]

Summary

go test -run <name> <pkg> over a package holding no such test runs nothing and exits 0, and the harness read that zero as "the guard is not held". So falsify printed TEST STILL PASSED for guards that were held all along: a verdict over an empty population, in the instrument this repository built to catch exactly that.

Measured on 2026-11-13: a mutation of a-transport-fault-is-bounded.json declared ./internal/core/emulator/ and named a test living in internal/cli. Run by hand against the package that holds the test, the same mutation failed in 5.02 s.

It fails towards alarm, which is the less dangerous direction, and it has a silent half: a mutation whose test was renamed keeps the old name, matches nothing, and reports TEST STILL PASSED for ever, until somebody "fixes" it by deleting the mutation. The guard then has no falsification and nothing says so.

What changed

Every declared test must appear in a go test -list of its declared package, or the spec is refused. One listing per run, grouped by package, memoised for the process. It runs inside falsify:lint, so prepush catches a stale name, and again before the tree is copied, so a void run costs half a second instead of a copy plus a suite.

The new state is kept apart from the three verdicts the harness already draws, and the comment at the call site says so in those words: a copy red before any mutation measured no guard; a refused spec measured no guard either; a mutation that applied and left its test green is a verdict. This one is none of them — the mutation was never applied and the test was never run.

It also asks at the second depth, and that was found by falsification rather than foreseen. go test -run 'TestX/no-such-sub' runs TestX, no subtest matches, the parent passes, exit 0 — the same defect one level down. 13 committed mutations name a subtest, nine of them rows of a single table in s3-bucket-configuration.json. Renaming one row would have silenced nine mutations with nothing to say so. Cost: three real parent runs, 0.9 s, so there was no cost argument for leaving it out.

Sweep of the committed specs

every declared fragment still applies: 1665 mutation(s) across 251 spec(s)
every declared test exists in the package its mutation names: 1665 name(s), 13 of them a subtest really entered under its parent

Exit 0. No offending triple at either depth. The issue asked for this sweep explicitly, because nobody knew whether a committed spec already named a test that does not exist.

Type of change

• Bug fix
• A route added or changed
• Refactor
• Documentation
• Chore / tooling
• Security / supply chain

What the contradicteurs found, and what it cost

Two roles, two cycles each, which is the maximum the project allows. Four BLOCKERs, all the same shape: the guard this lot adds was not held by its own tests.

• covered_paths' package filter could be replaced by return set(known) and the selftest stayed green, because a one-package module makes "in this package" and "anywhere in the listing" the same set. That is the very triple falsify reads "no tests to run" as a guard that is not held #923 exists for.
• _unbuildable_case() passed without the branch it exists for, because a broken package makes the name missing anyway.
• The pre-copy wiring in main() was unmeasured: unknown = [] reprinted TEST STILL PASSED verbatim, measured end to end.
• A stale subtest name was still accepted, which is the second depth above.

Twelve selftest assertions now hold seven controls, and each was proved by mutating it in a copy outside the repository: the package filter, the untrustworthy listing of a tree that does not build, the pre-copy wiring, the subtest check and its accepting half, a parent that cannot answer, and an unusable pattern. That last one is a false proof rather than a void one: go test rejects the pattern, exits non-zero, and the replay reads non-zero as "the test bit".

The most instructive result is not in the diff. The author's first versions of two of those cases were green under both mutants, because the specs named file relatively while lint resolves it against the process's working directory, so both answered through the stale-fragment path and never reached the new check. It was found by mutating the new code, not by reading it. That class is #938.

Runs

run	verdict
mise run prepush	green, on the rebase onto 2ab0c24
mise run falsify:selftest	green, 32 assertions
python3 tools/falsify/falsify.py --lint tools/falsify/specs	green, the two lines above
one table row renamed in a copy of the specs	exit 2, naming the parent, the subtest and the consequence
mise run conformance:leg -- probe	green, --vm off
the 5 falsify specs mise run testplan names	all 5 exit 0

Mode: --vm off throughout. Nothing here touches a machine runtime, so no leg under FEINT_VM is owed.

Costs, with the invocation that produced them

Measured on 2026-11-14 on the maintainer's station. The invocation is part of the figure, because mise run uses the pinned Python 3.12.11 while a bare python3 resolves 3.12.13 on that station, which is a factor of two on a sub-second task:

• falsify:selftest: 1.37 to 1.44 s through mise run, 2.92 to 3.04 s through a bare python3, same 32 assertions either way, against 0.03 s for the build-free version.
• falsify:lint: 2.54 to 2.63 s through mise run, 2.77 to 2.89 s bare; the three subtest parent runs are 0.9 s of it.

An earlier draft published 3.0 s for the selftest and explained it with a mechanism the code does not have ("the module path is new on every run", where it is a constant). Both were withdrawn and remeasured; the counterfactual was measured too, and a module name the build cache had never seen costs the same, because the cache keys on the bytes compiled and not on the path.

Runs not played

mise run falsify:all, which belongs to the night. What it would add over the lint is that the new pre-copy refusal changes no existing verdict across all 251 specs. The five specs testplan names are the subset this diff can have broken, and they are green.

Three stale sentences went with it

tools/testplan/specs.go claimed 128 specs and 186 files, measured now at 251 and 295; falsify:selftest said "three mutations" where its own output prints four; lint()'s docstring named one half of what it now checks. The wider version of that is #937.

Left to their own issues

• falsify:lint and falsify:selftest are run by no pull request, and CONTRIBUTING says they are #936: falsify:lint and falsify:selftest are run by no pull request, while CONTRIBUTING.md says prepush is what a pull request refuses. This lot makes the lint the only thing between a renamed test and a false TEST STILL PASSED, and it lives in a hook a fresh clone does not have.
• Five hand-written numbers about the falsify specs are stale, and nothing recomputes them #937: five hand-written numbers nothing recomputes.
• A falsify selftest case can pass for the wrong reason: 2 is the answer for seven of them #938: a selftest case can pass for the wrong reason, 2 being the answer for seven of them, plus four behaviours of this file that no case holds.

Checklist

Always

• mise run check passes (inside prepush, on the rebase)
• No new external Go dependency; stdlib only, in Python as in Go
• internal/core still knows no provider
• Nothing in the diff could be written identically for another provider
• mise run testplan was run, and what it printed was run; the one run not played is named above

When a model wrote a substantive part of this

• An Assisted-by: trailer names the tool and the model version
• No field name and no response shape changes
• Every number published carries its measurement and its invocation

When a route is added or changed

N/A.

When behaviour a client can observe changes

• The two conformance pages gain the fifth outcome, in both languages

When .github/workflows/ is touched

N/A.

When a machine runtime is involved

N/A.

Related issues

Closes #923.

🤖 Generated with Claude Code