Found in the adversarial review of #120.
Record-count equality compares two glob counts, so a partial read passes silently
Where: scripts/fleet/prefix_census.py:28
test_census_and_stats_count_the_same_records compares census files with stats records. Both are len(record_paths()), so at the same revision they are equal by construction. prefix_census increments n before try: open(...) except Exception: continue. build_subsets skips unreadable files the same way and records nfiles nowhere. A scan that reads only part of the corpus still writes a full files count, exits 0, and passes the gate and the skill's 'files equals the step 3 count' check.
Failure scenario. During this review, another process deleted a shared scratch snapshot while build_subsets.py was reading it. The run exited 0 having read 312,792 of 429,293 ProteinTraitsMech records. It wrote 20 changed asset files (ProteinTraitsMech--PATO total 81 -> 27, labels changed), and nothing in the pipeline or tests flagged it. The same can happen when a shared checkout is switched or pulled mid-scan, or on a permissions error. The census would still report files=429293 with prefix counts from fewer files.
Evidence. Contaminated run log /private/tmp/claude-501/review-pr120/det/run2.log: 'ProteinTraitsMech files 312792 unlinked 0 terms 6037' (clean run: 429293 / 7050), exit 0. Census fixture with one chmod-000 record under /private/tmp/claude-501/review-pr120/tg-f66c2125/unreadable: {'TraitMech': {'files': 2, 'prefixes': {'METPO': 1}}}. The unread file's GO:0000001 is missing, but files still says 2.
Verifier (confirmed, low). Every factual claim reproduces. A scan that fails to open some records still writes the full glob count as files and exits 0. build_subsets.py only prints how many files it actually read and never saves it. The PR's new count-equality test and the skill's "files equals the step 3 count" check both compare len(record_paths()) with len(record_paths()), so at one revision they are equal by construction. They cannot detect a partial read.
The revision gates don't help either: an unreadable file leaves the checkout clean. The census counts its vocabulary terms from fewer files, the subset and cell files are written with wrong totals, and every provenance assertion still passes.
Three caveats keep the severity at low:
- The swallowing
except Exception: continue in both scan scripts predates this PR (it is identical on main). What the PR adds is a check that looks like it verifies the read but doesn't. The skill is the misleading part, because it calls this a side-effect check rather than an exit-code check. The test's own comment only claims to catch partial reruns, and it does catch those.
- A persistent permissions error is not always silent. For Mechs whose schema tracks review (TraitMech in the fixture), mech_stats.py opens every record without a try and exits 1 with PermissionError. It is silent for the Mechs that don't track review (CultureMech, CommunityMech, MediaIngredientMech), and for any transient failure, such as a snapshot deleted, switched or pulled during the census or build_subsets and restored before mech_stats runs.
- The contaminated run was a reviewer's scratch run. The committed PR data is unaffected: census and stats agree for all Mechs, and ProteinTraitsMech--PATO has total 81, the clean value.
So this is a real hole in the new verification, but it takes an environmental fault during the scan to trigger, and nothing in the shipped data is wrong.
Suggested fix. Count files actually read, and fail (SystemExit) if any globbed record could not be opened, in both prefix_census.py and build_subsets.py. Optionally record build_subsets' per-Mech nfiles in subsets_summary.json and assert it equals census files.
Found in the adversarial review of #120.
Record-count equality compares two glob counts, so a partial read passes silently
Where:
scripts/fleet/prefix_census.py:28test_census_and_stats_count_the_same_records compares census
fileswith statsrecords. Both are len(record_paths()), so at the same revision they are equal by construction. prefix_census incrementsnbeforetry: open(...) except Exception: continue. build_subsets skips unreadable files the same way and records nfiles nowhere. A scan that reads only part of the corpus still writes a fullfilescount, exits 0, and passes the gate and the skill's 'files equals the step 3 count' check.Failure scenario. During this review, another process deleted a shared scratch snapshot while build_subsets.py was reading it. The run exited 0 having read 312,792 of 429,293 ProteinTraitsMech records. It wrote 20 changed asset files (ProteinTraitsMech--PATO total 81 -> 27, labels changed), and nothing in the pipeline or tests flagged it. The same can happen when a shared checkout is switched or pulled mid-scan, or on a permissions error. The census would still report files=429293 with prefix counts from fewer files.
Evidence. Contaminated run log /private/tmp/claude-501/review-pr120/det/run2.log: 'ProteinTraitsMech files 312792 unlinked 0 terms 6037' (clean run: 429293 / 7050), exit 0. Census fixture with one chmod-000 record under /private/tmp/claude-501/review-pr120/tg-f66c2125/unreadable: {'TraitMech': {'files': 2, 'prefixes': {'METPO': 1}}}. The unread file's GO:0000001 is missing, but files still says 2.
Verifier (confirmed, low). Every factual claim reproduces. A scan that fails to open some records still writes the full glob count as
filesand exits 0. build_subsets.py only prints how many files it actually read and never saves it. The PR's new count-equality test and the skill's "filesequals the step 3 count" check both compare len(record_paths()) with len(record_paths()), so at one revision they are equal by construction. They cannot detect a partial read.The revision gates don't help either: an unreadable file leaves the checkout clean. The census counts its vocabulary terms from fewer files, the subset and cell files are written with wrong totals, and every provenance assertion still passes.
Three caveats keep the severity at low:
except Exception: continuein both scan scripts predates this PR (it is identical on main). What the PR adds is a check that looks like it verifies the read but doesn't. The skill is the misleading part, because it calls this a side-effect check rather than an exit-code check. The test's own comment only claims to catch partial reruns, and it does catch those.So this is a real hole in the new verification, but it takes an environmental fault during the scan to trigger, and nothing in the shipped data is wrong.
Suggested fix. Count files actually read, and fail (SystemExit) if any globbed record could not be opened, in both prefix_census.py and build_subsets.py. Optionally record build_subsets' per-Mech nfiles in subsets_summary.json and assert it equals census files.