Let the census be imported, and pin the three prefix lists to each other

[realmarcin]

Closes #95. Unblocks the guard half of #84.

The bug

scripts/fleet/prefix_census.py had its scan loop and its json.dump at module level, with no __main__ guard. So importing it ran the whole census — every record in all nine corpora — and overwrote _fleet/data/prefix_census.json.

I hit this for real while inspecting the prefix list:

import prefix_census as pc
print(pc.P.split("|"))        # only wanted the constant

Four minutes later the working tree held a census dated the 21st over a fleet_data.json, subsets_summary.json and assets/fleet/** from the 20th — the page would have dated itself to today above a heatmap built from yesterday. Reverted with git checkout --.

Why it was worth stopping to fix

The pipeline carries three independently hand-maintained prefix lists that must agree:

list	in	decides	size
P	prefix_census.py	what is counted at all	67
VOC	build_data.py	which vocabularies become heatmap columns	22
PREF / rx / STRICT	build_subsets.py	which cells and edges get clickable record lists	21

A mismatch is silent in both directions: a column whose vocabulary is never counted renders as a stripe of zeros, and a vocabulary counted but absent from VOC never reaches the page. That is the failure mode behind #84 — TOGO is counted by nothing and shown by nothing, and no check noticed.

A test would pin this cheaply. It could not be written, because reading P cost a four-minute scan and corrupted the data directory. So this defect was the blocker, not merely a wart.

What changed

The scan moved into census() behind a __main__ guard. P, rx and norm stay at module level, which is what a test needs.

import cost   4 min + a dirty working tree   →   0.046s, writes nothing

Four tests now pin the invariants — all three hold today:

• every heatmap column is a vocabulary the census counts ✅
• every clickable-cell prefix is a vocabulary the census counts ✅
• columns and cells name the same vocabularies, once roots.CITATION is set aside ✅
• importing the census module neither scans nor writes ✅

VOC and PREF are read by parsing their literals out of source rather than importing, since those two scripts also scan on import. The parse asserts it found a plausible list, so a reformat fails the test loudly instead of quietly checking nothing.

Canary

The guard could have broken the entry point, so I ran the script for real before committing: 2m48s, all nine corpora, every Mech non-empty, exit 0, well-formed output. Then restored the committed census — that run was a partial refresh and a real one means the whole pipeline. Verified byte-identical to HEAD afterwards.

Worth noting from that run: the corpora have moved again since the 20th — CommunityMech 364 → 393, TraitMech 694 → 722, MediaIngredientMech 2951 → 2952. Not folded in here; a refresh is a full-pipeline job.

Checks

27 tests (four added), assemble_page.py --check clean, working tree touches only the two source files.

🤖 Generated with Claude Code

[realmarcin]

Review pass

Two dimensions (refactor behaviour, test quality), findings independently verified by mutation-testing scratch copies. 5 raised, 3 confirmed — and the serious ones were in the tests I had just written, not in the refactor.

Confirmed and fixed in d2067a3

1. The guard regression test could not fail. (high + medium, same root cause)

test_importing_the_census_module_does_not_scan_or_write snapshotted the census, then called importlib.reload. But setUp had already imported the module — so with the guard removed, the scan ran during setUp, before the snapshot, and the reload rewrote identical bytes. The test compared a corrupted file against itself and passed.

Verified against a copy with the guard replaced by a bare main(): the old assertion passed. The test pinning #95 was vacuous.

It now imports in a fresh interpreter via subprocess, with a 60-second timeout as the real teeth — a scan takes minutes, so an unguarded module fails long before it finishes writing. Against the same mutant it now raises TimeoutExpired and fails.

2. The accepted-prefix set was too loose. (medium)

I built it as P ∪ norm.values(), which admits the 15 raw spellings norm exists to fold away:

CHEMBL.COMPOUND, IPR, MicrO, PFAM, UniProtKB, bacdive.isolation_source, cas,
doi, drugbank, gold.ecosystem, mediadive.compound, mesh, mibig, npatlas,
pubchem.compound

None is ever emitted as a key. A heatmap column named any of them would have passed the check and rendered as a stripe of zeros — precisely the failure the test exists to catch. It now takes what the census can actually emit (norm applied to every alternative in P): 53, not 68.

Mutation results

A test that cannot fail is worse than no test, so all four were checked against deliberately broken copies:

mutation	expected	result
remove the __main__ guard	import test fails	✅ TimeoutExpired (old version: passed)
add UniProtKB to VOC	column test fails	✅ fails (loose version: would pass)
drop CHEBI from PREF	symmetry test fails	✅ fails

Run against copies in the scratchpad; the repository was untouched throughout (git status clean, census byte-identical).

Refuted

• "A fourth list (build_subsets' own rx/NORM) stays unguarded" — a request for a test this PR never claimed to write, and the verifier found its supporting reasoning wrong.
• "literal() checks the source literal, not the runtime list" — true and deliberate; both scripts scan on import, which is why the parse asserts it found a plausible list.

Checked myself

• CI runs only unittest, refresh_manifest --check and assemble_page --check — nothing imports or execs the census, so the guard breaks no workflow step.
• Semantic equivalence of the moved loop, proved by comparing the for f body's AST between main and HEAD: identical. The original's odd 12-space indentation happens to be exactly right under the new function.

27 tests pass, --check clean.

[realmarcin]

Addendum — the final aggregation raised the count to four

My review comment above reported 3 confirmed. The workflow's final tally is 4 of 5; the fourth is fixed in 39122af.

literal() read the source text, not the module's actual list. It regex-matched the literal and eval'd the match, so a reassignment on the next line was invisible:

VOC=["CHEBI", ...]
VOC = VOC + ["BOGUSVOC"]     # the test never sees this

The runtime VOC then carries a column the census never counts — exactly what the membership test exists to catch — while the test reads the stale literal, finds 22 entries, and passes. The assertGreater(len, 10) guard cannot help: a mis-parse that still leaves eleven-plus entries is silent.

It now parses with ast, requires exactly one module-level binding of the name, and reads it with literal_eval, which refuses anything that is not a literal. Against a copy carrying that exact mutation it fails with VOC is bound 2 times in build_data.py; the regex version passed.

Full mutation results across the four tests, all against scratch copies:

mutation	result
remove the __main__ guard	✅ fails (TimeoutExpired) — old version passed
add UniProtKB to VOC	✅ fails — loose version would pass
drop CHEBI from PREF	✅ fails
VOC = VOC + ["BOGUSVOC"] after the literal	✅ fails — regex version passed

Every one of the four defects was in the tests I wrote, not in the refactor. That is the right ratio for a PR whose entire value is a guard: the guard has to be the thing under suspicion.

One acknowledged limitation, recorded rather than fixed: importing the two scripts would beat parsing them, but build_subsets.py scans records at import and builds MECHS at module level. That is the next #95, and not this PR's business.

27 tests, --check clean.