The vocabulary census misses most of the fleet's largest namespaces, not just TOGO

[realmarcin]

Scope corrected 2026-09-20. This was filed as "three prefixes are missing". An ignore-independent scan of every record in all nine measured corpora shows the omission is far wider: the census misses twelve of ProteinTraitsMech's namespaces, several larger than anything it does count.

What the census does not count

Structured CURIEs only (id:/list-item position), excluding each Mech's own minted ids and schema metamodel prefixes (biolink, rdfs, skos):

occurrences	prefix	Mech
617,470	CATH	ProteinTraitsMech
446,166	CDD	ProteinTraitsMech
336,591	PROSITE	ProteinTraitsMech
152,324	ECOD	ProteinTraitsMech
68,418	SCOP	ProteinTraitsMech
44,728	IEDB	ProteinTraitsMech
40,653	GOC	ProteinTraitsMech
27,720	AlphaFoldDB	ProteinTraitsMech
20,000	OrthoDB	ProteinTraitsMech
17,835	SMART	ProteinTraitsMech
15,541	PANTHER	ProteinTraitsMech
14,863	pdb.ligand	ProteinTraitsMech
3,611	UNII	AntibioticMech
3,449	genbank	NaturalProductMech
3,236	komodo.medium	CultureMech
2,833	TOGO	CultureMech
2,829	mediadive.medium	CultureMech
2,058	pubchem.aid	NaturalProductMech
1,732	reaxys	AntibioticMech
1,232	pubchem	NaturalProductMech
1,176	mediadive.solution	CultureMech
1,062	patent	AntibioticMech

Smaller: kegg.compound 862, wikipedia.en 738, kegg.drug 598, uniprot.location 583, MEDIADB 459, drugcentral 435, kgmicrobe.ingredient 366, kgmicrobe.compound 232, CCAP 76, cyanometdb 56, bergey 55, chembl 41, UO 17, lotus 9, RNAcentral 3, RCSB_PDB 3, CryoETDataPortal 2, HAO 2.

For scale: CATH alone is the fleet's fourth-largest namespace, behind only InterPro (1.56M), UniProt (770k) and PMID (658k). ProteinTraitsMech's card already names CATH, SCOPe, ECOD, PROSITE and CDD among its sources, so the page advertises groundings its own heatmap does not count.

Two genuine normalisation bugs

• pubchem (lowercase, 1,232 in NaturalProductMech) is not folded into PubChem, although pubchem.compound is. The same registry is counted under one spelling and dropped under another.
• RCSB_PDB is not folded into PDB.

Candidates for the same treatment: kegg.compound/kegg.drug → KEGG, mediadive.medium/.solution/.ingredient → MediaDive (only mediadive.compound is mapped today).

Why this is not a quick fix

The alternation P is a hand-maintained allowlist, so every omission is silent — nothing fails when a namespace is missed. Extending it is not mechanical, because it forces a definition the pipeline has never written down: what counts as a vocabulary?

• Each Mech's own minted ids (proteintraitsmech: 91,559, CultureMech: 7,951, naturalproductmech: 6,604 …) are identity, not citation, and must stay out.
• Schema metamodel prefixes (biolink 184,587, skos, rdfs) describe the model, not the records.
• Cross-Mech references (MediaIngredientMech: 1,166 in CultureMech) are already the cross-reference list's job, not the heatmap's.
• Registry identifiers (genbank, patent, reaxys, UNII) are not ontologies. The page's own note already hedges on this for IMG and BacDive.

Adding twelve columns also reshapes the heatmap: the vocabulary tile jumps, column ordering shifts, and ProteinTraitsMech's row would dominate the scale even more than it already does.

Suggested approach

1. Decide the inclusion rule and write it in roots.py beside CITATION, which already records one such decision.
2. Derive the allowlist from each Mech's schema prefixes: block rather than hand-maintaining P, so a new namespace cannot be silently dropped again.
3. Fix the two normalisation bugs regardless — they are wrong under any rule.
4. Rerun prefix_census.py → build_subsets.py → build_data.py → assemble_page.py and look at the heatmap before committing.

Evidence produced by scanning all 448,844 records across the nine measured corpora (~4 min). Not fixed in #89: that branch refreshes the numbers within the current methodology, and this changes the methodology and the figure.

[realmarcin]

The shape of this is different from how it was filed

Tracing the pipeline turned up something the issue does not say: the fleet page carries three independently hand-maintained prefix lists, not one.

list	in	decides	size
P (regex alternation)	prefix_census.py	what gets counted at all	67
VOC	build_data.py	which vocabularies become heatmap columns	22
PREF / rx / STRICT	build_subsets.py	which cells and edges get clickable record lists	21

This matters for how the issue gets fixed, in two ways.

Adding prefixes to P does not widen the heatmap. I had assumed it would, and warned above that twelve new columns would reshape the figure. It would not: the heatmap is capped by VOC, which is a separate list. So the counting fix and the column decision are independent changes, and the counting fix is safe on its own. That removes the reason I gave for deferring it.

Two effects the counting fix would still have, both correct:

• The "vocabularies in dated census" tile rises from 45 toward ~65, since it counts distinct prefixes the census saw.
• Variant spellings folding into existing columns change two real cells: kegg.compound + kegg.drug → KEGG gives AntibioticMech 1,460 where it currently shows none, and RCSB_PDB → PDB adds 3 to CellStructureMech. Those are corrections, not additions.

The column question is the separate, harder one. The census counts 45 vocabularies; the heatmap shows 22. So CATH's 617,470 occurrences are invisible for two reasons stacked — not counted by P, and not listed in VOC — and fixing only the first leaves it invisible. Deciding which of ~65 deserve a column interacts directly with #58 (numbers already at ~10px to buy the fit) and #55 (row labels ellipsizing at minimum width). That is a layout decision, not a prefix decision.

A guard is now in place

The three lists are consistent today — I checked, and all three invariants hold:

• every VOC column is something P counts ✅
• every PREF prefix is something P counts ✅
• ignoring citation prefixes, VOC and PREF name the same vocabularies, so no column is dead and no record list is orphaned ✅

That was untestable until now, because importing prefix_census to read P ran a four-minute scan and overwrote the committed census (#95). With that guarded, four tests pin the invariants, so the three lists can no longer drift apart silently — which is the failure mode that let TOGO go missing in the first place.

Suggested split of what remains here:

1. Counting — extend P, fix the pubchem/RCSB_PDB normalisation bugs, rerun the pipeline. Safe, no layout impact.
2. Columns — decide what the heatmap can show, with Heatmap numbers dropped to roughly 10px to buy the fit #58 and Heatmap row label ellipsizes at the table's minimum width with no way to read it #55 on the table. Blocked on that decision, not on this one.