Found in the adversarial review of #120.
Audit check covers Mech shas only: CLAW pin, card_records and merged_prs in site_audit are unchecked despite the PR's 'agree on revisions and record counts'
Where: tests/test_fleet_page.py:440
The PR body says RefreshProvenanceTests 'requires the census, the stats and site_audit.json to agree on revisions and record counts'. Only the audit's sha is compared, and only for Mechs in mech_stats. Unchecked: the audit's record figure (card_records, which by design equals the template card figure), its merged_prs (which SKILL.md step 7 says to derive from mech_stats), and CLAW's audit sha against manifest.json source.revision. Extra stale repos in the audit are also unchecked.
Failure scenario. The next refresh updates the shas in site_audit.json but carries over last run's card_records and merged_prs, or leaves CLAW at an old sha after refresh_manifest.py re-pins the manifest. The provenance record then misstates what was audited and the gate stays green.
Evidence. mutate.py in the isolated copy: M9 CLAW audit sha != manifest source.revision -> OK. M10 every audit merged_prs -50 and card_records -7, shas current -> OK. M15 audit gains a non-member repo -> OK. Today the values do agree: every audit merged_prs equals stats, card_records equals the template cards (CommunityMech 422 vs stats 426 is expected), and CLAW 2637172 equals manifest.json.
Verifier (confirmed, low). Every factual part of the finding reproduces. The only audit check is the per-Mech sha, and it runs only for repos that appear in mech_stats.json. The audit's merged_prs, card_records, the CLAW sha (against manifest.json source.revision) and any extra or stale repo entries are all unchecked. site_audit.json is not read by any other script or CI job, so these fields have no gate at all. Mutating them leaves the whole 33-test suite green. A control mutation of a Mech sha does fail, which shows the harness would catch the covered case.
Two caveats keep the severity low.
- What the PR overclaims. The PR sentence can be read distributively: revisions agree across all three files, and record counts agree between the census and the stats, which is what the code checks. The audit's card_records cannot equal the census/stats count by design (CommunityMech 422 vs 426), so no record-count equality with the audit was ever achievable. The overclaim is the loose wording, not a broken invariant. SKILL.md and _fleet/README.md describe the scope accurately as revisions/SHAs only.
- Current data and impact. Today every unchecked value is correct: merged_prs match the stats, card_records match the cards, and CLAW 2637172 matches the manifest. site_audit.json is not rendered on the site. The failure is future-only: on a later refresh, the provenance record could keep last run's merged_prs or card_records, or a stale CLAW sha, and the tests would not notice. It misleads a reader of the JSON, not a visitor to /mechs/.
The suggested fix is sound, and the functions it needs exist: check_cards.cards(template) is at scripts/fleet/check_cards.py:61, and manifest['source']['revision'] is present. Alternatively, the PR text could drop "and record counts" or scope it to census vs stats.
Suggested fix. Extend test_the_audit_pins_the_revisions_the_stats_counted to assert audit merged_prs == stats merged_prs and audit card_records == check_cards.cards(template)[mech]. Also assert audit['culturebotai-claw'].sha == manifest['source']['revision'] and that the audit's repo set equals the stats repos plus culturebotai-claw. Otherwise drop 'record counts' from the PR claim.
Found in the adversarial review of #120.
Audit check covers Mech shas only: CLAW pin, card_records and merged_prs in site_audit are unchecked despite the PR's 'agree on revisions and record counts'
Where:
tests/test_fleet_page.py:440The PR body says RefreshProvenanceTests 'requires the census, the stats and site_audit.json to agree on revisions and record counts'. Only the audit's
shais compared, and only for Mechs in mech_stats. Unchecked: the audit's record figure (card_records, which by design equals the template card figure), its merged_prs (which SKILL.md step 7 says to derive from mech_stats), and CLAW's audit sha against manifest.json source.revision. Extra stale repos in the audit are also unchecked.Failure scenario. The next refresh updates the shas in site_audit.json but carries over last run's card_records and merged_prs, or leaves CLAW at an old sha after refresh_manifest.py re-pins the manifest. The provenance record then misstates what was audited and the gate stays green.
Evidence. mutate.py in the isolated copy: M9 CLAW audit sha != manifest source.revision -> OK. M10 every audit merged_prs -50 and card_records -7, shas current -> OK. M15 audit gains a non-member repo -> OK. Today the values do agree: every audit merged_prs equals stats, card_records equals the template cards (CommunityMech 422 vs stats 426 is expected), and CLAW 2637172 equals manifest.json.
Verifier (confirmed, low). Every factual part of the finding reproduces. The only audit check is the per-Mech
sha, and it runs only for repos that appear in mech_stats.json. The audit's merged_prs, card_records, the CLAW sha (against manifest.json source.revision) and any extra or stale repo entries are all unchecked. site_audit.json is not read by any other script or CI job, so these fields have no gate at all. Mutating them leaves the whole 33-test suite green. A control mutation of a Mech sha does fail, which shows the harness would catch the covered case.Two caveats keep the severity low.
The suggested fix is sound, and the functions it needs exist: check_cards.cards(template) is at scripts/fleet/check_cards.py:61, and manifest['source']['revision'] is present. Alternatively, the PR text could drop "and record counts" or scope it to census vs stats.
Suggested fix. Extend test_the_audit_pins_the_revisions_the_stats_counted to assert audit merged_prs == stats merged_prs and audit card_records == check_cards.cards(template)[mech]. Also assert audit['culturebotai-claw'].sha == manifest['source']['revision'] and that the audit's repo set equals the stats repos plus culturebotai-claw. Otherwise drop 'record counts' from the PR claim.