Let the card check tell a site that grew from a wrong card

.claude/skills/review-open-issues/SKILL.md

@@ -216,8 +216,12 @@ pull request, so it can be red without blocking anything:
 python scripts/fleet/check_cards.py   # card headline figures vs each Mech's site
 ```
 
-A drifted card therefore shows up as a failed scheduled run, not a failed PR
-check. Look at the latest scheduled run before crediting the cards as current.
+A stale card therefore shows up as a failed scheduled run, not a failed PR
+check. A card behind a fast Mech only warns ("grew") for 14 days after the
+refresh's pins, and only while its site is at most half as large again as the
+card; past either limit it fails as STALE (#263). So a green run does not mean
+every card equals its site: read the run's log, not just its colour, before
+crediting the cards as current.
 
 An issue asserting a defect that one of these already blocks is P2 unless it
 shows the gate is porous — and they have been porous: a test can pass because

.claude/skills/update-xmech-page/SKILL.md

@@ -232,8 +232,14 @@ Rewrite `_fleet/data/site_audit.json` for the run: per repository the pinned
 test compares a value with itself, #125) and its commit date, the URL each card figure is read from, the figure, the
 sha256 of the fetched HTML and of any data file, merged PRs, and short notes on
 how the site figure relates to the repo count. Set `checked_at_utc`,
-`local_date`, `pinned_at_utc` and `scope`. Derive the mechanical fields rather
-than typing them: the figure through `check_cards.published()`, merged PRs from
+`local_date`, `pinned_at_utc` (ISO, with its offset) and `scope`. Record
+`figure_at_pin` for every source with a committed copy: read the copy at the pin
+with `check_cards.figure()`, the nightly's own parser with `REGIONS` applied,
+never from the template. The provenance tests require it to equal each card, so
+a mistyped card fails on the PR, and the nightly reports WRONG from it (#231).
+ProteinTraitsMech's data file is built in CI and has none. Derive the other
+mechanical fields rather than typing them: the live figure through
+`check_cards.read_source()`, merged PRs from
 `mech_stats.json`, SHAs and commit dates from the pins, and assert that the
 pins equal the stats' `source_revision` before writing. Hash the served page as
 committed at the pin too (`git show <sha>:pages/index.html`, or `docs/`), record
@@ -253,7 +259,7 @@ python3 scripts/fleet/assemble_page.py
 python3 -m unittest discover -s tests -v
 python3 scripts/fleet/assemble_page.py --check
 python3 scripts/fleet/refresh_manifest.py --claw-root "$SNAP/claw" --check
-python3 scripts/fleet/check_cards.py        # 0 drifted, except sites that moved past their pin (below)
+python3 scripts/fleet/check_cards.py        # exit 0; "grew" lines are sites that moved past their pin (below)
 ```
 
 Rerun `check_cards.py` immediately before opening the PR and again before any
@@ -264,8 +270,12 @@ When a site has moved past its pin, do not re-pin that one Mech: the census and
 overlaps are computed across Mechs, so a single re-pin is a partial rerun, and a
 fast Mech moves again before the rerun finishes. Keep the page a consistent
 snapshot at the pins, record the live figure as `site_figure_at_check` in that
-Mech's `site_audit.json` entry, and say in the PR which cards will show as
-drifted. Re-pin everything only if the drift is large enough to mislead.
+Mech's `site_audit.json` entry, and say in the PR which cards the check reports
+as grown. That stays a warning for `GRACE_DAYS` (14) after `pinned_at_utc` while
+the site is at most half as large again as the card; past either limit the
+check reports STALE and fails, and the page is due a full refresh. A card that
+differs from its audit's `figure_at_pin` fails as WRONG however far the site has
+moved (step 7).
 
 Emoji headings render with a leading hyphen in their id on GitHub Pages. Verify
 anchors against the deployed HTML, not a local kramdown.

.github/workflows/fleet-page.yml

@@ -33,5 +33,8 @@ jobs:
       # are hand-curated from ten sites that publish on their own cadence, so a
       # blocking check would make an unrelated docs fix unmergeable whenever a
       # Mech shipped records overnight. Nightly red is the right signal.
-      - if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
+      # !cancelled() runs it even when an earlier step failed: a mistyped card
+      # fails the provenance tests first, and the card report, with its WRONG
+      # or AUDIT line and every other card's verdict, must still print (#239).
+      - if: ${{ !cancelled() && (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') }}
         run: python scripts/fleet/check_cards.py

_fleet/README.md

@@ -37,13 +37,44 @@ against the page it cites and is the one script here that needs the network:
 python3 scripts/fleet/check_cards.py
 ```
 
-It exits 1 on a figure that differs from the site and only warns on a page it
-could not read, and `SOURCES` at its top pins where each Mech publishes its
-count. It runs on the workflow's nightly schedule, not on pull requests, so a
-Mech shipping records overnight does not block an unrelated change; a nightly
-red means the card figures in `mechs_template.md` and the `MECHS` block in
-`fleet_fragment.html` need refreshing together. A new card needs a `SOURCES`
-entry; a test enforces that.
+`SOURCES` at its top pins where each Mech publishes its count, and `REGIONS`
+restricts a source to the part that states it where the same words appear
+elsewhere (CultureMech's generated README block). The page is a snapshot at a
+refresh's pins, so a site ahead of its card only warns ("grew") for
+`GRACE_DAYS` (14) after the pins in `site_audit.json`, and only while the site is
+at most `MAX_LEAD` (50%) ahead. Past either limit it fails as STALE. It also
+fails when a card differs from `figure_at_pin`, the figure `site_audit.json`
+records its source stating at the pin (the card was never right), when a site is
+behind its card, when a source answers a 4xx other than a throttle or no longer
+states a figure the parser can read, when a card lacks exactly one headline
+figure or has no `SOURCES` entry, when the audit is missing or malformed, its
+pin time is missing, unreadable or in the future, or a source lacks a
+whole-number `figure_at_pin` (all but ProteinTraitsMech, whose file is built in
+CI), and
+when more than half the sources could not be fetched. One site's outage or
+throttle only warns (#148, #115, #176, #217-#220, #231, #232). It runs on the
+workflow's nightly schedule, not on pull requests, so a Mech shipping records
+overnight does not block an unrelated change. The run's closing line names the
+remedy for each failing verdict (#235):
+
+- STALE or SHRANK: the card figures in `mechs_template.md` and the `MECHS` block
+  in `fleet_fragment.html` need a full refresh, since the page is a snapshot.
+- WRONG: a card, or the audit's `figure_at_pin`, was mistyped. Correct the card
+  and every other occurrence of its figure, found by grepping the tree for it as
+  step 6 of the update skill does: the MECHS `records:` and `extra:` text in
+  `fleet_fragment.html`, cross-references, `card_records` in `site_audit.json`
+  and the pages that repeat it. Or correct `figure_at_pin` if that is what was
+  wrong. Then rerun `assemble_page.py`; no re-pin (#248, #258). The unit tests catch this on the
+  PR and in the nightly, which still runs the card check after a failed test
+  step so its report prints (#239, #240).
+- GONE or CHANGED: a `SOURCES` entry needs repointing.
+- MARKUP, UNCARDED or AUDIT: fix the card markup, `SOURCES` or `site_audit.json`.
+- UNCHECKED: most sites could not be reached; rerun before changing anything.
+
+A new card needs a `SOURCES` entry; a test enforces that. The cards
+are read by `scripts/fleet/card_markup.py`, the one parser the assembler, this
+check and the tests share, and the assembler refuses a card without exactly one
+headline figure (#114, #218).
 
 The `Fleet page` workflow checks pull requests, pushes and the live CLAW manifest
 daily. It detects changes to membership, capability declarations (including
@@ -180,7 +211,10 @@ its site lists 422 communities, while its record glob also takes four isolate
 records, so the census and `mech_stats.json` count 426. CellStructureMech and
 TraitMech published new records after the pins were taken; their cards keep the
 pinned figures, and `site_audit.json` records what the two sites showed when it
-was written. `check_cards.py` will report both as drifted until the next refresh.
+was written. `check_cards.py` reports both as grown, a warning, until 14 days
+after the pins or until a site is half as large again as its card, whichever
+comes first; CellStructureMech, adding about two records an hour, reaches the
+second within a week.
 NaturalProductMech's landing page and MediaIngredientMech's data file also
 changed after the pins without changing their figures; the audit records each
 live hash beside the hash of the committed copy at the pin.

_fleet/data/site_audit.json

@@ -10,6 +10,7 @@
    "commit_date": "2026-09-25T01:30:31Z",
    "readme_url": "https://github.com/CultureBotAI/AntibioticMech/blob/66d68c2a8d99230ab729267e6cd8ae1051b85225/README.md",
    "card_records": 2939,
+   "figure_at_pin": 2939,
    "merged_prs": 422,
    "site": "https://culturebotai.github.io/AntibioticMech/pages/index.html",
    "site_html_sha256": "426d439768e46f5e3c65b0241762359d06dffb3b217a3bba01e2552b65991ae9",
@@ -22,6 +23,7 @@
    "commit_date": "2026-09-25T01:58:24Z",
    "readme_url": "https://github.com/CultureBotAI/CellStructureMech/blob/42cc23b90b6c48c49efd94b7ee6efb808dded7e6/README.md",
    "card_records": 542,
+   "figure_at_pin": 542,
    "merged_prs": 611,
    "site": "https://culturebotai.github.io/CellStructureMech/pages/index.html",
    "site_html_sha256": "aa3a2b8d51d6cd1bbac33c472b0b40279538779c514f384437f818b3c665d679",
@@ -35,6 +37,7 @@
    "commit_date": "2026-09-24T14:00:15Z",
    "readme_url": "https://github.com/CultureBotAI/CommunityMech/blob/8505a56d644b02fe87f776be9d667db4cf3f5c5d/README.md",
    "card_records": 422,
+   "figure_at_pin": 422,
    "merged_prs": 578,
    "site": "https://culturebotai.github.io/CommunityMech/",
    "site_html_sha256": "5c7aba370a7e7bd50590eb16005c4e3bdd31b1f17d933b8106de7506d67d07dd",
@@ -54,6 +57,7 @@
    "commit_date": "2026-09-22T06:24:22Z",
    "readme_url": "https://github.com/CultureBotAI/CultureMech/blob/faaf033b8c5386aaf2cb6f28fc527678d00cf83b/README.md",
    "card_records": 6288,
+   "figure_at_pin": 6288,
    "merged_prs": 253,
    "site": "https://raw.githubusercontent.com/CultureBotAI/CultureMech/main/README.md",
    "site_html_sha256": "9bc06d232a67fa55cdef539fec2f52fa2644f7cf50e2820e223cc26106ee6f51",
@@ -66,6 +70,7 @@
    "commit_date": "2026-09-25T01:40:32Z",
    "readme_url": "https://github.com/CultureBotAI/HabitatMech/blob/b16e3099478a7ea55b82142836e9a118ae062cf6/README.md",
    "card_records": 3206,
+   "figure_at_pin": 3206,
    "merged_prs": 417,
    "site": "https://culturebotai.github.io/HabitatMech/pages/index.html",
    "site_html_sha256": "53564300e6d0aaf2a5b1144fceab088024586941f197113c57c42c17eefbc8e9",
@@ -78,6 +83,7 @@
    "commit_date": "2026-09-24T08:38:41Z",
    "readme_url": "https://github.com/CultureBotAI/MediaIngredientMech/blob/dfce1c9342ca7aec41b50d0f6db8adb950bb72fe/README.md",
    "card_records": 2953,
+   "figure_at_pin": 2953,
    "merged_prs": 379,
    "site": "https://culturebotai.github.io/MediaIngredientMech/",
    "site_html_sha256": "e98a1ac81f8faafad3aae291782b32e22e78e8f20cd822effc4cba4c22ce7b09",
@@ -92,6 +98,7 @@
    "commit_date": "2026-09-24T07:48:18Z",
    "readme_url": "https://github.com/CultureBotAI/NaturalProductMech/blob/36662aa8fcdcac9d3ddf4acba0f430e39bc2da2e/README.md",
    "card_records": 3115,
+   "figure_at_pin": 3115,
    "merged_prs": 84,
    "site": "https://culturebotai.github.io/NaturalProductMech/pages/index.html",
    "site_html_sha256": "28d335df6bfeb065940fad9e5ae783d6cd3ed7eacabd8c46c94ee4a9aecdbabd",
@@ -117,6 +124,7 @@
    "commit_date": "2026-09-21T08:37:02Z",
    "readme_url": "https://github.com/CultureBotAI/TaxonMech/blob/972fbd7b85a052c6b95bf1437f672168bd6d8970/README.md",
    "card_records": 625960,
+   "figure_at_pin": 625960,
    "merged_prs": 18,
    "site": "https://culturebotai.github.io/TaxonMech/pages/index.html",
    "site_html_sha256": "53aab681e40bafb9326b320858660a4273c7df8b9a19ad561d4730b5b32c6a89",
@@ -129,6 +137,7 @@
    "commit_date": "2026-09-22T15:30:43Z",
    "readme_url": "https://github.com/CultureBotAI/TraitMech/blob/a21c5aa34a876c43ea1e8053a58bc63d53be30a1/README.md",
    "card_records": 763,
+   "figure_at_pin": 763,
    "merged_prs": 656,
    "site": "https://culturebotai.github.io/TraitMech/pages/index.html",
    "site_html_sha256": "21fd928ae51ddaca7a9f1e5cafce81a37b055c97ab8a83442ba19021acc2daeb",

scripts/fleet/assemble_page.py

@@ -6,6 +6,7 @@
 from pathlib import Path
 import re
 
+from card_markup import card_figures, card_names, markup_problems
 from refresh_manifest import validate
 
 REPO = Path(__file__).resolve().parents[2]
@@ -50,11 +51,8 @@ def number_word(value: int) -> str:
     return WORDS[value] if 0 <= value < len(WORDS) else f"{value:,}"
 
 
-CARD_RECORDS = re.compile(r'<div class="num"><b>([\d,]+)</b>')
-
-
 def fleet_records(template):
-    """What the Mech cards add up to.
+    """What the Mech cards add up to, one figure per card.
 
     The tile used to carry its own typed figure and drifted away from the
     cards it was meant to total: it read 448,724 while the ten cards summed to
@@ -68,11 +66,21 @@ def fleet_records(template):
     The ten are not ten counts of the same thing: the cards call theirs taxon
     records, published recipes, natural product structures and so on. The tile
     says "curated entries" rather than "records" for that reason (#82).
+
+    Keyed by Mech and read card by card through card_markup, the parser
+    check_cards.py also uses, so the total and the nightly check cannot read the
+    markup differently (#114).
     """
-    counts = [int(n.replace(",", "")) for n in CARD_RECORDS.findall(template)]
-    if not counts:
+    # Exactly one figure per card, and none outside the cards: a second stat
+    # tile used to replace a card's headline in the total without failing (#218).
+    problems = markup_problems(template)
+    if problems:
+        raise ValueError("Every Mech card must carry a record count, exactly once: "
+                         + "; ".join(f"{mech}: {why}" for mech, why in problems))
+    figures = card_figures(template)
+    if not figures:
         raise ValueError("No Mech card record counts found")
-    return counts
+    return figures
 
 
 def assemble(template, fragment, data, snapshot, stats, census):
@@ -81,7 +89,7 @@ def assemble(template, fragment, data, snapshot, stats, census):
     badges = re.findall(r"<!--FLEET_BADGE:([^>]+)-->", template)
     if len(badges) != len(names) or set(badges) != names:
         raise ValueError("Mech cards must match canonical fleet membership exactly")
-    cards = re.findall(r'<article\b[^>]*\bdata-mech="([^"]+)"', template)
+    cards = card_names(template)
     if len(cards) != len(names) or set(cards) != names:
         raise ValueError("Actual Mech cards must match canonical fleet membership exactly")
     metadata = fragment.split("var MECHS = {", 1)[1].split("\n  };", 1)[0]
@@ -114,7 +122,7 @@ def assemble(template, fragment, data, snapshot, stats, census):
     if counted != names:
         raise ValueError("Mech stats must cover canonical fleet membership exactly")
     counts = fleet_records(template)
-    if len(counts) != len(names):
+    if set(counts) != names:
         raise ValueError("Every Mech card must carry a record count")
     # The census is a dated scan, so its vocabulary tally is labelled with its own
     # run date rather than as current, and its coverage is stated below.
@@ -127,7 +135,7 @@ def assemble(template, fragment, data, snapshot, stats, census):
     tokens = {
         "<!--FLEET_COUNT-->": str(len(names)),
         "<!--FLEET_COUNT_WORD-->": number_word(len(names)),
-        "<!--FLEET_RECORDS_TOTAL-->": f"{sum(counts):,}",
+        "<!--FLEET_RECORDS_TOTAL-->": f"{sum(counts.values()):,}",
         "<!--FLEET_VOCAB_COUNT-->": f"{len(vocabularies):,}",
         # "all ten Mechs" once the census reaches every member, which it has
         # since TaxonMech was added (#87); "nine of the ten Mechs" otherwise.

scripts/fleet/card_markup.py

@@ -0,0 +1,68 @@
+"""The one parser for the Mech cards in _fleet/mechs_template.md.
+
+assemble_page.py, check_cards.py and the tests used to read the card markup with
+three separate regexes, and check_cards.py paired each Mech with the *next*
+figure in the file, so a card missing its figure silently took its neighbour's
+(#114). Everything now reads the cards here, one <article> at a time.
+"""
+from __future__ import annotations
+
+import collections
+import re
+
+ARTICLE = re.compile(r'<article\b[^>]*\bdata-mech="([^"]+)"[^>]*>(.*?)</article>', re.S)
+# A card's headline tile, counted whatever it holds, so a malformed second tile
+# cannot slip past the one-per-card rule (#261).
+TILE = re.compile(r'<div class="num">')
+# The figure in a tile: plain digits, or digits grouped in threes by commas, so
+# "<b>,</b>" is unreadable rather than int("") (#261) and "32,06" is unreadable
+# rather than 3206 (#264).
+FIGURE = re.compile(r'<div class="num"><b>(\d{1,3}(?:,\d{3})+|\d+)</b>')
+
+
+def card_names(template: str) -> list[str]:
+    """Every card's Mech, in page order, duplicates kept."""
+    return [mech for mech, _ in ARTICLE.findall(template)]
+
+
+def markup_problems(template: str) -> list[tuple[str, str]]:
+    """(Mech, what is wrong) for every card that does not state exactly one figure.
+
+    A card with two figures used to be read as its first, and a figure outside
+    every card was ignored, so a second stat tile changed the fleet total without
+    failing anything (#218). "-" stands for a figure that belongs to no card.
+    """
+    problems = []
+    for mech, body in ARTICLE.findall(template):
+        tiles = len(TILE.findall(body))
+        if tiles == 0:
+            problems.append((mech, "card has no headline figure"))
+        elif tiles > 1:
+            problems.append((mech, f"card has {tiles} headline figures; it must have exactly one"))
+        elif not FIGURE.search(body):
+            problems.append((mech, "card's headline figure is unreadable"))
+    for mech, count in collections.Counter(card_names(template)).items():
+        if count > 1:
+            # The nightly would check only one of them (#262).
+            problems.append((mech, f"{count} cards for one Mech"))
+    outside = len(TILE.findall(ARTICLE.sub("", template)))
+    if outside:
+        problems.append(("-", f"{outside} headline figure(s) outside any card"))
+    return problems
+
+
+def card_figures(template: str) -> dict[str, int]:
+    """Each card's headline figure, keyed by Mech.
+
+    Read inside the card's own <article>, so a card without a figure is absent
+    rather than borrowing the next card's. A card with more than one headline
+    tile, an unreadable one, or a second card for the same Mech is absent too;
+    markup_problems() says which cards those are.
+    """
+    names = collections.Counter(card_names(template))
+    figures: dict[str, int] = {}
+    for mech, body in ARTICLE.findall(template):
+        hits = FIGURE.findall(body)
+        if names[mech] == 1 and len(TILE.findall(body)) == 1 and len(hits) == 1:
+            figures[mech] = int(hits[0].replace(",", ""))
+    return figures