From 75223f7dc04f08ffde4f09f96d4d304aa28fb6a9 Mon Sep 17 00:00:00 2001 From: "marcin p. joachimiak" <4625870+realmarcin@users.noreply.github.com> Date: Fri, 25 Sep 2026 13:15:54 -0700 Subject: [PATCH 1/7] Tell the update skill to remove its snapshot after the merge A refresh builds sparse shared clones of every Mech in the scratchpad, about 6 GB, and nothing removed them (#174). Step 11 removes the snapshot once the PR has merged, guarded against an unset variable, and step 2 gives its real size. Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/skills/update-xmech-page/SKILL.md | 26 +++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/.claude/skills/update-xmech-page/SKILL.md b/.claude/skills/update-xmech-page/SKILL.md index b904c8a..265026a 100644 --- a/.claude/skills/update-xmech-page/SKILL.md +++ b/.claude/skills/update-xmech-page/SKILL.md @@ -102,8 +102,10 @@ git -C "$SNAP/mechs/$m" checkout -q --detach "$sha" The directories each Mech needs are its `roots.RECORD_GLOBS` directories (plus `mech_stats.EXTRA_GLOBS` for Mechs outside the census), `src` for the schema that `mech_stats.py` reads, and HabitatMech's `pages/habitats`, which -`build_subsets.py` matches record links against. About 3 GB and a million -files; TaxonMech and ProteinTraitsMech are most of it. The detached checkout +`build_subsets.py` matches record links against. About 6 GB and a million +files (5.7 GB at the 2026-09-24 refresh); TaxonMech (3.2 GB) and +ProteinTraitsMech (2.3 GB) are most of it. Keep it until the PR merges, because +reruns after review fixes go against the same pins; step 11 removes it. The detached checkout works although the pin exists only in the source's remote-tracking refs, because a shared clone borrows the source's whole object store. @@ -271,6 +273,26 @@ explicit go-ahead in the current conversation.** After a merge, delete the branc locally and remotely, and close issues the PR resolved that GitHub did not auto-close. GitHub honours only the first number after a closing keyword. +### 11. Remove the snapshot + +After the merge and the branch deletion, and not before: until then a review fix +means a rerun at the same pins, which needs the snapshot. Remove `$SNAP` whole, +the Mech clones, the CLAW clone, `revisions.json`, the scan logs and any +before/after copies of the derived data: + +```bash +du -sh "${SNAP:?}" # about 6 GB; say it in the report +rm -rf "${SNAP:?}" +``` + +`${SNAP:?}` stops the command if `SNAP` is unset rather than expanding to a bare +`rm -rf`. Only `$SNAP` goes. Never `$SRC`: those are the shared checkouts other +sessions use. Removing the shared clones is safe for `$SRC`, because a +`--shared` clone borrows the source's object store and the source knows nothing +about it; the reverse, pruning or deleting `$SRC` while a clone exists, is what +would break a clone. A second refresh in the same session builds a fresh +snapshot at its own pins, so nothing is lost by removing the first (#174). + ## Related - `_fleet/README.md` — the pipeline and what each script reads and writes. From 84f69b466c69f392c17d95f644d65a38d909edfb Mon Sep 17 00:00:00 2001 From: "marcin p. joachimiak" <4625870+realmarcin@users.noreply.github.com> Date: Fri, 25 Sep 2026 13:31:12 -0700 Subject: [PATCH 2/7] Give the snapshot its own directory, and remove only that The skill named the session scratchpad itself as $SNAP, so step 11's rm -rf could take the scratchpad and every worktree in it (#227). Step 1 now creates a dedicated directory and refuses to reuse one, the PR records its absolute path because the merge may come in another session (#228), and step 11 removes the path from the PR only if it holds revisions.json, mechs/ and claw/. Step 4's timings follow _fleet/README.md. Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/skills/update-xmech-page/SKILL.md | 43 ++++++++++++++++------- 1 file changed, 30 insertions(+), 13 deletions(-) diff --git a/.claude/skills/update-xmech-page/SKILL.md b/.claude/skills/update-xmech-page/SKILL.md index 265026a..1db9bd6 100644 --- a/.claude/skills/update-xmech-page/SKILL.md +++ b/.claude/skills/update-xmech-page/SKILL.md @@ -54,8 +54,18 @@ moves for two reasons at once. ## Procedure -Work from the site root. Put the snapshot and logs in the session scratchpad, -here `$SNAP`. +Work from the site root. The snapshot and logs go in a directory of their own +inside the session scratchpad, never the scratchpad itself, which also holds +other work (worktrees of open PRs, among other things). Create it fresh: + +```bash +SNAP=/xmech-refresh-$(date +%Y%m%d) +mkdir "$SNAP" || exit 1 # refuse to reuse a directory that already exists +``` + +Shell variables do not survive between tool calls, so write the absolute path +down: it goes in the PR body and the report (step 10), and step 11 needs it +after the merge, which may come in another session (#227, #228). ### 1. Branch, locate the checkouts, pin CLAW and refresh the manifest @@ -129,8 +139,8 @@ Run each stage the same way, in the same environment and from the same launcher, and verify its output before starting the next. The first stage is the canary: ```bash -MECHS_ROOT=$SNAP/mechs python3 scripts/fleet/prefix_census.py # ~2-4 min -MECHS_ROOT=$SNAP/mechs python3 scripts/fleet/build_subsets.py # ~2-4 min +MECHS_ROOT=$SNAP/mechs python3 scripts/fleet/prefix_census.py # ~8 min +MECHS_ROOT=$SNAP/mechs python3 scripts/fleet/build_subsets.py # longer than the census python3 scripts/fleet/build_data.py MECHS_ROOT=$SNAP/mechs python3 scripts/fleet/mech_stats.py # needs gh ``` @@ -265,8 +275,9 @@ against its evidence. An unexplained change is a finding, not noise. ### 10. PR, review, issues Open the PR with the pins, what changed and why, the canary result, what was -left out and why (methodology issues stay open), and the evidence table for -hand-curated edits. Then review it adversarially as a separate read-only pass, +left out and why (methodology issues stay open), the evidence table for +hand-curated edits, and the snapshot's absolute path and size ("Snapshot: +``, 5.7 GB, remove after merge"). Then review it adversarially as a separate read-only pass, file every finding as an issue, fix the ones that belong in this PR, and leave the rest filed with a reason. Report and stop: **do not merge without the user's explicit go-ahead in the current conversation.** After a merge, delete the branch @@ -276,17 +287,23 @@ auto-close. GitHub honours only the first number after a closing keyword. ### 11. Remove the snapshot After the merge and the branch deletion, and not before: until then a review fix -means a rerun at the same pins, which needs the snapshot. Remove `$SNAP` whole, -the Mech clones, the CLAW clone, `revisions.json`, the scan logs and any -before/after copies of the derived data: +means a rerun at the same pins, which needs the snapshot. Take the path from the +PR body, not from a shell variable, and remove that directory whole: the Mech +clones, the CLAW clone, `revisions.json`, the scan logs and any before/after +copies of the derived data. Remove it only if it is recognizably a snapshot: ```bash -du -sh "${SNAP:?}" # about 6 GB; say it in the report -rm -rf "${SNAP:?}" +SNAP= +test -f "${SNAP:?}/revisions.json" && test -d "$SNAP/mechs" && test -d "$SNAP/claw" \ + || { echo "not a refresh snapshot: $SNAP"; exit 1; } +du -sh "$SNAP" # about 6 GB; say it in the report +rm -rf "$SNAP" ``` -`${SNAP:?}` stops the command if `SNAP` is unset rather than expanding to a bare -`rm -rf`. Only `$SNAP` goes. Never `$SRC`: those are the shared checkouts other +`${SNAP:?}` stops the command if `SNAP` is unset, and the three tests stop it if +the path is the scratchpad or anything else that is not a snapshot. If the +directory is already gone, say so rather than searching for another. Only +`$SNAP` goes. Never `$SRC`: those are the shared checkouts other sessions use. Removing the shared clones is safe for `$SRC`, because a `--shared` clone borrows the source's object store and the source knows nothing about it; the reverse, pruning or deleting `$SRC` while a clone exists, is what From 7d168a72ae878722d4e6903e55f39db0068a40b4 Mon Sep 17 00:00:00 2001 From: "marcin p. joachimiak" <4625870+realmarcin@users.noreply.github.com> Date: Fri, 25 Sep 2026 13:42:55 -0700 Subject: [PATCH 3/7] Give the pipeline's measured timings, and say how to rebuild a removed snapshot build_subsets.py takes about six minutes over the ten-Mech snapshot, less than the census's eight, in every logged run at the #120 pins (#230). Step 11 also says a later PR can rebuild the snapshot from the pins in site_audit.json. Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/skills/update-xmech-page/SKILL.md | 6 ++++-- _fleet/README.md | 5 +++-- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/.claude/skills/update-xmech-page/SKILL.md b/.claude/skills/update-xmech-page/SKILL.md index 1db9bd6..4737f55 100644 --- a/.claude/skills/update-xmech-page/SKILL.md +++ b/.claude/skills/update-xmech-page/SKILL.md @@ -140,7 +140,7 @@ and verify its output before starting the next. The first stage is the canary: ```bash MECHS_ROOT=$SNAP/mechs python3 scripts/fleet/prefix_census.py # ~8 min -MECHS_ROOT=$SNAP/mechs python3 scripts/fleet/build_subsets.py # longer than the census +MECHS_ROOT=$SNAP/mechs python3 scripts/fleet/build_subsets.py # ~6 min python3 scripts/fleet/build_data.py MECHS_ROOT=$SNAP/mechs python3 scripts/fleet/mech_stats.py # needs gh ``` @@ -308,7 +308,9 @@ sessions use. Removing the shared clones is safe for `$SRC`, because a `--shared` clone borrows the source's object store and the source knows nothing about it; the reverse, pruning or deleting `$SRC` while a clone exists, is what would break a clone. A second refresh in the same session builds a fresh -snapshot at its own pins, so nothing is lost by removing the first (#174). +snapshot at its own pins, and a later PR that must rerun at these pins can +rebuild this one with step 2 from the pins in `_fleet/data/site_audit.json`, so +nothing is lost by removing it (#174). ## Related diff --git a/_fleet/README.md b/_fleet/README.md index ad20bfe..f5f7622 100644 --- a/_fleet/README.md +++ b/_fleet/README.md @@ -123,8 +123,9 @@ python3 scripts/fleet/mech_stats.py # _fleet/data/mech_stats.json (needs g python3 scripts/fleet/assemble_page.py # mechs.md ``` -The census takes about eight minutes and `build_subsets.py` longer, dominated by -TaxonMech's ~626k and ProteinTraitsMech's ~430k records. +The census takes about eight minutes and `build_subsets.py` about six over the +ten-Mech snapshot, both dominated by TaxonMech's ~626k and ProteinTraitsMech's +~430k records (#230). Jekyll ignores `_fleet/` (leading underscore) and `scripts/` is excluded in `_config.yml`. Record links resolve to each Mech's published page where one exists (TraitMech, From ffe1d3210bca4f291a0628e0b9122c9586165c15 Mon Sep 17 00:00:00 2001 From: "marcin p. joachimiak" <4625870+realmarcin@users.noreply.github.com> Date: Fri, 25 Sep 2026 14:04:07 -0700 Subject: [PATCH 4/7] Stop step 11 on anything but snapshot data, and spell out a rebuild at the same pins Step 11 said step 2 could rebuild a removed snapshot, but step 2 makes only the Mech clones; a rerun at the same pins also needs CLAW at the audited sha and a revisions.json, and the skill now says how (#236). The removal also stops if the directory holds anything but the clones, revisions.json and logs, and the preamble says to keep working files outside it (#237). Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/skills/update-xmech-page/SKILL.md | 26 +++++++++++++++++------ 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/.claude/skills/update-xmech-page/SKILL.md b/.claude/skills/update-xmech-page/SKILL.md index 4737f55..8676415 100644 --- a/.claude/skills/update-xmech-page/SKILL.md +++ b/.claude/skills/update-xmech-page/SKILL.md @@ -65,7 +65,10 @@ mkdir "$SNAP" || exit 1 # refuse to reuse a directory that already exists Shell variables do not survive between tool calls, so write the absolute path down: it goes in the PR body and the report (step 10), and step 11 needs it -after the merge, which may come in another session (#227, #228). +after the merge, which may come in another session (#227, #228). Keep only the +snapshot and its logs in `$SNAP`; working files such as drafts, review records +and helper scripts go elsewhere in the scratchpad, because step 11 removes the +directory whole (#237). ### 1. Branch, locate the checkouts, pin CLAW and refresh the manifest @@ -296,21 +299,30 @@ copies of the derived data. Remove it only if it is recognizably a snapshot: SNAP= test -f "${SNAP:?}/revisions.json" && test -d "$SNAP/mechs" && test -d "$SNAP/claw" \ || { echo "not a refresh snapshot: $SNAP"; exit 1; } +extra=$(ls -A "$SNAP" | grep -v -x -e revisions.json -e mechs -e claw -e '.*\.log') +[ -z "$extra" ] || { printf 'not snapshot data; move it out or decide first:\n%s\n' "$extra"; exit 1; } du -sh "$SNAP" # about 6 GB; say it in the report rm -rf "$SNAP" ``` `${SNAP:?}` stops the command if `SNAP` is unset, and the three tests stop it if -the path is the scratchpad or anything else that is not a snapshot. If the -directory is already gone, say so rather than searching for another. Only +the path is the scratchpad or anything else that is not a snapshot. The listing +stops it if the directory holds anything but the clones, `revisions.json` and +logs: a script or record left there cannot be rebuilt from the pins (#237). If +the directory is already gone, say so rather than searching for another. Only `$SNAP` goes. Never `$SRC`: those are the shared checkouts other sessions use. Removing the shared clones is safe for `$SRC`, because a `--shared` clone borrows the source's object store and the source knows nothing about it; the reverse, pruning or deleting `$SRC` while a clone exists, is what -would break a clone. A second refresh in the same session builds a fresh -snapshot at its own pins, and a later PR that must rerun at these pins can -rebuild this one with step 2 from the pins in `_fleet/data/site_audit.json`, so -nothing is lost by removing it (#174). +would break a clone. A second refresh builds a fresh snapshot at its own pins +(#174). A later PR that must rerun at these same pins builds one too, from the +pins committed in `_fleet/data/site_audit.json`: create `$SNAP` as above, clone +CLAW with step 1's three `git` commands but at the audit's `culturebotai-claw` +sha rather than `main`, and run `refresh_manifest.py` only with `--check`, so the +committed manifest stays at its pin. Write `$SNAP/revisions.json` from the +audit's shas, commit dates and `pinned_at_utc`, keyed by Mech name, not repository name +(ProteinTraitsMech's repository is `proteintraitsmech`). Then run step 2 for +each Mech (#236). ## Related From 1226a31ca09735047e8f0e042edda92c5eaad3f7 Mon Sep 17 00:00:00 2001 From: "marcin p. joachimiak" <4625870+realmarcin@users.noreply.github.com> Date: Fri, 25 Sep 2026 14:15:36 -0700 Subject: [PATCH 5/7] Keep before/after copies out of the snapshot, and give revisions.json's shape Step 11 listed before/after copies of the derived data among what it removes while its guard stops on them; they are working files and belong outside $SNAP (#245). The same-pins rebuild recipe now shows the revisions.json shape step 7 reads and says to set SRC before step 2 (#246). Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/skills/update-xmech-page/SKILL.md | 25 +++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/.claude/skills/update-xmech-page/SKILL.md b/.claude/skills/update-xmech-page/SKILL.md index 8676415..e3c9d3a 100644 --- a/.claude/skills/update-xmech-page/SKILL.md +++ b/.claude/skills/update-xmech-page/SKILL.md @@ -66,9 +66,10 @@ mkdir "$SNAP" || exit 1 # refuse to reuse a directory that already exists Shell variables do not survive between tool calls, so write the absolute path down: it goes in the PR body and the report (step 10), and step 11 needs it after the merge, which may come in another session (#227, #228). Keep only the -snapshot and its logs in `$SNAP`; working files such as drafts, review records -and helper scripts go elsewhere in the scratchpad, because step 11 removes the -directory whole (#237). +snapshot and its logs in `$SNAP`; working files, including before/after copies +of the derived data for step 9, drafts, review records and helper scripts, go +elsewhere in the scratchpad, because step 11 removes the directory whole and +stops if it finds anything else there (#237, #245). ### 1. Branch, locate the checkouts, pin CLAW and refresh the manifest @@ -292,8 +293,8 @@ auto-close. GitHub honours only the first number after a closing keyword. After the merge and the branch deletion, and not before: until then a review fix means a rerun at the same pins, which needs the snapshot. Take the path from the PR body, not from a shell variable, and remove that directory whole: the Mech -clones, the CLAW clone, `revisions.json`, the scan logs and any before/after -copies of the derived data. Remove it only if it is recognizably a snapshot: +clones, the CLAW clone, `revisions.json` and the scan logs. Remove it only if +it is recognizably a snapshot and holds nothing else: ```bash SNAP= @@ -320,9 +321,17 @@ pins committed in `_fleet/data/site_audit.json`: create `$SNAP` as above, clone CLAW with step 1's three `git` commands but at the audit's `culturebotai-claw` sha rather than `main`, and run `refresh_manifest.py` only with `--check`, so the committed manifest stays at its pin. Write `$SNAP/revisions.json` from the -audit's shas, commit dates and `pinned_at_utc`, keyed by Mech name, not repository name -(ProteinTraitsMech's repository is `proteintraitsmech`). Then run step 2 for -each Mech (#236). +audit, in the shape step 1 writes and step 7 reads: + +```json +{"pinned_at_utc": "", + "mechs": {"": {"repo": "", "sha": "", "commit_date": ""}}, + "claw": ""} +``` + +keyed by Mech name, not repository name (ProteinTraitsMech's repository is +`proteintraitsmech`). Then set `SRC` as in step 1 and run step 2 for each Mech +(#236, #246). ## Related From ccda198734ff0ebb2a91f5e96bc7381270a60315 Mon Sep 17 00:00:00 2001 From: "marcin p. joachimiak" <4625870+realmarcin@users.noreply.github.com> Date: Fri, 25 Sep 2026 14:27:20 -0700 Subject: [PATCH 6/7] Name snapshot logs .log, the only logs step 11 removes The preamble allowed any log in $SNAP while step 11's guard accepts only *.log; the preamble and step 4 now say to use that suffix (#251). Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/skills/update-xmech-page/SKILL.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.claude/skills/update-xmech-page/SKILL.md b/.claude/skills/update-xmech-page/SKILL.md index e3c9d3a..7692fc9 100644 --- a/.claude/skills/update-xmech-page/SKILL.md +++ b/.claude/skills/update-xmech-page/SKILL.md @@ -66,7 +66,7 @@ mkdir "$SNAP" || exit 1 # refuse to reuse a directory that already exists Shell variables do not survive between tool calls, so write the absolute path down: it goes in the PR body and the report (step 10), and step 11 needs it after the merge, which may come in another session (#227, #228). Keep only the -snapshot and its logs in `$SNAP`; working files, including before/after copies +snapshot and its logs, each named `.log`, in `$SNAP`; working files, including before/after copies of the derived data for step 9, drafts, review records and helper scripts, go elsewhere in the scratchpad, because step 11 removes the directory whole and stops if it finds anything else there (#237, #245). @@ -160,7 +160,8 @@ byte-identical; `SubsetDeterminismTests` checks that on a fixture. Long scripts piped to `tail` print nothing until they exit. Check the process, not the empty log. Exit codes through pipes are the last command's, so use -`${PIPESTATUS[0]}` or write to a log file. +`${PIPESTATUS[0]}` or write to a log file: `$SNAP/.log`, since step 11 +removes only logs with that suffix (#251). ### 5. Re-check the hand-curated layer From 5b608446432d26cf187753af77c39c60b4f69bf7 Mon Sep 17 00:00:00 2001 From: "marcin p. joachimiak" <4625870+realmarcin@users.noreply.github.com> Date: Fri, 25 Sep 2026 14:32:55 -0700 Subject: [PATCH 7/7] Compare step 11's entries literally, and let a clean snapshot pass under set -e The listing matched names as regular expressions, so revisionsXjson passed as revisions.json, and it took a directory named *.log for a log (#252). It also exited 1 when nothing was extra, which stops a set -e shell before the removal (#253). A case loop compares literally, requires a *.log to be a regular file and exits 0 on a clean snapshot; tested under bash 3.2, zsh and sh with set -e. Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/skills/update-xmech-page/SKILL.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/.claude/skills/update-xmech-page/SKILL.md b/.claude/skills/update-xmech-page/SKILL.md index 7692fc9..1f35615 100644 --- a/.claude/skills/update-xmech-page/SKILL.md +++ b/.claude/skills/update-xmech-page/SKILL.md @@ -301,7 +301,13 @@ it is recognizably a snapshot and holds nothing else: SNAP= test -f "${SNAP:?}/revisions.json" && test -d "$SNAP/mechs" && test -d "$SNAP/claw" \ || { echo "not a refresh snapshot: $SNAP"; exit 1; } -extra=$(ls -A "$SNAP" | grep -v -x -e revisions.json -e mechs -e claw -e '.*\.log') +extra=$(ls -A "$SNAP" | while IFS= read -r name; do + case "$name" in + (revisions.json|mechs|claw) ;; + (*.log) [ -f "$SNAP/$name" ] || printf '%s\n' "$name" ;; + (*) printf '%s\n' "$name" ;; + esac +done) [ -z "$extra" ] || { printf 'not snapshot data; move it out or decide first:\n%s\n' "$extra"; exit 1; } du -sh "$SNAP" # about 6 GB; say it in the report rm -rf "$SNAP" @@ -310,7 +316,11 @@ rm -rf "$SNAP" `${SNAP:?}` stops the command if `SNAP` is unset, and the three tests stop it if the path is the scratchpad or anything else that is not a snapshot. The listing stops it if the directory holds anything but the clones, `revisions.json` and -logs: a script or record left there cannot be rebuilt from the pins (#237). If +logs: a script or record left there cannot be rebuilt from the pins (#237). It +compares names literally and accepts a `*.log` only if it is a regular file, and +finding nothing extra exits 0, so the snippet also runs under `set -e`; the +`(pattern)` form keeps macOS's bash 3.2 from misreading a `case` inside `$( )` +(#252, #253). If the directory is already gone, say so rather than searching for another. Only `$SNAP` goes. Never `$SRC`: those are the shared checkouts other sessions use. Removing the shared clones is safe for `$SRC`, because a