diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 98b1774..e04c550 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -120,12 +120,38 @@ Every skill, agent and hook is held to these; a change that breaks one says why. header row, so rename or renumber any of them only with those citations. - The Python environment is uv's (`pyproject.toml`, `.python-version`; the devcontainer and CI run `uv sync`). The dev tools' pins live in `pyproject.toml` only; `uv.lock` is not committed. -- Before a pull request: `uv run python -m unittest discover -s tests`, the five - `tests/*.sh`, and `uv run python scripts/check_static.py`, so the tests use the project's - environment outside the devcontainer too; CI also runs codespell, `claude plugin validate .`, and the unit tests and `check_static.py` - on Python 3.9, the README's floor. +- Before a pull request: `bash tests/checks.sh`, scoped as the next bullet says. A change to what a skill, agent or hook does fills the description's `Origin:`, `Sandbox:` and `Canary:` lines, as the README's "Changing the tooling" says; a `skill-feedback` issue is where a complaint about a skill starts. +- Checking scales with the depth of the change. CI runs every check on every pull request, so + locally `bash tests/checks.sh` runs the tests that name a changed file or a script loading it, + and the static checks, as its header sets out; `--all` reproduces CI when it is red. Run the + checks serially: in parallel they produce spurious errors. + + | Change | Before review | Sandbox | Adversarial review | + | --- | --- | --- | --- | + | Docs, ROADMAP, CI, devcontainer, the checks runner | `checks.sh` (static only) | None | None; the PR reviewer | + | A script, hook or test that changes no skill's output | `checks.sh` | None | One round | + | A skill's or agent's behavior or wording | `checks.sh` | `compare` on the scenarios that state the change, one Haiku batch; `--check --repeat 2` before the merge | Rounds until clean | + | A gate, or anything that posts or writes outside the repository | `checks.sh --all` | As above, plus the gate's scenarios; the week's canary | Rounds until clean | + + The README's "Changing the tooling" owns the `Origin:`, `Sandbox:` and `Canary:` lines a pull + request carries; this table only says what to run. + + While iterating, `bash tests/checks.sh ...` runs one module; the default runs before + review and after its fixes, not after every edit. +- Usage per turn grows with the context the turn carries, so long sessions and agents are the + cost, not the checks or the sandbox: + - One agent per phase, started fresh: implement, gather sandbox evidence, fix review findings. + Each gets the plan, the review file or the scenario list and reports through files. A new + ticket starts a new main session. + - An agent doing mechanical work, such as running a sandbox batch and tabulating its summaries + or rerunning one failing test against a known fix, is spawned with `model: sonnet`. Planning, + implementing and reviewing keep the session's model. + - A command that takes minutes runs in the background and is waited on through its completion + notice, never a `sleep`, `pgrep` or repeated `tail` loop. + - Reading is batched: several `sed -n` ranges or greps in one call, and a wide search goes to + an Explore agent. - `ROADMAP.md` holds deferred and planned work. An item leaves it when it lands, and a change that defers or lands one updates it in the same change. diff --git a/README.md b/README.md index 29122e8..f6adbae 100644 --- a/README.md +++ b/README.md @@ -910,6 +910,14 @@ CI (`.github/workflows/checks.yml`) runs, on every pull request: - codespell, with `en-GB_to_en-US`, for American spelling. - `claude plugin validate .`, on the plugin and marketplace manifests. +Locally, `bash tests/checks.sh` runs the tests of what the branch changed against +`origin/main`, with `check_static.py` and codespell, one step after another, printing a line per +step and the output only of a step that fails; its header says how a changed file maps to tests. +`bash tests/checks.sh next_phase where` runs only `tests/test_next_phase.py` and +`tests/test_where.sh`, and `--all` runs every step CI runs except its gitleaks scan of the +history, with the installed `claude` in place of CI's pinned one. How much checking each kind of +change gets is in `.claude/CLAUDE.md`. + To try a skill change before its pull request, run it in [`sandbox/`](sandbox/README.md): the skill's scenario on a worktree of your branch beside the current plugin, repeated, then compared, and `--check --repeat 2` on it before the merge. Nothing there runs in CI. diff --git a/tests/checks.sh b/tests/checks.sh new file mode 100755 index 0000000..d720263 --- /dev/null +++ b/tests/checks.sh @@ -0,0 +1,134 @@ +#!/usr/bin/env bash +# Runs the pull request checks one after another and prints one line per step; a failed step +# also prints the last 40 lines of its output, and the full logs stay in the directory the last +# line names. Serial on purpose: run together, test_profile, test_window_merges and +# test_surface_diff fail on each other's temporary state. +# +# bash tests/checks.sh the tests of what the branch changes against origin/main, +# committed or not, then check_static.py and codespell: +# - a changed tests/test_* file; +# - for a script, hook or sandbox .py or .sh, every test file +# naming it, and every test file naming a script, hook or +# sandbox file that names it; +# - for any other file under scripts/, hooks/, sandbox/ or +# standards/templates/, every test file naming it, or the +# whole suite when none does; +# - the whole suite for tests/fixtures/, pyproject.toml or +# .python-version. +# Skills, agents and docs get the static checks only. +# bash tests/checks.sh NAME... only tests/test_NAME.py or tests/test_NAME.sh for each NAME, +# such as `next_phase where` +# bash tests/checks.sh --all CI's steps but its gitleaks scan: the mcp-atlassian cache, the +# unit tests, every tests/test_*.sh, check_static.py, both +# again on Python 3.9, codespell and `claude plugin validate .`, +# the last with the installed claude, not CI's pinned version +# +# Exits 0 when every step passed, 1 when one failed, and 2 for a NAME with no test file, a +# missing tool, or no origin/main to compare with. +cd "$(dirname "$0")/.." || exit 2 +# The tests make hundreds of throwaway commits. Without the person's git configuration they run +# as on CI, which has none, and unsigned: a signing key there makes each commit about twice as slow. +export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 +logs=$(mktemp -d "${TMPDIR:-/tmp}/checks.XXXXXX") || exit 2 +home="" +failed=0 n=0 +trap '[ "$failed" = 1 ] || rm -rf "$logs"; [ -z "$home" ] || rm -rf "$home"' EXIT + +step() { + local name=$1; shift + n=$((n + 1)) + local log="$logs/$n-${name//[^A-Za-z0-9_.-]/_}.log" start=$SECONDS + if "$@" >"$log" 2>&1; then + printf 'ok %-28s %3ss\n' "$name" $((SECONDS - start)) + else + printf 'FAIL %-28s %3ss\n' "$name" $((SECONDS - start)) + tail -n 40 "$log" | sed 's/^/ /' + failed=1 + fi +} + +need() { command -v "$1" >/dev/null || { echo "checks.sh: $1 is not on PATH" >&2; exit 2; }; } + +run_test() { + case $1 in + *.py) step "$(basename "$1")" uv run python -m unittest discover -s tests -p "$(basename "$1")" ;; + *.sh) step "$(basename "$1")" bash "$1" ;; + esac +} + +naming() { grep -l -F "$1" tests/test_*.py tests/test_*.sh 2>/dev/null; } + +static() { + step "check_static" uv run python scripts/check_static.py + step "codespell" uv run codespell --builtin clear,rare,en-GB_to_en-US --skip "./.git,./tests/fixtures" . +} + +whole_suite() { + step "unit tests" uv run python -m unittest discover -s tests + for t in tests/test_*.sh; do + run_test "$t" + done +} + +need uv +if [ "$1" = --all ]; then + need uvx + need claude + step "mcp-atlassian cache" uvx --from mcp-atlassian@0.23.1 python -c pass + whole_suite + static + step "unit tests, python 3.9" uv run --isolated --python 3.9 python -m unittest discover -s tests + step "check_static, python 3.9" uv run --isolated --python 3.9 python scripts/check_static.py + home=$(mktemp -d) || exit 2 + step "plugin validate" env HOME="$home" claude plugin validate . +elif [ $# -gt 0 ]; then + tests=() + for name in "$@"; do + if [ -f "tests/test_$name.py" ]; then + tests+=("tests/test_$name.py") + elif [ -f "tests/test_$name.sh" ]; then + tests+=("tests/test_$name.sh") + else + echo "checks.sh: no tests/test_$name.py or tests/test_$name.sh" >&2 + exit 2 + fi + done + for t in "${tests[@]}"; do + run_test "$t" + done +else + base=$(git merge-base origin/main HEAD 2>/dev/null) \ + || { echo "checks.sh: no origin/main to compare with; name the tests or pass --all" >&2; exit 2; } + selected=$logs/selected + whole=0 + : >"$selected" + while IFS= read -r f; do + [ -n "$f" ] || continue + name=$(basename "$f") + case $f in + tests/fixtures/* | pyproject.toml | .python-version) whole=1 ;; + tests/test_*.py | tests/test_*.sh) [ -f "$f" ] && echo "$f" >>"$selected" ;; + scripts/*.py | scripts/*.sh | hooks/*.sh | sandbox/*.py) + naming "$name" >>"$selected" + for loader in $(grep -l -F "$name" scripts/*.py scripts/*.sh hooks/*.sh sandbox/*.py 2>/dev/null); do + [ "$loader" = "$f" ] || naming "$(basename "$loader")" >>"$selected" + done ;; + scripts/* | hooks/* | sandbox/* | standards/templates/*) + naming "$name" >>"$selected" || whole=1 ;; + esac + done < <({ git diff --name-only --no-renames "$base"; git ls-files --others --exclude-standard; } | sort -u) + if [ "$whole" = 1 ]; then + whole_suite + else + while IFS= read -r t; do + run_test "$t" + done < <(sort -u "$selected") + fi + static +fi + +if [ "$failed" = 1 ]; then + echo "full logs: $logs" + exit 1 +fi +echo "all $n passed"