Finish the code review article plan: noise ledger, second reads, replacement rounds, finding evals

.github/workflows/evals.yml

@@ -0,0 +1,39 @@
+name: evals
+
+on:
+  pull_request:
+    paths:
+      - "skills/underwrite/SKILL.md"
+      - "evals/**"
+
+permissions:
+  contents: read
+
+jobs:
+  evals:
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+    env:
+      # Each run spends model budget, so it waits for the owner to add the key; fork PRs never see it.
+      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+      GH_TOKEN: ${{ github.token }}
+      DISABLE_AUTOUPDATER: "1"
+    steps:
+      - name: Without a model key the walk cannot run
+        if: env.ANTHROPIC_API_KEY == ''
+        run: echo "::notice::ANTHROPIC_API_KEY is not set, so no eval walk ran"
+      - if: env.ANTHROPIC_API_KEY != ''
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      - if: env.ANTHROPIC_API_KEY != ''
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.13"
+      - if: env.ANTHROPIC_API_KEY != ''
+        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: "22"
+          package-manager-cache: false
+      - if: env.ANTHROPIC_API_KEY != ''
+        run: npm install --global @anthropic-ai/claude-code@2.1.251 --no-audit --no-fund
+      - if: env.ANTHROPIC_API_KEY != ''
+        run: python3 evals/run.py --bare --max-budget-usd 3 --baseline evals/baseline.json

README.md

@@ -68,6 +68,14 @@ PRIOR  #2 pinned break-check to 0.6.0 citing this exact failure mode
 FIX    npx --yes @arethetypeswrong/cli@0.18.5
 ```
 
+Before a flag is written, a fresh-context agent reads it against the frozen diff and blobs
+alone and answers three questions: did this PR introduce it, is it real, and would a senior
+engineer stop at it. A flag the PR did not introduce, or that is not real, is demoted to
+`UNVERIFIED` with the second reading beside it, so a wrong flag costs a glance rather than a
+decision. A doubt about whether it is worth stopping for stays beside a confirmed flag
+instead, because that call is yours. A flag written where no second reader was available says
+"Not read twice" rather than passing for a checked one.
+
 A flag in a PR session has to land on a line the diff touches: the store refuses one on a
 line the PR never put in front of you, naming the nearest touched line. That holds a flag
 to the change without proving the change caused what it flags.
@@ -295,6 +303,16 @@ diff, trusted context, and object bundle are checked by byte count and SHA-256.
 Controller cleanliness is rechecked immediately before capture freezes the target. It is
 a point-in-time precondition, not an attestation against a concurrent local writer.
 
+When the PR moves, the old session stays as it was and a replacement starts beside it at
+`<session>-r2/`. `link-predecessor` records the round, the lines the PR touches that changed
+since the walked head, and every finding the reviewer settled in an earlier round. The
+replacement walks only those moved lines, the page carries the earlier calls forward, and
+the store refuses a flag the reviewer already dropped while its line reads the same. By the
+fourth round the recommendation says to reconsider the approach rather than fix again.
+`history` returns what earlier sessions on the same repository learned, their facts and
+recommendation and the findings they left on the paths a new one touches, so the second
+review of an area starts from the first.
+
 Linked creation is deterministic for one source action and refuses a conflicting second
 authorization. A complete child is atomically published at its deterministic path, so a
 crash cannot expose a partial final session. Request reservation reuses a nonfailed attempt;
@@ -306,6 +324,14 @@ moved target before the update blocks landing and requires a replacement source
 A move detected after the local update preserves the landed commit and reports that a
 replacement is required, without causing another effect.
 
+`sessionctl.py stats ~/.claude/reviews` is the noise ledger. For every stored session and
+in total it counts beats, flags raised, flags still open, accepted, decided, dropped by
+reason, unverified beats and their share, flags a second read confirmed or disputed, and
+questions asked. A drop recorded before reasons existed counts as unstated. It reads each
+session from a backup copy, so counting never upgrades an old session in place. Read it
+before and after a change to the skill: a change that raises the dropped share is a
+regression even when it finds more.
+
 `render-report.py` turns a session into the page and validates it on the way through. A
 flag with no fix, a clean beat with no proof, or a proof naming no command gets an
 `UNPROVEN` chip and exit 2. During a live walk, pending and failed delivery are visible
@@ -321,6 +347,29 @@ unsnappable ones into the body rather than dropping them.
 Iterating on the page design means editing `skills/underwrite/assets/report.css`, or
 passing `--css` to try something without a commit.
 
+## Evals
+
+`evals/cases/` holds pull requests with a known answer: misses, where a defect shipped and a
+later PR fixed it, and clean PRs that nothing later had to fix. `evals/run.py` clones each at
+its base, walks it headlessly with this checkout's skill in eval mode, and scores the session
+with `evals/score.py`. A flag the second read demoted does not stand, so it counts neither
+for a miss nor against a clean case. Nothing is posted: every case is a merged PR, which freezes to report
+audience, and eval mode delivers nothing.
+
+```bash
+python3 evals/run.py                         # every case
+python3 evals/run.py underwrite-83-legacy-guard --keep --max-budget-usd 3
+```
+
+A case passes when every expected finding is found, by place and by the keywords of its class,
+and nothing else stands. `--baseline evals/baseline.json` fails only on a regression: a case
+that passed and now fails, or more misses or false alarms than the committed baseline. Keep a
+change to `SKILL.md` only when it passes that, and commit a new baseline when it improves. The
+`evals` workflow runs this on pull requests that touch the skill or the evals, but only once
+an `ANTHROPIC_API_KEY` secret is set, because each run spends model budget; CI checks the cases
+and the scorer either way. `--bare` keeps hooks and personal instructions
+out of the run and needs `ANTHROPIC_API_KEY`. `docs/misses.md` records what each miss lacked.
+
 ## License
 
 MIT

docs/misses.md

@@ -0,0 +1,78 @@
+# Misses
+
+A miss is a defect that shipped through a pull request and was found later. Each one here is
+an eval case under `evals/cases/`, and each entry asks the question the Anthropic code review
+write-up asks of every miss: what did the reviewer not have in front of it? A change to the
+skill is kept only when `evals/run.py` still finds every miss without raising more false
+alarms on the clean cases.
+
+## #83 refused legacy sessions in `links`
+
+- **Shipped:** `attested_implementation.py:561` refused any source without `session.sqlite3`.
+  A legacy session is `session.json` plus `beats/` and no database, and its first open is the
+  migration, so the guard refused the one shape that converts itself.
+- **Found by:** #84, which moved the predicate into `SessionStore.__init__`, the one place
+  every caller passes through.
+- **What was missing:** which on-disk shapes count as a session was written down nowhere a
+  reader of `links` would look. The predicate looked obviously right in isolation.
+- **Case:** `underwrite-83-legacy-guard`. No walk of #83 was stored, so this enters as a case
+  rather than a replayed session.
+
+## #76 canonicalized slots on write and compared them raw
+
+- **Shipped:** `session_store.py:1550` canonicalized incoming slot keys, while line 907 skipped
+  the repair for pinned beats. A pinned beat read with `get-beat` and written straight back
+  then counted as a change, deleted the `fix` slot and voided the authorization.
+- **Found by:** #79, which compares the stored beat as the store spells it.
+- **What was missing:** the invariant that a comparison reads both sides through the same
+  spelling. The two halves sat 600 lines apart, and the diff showed each one as correct.
+- **Case:** `underwrite-76-normalize-compare`.
+
+## Candidates not admitted
+
+- **cornermind #29:** `migrate.yml` allowed `workflow_dispatch` from any branch, later guarded
+  in cornermind #54. The `production-db` environment already allowed only `main`, so the fix
+  was defense in depth, not a shipped defect.
+- **ai-kit #15:** the break-check gate could be bypassed by editing its own config, fixed in
+  ai-kit #17. The PR body named the gap and deferred it, so a walk that stayed quiet was not
+  wrong.
+
+## Runs
+
+| Date | Skill | Case | Result | Cost |
+|---|---|---|---|---|
+| 2026-10-21 | second read demotes on any no | `underwrite-17-sigterm` | fail: one confirmed flag on a weak race test | $1.64 |
+| 2026-10-21 | second read demotes on any no | `underwrite-83-legacy-guard` | fail: the walk flagged `:561`, the second read demoted it as not worth stopping | $1.89 |
+| 2026-10-21 | demotes only on not introduced or not real | `underwrite-83-legacy-guard` | pass: `:561` flagged and kept, worth doubt noted in its evidence | $2.20 |
+
+| 2026-10-21 | demotes only on not introduced or not real, before the completion marker | all five | 2 of 5 pass, 1 miss, 2 false alarms | $7.74 |
+| 2026-10-21 | final skill, baseline in `evals/baseline.json` | all five | 3 of 5 pass, 1 miss, 2 false alarms | $9.24 |
+
+In the baseline run `underwrite-83-legacy-guard`, `underwrite-12-host-guard` and
+`ai-kit-5-core-deps` pass. `underwrite-76-normalize-compare` raised the right class of finding,
+an upper-case slot read from before the repair, but at `session_store.py:2485`, far from the
+defect at `:907` and `:1550`, so it scores as one miss and one false alarm; it is the live
+miss for the next skill change to win back. `underwrite-17-sigterm` raised the same weak race
+test flag on every run. `ai-kit-5-core-deps` raised an alarm in the run before and none in
+this one; that alarm, that the guard never checks it parsed a dynamic import, is the class of
+finding the original reviewer accepted in the stored session of that PR, so the case may be
+mislabeled clean, and it stays until someone decides.
+
+Walks are not deterministic: the same skill moved `ai-kit-5-core-deps` between pass and fail
+across two runs. The baseline is one run, so a lone flip on a regression check is a reason to
+rerun that case before believing it. Four walks of one run first stopped early because
+exploratory commands were denied, which is why the runner now tells the walk what it may run.
+
+The second run in the table is why the second read no longer demotes on worth. A replay of the six flags
+stored before the second read existed agreed: it disputed two on worth, and the reviewer had
+acted on one of them. The `underwrite-17-sigterm` flag is a fair question about a test that
+catches the old bug only about one run in seven; it stands as a false alarm because nothing
+later had to fix #17.
+
+## Adding a miss
+
+1. Write `evals/cases/<repo>-<number>-<slug>.json` with the PR's base and head, the defect's
+   `path` and `lines` as they read at the head, and the fix as `evidence`. The lines must sit
+   inside the PR's diff, or no walk can flag them.
+2. Add an entry here naming what the reviewer did not have in front of it.
+3. Run `python3 evals/run.py <case-id>` on the current skill and record the result.

evals/baseline.json

@@ -0,0 +1,118 @@
+{
+  "results": [
+    {
+      "beats": 4,
+      "case": "ai-kit-5-core-deps",
+      "cost_usd": 1.5828408000000003,
+      "demoted": 0,
+      "false_alarms": [],
+      "finished": true,
+      "flags": 0,
+      "found": [],
+      "kind": "clean",
+      "missed": [],
+      "pass": true
+    },
+    {
+      "beats": 3,
+      "case": "underwrite-12-host-guard",
+      "cost_usd": 1.8406928000000002,
+      "demoted": 0,
+      "false_alarms": [],
+      "finished": true,
+      "flags": 0,
+      "found": [],
+      "kind": "clean",
+      "missed": [],
+      "pass": true
+    },
+    {
+      "beats": 1,
+      "case": "underwrite-17-sigterm",
+      "cost_usd": 1.7215960000000001,
+      "demoted": 0,
+      "false_alarms": [
+        {
+          "claim": "the race test passes on the old ordering most runs",
+          "n": 1,
+          "where": "tests/test_serve.py:434"
+        }
+      ],
+      "finished": true,
+      "flags": 1,
+      "found": [],
+      "kind": "clean",
+      "missed": [],
+      "pass": false
+    },
+    {
+      "beats": 5,
+      "case": "underwrite-76-normalize-compare",
+      "cost_usd": 2.2239962,
+      "demoted": 0,
+      "false_alarms": [
+        {
+          "claim": "a linked child seeded before v6 from an uppercase beat can no longer be resumed",
+          "n": 5,
+          "where": "skills/underwrite/scripts/session_store.py:2485"
+        }
+      ],
+      "finished": true,
+      "flags": 1,
+      "found": [],
+      "kind": "miss",
+      "missed": [
+        {
+          "class": "slots canonicalized on write but compared raw, so a pinned beat written back unchanged counts as a change",
+          "keywords": [
+            "canonical",
+            "spelling",
+            "upper",
+            "pinned",
+            "raw"
+          ],
+          "lines": [
+            907,
+            1550
+          ],
+          "path": "skills/underwrite/scripts/session_store.py"
+        }
+      ],
+      "pass": false
+    },
+    {
+      "beats": 4,
+      "case": "underwrite-83-legacy-guard",
+      "cost_usd": 1.8666264000000004,
+      "demoted": 0,
+      "false_alarms": [],
+      "finished": true,
+      "flags": 1,
+      "found": [
+        {
+          "beat": 1,
+          "class": "an existence guard keyed on the new storage format refuses a legacy session.json session",
+          "keywords": [
+            "legacy",
+            "session.json",
+            "pre-sqlite",
+            "migrat"
+          ],
+          "lines": [
+            561
+          ],
+          "path": "skills/underwrite/scripts/attested_implementation.py"
+        }
+      ],
+      "kind": "miss",
+      "missed": [],
+      "pass": true
+    }
+  ],
+  "summary": {
+    "cases": 5,
+    "false_alarms": 2,
+    "missed": 1,
+    "passed": 3
+  }
+}

evals/cases/ai-kit-5-core-deps.json

@@ -0,0 +1,10 @@
+{
+  "id": "ai-kit-5-core-deps",
+  "kind": "clean",
+  "repo": "radkode/ai-kit",
+  "number": 5,
+  "base_sha": "49900cb40b08c88bab66be4e73a510b07cdc9b5a",
+  "head_sha": "1b007a07504df51eb1f90725b52713a3dc5e6115",
+  "expected": [],
+  "evidence": "scripts/check-core-deps.mjs unchanged since merge"
+}

evals/cases/underwrite-12-host-guard.json

@@ -0,0 +1,10 @@
+{
+  "id": "underwrite-12-host-guard",
+  "kind": "clean",
+  "repo": "radkode/underwrite",
+  "number": 12,
+  "base_sha": "6c011f38c8a8311e8b4f309e542e50b77c72f5d7",
+  "head_sha": "cf3e876d2936a29b7e861a07354db4c7a1cf4a17",
+  "expected": [],
+  "evidence": "no later fix or revert touches its lines"
+}

evals/cases/underwrite-17-sigterm.json

@@ -0,0 +1,10 @@
+{
+  "id": "underwrite-17-sigterm",
+  "kind": "clean",
+  "repo": "radkode/underwrite",
+  "number": 17,
+  "base_sha": "5c7bae2e6e0bb4600d688575b66a3cc811651239",
+  "head_sha": "22d4ab5440d5ec5c17e769fd6b4d7513994d3126",
+  "expected": [],
+  "evidence": "its SIGTERM block was later touched only by tests (#23) and a feature (#63)"
+}

evals/cases/underwrite-76-normalize-compare.json

@@ -0,0 +1,26 @@
+{
+  "id": "underwrite-76-normalize-compare",
+  "kind": "miss",
+  "repo": "radkode/underwrite",
+  "number": 76,
+  "base_sha": "80bdf166d0f40f214b37166f8d7994b6ee19c47e",
+  "head_sha": "ee01410253691c0cde9e3f17251ca33c1d8a867b",
+  "expected": [
+    {
+      "path": "skills/underwrite/scripts/session_store.py",
+      "lines": [
+        907,
+        1550
+      ],
+      "class": "slots canonicalized on write but compared raw, so a pinned beat written back unchanged counts as a change",
+      "keywords": [
+        "canonical",
+        "spelling",
+        "upper",
+        "pinned",
+        "raw"
+      ]
+    }
+  ],
+  "evidence": "https://github.com/radkode/underwrite/pull/79"
+}

evals/cases/underwrite-83-legacy-guard.json

@@ -0,0 +1,24 @@
+{
+  "id": "underwrite-83-legacy-guard",
+  "kind": "miss",
+  "repo": "radkode/underwrite",
+  "number": 83,
+  "base_sha": "90e432561bdd16932be9bbb9500189c4bdc78450",
+  "head_sha": "ef1fa2222f1b6481cae6c85528e1de5383cb4bc1",
+  "expected": [
+    {
+      "path": "skills/underwrite/scripts/attested_implementation.py",
+      "lines": [
+        561
+      ],
+      "class": "an existence guard keyed on the new storage format refuses a legacy session.json session",
+      "keywords": [
+        "legacy",
+        "session.json",
+        "pre-sqlite",
+        "migrat"
+      ]
+    }
+  ],
+  "evidence": "https://github.com/radkode/underwrite/pull/84"
+}