From 0011231984797af34e656309f0f2dbc33f5488dc Mon Sep 17 00:00:00 2001 From: Daniel McCoy Stephenson Date: Tue, 25 Aug 2026 01:14:13 -0600 Subject: [PATCH 1/3] Make the checkout form Phase 4's default Tests-fix revert and gate rung 1 The Tests-fix rubric item prescribed `git stash push` on a tree that Phase 3 has already committed and Phase 4 has already pushed, so the stash saved nothing, the check ran against the unmodified tree, and the no-op read as a false-negative in the test. The checkout form is now the default, stash is demoted to the genuinely-uncommitted case, and the revert must be confirmed to have changed the tree before either half is trusted. The fallback ladder loses its "retry with checkout first" preamble, since the retry is now the default, and rung 1 gains an entry condition: a headless or timeout-bounded dispatch must not push a head that undoes its own fix, because a killed session leaves nothing to restore it. Closes #116 Closes #132 --- create-dev-loop.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/create-dev-loop.md b/create-dev-loop.md index 3f059b7..dcaad9f 100644 --- a/create-dev-loop.md +++ b/create-dev-loop.md @@ -362,15 +362,17 @@ Perform a self-review. This step is anchored on external signals ({{EXTERNAL_SIG Universal rubric: - **Scope:** every file modified is necessary for one of the issues in `Closes #N` (no unrelated formatting, renames, or comment churn). - **Tests-new:** every new public method/function has at least one test that exercises it. - - **Tests-fix (empirical, not judged):** for each bug fix, temporarily revert the fix (`git stash push -- `), run the new/changed tests and confirm they **FAIL**, then `git stash pop` and confirm they **PASS**. A regression test that still passes with the fix stashed is a false-negative (common when the "after" state is indistinguishable from the "before") — use distinct/sentinel data so the failure is observable. Do not score this from reasoning alone. When the local anchor cannot run (see UNVERIFIED above), use the fallback ladder below instead of skipping this item. + - **Tests-fix (empirical, not judged):** for each bug fix, temporarily revert the fix, run the new/changed tests and confirm they **FAIL**, then restore the fix and confirm they **PASS**. By the time this item is reached the fix is normally already committed — Phase 3 commits it and Phase 4 has pushed the branch and opened the PR — so the working tree is clean and the **checkout form is the default**: `git checkout origin/{{DEFAULT_BRANCH}} -- ` to revert, `git checkout HEAD -- ` to restore. Reach for `git stash push -- ` / `git stash pop` only in the rarer case where the fix is genuinely still uncommitted. **Confirm the revert actually changed the tree** (`git status --porcelain`, or `git diff --stat`) before trusting either half of the result: `git stash push` on a clean tree saves nothing and reports success, the check then runs against the *unmodified* (fixed) tree and passes, and `git stash pop` fails with `No stash entries found` — scored naively that sequence reads as "reverted, still passed → false negative in the test" when in fact nothing was ever reverted. A regression test that still passes with the fix *genuinely* reverted is a real false-negative (common when the "after" state is indistinguishable from the "before") — use distinct/sentinel data so the failure is observable. Do not score this from reasoning alone. When the local anchor cannot run (see UNVERIFIED above), use the fallback ladder below instead of skipping this item. - **Sibling structure:** every new file matches the section/structure conventions of its directory siblings (Phase 3 rule). - **Sibling renames:** every renamed identifier in a parallel pair/series has its siblings renamed in the same commit (Phase 3 rule). - **Docs:** every row in the Phase 7 documentation sources-of-truth table reflects the new behavior. - **Issue resolution:** every `Closes #N` issue's named surface area is actually changed; no issue is partially resolved while claiming closure. - **{{EXTERNAL_SIGNAL_LABEL}}:** the external anchor is green on the PR head (re-confirms step 1). - **Tests-fix fallback ladder when the local anchor cannot run.** The stash-and-run experiment above needs a live local anchor. CI on the fixed tree alone cannot substitute for it — CI only ever runs the *fixed* state and can never reproduce the stashed-revert half. **Before concluding the local anchor cannot run at all, retry with a targeted checkout-based revert** (`git checkout -- `, run tests locally, then `git checkout HEAD -- ` to restore) — `git stash` can fail for reasons unrelated to the tool/interpreter itself (a dirty working tree, submodule state), and checkout avoids those. If that runs, it *is* the stash-and-run experiment (just via a different git mechanism) — score Tests-fix from it directly and skip the ladder below entirely. Only enter this ladder when the tool/interpreter itself is unavailable or broken (the UNVERIFIED case above), where no local technique — stash or checkout — can execute: - 1. **CI-based temporary revert.** Push a commit that reverts only the production fix while keeping the new/changed tests, with a commit message that makes recovery undoable by a future session without investigation, e.g. `TEMP: revert to prove regression tests fail — MUST be reverted before merge, see `; confirm CI goes **red** naming exactly the new regression tests; then `git reset --hard` back to the verified fix commit and force-push, confirming CI goes **green** again. Strongest available substitute — costs two CI round-trips and leaves a real broken commit on the branch until the reset completes, so do not leave a session mid-ladder (see Phase 1's orphaned-PR handling). + **Tests-fix fallback ladder when the local anchor cannot run.** The revert-and-run experiment above needs a live local anchor. {{EXTERNAL_SIGNAL_LABEL}} on the fixed tree alone cannot substitute for it — it only ever runs the *fixed* state and can never reproduce the reverted half. Enter this ladder **only** when the tool/interpreter itself is unavailable or broken (the UNVERIFIED case above), so that no local technique can execute. A failure of one git mechanism is not that condition: if `git stash` refuses (a clean tree, a dirty tree, submodule state) but `git checkout` works — or the reverse — that *is* the experiment above via a different mechanism, so score Tests-fix from it directly and do not enter the ladder. + 1. **{{EXTERNAL_SIGNAL_LABEL}}-based temporary revert.** Push a commit that reverts only the production fix while keeping the new/changed tests, with a commit message that makes recovery undoable by a future session without investigation, e.g. `TEMP: revert to prove regression tests fail — MUST be reverted before merge, see `; confirm {{EXTERNAL_SIGNAL_LABEL}} goes **red** naming exactly the new regression tests; then `git reset --hard` back to the verified fix commit and force-push, confirming it goes **green** again. Strongest available substitute — costs two round-trips and leaves a real broken commit on the branch until the reset completes, so do not leave a session mid-ladder (see Phase 1's orphaned-PR handling). + + **Entry condition — do not use rung 1 under a headless or timeout-bounded dispatch.** Rung 1 is only safe when the session is certain to survive both round-trips. Where no human is present and the run may be killed at any point (`gardener tend` and equivalents — the same condition the autonomous-batch rule in Phase 5 keys on), a killed session leaves a pushed head that deliberately undoes its own fix, and nothing restores it until a later cycle happens to notice: precisely the stuck-mid-revert state Phase 1's orphaned-PR handling exists to repair. Skip rung 1 in that case and fall through to rung 2, or to rung 3's hand-off. 2. **Pre-existing test changed by the fix.** If a test written before the fix asserted the old behavior and the fix's diff changes that test's assertion to the new behavior, that diff is itself a recorded FAIL→PASS — quote the test name and the changed assertion instead of re-running it. 3. **Neither is available.** Score Tests-fix **FAIL** and do not auto-merge — hand to a human. A bug-fix PR whose regression evidence can't be established by either rung above does not clear the regression gate. {{#if SELF_REVIEW_RUBRIC}} @@ -390,7 +392,7 @@ Perform a self-review. This step is anchored on external signals ({{EXTERNAL_SIG # Self-review rubric: # - Scope: PASS — # - Tests-new: PASS — - # - Tests-fix: PASS — stash-and-run confirmed FAIL→PASS + # - Tests-fix: PASS — checkout-revert-and-run confirmed FAIL→PASS # - ... # # @@ -459,7 +461,7 @@ Only proceed when a complete pass finds nothing wrong. ### Phase 8 — Merge -**Regression gate.** For each issue in `Closes #N` that is a bug fix or describes incorrect behavior, verify the diff includes a new or modified test (or, for projects whose external anchor is manual validation, a new validation step) that exercises the fix — and that it was confirmed empirically (the Phase 4 stash-and-run, or its fallback ladder when the local anchor cannot run: FAIL/RED with the fix reverted, PASS/GREEN with it restored), not by reasoning alone. If absent, do not merge — either add the regression coverage or reclassify the issue. Per RESEARCH.md §3, regression evidence is the only way to distinguish a real fix from a coincidental patch. +**Regression gate.** For each issue in `Closes #N` that is a bug fix or describes incorrect behavior, verify the diff includes a new or modified test (or, for projects whose external anchor is manual validation, a new validation step) that exercises the fix — and that it was confirmed empirically (the Phase 4 revert-and-run, or its fallback ladder when the local anchor cannot run: FAIL/RED with the fix reverted, PASS/GREEN with it restored), not by reasoning alone. If absent, do not merge — either add the regression coverage or reclassify the issue. Per RESEARCH.md §3, regression evidence is the only way to distinguish a real fix from a coincidental patch. **Do-not-auto-merge path check.** Before invoking `gh pr merge`, list the files this PR modifies and check them against the do-not-auto-merge list. If any modified path matches, do not merge automatically — leave the PR open and report to the user for manual review. From 167e2f828447b352e52235ff807d323de7789303 Mon Sep 17 00:00:00 2001 From: Daniel McCoy Stephenson Date: Tue, 25 Aug 2026 01:15:48 -0600 Subject: [PATCH 2/3] Record PR #136 under RESEARCH.md findings 1 and 3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Finding 1's entry covers both defects: a revert that silently does nothing returns Tests-fix to unanchored judgment while still reporting an anchor's verdict, and a ladder rung whose interruption leaves the head undoing its own fix damages the anchor it substitutes for. Finding 3's entry covers the consequence for the regression gate it motivated in PR #31 — manufactured evidence cannot distinguish a real fix from a coincidental patch. --- RESEARCH.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/RESEARCH.md b/RESEARCH.md index e02960e..88fe7cc 100644 --- a/RESEARCH.md +++ b/RESEARCH.md @@ -50,6 +50,7 @@ Last reviewed: 2026-08-04. - PR #67 (Add Tests-fix CI-revert fallback ladder): shipped 2026-07-26. Observed effect: pending — needs N cycles of data. Closes a gap left by PR #54: the Tests-fix rubric item required a local `git stash` experiment with no substitute when the local anchor can't run, even though the same PR's UNVERIFIED handling fully anticipates that case. Without a defined fallback, Tests-fix silently degraded back to unanchored judgment on exactly the repos this finding is about — confirmed twice, in two separate PRs opened by a generated child skill running against a repo whose local test anchor could not be run. - PR #119 (Harden Phase 8's rebase fence and state the merge-authorization precedence rule): shipped 2026-08-22. Observed effect: pending — needs N cycles of data. Two more ways the anchor silently weakens at the merge phase: the post-rebase verification ran the test command alone, so a required check covering a linter or static analysis was never part of the signal a force-push committed to (#115); and nothing stated whether a run-level merge pre-authorization could stand in for a Phase 4 UNVERIFIED result (#118). Both are the same failure this finding's PR #54 entry records for CI scope — an anchor that does not cover the changed surface is not an anchor — applied to permission rather than coverage in the second case. - PR #121 (Define the structurally-red-anchor path and require the anchor to have executed tests): shipped 2026-08-16. Observed effect: pending — needs N cycles of data. Extends the same argument as PR #54's "green CI is not verification when CI's scope excludes the changed files" to two further ways an anchor can carry no information: a required job that has *never* succeeded on the base branch (observed on a repo whose CI jobs have failed on empty credentials since February 2026, so every cycle terminated in a hand-off regardless of quality), and a build tool that reports success having executed zero tests (`:test NO-SOURCE` still prints `BUILD SUCCESSFUL`). Both were previously read as ordinary signals — the first as a blocking red, the second as a green — which is unanchored critique wearing an anchor's clothes. +- PR #136 (Make the checkout form Phase 4's default Tests-fix revert and gate ladder rung 1): shipped 2026-08-25. Observed effect: pending — needs N cycles of data. Repairs the mechanism PR #54's entry above introduced. The prescribed `git stash push` revert ran against a tree Phase 3 had already committed and Phase 4 had already pushed, so it saved nothing, the check then passed against the still-fixed tree, and the no-op was naturally scored as a false-negative in the *test* — an anchor reporting a verdict it never measured, which is precisely the failure this finding warns about. The checkout form becomes the default because it is correct in the state Phase 4 actually finds, and the revert must now be confirmed to have changed the tree before either half is trusted. Separately, the ladder rung PR #67 added is gated against headless dispatch, where an interrupted session leaves the branch head undoing its own fix and the substitute anchor damages the thing it substitutes for. --- @@ -97,6 +98,7 @@ Last reviewed: 2026-08-04. - PR #31 (Localization + regression gates): shipped 2026-05-25. Observed effect: pending — needs N cycles of data. Adds a Phase 3 "Localization verification" step (enumerate files, `test -f`, grep for the named surface area) and a Phase 8 regression gate (bug-fix `Closes #N` requires a new/modified test or validation step). Generalized to projects whose external anchor is manual validation rather than tests. - PR #52 (Add documentation-accuracy and unit-test-expansion work-mode stages to Phase 2): shipped 2026-06-08. Observed effect: pending — needs N cycles of data. Adds Stage B, a characterization-test work-mode that locks in current behavior and builds the regression guards this finding argues are the only way to distinguish a real fix from a coincidental patch — and forbids changing production code under a test-only cycle, so an apparent bug becomes a filed issue rather than an unverified patch. - PR #108 (Broaden Step 2 to find the PR template under every accepted spelling): shipped 2026-08-09. Observed effect: pending — needs N cycles of data. Widens Step 2's PR-template bullet from a single lowercase path to every spelling and location GitHub accepts. Applies the finding one level earlier than PR #31 does: before a generated skill can localize within its repo, Step 2 must localize the repo's own convention files, and a silently under-matching file list loses that input with no trace. +- PR #136 (Make the checkout form Phase 4's default Tests-fix revert and gate ladder rung 1): shipped 2026-08-25. Observed effect: pending — needs N cycles of data. Protects the regression gate this finding motivated in PR #31. That gate is satisfied by the Phase 4 Tests-fix experiment, which prescribed a `git stash push` revert that saves nothing on the clean tree Phase 4 always finds; the resulting evidence was manufactured rather than collected, so a coincidental patch and a real fix were indistinguishable exactly where this finding says they must be told apart. The checkout form becomes the default, and the revert must be confirmed to have changed the tree before the FAIL→PASS pair is scored. --- From 94f494c3760d0c2c5a58016956dad22004d7dc8c Mon Sep 17 00:00:00 2001 From: Daniel McCoy Stephenson Date: Tue, 25 Aug 2026 01:17:20 -0600 Subject: [PATCH 3/3] Keep rung 1 CI-specific and revert to the merge base, not the branch tip Two self-review findings. Substituting {{EXTERNAL_SIGNAL_LABEL}} into the ladder preamble and rung 1 was wrong: rung 1 pushes a commit and reads a check result, so it is inherently CI-specific and cannot be performed at all where the anchor is a manual checklist. The literal wording is restored and a sentence now states that unavailability outright. The checkout form said `git checkout origin/ -- `, which also pulls in anything that landed on the default branch since the branch was cut. The merge base is named as the correct revert target. --- create-dev-loop.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/create-dev-loop.md b/create-dev-loop.md index dcaad9f..3e6cd91 100644 --- a/create-dev-loop.md +++ b/create-dev-loop.md @@ -362,15 +362,15 @@ Perform a self-review. This step is anchored on external signals ({{EXTERNAL_SIG Universal rubric: - **Scope:** every file modified is necessary for one of the issues in `Closes #N` (no unrelated formatting, renames, or comment churn). - **Tests-new:** every new public method/function has at least one test that exercises it. - - **Tests-fix (empirical, not judged):** for each bug fix, temporarily revert the fix, run the new/changed tests and confirm they **FAIL**, then restore the fix and confirm they **PASS**. By the time this item is reached the fix is normally already committed — Phase 3 commits it and Phase 4 has pushed the branch and opened the PR — so the working tree is clean and the **checkout form is the default**: `git checkout origin/{{DEFAULT_BRANCH}} -- ` to revert, `git checkout HEAD -- ` to restore. Reach for `git stash push -- ` / `git stash pop` only in the rarer case where the fix is genuinely still uncommitted. **Confirm the revert actually changed the tree** (`git status --porcelain`, or `git diff --stat`) before trusting either half of the result: `git stash push` on a clean tree saves nothing and reports success, the check then runs against the *unmodified* (fixed) tree and passes, and `git stash pop` fails with `No stash entries found` — scored naively that sequence reads as "reverted, still passed → false negative in the test" when in fact nothing was ever reverted. A regression test that still passes with the fix *genuinely* reverted is a real false-negative (common when the "after" state is indistinguishable from the "before") — use distinct/sentinel data so the failure is observable. Do not score this from reasoning alone. When the local anchor cannot run (see UNVERIFIED above), use the fallback ladder below instead of skipping this item. + - **Tests-fix (empirical, not judged):** for each bug fix, temporarily revert the fix, run the new/changed tests and confirm they **FAIL**, then restore the fix and confirm they **PASS**. By the time this item is reached the fix is normally already committed — Phase 3 commits it and Phase 4 has pushed the branch and opened the PR — so the working tree is clean and the **checkout form is the default**: `git checkout origin/{{DEFAULT_BRANCH}} -- ` to revert, `git checkout HEAD -- ` to restore. Revert to the **merge base** (`git merge-base HEAD origin/{{DEFAULT_BRANCH}}`) rather than the branch tip wherever `{{DEFAULT_BRANCH}}` has moved on since the branch was cut, so the experiment isolates this PR's change instead of also pulling in whatever else landed meanwhile. Reach for `git stash push -- ` / `git stash pop` only in the rarer case where the fix is genuinely still uncommitted. **Confirm the revert actually changed the tree** (`git status --porcelain`, or `git diff --stat`) before trusting either half of the result: `git stash push` on a clean tree saves nothing and reports success, the check then runs against the *unmodified* (fixed) tree and passes, and `git stash pop` fails with `No stash entries found` — scored naively that sequence reads as "reverted, still passed → false negative in the test" when in fact nothing was ever reverted. A regression test that still passes with the fix *genuinely* reverted is a real false-negative (common when the "after" state is indistinguishable from the "before") — use distinct/sentinel data so the failure is observable. Do not score this from reasoning alone. When the local anchor cannot run (see UNVERIFIED above), use the fallback ladder below instead of skipping this item. - **Sibling structure:** every new file matches the section/structure conventions of its directory siblings (Phase 3 rule). - **Sibling renames:** every renamed identifier in a parallel pair/series has its siblings renamed in the same commit (Phase 3 rule). - **Docs:** every row in the Phase 7 documentation sources-of-truth table reflects the new behavior. - **Issue resolution:** every `Closes #N` issue's named surface area is actually changed; no issue is partially resolved while claiming closure. - **{{EXTERNAL_SIGNAL_LABEL}}:** the external anchor is green on the PR head (re-confirms step 1). - **Tests-fix fallback ladder when the local anchor cannot run.** The revert-and-run experiment above needs a live local anchor. {{EXTERNAL_SIGNAL_LABEL}} on the fixed tree alone cannot substitute for it — it only ever runs the *fixed* state and can never reproduce the reverted half. Enter this ladder **only** when the tool/interpreter itself is unavailable or broken (the UNVERIFIED case above), so that no local technique can execute. A failure of one git mechanism is not that condition: if `git stash` refuses (a clean tree, a dirty tree, submodule state) but `git checkout` works — or the reverse — that *is* the experiment above via a different mechanism, so score Tests-fix from it directly and do not enter the ladder. - 1. **{{EXTERNAL_SIGNAL_LABEL}}-based temporary revert.** Push a commit that reverts only the production fix while keeping the new/changed tests, with a commit message that makes recovery undoable by a future session without investigation, e.g. `TEMP: revert to prove regression tests fail — MUST be reverted before merge, see `; confirm {{EXTERNAL_SIGNAL_LABEL}} goes **red** naming exactly the new regression tests; then `git reset --hard` back to the verified fix commit and force-push, confirming it goes **green** again. Strongest available substitute — costs two round-trips and leaves a real broken commit on the branch until the reset completes, so do not leave a session mid-ladder (see Phase 1's orphaned-PR handling). + **Tests-fix fallback ladder when the local anchor cannot run.** The revert-and-run experiment above needs a live local anchor. CI on the fixed tree alone cannot substitute for it — CI only ever runs the *fixed* state and can never reproduce the reverted half. Enter this ladder **only** when the tool/interpreter itself is unavailable or broken (the UNVERIFIED case above), so that no local technique can execute. A failure of one git mechanism is not that condition: if `git stash` refuses (a clean tree, a dirty tree, submodule state) but `git checkout` works — or the reverse — that *is* the experiment above via a different mechanism, so score Tests-fix from it directly and do not enter the ladder. + 1. **CI-based temporary revert.** Push a commit that reverts only the production fix while keeping the new/changed tests, with a commit message that makes recovery undoable by a future session without investigation, e.g. `TEMP: revert to prove regression tests fail — MUST be reverted before merge, see `; confirm CI goes **red** naming exactly the new regression tests; then `git reset --hard` back to the verified fix commit and force-push, confirming CI goes **green** again. Strongest available substitute — costs two CI round-trips and leaves a real broken commit on the branch until the reset completes, so do not leave a session mid-ladder (see Phase 1's orphaned-PR handling). Requires an automated remote signal, so this rung is unavailable where {{EXTERNAL_SIGNAL_LABEL}} names a manual checklist rather than CI. **Entry condition — do not use rung 1 under a headless or timeout-bounded dispatch.** Rung 1 is only safe when the session is certain to survive both round-trips. Where no human is present and the run may be killed at any point (`gardener tend` and equivalents — the same condition the autonomous-batch rule in Phase 5 keys on), a killed session leaves a pushed head that deliberately undoes its own fix, and nothing restores it until a later cycle happens to notice: precisely the stuck-mid-revert state Phase 1's orphaned-PR handling exists to repair. Skip rung 1 in that case and fall through to rung 2, or to rung 3's hand-off. 2. **Pre-existing test changed by the fix.** If a test written before the fix asserted the old behavior and the fix's diff changes that test's assertion to the new behavior, that diff is itself a recorded FAIL→PASS — quote the test name and the changed assertion instead of re-running it.