Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions RESEARCH.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ Last reviewed: 2026-08-04.
- PR #54 (Apply consolidated template-rule backlog, #44/#46/#47): shipped 2026-06-14. Observed effect: pending — needs N cycles of data. Sharpens the external-anchor definition (#44: UNVERIFIED handling when the anchor can't run; #46: green CI is not verification when CI's scope excludes the changed files) and grounds the Tests-fix rubric item empirically (#47: stash-and-run FAIL→PASS, not scored from reasoning alone) — directly targeting the self-preference failure mode this finding's evidence documents for judgment-based rubric scoring.
- PR #57 (Fix documentation drift across the three sources of truth): shipped 2026-07-25. Observed effect: pending — needs N cycles of data. Restores `{{EXTERNAL_SIGNAL_LABEL}}` in the Phase 1 carried-over-PR bullet, where PR #54 had reintroduced a hardcoded "CI must be green first" after PR #28 generalized the anchor. An anchor named `CI` in a repo that has no CI is an anchor the loop cannot check, which degrades the self-review to exactly the unanchored critique this finding shows is neutral-to-harmful.
- PR #67 (Add Tests-fix CI-revert fallback ladder): shipped 2026-07-26. Observed effect: pending — needs N cycles of data. Closes a gap left by PR #54: the Tests-fix rubric item required a local `git stash` experiment with no substitute when the local anchor can't run, even though the same PR's UNVERIFIED handling fully anticipates that case. Without a defined fallback, Tests-fix silently degraded back to unanchored judgment on exactly the repos this finding is about — confirmed twice, in two separate PRs opened by a generated child skill running against a repo whose local test anchor could not be run.
- PR #119 (Harden Phase 8's rebase fence and state the merge-authorization precedence rule): shipped 2026-08-22. Observed effect: pending — needs N cycles of data. Two more ways the anchor silently weakens at the merge phase: the post-rebase verification ran the test command alone, so a required check covering a linter or static analysis was never part of the signal a force-push committed to (#115); and nothing stated whether a run-level merge pre-authorization could stand in for a Phase 4 UNVERIFIED result (#118). Both are the same failure this finding's PR #54 entry records for CI scope — an anchor that does not cover the changed surface is not an anchor — applied to permission rather than coverage in the second case.
- PR #121 (Define the structurally-red-anchor path and require the anchor to have executed tests): shipped 2026-08-16. Observed effect: pending — needs N cycles of data. Extends the same argument as PR #54's "green CI is not verification when CI's scope excludes the changed files" to two further ways an anchor can carry no information: a required job that has *never* succeeded on the base branch (observed on a repo whose CI jobs have failed on empty credentials since February 2026, so every cycle terminated in a hand-off regardless of quality), and a build tool that reports success having executed zero tests (`:test NO-SOURCE` still prints `BUILD SUCCESSFUL`). Both were previously read as ordinary signals — the first as a blocking red, the second as a green — which is unanchored critique wearing an anchor's clothes.

---
Expand Down
19 changes: 14 additions & 5 deletions create-dev-loop.md
Original file line number Diff line number Diff line change
Expand Up @@ -486,6 +486,8 @@ git checkout {{DEFAULT_BRANCH}} && git pull

**Review-ready hand-off is a valid terminal state — not a failed cycle.** On a repo with no autonomous merge path (branch-protected `{{DEFAULT_BRANCH}}` requiring an approving review, and/or repository auto-merge disabled), a clean, anchor-green PR's true terminal state is *open + self-review posted + awaiting human approval*. Expect `gh pr merge` → `the base branch policy prohibits the merge` and `--auto` → `Auto merge is not allowed for this repository`. **Never use `--admin`** to bypass a deliberately-configured human gate, and a later self-audit must not score "could not auto-merge" as a failure. A do-not-auto-merge path match blocks **autonomous** merge only: if the human codeowner explicitly authorizes the merge after being shown which protected path matched, that authorization satisfies the hold — proceed (still run the Phase 7 docs sweep + the regression gate above first), and state in the merge report which protected path was overridden and by whose authorization.

**Standing merge authorization does not satisfy the anchor gate.** A run-level merge pre-authorization granted before this PR existed (e.g. a headless dispatch launched with a blanket "may merge" flag) conveys *permission*, not *validation* — where Phase 4 has marked the PR UNVERIFIED because the anchor could not run on anchor-relevant files, the hand-off stands regardless of merge authority. The codeowner exception above is retrospective by construction: it applies only to authorization given *after* the specific protected path has been surfaced to the codeowner, so a flag passed before the PR was written does not satisfy it. The terminal state in that case is open + self-review posted + awaiting a human, which the paragraph above already declares valid rather than failed.

**A structurally-red anchor is a named blocking condition, not an ordinary hand-off.** Where Phase 1 recorded a required workflow that has never succeeded on `{{DEFAULT_BRANCH}}`, the merge gate cannot be satisfied by any change this loop could write, so the hand-off must say that rather than reporting "awaiting review" — the two have entirely different remedies. State which jobs were assessed as carrying no signal for this diff and on what grounds, which jobs carried signal and were green, and the issue tracking the structural failure (file one on the project repo if none exists, and link it). The point is that the owner sees the single action that would unblock every future cycle, instead of a hand-off that reads like a queue.

**Worktree note:** under a `git worktree` workflow (main checkout kept on `{{DEFAULT_BRANCH}}`), `gh pr merge --delete-branch`'s *local* branch deletion can fail with `fatal: '{{DEFAULT_BRANCH}}' is already checked out` — the merge and remote-branch deletion still succeed. Verify the real state with `gh pr view <number> --json state` and delete the remote branch separately rather than treating the local error as a failed merge.
Expand Down Expand Up @@ -565,13 +567,20 @@ Return to Phase 1.
**A `gh` command fails with a transient network/transport error (`http2: client conn could not be established`, `unexpected EOF`):** retry once or twice before treating it as blocked — do not treat the first failure as terminal and silently drop a self-review, commit, or comment.
**`gh pr create` fails with "you must first push the current branch to a remote" despite a successful `git push -u`:** the clone's fetch refspec may be restricted to the default branch only (common in gardener-managed or otherwise restricted dedicated checkouts) — confirm with `git config --get-all remote.origin.fetch`. A restricted refspec means `git fetch` never populates a remote-tracking ref for feature branches, so `gh pr create`'s and `git branch --set-upstream-to`'s auto-detection both fail even though the push and upstream config succeeded. Pass `--head <branch>` explicitly to `gh pr create` to bypass the remote-tracking-ref lookup rather than retrying the push.
{{#if REVIEWER}}**Reviewer addition fails:** proceed to the self-review step in Phase 4. Do not skip to Phase 5 without posting self-review comments — Phase 6 addresses those comments like any other review.{{/if}}
**Branch is behind main or has a merge conflict at Phase 8:** rebase onto main, re-run tests, and force-push before retrying the merge:
**Branch is behind main or has a merge conflict at Phase 8:** rebase onto main, re-run the **same verification set Phase 3 runs** — not just the test command — and force-push before retrying the merge:
\`\`\`bash
git fetch origin
git rebase origin/{{DEFAULT_BRANCH}}
{{COMPILE_CMD}}
{{TEST_CMD}}
git push --force-with-lease
{{#if LINT_CMD}}{{LINT_CMD}}{{/if}}
git ls-remote origin refs/heads/<branch> # full 40-char SHA for the lease below
git push origin <branch>:<branch> --force-with-lease=<branch>:<full-40-char-sha>
\`\`\`
**Verify with everything the required checks run, not a subset.** A force-push commits to the result, so a repository whose required checks also run a formatter, linter or static analysis will otherwise go red on a task the test command never covered, minutes later, from CI. Where the repository's required checks exceed the Phase 3 verification set, run those too rather than trusting the subset.

**`(stale info)` has three unrelated causes; only one justifies backing off.** A bare `--force-with-lease` compares against a maintained remote-tracking ref, which a restricted clone does not have (fetch refspec covering `{{DEFAULT_BRANCH}}` only — see the `gh pr create` entry above); the push is rejected as `! [rejected] <branch> -> <branch> (stale info)`. The explicit `--force-with-lease=<branch>:<sha>` form fixes that, but the expected value is compared against the remote ref literally and abbreviations are never expanded — so a short SHA copied from `git log --oneline` is rejected with the *same* message. A restricted fetch refspec, an abbreviated SHA, and a genuine concurrent push therefore all read identically. Do not treat `(stale info)` as "another session pushed" until the expected SHA has been confirmed **full 40-character** and re-read from the remote at push time (`git ls-remote`; `git rev-parse origin/<branch>` also yields the full form but only reflects the last fetch, so it is the weaker source where the lease is meant to be meaningful). Never fall back to a plain `--force`, which clobbers unseen work in exactly the one case that warranted stopping.

If the rebase produces conflicts that cannot be resolved automatically, close the PR and delete the branch, then return to Phase 1:
\`\`\`bash
gh pr close <number> --comment "Closing: unresolvable merge conflict after rebase."
Expand Down Expand Up @@ -606,9 +615,9 @@ Use findings from Step 2 to substitute each `{{placeholder}}`. The table below c
| `SKILL_REPO_OWNER` | `gh api user -q .login` — the authenticated GitHub account that ran `/create-dev-loop`, i.e. the owner of the skill-tracker repo created in Step 6. Not necessarily the same as `GITHUB_OWNER` when the target project lives under an org. |
| `DEFAULT_BRANCH` | `gh repo view --json defaultBranchRef -q .defaultBranchRef.name` |
| `BRANCH_PREFIX` | From CONTRIBUTING.md, or `feature` if not specified |
| `COMPILE_CMD` | Fastest command that catches syntax/import errors without running tests. For Maven: `MVN=$([ -f ./mvnw ] && echo ./mvnw \|\| echo mvn) && $MVN compile`. **If the repo has no build system** (docs-only, config-only, single-template repos), substitute the cheapest mechanical consistency check that would actually catch breakage — e.g. a `grep` for unresolved placeholders, `jq -e . <config>`, a YAML/schema lint. If no such check exists, substitute a shell comment naming what gates correctness instead (e.g. `# no build system — see the validation steps below`) so the fence stays valid bash. |
| `COMPILE_CMD` | Fastest command that catches syntax/import errors without running tests. For Maven: `MVN=$([ -f ./mvnw ] && echo ./mvnw \|\| echo mvn) && $MVN compile`. **If the repo has no build system** (docs-only, config-only, single-template repos), substitute the cheapest mechanical consistency check that would actually catch breakage — e.g. a `grep` for unresolved placeholders, `jq -e . <config>`, a YAML/schema lint. If no such check exists, substitute a shell comment naming what gates correctness instead (e.g. `# no build system — see the validation steps below`) so the fence stays valid bash. Substitute the same text in both places it appears (Phase 3 verify block, Phase 8 rebase fence). |
| `TEST_CMD` | Full test suite command from CI workflow or README. For Maven: `$MVN test` (reuse the `MVN` variable set above). **If the repo has no automated test suite** — confirm it (`CLAUDE.md` says so, no test directory, no CI test job); don't assume — substitute a shell comment pointing at the project's manual validation steps, e.g. `# re-run the Phase 3 manual validation checklist`, naming the same check `EXTERNAL_SIGNAL_CMD` renders. Substitute the same text everywhere `{{TEST_CMD}}` appears (Phase 3 verify block, Phase 8 rebase fence). |
| `LINT_CMD` | Linter/formatter command if present in CI; omit section if absent |
| `LINT_CMD` | Linter/formatter command if present in CI; omit section if absent. Appears in both the Phase 3 verify block and the Phase 8 rebase fence — include or omit it in both, so the post-rebase verification covers the same tasks the required checks do |
| `VALIDATION_NOTE` | Only when `COMPILE_CMD` or `TEST_CMD` is a shell comment rather than a real command: a one-line parenthetical placed directly under the Phase 3 verify fence, stating what actually gates correctness — e.g. `(no automated test command or linter exists for this repo — the manual validation checklist in Phase 4 is the real gate)`. Omit the conditional block entirely when both are real commands. |
| `REVALIDATE_INSTRUCTION` | Imperative clause opening the Phase 6 re-verification sentence (capitalized, no trailing period; it is followed by "after every fix"). With a test suite: `Run \`$MVN test\``. Without one: `Re-run the Phase 3 manual validation checklist`. Must name the same check as `TEST_CMD` and `EXTERNAL_SIGNAL_CMD`. |
| `EXTERNAL_SIGNAL_LABEL` | Short label for the anchor used in Phase 4 self-review. `CI` when the repo has CI workflows; `manual validation` when the project uses a fixture or test-command checklist (e.g. a doc-only repo); otherwise a project-specific phrase like `snapshot regression suite`. |
Expand All @@ -624,7 +633,7 @@ Use findings from Step 2 to substitute each `{{placeholder}}`. The table below c
| `TEMPLATE_VERSION` | Short commit SHA of the `create-dev-loop` repo at generation time. Capture with `git -C <path-to-create-dev-loop> rev-parse --short HEAD`. Becomes part of an HTML comment at the top of the generated skill so future cycles can detect template drift. |
| `GENERATED_AT` | ISO-8601 UTC timestamp at generation time. Capture with `date -u +%Y-%m-%dT%H:%M:%SZ`. Pairs with `TEMPLATE_VERSION` in the HTML-comment header. |

**When the repo has no build system or no automated test suite**, `COMPILE_CMD`, `TEST_CMD`, `REVALIDATE_INSTRUCTION`, `VALIDATION_NOTE`, and `EXTERNAL_SIGNAL_CMD` must all name the *same* check — whatever the project actually gates correctness on. Also reword the Edge cases entries that name tests — "Tests fail during implementation", "Tests fail after addressing a comment", "A test fails intermittently", and the "re-run tests" step of "Branch is behind main or has a merge conflict at Phase 8" — so they refer to that check rather than a test suite; keep the entry order, wording of the surrounding rules, and phase numbers unchanged.
**When the repo has no build system or no automated test suite**, `COMPILE_CMD`, `TEST_CMD`, `REVALIDATE_INSTRUCTION`, `VALIDATION_NOTE`, and `EXTERNAL_SIGNAL_CMD` must all name the *same* check — whatever the project actually gates correctness on. Also reword the Edge cases entries that name tests — "Tests fail during implementation", "Tests fail after addressing a comment", "A test fails intermittently", and the post-rebase verification step of "Branch is behind main or has a merge conflict at Phase 8" — so they refer to that check rather than a test suite; keep the entry order, wording of the surrounding rules, and phase numbers unchanged.

For `SCAN_CHECKLIST`, always include these universal items plus any repo-specific ones:
- Missing tests on new public methods
Expand Down
Loading