You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Phase 1's carried-over-PR handling in the generated-skill template branches on
whether the open PR's checks are green:
If the PR is still valid ({{EXTERNAL_SIGNAL_LABEL}} green, no conflicts), first
confirm a Phase 4 self-review was actually posted …
and Phase 4 step 1 anchors the self-review rubric on gh pr checks <number> --watch.
Both treat the anchor as binary: green, or red-and-must-be-fixed. GitHub has a
third state that neither covers.
When a commit is pushed to a PR branch by an account whose workflow runs require
manual approval — a first-time contributor, or an agent account such as copilot-swe-agent — the run is created with conclusion action_required and
executes nothing (0s duration). The workflow never ran; no job produced a result.
The failure mode is that this does not surface as a red check. It surfaces as an absence:
gh pr checks <number> prints no checks reported on the '<branch>' branch
gh pr checks <number> --watch returns that same line immediately rather than
waiting, so a step written as "wait for the anchor to go green" returns at once
having established nothing
gh run list --branch <branch> still shows a success run — but on an older
head SHA, so a glance at the run history reads as a healthy branch
Nothing in Phase 1 or Phase 4 distinguishes "the anchor has not been run" from
"the anchor has no jobs configured for this branch". An unattended cycle can
therefore score the Phase 4 {{EXTERNAL_SIGNAL_LABEL}} rubric item off a run that
never executed, or read --watch returning instantly as confirmation.
Observed instance
Stephenson-Software/create-dev-loop PR #119. A merge-conflict resolution was
pushed to the branch by copilot-swe-agent as 26b149f; run 32547957642 on
that SHA completed action_required in 0s. gh pr checks 119 reported no checks
reported, while gh run list showed a success CI run two commits back on 9d7af16. The condition was resolved by pushing a further commit from an
already-permitted account, which retriggered the workflow with no approval gate
and produced a real green on the new head.
Suggested direction
Two changes, both small:
Phase 1 carried-over-PR handling — add a third branch alongside the
existing green / stale-or-conflicted cases and the TEMP: mid-revert check:
an anchor run in a not-executed state (action_required, waiting, or stale) on the PR head is neither green nor red, and the PR is not ready for
the Phase 5 jump. Where the loop's own account can push, adding a commit or
re-running the workflow is enough to clear it; where it cannot, this is a
hand-off naming the approval gate.
Phase 4 step 1 — state that the anchor must be confirmed against the PR head SHA, and that no checks reported is not a pass. This parallels
the rule already added by PR Require unique scratch filenames and define the structurally-red-anchor path #121 for a build tool that exits 0 having
executed zero tests: a status that reports success without having run anything
verifies nothing. The same sentence covers both, and the existing
"Confirm the anchor actually executed tests" paragraph at Phase 3 is the
natural place to point at.
A useful check for either: compare gh run list --branch <branch> --json headSha,status,conclusion
against the head SHA from gh pr view <number> --json headRefOid, rather than
trusting the aggregate status line.
Research grounding
RESEARCH.md §1 (Self-critique without an external signal is unreliable,
confidence: high). This is the same failure the finding's PR #121 Implementations
entry already records — an anchor that carries no information being read as one
that does — reached by a third route. The existing entry covers a job that has
never succeeded on the base branch and a build tool that reports success having
executed zero tests; a run that was created but never permitted to execute is a
further instance, and would extend that entry rather than needing a new finding.
Problem
Phase 1's carried-over-PR handling in the generated-skill template branches on
whether the open PR's checks are green:
and Phase 4 step 1 anchors the self-review rubric on
gh pr checks <number> --watch.Both treat the anchor as binary: green, or red-and-must-be-fixed. GitHub has a
third state that neither covers.
When a commit is pushed to a PR branch by an account whose workflow runs require
manual approval — a first-time contributor, or an agent account such as
copilot-swe-agent— the run is created with conclusionaction_requiredandexecutes nothing (0s duration). The workflow never ran; no job produced a result.
The failure mode is that this does not surface as a red check. It surfaces as an
absence:
gh pr checks <number>printsno checks reported on the '<branch>' branchgh pr checks <number> --watchreturns that same line immediately rather thanwaiting, so a step written as "wait for the anchor to go green" returns at once
having established nothing
gh run list --branch <branch>still shows asuccessrun — but on an olderhead SHA, so a glance at the run history reads as a healthy branch
Nothing in Phase 1 or Phase 4 distinguishes "the anchor has not been run" from
"the anchor has no jobs configured for this branch". An unattended cycle can
therefore score the Phase 4
{{EXTERNAL_SIGNAL_LABEL}}rubric item off a run thatnever executed, or read
--watchreturning instantly as confirmation.Observed instance
Stephenson-Software/create-dev-loopPR #119. A merge-conflict resolution waspushed to the branch by
copilot-swe-agentas26b149f; run32547957642onthat SHA completed
action_requiredin 0s.gh pr checks 119reported no checksreported, while
gh run listshowed asuccessCI run two commits back on9d7af16. The condition was resolved by pushing a further commit from analready-permitted account, which retriggered the workflow with no approval gate
and produced a real green on the new head.
Suggested direction
Two changes, both small:
Phase 1 carried-over-PR handling — add a third branch alongside the
existing green / stale-or-conflicted cases and the
TEMP:mid-revert check:an anchor run in a not-executed state (
action_required,waiting, orstale) on the PR head is neither green nor red, and the PR is not ready forthe Phase 5 jump. Where the loop's own account can push, adding a commit or
re-running the workflow is enough to clear it; where it cannot, this is a
hand-off naming the approval gate.
Phase 4 step 1 — state that the anchor must be confirmed against the
PR head SHA, and that
no checks reportedis not a pass. This parallelsthe rule already added by PR Require unique scratch filenames and define the structurally-red-anchor path #121 for a build tool that exits 0 having
executed zero tests: a status that reports success without having run anything
verifies nothing. The same sentence covers both, and the existing
"Confirm the anchor actually executed tests" paragraph at Phase 3 is the
natural place to point at.
A useful check for either: compare
gh run list --branch <branch> --json headSha,status,conclusionagainst the head SHA from
gh pr view <number> --json headRefOid, rather thantrusting the aggregate status line.
Research grounding
RESEARCH.md §1 (Self-critique without an external signal is unreliable,
confidence: high). This is the same failure the finding's PR #121 Implementations
entry already records — an anchor that carries no information being read as one
that does — reached by a third route. The existing entry covers a job that has
never succeeded on the base branch and a build tool that reports success having
executed zero tests; a run that was created but never permitted to execute is a
further instance, and would extend that entry rather than needing a new finding.
This issue body was drafted during a Gardener session (https://github.com/Stephenson-Software/gardener).
drafted by Claude on behalf of Daniel Stephenson