Skip to content

An approval-gated CI run reads as 'no checks reported', which is neither the green nor the red Phases 1 and 4 branch on #129

Description

@dmccoystephenson

Problem

Phase 1's carried-over-PR handling in the generated-skill template branches on
whether the open PR's checks are green:

If the PR is still valid ({{EXTERNAL_SIGNAL_LABEL}} green, no conflicts), first
confirm a Phase 4 self-review was actually posted …

and Phase 4 step 1 anchors the self-review rubric on gh pr checks <number> --watch.
Both treat the anchor as binary: green, or red-and-must-be-fixed. GitHub has a
third state that neither covers.

When a commit is pushed to a PR branch by an account whose workflow runs require
manual approval — a first-time contributor, or an agent account such as
copilot-swe-agent — the run is created with conclusion action_required and
executes nothing (0s duration). The workflow never ran; no job produced a result.
The failure mode is that this does not surface as a red check. It surfaces as an
absence:

  • gh pr checks <number> prints no checks reported on the '<branch>' branch
  • gh pr checks <number> --watch returns that same line immediately rather than
    waiting, so a step written as "wait for the anchor to go green" returns at once
    having established nothing
  • gh run list --branch <branch> still shows a success run — but on an older
    head SHA, so a glance at the run history reads as a healthy branch

Nothing in Phase 1 or Phase 4 distinguishes "the anchor has not been run" from
"the anchor has no jobs configured for this branch". An unattended cycle can
therefore score the Phase 4 {{EXTERNAL_SIGNAL_LABEL}} rubric item off a run that
never executed, or read --watch returning instantly as confirmation.

Observed instance

Stephenson-Software/create-dev-loop PR #119. A merge-conflict resolution was
pushed to the branch by copilot-swe-agent as 26b149f; run 32547957642 on
that SHA completed action_required in 0s. gh pr checks 119 reported no checks
reported, while gh run list showed a success CI run two commits back on
9d7af16. The condition was resolved by pushing a further commit from an
already-permitted account, which retriggered the workflow with no approval gate
and produced a real green on the new head.

Suggested direction

Two changes, both small:

  1. Phase 1 carried-over-PR handling — add a third branch alongside the
    existing green / stale-or-conflicted cases and the TEMP: mid-revert check:
    an anchor run in a not-executed state (action_required, waiting, or
    stale) on the PR head is neither green nor red, and the PR is not ready for
    the Phase 5 jump. Where the loop's own account can push, adding a commit or
    re-running the workflow is enough to clear it; where it cannot, this is a
    hand-off naming the approval gate.

  2. Phase 4 step 1 — state that the anchor must be confirmed against the
    PR head SHA, and that no checks reported is not a pass. This parallels
    the rule already added by PR Require unique scratch filenames and define the structurally-red-anchor path #121 for a build tool that exits 0 having
    executed zero tests: a status that reports success without having run anything
    verifies nothing. The same sentence covers both, and the existing
    "Confirm the anchor actually executed tests" paragraph at Phase 3 is the
    natural place to point at.

A useful check for either: compare gh run list --branch <branch> --json headSha,status,conclusion
against the head SHA from gh pr view <number> --json headRefOid, rather than
trusting the aggregate status line.

Research grounding

RESEARCH.md §1 (Self-critique without an external signal is unreliable,
confidence: high). This is the same failure the finding's PR #121 Implementations
entry already records — an anchor that carries no information being read as one
that does — reached by a third route. The existing entry covers a job that has
never succeeded on the base branch and a build tool that reports success having
executed zero tests; a run that was created but never permitted to execute is a
further instance, and would extend that entry rather than needing a new finding.

This issue body was drafted during a Gardener session (https://github.com/Stephenson-Software/gardener).


drafted by Claude on behalf of Daniel Stephenson

Metadata

Metadata

Assignees

No one assigned

    Labels

    template-ruleShould be promoted into create-dev-loop.md

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions