Skip to content

Fork review gate: enforce the owner's rules (#82, #99, #100) - #943

Open
scuttletoby wants to merge 1 commit into
mainfrom
fork-ci/review-gate-enforce
Open

scuttletoby wants to merge 1 commit into
mainfrom
fork-ci/review-gate-enforce

Conversation

@scuttletoby

@scuttletoby scuttletoby commented Nov 22, 2026 •

Copy link
Copy Markdown
Owner

Part of #82, part of #99, part of #100 (see "Why not Fixes" below).

What

The review block of .github/fork-pr-queue.json goes from shadow to enforce, for exactly the rules the owner decided on 2026-11-14:

One rule change, found by the dry run. .github/** made two bookkeeping lists critical: .github/fork-seams.json (seam announcements) and .github/fork-ci-tests.txt (the test list). Most fork pull requests add a line to one of them, so plain app changes needed only the gate lenses and no code lens. That is wrong under #82, which covers "workflow files or gate settings". The glob dialect has no negation, and the order check refuses a normal rule above a critical one. So the gate rule now names the .github files one by one: .github/*/*/** (every subfolder: workflows, actions, scripts, issue templates, …), *.md, *.td, *.yml, *.yaml and each JSON or text setting file. A test walks .github and fails when a new top-level file is in neither the gate rule nor the two-entry bookkeeping list. So a new gate file cannot stay normal unnoticed. (An earlier .github/*/** draft was caught by that test: ** matches zero folders.)

What enforce does on the fork today: nothing yet. No fork workflow calls the head gate. There is no gate App (#81) and no head-gate environment, and the main ruleset does not require head_check. The switch puts the config in its decided end state, so the gate enforces it from its first run once those parts exist. Until then, lens discipline on fork pull requests stays a process rule.

Dry run: the newest 22 merged fork pull requests

How it was run: the gate's own head_check.decide(), with this branch's config forced to each mode. Base = the squash commit's first parent, head = the PR head as merged. Comments and reviews came from gh pr view / GraphQL (read-only, about 3 calls per PR). "Old rule" = .github/**.

PR Paths judged (old rule → new) Enforce (new rule) Why Right?
#939 critical (seam-check script) refuse merge-path reported a major: (the "Fixes" wording) and no later report cleared it (#99) yes
#938 normal pass yes
#922 critical (.github/scripts) refuse code r2 says NOT SAFE (read as a blocker) and the fix was pushed with no new report (#99). Shadow fails it too: the head is not the read change yes
#920 critical (release config + script) pass yes
#917 normal pass yes
#912 normal pass yes
#909 critical → normal (seam list) refuse the head is not the change read (a push after the review); shadow fails it too yes
#907 critical → normal (seam list only) pass (old rule: refuse) old rule: no merge-path/state on a seam-announcement PR old refusal wrong, fixed by the rule
#874 critical (fork-ci/**) refuse only merge-path read it; state missing (#82) yes
#847 critical → normal (test list only) refuse lens b says NOT SAFE; the override came from the owner's login, which does not count yet (#78) yes
#845 normal refuse lenses named a/b (not config names); round 3 without approach (#100) yes
#843 normal refuse only state; code missing yes
#840 critical → normal refuse old free-format ## Lens A comments (no lens header) yes
#835 critical → normal refuse old free-format lens comments yes
#833 critical (fork-release.yml) refuse no lens comment at all yes
#832 critical → normal (seam list) refuse no lens comment at all; a normal change needs code yes
#827 critical → normal (seam list) refuse lenses security, state; the config names code yes
#826 critical → normal (seam list) refuse the head is not the change read; shadow fails it too yes
#825 normal refuse old free-format lens comments yes
#824 normal refuse old free-format lens comments yes
#823 normal refuse code r3 without approach (#100) yes
#820 critical → normal refuse old free-format lens comments yes

Under the old rule, 8 more PRs needed merge-path+state instead of code. Under the new rule, every refusal has a real cause: the comments do not follow the lens format (the review skill from #845 came later), a lens is missing, a finding is still open, a round is missing, or a push came after the review. Shadow (today's rules) already fails 9 of the 22 (#922 #909 #840 #835 #826 #825 #824 #823 #820). They merged because the fork runs no gate.

Tests

  • review_judge.test.py ForkConfig:
    • the config parses in enforce with 3/4 rounds, findings true, no override author and no order warning;
    • workflow, gate, config and lens files are critical with two lenses;
    • every top-level .github entry is critical except the two bookkeeping lists, and a new unsorted file fails;
    • an app change plus a seam or test-list line needs code.
  • head_check.test.py ForkConfigRules (+ Bare), on a base holding the fork's real config and lens prompts:

State question

  • The switch lands while PRs are open. Nothing reads the fork's config in CI today, so no open PR gets stuck. Once a gate caller exists, the gate judges each open PR at its next run (dispatch the backfill right after). A PR with old free-format comments (## Lens A) or old lens names (a, security) then fails. It needs new reports in the lens: <config name> · reviewer: … · read at: <head sha> format with blocker:/major: lines (the fork-kit/review skill writes them), on its current head. A PR with an open finding needs a new clean report from that lens. A PR past round 4 needs a person's override, which needs Owner question: may your own GitHub login leave the agent machine? #78 first, or it is closed.
  • An older release reads the config. No caller pins any ci/vN against this file. Releases ci/v15–ci/v20 parse this config (they know rounds and findings as keys, and it has no override_authors) and enforce the path rules (Owner question: extra reviewer for workflow and gate changes? #82). They do not judge findings or rounds, so Owner question: should the merge gate block a PR when a reviewer reported a blocker or major finding? #99 and Owner question: cap review rounds at 4, then a person decides (override or close)? #100 hold only from ci/v21 (latest ci/v22). A caller must pin ci/v21 or later for all three rules.
  • It runs again / stops half-way. A config change judges nothing by itself. A run that stops leaves the previous verdict, and the next event or a backfill judges again.
  • Rollback to shadow. One PR sets "mode": "shadow". While the base is in enforce, that PR is judged by the enforce rules, so it needs merge-path and state (it changes the config). After it merges, the rows only report, as the shadow test shows.

Why not "Fixes"

Each issue is covered as config plus tests. On the fork, though, nothing enforces it until a gate caller, the shared gate App (#81) and a ruleset entry for head_check exist. The "a person's override" half of #99 and #100 waits on #78 (override_authors stays empty while agents post from the owner's login).

Docs

General vs project

  • General: nothing in the gate code changed. The README note and the drift-test pattern (walk the folder, every entry sorted) apply to any project.
  • Project (the fork as a user of its own kit): .github/fork-pr-queue.json and the tests that pin what it decides for the fork.

Checks run

  • review_judge.test.py: 51 OK
  • review_config.test.py: 51 OK
  • head_check.test.py: 258 OK, 1 skipped
  • fork-contracts.test.py: 716 ok
  • seam check against origin/main: only the pinned seams
  • vp fmt --check on the touched JSON and Markdown: clean
  • actionlint: not run, because no workflow file changed

🤖 Generated with Claude Code

The review block of .github/fork-pr-queue.json moves from shadow to enforce:
workflow and gate-setting files need the merge-path and state lenses (#82),
an open blocker or major fails the head (#99), the approach lens joins from
round 3 and round 5 needs a person (#100). No override_authors yet (#78).

A dry run over the newest merged pull requests showed the old .github/**
rule made seam announcements and test-list lines critical, so plain app
changes needed the gate lenses and no code lens. The gate rule now names the
.github files one by one; the seam list and the test list stay normal, and a
test fails when a new .github file is in neither list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: merge-path · reviewer: lens agent (Opus 5.5) · read at: 0795fae
verdict: not safe to merge
covered: the diff (.github/fork-pr-queue.json, both test files, the pr-queue README); the owner decisions on #82, #99, #100 and #78; review_judge.judge and review_glob (dialect: * stays in one segment and matches dot-names, case-sensitive, ** takes zero or more segments); renames and deletes (both paths are judged); every reader of the config (no fork workflow calls fork-head-gate.yml or fork-pr-queue.yml; the main ruleset requires fork-ci-result, "No database schema changes" and "Outside pull request", not head_check; the SkyRPG queue reads its own config file); the test list and seam list now marked normal.
checks: ran review_judge.test.py (51 OK) and head_check.test.py (258 OK, 1 skipped). Added an unsorted .github/fork-new-gate.json: the new sorting test fails with the right message, and the "Fork action and fleet walls tests" job that runs it has no path filter, so it runs on every same-repo pull request. Probe on the fork's real config: an app change plus a one-line edit to .github/SECURITY.md is judged critical with lenses merge-path and state only; a code lens report with an open major that was already posted still fails the head.
state: runs again: the config judges nothing by itself and the same head gets the same verdict. Stopped half-way: the previous verdict stays. Existing state: no gate caller and no ruleset entry, so open pull requests and main's merges and releases are not blocked now. Rollback: the tests pin enforce, so a rollback pull request must also change two tests (minor 2).
blocker: none
major: a change that touches any gate file no longer needs the code lens for the rest of its diff, so the config is looser than #82 ("one extra review"). Example: an app change plus one line in .github/SECURITY.md passes with merge-path and state alone; nobody has to read the app code for correctness. The judge takes lenses only from the highest weight hit (fork-ci/actions/pr-queue/review_judge.py:172), and the gate rule lists merge-path and state (.github/fork-pr-queue.json:41). The pull request uses this same argument ("the gate lenses alone never read app code") to make the seam list and test list normal, but the general case stays open, and review_judge.test.py pins it for app plus seam list plus ci.yml. One possible fix in general code: require the lenses of every rule that matched, which gives code plus merge-path plus state on mixed changes and leaves pure gate or pure app changes as they are. Or the owner decides that the gate lenses replace code on mixed changes, and the README says so.
minor: 1. fork-kit/** stays normal (.github/fork-pr-queue.json:39 only adds fork-ci/), and so do fork-kit/gh-apps (where the gate App's key is kept, from the gate-trust design) and fork-kit/review (the lens-format checker). Both are arguably "gate settings" under #82. The old rule had the same gap, so this is a follow-up, not caused by this pull request. 2. Rolling back to shadow is not "one PR sets mode shadow": head_check.test.py:1589 and review_judge.test.py:451 assert enforce and fail until they change too (that change is critical anyway). The README "Roll back" step and the PR body's State question should say so. 3. I checked the newly normal files: changing the test list or the seam list cannot make fork-ci-result pass for a gate test that did not run. An unlisted test, a test that was listed and then deleted, a missing run record, or a run by file that does not name the test all fail, and a may skip line lives in the test file under fork-ci/ or .github/scripts (critical). So their normal weight is fine.

@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: state · reviewer: state lens agent (Opus 5.5) · read at: 0795fae
covered: the diff (.github/fork-pr-queue.json, the README "Turning it on" step 5, the new tests in review_judge.test.py and head_check.test.py); the owner decisions on issues 82, 99, 100 and 78; review_config.py, review_glob.py, head_check.py and README sections "Findings and rounds", "Lens comments" and "Turning it on" at this head; review_config.py and head_check.py at the tags ci/v14 to ci/v22.
checks: ran review_judge.test.py (51 OK) and head_check.test.py (258 OK, 1 skipped) at this head; mutation: removed .github/fork-upstream.json from the gate rule, and test_every_github_file_is_critical_but_the_two_bookkeeping_lists failed naming that file (restored after); read the glob dialect: .github/*/*/** matches every file in a subfolder (** takes zero segments), and the drift test judges a subfolder as <folder>/x; checked ci/v15 to ci/v20 head_check.py: no mention of findings or rounds.
state: nothing reads this config in CI today, so merging changes no verdict. Once a gate caller exists: an older release (ci/v15 to ci/v20) parses the config and enforces the path rules (82), but silently does not judge findings or rounds (99, 100): it fails open, with no summary row saying so (minor 1). ci/v14 or older refuses the block (fail closed, the README owner step). Runs again: a config change judges nothing; a re-run of the gate on the same head gives the same answer. Stopped half-way: the previous verdict stays until the next event or a backfill. Open pull requests with old free-format lens comments fail at their next gate run; their ways out are delete, not a lens:, or an override (an override from the owner's login still covers a read, only clearing findings and the cap needs a listed person). A review older than a push fails ("not the change read"), unless an override for the new head vouches for it (minor 3). Rollback to shadow: the rollback pull request changes the config, so it needs clean merge-path and state reports within 4 rounds; if the gate itself is broken so that every head fails, that pull request fails too and only the README owner step (a direct push of the revert) gets out. Round counting: edited and hidden lens comments still count, so editing cannot lower the round; deleting can (minor 2).
blocker: none
major: none
minor: 1. Older releases fail open on 99 and 100. ci/v15 to ci/v20 know the keys rounds and findings (KEYS in their review_config.py) but their head_check.py never reads them, so a caller pinned to one of them, or a rollback that pins back to one (README step 6, fork-ci/actions/pr-queue/README.md:419), runs this "enforce" config with no findings or round judging and no row saying so. The PR body says "a caller must pin ci/v21 or later", but the README paragraph this PR adds (README.md:400-417) and step 6 do not: step 6 only warns about releases that refuse a key. Cheapest fix: write "override_authors": [] in the fork's config, which ci/v20 and older refuse as an unknown key, so they fail closed instead of open (the test test_it_is_in_enforce asserts the key is absent and would change). Or at least add the ci/v21 floor to step 6 and to the fork paragraph.
minor: 2. The round cap (100) does not hold against lens authors on the fork. Rounds count from the lens comments that still exist; a deleted comment is gone and only lens authors can delete one (README.md:269). On the fork every lens author is the owner's login, which every agent posts from (78), so an agent can delete old round comments, or close the pull request and open a new one, and start again at round 1. This is the gate's design, not this diff, but the PR body's "past round 4 only a person's override passes" is true only once 78 is done. Worth one sentence in the README fork paragraph, or on issue 100.
minor: 3. A push after the review: the head fails, unless an override for the new head (prefix override for head, association OWNER) vouches for it. That override does not need a listed person: with 78 open, any agent's override lets a pushed head pass on the old reads, while open findings still block (head_check.py:213 via lens.Review.reviewed_head). By design (the nit-only quick check), and the README says it, but it means "only a person decides" in this PR covers findings and the cap, not unread pushes.
minor: 4. No test runs the rollback pull request itself: a base in enforce and a head that sets "mode": "shadow", which should need merge-path and state. It follows from the config being a built-in protected path and from test_workflow_queue_gate_config_and_lens_files_are_critical (the config is in its list), so this is only a gap in the head_check.test.py ForkConfigRules class, which covers the base-in-shadow side only.
verdict: safe to merge

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant