Repository navigation
fix(ci): stop serializing IPR checks behind signature writes - #8224
Merged
Merged
Conversation
The required "IPR Policy / Signature" status came from a workflow with a repo-wide concurrency group. Every PR push's check-only run, and every PR comment, waited in that one group. Each run held the slot while it also waited for an ubuntu-latest runner. On 2026-10-15 that queue completed one check every 20-45 minutes (slot waits up to 135 min) and blocked every merge in the repo. Split the workflow (and the cross-repo callable) into two jobs: - check (pull_request_target): reads the ledger and posts the status. It never writes. It uses a per-PR group with cancel-in-progress. - record (sign-phrase comments only): writes the ledger and keeps the repo-wide adcp-ipr-signature-write group with queue: max. Because the group is on this job, unrelated comments no longer queue in it. The check and the write no longer share a group, so close the race in the script. The check re-reads origin/main after it posts pending. The write re-marks the PR's latest head after its push. Every interleaving ends with success for a signed author. A failed re-read leaves the status pending (fail closed). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Ladon verdict: Escalate to human review
Escalate — gated paths require human/CODEOWNERS review.
This PR modifies workflow files under .github/workflows/**, which sit behind a hard, deterministic approval gate (gated_paths: true). The required review status is REVIEW_REQUIRED, not APPROVED, so the gate is not satisfied and the PR cannot be auto-approved.
Escalation reasons
.github/workflows/ipr-agreement.yml(modified) and.github/workflows/ipr-check-callable.yml(modified) match the gated glob.github/workflows/**; a human/CODEOWNERS approval is required before merge.
The reviewer found no blocking or medium findings — the change itself is a clean operational fix splitting the IPR workflow into a read-only check lane and a serialized record lane, with status-race and fail-closed invariants verified and new test coverage added. Once review_decision becomes APPROVED, this PR should fall through to a normal approve.
Why human review
- .github/workflows/ipr-agreement.yml (modified) matches gated path .github/workflows/** — human/CODEOWNERS review required
- .github/workflows/ipr-check-callable.yml (modified) matches gated path .github/workflows/** — human/CODEOWNERS review required
- gated_paths is true and review_decision is REVIEW_REQUIRED (not APPROVED)
- This PR touches a path under a hard, non-overridable approval gate (.github/workflows/ipr-agreement.yml (modified) matches
.github/workflows/**; .github/workflows/ipr-check-callable.yml (modified) matches.github/workflows/**) and the current GitHub review decision is 'REVIEW_REQUIRED', not APPROVED. This is a hard gate enforced in code — Ladon cannot auto-approve until a human/CODEOWNERS approval is recorded, regardless of how clean the diff is.
bokelley
added a commit
that referenced
this pull request
Oct 17, 2026
… races it (#8256) Several AAO repos call ipr-check-callable.yml, and each writes the shared signatures/ipr-signatures.json on adcp's main. Actions concurrency groups are per repo, so the write group does not serialize writes across repos. When two signatures were recorded at the same time, the second push was rejected and the script ran `git pull --rebase`. Both commits append to the end of the same JSON array, so the rebase conflicted and the record job failed. On a rejected push, the script now resets to the fresh remote head and applies the signature to that ledger again instead of replaying its commit. If the author is already recorded there (for example, they signed on another repo's PR at the same moment), it skips the write. It retries up to 5 times with exponential backoff and jitter, then fails with the last push error. The ledger format and the post-write status behaviour from #8224 are unchanged: the commented head and the PR's latest head are still marked success after the write. Tests run real git against a local bare remote. They cover a concurrent append from another repo, the same author recorded concurrently, retry exhaustion with backoff, and no success status when recording fails. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
IPR Policy / Signatureis a required status onmain. It came fromipr-agreement.yml, which used a repo-wide concurrency group (adcp-ipr-signature-write,queue: max). That one group held:pull_request_targetpath incheck-and-record.mjsonly reads.if, and the group sat at workflow level, so comments that were going to be skipped still waited their turn.Each queued run got the slot first and then waited for an
ubuntu-latestrunner while still holding it. The org's hosted runners were pinned at the 60-job cap, so the IPR queue drained one runner-wait at a time.Job timestamps from 2026-10-15:
Change
ipr-agreement.ymland the cross-repoipr-check-callable.ymlare now split into two jobs, each with its own concurrency:checkpull_request_targetadcp-ipr-check-pr-<number>,cancel-in-progress: true. Unrelated PRs never wait on each other, and a newer push cancels a superseded check.recordadcp-ipr-signature-write,cancel-in-progress: false,queue: max(same as before).Because the group is now on
recordrather than the workflow, a comment that fails the sign-phraseifis skipped instead of queuing in the write group.Keeping the record correct without a shared group
checkandrecordno longer serialize against each other, so a signature can land between a check's checkout and its status write.scripts/ipr/check-and-record.mjsnow closes that race:pending, then re-readssignatures/ipr-signatures.jsonfrom the tip oforigin/main(git fetch --depth=1+git show FETCH_HEAD:). If the author has signed by then, it postssuccess.success.Either the check's re-read happens after the signature push, and it sees the signature, or it happens before. In the second case the write's later head re-read sees the new head and overwrites
pending. Either way a signed author's latest head ends atsuccess.A failed re-read returns
nulland the status stayspending(fail closed). No unsigned author can getsuccess.The ledger write path itself is unchanged: same file format, same commit, same rebase-retry push.
Other changes:
main()only when invoked directly, comparing real paths. A test checks that a direct invocation still runsmain(), so this can't become a silent no-op.governance/ipr-bot-setup.mdwas stale: it said the adcp flow usesGITHUB_TOKENwith "no App involvement". It now describes the token and the two-job split.Downstream impact
The callable gets the same split, so downstream repos' check-run names become
check / checkandcheck / record. The required context is the commit statusIPR Policy / Signature, which is unchanged. No caller changes are needed.Not changed
Verification
npx vitest run --dir tests/ tests/ipr-check-and-record.test.mjs tests/ipr-ledger-remote.test.mjs: 35 passed. Coverage:pendingthensuccess), re-read failure (stayspending), bot author, never writesreadLatest()smoke test against a real git remote, with a signature pushed after the clone: checkout shows[], latest shows[101].actionlint1.7.12: clean exceptunexpected key "queue". That key was already onmain(actionlint doesn't know the 2026-05-07queueoption yet), and GitHub's docs listqueueforjobs.<job_id>.concurrency. I also confirmed GitHub accepts it live: a throwaway branch with the same two-job shape (job-levelqueue: max+ per-PR cancel group) ran withrecordsucceeding andcheckskipped (run 38782529231). The branch has been deleted.After merge
recordshows as skipped with no wait.checkruns for several PRs at the same time.Companion PR: #8225 (runner-demand reductions: draft-skip for storyboards, cancel superseded scans).
🤖 Generated with Claude Code