Skip to content

fix(ci): stop serializing IPR checks behind signature writes - #8224

Merged
bokelley merged 1 commit into
mainfrom
fix/ci-ipr-check-without-write-lock
Oct 15, 2026
Merged

bokelley merged 1 commit into
mainfrom
fix/ci-ipr-check-without-write-lock

Conversation

@bokelley

@bokelley bokelley commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem

IPR Policy / Signature is a required status on main. It came from ipr-agreement.yml, which used a repo-wide concurrency group (adcp-ipr-signature-write, queue: max). That one group held:

  • every PR push's check-only run. Those runs never write the ledger, because the pull_request_target path in check-and-record.mjs only reads.
  • every PR comment, including bot comments. The sign-phrase filter was a job-level if, and the group sat at workflow level, so comments that were going to be skipped still waited their turn.

Each queued run got the slot first and then waited for an ubuntu-latest runner while still holding it. The org's hosted runners were pinned at the 60-job cap, so the IPR queue drained one runner-wait at a time.

Job timestamps from 2026-10-15:

run created job created (got slot) job started (got runner) slot wait runner wait exec
09:39:39 09:44:12 09:54:11 4.5m 10.0m 84s
09:39:41 09:55:35 10:14:33 15.9m 19.0m 104s
09:40:59 10:16:18 10:58:55 35.3m 42.6m 93s
09:43:55 11:00:29 11:23:40 76.6m 23.2m 103s
09:44:42 (#8208) 11:39:33 11:44:10 114.8m 4.6m 95s
10:13:47 12:25:39 12:25:42 131.9m 0.1m 207s
  • Each check runs for about 90 seconds, but slot waits reached about 135 minutes.
  • 18 skipped comment runs also sat in the group for up to about 2 hours.
  • Every PR merge was blocked for that time.

Change

ipr-agreement.yml and the cross-repo ipr-check-callable.yml are now split into two jobs, each with its own concurrency:

job trigger concurrency
check pull_request_target adcp-ipr-check-pr-<number>, cancel-in-progress: true. Unrelated PRs never wait on each other, and a newer push cancels a superseded check.
record comments containing the sign phrase adcp-ipr-signature-write, cancel-in-progress: false, queue: max (same as before).

Because the group is now on record rather than the workflow, a comment that fails the sign-phrase if is skipped instead of queuing in the write group.

Keeping the record correct without a shared group

check and record no longer serialize against each other, so a signature can land between a check's checkout and its status write. scripts/ipr/check-and-record.mjs now closes that race:

  • Check path: if the checkout shows the author as unsigned, it posts pending, then re-reads signatures/ipr-signatures.json from the tip of origin/main (git fetch --depth=1 + git show FETCH_HEAD:). If the author has signed by then, it posts success.
  • Write path: after pushing the signature, it re-reads the PR head. It marks both the commented head and the current head success.

Either the check's re-read happens after the signature push, and it sees the signature, or it happens before. In the second case the write's later head re-read sees the new head and overwrites pending. Either way a signed author's latest head ends at success.

A failed re-read returns null and the status stays pending (fail closed). No unsigned author can get success.

The ledger write path itself is unchanged: same file format, same commit, same rebase-retry push.

Other changes:

  • The handlers now take a small ledger interface so they can be unit-tested. The script runs main() only when invoked directly, comparing real paths. A test checks that a direct invocation still runs main(), so this can't become a silent no-op.
  • governance/ipr-bot-setup.md was stale: it said the adcp flow uses GITHUB_TOKEN with "no App involvement". It now describes the token and the two-job split.

Downstream impact

The callable gets the same split, so downstream repos' check-run names become check / check and check / record. The required context is the commit status IPR Policy / Signature, which is unchanged. No caller changes are needed.

Not changed

  • The required check and its context name.
  • Ledger format, bot detection, and the author/commenter identity check.

Verification

  • npx vitest run --dir tests/ tests/ipr-check-and-record.test.mjs tests/ipr-ledger-remote.test.mjs: 35 passed. Coverage:
    • check path: signed, unsigned, signed-meanwhile (pending then success), re-read failure (stays pending), bot author, never writes
    • write path: head moved (both SHAs marked), head unchanged, wrong signer
    • the CLI entry point
    • the concurrency shape of both workflows
  • readLatest() smoke test against a real git remote, with a signature pushed after the clone: checkout shows [], latest shows [101].
  • actionlint 1.7.12: clean except unexpected key "queue". That key was already on main (actionlint doesn't know the 2026-05-07 queue option yet), and GitHub's docs list queue for jobs.<job_id>.concurrency. I also confirmed GitHub accepts it live: a throwaway branch with the same two-job shape (job-level queue: max + per-PR cancel group) ran with record succeeding and check skipped (run 38782529231). The branch has been deleted.
  • The pre-commit hook passed (root unit suite and typecheck).

After merge

  • Confirm a non-sign PR comment produces a run where record shows as skipped with no wait.
  • Confirm check runs for several PRs at the same time.

Companion PR: #8225 (runner-demand reductions: draft-skip for storyboards, cancel superseded scans).

🤖 Generated with Claude Code

The required "IPR Policy / Signature" status came from a workflow with a
repo-wide concurrency group. Every PR push's check-only run, and every PR
comment, waited in that one group. Each run held the slot while it also
waited for an ubuntu-latest runner. On 2026-10-15 that queue completed one
check every 20-45 minutes (slot waits up to 135 min) and blocked every
merge in the repo.

Split the workflow (and the cross-repo callable) into two jobs:

- check (pull_request_target): reads the ledger and posts the status.
  It never writes. It uses a per-PR group with cancel-in-progress.
- record (sign-phrase comments only): writes the ledger and keeps the
  repo-wide adcp-ipr-signature-write group with queue: max. Because the
  group is on this job, unrelated comments no longer queue in it.

The check and the write no longer share a group, so close the race in
the script. The check re-reads origin/main after it posts pending. The
write re-marks the PR's latest head after its push. Every interleaving
ends with success for a signed author. A failed re-read leaves the status
pending (fail closed).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Escalate to human review

Escalate — gated paths require human/CODEOWNERS review.

This PR modifies workflow files under .github/workflows/**, which sit behind a hard, deterministic approval gate (gated_paths: true). The required review status is REVIEW_REQUIRED, not APPROVED, so the gate is not satisfied and the PR cannot be auto-approved.

Escalation reasons

  • .github/workflows/ipr-agreement.yml (modified) and .github/workflows/ipr-check-callable.yml (modified) match the gated glob .github/workflows/**; a human/CODEOWNERS approval is required before merge.

The reviewer found no blocking or medium findings — the change itself is a clean operational fix splitting the IPR workflow into a read-only check lane and a serialized record lane, with status-race and fail-closed invariants verified and new test coverage added. Once review_decision becomes APPROVED, this PR should fall through to a normal approve.

Why human review

  • .github/workflows/ipr-agreement.yml (modified) matches gated path .github/workflows/** — human/CODEOWNERS review required
  • .github/workflows/ipr-check-callable.yml (modified) matches gated path .github/workflows/** — human/CODEOWNERS review required
  • gated_paths is true and review_decision is REVIEW_REQUIRED (not APPROVED)
  • This PR touches a path under a hard, non-overridable approval gate (.github/workflows/ipr-agreement.yml (modified) matches .github/workflows/**; .github/workflows/ipr-check-callable.yml (modified) matches .github/workflows/**) and the current GitHub review decision is 'REVIEW_REQUIRED', not APPROVED. This is a hard gate enforced in code — Ladon cannot auto-approve until a human/CODEOWNERS approval is recorded, regardless of how clean the diff is.

@aao-secretariat aao-secretariat Bot added the ladon/needs-human-review Ladon has escalated this PR for human review. label Oct 15, 2026
@bokelley
bokelley merged commit 6741fe4 into main Oct 15, 2026
32 checks passed
@bokelley
bokelley deleted the fix/ci-ipr-check-without-write-lock branch October 15, 2026 18:25
bokelley added a commit that referenced this pull request Oct 17, 2026
… races it (#8256)

Several AAO repos call ipr-check-callable.yml, and each writes the shared
signatures/ipr-signatures.json on adcp's main. Actions concurrency groups
are per repo, so the write group does not serialize writes across repos.
When two signatures were recorded at the same time, the second push was
rejected and the script ran `git pull --rebase`. Both commits append to
the end of the same JSON array, so the rebase conflicted and the record
job failed.

On a rejected push, the script now resets to the fresh remote head and
applies the signature to that ledger again instead of replaying its
commit. If the author is already recorded there (for example, they signed
on another repo's PR at the same moment), it skips the write. It retries
up to 5 times with exponential backoff and jitter, then fails with the
last push error.

The ledger format and the post-write status behaviour from #8224 are
unchanged: the commented head and the PR's latest head are still marked
success after the write.

Tests run real git against a local bare remote. They cover a concurrent
append from another repo, the same author recorded concurrently, retry
exhaustion with backoff, and no success status when recording fails.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ladon/needs-human-review Ladon has escalated this PR for human review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant