[bokelley]

Diagnosis (2026-10-15)

The backlog was caused by runner capacity, not a stuck workflow.

Org-wide hosted concurrency hit the cap.

• Summing running ubuntu-latest jobs across adcp, adcp-client, adcp-client-python, adcp-go and actions gives 58 to 60 from 09:24 to 12:21 UTC, never above 60.
• That points to an org concurrent-job limit of 60, which matches GitHub's Team plan for standard Linux runners. I couldn't read plan or billing, so 60 is inferred from the observed ceiling.
• adcp got 29 to 52 of those slots. adcp-client-python and adcp-client used the rest.

adcp demand outran the cap.

measure	value
PR pushes (heads), 09:25 to 11:00	30
jobs from those pushes	1036
runner-minutes from those pushes	5991
time to drain at 45 to 60 slots	100 to 133 min
peak queued jobs	about 390 (10:21)
job queue wait, 09:00 to 13:00	p50 22 min, p90 48 min, max 90 min

Per PR push: p50 48 jobs and 293 runner-minutes, p90 60 jobs and 370 runner-minutes. One push alone can nearly fill the org's 60 slots.

Where today's 12,002 runner-minutes went:

workflow	share
Training Agent Storyboards (PR)	49%
Build Check (PR)	24%
Broken Links	2.5%
Validate schema bundle	2.5%
CodeQL	2.1%
everything else	about 1% each

Within storyboards, the 8 sales orchestrators average 16 minutes each, and 13 tenant jobs average 6 minutes.

Waste that was already low:

• Build Check, Storyboards, Dependency Review and schema-bundle validation already cancel superseded PR runs.
• Superseded runs that still ran to completion cost only 47 runner-minutes today.
• Duplicate push + pull_request triggers don't double-run, because PR branches don't match the push branch filters.
• Draft PRs are the larger avoidable cost: about 1411 runner-minutes today (15% of PR minutes). 849 of those were the storyboard matrix.

Change

file	change	expected effect
training-agent-storyboards.yml	Every job is skipped on draft PRs (github.event.pull_request.draft != true). ready_for_review is added to the PR trigger types, so the full matrix runs when a draft is marked ready. Push runs are unchanged.	About 22 jobs and 200+ runner-minutes saved per draft push (about 850 runner-minutes today). This workflow is not a required check (main's required checks are IPR, TypeScript Build, Check for changeset, Server integration tests, No duplicate migration numbers), so skipped draft runs can't block a merge.
codeql.yml, semgrep.yml, broken-links.yml, migration-smoke-test.yml	Add concurrency that cancels a PR's superseded run. Non-PR runs (push, schedule, workflow_run) each get a unique group (github.run_id), so main-branch analyses are never cancelled or evicted from a pending slot.	Small in steady state (superseded waste was low today). During bursts it stops superseded 7 to 10 minute jobs that are still queued from taking runners. CodeQL's code-scanning merge rule still gets an analysis for every PR head.
.changeset/skip-storyboards-on-draft-prs.md	Patch changeset.	check-changeset-protocol-scope.cjs scopes training-agent-storyboards.yml as protocol, and CI-only storyboard changes have carried patch changesets before (#6788).

Deliberately not changed

• Build Check: no draft skip or path filter.
  • It produces two required checks. A skipped required job reports as passing, so a draft that flips to ready could briefly show green before its real run starts.
  • Docs changes really are tested there (docs-nav, doc-compliance drift, canonical stages). The pre-commit hook also treats docs/ as needing the full server suite.
• AI PR Review (Ladon): no concurrency added. Changes to ai-review.yml are gated to human review by design, and its superseded waste was 7 minutes today.

Capacity is the remaining bottleneck

These changes cut demand, but a burst of 30 or more non-draft pushes will still exceed 60 slots. This needs a decision outside this PR:

1. Larger or self-hosted runners for the storyboard matrix (half the minutes). Options include GitHub larger runners with their own runner-group concurrency, or self-hosted runners via ARC or a hosted-runner vendor. This takes those jobs out of the shared 60-job pool.
2. A GitHub Enterprise plan. Its documented limit for standard Linux runners is 500 concurrent jobs.
3. Run the full storyboard matrix only in a merge queue, or on a label. PR pushes would get a smaller smoke subset. This changes the review workflow.

Companion PR: #8224 (IPR checks no longer serialize behind the runner queue).

Verification

• npx vitest run --dir tests/ tests/ci-runner-demand-workflows.test.mjs: 6 passed. The test asserts the draft guard on every storyboard job, the ready_for_review trigger, and the PR-only cancel shape.
• actionlint 1.7.12: clean on all five workflows.
• The changeset gates passed: check-changeset-protocol-scope.cjs and changesets status --since=origin/main.
• The pre-commit hook passed.

🤖 Generated with Claude Code

[aao-secretariat]

Ladon verdict: Escalate to human review

Escalate — gated paths require human/CODEOWNERS approval.

This PR modifies five files under .github/workflows/**, which are under a hard, path-based approval gate (gated_paths: true). The current review_decision is REVIEW_REQUIRED, not APPROVED, so per decision-table row 2 the outcome must be escalate.

The diff itself is clean: no Critical/High/Medium findings. Concurrency blocks added to broken-links, codeql, migration-smoke-test, and semgrep correctly cancel only superseded PR runs while leaving push/schedule/workflow_run runs with unique groups; the training-agent-storyboards draft-skip uses the correct non-blocking job-level if pattern with ready_for_review coverage and a matching test; the patch changeset is in scope. Nothing blocks on the merits.

Escalation reasons

• Workflow files under .github/workflows/** are gated; review_decision is REVIEW_REQUIRED. Human/CODEOWNERS approval is required before merge:
  • .github/workflows/broken-links.yml
  • .github/workflows/codeql.yml
  • .github/workflows/migration-smoke-test.yml
  • .github/workflows/semgrep.yml
  • .github/workflows/training-agent-storyboards.yml

Why human review

• Gated paths: .github/workflows/** files (broken-links.yml, codeql.yml, migration-smoke-test.yml, semgrep.yml, training-agent-storyboards.yml) are under a hard approval gate and review_decision is REVIEW_REQUIRED — human/CODEOWNERS approval required before merge.
• This PR touches a path under a hard, non-overridable approval gate (.github/workflows/broken-links.yml (modified) matches .github/workflows/**; .github/workflows/codeql.yml (modified) matches .github/workflows/**; .github/workflows/migration-smoke-test.yml (modified) matches .github/workflows/**; .github/workflows/semgrep.yml (modified) matches .github/workflows/**; .github/workflows/training-agent-storyboards.yml (modified) matches .github/workflows/**) and the current GitHub review decision is 'REVIEW_REQUIRED', not APPROVED. This is a hard gate enforced in code — Ladon cannot auto-approve until a human/CODEOWNERS approval is recorded, regardless of how clean the diff is.