Skip to content

Move >30 s CI suites to label-gated jobs (ci:slow:*) - #97

Merged
offdev merged 2 commits into
masterfrom
fix/ci-slow-suite-jobs
Oct 13, 2026
Merged

offdev merged 2 commits into
masterfrom
fix/ci-slow-suite-jobs

Conversation

@offdev

@offdev offdev commented Oct 13, 2026 •

Copy link
Copy Markdown
Owner

What this changes

Measured on the latest master run 38312355303 (2026-10-12, all 11 jobs green), every ctest entry that exceeded 30 s in a lane has been moved out of the main job into a dedicated job of its own (one job per suite per lane). The new jobs run only when gated:

  • PR lane (ci-pull.yml): each slow suite has a repo label, created via the GitHub API:
    label entry lanes
    ci:slow:sim laige-sim_tests gcc clang asan tsan
    ci:slow:render laige-render_tests gcc clang asan tsan
    ci:slow:particle particle_render gcc clang tsan
    ci:slow:zero-alloc zero_alloc asan tsan
    ci:slow:ecs-stress ecs_stress asan tsan
    ci:slow:bench-50k laige_bench_render_50k_fpx16 / _fp32 gcc clang
  • Merge lane (ci.yml): the same suites are gated by workflow_dispatch checkboxes (slow-sim ... slow-bench-50k); plain pushes to master never run them.

The main jobs' ctest excludes the moved entries (-E) and the slow jobs select theirs (-R): every entry runs exactly once per gated run, never twice, never ungated.

Per-lane evidence (run 38312355303, seconds):

entry gcc clang asan tsan
laige_bench_render_50k_fpx16 577.05 578.42 – –
laige_bench_render_50k_fp32 593.82 566.52 – –
laige-sim_tests 59.97 53.79 124.03 159.52
laige-render_tests 53.83 44.45 43.20 60.41
particle_render 38.76 34.67 28.84 (stays) 39.56
zero_alloc 16.16 (stays) 14.39 (stays) 43.76 54.75
ecs_stress 16.34 (stays) 14.45 (stays) 42.87 54.70

No entry exceeds 30 s on windows/macOS (worst: laige-sim_tests 27.11 s on macos-intel), so those jobs keep the full suite and are unchanged.

Timeouts: the main Linux jobs revert to 10 min (the gcc/clang 45-min and tsan 30-min caps existed only for the moved suites); the slow suite jobs carry 20 min, the bench jobs 30 min.

Milestone gate (AGENTS.md §5, new subsection): a milestone-gate PR MUST carry ALL six ci:slow:* labels so every slow job runs and passes before merge; after the gate merges, a workflow_dispatch on master with all six checkboxes re-runs the slow matrix. Note: a label added after a run is created never re-triggers the run, so the labels must be on the PR before the run starts.

Docs updated in the same change: AGENTS.md §5, PRD §14 Perf row, roadmap change log. No engine code change — only the two workflows and the three docs.

CI validation (GitHub runs a PR's workflow from the merge commit, i.e. the PR's changes applied on base, so this PR's own runs validate the new PR-lane workflow):

  • Run 38367352602 (no ci:slow:* labels): all 11 fast jobs green, all 19 slow jobs correctly skipped.
  • Run 38369658151 (all six ci:slow:* labels, added only for this gate validation and removed afterwards): all 30 jobs green. The slow suite jobs finished in 1:46 to 5:23 (worst: TSan laige-sim_tests 5:23) and the bench jobs in 6:24 to 11:34, each well inside the 20/30-min caps. The first pass of this run had one unrelated failure: the unchanged macOS arm64 job hit its 20-min shared-runner cap (tests 1-120 of 123 passed, the job was cancelled while test 121 was starting); the rerun of that job passed in 16:08.

Measured on the latest master run 38312355303 (2026-10-12): every ctest entry that exceeded 30 s in a lane moved out of the main job into a dedicated job of its own (one job per suite per lane), gated by a repo label in the PR lane (ci:slow:sim, ci:slow:render, ci:slow:particle, ci:slow:zero-alloc, ci:slow:ecs-stress, ci:slow:bench-50k) and by a workflow_dispatch checkbox in the merge lane. The main jobs' ctest excludes the moved entries (-E) and the slow jobs select theirs (-R), so every entry runs exactly once per gated run and never ungated.

Slow set: laige-sim_tests + laige-render_tests (gcc/clang/asan/tsan), particle_render (gcc/clang/tsan; 28.84 s on asan stays in the main asan job), zero_alloc + ecs_stress (asan/tsan; under 30 s on gcc/clang), laige_bench_render_50k_fpx16/_fp32 (gcc/clang). No entry exceeds 30 s on windows/macOS (worst: laige-sim_tests 27.11 s on macos-intel), so those jobs keep the full suite unchanged.

Timeouts: the main Linux jobs revert to 10 min (the gcc/clang 45-min and tsan 30-min caps existed only for the moved suites); the slow suite jobs carry 20 min, the bench jobs 30 min.

Milestone gate: a milestone-gate PR MUST carry ALL six ci:slow:* labels so every slow job runs and passes before merge (AGENTS.md section 5, new 'Slow CI suites and the milestone gate' subsection; the labels were created via the GitHub API). The PRD section 14 Perf row and the roadmap change log were updated in this change. No engine code change.
@offdev offdev added ci:windows ci:macos ci:slow:sim Gates the slow sim jobs; milestone gates carry all ci:slow:* labels. ci:slow:render Gates the slow render jobs; milestone gates carry all ci:slow:* labels. ci:slow:particle Gates the slow particle jobs; milestone gates carry all ci:slow:* labels. ci:slow:zero-alloc Gates the slow zero-alloc jobs; milestone gates carry all ci:slow:* labels. ci:slow:ecs-stress Gates the slow ecs-stress jobs; milestone gates carry all ci:slow:* labels. ci:slow:bench-50k Gates the slow bench-50k-fpx16/fp32 jobs; milestone gates carry all ci:slow:* labels. labels Oct 13, 2026
@offdev offdev removed ci:slow:sim Gates the slow sim jobs; milestone gates carry all ci:slow:* labels. ci:slow:render Gates the slow render jobs; milestone gates carry all ci:slow:* labels. ci:slow:particle Gates the slow particle jobs; milestone gates carry all ci:slow:* labels. ci:slow:zero-alloc Gates the slow zero-alloc jobs; milestone gates carry all ci:slow:* labels. ci:slow:ecs-stress Gates the slow ecs-stress jobs; milestone gates carry all ci:slow:* labels. ci:slow:bench-50k Gates the slow bench-50k-fpx16/fp32 jobs; milestone gates carry all ci:slow:* labels. labels Oct 13, 2026
@offdev
offdev merged commit 0298b8c into master Oct 13, 2026
59 of 60 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant