Skip to content

Extend the CI build-job timeout caps for M2-PERF-01 (Linux 10→45, macOS arm64 15→20) - #94

Merged
offdev merged 2 commits into
masterfrom
fix/ci-linux-timeout-render-50k
Oct 12, 2026
Merged

offdev merged 2 commits into
masterfrom
fix/ci-linux-timeout-render-50k

Conversation

@offdev

@offdev offdev commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

Extend the CI build-job timeout caps for M2-PERF-01 (Linux 10→45, macOS arm64 15→20)

Small CI-only PR that unblocks the M2-PERF-01 feature PR
(feat/m2-perf-01-render-50k): no engine code, no new tests — only
timeout-minutes caps in both lanes: 10 → 45 on the two Linux build
jobs (linux-gcc, linux-clang), and 15 → 20 on the macOS arm64
job (which hit its cap on this PR's first runs — see below).

Why

The M2-PERF-01 step adds two ctest entries
(laige_bench_render_50k_fpx16 / _fp32) that each render 4000
offscreen frames (3000 measured + 1000 warmup) of the ~116k-item
M2-SCENE-01 reference scene in software GL (Mesa surfaceless EGL +
llvmpipe on the ubuntu-24.04 reference machine). The 10-min build
budget (PRD §8.1, M0-CI-01) cannot contain the workload without
shrinking its canonical 3000-run / 1000-warmup shape:

  • the local canonical-tree CPU window measures ~50 ms/frame at -O0
    (CPU-only measurement of the measured batch stage — 4000 frames,
    both backends, zero-allocation asserted per frame);
  • the reference lane's recorded CPU factor vs the local canonical
    tree is ≈1.9–2.3× (m2-particle-render-budget-rebaseline.md);
  • the software-GL submit stage is on top (464k instanced quads/frame
    at 1280×720).

Estimated ~15–25 min for the two entries on the reference lane,
before the job's other steps. (The entry's own ctest TIMEOUT 1800
still bounds each test; the job cap only adds headroom for the
shared-runners tail.)

The macOS arm64 bump (15 → 20)

This PR's first lane (run 38245828701) hit the macOS arm64 job's
existing 15-min cap on all three attempts: each attempt finished its
full build + the 122/122 ctest suite at 15:00–15:10, and the cap
fired seconds before the job's final cleanup (the identical macOS
Intel job finished in 7:50 on the same run). The job builds the
unchanged master tree — this is shared-runner tail latency, not a
regression from this change — but it blocks this PR's green, so the
cap moves with the evidence (the M2-UI-01 10→15 precedent). macOS
Intel keeps 15 (no cap contact).

Scope

  • ci.yml (merge lane) + ci-pull.yml (PR lane): linux-gcc and
    linux-clang (45), and macos-arm64 (20). The sanitizer jobs
    carry no render-50k entries (the CMake gate excludes them —
    instrumentation inflates the absolute cost) and keep their caps;
    macos-intel keeps 15.
  • Documented exceptions in both files' headers, in the shape of the
    existing macOS exception (M2-UI-01): a job that runs to its cap is
    investigated; the caps are not a silent relaxation of the build
    budget — the budget the render-50k entries guard is their own
    absolute target in budgets.json.

Note on workflow propagation

GitHub runs a PR's workflow from the BASE branch, so this change
takes effect for PRs opened after it lands. Two consequences:

  • This PR's own lane still runs the BASE workflow (macOS arm64
    15 min, Linux 10 min) — its runs will show the known cap contact
    (and its macOS arm64 job may hit the 15-min wall one last time,
    the ~15:05 build time vs the 15-min cap). The merge lane
    (post-merge, on the new workflow) is the authoritative green.
  • The M2-PERF-01 feature PR relies on it: its lane's Linux jobs
    would otherwise hit the 10-min wall before the render-50k entries
    can report. Merge this before the feature PR is reviewed for
    green.

The M2-PERF-01 step (the render-50k bench suite) adds two ctest
entries that render 4000 offscreen frames of the ~116k-item
reference scene in software GL (Mesa surfaceless EGL + llvmpipe on
the ubuntu-24.04 reference machine). The 10-min build budget
(PRD 8.1, M0-CI-01) cannot contain the workload without shrinking
its canonical 3000-run / 1000-warmup shape: the local
canonical-tree CPU window measures ~50 ms/frame at -O0 (CPU-only,
the batch stage alone), the reference lane's recorded CPU factor is
1.9-2.3x (m2-particle-render-budget-rebaseline.md), and the
software-GL submit stage is on top -- estimated ~15-25 min for the
two backends' entries, before the job's other steps.

timeout-minutes 10 -> 45 on the two Linux build jobs (linux-gcc,
linux-clang) in both ci.yml (merge lane) and ci-pull.yml (PR lane);
the entry's own ctest TIMEOUT (1800 s) still bounds each test, and
the absolute budget the entries guard is its own target in
budgets.json. This is a documented exception to the build budget,
in the shape of the macOS 15-min exception (ci.yml header), not a
silent relaxation: a job that runs to the 45-min cap is
investigated.

Note: GitHub runs a PR's workflow from the BASE branch, so this
change takes effect for PRs opened after it lands (the M2-PERF-01
feature PR relies on it -- its lane's Linux jobs would otherwise hit
the 10-min wall before the render-50k entries report).
…rst lane)

Run 38245828701 (this PR, 2026-10-12) hit the macOS arm64 job's
15-min cap on all three attempts: each attempt finished its full
build + the 122/122 ctest suite at 15:00-15:10, the cap firing
seconds before the job's final cleanup (the identical macOS Intel
job finished in 7:50 on the same run). The job builds the unchanged
master tree — shared-runner tail latency, not a regression — but the
repeated cap contact blocks this PR's green, so the cap moves with
the evidence (the M2-UI-01 10->15 precedent). macOS Intel keeps 15
(no cap contact). Headers + per-job comments updated in both lanes.
@offdev offdev changed the title Extend the Linux build jobs' CI timeout 10 to 45 min for M2-PERF-01 Extend the CI build-job timeout caps for M2-PERF-01 (Linux 10→45, macOS arm64 15→20) Oct 12, 2026
@offdev
offdev merged commit e8903b9 into master Oct 12, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant