From db9611b14914564e1903228262bfb2f8bac3da11 Mon Sep 17 00:00:00 2001 From: Pascal Severin Date: Mon, 12 Oct 2026 13:19:22 +0200 Subject: [PATCH 1/2] Extend the Linux build jobs' CI timeout 10 to 45 min for M2-PERF-01 The M2-PERF-01 step (the render-50k bench suite) adds two ctest entries that render 4000 offscreen frames of the ~116k-item reference scene in software GL (Mesa surfaceless EGL + llvmpipe on the ubuntu-24.04 reference machine). The 10-min build budget (PRD 8.1, M0-CI-01) cannot contain the workload without shrinking its canonical 3000-run / 1000-warmup shape: the local canonical-tree CPU window measures ~50 ms/frame at -O0 (CPU-only, the batch stage alone), the reference lane's recorded CPU factor is 1.9-2.3x (m2-particle-render-budget-rebaseline.md), and the software-GL submit stage is on top -- estimated ~15-25 min for the two backends' entries, before the job's other steps. timeout-minutes 10 -> 45 on the two Linux build jobs (linux-gcc, linux-clang) in both ci.yml (merge lane) and ci-pull.yml (PR lane); the entry's own ctest TIMEOUT (1800 s) still bounds each test, and the absolute budget the entries guard is its own target in budgets.json. This is a documented exception to the build budget, in the shape of the macOS 15-min exception (ci.yml header), not a silent relaxation: a job that runs to the 45-min cap is investigated. Note: GitHub runs a PR's workflow from the BASE branch, so this change takes effect for PRs opened after it lands (the M2-PERF-01 feature PR relies on it -- its lane's Linux jobs would otherwise hit the 10-min wall before the render-50k entries report). --- .github/workflows/ci-pull.yml | 18 +++++++++++++----- .github/workflows/ci.yml | 22 +++++++++++++++++++--- 2 files changed, 32 insertions(+), 8 deletions(-) diff --git a/.github/workflows/ci-pull.yml b/.github/workflows/ci-pull.yml index c2869ee..b3d5812 100644 --- a/.github/workflows/ci-pull.yml +++ b/.github/workflows/ci-pull.yml @@ -25,9 +25,11 @@ # # Jobs, runner images, steps, and the build-budget timeouts are the # same as in ci.yml — including the macOS exception (the two macOS -# jobs carry timeout-minutes: 15, the shared-runners tail latency), -# documented in ci.yml's header (the full matrix documentation and -# the Windows multi-config note). The one difference: the sanitizer +# jobs carry timeout-minutes: 15, the shared-runners tail latency) +# and the Linux exception (the two Linux build jobs carry +# timeout-minutes: 45, M2-PERF-01's render-50k gate), both documented +# in ci.yml's header (the full matrix documentation and the Windows +# multi-config note). The one difference: the sanitizer # jobs upload no artifacts here. The pull_request GITHUB_TOKEN is # read-only (contents:read for same-repo PRs, read-only for fork PRs — # Actions cannot grant actions:write on this event), so the report @@ -70,7 +72,10 @@ jobs: linux-gcc: name: Linux x64 (g++) runs-on: ubuntu-24.04 - timeout-minutes: 10 + # 45 instead of the 10-min build budget: M2-PERF-01's render-50k + # gate (this job's Test step) — see ci.yml's header (the Linux + # exception) and the linux-gcc note there. + timeout-minutes: 45 steps: - uses: actions/checkout@v4 - name: Install GLFW X11 headers (laige-render) @@ -97,7 +102,10 @@ jobs: linux-clang: name: Linux x64 (clang++) runs-on: ubuntu-24.04 - timeout-minutes: 10 + # 45 instead of the 10-min build budget: M2-PERF-01's render-50k + # gate (this job's Test step) — see ci.yml's header (the Linux + # exception) and the linux-gcc note there. + timeout-minutes: 45 steps: - uses: actions/checkout@v4 - name: Install GLFW X11 headers (laige-render) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index eaefd64..98332f4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -85,7 +85,13 @@ # so a job (and therefore the whole matrix, which runs the jobs in # parallel) longer than the budget fails instead of dragging on. The # include-lint job is a lint job, not a build, and carries its own -# timeout-minutes: 5. +# timeout-minutes: 5. Two documented exceptions carry a wider cap: the +# two macOS build jobs (15, below) and the two Linux build jobs (45 — +# M2-PERF-01's render-50k gate renders 4000 offscreen frames of the +# ~116k-item reference scene in software GL on this machine's +# llvmpipe stack; estimated ~15-25 min on the reference lane from the +# local canonical-tree CPU window (~50 ms/frame at -O0) and the lane's +# recorded 1.9-2.3x CPU factor, plus the submit stage). # # macOS exception: the two macOS build jobs carry timeout-minutes: 15 # instead. The shared GitHub macOS runners' tail latency (runner @@ -153,7 +159,14 @@ jobs: linux-gcc: name: Linux x64 (g++) runs-on: ubuntu-24.04 - timeout-minutes: 10 + # 45 instead of the 10-min build budget: M2-PERF-01's render-50k + # gate renders 4000 offscreen frames of the ~116k-item reference + # scene in software GL (this job's Test step) — estimated ~15-25 + # min on the reference lane from the local canonical-tree CPU + # window (~50 ms/frame at -O0) and the lane's recorded 1.9-2.3x + # CPU factor (m2-particle-render-budget-rebaseline.md), plus the + # llvmpipe submit stage. See the header's Linux exception. + timeout-minutes: 45 steps: - uses: actions/checkout@v4 - name: Install GLFW X11 headers (laige-render) @@ -180,7 +193,10 @@ jobs: linux-clang: name: Linux x64 (clang++) runs-on: ubuntu-24.04 - timeout-minutes: 10 + # 45 instead of the 10-min build budget: M2-PERF-01's render-50k + # gate (this job's Test step) — see the linux-gcc note and the + # header's Linux exception. + timeout-minutes: 45 steps: - uses: actions/checkout@v4 - name: Install GLFW X11 headers (laige-render) From 939d289767ee440fa3d11743ce10fe4b46a333e5 Mon Sep 17 00:00:00 2001 From: Pascal Severin Date: Mon, 12 Oct 2026 14:57:47 +0200 Subject: [PATCH 2/2] Raise the macOS arm64 job cap 15 to 20 min (evidenced by this PR's first lane) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run 38245828701 (this PR, 2026-10-12) hit the macOS arm64 job's 15-min cap on all three attempts: each attempt finished its full build + the 122/122 ctest suite at 15:00-15:10, the cap firing seconds before the job's final cleanup (the identical macOS Intel job finished in 7:50 on the same run). The job builds the unchanged master tree — shared-runner tail latency, not a regression — but the repeated cap contact blocks this PR's green, so the cap moves with the evidence (the M2-UI-01 10->15 precedent). macOS Intel keeps 15 (no cap contact). Headers + per-job comments updated in both lanes. --- .github/workflows/ci-pull.yml | 12 +++++++++--- .github/workflows/ci.yml | 37 +++++++++++++++++++++++------------ 2 files changed, 33 insertions(+), 16 deletions(-) diff --git a/.github/workflows/ci-pull.yml b/.github/workflows/ci-pull.yml index b3d5812..0c722e0 100644 --- a/.github/workflows/ci-pull.yml +++ b/.github/workflows/ci-pull.yml @@ -24,8 +24,9 @@ # section 5: the local Linux test gate). # # Jobs, runner images, steps, and the build-budget timeouts are the -# same as in ci.yml — including the macOS exception (the two macOS -# jobs carry timeout-minutes: 15, the shared-runners tail latency) +# same as in ci.yml — including the macOS exceptions (macOS arm64 +# carries timeout-minutes: 20, macOS Intel 15 — the shared-runners +# tail latency) # and the Linux exception (the two Linux build jobs carry # timeout-minutes: 45, M2-PERF-01's render-50k gate), both documented # in ci.yml's header (the full matrix documentation and the Windows @@ -248,7 +249,12 @@ jobs: runs-on: macos-15 # The shared macOS runners' tail latency exceeds the 10-min build # budget (the macOS exception, documented in ci.yml's header). - timeout-minutes: 15 + # 15 -> 20: this PR's run 38245828701 (2026-10-12) hit the 15-min + # cap on all three attempts — each attempt finished its full + # build + the 122/122 ctest suite at 15:00-15:10 and the cap + # fired seconds before the job's final cleanup (the identical + # macOS Intel job finished in 7:50 on the same run). + timeout-minutes: 20 steps: - uses: actions/checkout@v4 - name: Configure diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 98332f4..041072f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -85,26 +85,32 @@ # so a job (and therefore the whole matrix, which runs the jobs in # parallel) longer than the budget fails instead of dragging on. The # include-lint job is a lint job, not a build, and carries its own -# timeout-minutes: 5. Two documented exceptions carry a wider cap: the -# two macOS build jobs (15, below) and the two Linux build jobs (45 — +# timeout-minutes: 5. Two documented exceptions carry wider caps: the +# two macOS build jobs (arm64 20, Intel 15 — below) and the two Linux +# build jobs (45 — # M2-PERF-01's render-50k gate renders 4000 offscreen frames of the # ~116k-item reference scene in software GL on this machine's # llvmpipe stack; estimated ~15-25 min on the reference lane from the # local canonical-tree CPU window (~50 ms/frame at -O0) and the lane's # recorded 1.9-2.3x CPU factor, plus the submit stage). # -# macOS exception: the two macOS build jobs carry timeout-minutes: 15 -# instead. The shared GitHub macOS runners' tail latency (runner -# preemption) exceeds the 10-min budget: the M2-UI-01 PR lane (run -# 38138278150) saw three consecutive 10-min cancellations of the +# macOS exceptions: the shared GitHub macOS runners' tail latency +# (runner preemption) exceeds the 10-min budget: the M2-UI-01 PR lane +# (run 38138278150) saw three consecutive 10-min cancellations of the # macOS arm64 job (11m25s in Build, 10m18s in Test, 10m32s in Build) # while the identical build + test finished in 5m37s on the macOS -# Intel lane. The 15-min cap gives the shared runners headroom without -# hiding a real hang (a job that runs 15 min is still well past the -# 10-min budget and gets investigated). Raised on user decision -# 2026-10-11; if macOS lanes keep hitting the 15-min wall, the -# evidence is persistent runner degradation, not a code regression — -# revisit the budget, never silently lower the cap back. +# Intel lane. Raised on user decision 2026-10-11: macOS arm64 now +# carries timeout-minutes: 20, macOS Intel 15 — each cap gives the +# shared runners headroom without hiding a real hang (a job at its +# cap is still well past the 10-min budget and gets investigated). +# The arm64 15 -> 20 step (2026-10-12): this PR's run 38245828701 hit +# the 15-min cap on all three attempts — each finished its full build +# + the 122/122 ctest suite at 15:00-15:10, the cap firing seconds +# before the job's final cleanup (the identical macOS Intel job +# finished in 7:50 on the same run). If macOS lanes keep hitting +# their walls, the evidence is persistent runner degradation, not a +# code regression — revisit the budget, never silently lower the cap +# back. # # Windows note: CMake's default generator there is the multi-config # "Visual Studio 17 2022" generator, which ignores CMAKE_BUILD_TYPE and @@ -376,7 +382,12 @@ jobs: runs-on: macos-15 # The shared macOS runners' tail latency exceeds the 10-min build # budget (the macOS exception, documented in this file's header). - timeout-minutes: 15 + # 15 -> 20: this PR's run 38245828701 (2026-10-12) hit the 15-min + # cap on all three attempts — each attempt finished its full + # build + the 122/122 ctest suite at 15:00-15:10 and the cap + # fired seconds before the job's final cleanup (the identical + # macOS Intel job finished in 7:50 on the same run). + timeout-minutes: 20 steps: - uses: actions/checkout@v4 - name: Configure