diff --git a/.github/workflows/ci-pull.yml b/.github/workflows/ci-pull.yml index 0c722e0..407e0bc 100644 --- a/.github/workflows/ci-pull.yml +++ b/.github/workflows/ci-pull.yml @@ -1,13 +1,13 @@ # ============================================================================ # Laige CI — full P0 platform matrix on pull request (M0-CI-01, -# re-baselined 2026-10-11) +# re-baselined 2026-10-11, slow-suite split 2026-10-13) # # Roadmap: roadmap/M0-foundations.md, step M0-CI-01 # PRD refs: §6 (P0 platforms, AC-6.1), §14 (cadence), §8.1 (build budget) # -# Cadence (PRD §14, as amended 2026-10-11 — "test everything, all the -# time"): this workflow runs on EVERY pull request, and ALL eleven jobs -# run — unconditionally: +# Cadence (PRD §14, as amended 2026-10-13 — "fast lane + gated slow +# suites"): this workflow runs on EVERY pull request. The eleven FAST +# jobs run unconditionally: # # * the five P0 OS jobs (linux-gcc, linux-clang, windows-msvc, # macos-arm64, macos-intel); @@ -15,28 +15,70 @@ # * the four platform-independent tooling jobs (include-lint, # determinism-lint, api-manifest, detcheck). # -# The former ci:windows / ci:macos label selector is REMOVED: no job is -# gated on a label and no test is ever skipped. Linux is no longer -# "verified locally only" — it runs in the PR lane exactly as in the -# merge lane (ci.yml), so a PR that breaks a Linux tree fails in the -# PR lane before it can reach master. Local verification of the Linux -# trees before pushing remains a developer obligation (AGENTS.md, -# section 5: the local Linux test gate). +# The SLOW suites do NOT run on a normal PR. Every ctest entry that took +# longer than 30 s in the latest master run (38312355303, 2026-10-12) +# has been moved out of the main job into a dedicated job of its own +# (one per lane), and those jobs run ONLY when the PR carries the +# matching ci:slow:* label: +# +# label ctest entry(es) lanes +# ci:slow:sim laige-sim_tests gcc clang asan tsan +# ci:slow:render laige-render_tests gcc clang asan tsan +# ci:slow:particle particle_render gcc clang tsan +# ci:slow:zero-alloc zero_alloc asan tsan +# ci:slow:ecs-stress ecs_stress asan tsan +# ci:slow:bench-50k laige_bench_render_50k_fpx16 / _fp32 gcc clang +# +# Evidence (run 38312355303, per-entry times in s, threshold 30 s): +# laige_bench_render_50k_fpx16 gcc 577.05 clang 578.42 +# laige_bench_render_50k_fp32 gcc 593.82 clang 566.52 +# laige-sim_tests gcc 59.97 clang 53.79 asan 124.03 tsan 159.52 +# laige-render_tests gcc 53.83 clang 44.45 asan 43.20 tsan 60.41 +# particle_render gcc 38.76 clang 34.67 asan 28.84 tsan 39.56 +# zero_alloc gcc 16.16 clang 14.39 asan 43.76 tsan 54.75 +# ecs_stress gcc 16.34 clang 14.45 asan 42.87 tsan 54.70 +# The threshold is per lane (zero_alloc / ecs_stress stay in the main +# gcc/clang jobs, particle_render in the main asan job — see the +# evidence). No entry exceeds 30 s on windows/macOS (worst: +# laige-sim_tests 27.11 s on macos-intel), so those lanes have no slow +# jobs and their jobs are unchanged. +# +# The main jobs' ctest EXCLUDES the moved entries (-E) and each slow +# job SELECTS its entry (-R): every entry runs exactly once per gated +# run, never twice, and never ungated. The suite list is re-derived +# from the latest master run whenever a suite crosses (or leaves) the +# 30 s threshold — see AGENTS.md §5, "Slow CI suites and the +# milestone gate". +# +# MILESTONE GATE: a milestone-gate PR MUST carry ALL six ci:slow:* +# labels so every slow job runs and passes before merge (AGENTS.md +# §5). +# +# Label mechanics (the old ci:windows / ci:macos precedent, still the +# standing convention for those two labels even though no job reads +# them): a run snapshots the PR's labels when it starts, and adding a +# label later never re-triggers a finished or running run — so the +# ci:slow:* labels must be on the PR BEFORE the run is created (a +# push to the PR branch after adding the labels starts a fresh run). +# +# Local verification of the Linux trees before pushing remains a +# developer obligation (AGENTS.md, section 5: the local Linux test +# gate). # # Jobs, runner images, steps, and the build-budget timeouts are the # same as in ci.yml — including the macOS exceptions (macOS arm64 # carries timeout-minutes: 20, macOS Intel 15 — the shared-runners -# tail latency) -# and the Linux exception (the two Linux build jobs carry -# timeout-minutes: 45, M2-PERF-01's render-50k gate), both documented -# in ci.yml's header (the full matrix documentation and the Windows -# multi-config note). The one difference: the sanitizer -# jobs upload no artifacts here. The pull_request GITHUB_TOKEN is -# read-only (contents:read for same-repo PRs, read-only for fork PRs — -# Actions cannot grant actions:write on this event), so the report -# files are not archived; the tee'd ctest output keeps every sanitizer -# report in the job log, which is enough to diagnose a red PR. The -# merge lane (ci.yml) still archives them as artifacts. +# tail latency), both documented in ci.yml's header (the full matrix +# documentation and the Windows multi-config note). The Linux fast jobs +# carry timeout-minutes: 10 (the pre-split 45-min exception is gone — +# the render-50k bench moved to the label-gated slow BENCH jobs, which +# carry 30). The one difference from ci.yml: the sanitizer jobs upload +# no artifacts here, neither fast nor slow. The pull_request +# GITHUB_TOKEN is read-only (contents:read for same-repo PRs, read-only +# for fork PRs — Actions cannot grant actions:write on this event), so +# the report files are not archived; the tee'd ctest output keeps every +# sanitizer report in the job log, which is enough to diagnose a red PR. +# The merge lane (ci.yml) still archives them as artifacts. # # Fuzz lane (PRD §14 "every commit (bounded)"; M0-TEST-01): no separate # fuzz job — the `fuzz_json_parse` ctest entry (1000 deterministic runs @@ -73,10 +115,12 @@ jobs: linux-gcc: name: Linux x64 (g++) runs-on: ubuntu-24.04 - # 45 instead of the 10-min build budget: M2-PERF-01's render-50k - # gate (this job's Test step) — see ci.yml's header (the Linux - # exception) and the linux-gcc note there. - timeout-minutes: 45 + # 10-min build budget (reverted from 45 on 2026-10-13): the render-50k + # bench and the other >30 s entries moved to the label-gated slow + # jobs (linux-gcc-slow / linux-gcc-bench, below). The heaviest + # remaining entries are the filtered sim/render suites (≤ 26 s) + + # the build (~1.5 min) — ~5 min total on the reference lane. + timeout-minutes: 10 steps: - uses: actions/checkout@v4 - name: Install GLFW X11 headers (laige-render) @@ -98,15 +142,19 @@ jobs: - name: Build run: cmake --build build -j - name: Test (unit) - run: ctest --test-dir build --output-on-failure + # -E excludes the >30 s entries that moved to the dedicated + # label-gated jobs (2026-10-13 split, see the header): they are + # selected there with -R, so each entry runs exactly once per + # gated run. + run: | + ctest --test-dir build --output-on-failure \ + -E '^(laige-sim_tests|laige-render_tests|particle_render|laige_bench_render_50k_fpx16|laige_bench_render_50k_fp32)$' linux-clang: name: Linux x64 (clang++) runs-on: ubuntu-24.04 - # 45 instead of the 10-min build budget: M2-PERF-01's render-50k - # gate (this job's Test step) — see ci.yml's header (the Linux - # exception) and the linux-gcc note there. - timeout-minutes: 45 + # 10-min build budget — see the linux-gcc note and the header. + timeout-minutes: 10 steps: - uses: actions/checkout@v4 - name: Install GLFW X11 headers (laige-render) @@ -128,14 +176,22 @@ jobs: - name: Build run: cmake --build build -j - name: Test (unit) - run: ctest --test-dir build --output-on-failure + # -E excludes the >30 s entries that moved to the dedicated + # label-gated jobs (2026-10-13 split, see the header): they are + # selected there with -R, so each entry runs exactly once per + # gated run. + run: | + ctest --test-dir build --output-on-failure \ + -E '^(laige-sim_tests|laige-render_tests|particle_render|laige_bench_render_50k_fpx16|laige_bench_render_50k_fp32)$' linux-asan: name: Linux x64 ASan+UBSan (clang++) runs-on: ubuntu-24.04 - timeout-minutes: 10 + # 10-min build budget: the >30 s entries moved to the label-gated + # linux-asan-slow job (2026-10-13 split, see the header). # No artifact step here: the pull_request token is read-only (see # the header). The tee keeps every report in the job log. + timeout-minutes: 10 steps: - uses: actions/checkout@v4 - name: Install GLFW X11 headers (laige-render) @@ -160,23 +216,29 @@ jobs: # Every sanitizer report is fatal to the test process: ASan via # abort_on_error=1:halt_on_error=1 here, UBSan via # -fno-sanitize-recover=all (wired by CMake). detect_leaks=1 (the - # Linux default) is stated explicitly. + # Linux default) is stated explicitly. -E excludes the >30 s + # entries that moved to the label-gated linux-asan-slow job + # (2026-10-13 split, see the header). env: ASAN_OPTIONS: "abort_on_error=1:halt_on_error=1:detect_leaks=1" run: | set -o pipefail - ctest --test-dir build-asan --output-on-failure 2>&1 | tee asan-ctest-output.txt + ctest --test-dir build-asan --output-on-failure \ + -E '^(laige-sim_tests|laige-render_tests|zero_alloc|ecs_stress)$' \ + 2>&1 | tee asan-ctest-output.txt linux-tsan: name: Linux x64 TSan (clang++) - # 30 min (not 10): TSan is the slowest lane by design (~5-15x - # instrumentation overhead), and the suite has grown past the old - # margin (the documented rationale, with the evidence runs, is in - # the linux-tsan job of ci.yml). + # 10-min build budget: the >30 s entries moved to the label-gated + # linux-tsan-slow job (2026-10-13 split, see the header). Before the + # split the full TSan suite needed 30 min (the documented rationale + # is in ci.yml's linux-tsan note); the heaviest REMAINING entries + # are archetype (~26 s) and laige_run_smoke (~17 s) plus the build + # (~2 min). # No artifact step here: the pull_request token is read-only (see # the header). The tee keeps the race report in the job log. runs-on: ubuntu-24.04 - timeout-minutes: 30 + timeout-minutes: 10 steps: - uses: actions/checkout@v4 - name: Install GLFW X11 headers (laige-render) @@ -201,14 +263,520 @@ jobs: # TSAN_OPTIONS=halt_on_error=1 is applied per test by tests/ # (M0-BUILD-01), so the first data race report kills the test # process and ctest fails. The race report is printed to stderr - # and captured by the tee below. + # and captured by the tee below. -E excludes the >30 s entries + # that moved to the label-gated linux-tsan-slow job (2026-10-13 + # split, see the header). + run: | + set -o pipefail + ctest --test-dir build-tsan --output-on-failure \ + -E '^(laige-sim_tests|laige-render_tests|particle_render|zero_alloc|ecs_stress)$' \ + 2>&1 | tee tsan-ctest-output.txt + + # ------------------------------------------------------------------------ + # Slow suites (2026-10-13 split; see the header). Each job below is one + # ctest entry that exceeded 30 s in the latest master run: one dedicated + # job per suite per lane (the matrix context is not available in a + # job-level `if:`, so one block per suite). The main job's ctest + # excludes the entry, so it runs exactly once per gated run, here. + # ------------------------------------------------------------------------ + linux-gcc-sim: + name: "Linux x64 (g++) slow: laige-sim_tests" + runs-on: ubuntu-24.04 + # 20: build (~1.5 min) + laige-sim_tests (59.97 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: contains(github.event.pull_request.labels.*.name, 'ci:slow:sim') + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-gcc). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=g++ + - name: Build + run: cmake --build build -j + - name: "Test (slow: laige-sim_tests)" + run: ctest --test-dir build --output-on-failure -R '^laige-sim_tests$' + + linux-gcc-render: + name: "Linux x64 (g++) slow: laige-render_tests" + runs-on: ubuntu-24.04 + # 20: build (~1.5 min) + laige-render_tests (53.83 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: contains(github.event.pull_request.labels.*.name, 'ci:slow:render') + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-gcc). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=g++ + - name: Build + run: cmake --build build -j + - name: "Test (slow: laige-render_tests)" + run: ctest --test-dir build --output-on-failure -R '^laige-render_tests$' + + linux-gcc-particle: + name: "Linux x64 (g++) slow: particle_render" + runs-on: ubuntu-24.04 + # 20: build (~1.5 min) + particle_render (38.76 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: contains(github.event.pull_request.labels.*.name, 'ci:slow:particle') + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-gcc). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=g++ + - name: Build + run: cmake --build build -j + - name: "Test (slow: particle_render)" + run: ctest --test-dir build --output-on-failure -R '^particle_render$' + + linux-gcc-bench-fpx16: + name: "Linux x64 (g++) slow bench: laige_bench_render_50k_fpx16" + runs-on: ubuntu-24.04 + # 30: build (~1.5 min) + the render-50k bench (577.05 s in the evidence + # run) plus runner variance. The pre-split 45-min exception minus + # the rest of the suite. + timeout-minutes: 30 + if: contains(github.event.pull_request.labels.*.name, 'ci:slow:bench-50k') + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-gcc). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=g++ + - name: Build + run: cmake --build build -j + - name: "Test (slow bench: laige_bench_render_50k_fpx16)" + run: ctest --test-dir build --output-on-failure -R '^laige_bench_render_50k_fpx16$' + + linux-gcc-bench-fp32: + name: "Linux x64 (g++) slow bench: laige_bench_render_50k_fp32" + runs-on: ubuntu-24.04 + # 30: build (~1.5 min) + the render-50k bench (593.82 s in the evidence + # run) plus runner variance. + timeout-minutes: 30 + if: contains(github.event.pull_request.labels.*.name, 'ci:slow:bench-50k') + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-gcc). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=g++ + - name: Build + run: cmake --build build -j + - name: "Test (slow bench: laige_bench_render_50k_fp32)" + run: ctest --test-dir build --output-on-failure -R '^laige_bench_render_50k_fp32$' + + linux-clang-sim: + name: "Linux x64 (clang++) slow: laige-sim_tests" + runs-on: ubuntu-24.04 + # 20: build (~1.5 min) + laige-sim_tests (53.79 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: contains(github.event.pull_request.labels.*.name, 'ci:slow:sim') + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-clang). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ + - name: Build + run: cmake --build build -j + - name: "Test (slow: laige-sim_tests)" + run: ctest --test-dir build --output-on-failure -R '^laige-sim_tests$' + + linux-clang-render: + name: "Linux x64 (clang++) slow: laige-render_tests" + runs-on: ubuntu-24.04 + # 20: build (~1.5 min) + laige-render_tests (44.45 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: contains(github.event.pull_request.labels.*.name, 'ci:slow:render') + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-clang). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ + - name: Build + run: cmake --build build -j + - name: "Test (slow: laige-render_tests)" + run: ctest --test-dir build --output-on-failure -R '^laige-render_tests$' + + linux-clang-particle: + name: "Linux x64 (clang++) slow: particle_render" + runs-on: ubuntu-24.04 + # 20: build (~1.5 min) + particle_render (34.67 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: contains(github.event.pull_request.labels.*.name, 'ci:slow:particle') + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-clang). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ + - name: Build + run: cmake --build build -j + - name: "Test (slow: particle_render)" + run: ctest --test-dir build --output-on-failure -R '^particle_render$' + + linux-clang-bench-fpx16: + name: "Linux x64 (clang++) slow bench: laige_bench_render_50k_fpx16" + runs-on: ubuntu-24.04 + # 30: build (~1.5 min) + the render-50k bench (578.42 s in the evidence + # run) plus runner variance. + timeout-minutes: 30 + if: contains(github.event.pull_request.labels.*.name, 'ci:slow:bench-50k') + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-clang). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ + - name: Build + run: cmake --build build -j + - name: "Test (slow bench: laige_bench_render_50k_fpx16)" + run: ctest --test-dir build --output-on-failure -R '^laige_bench_render_50k_fpx16$' + + linux-clang-bench-fp32: + name: "Linux x64 (clang++) slow bench: laige_bench_render_50k_fp32" + runs-on: ubuntu-24.04 + # 30: build (~1.5 min) + the render-50k bench (566.52 s in the evidence + # run) plus runner variance. + timeout-minutes: 30 + if: contains(github.event.pull_request.labels.*.name, 'ci:slow:bench-50k') + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-clang). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ + - name: Build + run: cmake --build build -j + - name: "Test (slow bench: laige_bench_render_50k_fp32)" + run: ctest --test-dir build --output-on-failure -R '^laige_bench_render_50k_fp32$' + + linux-asan-sim: + name: "Linux x64 ASan+UBSan (clang++) slow: laige-sim_tests" + runs-on: ubuntu-24.04 + # 20: build (~2.5 min) + laige-sim_tests (124.03 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: contains(github.event.pull_request.labels.*.name, 'ci:slow:sim') + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-asan). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure (ASan+UBSan) + run: cmake -S . -B build-asan -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ -DLAIGE_ASAN=ON + - name: Build (ASan+UBSan) + run: cmake --build build-asan -j + - name: "Test (slow: laige-sim_tests, ASan+UBSan)" + env: + ASAN_OPTIONS: "abort_on_error=1:halt_on_error=1:detect_leaks=1" + run: | + set -o pipefail + ctest --test-dir build-asan --output-on-failure \ + -R '^laige-sim_tests$' 2>&1 | tee asan-ctest-output.txt + + linux-asan-render: + name: "Linux x64 ASan+UBSan (clang++) slow: laige-render_tests" + runs-on: ubuntu-24.04 + # 20: build (~2.5 min) + laige-render_tests (43.20 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: contains(github.event.pull_request.labels.*.name, 'ci:slow:render') + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-asan). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure (ASan+UBSan) + run: cmake -S . -B build-asan -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ -DLAIGE_ASAN=ON + - name: Build (ASan+UBSan) + run: cmake --build build-asan -j + - name: "Test (slow: laige-render_tests, ASan+UBSan)" + env: + ASAN_OPTIONS: "abort_on_error=1:halt_on_error=1:detect_leaks=1" + run: | + set -o pipefail + ctest --test-dir build-asan --output-on-failure \ + -R '^laige-render_tests$' 2>&1 | tee asan-ctest-output.txt + + linux-asan-zero-alloc: + name: "Linux x64 ASan+UBSan (clang++) slow: zero_alloc" + runs-on: ubuntu-24.04 + # 20: build (~2.5 min) + zero_alloc (43.76 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: contains(github.event.pull_request.labels.*.name, 'ci:slow:zero-alloc') + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-asan). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure (ASan+UBSan) + run: cmake -S . -B build-asan -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ -DLAIGE_ASAN=ON + - name: Build (ASan+UBSan) + run: cmake --build build-asan -j + - name: "Test (slow: zero_alloc, ASan+UBSan)" + env: + ASAN_OPTIONS: "abort_on_error=1:halt_on_error=1:detect_leaks=1" + run: | + set -o pipefail + ctest --test-dir build-asan --output-on-failure \ + -R '^zero_alloc$' 2>&1 | tee asan-ctest-output.txt + + linux-asan-ecs-stress: + name: "Linux x64 ASan+UBSan (clang++) slow: ecs_stress" + runs-on: ubuntu-24.04 + # 20: build (~2.5 min) + ecs_stress (42.87 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: contains(github.event.pull_request.labels.*.name, 'ci:slow:ecs-stress') + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-asan). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure (ASan+UBSan) + run: cmake -S . -B build-asan -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ -DLAIGE_ASAN=ON + - name: Build (ASan+UBSan) + run: cmake --build build-asan -j + - name: "Test (slow: ecs_stress, ASan+UBSan)" + env: + ASAN_OPTIONS: "abort_on_error=1:halt_on_error=1:detect_leaks=1" + run: | + set -o pipefail + ctest --test-dir build-asan --output-on-failure \ + -R '^ecs_stress$' 2>&1 | tee asan-ctest-output.txt + + linux-tsan-sim: + name: "Linux x64 TSan (clang++) slow: laige-sim_tests" + runs-on: ubuntu-24.04 + # 20: build (~2 min) + laige-sim_tests (159.52 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: contains(github.event.pull_request.labels.*.name, 'ci:slow:sim') + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-tsan). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure (TSan) + run: cmake -S . -B build-tsan -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ -DLAIGE_TSAN=ON + - name: Build (TSan) + run: cmake --build build-tsan -j + - name: "Test (slow: laige-sim_tests, TSan)" + run: | + set -o pipefail + ctest --test-dir build-tsan --output-on-failure \ + -R '^laige-sim_tests$' 2>&1 | tee tsan-ctest-output.txt + + linux-tsan-render: + name: "Linux x64 TSan (clang++) slow: laige-render_tests" + runs-on: ubuntu-24.04 + # 20: build (~2 min) + laige-render_tests (60.41 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: contains(github.event.pull_request.labels.*.name, 'ci:slow:render') + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-tsan). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure (TSan) + run: cmake -S . -B build-tsan -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ -DLAIGE_TSAN=ON + - name: Build (TSan) + run: cmake --build build-tsan -j + - name: "Test (slow: laige-render_tests, TSan)" + run: | + set -o pipefail + ctest --test-dir build-tsan --output-on-failure \ + -R '^laige-render_tests$' 2>&1 | tee tsan-ctest-output.txt + + linux-tsan-particle: + name: "Linux x64 TSan (clang++) slow: particle_render" + runs-on: ubuntu-24.04 + # 20: build (~2 min) + particle_render (39.56 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: contains(github.event.pull_request.labels.*.name, 'ci:slow:particle') + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-tsan). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure (TSan) + run: cmake -S . -B build-tsan -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ -DLAIGE_TSAN=ON + - name: Build (TSan) + run: cmake --build build-tsan -j + - name: "Test (slow: particle_render, TSan)" + run: | + set -o pipefail + ctest --test-dir build-tsan --output-on-failure \ + -R '^particle_render$' 2>&1 | tee tsan-ctest-output.txt + + linux-tsan-zero-alloc: + name: "Linux x64 TSan (clang++) slow: zero_alloc" + runs-on: ubuntu-24.04 + # 20: build (~2 min) + zero_alloc (54.75 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: contains(github.event.pull_request.labels.*.name, 'ci:slow:zero-alloc') + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-tsan). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure (TSan) + run: cmake -S . -B build-tsan -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ -DLAIGE_TSAN=ON + - name: Build (TSan) + run: cmake --build build-tsan -j + - name: "Test (slow: zero_alloc, TSan)" + run: | + set -o pipefail + ctest --test-dir build-tsan --output-on-failure \ + -R '^zero_alloc$' 2>&1 | tee tsan-ctest-output.txt + + linux-tsan-ecs-stress: + name: "Linux x64 TSan (clang++) slow: ecs_stress" + runs-on: ubuntu-24.04 + # 20: build (~2 min) + ecs_stress (54.70 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: contains(github.event.pull_request.labels.*.name, 'ci:slow:ecs-stress') + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-tsan). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure (TSan) + run: cmake -S . -B build-tsan -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ -DLAIGE_TSAN=ON + - name: Build (TSan) + run: cmake --build build-tsan -j + - name: "Test (slow: ecs_stress, TSan)" run: | set -o pipefail - ctest --test-dir build-tsan --output-on-failure 2>&1 | tee tsan-ctest-output.txt + ctest --test-dir build-tsan --output-on-failure \ + -R '^ecs_stress$' 2>&1 | tee tsan-ctest-output.txt windows-msvc: name: Windows x64 (MSVC 2022) runs-on: windows-2022 + # Unchanged by the 2026-10-13 slow-suite split: no ctest entry + # exceeds 30 s on this lane (worst: laige-sim_tests 16.71 s in the + # evidence run), so the full suite stays in the main job. timeout-minutes: 10 steps: - uses: actions/checkout@v4 @@ -249,11 +817,9 @@ jobs: runs-on: macos-15 # The shared macOS runners' tail latency exceeds the 10-min build # budget (the macOS exception, documented in ci.yml's header). - # 15 -> 20: this PR's run 38245828701 (2026-10-12) hit the 15-min - # cap on all three attempts — each attempt finished its full - # build + the 122/122 ctest suite at 15:00-15:10 and the cap - # fired seconds before the job's final cleanup (the identical - # macOS Intel job finished in 7:50 on the same run). + # Unchanged by the 2026-10-13 slow-suite split: no ctest entry + # exceeds 30 s on this lane (worst: laige-sim_tests 21.60 s in the + # evidence run), so the full suite stays in the main job. timeout-minutes: 20 steps: - uses: actions/checkout@v4 @@ -269,6 +835,9 @@ jobs: runs-on: macos-14 # The shared macOS runners' tail latency exceeds the 10-min build # budget (the macOS exception, documented in ci.yml's header). + # Unchanged by the 2026-10-13 slow-suite split: no ctest entry + # exceeds 30 s on this lane (worst: laige-sim_tests 27.11 s in the + # evidence run), so the full suite stays in the main job. timeout-minutes: 15 steps: - uses: actions/checkout@v4 @@ -302,8 +871,8 @@ jobs: # src/laige-sim/**, with per-line LAIGE-DETERM-EXCEPTION markers as # the documented false-positive policy (docs/concepts/determinism.md). # Platform-independent: Python 3 stdlib only, no setup step. The - # CTest suite runs it against fixture trees and the real tree in - # every P0 job as well (tests/tools, tests `determinism-lint-*`). + # CTest suite runs it against the real tree in every P0 job as well + # (tests/tools, tests `determinism-lint-*`). name: Determinism source scan (sim module) runs-on: ubuntu-24.04 timeout-minutes: 5 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 041072f..cd87bb9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -4,29 +4,68 @@ # Roadmap: roadmap/M0-foundations.md, step M0-CI-01 # PRD refs: §6 (P0 platforms, AC-6.1), §14 (cadence), §8.1 (build budget) # -# Cadence (PRD §14, as amended 2026-10-11 — "test everything, all the -# time"): -# * this workflow runs on every push to the default branch (master) — -# i.e. on merge — and on manual workflow_dispatch: -# ALL eleven jobs run (the seven P0 OS/variant jobs — the five P0 OS +# Cadence (PRD §14, as amended 2026-10-13 — "fast lane + gated slow +# suites"): +# * This workflow runs on every push to the default branch (master) — +# i.e., on merge — and on manual workflow_dispatch. The eleven FAST +# jobs always run: the seven P0 OS/variant jobs — the five P0 OS # jobs plus the two sanitizer lanes (M0-CI-02) — and the four # platform-independent tooling jobs: the include-graph lint with # dependency-count metric (M0-CI-03), the public API manifest drift # check (M0-TOOL-01), the determinism source scan (M1-DET-01), and -# the determinism check (M0-TOOL-02)). +# the determinism check (M0-TOOL-02). # -# Fuzz lane (PRD §14 "every commit (bounded)"; M0-TEST-01): there is no -# separate fuzz job — the `fuzz_json_parse` ctest entry (1000 -# deterministic runs of laige-fuzz on the json_parse target) runs inside -# every build job's ctest above, instrumented in the ASan tree. Seed and -# nightly-long-run conventions: docs/testing.md. -# * Pull requests run the SAME eleven jobs, unconditionally, in the -# companion workflow .github/workflows/ci-pull.yml. The former -# ci:windows / ci:macos label selector is removed: no job is gated -# on a label and no test is ever skipped (Linux included). Local -# verification of the Linux trees before pushing remains a -# developer obligation (AGENTS.md, section 5: the local Linux test -# gate). +# * The SLOW suites do NOT run in the push lane. Every ctest entry +# that took longer than 30 s in the latest master run (38312355303, +# 2026-10-12) has been moved out of the main job into a dedicated +# job of its own (one per lane), and those jobs run ONLY when the +# matching workflow_dispatch checkbox is ticked — the merge-lane +# equivalent of the ci:slow:* PR labels used by ci-pull.yml: +# +# checkbox (dispatch input) / label ctest entry(es) lanes +# slow-sim / ci:slow:sim laige-sim_tests gcc clang asan tsan +# slow-render / ci:slow:render laige-render_tests gcc clang asan tsan +# slow-particle / ci:slow:particle particle_render gcc clang tsan +# slow-zero-alloc / ci:slow:zero-alloc zero_alloc asan tsan +# slow-ecs-stress / ci:slow:ecs-stress ecs_stress asan tsan +# slow-bench-50k / ci:slow:bench-50k laige_bench_render_50k_fpx16, +# laige_bench_render_50k_fp32 gcc clang +# +# Evidence (run 38312355303, per-entry times in s, threshold 30 s): +# laige_bench_render_50k_fpx16 gcc 577.05 clang 578.42 +# laige_bench_render_50k_fp32 gcc 593.82 clang 566.52 +# laige-sim_tests gcc 59.97 clang 53.79 asan 124.03 tsan 159.52 +# laige-render_tests gcc 53.83 clang 44.45 asan 43.20 tsan 60.41 +# particle_render gcc 38.76 clang 34.67 asan 28.84 tsan 39.56 +# zero_alloc gcc 16.16 clang 14.39 asan 43.76 tsan 54.75 +# ecs_stress gcc 16.34 clang 14.45 asan 42.87 tsan 54.70 +# The threshold is per lane: zero_alloc / ecs_stress stay in the +# main gcc/clang jobs (under 30 s there) and move only on the +# sanitizer lanes; particle_render stays in the main asan job +# (28.84 s). No entry exceeds 30 s on windows/macOS (worst: +# laige-sim_tests 27.11 s on macos-intel), so those lanes have no +# slow jobs and their jobs are unchanged. +# +# The main jobs' ctest EXCLUDES the moved entries (-E) and each +# slow job SELECTS its entry (-R): every entry runs exactly once +# per gated run, never twice, and never ungated. The suite list is +# re-derived from the latest master run whenever a suite crosses +# (or leaves) the 30 s threshold — see AGENTS.md §5, "Slow CI +# suites and the milestone gate". +# +# * MILESTONE GATE: a milestone-gate PR MUST carry ALL six ci:slow:* +# labels so every slow job runs and passes before merge (AGENTS.md +# §5). After the gate merges, a workflow_dispatch here with all six +# checkboxes ticked re-runs the slow matrix on master. +# +# * Pull requests run the SAME matrix in the companion workflow +# .github/workflows/ci-pull.yml: the fast jobs run unconditionally +# on every PR, the slow jobs only when the PR carries the matching +# ci:slow:* label. A label added after a run is created never +# re-triggers that run — the label must be on the PR before the +# run starts. Local verification of the Linux trees before pushing +# remains a developer obligation (AGENTS.md, section 5: the local +# Linux test gate). # # One job per P0 OS/variant (PRD §6). Each job runs the canonical flow from # docs/getting-started/building.md — configure → build → ctest (unit tests @@ -79,20 +118,25 @@ # asan-reports/ (ASAN_OPTIONS log_path below); # - /Testing/Temporary/LastTest.log keeps ctest's full raw # output of every test. +# The label-gated slow jobs on the sanitizer lanes carry the same +# archive step (per-suite artifact names, below). # # Build budget (PRD §8.1: clean build ≤ 10 min on CI; roadmap M0-CI-01 -# "matrix runs in < 10 min"): every build job carries timeout-minutes: 10, -# so a job (and therefore the whole matrix, which runs the jobs in -# parallel) longer than the budget fails instead of dragging on. The -# include-lint job is a lint job, not a build, and carries its own -# timeout-minutes: 5. Two documented exceptions carry wider caps: the -# two macOS build jobs (arm64 20, Intel 15 — below) and the two Linux -# build jobs (45 — -# M2-PERF-01's render-50k gate renders 4000 offscreen frames of the -# ~116k-item reference scene in software GL on this machine's -# llvmpipe stack; estimated ~15-25 min on the reference lane from the -# local canonical-tree CPU window (~50 ms/frame at -O0) and the lane's -# recorded 1.9-2.3x CPU factor, plus the submit stage). +# "matrix runs in < 10 min"): every fast build job carries +# timeout-minutes: 10, so a job (and therefore the whole matrix, which +# runs the jobs in parallel) longer than the budget fails instead of +# dragging on. The include-lint job is a lint job, not a build, and +# carries its own timeout-minutes: 5. Two documented exceptions carry +# wider caps: the two macOS build jobs (arm64 20, Intel 15 — below) and +# the two slow BENCH jobs (30 — M2-PERF-01's render-50k gate renders +# 4000 offscreen frames of the ~116k-item reference scene in software +# GL on this machine's llvmpipe stack: ~10 min of bench per backend on +# the reference lane plus ~3 min of build; the pre-split main jobs +# carried 45 for that work plus the rest of the suite). The main Linux +# jobs' caps reverted to 10 on 2026-10-13 with the split: their heaviest +# remaining entries are the filtered sim/render suites (≤ 26 s on the +# gcc lane) plus the build (~1.5 min), ~5 min total on the reference +# lane. # # macOS exceptions: the shared GitHub macOS runners' tail latency # (runner preemption) exceeds the 10-min budget: the M2-UI-01 PR lane @@ -139,7 +183,11 @@ # P0 platforms, as does the ungated sim-tick smoke # (laige_bench_sim_tick_smoke): it exercises the workload and asserts # its bit-identical fpx16_16 final_hash on every platform, without -# checking timing. +# checking timing. The render-50k budget entries +# (laige_bench_render_50k_fpx16 / _fp32) are the SLOW bench jobs above — +# reference-platform-scoped, gated by ci:slow:bench-50k / the +# slow-bench-50k checkbox, and run at every milestone gate (AGENTS.md +# §5). # # AC-6.1 (same source tree, same feature set, no per-OS feature flags): no # build job passes feature flags; every build job builds the identical @@ -153,6 +201,31 @@ on: push: branches: [master] workflow_dispatch: + inputs: + slow-sim: + description: "Run the slow laige-sim_tests jobs (gcc, clang, asan, tsan)" + type: boolean + default: false + slow-render: + description: "Run the slow laige-render_tests jobs (gcc, clang, asan, tsan)" + type: boolean + default: false + slow-particle: + description: "Run the slow particle_render jobs (gcc, clang, tsan)" + type: boolean + default: false + slow-zero-alloc: + description: "Run the slow zero_alloc jobs (asan, tsan)" + type: boolean + default: false + slow-ecs-stress: + description: "Run the slow ecs_stress jobs (asan, tsan)" + type: boolean + default: false + slow-bench-50k: + description: "Run the slow laige_bench_render_50k jobs (gcc, clang)" + type: boolean + default: false permissions: contents: read @@ -165,14 +238,12 @@ jobs: linux-gcc: name: Linux x64 (g++) runs-on: ubuntu-24.04 - # 45 instead of the 10-min build budget: M2-PERF-01's render-50k - # gate renders 4000 offscreen frames of the ~116k-item reference - # scene in software GL (this job's Test step) — estimated ~15-25 - # min on the reference lane from the local canonical-tree CPU - # window (~50 ms/frame at -O0) and the lane's recorded 1.9-2.3x - # CPU factor (m2-particle-render-budget-rebaseline.md), plus the - # llvmpipe submit stage. See the header's Linux exception. - timeout-minutes: 45 + # Back to the 10-min build budget on 2026-10-13: the render-50k bench + # and the other >30 s entries moved to the label-gated slow jobs + # (linux-gcc-slow / linux-gcc-bench, below). The heaviest remaining + # entries are the filtered sim/render suites (≤ 26 s) + the build + # (~1.5 min) — ~5 min total on the reference lane. + timeout-minutes: 10 steps: - uses: actions/checkout@v4 - name: Install GLFW X11 headers (laige-render) @@ -194,15 +265,20 @@ jobs: - name: Build run: cmake --build build -j - name: Test (unit) - run: ctest --test-dir build --output-on-failure + # -E excludes the >30 s entries that moved to the dedicated + # label-gated jobs (2026-10-13 split, see the header): they are + # selected there with -R, so each entry runs exactly once per + # gated run. + run: | + ctest --test-dir build --output-on-failure \ + -E '^(laige-sim_tests|laige-render_tests|particle_render|laige_bench_render_50k_fpx16|laige_bench_render_50k_fp32)$' linux-clang: name: Linux x64 (clang++) runs-on: ubuntu-24.04 - # 45 instead of the 10-min build budget: M2-PERF-01's render-50k - # gate (this job's Test step) — see the linux-gcc note and the - # header's Linux exception. - timeout-minutes: 45 + # Back to the 10-min build budget on 2026-10-13 — see the linux-gcc + # note and the header. + timeout-minutes: 10 steps: - uses: actions/checkout@v4 - name: Install GLFW X11 headers (laige-render) @@ -224,11 +300,19 @@ jobs: - name: Build run: cmake --build build -j - name: Test (unit) - run: ctest --test-dir build --output-on-failure + # -E excludes the >30 s entries that moved to the dedicated + # label-gated jobs (2026-10-13 split, see the header): they are + # selected there with -R, so each entry runs exactly once per + # gated run. + run: | + ctest --test-dir build --output-on-failure \ + -E '^(laige-sim_tests|laige-render_tests|particle_render|laige_bench_render_50k_fpx16|laige_bench_render_50k_fp32)$' linux-asan: name: Linux x64 ASan+UBSan (clang++) runs-on: ubuntu-24.04 + # 10-min build budget: the >30 s entries moved to the label-gated + # linux-asan-slow job (2026-10-13 split, see the header). timeout-minutes: 10 # Job-scoped token: the archive step below needs actions:write, which # the workflow-level contents:read token does not grant (GitHub @@ -261,12 +345,16 @@ jobs: # abort_on_error=1:halt_on_error=1 here, UBSan via # -fno-sanitize-recover=all (wired by CMake). detect_leaks=1 (the # Linux default) is stated explicitly. log_path keeps one report - # file per test process for the artifact upload below. + # file per test process for the artifact upload below. -E + # excludes the >30 s entries that moved to the label-gated + # linux-asan-slow job (2026-10-13 split, see the header). env: ASAN_OPTIONS: "abort_on_error=1:halt_on_error=1:detect_leaks=1:log_path=${{ github.workspace }}/asan-reports/asan" run: | set -o pipefail - ctest --test-dir build-asan --output-on-failure 2>&1 | tee asan-ctest-output.txt + ctest --test-dir build-asan --output-on-failure \ + -E '^(laige-sim_tests|laige-render_tests|zero_alloc|ecs_stress)$' \ + 2>&1 | tee asan-ctest-output.txt - name: Archive sanitizer reports # Runs green or red: on a red run the reports live in this # artifact. The job-level token grants actions:write. @@ -282,19 +370,13 @@ jobs: linux-tsan: name: Linux x64 TSan (clang++) runs-on: ubuntu-24.04 - # 30 min (not 10): TSan is the slowest lane by design (~5-15x - # instrumentation overhead), and the suite has grown past the old - # margin — the M2-ISO-01 10k-scene property suite (iso_depth_key - # entry) plus the M2-GL-02 3000-frame integration run. Evidence: - # the master merge run of PR #60 took 8.8 min (96 entries: setup + - # build ~1.7 min, the 96-entry TSan test step ~7.1 min); the - # M2-ISO-01 PR run needed ~11 min (setup + build 3.7 min on a - # slower runner, the 97-entry test step ~7.2 min — laige-sim_tests - # alone 177 s) and was CANCELLED at the 10-min cap ~50 s before - # completion. 30 min keeps a large margin for runner variance and - # the suite's continued growth without weakening the gate - # (M2-ISO-01, 2026-09-30). - timeout-minutes: 30 + # 10-min build budget: the >30 s entries moved to the label-gated + # linux-tsan-slow job (2026-10-13 split, see the header). Before the + # split the full TSan suite (incl. laige-sim_tests at ~2.7 min and + # the render suites) needed 30 min; the heaviest REMAINING entries + # are archetype (~26 s) and laige_run_smoke (~17 s) plus the build + # (~2 min) — ~5 min total, inside 10 with runner-variance margin. + timeout-minutes: 10 # Job-scoped token: the archive step below needs actions:write (see # the linux-asan job comment). permissions: @@ -324,10 +406,14 @@ jobs: # TSAN_OPTIONS=halt_on_error=1 is applied per test by tests/ # (M0-BUILD-01), so the first data race report kills the test # process and ctest fails. The race report is printed to stderr - # and captured by the tee below. + # and captured by the tee below. -E excludes the >30 s entries + # that moved to the label-gated linux-tsan-slow job (2026-10-13 + # split, see the header). run: | set -o pipefail - ctest --test-dir build-tsan --output-on-failure 2>&1 | tee tsan-ctest-output.txt + ctest --test-dir build-tsan --output-on-failure \ + -E '^(laige-sim_tests|laige-render_tests|particle_render|zero_alloc|ecs_stress)$' \ + 2>&1 | tee tsan-ctest-output.txt - name: Archive sanitizer reports # Runs green or red: on a red run the race report lives in this # artifact. The job-level token grants actions:write. @@ -339,9 +425,650 @@ jobs: tsan-ctest-output.txt build-tsan/Testing/Temporary/LastTest.log + # ------------------------------------------------------------------------ + # Slow suites (2026-10-13 split; see the header). Each job below is one + # ctest entry that exceeded 30 s in the latest master run: one dedicated + # job per suite per lane (the matrix context is not available in a + # job-level `if:`, so one block per suite). The main job's ctest + # excludes the entry, so it runs exactly once per gated run, here. + # ------------------------------------------------------------------------ + linux-gcc-sim: + name: "Linux x64 (g++) slow: laige-sim_tests" + runs-on: ubuntu-24.04 + # 20: build (~1.5 min) + laige-sim_tests (59.97 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: github.event.inputs.slow-sim == 'true' + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-gcc). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=g++ + - name: Build + run: cmake --build build -j + - name: "Test (slow: laige-sim_tests)" + run: ctest --test-dir build --output-on-failure -R '^laige-sim_tests$' + + linux-gcc-render: + name: "Linux x64 (g++) slow: laige-render_tests" + runs-on: ubuntu-24.04 + # 20: build (~1.5 min) + laige-render_tests (53.83 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: github.event.inputs.slow-render == 'true' + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-gcc). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=g++ + - name: Build + run: cmake --build build -j + - name: "Test (slow: laige-render_tests)" + run: ctest --test-dir build --output-on-failure -R '^laige-render_tests$' + + linux-gcc-particle: + name: "Linux x64 (g++) slow: particle_render" + runs-on: ubuntu-24.04 + # 20: build (~1.5 min) + particle_render (38.76 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: github.event.inputs.slow-particle == 'true' + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-gcc). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=g++ + - name: Build + run: cmake --build build -j + - name: "Test (slow: particle_render)" + run: ctest --test-dir build --output-on-failure -R '^particle_render$' + + linux-gcc-bench-fpx16: + name: "Linux x64 (g++) slow bench: laige_bench_render_50k_fpx16" + runs-on: ubuntu-24.04 + # 30: build (~1.5 min) + the render-50k bench (577.05 s in the evidence + # run) plus runner variance. The pre-split 45-min exception minus + # the rest of the suite. + timeout-minutes: 30 + if: github.event.inputs.slow-bench-50k == 'true' + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-gcc). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=g++ + - name: Build + run: cmake --build build -j + - name: "Test (slow bench: laige_bench_render_50k_fpx16)" + run: ctest --test-dir build --output-on-failure -R '^laige_bench_render_50k_fpx16$' + + linux-gcc-bench-fp32: + name: "Linux x64 (g++) slow bench: laige_bench_render_50k_fp32" + runs-on: ubuntu-24.04 + # 30: build (~1.5 min) + the render-50k bench (593.82 s in the evidence + # run) plus runner variance. + timeout-minutes: 30 + if: github.event.inputs.slow-bench-50k == 'true' + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-gcc). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=g++ + - name: Build + run: cmake --build build -j + - name: "Test (slow bench: laige_bench_render_50k_fp32)" + run: ctest --test-dir build --output-on-failure -R '^laige_bench_render_50k_fp32$' + + linux-clang-sim: + name: "Linux x64 (clang++) slow: laige-sim_tests" + runs-on: ubuntu-24.04 + # 20: build (~1.5 min) + laige-sim_tests (53.79 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: github.event.inputs.slow-sim == 'true' + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-clang). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ + - name: Build + run: cmake --build build -j + - name: "Test (slow: laige-sim_tests)" + run: ctest --test-dir build --output-on-failure -R '^laige-sim_tests$' + + linux-clang-render: + name: "Linux x64 (clang++) slow: laige-render_tests" + runs-on: ubuntu-24.04 + # 20: build (~1.5 min) + laige-render_tests (44.45 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: github.event.inputs.slow-render == 'true' + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-clang). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ + - name: Build + run: cmake --build build -j + - name: "Test (slow: laige-render_tests)" + run: ctest --test-dir build --output-on-failure -R '^laige-render_tests$' + + linux-clang-particle: + name: "Linux x64 (clang++) slow: particle_render" + runs-on: ubuntu-24.04 + # 20: build (~1.5 min) + particle_render (34.67 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + if: github.event.inputs.slow-particle == 'true' + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-clang). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ + - name: Build + run: cmake --build build -j + - name: "Test (slow: particle_render)" + run: ctest --test-dir build --output-on-failure -R '^particle_render$' + + linux-clang-bench-fpx16: + name: "Linux x64 (clang++) slow bench: laige_bench_render_50k_fpx16" + runs-on: ubuntu-24.04 + # 30: build (~1.5 min) + the render-50k bench (578.42 s in the evidence + # run) plus runner variance. + timeout-minutes: 30 + if: github.event.inputs.slow-bench-50k == 'true' + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-clang). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ + - name: Build + run: cmake --build build -j + - name: "Test (slow bench: laige_bench_render_50k_fpx16)" + run: ctest --test-dir build --output-on-failure -R '^laige_bench_render_50k_fpx16$' + + linux-clang-bench-fp32: + name: "Linux x64 (clang++) slow bench: laige_bench_render_50k_fp32" + runs-on: ubuntu-24.04 + # 30: build (~1.5 min) + the render-50k bench (566.52 s in the evidence + # run) plus runner variance. + timeout-minutes: 30 + if: github.event.inputs.slow-bench-50k == 'true' + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-clang). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ + - name: Build + run: cmake --build build -j + - name: "Test (slow bench: laige_bench_render_50k_fp32)" + run: ctest --test-dir build --output-on-failure -R '^laige_bench_render_50k_fp32$' + + linux-asan-sim: + name: "Linux x64 ASan+UBSan (clang++) slow: laige-sim_tests" + runs-on: ubuntu-24.04 + # 20: build (~2.5 min) + laige-sim_tests (124.03 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + # Job-scoped token: the archive step needs actions:write (see + # linux-asan). + permissions: + contents: read + actions: write + if: github.event.inputs.slow-sim == 'true' + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-asan). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure (ASan+UBSan) + run: cmake -S . -B build-asan -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ -DLAIGE_ASAN=ON + - name: Build (ASan+UBSan) + run: cmake --build build-asan -j + - name: "Test (slow: laige-sim_tests, ASan+UBSan)" + env: + ASAN_OPTIONS: "abort_on_error=1:halt_on_error=1:detect_leaks=1:log_path=${{ github.workspace }}/asan-reports/asan" + run: | + set -o pipefail + ctest --test-dir build-asan --output-on-failure \ + -R '^laige-sim_tests$' 2>&1 | tee asan-ctest-output.txt + - name: Archive sanitizer reports + # Runs green or red: on a red run the reports live in this + # artifact (one artifact per suite, per run). + if: always() + uses: actions/upload-artifact@v4 + with: + name: linux-asan-slow-sim + path: | + asan-ctest-output.txt + asan-reports/ + build-asan/Testing/Temporary/LastTest.log + + linux-asan-render: + name: "Linux x64 ASan+UBSan (clang++) slow: laige-render_tests" + runs-on: ubuntu-24.04 + # 20: build (~2.5 min) + laige-render_tests (43.20 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + # Job-scoped token: the archive step needs actions:write (see + # linux-asan). + permissions: + contents: read + actions: write + if: github.event.inputs.slow-render == 'true' + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-asan). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure (ASan+UBSan) + run: cmake -S . -B build-asan -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ -DLAIGE_ASAN=ON + - name: Build (ASan+UBSan) + run: cmake --build build-asan -j + - name: "Test (slow: laige-render_tests, ASan+UBSan)" + env: + ASAN_OPTIONS: "abort_on_error=1:halt_on_error=1:detect_leaks=1:log_path=${{ github.workspace }}/asan-reports/asan" + run: | + set -o pipefail + ctest --test-dir build-asan --output-on-failure \ + -R '^laige-render_tests$' 2>&1 | tee asan-ctest-output.txt + - name: Archive sanitizer reports + # Runs green or red: on a red run the reports live in this + # artifact (one artifact per suite, per run). + if: always() + uses: actions/upload-artifact@v4 + with: + name: linux-asan-slow-render + path: | + asan-ctest-output.txt + asan-reports/ + build-asan/Testing/Temporary/LastTest.log + + linux-asan-zero-alloc: + name: "Linux x64 ASan+UBSan (clang++) slow: zero_alloc" + runs-on: ubuntu-24.04 + # 20: build (~2.5 min) + zero_alloc (43.76 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + # Job-scoped token: the archive step needs actions:write (see + # linux-asan). + permissions: + contents: read + actions: write + if: github.event.inputs.slow-zero-alloc == 'true' + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-asan). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure (ASan+UBSan) + run: cmake -S . -B build-asan -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ -DLAIGE_ASAN=ON + - name: Build (ASan+UBSan) + run: cmake --build build-asan -j + - name: "Test (slow: zero_alloc, ASan+UBSan)" + env: + ASAN_OPTIONS: "abort_on_error=1:halt_on_error=1:detect_leaks=1:log_path=${{ github.workspace }}/asan-reports/asan" + run: | + set -o pipefail + ctest --test-dir build-asan --output-on-failure \ + -R '^zero_alloc$' 2>&1 | tee asan-ctest-output.txt + - name: Archive sanitizer reports + # Runs green or red: on a red run the reports live in this + # artifact (one artifact per suite, per run). + if: always() + uses: actions/upload-artifact@v4 + with: + name: linux-asan-slow-zero-alloc + path: | + asan-ctest-output.txt + asan-reports/ + build-asan/Testing/Temporary/LastTest.log + + linux-asan-ecs-stress: + name: "Linux x64 ASan+UBSan (clang++) slow: ecs_stress" + runs-on: ubuntu-24.04 + # 20: build (~2.5 min) + ecs_stress (42.87 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + # Job-scoped token: the archive step needs actions:write (see + # linux-asan). + permissions: + contents: read + actions: write + if: github.event.inputs.slow-ecs-stress == 'true' + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-asan). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure (ASan+UBSan) + run: cmake -S . -B build-asan -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ -DLAIGE_ASAN=ON + - name: Build (ASan+UBSan) + run: cmake --build build-asan -j + - name: "Test (slow: ecs_stress, ASan+UBSan)" + env: + ASAN_OPTIONS: "abort_on_error=1:halt_on_error=1:detect_leaks=1:log_path=${{ github.workspace }}/asan-reports/asan" + run: | + set -o pipefail + ctest --test-dir build-asan --output-on-failure \ + -R '^ecs_stress$' 2>&1 | tee asan-ctest-output.txt + - name: Archive sanitizer reports + # Runs green or red: on a red run the reports live in this + # artifact (one artifact per suite, per run). + if: always() + uses: actions/upload-artifact@v4 + with: + name: linux-asan-slow-ecs-stress + path: | + asan-ctest-output.txt + asan-reports/ + build-asan/Testing/Temporary/LastTest.log + + linux-tsan-sim: + name: "Linux x64 TSan (clang++) slow: laige-sim_tests" + runs-on: ubuntu-24.04 + # 20: build (~2 min) + laige-sim_tests (159.52 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + # Job-scoped token: the archive step needs actions:write (see + # linux-tsan). + permissions: + contents: read + actions: write + if: github.event.inputs.slow-sim == 'true' + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-tsan). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure (TSan) + run: cmake -S . -B build-tsan -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ -DLAIGE_TSAN=ON + - name: Build (TSan) + run: cmake --build build-tsan -j + - name: "Test (slow: laige-sim_tests, TSan)" + run: | + set -o pipefail + ctest --test-dir build-tsan --output-on-failure \ + -R '^laige-sim_tests$' 2>&1 | tee tsan-ctest-output.txt + - name: Archive sanitizer reports + # Runs green or red: on a red run the reports live in this + # artifact (one artifact per suite, per run). + if: always() + uses: actions/upload-artifact@v4 + with: + name: linux-tsan-slow-sim + path: | + tsan-ctest-output.txt + build-tsan/Testing/Temporary/LastTest.log + + linux-tsan-render: + name: "Linux x64 TSan (clang++) slow: laige-render_tests" + runs-on: ubuntu-24.04 + # 20: build (~2 min) + laige-render_tests (60.41 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + # Job-scoped token: the archive step needs actions:write (see + # linux-tsan). + permissions: + contents: read + actions: write + if: github.event.inputs.slow-render == 'true' + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-tsan). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure (TSan) + run: cmake -S . -B build-tsan -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ -DLAIGE_TSAN=ON + - name: Build (TSan) + run: cmake --build build-tsan -j + - name: "Test (slow: laige-render_tests, TSan)" + run: | + set -o pipefail + ctest --test-dir build-tsan --output-on-failure \ + -R '^laige-render_tests$' 2>&1 | tee tsan-ctest-output.txt + - name: Archive sanitizer reports + # Runs green or red: on a red run the reports live in this + # artifact (one artifact per suite, per run). + if: always() + uses: actions/upload-artifact@v4 + with: + name: linux-tsan-slow-render + path: | + tsan-ctest-output.txt + build-tsan/Testing/Temporary/LastTest.log + + linux-tsan-particle: + name: "Linux x64 TSan (clang++) slow: particle_render" + runs-on: ubuntu-24.04 + # 20: build (~2 min) + particle_render (39.56 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + # Job-scoped token: the archive step needs actions:write (see + # linux-tsan). + permissions: + contents: read + actions: write + if: github.event.inputs.slow-particle == 'true' + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-tsan). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure (TSan) + run: cmake -S . -B build-tsan -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ -DLAIGE_TSAN=ON + - name: Build (TSan) + run: cmake --build build-tsan -j + - name: "Test (slow: particle_render, TSan)" + run: | + set -o pipefail + ctest --test-dir build-tsan --output-on-failure \ + -R '^particle_render$' 2>&1 | tee tsan-ctest-output.txt + - name: Archive sanitizer reports + # Runs green or red: on a red run the reports live in this + # artifact (one artifact per suite, per run). + if: always() + uses: actions/upload-artifact@v4 + with: + name: linux-tsan-slow-particle + path: | + tsan-ctest-output.txt + build-tsan/Testing/Temporary/LastTest.log + + linux-tsan-zero-alloc: + name: "Linux x64 TSan (clang++) slow: zero_alloc" + runs-on: ubuntu-24.04 + # 20: build (~2 min) + zero_alloc (54.75 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + # Job-scoped token: the archive step needs actions:write (see + # linux-tsan). + permissions: + contents: read + actions: write + if: github.event.inputs.slow-zero-alloc == 'true' + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-tsan). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure (TSan) + run: cmake -S . -B build-tsan -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ -DLAIGE_TSAN=ON + - name: Build (TSan) + run: cmake --build build-tsan -j + - name: "Test (slow: zero_alloc, TSan)" + run: | + set -o pipefail + ctest --test-dir build-tsan --output-on-failure \ + -R '^zero_alloc$' 2>&1 | tee tsan-ctest-output.txt + - name: Archive sanitizer reports + # Runs green or red: on a red run the reports live in this + # artifact (one artifact per suite, per run). + if: always() + uses: actions/upload-artifact@v4 + with: + name: linux-tsan-slow-zero-alloc + path: | + tsan-ctest-output.txt + build-tsan/Testing/Temporary/LastTest.log + + linux-tsan-ecs-stress: + name: "Linux x64 TSan (clang++) slow: ecs_stress" + runs-on: ubuntu-24.04 + # 20: build (~2 min) + ecs_stress (54.70 s in the evidence run), + # plus runner variance. + timeout-minutes: 20 + # Job-scoped token: the archive step needs actions:write (see + # linux-tsan). + permissions: + contents: read + actions: write + if: github.event.inputs.slow-ecs-stress == 'true' + steps: + - uses: actions/checkout@v4 + - name: Install GLFW X11 headers (laige-render) + # Same rationale as the main job (see linux-tsan). + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq \ + libx11-dev libxcursor-dev libxrandr-dev \ + libxinerama-dev libxi-dev libxext-dev \ + libegl1 libgl1 libegl-mesa0 libglx-mesa0 libgl1-mesa-dri + - name: Configure (TSan) + run: cmake -S . -B build-tsan -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ -DLAIGE_TSAN=ON + - name: Build (TSan) + run: cmake --build build-tsan -j + - name: "Test (slow: ecs_stress, TSan)" + run: | + set -o pipefail + ctest --test-dir build-tsan --output-on-failure \ + -R '^ecs_stress$' 2>&1 | tee tsan-ctest-output.txt + - name: Archive sanitizer reports + # Runs green or red: on a red run the reports live in this + # artifact (one artifact per suite, per run). + if: always() + uses: actions/upload-artifact@v4 + with: + name: linux-tsan-slow-ecs-stress + path: | + tsan-ctest-output.txt + build-tsan/Testing/Temporary/LastTest.log + windows-msvc: name: Windows x64 (MSVC 2022) runs-on: windows-2022 + # Unchanged by the 2026-10-13 slow-suite split: no ctest entry + # exceeds 30 s on this lane (worst: laige-sim_tests 16.71 s in the + # evidence run), so the full suite stays in the main job. timeout-minutes: 10 steps: - uses: actions/checkout@v4 @@ -387,6 +1114,9 @@ jobs: # build + the 122/122 ctest suite at 15:00-15:10 and the cap # fired seconds before the job's final cleanup (the identical # macOS Intel job finished in 7:50 on the same run). + # Unchanged by the 2026-10-13 slow-suite split: no ctest entry + # exceeds 30 s on this lane (worst: laige-sim_tests 21.60 s in the + # evidence run), so the full suite stays in the main job. timeout-minutes: 20 steps: - uses: actions/checkout@v4 @@ -402,6 +1132,9 @@ jobs: runs-on: macos-14 # The shared macOS runners' tail latency exceeds the 10-min build # budget (the macOS exception, documented in this file's header). + # Unchanged by the 2026-10-13 slow-suite split: no ctest entry + # exceeds 30 s on this lane (worst: laige-sim_tests 27.11 s in the + # evidence run), so the full suite stays in the main job. timeout-minutes: 15 steps: - uses: actions/checkout@v4 @@ -502,9 +1235,8 @@ jobs: # stated per ARCH-010). # # Like include-lint and api-manifest, the job runs on every PR and - # merge unconditionally (no label selector — see the header) — it is - # a tooling check, not an additional P0 OS build (the PRD §14 - # cadence is unchanged). + # merge unconditionally (it is a tooling check, not an additional P0 + # OS build — the PRD §14 cadence is unchanged). name: Determinism check runs-on: ubuntu-24.04 timeout-minutes: 20 diff --git a/AGENTS.md b/AGENTS.md index d7cdf75..b8baf99 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -161,6 +161,47 @@ GitHub without the gate is a rule violation (CORE-008): the PR lane all four Linux trees, but the gate exists so a red Linux tree is caught on the developer machine before it reaches GitHub. + ### Slow CI suites and the milestone gate (2026-10-13) + + The CI lanes split into a fast lane (every PR, every merge) and + label-gated slow jobs. Every ctest entry that took longer than 30 s + in a merge-lane run has been moved out of the main job into a + dedicated job of its own, and it MUST NOT run unless its gate is + provided. The current slow set, measured on the latest master run + (38312355303, 2026-10-12): + + - `laige-sim_tests` — gcc, clang, asan, tsan — label `ci:slow:sim` + - `laige-render_tests` — gcc, clang, asan, tsan — label + `ci:slow:render` + - `particle_render` — gcc, clang, tsan — label `ci:slow:particle` + - `zero_alloc` — asan, tsan — label `ci:slow:zero-alloc` + - `ecs_stress` — asan, tsan — label `ci:slow:ecs-stress` + - `laige_bench_render_50k_fpx16` / `laige_bench_render_50k_fp32` — + gcc, clang — label `ci:slow:bench-50k` + + No entry exceeds 30 s on the windows/macOS lanes (worst: + `laige-sim_tests` 27.11 s on macos-intel), so those jobs keep the full + suite and have no slow jobs. The main jobs' `ctest` excludes the moved + entries (`-E`) and the slow jobs select theirs (`-R`): every entry + runs exactly once per gated run, never twice, and never ungated. + + **Milestone gate.** The slow suites MUST run to finalize a milestone + gate: before a milestone-gate PR is merged, the PR MUST carry ALL + six `ci:slow:*` labels so every slow job runs and passes (a label + added after a run is created never re-triggers it — the labels must + be on the PR before the run starts). After the gate merges, a + `workflow_dispatch` on master with all six slow checkboxes ticked + re-runs the slow matrix on master. The local Linux test gate above is + unaffected: it still runs the complete unfiltered `ctest` of all four + trees locally, slow suites included. + + **Threshold maintenance.** The slow set is re-derived from the latest + master run whenever a suite crosses the 30 s threshold: a main-job + entry that exceeds 30 s moves to a dedicated label-gated job in the + next PR, and a gated entry that is back under 30 s in the latest + master run returns to its main job. The per-lane evidence (the entry + times above) MUST be recorded in the workflow header of the change. + ## 6. Architecture boundaries - **ARCH-001:** Keep simulation, rendering, platform, asset, audio, networking, diff --git a/PRD.md b/PRD.md index e25073f..e6dbdbc 100644 --- a/PRD.md +++ b/PRD.md @@ -426,7 +426,7 @@ Developed with the expectation that a significant share of game code is written | Unit | Per-module, ASan/UBSan | Every commit | | Integration | ECS↔sim↔render↔net contracts; replay determinism; protocol fuzz; isometric picking/depth-key golden tests | Every commit | | Fuzz | Asset import, network packets, config parse | Every commit (bounded), nightly (long) | -| Perf | Budget suite (§8.1) on pinned CI hardware, incl. worst-case isometric scene | Every PR (subset), nightly (full) | +| Perf | Budget suite (§8.1) on pinned CI hardware, incl. worst-case isometric scene — the render-50k budget entries run label-gated (`ci:slow:bench-50k`) and always at the milestone gate (AGENTS.md §5) | Every PR (subset; render-50k label-gated), nightly (full) | | Determinism | Two-target bit-identity check (seeded scenarios) | Every merge | | Soak | 72 h auto-gameplay + abuse scenarios per release candidate | Per release | | Cross-OS | All P0 platforms in CI; one per PR, all per merge | Per PR/merge | diff --git a/roadmap/README.md b/roadmap/README.md index ea0ea04..ce861c1 100644 --- a/roadmap/README.md +++ b/roadmap/README.md @@ -256,6 +256,7 @@ One line per completed (or split/renumbered) step. | 2026-10-12 | M2-SCENE-01 | `2451424` | Reference isometric scene generator (PRD §4 — the reference scene for all visual/perf tests; M2-SCENE-01 scope, nothing else): **API** — NEW header-only `laige/render/iso_reference_scene.h` (no GL calls): `buildIsoReferenceScene(seed, variant)` the deterministic worst-case scene builder (the 50k-sprite §8.1 Full variant + the 1k-sprite Small variant — 256×256 / 32×32 stepped-terrain grids of uniform-height plateaus, sprites standing on their PRNG-chosen tiles at exact 2^-16 positions, 3 preset-factor parallax layers, 8 continuous emitters, a 2-widget UI on the 1280×720 reference screen); same seed, same bytes, every build (ARCH-010 — Prng substreams + the exact dyadic position scale, no FMA-susceptible composition); the scene serializes to a versioned little-endian binary with an untrusted-input reader (full domain validation + exact length, `MalformedInput` on any violation, ARCH-007 / SCALE-004; the fuzz gate `fuzz_scene_iso_parse`, TEST-005); `loadIsoReferenceScene` builds the engine-side world (TileMap + ParallaxLayers + ParticleSystem + UiTree + StringTable + the 4 precomputed frame UVs); `declareIsoReferenceScene` the one-call whole-world pass (O(frame items), zero allocation — the scene's 14-group draw-call property under the 30-draw-call budget asserted by `IsoReferenceSceneLoadAndBatch`); `laige-scene-iso` (tools/scene-iso) the headless generator CLI (atomic temp + rename write, round-trip self-check, machine-greppable summary) + the `scene_iso_cli` ctest determinism gate (two same-seed invocations per variant, byte-identical); 22 tests in `scene_iso_tests`; **engine bug fixed** — `StringTable::operator=(StringTable&&)` infinite recursion in `text_items.h` + regression test `TextStringTable.MoveAssignment`; **budget** — no `budgets.json` entry (the per-frame declare cost is part of the composite 50k render-CPU budget, M2-PERF-01 — the pin count stays 17); **local verification** — the 4-tree Linux gate green (115/115, 115/115, 112/112, 112/112 minus the 9 GL-creating entries per the GPU policy), `laige-api.json` stable at 1815 symbols / 48 headers, CI run 38183074975 all 11 jobs green; **Progress Board M2 27/33 (total 73/194) — row added retroactively by the M2-PERF-01 PR** (the M2-SCENE-01 step landed without updating the board or this log — the M2-ISO-03 index-row precedent) | | 2026-10-12 | M2-PERF-01 | `—` / PR (open) | 50k-sprite budget test (PRD §8.1; M2-PERF-01 scope, nothing else): **suite** — `laige-bench --suite=render-50k` in `tools/bench/laige-bench.cpp` (tool-only — no public headers, no engine change): per frame the measured window spans the batch stage only (world `beginFrame` + `declareIsoReferenceScene` + `build`, then UI `beginFrame` + `declareUi` + `build` on the 1280×720 reference screen); particle `update()` and both renderer `submit()` calls sit outside the window (the CPU budget is the batch stage), the world+UI `frameStats().drawCalls` sum feeds the draw-call budget as a max, and per-frame allocations are asserted 0 in-suite (the M1-ALLOC-01 alloc watch extended to the batcher — a violating frame aborts the run, exit 1, CORE-008); CPU-only degradation without offscreen GL 3.3 (the window + zero-alloc still measure; the draw-call check fails `NO_SAMPLES` loudly); **tests** — two ctest entries `laige_bench_render_50k_fpx16` / `_fp32` (Linux non-sanitizer scope — the CMake gate keeps them out of the ASan/TSan trees, instrumentation inflates the absolute cost; the canonical 3000 measured + 1000 warm-up frames, `--budget=sprites_50k_draw_calls --budget=sprites_50k_cpu`, ctest `TIMEOUT 1800`, `LAIGE_BUDGETS_PATH`); **budget** — first measurement of `sprites_50k_cpu` + `sprites_50k_draw_calls` (CI run 38257115294, commit `23a27ab`): 14 draw calls/frame on every backend and toolchain (2.1× inside the 30 budget), 60.4450 ms mean batch stage on the CI reference lane (worse backend fp32_pinned; the g++ lane 69.4684 ms), zero bad-alloc frames — the 2.0 ms target predates the concrete M2-SCENE-01 workload, so the first CI run fails it by design; calibrated in the same PR: PRD §8.1 **v0.5** (`≤ 2 ms CPU` → `≤ 100 ms CPU`, §4 acceptance line with it), `budgets.json` `sprites_50k_cpu` `target` 2.0 → **100.0**, `measured` 0 → **60.4450**, `sprites_50k_draw_calls` `measured` 0 → **14** (`target` 30 unchanged); **new baseline** `docs/benchmarks/baselines/m2-render-50k.md` (the twelfth baseline — full AGENTS §12 metadata + verbatim CI reports) + the baselines index entry; **docs** — `docs/benchmarks/budget_harness.md` (the suite table, the `histogramFor` hook, `--math`, the exit-1 contract, determinism), the §8.1 composite-budget references in `iso_reference_scene.md` / `sprite_batcher.md` / `sprite_renderer.md`, this file's stale entry names fixed (`render_50k_drawcalls` → `sprites_50k_draw_calls`, `render_50k_cpu` → `sprites_50k_cpu`); **local verification** — the 4-tree Linux gate green ×4 (115/115, 115/115, 112/112, 112/112 — 126 entries, 11 excluded: the 9 GL-creating entries per the GPU policy + the 2 new entries on the sanitizer-excluded CMake gate); **CI** — the first lanes caught two setup bugs, both fail-loud aborts (CORE-008): the UI renderer's atlas-registry domain `[0, maxAtlases)` vs the scene's sparse UI atlas id 100 (registry sized `kIsoRefSceneUiFontAtlas + 1`), and the world fg parallax atlas 8 unbound (the bind loop now covers the scene's 0..8 atlas-id plan); full lane run 38257115294: both Linux jobs 124/126 (the only failures are the two budget entries — the documented first measurement), all other 9 jobs green; the step's CI entries required the companion CI PR (#94 — Linux build-job cap 10→45 min), merged first; Progress Board M2 28/33 (total 74/194) | | 2026-10-12 | M2-PROF-01 | `—` / PR (open) | CPU/GPU render timing (RENDER-005 — CPU and GPU measured independently; M2-PROF-01 scope, nothing else): **API** — `SpriteBatcher` gained the per-frame CPU phase times (`SpriteBatchTiming` via `frameTiming()`: `batchMs` = the frame window, `beginFrame()`/create up to `build()` start — the declaration work; `sortMs` = `build()` itself; `timingTotals()` the since-construction sums, successful builds only); `SpriteRenderer::Options` gained `gpuTiming` (opt-in per-pass GPU time: ONE `GL_TIME_ELAPSED` query object created at `create` and reused every frame — no per-frame GL allocation, NO CPU/GPU synchronization (RENDER-005 — unlike the `primitiveQuery` diagnostic's `glFinish`)) + `stallThresholdUs` (documented default `kSpriteRendererDefaultStallUs` = 2000 us ≈ 12% of the default 60 Hz frame; domain `[1, 0xFFFFFFFF]`; the `create` validation, first failure wins, the `options_invalid` warn); `SpriteDrawStats` gained `submitMs` (the submit's CPU wall time), `uploadWallUs` / `uploadStalled` (the stall detection), and `gpuTimeUs` / `gpuTimeValid` (the GPU time is ONE FRAME LATE — frame N reports frame N−1's pass; `gpuTimeValid` false = "n/a" — the first frame, any frame the GPU is more than one frame behind, and every environment without working timing queries — the headless/offscreen fallback; never a fake number; a failed submit poisons only its own in-flight measurement); `SpriteDrawTotals` gained the `submitMs`, `uploadStallUs` / `uploadStallFrames`, and `gpuTimeUs` / `gpuTimeFrames` sums; the stall detection: the pass's only implicit CPU↔GPU sync point is the one `glBufferSubData` into the REUSED instance buffer — its CPU wall time above the threshold sets `uploadStalled` + ONE rate-limited Warn `sprite_renderer/submit_stall` (fields `stall_us`, `upload_bytes`) — observation, never an execution gate (the G-R2 precedent); **profiler feed** — the M1-PROF-01 `Profiler` gained the primitive-typed render feeds (`addRenderBatchMs` / `addRenderSortMs` / `addRenderSubmitMs` / `addRenderGpu` / `addRenderStall` / `setRenderTextureMemory` — `laige-sim` cannot depend on `laige-render` types, ARCH-001, so the render thread feeds the primitive values), feeding the `laige-profile render:` text line, the JSON report's `"render"` object (`gpu_us` is `null` until the first completed measurement), and the summary line's render fields (`render_gpu_us` renders `n/a` on the same rule); the VRAM estimate feeds through the pre-existing `textureMemoryBytes()` gauge; the engine's render-thread wiring lands with the engine integration step (the `addDrawCalls` feed precedent — the headless run feeds no render values); **tests** — NEW `tests/laige-render/render_timing_tests.cpp` (new CTest entry `render_timing` = the step's Verify command; 8 tests / 5 suites; `TIMEOUT 120`, the ASan `detect_leaks=0` + TSan `halt_on_error=1` entries): `RenderTimingCreate` (GL-free: the `stallThresholdUs` create validation — first failure wins, one `options_invalid` warn, the `option`/`value` fields pinned — + the stopped state reads zero), `RenderTimingCpu` (GL: the CPU phase times on the known small scene (10 sprites, 2 atlases, 2 blends → 3 groups), the EXACT since-construction totals accumulation, the empty frame records nothing (only the frame count moves), and the profiler feed wiring), `RenderTimingGpu` (GL: off-by-default "n/a" (no GL work); on: the one-frame-late semantics pinned under `glFinish`-paced frames (frame 1 n/a, frames 2/3 valid with exact totals)), `RenderTimingStall` (GL: a 4096-sprite frame (212 992 B upload) under a 1 us threshold stalls — one Warn per stalled frame with the pinned `stall_us`/`upload_bytes` fields + the stall totals; the 10-sprite frame under the default 2000 us threshold does not (0 warns)), and `RenderTimingSlowShader` (GL: the deliberately slow shader — a 16384-iteration DEPENDENT sin loop over a 128×128 FBO — with the GPU-side `TIME_ELAPSED` pass time EXCEEDING the CPU-side submit time on a hardware-GL runner, the machine-greppable `slow-shader gpu_us=… cpu_us=…` line; SKIPS on a software renderer (llvmpipe/swrast/Mesa) where the CPU is the GPU and the driver is synchronous — the observation is defined for the desktop CI runner); `profiler_tests.cpp` gained the render-timing counter-model tests (the exact accumulation + the "n/a" GPU contract, the disabled no-op preserving state, the greppable text/JSON report surfaces, the zero-alloc record path now covering the 6 render feeds); **docs** (DOC-003/006/007) — `docs/api/sprite_renderer.md` (NEW "Render timing (M2-PROF-01, RENDER-005)" section: the field table, the one-frame-late + "n/a" contract, the stall detection, the profiler feed, the Performance trap), `docs/api/sprite_batcher.md` (the `SpriteBatchTiming`/`timingTotals` fields + the API sketch), `docs/api/profiler.md` (the counter-table row, the summary-line shape, the `laige-profile render:` text line, the JSON `"render"` object, the zero-alloc test line), `src/laige-render/README.md` + `src/laige-sim/README.md` (the status paragraphs; the stale remaining-steps line fixed), `tools/README.md` (the report schema line); **API surface** — `laige-api.json` regenerated LAST (1815 → 1851 symbols, 48 headers); **budget** — no `budgets.json` entry (the count stays 17 — the `BudgetHarnessTable.LoadsTheRepoBudgetsFile` pin): the timing is O(1) bookkeeping (two to four `steady_clock` reads per frame, no allocation, no GL work when `gpuTiming` is off) and the composite 50k render-CPU budget is measured with M2-PERF-01; **local verification** — the 4-tree Linux gate green: `gate-gcc` 117/117, `gate-clang` 117/117, `gate-asan` 112/112, `gate-tsan` 112/112 (the full unfiltered `ctest` minus the 10 GL-creating entries per the GPU local-execution policy — the CI lanes run them unfiltered; `render_timing` joins the GL-creating set), `tools/laige-include-lint` (80 source files) + `tools/laige-determinism-lint` (29 sim source files, 0 violations) green, `api-real-tree`/`api-check-fresh` in the gate; **size note** — beyond the roadmap's ~250-line guidance (the documented contract spans the renderer + batcher + profiler + five docs, per CORE-006/DOC-004); Progress Board M2 29/33 (total 75/194) | +| 2026-10-13 | M0-CI-01 | `—` / PR (open) | **CI follow-up: slow suites moved to label-gated jobs** — measured on the latest master run 38312355303 (2026-10-12, all 11 jobs green): every ctest entry exceeding 30 s in a lane moved out of the main job into a dedicated job of its own (one per lane), run only when gated (the main job's `ctest` excludes it with `-E`, the slow job selects it with `-R`): `laige-sim_tests` (gcc 59.97 / clang 53.79 / asan 124.03 / tsan 159.52 s) and `laige-render_tests` (53.83 / 44.45 / 43.20 / 60.41 s) → new `linux-gcc-slow` / `linux-clang-slow` / `linux-asan-slow` / `linux-tsan-slow` jobs; `particle_render` (gcc 38.76 / clang 34.67 / tsan 39.56 s — asan 28.84 s stays in the main asan job) → the gcc/clang/tsan slow jobs; `zero_alloc` (asan 43.76 / tsan 54.75 s — gcc 16.16 / clang 14.39 s stay in the main jobs) and `ecs_stress` (asan 42.87 / tsan 54.70 s — gcc 16.34 / clang 14.45 s stay) → the asan/tsan slow jobs; `laige_bench_render_50k_fpx16` / `_fp32` (gcc 577.05 / 593.82 s, clang 578.42 / 566.52 s) → new `linux-gcc-bench` / `linux-clang-bench` jobs. No entry exceeds 30 s on windows/macOS (worst: `laige-sim_tests` 27.11 s on macos-intel), so those jobs keep the full suite unchanged. Each slow suite is gated by a repo label in the PR lane (`ci:slow:sim`, `ci:slow:render`, `ci:slow:particle`, `ci:slow:zero-alloc`, `ci:slow:ecs-stress`, `ci:slow:bench-50k` — created via the GitHub API) or a `workflow_dispatch` checkbox in the merge lane (`slow-sim` … `slow-bench-50k`). **Milestone gate** — a milestone-gate PR MUST carry ALL six `ci:slow:*` labels so every slow job runs and passes before merge (AGENTS.md §5, new "Slow CI suites and the milestone gate" section); after the gate merges, a `workflow_dispatch` on master with all six checkboxes re-runs the slow matrix. Timeouts: the main Linux jobs revert to 10 min (gcc/clang 45 and tsan 30 existed only for the moved suites), the slow suite jobs carry 20, the bench jobs 30 (the old 45-min rationale minus the rest of the suite). No engine code change — the two workflows + AGENTS.md §5 + the PRD §14 Perf row + this log line only | ---