You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
MG critical path: x86-main 55 min and shard 2 54 min vs shard 3 37 min (run 39301042731) #5769
Owner: aprender-a3 (routes the workflow edit; needs a non-flow author and a quorum, since ci/sections.yml content is gate content). Measured by aprender-flow from the job API.
MG 39301042731 (merge_group, #5765's batch), created 18:15:48Z, success 19:12:31Z = 57 min.
Queue wait: ~0 (every job started within 4 s of creation; runners intel-clean-room-4/9/11/14, gx10-pool3). Caveat: started_at has been unreliable for queued jobs, but the 37-54 min job durations below leave no room for a long wait.
workspace-test-shard (3): 36.9 min. (1): 40.8 min. (2): 54.2 min. x86-main: 55.2 min. Tail after the heavy jobs: workspace-test 0.3, determinism 1.3, gate 0.1 = ~2 min.
Critical path = x86-main (55.2) ~ shard (2) (54.2). Shard 3 sits idle for ~17 min before the fan-in.
Five whys: why 57 min? the slower of shard 2 / x86-main ends at ~55 min. Why shard 2 slow? hash:N/3 partitions by test name, not cost; shard 2 got the heavier tests (17 min over shard 3). Why x86-main 55? it runs the whole x86 build/gate section in parallel with the shards, and nobody has timed its steps. Why not seen? per-step times exist only for shard 1.
Countermeasure (do both; each is a one-line workflow change): (a) rebalance the partition by measured cost (move the heavy steps/tests from shard 2 to shard 3); (b) time x86-main per step and shorten it. Rebalancing alone gains ~0: x86-main is co-critical at 55 min, so (b) first. Expected: 57 -> ~45 min if both land. Falsifier: next 3 MG runs <= 47 min. Related: #5394, #5764, p2-shard-split-proposal.
Per-section times (x86-main job logs, 3 MG runs 16:14Z / 17:16Z / 18:16Z, seconds): guard-cargo 3361 / 3160 / 3048 (51-56 min = the whole job, the long pole); guard-tree 2063 / 2108 / 2290; determinism[X64] 1454 / 1509 / 1781; sov.lint ~1450-1494; sov.test ~1418-1470; vendored-schemas 267-483; sov.security 61-85; sov.gate / sov.provenance <30. Sections run in parallel, so job time = guard-cargo. Splitting guard-cargo in two halves (or moving it off the critical path) leaves guard-tree (34-38 min) as the next pole: saves ~15-17 min per MG run. This is the first target, ahead of the shard rebalance.
Activity
noahgift commentedon Oct 19, 2026
Per-section times (x86-main job logs, 3 MG runs 16:14Z / 17:16Z / 18:16Z, seconds): guard-cargo 3361 / 3160 / 3048 (51-56 min = the whole job, the long pole); guard-tree 2063 / 2108 / 2290; determinism[X64] 1454 / 1509 / 1781; sov.lint ~1450-1494; sov.test ~1418-1470; vendored-schemas 267-483; sov.security 61-85; sov.gate / sov.provenance <30. Sections run in parallel, so job time = guard-cargo. Splitting guard-cargo in two halves (or moving it off the critical path) leaves guard-tree (34-38 min) as the next pole: saves ~15-17 min per MG run. This is the first target, ahead of the shard rebalance.
ci(guard-cargo): rebalance shard b on two runs' mean (A 1523 s / B 15…
ci(guard-cargo): count --shard calls in claims.rs; look the shard fil…