Repository navigation
Conversation
A --branch clone misses the pipeline SHA once the branch moves on, and its exit 128 was retried twice for nothing. GitLab serves any SHA (Gitaly sets uploadpack.allowAnySHA1InWant), so fetch the pipeline commit directly.
The package "prepare code" already uses it; it drops the apt update and getcomposer.org fetch. An apt hang there delayed every tracer test job by 59 min (pipeline 144816042).
A stalled deb.debian.org fetch hung apt-get update until the 1 h job timeout (4 jobs, 60 min each). Cap each apt-get call at 300 s with a 30 s HTTP timeout; the existing exit 75 paths then retry the job.
Octane's "auto" starts one worker and one task worker per node vCPU, so 128-vCPU CI nodes boot 256 Laravel workers and OOM the 6Gi job (6/6 there, 981/981 pass at <=96). Use 2 workers and 1 task worker, like the Swoole and RoadRunner harnesses.
Run the appsec test cmake and make xtest through run-with-retryable-download.sh, as build-appsec.sh does. Fetch the FrankenPHP tarball with download-file.sh and build it as its own script line, so its exit 75 reaches GitLab.
choco exits 0 after a feed error ("installed 0/0 packages"), and the job then failed with exit 1 on the missing php (job 2129816747).
Exit 75 instead, which the default retry covers.
The gradle builds run the appsec CMake FetchContent outside build-appsec.sh, so a Depot 503 failed with exit 1 (job 2137703728). Run the gradle test calls of .appsec_integration_tests and helper-rust integration coverage through run-with-retryable-download.sh.
bash 4.2 turns any errexit status inside the runner's eval into 1, so verify centos lost install.sh's exit 75. End that line, and the libdatadog-override git steps (128), with `|| exit $?`, as #4318 did for compile appsec.
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 1b0ddcf | Docs | View more details | Give us feedback! |
The mirror's composer:2 is Composer 2.5.1, which stalls behind the proxy and never retries curl error 28 (job 2148933797: 300 s). Both prepare code jobs now use composer:2.10.3 pinned by digest, so the package job no longer self-updates; make generate output is byte-identical.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reliability quick fixes for observed CI infra failures (Track R). One commit per item:
$CI_COMMIT_SHA(depth 50) instead of cloning--branch. Once a branch moved, the checkout exited 128 and was retried twice for nothing (42 attempts in 50 branch pipelines).composer self-updateand no getcomposer.org fetch. The mirror'scomposer:2is 2.5.1, which stalls behind the proxy and never retries curl error 28. The tracer job also drops its apt step; one apt hang there delayed every tracer test job by 59 min._generated_*.phpis byte-identical.timeout 300, 30 s HTTP timeout). Stalled mirrors used to hang until the 1 h job timeout; now they exit 75 and retry.cmakeandmake xtestlines and the appsec gradle test calls run throughrun-with-retryable-download.sh, which also retries libddwaf download failures inmake xtest.install-frankenphp.shfetches the tarball withdownload-file.shand now runs as its own script line, so its 75 reaches GitLab.autobooted 2 × node vCPUs Laravel workers and OOMed the 6Gi job on 128-vCPU nodes.|| exit $?to the verify centos line and the libdatadog-override git steps. bash 4.2 turns an errexit status into 1, which loses the 75 and 128 exit codes.To check in this PR's pipeline:
Fetching <sha>.--workers=2 --task-workers=1and 3 Laravel worker PIDs.