Repository navigation
ci: retry release jobs GitHub gave no runner - #97
Merged
Merged
Conversation
The release failed on 2026-11-06, -09 and -10 with every build passing: 2-3 macOS arm64 jobs each time were dropped after ~3 h with "The job was not acquired by Runner of type hosted even after multiple attempts", and all-builds failed the run. This calls the engine's retry.yml on the Release run's completion, which re-runs only the failed jobs (up to two times) when that is the only failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ryan-morris
added a commit
to devenvy/ffmpeg-build
that referenced
this pull request
Nov 10, 2026
…etry runner-less release jobs and mirror outages (#128) ## Summary Fixes found while checking what the release archives actually carry, and why devenvy/ffmpeg's and this repository's CI keep failing. ### 1. Static toolchain runtimes in the SBOM and THIRD-PARTY-NOTICES.txt `angeltrax/ffmpeg` `internal-9.0.2.3` (engine 0.0.22) has x265 (C++) built in. On **linux-musl-x64** `libavcodec.so` has no `NEEDED libstdc++.so` but carries C++ runtime symbols. On **win-x64** the DLLs import no `libstdc++-6.dll` or libgcc DLL. The runtimes are linked statically, but neither the SBOM nor the notices listed them. - **Conditional ships.** A `ships:` entry can carry `linked:` (archive names or driver flags) and `compiler:`. After FFmpeg configure, staging reads `ffbuild/config.mak`. It also runs `$(LD) -shared <LDFLAGS> -###` on one of the build's objects, which prints the libraries and search folders the driver adds itself without linking anything. - **Resolution.** A plain `-l<name>` counts as the archive ld would really pick: the `-L` order, then the driver's folders; shared or import library first unless the link is static; GNU ld scripts are followed. - **Recording rule.** An entry is recorded only where that link shows it; a build without C++ gets no C++ runtime. - **Recorded now:** | Platform | Runtimes | | --- | --- | | musl x64/arm64 | libstdc++, libgcc, LLVM libunwind (`llvm-libunwind-static`; `Apache-2.0 WITH LLVM-exception AND (NCSA OR MIT)`) | | win-x64 (gcc) | libstdc++, libgcc, libwinpthread | | win-x64 (MSYS2 CLANG64) | libc++, libunwind, compiler-rt builtins | | win-arm64 | libc++, libunwind, compiler-rt builtins (llvm-mingw names them on every link) | | linux-x64/arm64 (glibc) | libgcc, and `libstdc++_nonshared.a`: gcc-toolset-14's `libstdc++.so` is a linker script that links it statically wherever `-lstdc++` is used | | linux-armhf | libgcc | - **Versions** are recorded only for runtimes that ship with the compiler (`-dumpfullversion` / `-dumpversion`, confirmed by package ownership in each image), not for separately packaged ones (Alpine libunwind, winpthreads, MSYS2 libc++). - **Notices** come from each toolchain's own files where it has them. Where it doesn't, `license-texts/` holds verbatim upstream texts: GCC 15.2.0 `COPYING3` and `COPYING.RUNTIME`, LLVM 22.1.3 `libunwind/LICENSE.TXT`. - **SBOM relationship:** archive DEPENDS_ON runtime, with the evidence in the comment (as for static recipe libraries). libwinpthread moves from BUILD_DEPENDENCY_OF to that too. - `docs/building.md` lists what the inventory still doesn't cover, and why: Apple's and the NDK's implicit compiler-rt, the NDK's static libunwind, which archive members are actually used. ### 2. Static runtime links that silently went dynamic, fixed in the link environment - configure's own `-lstdc++` (rubberband, openmpt, snappy...) isn't reached by `beforeFfmpeg`'s `.pc` rewrite. On cross win-x64 it produced a DLL importing `libstdc++-6.dll`, and on musl `NEEDED libstdc++.so.6`. Staging rejected both, so these were build failures, not silent gaps. - `beforeFfmpeg` now copies the compiler's own static archives (`-print-file-name`) into `work/static-runtime` and adds `-L` for it, so a plain `-l` resolves to the static archive. No configure output or upstream file is edited. - `-lunwind` becomes `-l:libunwind.a` on musl, and the Windows rewrite follows the compiler: libc++ and libunwind on clang, libstdc++ on gcc. ### 3. `retry.yml`: re-run release jobs GitHub never gave a runner devenvy/ffmpeg's release failed on 2026-11-06, -09 and -10 with every build passing. Each time, 2–3 macOS arm64 jobs were dropped after ~3 h: *"The job was not acquired by Runner of type hosted even after multiple attempts"*. That's hosted macOS capacity (38 macOS jobs, 5 concurrent on the free plan). - The new reusable `retry.yml` is called from a `workflow_run` workflow (`examples/workflows/retry.yml`; devenvy/ffmpeg#97). - When every failed job carries that annotation, it runs `gh run rerun --failed`, up to `max-attempts` (default 3). Any other failure is left alone. - `actions: write`, `checks: read`; it runs no code. ### 4. Toolchain image builds outlast a package-mirror outage Toolchains run 42543300431 failed on `main`: the linux-arm64 image's `dnf install` got an empty `repomd.xml` from an AlmaLinux mirror mid-sync. `ensureImage` retried 3 times within about 13 s and failed. - `retry()` takes its first wait as an option; the 4 s default leaves `fetch.ts` unchanged. - Image builds get 4 tries, about 30 s, 60 s and 120 s apart. `FFMPEG_BUILD_IMAGE_ATTEMPTS` and `FFMPEG_BUILD_IMAGE_RETRY_MS` override them, and the tests use fake timers. - The package managers are unchanged, because none has an option for this failure. dnf's `retries=` covers package downloads only, and `skip_if_unavailable` would drop baseos. ## Testing - `guidelines:check`, `lint`, `format:check`, `typecheck`, `knip`: pass. - `test:ci`: 197 files, 4705 tests pass (3 files skipped, among them the Docker build test). The SBOM set with the official validator (`spdx-tools==0.8.5`): 57 tests pass, 3 macOS-only tests skipped. - Probed in the pinned images: Alpine and cross-windows driver lines; the `libstdc++-6.dll` import and the musl NEEDED entry before the fix; static resolution after it; posix-thread mingw's implicit `-lpthread`; gcc-toolset's linker script. The new logic's unit tests use fixtures built from that probe output. - `retry.yml` dry-run against devenvy/ffmpeg: runs 42846005330, 42665036518 and 42239484169 would be re-run; the successful 42489100542 would not. actionlint is clean; zizmor's only finding is the example's `@v0`, like the other examples. - **Not run:** a full FFmpeg build on any platform; an MSYS2 install (its licence paths come from the PKGBUILDs, and CLANG64 resolution is untested); the linux-arm64 and armhf driver lines. This PR's Toolchains and release CI are the first real builds. - Changing `platforms.yml` and `src/build` changes every toolchain identity, so library caches start cold. ## Not in this PR yet The minimatch alert (Dependabot #1 and code-scanning #88, GHSA-g3hv-75wg-m249) is still open. Overriding minimatch to 11 breaks `lint`: eslint-plugin-import-x 4.17.1 (through `@devenvy/code-guidelines`) imports `makeRe`, which minimatch 11 removed. Decision pending. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ryan-morris
marked this pull request as ready for review
November 10, 2026 23:21
ryan-morris
added a commit
that referenced
this pull request
Nov 11, 2026
## Summary Moves to **ffmpeg-build 0.0.24** (devenvy/ffmpeg-build #128, #129, #130). Every archive's SBOM and `THIRD-PARTY-NOTICES.txt` now cover everything the toolchain puts into the shipped binaries: - statically linked libstdc++/libgcc (musl, Windows), `libstdc++_nonshared` (glibc), LLVM libunwind (musl); - the mingw-w64 C runtime on Windows, with its ZPL notice (previously missing); - glibc's `libc_nonshared.a` and crt objects, and musl's crt objects; - the NDK's and Apple's compiler-rt and libunwind; - the SBOM also lists the system libraries each binary loads (glibc, UCRT, libSystem, ...) as "provided by the operating system, not shipped". The engine also re-runs release jobs that GitHub gave no runner (the 11-06/-09/-10 release failures), and retries toolchain image builds long enough to outlast a package mirror's sync. ## Changes - **All engine workflow pins → v0.0.24** (`85cbcfa`): `ci.yml` (check, automerge), `release.yml`, `update-lock.yml`, and `retry.yml`. #97 had left retry pinned to a pull-request commit until the engine was released. - **`ffmpeg.lock`** from `ffmpeg-build update`: engine 0.0.23 → 0.0.24, no library changes. Every release is therefore due and gets built again with the new engine on the next Release run. ## Testing - `ffmpeg-build check` (0.0.24): passes. - actionlint: clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Release workflow failed on 2026-11-06, -09 and -10 with no build failure. Each time, 2–3 macOS arm64 jobs waited about 3 h and were dropped: "The job was not acquired by Runner of type hosted even after multiple attempts". That's hosted macOS capacity: a release queues 38 macOS jobs on a plan that runs 5 at a time.
all-buildsthen failed the run, and the release waited for the next day's schedule..github/workflows/retry.ymlruns when a Release run completes and calls the engine's new reusableretry.yml(devenvy/ffmpeg-build#128). When every failed job carries that annotation, it re-runs only the failed jobs (gh run rerun --failed), up to two times. Any other failure is left for review. It checks out and runs no code:actions: write,checks: read.Before merging
Draft until devenvy/ffmpeg-build#128 is released. The
uses:pin points at that PR's branch commit (acbec80), so this file can be reviewed and linted now. Re-pin it to the engine release's commit (@<sha> # v0.0.x, likerelease.yml) before merging.Testing
dangerous-triggersis suppressed with its reason: the workflow runs no code).workflow_runonly fires from the default branch.🤖 Generated with Claude Code