fix(deps): ship zsmooth per CPU baseline, so pre-AVX2 machines work (#82) - #88
Merged
Conversation
) Upstream builds zsmooth for an AVX2 baseline with no runtime dispatch, so the bundled binary loaded fine and then died with an illegal instruction (0xC000001D) the instant a filter ran on any pre-2013 x86 CPU. Confirmed by the reporter on a Celeron J4105 and a Core i7 870, neither of which has AVX at all. It hit CCD, Cnr4, SpotLess/RemoveDirt, mClean and TemporalDegrain2; QTGMC was unaffected, since havsfunc uses rgvs. x86 bundles now carry both a haswell and an x86_64_v2 build in vapoursynth/zsmooth/, outside the autoload directory, and the worker loads exactly one by path. They cannot share a directory — each registers the namespace `zsmooth`, so whichever autoloads second is rejected — and VAPOURSYNTH_EXTRA_PLUGIN_PATH takes only one directory, verified. Loading by path rather than under a forcens alias keeps the namespace intact, so no call site changes and the vendored modules stay on the fast build too. Shipping one portable build instead was measured and rejected: x86_64 runs CCD at 0.50x and Cnr4 at 0.33x of haswell (x86_64_v2 0.72x / 0.71x), i.e. 2-3x on the two filters the Chroma Denoise pass is made of, charged to every modern machine. A locally built -Dcpu=haswell matched the shipped binary within 3%, so the comparison is like-for-like. The two builds produce identical chroma, so the fallback costs throughput and nothing else. The portable build has no upstream asset, so Windows and Linux compile it with a pinned Zig toolchain — Windows' first from-source plugin, which needs no MSVC. macOS x64 builds both, which also makes Intel Macs fast for the first time: that arch had always used Zig's default SSE2 baseline. No selected build means no LoadPlugin, so a newer worker still works against a deps bundle from before the split (verified end to end both ways). Also decode Windows NTSTATUS crash codes in the reported error. A bare "exit code -1073741795" was all this report had to go on, and 0xC0000005 (issue #83's CTMF crash) reads almost identically. Guards: test_154/test_155 cover both scripts and both bundle layouts; zsmooth_never_offers_a_build_this_cpu_cannot_run runs on every platform whatever hardware CI draws; the Dart side loads the chosen build and renders a frame, since the fault is in the kernel. deps-expected-plugins.json entries may now be bundle-relative paths, understood by all three packaging guards. Needs deps 1.10.0 published before merge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… zsmooth Two blockers for the deps 1.10.0 build, found by running it. FFmpeg had silently diverged across the bundles, and had been that way for months: Windows master N-125978 (post-9.0) BtbN master-latest, UNPINNED macOS x64 9.0.1 evermeet getrelease, UNPINNED macOS arm64 9.0.1 martin-riedl latest, UNPINNED Linux 7.1, two majors behind BtbN n7.1, pinned FFmpeg is what interprets everything in pipeline_executor.rs — the accumulated -vf chain, the colour flags, -ss/-frames:v trimming, the hardware-encoder options — so a per-OS FFmpeg is a per-OS encoder. All four now pin 9.0, which is where three of them already were. The Linux pin is also what broke: BtbN publish every series to one rolling `latest` tag and drop old ones as they age, so n7.1 became a 404 and the deps build died at `tar: Error is not recoverable`. It read as a tar bug because curl had no -f and wrote the nine-byte "Not Found" body into the tarball. Every FFmpeg fetch now uses -f, and each script runs `ffmpeg -version` on what it actually installed and fails if it is not the pinned series — pinning alone cannot stop an upstream moving a URL. ffmpeg_version_pin_test.dart fails the push gate if the three pins disagree, if one stops being a bare major.minor, or if a script stops asserting; without it the pins are only comments. The parser rejects a `master` build outright, so reverting to an unpinned URL is red. Separately, the macOS x64 haswell zsmooth added in the previous commit did not compile. zsmooth's Zig fftw port sets HAVE_MEMALIGN on every non-Windows target, but macOS has no memalign() — it is declared in <malloc.h>, which the same file already knows macOS lacks (HAVE_MALLOC_H is gated on !is_mac). kalloc.c only reaches that branch at MIN_ALIGNMENT 32, i.e. with AVX enabled, so it is invisible at every SSE-level baseline and killed only the haswell build. HAVE_POSIX_MEMALIGN is already true, so clearing it falls through to posix_memalign. Patched by cloning the fftw fork at the ref build.zig.zon names and repointing it as a path dependency, which takes no hash and so survives a cache wipe; pre- and post-checks hard-fail so an upstream fix surfaces as an error telling you to drop the patch. Verified by cross-compiling from Windows (-Dtarget=x86_64-macos.12.0 -Dcpu=haswell): reproduces the failure exactly, and builds clean with the patch. The target does not have to be the host, which is far cheaper than a macOS CI round trip. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The interpolated path "$FullTargetDir\ffmpeg\ffmpeg.exe" reached the file with its \f collapsed into a literal form feed (0x0C), so the check ran against "$FullTargetDir<FF>fmpeg<FF>fmpeg.exe" and the deps build died immediately after a successful download. Join-Path instead: no backslashes to escape, so the whole class of corruption is gone rather than fixed once. Verified by running the real section against a scratch target — it fetches n9.0, installs, and reports "FFmpeg 9.0 installed (matches the pinned 9.0 series)" for n9.0.1-11-ge47273f4d9. That harness is also what caught the git 2>&1 bug earlier; PowerShell sections in this script are worth running before pushing, because a deps build is a slow way to find a typo. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The sed backreference in assert_ffmpeg_series arrived in the file as a literal 0x01, so the check read an empty version and failed every Linux and macOS deps build with "FFmpeg is but this bundle pins 9.0" — after a download that had worked. Same class as the form feed in the Windows path check: escapes mangled in transit rather than a logic error. Rewritten with grep -oE and a [.] bracket expression, so the parser contains no backslashes at all and there is nothing left to mangle. Exercised against real binaries rather than reasoned about: a BtbN n9.0.1 build passes, the unpinned master build in deps/ is rejected (it reports N-125978-..., which is exactly the drift this is meant to catch), and a missing binary is rejected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #82.
The bug
Upstream builds zsmooth for an AVX2 baseline with no runtime dispatch (only
haswellandznver4assets exist for x86), so the bundled binary loads fine and then dies with an illegal instruction the instant a filter runs on any pre-2013 x86 CPU.Confirmed by the reporter on a Celeron J4105 and a Core i7 870, neither of which has AVX at all. The symptom was a bare
vspipe exited with exit code -1073741795—0xC000001DSTATUS_ILLEGAL_INSTRUCTION, distinct from the0xC0000005access violation of the CTMF crash in #83. vspipe prints nothing on a native crash, so nothing anywhere named zsmooth.Affected everything reaching
core.zsmooth.*: CCD, Cnr4, SpotLess → RemoveDirt, Noise Reduction → mClean and TemporalDegrain2, and hybrid_mv. QTGMC was not affected — havsfunc usesrgvs.linux-x64took the same haswell asset and had the same bug.The fix
x86 bundles now ship both builds in
vapoursynth/zsmooth/, outside the autoload directory, and the worker loads exactly one by path.zsmooth, so whichever autoloads second is rejected.VAPOURSYNTH_EXTRA_PLUGIN_PATHtakes only one directory (verified: a;-separated pair silently loads just the first), and on macOS/Linuxvapoursynth/pluginsis autoloaded implicitly by R78 anyway.forcensalias, so the namespace stayszsmoothand no call site changed. An alias would have needed a_zs()indirection throughmclean.py,removedirt.py,temporaldegrain2.pyandhybrid_mv.py— and left AVX2 users on the slow build in all four.LoadPlugin, so a newer worker still works against a deps ≤ 1.9.0 bundle that autoloads a single zsmooth. The app can be upgraded before the deps download finishes, so that window is real.Why not just ship one portable build
Measured (Zig 0.15.2, same source, 720x576, fps, best of 3), as a fraction of haswell speed:
x86_64x86_64_v2That is 2–3x on the two filters the Chroma Denoise pass is made of, charged to every modern machine to serve the rare old one. Ratios hold at 1080p and multithreaded. A locally built
-Dcpu=haswellmatched the shipped binary within ±3%, so this is like-for-like and not a toolchain artefact.x86_64_v2is the fallback rather than plainx86_64: 1.4–2.1x faster on the filters that matter, 0.6 MB smaller, and it covers everything from Nehalem (2009) on — including both CPUs in the report.The two builds produce identical chroma, so falling back costs throughput and nothing else.
Side effects
format_exit_status:0xC000001Dnow reads "illegal instruction: a plugin used a CPU feature this machine does not have" instead of-1073741795.Verification
2>&1ongit cloneunder$ErrorActionPreference = "Stop"turns ordinary progress into a terminating error in PowerShell 5.1. That would have been a red deps build.Guards:
test_154/test_155(both scripts, both bundle layouts),zsmooth_never_offers_a_build_this_cpu_cannot_run(runs on every platform whatever hardware CI draws), and a Dart test that loads the chosen build and renders a frame — the fault is in the kernel, so constructing the node proves nothing.Merge order
Needs deps 1.10.0 published before merge —
deps-version.jsonalready points at it.🤖 Generated with Claude Code
Also in this PR: FFmpeg pinned to 9.0 everywhere
Not planned — found by running the deps build. Two blockers surfaced, and both had to be fixed here because deps 1.10.0 cannot be built without them.
FFmpeg had silently diverged across the bundles
master-latest, unpinnedgetrelease, unpinnedlatest, unpinnedn7.1, pinnedFFmpeg is what interprets everything in
pipeline_executor.rs— the accumulated-vfchain, the colour metadata flags,-ss/-frames:vtrimming, the hardware-encoder options. A per-OS FFmpeg is a per-OS encoder, so the same job could encode differently depending on where it ran. All four now pin 9.0, which is where three of them already were.The Linux pin is also what broke the build: BtbN publish every series to one rolling
latesttag and drop old ones as they age, son7.1became a 404. It surfaced astar: Error is not recoverablebecausecurlhad no-fand wrote the nine-byteNot Foundbody into the tarball.A version pin against a rolling tag is not a pin — n7.1 was always going to 404, it was only a question of when. The fix pins the series (
n9.0-latest), which is how all three upstreams actually publish.Two guards, both needed:
assert_ffmpeg_seriesin each script runsffmpeg -versionon what was actually installed and fails if it isn't the pinned series. Pinning alone can't stop an upstream moving a URL.app/test/ffmpeg_version_pin_test.dart(push gate) fails if the three pins disagree, if one stops being a baremajor.minor, or if a script stops asserting. Without it the pins are just comments.The parser rejects a
masterbuild outright, so reverting any platform to an unpinned URL is a red build rather than a silent regression.macOS x64 haswell zsmooth didn't compile
The build added in the first commit failed inside zsmooth's Zig FFTW port:
The port sets
HAVE_MEMALIGNon every non-Windows target, but macOS has nomemalign()— it's declared in<malloc.h>, which the same file already knows macOS lacks (HAVE_MALLOC_His gated on!is_mac).kalloc.conly reaches that branch atMIN_ALIGNMENT32, i.e. with AVX enabled — so it's invisible at every SSE-level baseline and killed only the haswell build.HAVE_POSIX_MEMALIGNis already true, so clearing it falls through toposix_memalign.Patched by cloning the FFTW fork at the ref
build.zig.zonnames and repointing it as a path dependency — path deps take no hash, so it survives a cache wipe, unlike editing Zig's global package cache. Pre- and post-checks hard-fail, so an upstream fix surfaces as an error telling you to drop the patch rather than silently doing nothing.Verified by cross-compiling from Windows (
-Dtarget=x86_64-macos.12.0 -Dcpu=haswell): reproduces the failure exactly, and builds clean with the patch. The target doesn't have to be the host — much cheaper than a macOS CI round trip.