Repository navigation
Conversation
Codecov Report❌ Patch coverage is
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 1 file with indirect coverage changes 🚀 New features to boost your workflow:
|
jesup
force-pushed
the
decompress-codegen-parity
branch
from
November 6, 2026 16:52
fcfc7fb to
0297d6e
Compare
LLVM lowers inline thresholds for code that is cold in a PGO profile, which leaves calls like `Writer::subslice` in the sequence loop. Firefox PGO training does not exercise zstd, so this is the configuration it ships.
PGO builds turned the element-wise `fill` into two byte stores per entry.
LLVM vectorized the per-byte loop on x86-64 only without PGO, and not at all on aarch64.
`ZSTD_execSequence` already guarantees the sequence fits before `oend`, so the per-sequence `Writer` bounds check is redundant. Keep `new_seq_state` out of line: inlining it makes the x86-64 loop allocate registers noticeably worse. On targets other than x86_64, force it inline with the new `pgo` feature: out of line the sequence state stays in memory, which made trained and cold PGO builds up to 10% slower on Apple silicon. Without PGO forcing it inline is slower, so the compiler decides. The long sequence decoder always calls the out of line version.
The 16-byte over-copy relies on the same output slack as wildcopy.
Makes `is_last_sequence` a constant in the loop.
Matches C's BMI2_TARGET_ATTRIBUTE; without lzcnt, `ilog2` in the table builders compiles to bsr plus a fixup.
The mask sits on the state dependency chain; the bounds check does not, and never fails because every table is built by this crate.
jesup
force-pushed
the
decompress-codegen-parity
branch
from
November 10, 2026 17:13
0297d6e to
9d73ef9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on PR #500 (decompress-dictionaries), which builds on #499; only the last 3 commits are new.
lzcntandbmi1alongsidebmi2in the dynamically dispatched functions, as C'sBMI2_TARGET_ATTRIBUTEdoes; withoutlzcnt,ilog2in the table builders compiles tobsrplus a fixup. The runtime check now also requireslzcnt.u32load instead of three field loads (one instruction less per symbol on aarch64; x86 on unix uses the asm loop).These only pay off once #PR1 is in; on main alone the effect is lost in the cold-code penalty.
Measured on Zen 3 with a harness that decodes the way Firefox does (fresh DStream per response, 16 KB input chunks, 128 KB output buffer), cycles per decompressed byte. "Cold PGO" is instrumented with a profile that never decodes zstd, which is what Firefox ships, since its PGO training has no zstd content; "trained" includes zstd decoding in the training.
Upstream benchmark (non-PGO, rs/C): silesia-small 0.995 -> 0.992, re2-exhaustive 1.022 -> 1.012, zip64support 0.991 -> 0.983.
On an Apple M-series Mac (wall-clock, same harness), this PR and the PRs below it decode about 9% faster than main without PGO, 14% faster with cold PGO and 8% faster with trained PGO (about 10%, 16% and 9% for streaming non-dictionary inputs).
Each commit builds, is rustfmt-clean and passes
cargo test.