Repository navigation
Conversation
Codecov Report❌ Patch coverage is
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 1 file with indirect coverage changes 🚀 New features to boost your workflow:
|
jesup
force-pushed
the
decompress-fse-state-hint
branch
from
November 6, 2026 16:52
d2b7d1e to
305b7a6
Compare
LLVM lowers inline thresholds for code that is cold in a PGO profile, which leaves calls like `Writer::subslice` in the sequence loop. Firefox PGO training does not exercise zstd, so this is the configuration it ships.
PGO builds turned the element-wise `fill` into two byte stores per entry.
LLVM vectorized the per-byte loop on x86-64 only without PGO, and not at all on aarch64.
`ZSTD_execSequence` already guarantees the sequence fits before `oend`, so the per-sequence `Writer` bounds check is redundant. Keep `new_seq_state` out of line: inlining it makes the x86-64 loop allocate registers noticeably worse. On targets other than x86_64, force it inline with the new `pgo` feature: out of line the sequence state stays in memory, which made trained and cold PGO builds up to 10% slower on Apple silicon. Without PGO forcing it inline is slower, so the compiler decides. The long sequence decoder always calls the out of line version.
The 16-byte over-copy relies on the same output slack as wildcopy.
Makes `is_last_sequence` a constant in the loop.
Matches C's BMI2_TARGET_ATTRIBUTE; without lzcnt, `ilog2` in the table builders compiles to bsr plus a fixup.
The mask sits on the state dependency chain; the bounds check does not, and never fails because every table is built by this crate.
…able Not in the Long decoder, where it made register allocation in the loop worse.
jesup
force-pushed
the
decompress-fse-state-hint
branch
from
November 10, 2026 17:08
305b7a6 to
5c794a5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on PR #501 (decompress-codegen-parity); only the last commit is new.
Under the existing
unsafe-performance-experimentalfeature, tell LLVM that an FSE state is always below its table size, which removes the bounds check from the state lookup. This holds for every table this crate builds (decoded, RLE or default);unreachable_uncheckedis checked in debug builds, and decompression was fuzzed with the feature on.The hint is applied on x86_64 only. On Apple silicon, removing the bounds check made sequence decoding 4-8% slower (mostly the offset table lookup), so other targets keep the checked lookup.
The Long decoder (used for freshly loaded dictionaries) keeps the checked lookup: with the hint there, register allocation in its loop got worse (about a third more stack loads and stores executed) and dictionary decoding was 3-5% slower without a profile or with a cold one.
ZSTD_decodeSequencetakes a const parameter for this.Measured on Zen 3 with a harness that decodes the way Firefox does (fresh DStream per response, 16 KB input chunks, 128 KB output buffer), cycles per decompressed byte. "Cold PGO" is instrumented with a profile that never decodes zstd, which is what Firefox ships, since its PGO training has no zstd content; "trained" includes zstd decoding in the training. Feature enabled, against #PR2:
Upstream benchmark (non-PGO, rs/C): silesia-small 0.992 -> 0.980, re2-exhaustive 1.012 -> 1.006, zip64support 0.983 -> 0.971.
On an Apple M-series Mac (wall-clock, same harness), the whole stack through this PR decodes about 9% faster than main without PGO, 13% faster with cold PGO and 8% faster with trained PGO (about 10%, 15% and 9% for streaming non-dictionary inputs). Against C zstd 1.5.8 built with clang -O3 and LTO, that is about 8% slower without PGO, level with cold PGO and 11% slower with trained PGO. With the
pgofeature from the first PR enabled, the stack is 21% faster than main with cold PGO and 14% faster with trained PGO, which is about 9% faster than C and 4% slower than C. This PR itself changes nothing on aarch64, since the hint is x86_64 only.Tests pass with and without the feature, in release and debug.