[jesup]

When the decoder is cold in a PGO profile, LLVM lowers its inline thresholds, and the Rust port suffers much more than C because it relies on many small helpers being inlined (for example, Writer::subslice ends up as an out-of-line call in the sequence loop). The table builders also lose vectorization of their element-wise loops under PGO.

• Force-inline the small helpers on the hot paths (Writer, BIT_DStream_t, ZSTD_copy16, ZSTD_fseState::new).
• HUF/FSE table builders: explicit wide stores instead of fill(), fewer bounds checks, HUF_rescaleStats 8 bytes at a time, no ilog2 zero check per entry.
• A raw output pointer in the sequence loops; ZSTD_execSequence already guarantees the sequence fits before oend.
• A new pgo feature that forces new_seq_state inline on targets other than x86_64 (the long sequence decoder keeps the out of line version). Out of line, the sequence state stays in memory instead of registers, which is slower in PGO builds on aarch64. It is slower without PGO, so it is opt-in: enable it for PGO builds. x86_64 is unaffected and always keeps it out of line.

Measured on Zen 3 with a harness that decodes the way Firefox does (fresh DStream per response, 16 KB input chunks, 128 KB output buffer), cycles per decompressed byte. "Cold PGO" is instrumented with a profile that never decodes zstd, which is what Firefox ships, since its PGO training has no zstd content; "trained" includes zstd decoding in the training.

	main	this PR
web resources, cold PGO	3.47	3.08 (-11%)
small responses (~2.6 KB), cold PGO	7.90	6.34 (-20%)
silesia level 19, cold PGO	3.84	3.48 (-9%)
small responses, trained PGO	7.28	5.94 (-18%)
web resources, trained PGO	3.02	2.97 (-2%)

Upstream benchmark (non-PGO, rs/C cycles): silesia-small 1.053 -> 1.016, re2-exhaustive 1.034 -> 1.013, zip64support 1.029 -> 1.001.

On an Apple M-series Mac (wall-clock, same harness), this PR alone decodes about 3% faster than main without PGO, 9% faster with cold PGO and 3% faster with trained PGO. On top of the whole stack, enabling pgo makes streaming decompression of non-dictionary data about 11% faster with cold PGO and 10% faster with trained PGO, and about 4% slower without PGO.

Each commit builds, is rustfmt-clean and passes cargo test; the tip also passes debug tests.

[codecov]

Codecov Report

❌ Patch coverage is 92.80000% with 9 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
lib/decompress/zstd_decompress_block.rs	83.33%	9 Missing ⚠️

Flag	Coverage Δ
test-aarch64-apple-darwin	50.80% <92.80%> (+0.06%)	⬆️
test-aarch64-unknown-linux-gnu	49.38% <92.80%> (+0.07%)	⬆️
test-i686-unknown-linux-gnu	49.17% <92.80%> (+0.04%)	⬆️
test-x86_64-unknown-linux-gnu	50.55% <92.80%> (+0.05%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
lib/common/bitstream.rs	98.35% <ø> (-0.42%)	⬇️
lib/common/fse_decompress.rs	94.73% <100.00%> (+0.09%)	⬆️
lib/common/zstd_internal.rs	86.66% <ø> (ø)
lib/decompress/huf_decompress.rs	84.03% <100.00%> (+0.64%)	⬆️
lib/decompress/zstd_decompress_block.rs	77.21% <83.33%> (+0.13%)	⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[michielp1807]

It seems miri has detected some UB here, which I believe is caused by 15c8b70 because of reads past nbSymbols in huffWeight

[jesup]

Thanks, fixed. Ran miri across all the PRs I have up as well

[folkertdev]

This has a variety of kind of unrelated changes, some more controversial than others. I think to merge this I'd like to see it split up basically per-commit so we can test, benchmark, and review it.