From 4ea03764d7a6563154cf5bd861f113dc56c0d4a5 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sat, 7 Nov 2026 12:59:19 +1100 Subject: [PATCH 1/4] perf(byte-storage): write inputs under 128 B as a literals-only block The LZ4 encoder zeroes its 8 KiB hash table on every call and then searches for matches; below 128 B a cached value rarely holds a repeat, so both bought nothing. On the bench corpus the envelope grows 0.2% on average from 64 B to 127 B, and a 64 B store drops from 10,398 to 515 Ir/op. Adds store/retrieve/retrieve_data cases at 128 B, the first size the encoder takes, and ratchets the stale Ir budgets. --- README.md | 22 ++++++++++++------- SECURITY.md | 4 ++-- benches/perf_ir.rs | 7 +++--- benches/perf_ir_baselines.json | 33 +++++++++++++++------------- src/byte_storage.rs | 38 +++++++++++++++++++++------------ tests/compress_bound_32_bits.rs | 2 +- 6 files changed, 63 insertions(+), 43 deletions(-) diff --git a/README.md b/README.md index 6dd3456..4c89bc0 100644 --- a/README.md +++ b/README.md @@ -341,7 +341,7 @@ Single-threaded time per call of cachekit-core's own operations, and the payload | `encrypt_aes_gcm` | 526 ns · 1.95 GB/s | 13.79 µs · 4.75 GB/s | | `decrypt_aes_gcm` | 568 ns · 1.80 GB/s | 13.26 µs · 4.94 GB/s | -- `store` compresses with LZ4 (an input under 64 bytes is written as one literals-only LZ4 block, without running the compressor; with a format of at most 255 bytes it goes straight into the envelope, so its store allocates only the returned bytes; from 64 bytes to under 4 KiB the compressor reuses one 8 KiB hash table and one 4,524-byte output buffer per thread, kept for the life of the thread), checksums with xxHash3-64 and wraps both in the MessagePack envelope. `retrieve` checks the envelope, decompresses and verifies the checksum. +- `store` compresses with LZ4 (an input under 128 bytes is written as one literals-only LZ4 block, without running the compressor; with a format of at most 255 bytes it goes straight into the envelope, so its store allocates only the returned bytes; from 128 bytes to under 4 KiB the compressor reuses one 8 KiB hash table and one 4,524-byte output buffer per thread, kept for the life of the thread), checksums with xxHash3-64 and wraps both in the MessagePack envelope. `retrieve` checks the envelope, decompresses and verifies the checksum. - `encrypt_aes_gcm` and `decrypt_aes_gcm` are `ZeroKnowledgeEncryptor`'s AES-256-GCM calls with a 34-byte AAD. - The payload is MessagePack-encoded records (`benches/common/mod.rs`), the corpus `hot_path` and `perf_ir` also use. LZ4 compresses it to 0.57 of its size at 1 KiB and to 0.38 at 64 KiB. - Each figure is the median of 21 timed batches, from one run on the CPU named. Two runs of this code on the EPYC 7763 agreed within 2% on every cell but one (`decrypt_aes_gcm` at 64 KiB, 23%); three arm64 runs varied by up to 30% on a cell. @@ -371,7 +371,7 @@ The Criterion suite in `benches/hot_path.rs` needs the `encryption` feature (`ma | `tenant_keyring_decrypt/` | `TenantKeyring::decrypt_indexed`, key derived once at `for_tenant` | | `hkdf` | `derive_domain_key` (takes no payload, so it has no size) | -Sizes are 64 B, 1 KiB and 64 KiB. `store`, `retrieve` and `retrieve_data` also run at 40 B, below the 64-byte literals-only block threshold, and at 1 MiB. `ByteStorage` switches to its copying allocation pattern once the envelope or the compressed payload reaches 64 KiB. The 64 KiB payload compresses to a 24,702-byte envelope and never gets there; the 1 MiB one compresses to 383,893 bytes. One case at a time: `make perf-ir ARGS="--case store/1024"`. The bench binary is built with fat LTO, so its cases shape each other's code. Because both `retrieve` and `retrieve_data` call the envelope decode, it stays out of line, and `retrieve` costs 26 to 27 Ir/op more at 40 B to 1 KiB (2.8% at 40 B, rustc 1.99.1) than in a binary that calls `retrieve` alone. A consumer that calls only one of the two gets the decode inlined. +Sizes are 64 B, 1 KiB and 64 KiB. `store`, `retrieve` and `retrieve_data` also run at 40 B and at 128 B, the first size the LZ4 encoder takes (40 B and 64 B sit below the 128-byte literals-only block threshold), and at 1 MiB. `ByteStorage` switches to its copying allocation pattern once the envelope or the compressed payload reaches 64 KiB. The 64 KiB payload compresses to a 24,702-byte envelope and never gets there; the 1 MiB one compresses to 383,893 bytes. One case at a time: `make perf-ir ARGS="--case store/1024"`. The bench binary is built with fat LTO, so its cases shape each other's code. Because both `retrieve` and `retrieve_data` call the envelope decode, it stays out of line, and `retrieve` costs 26 to 27 Ir/op more at 40 B to 1 KiB (2.8% at 40 B, rustc 1.99.1) than in a binary that calls `retrieve` alone. A consumer that calls only one of the two gets the decode inlined. **Method** (`benches/perf_ir.rs`): each case runs in its own process under `valgrind --tool=cachegrind --cache-sim=no`, once with N calls and once with none, after the same setup and one warm-up call. N is 1,000, or 20 for a 1 MiB case. Ir/op is `(Ir[N] - Ir[0]) / N`, so process start, setup and exit cancel. The counts are exact, so N sets the run time, not the precision. The measured process gets an empty environment, so neither the shell nor a `VALGRIND_OPTS` changes what is counted. @@ -469,12 +469,17 @@ it. Measure as [Instruction counts](#instruction-counts) describes, and put the pull request. **Give small values their own path.** Most cached values are small, and a fixed per-call cost -dominates them. Below 64 B, `store` writes one literals-only LZ4 block instead of running the -compressor, which zeroes its hash table on every call: −80.8% instructions on a 63 B store -([#128]). The trade: compressible inputs under 64 B are stored uncompressed and grow. From 64 B -to under 4 KiB, the compressor reuses one hash table and one output scratch per thread: −27.5% on -a 64 B store ([#144]). `make perf-ir` has no case below 64 B or above 64 KiB, so a change that -branches on size also measures each side of its threshold and 1 MiB, as [#128] and [#144] did. +dominates them. Below 128 B, `store` writes one literals-only LZ4 block instead of running the +compressor, which zeroes its 8 KiB hash table on every call and then searches for matches: −80.8% +instructions on a 63 B store ([#128]), and −95.0% instructions and about −81% wall time on a +64 B store when the threshold rose from 64 B to 128 B ([#156]). The trade: compressible inputs +under 128 B are stored uncompressed and grow. The threshold sits where that trade starts to cost: +on the bench corpus a value under 128 B rarely holds a repeat, and its envelope grew 0.2% on +average, while the literal block would cost 4% at 144 B and 28% at 256 B. From 128 B to under +4 KiB, the compressor reuses one hash table and one output scratch per thread: −27.5% on a 64 B +store ([#144]), which then still took that path. `make perf-ir` measures `store`, `retrieve` and +`retrieve_data` at 40 B, 64 B, 128 B, 1 KiB, 64 KiB and 1 MiB, so a change that branches on a +size threshold adds a case on any side of it those sizes miss, as [#128] and [#144] did. Per-thread reuse has its own traps, all from [#144]: - Reuse stopped paying at about 6 KiB, because the copy out grows with the block while the saving @@ -550,6 +555,7 @@ change. [#131] changed `store` and [#132] `retrieve`. [#144] says wasm32 was not [#131]: https://github.com/cachekit-io/cachekit-core/pull/131 [#132]: https://github.com/cachekit-io/cachekit-core/pull/132 [#144]: https://github.com/cachekit-io/cachekit-core/pull/144 +[#156]: https://github.com/cachekit-io/cachekit-core/pull/156 [cachekit-rs#185]: https://github.com/cachekit-io/cachekit-rs/pull/185 ## License diff --git a/SECURITY.md b/SECURITY.md index d04f8d4..644a27b 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -55,7 +55,7 @@ This crate protects against: This crate does **not** protect against: - Side-channel attacks on the host system -- Payload bytes left in process memory: `store` compresses inputs from 64 B to under 4 KiB through a per-thread output buffer (4,524 B) that is not cleared between calls, so it holds bytes of earlier payloads for the life of the thread, as freed buffers do until they are reused +- Payload bytes left in process memory: `store` compresses inputs from 128 B to under 4 KiB through a per-thread output buffer (4,524 B) that is not cleared between calls, so it holds bytes of earlier payloads for the life of the thread, as freed buffers do until they are reused - Compromise of the master key - Denial of service via resource exhaustion (partial protection only) - Attacks requiring physical access @@ -113,7 +113,7 @@ the multiply, and a release build panicked when the compressed output outgrew the wrapped bound. This crate requires 0.13, which computes the bound in `u64`; the `wasm32` CI job stores and retrieves an input one byte past that boundary. -Below 64 bytes, `StorageEnvelope::new` does not call the encoder at all. It +Below 128 bytes, `StorageEnvelope::new` does not call the encoder at all. It writes the input as one literals-only LZ4 block, which every LZ4 block decoder reads. diff --git a/benches/perf_ir.rs b/benches/perf_ir.rs index d0e26da..bfad6ad 100644 --- a/benches/perf_ir.rs +++ b/benches/perf_ir.rs @@ -76,13 +76,14 @@ const MIB: usize = 1024 * 1024; /// A small value, a typical value, a large object (as `hot_path`). const SIZES: &[usize] = &[64, 1024, 64 * 1024]; /// ByteStorage picks its strategy by size, so store and retrieve (and -/// retrieve_data, which decodes the same envelope) also measure -/// 40 B, under the literals-only block threshold (64 B), and 1 MiB. Its +/// retrieve_data, which decodes the same envelope) also measure 40 B, 128 B +/// and 1 MiB. 40 B and 64 B sit under the literals-only block threshold +/// (128 B), 128 B is the first size the LZ4 encoder takes. Its /// large-payload switch compares the envelope (store) or the compressed /// payload (retrieve) with 64 KiB: the 64 KiB payload stores as a /// 24,702-byte envelope and never reaches it, the 1 MiB one as 383,893 bytes, /// past glibc's default mmap threshold too. -const STORAGE_SIZES: &[usize] = &[40, 64, 1024, 64 * 1024, MIB]; +const STORAGE_SIZES: &[usize] = &[40, 64, 128, 1024, 64 * 1024, MIB]; /// Operations and their payload sizes. HKDF derives from the master key and /// tenant id alone, so it has one case with no size. diff --git a/benches/perf_ir_baselines.json b/benches/perf_ir_baselines.json index 39190b2..ec4e12b 100644 --- a/benches/perf_ir_baselines.json +++ b/benches/perf_ir_baselines.json @@ -18,21 +18,24 @@ "prescan/1024": 910, "prescan/64": 815, "prescan/65536": 864, - "retrieve/1024": 7409, - "retrieve/1048576": 6679353, - "retrieve/40": 2637, - "retrieve/64": 2630, - "retrieve/65536": 391975, - "retrieve_data/1024": 7243, - "retrieve_data/1048576": 6679184, - "retrieve_data/40": 2471, - "retrieve_data/64": 2464, - "retrieve_data/65536": 391809, - "store/1024": 22934, - "store/1048576": 11912874, - "store/40": 1186, - "store/64": 10551, - "store/65536": 772704, + "retrieve/1024": 5739, + "retrieve/1048576": 6677798, + "retrieve/128": 1304, + "retrieve/40": 1005, + "retrieve/64": 1007, + "retrieve/65536": 390383, + "retrieve_data/1024": 5573, + "retrieve_data/1048576": 6677632, + "retrieve_data/128": 1138, + "retrieve_data/40": 839, + "retrieve_data/64": 841, + "retrieve_data/65536": 390217, + "store/1024": 22781, + "store/1048576": 11912713, + "store/128": 11082, + "store/40": 533, + "store/64": 515, + "store/65536": 772551, "tenant_keyring_decrypt/1024": 5608, "tenant_keyring_decrypt/64": 2623, "tenant_keyring_decrypt/65536": 235224 diff --git a/src/byte_storage.rs b/src/byte_storage.rs index c065011..a6ef771 100644 --- a/src/byte_storage.rs +++ b/src/byte_storage.rs @@ -72,17 +72,27 @@ const MAX_COMPRESSION_RATIO: u64 = 1000; pub(crate) const LARGE_PAYLOAD: usize = 64 * 1024; /// Inputs shorter than this are written as one literals-only LZ4 block instead -/// of going through `lz4_flex::compress`. +/// of going through the LZ4 encoder. /// -/// `lz4_flex::compress` zeroes an 8 KiB hash table on every call, which costs -/// more than the whole store for a tiny value, and below LZ4's 13-byte minimum -/// its output is literals-only anyway. The price is stored size: a compressible -/// input under the threshold is no longer shrunk; its block is the input plus -/// 1 byte (plus 2 from 15 bytes up). Compressed bytes are not canonical -/// (protocol `spec/wire-format.md`, "Compressed-byte reproducibility"), and -/// every reader decodes a literals-only block. +/// The encoder zeroes an 8 KiB hash table on every call, even the reused one +/// (`compress_into_with_table` clears it), then searches for matches. For a +/// small value both cost more than the rest of the store, and below LZ4's +/// 13-byte minimum its output is literals-only anyway. The price is stored +/// size: a compressible input under the threshold is no longer shrunk; its +/// block is the input plus 1 byte (plus 2 from 15 bytes up). Compressed bytes +/// are not canonical (protocol `spec/wire-format.md`, "Compressed-byte +/// reproducibility"), and every reader decodes a literals-only block. +/// +/// The threshold sits where that price starts. On the bench corpus (a stream +/// of MessagePack records about 130 bytes long, `benches/common`), a value +/// under 128 bytes rarely repeats anything LZ4 can match: its envelope was +/// 0.2% bigger on average as a literal block, while the store took about a +/// fifth of the encoder path's wall time from 64 to 127 bytes (515 against +/// 10,398 instructions at 64 bytes, `make perf-ir`). From the second +/// record on, the encoder finds matches: the literal block would cost 4% at +/// 144 bytes and 28% at 256, so the encoder keeps those. #[cfg(feature = "compression")] -const LITERAL_BLOCK_THRESHOLD: usize = 64; +const LITERAL_BLOCK_THRESHOLD: usize = 128; // The literal block writes the length in at most one extra byte, which holds // for literal lengths up to 15 + 254. @@ -211,7 +221,7 @@ fn compress_with_thread_table(data: &[u8]) -> Result, ByteStorageError> // `retrieve`, `retrieve_data` and `validate` decode into its private borrowing twin, // `borrowed::StorageEnvelope`, whose `decode_one_pass` hard-codes this field // order and these encodings. `store` encodes from another, `EnvelopeRef`, -// and writes the same bytes by hand for an input under 64 bytes with a format +// and writes the same bytes by hand for an input under 128 bytes with a format // of at most 255 bytes (`encode_literal_envelope`, sized by // `envelope_encoded_len`). Change all of them together; // `literal_envelope_matches_rmp_serde`, `encoded_len_matches_rmp_serde` and @@ -319,7 +329,7 @@ impl StorageEnvelope { /// `data`'s LZ4 block and checksum, with the input and block size checks. /// `StorageEnvelope::new` and `store`'s general path call it; the -/// sub-64-byte envelope `encode_literal_envelope` writes is too small to need +/// sub-128-byte envelope `encode_literal_envelope` writes is too small to need /// them. #[cfg(all(feature = "compression", feature = "checksum"))] fn compress_and_checksum(data: &[u8]) -> Result<(Vec, [u8; 8]), ByteStorageError> { @@ -449,11 +459,11 @@ fn envelope_encoded_len( /// written in one pass into a `Vec` allocated once at its final length: the /// literals-only block goes straight into the `bin` field, so no block is /// allocated and copied. The bytes are `rmp_serde`'s for the same envelope -/// (`literal_envelope_matches_rmp_serde`). The envelope is at most 343 bytes, +/// (`literal_envelope_matches_rmp_serde`). The envelope is at most 407 bytes, /// far below every size cap, so it needs none of `compress_and_checksum`'s /// checks. /// -/// `None` for an input of 64 bytes or more, or a format string longer than +/// `None` for an input of 128 bytes or more, or a format string longer than /// 255 bytes: `store` then takes its general path, which writes the same /// bytes. #[cfg(all(feature = "compression", feature = "checksum", feature = "messagepack"))] @@ -1114,7 +1124,7 @@ mod tests { // class input: the shared table must not have been upgraded by it. let sizes = [ LITERAL_BLOCK_THRESHOLD, - 100, + 200, 1024, 2048, TABLE_REUSE_LIMIT - 1, diff --git a/tests/compress_bound_32_bits.rs b/tests/compress_bound_32_bits.rs index 18db36b..05356b6 100644 --- a/tests/compress_bound_32_bits.rs +++ b/tests/compress_bound_32_bits.rs @@ -1,6 +1,6 @@ //! The LZ4 encoder's output bound on 32-bit targets. //! -//! `StorageEnvelope::new` sends every input of 64 B or more through +//! `StorageEnvelope::new` sends every input of 128 B or more through //! `lz4_flex`, which sizes its output as `16 + 4 + input_len * 110 / 100`. //! lz4_flex 0.12 computed that product in `usize`, so on a 32-bit target it //! overflowed for any input above 39,045,157 B, far inside the crate's 512 MiB From e938d303eee2fce0fc4903761fb880e38c917afe Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sat, 7 Nov 2026 13:07:28 +1100 Subject: [PATCH 2/4] test(byte-storage): pin the store tests to the threshold, trim duplicated figures Tests that straddled the old 64 B threshold with 63/64 now read LITERAL_BLOCK_THRESHOLD, so they still cover both sides of it, and the round-trip test's interior size moves above it. The rustdoc drops its instruction counts and the README points to the rustdoc instead of repeating the corpus figures. --- README.md | 12 +++++------- src/byte_storage.rs | 28 +++++++++++++++++++++------- 2 files changed, 26 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 4c89bc0..1fecf2f 100644 --- a/README.md +++ b/README.md @@ -473,13 +473,11 @@ dominates them. Below 128 B, `store` writes one literals-only LZ4 block instead compressor, which zeroes its 8 KiB hash table on every call and then searches for matches: −80.8% instructions on a 63 B store ([#128]), and −95.0% instructions and about −81% wall time on a 64 B store when the threshold rose from 64 B to 128 B ([#156]). The trade: compressible inputs -under 128 B are stored uncompressed and grow. The threshold sits where that trade starts to cost: -on the bench corpus a value under 128 B rarely holds a repeat, and its envelope grew 0.2% on -average, while the literal block would cost 4% at 144 B and 28% at 256 B. From 128 B to under -4 KiB, the compressor reuses one hash table and one output scratch per thread: −27.5% on a 64 B -store ([#144]), which then still took that path. `make perf-ir` measures `store`, `retrieve` and -`retrieve_data` at 40 B, 64 B, 128 B, 1 KiB, 64 KiB and 1 MiB, so a change that branches on a -size threshold adds a case on any side of it those sizes miss, as [#128] and [#144] did. +under 128 B are stored uncompressed and grow; the threshold sits where the bench corpus shows that +starting to cost (`LITERAL_BLOCK_THRESHOLD`'s rustdoc, [#156]). From 128 B to under 4 KiB, the +compressor reuses one hash table and one output scratch per thread ([#144]). A change that branches +on a size threshold measures each side of it, adding `make perf-ir` cases where the +[size list](#instruction-counts) misses one, as [#128], [#144] and [#156] did. Per-thread reuse has its own traps, all from [#144]: - Reuse stopped paying at about 6 KiB, because the copy out grows with the block while the saving diff --git a/src/byte_storage.rs b/src/byte_storage.rs index a6ef771..b521a0e 100644 --- a/src/byte_storage.rs +++ b/src/byte_storage.rs @@ -87,8 +87,7 @@ pub(crate) const LARGE_PAYLOAD: usize = 64 * 1024; /// of MessagePack records about 130 bytes long, `benches/common`), a value /// under 128 bytes rarely repeats anything LZ4 can match: its envelope was /// 0.2% bigger on average as a literal block, while the store took about a -/// fifth of the encoder path's wall time from 64 to 127 bytes (515 against -/// 10,398 instructions at 64 bytes, `make perf-ir`). From the second +/// fifth of the encoder path's wall time from 64 to 127 bytes. From the second /// record on, the encoder finds matches: the literal block would cost 4% at /// 144 bytes and 28% at 256, so the encoder keeps those. #[cfg(feature = "compression")] @@ -1100,7 +1099,7 @@ mod tests { LITERAL_BLOCK_THRESHOLD - 1, LITERAL_BLOCK_THRESHOLD, LITERAL_BLOCK_THRESHOLD + 1, - 100, + 200, 1000, 4103, ] { @@ -1554,7 +1553,14 @@ mod tests { } // `store` itself, either side of its hand-written literal envelope. let storage = ByteStorage::new(None); - for len in [0, 1, 40, 63, 64, 1024] { + for len in [ + 0, + 1, + 40, + LITERAL_BLOCK_THRESHOLD - 1, + LITERAL_BLOCK_THRESHOLD, + 1024, + ] { let stored = storage.store(&xorshift_bytes(len), None).unwrap(); assert!( borrowed::StorageEnvelope::decode_one_pass(&stored).is_some(), @@ -1793,8 +1799,8 @@ mod tests { // (input, whether its envelope is at or above LARGE_PAYLOAD) for (data, large) in [ (vec![], false), - (vec![b'a'; 63], false), - (vec![b'a'; 64], false), + (vec![b'a'; LITERAL_BLOCK_THRESHOLD - 1], false), + (vec![b'a'; LITERAL_BLOCK_THRESHOLD], false), ((0..70_000).map(|i| (i * 7919 % 251) as u8).collect(), false), // either side of the gate: LZ4 adds ~257 length bytes to a literal run (xorshift_bytes(LARGE_PAYLOAD - 512), false), @@ -1879,7 +1885,15 @@ mod tests { let storage = ByteStorage::new(None); // Warm this thread's LZ4 table and scratch, allocated on first use storage.store(&xorshift_bytes(1024), None).unwrap(); - for len in [0, 14, 15, 40, 63, 64, 1024] { + for len in [ + 0, + 14, + 15, + 40, + LITERAL_BLOCK_THRESHOLD - 1, + LITERAL_BLOCK_THRESHOLD, + 1024, + ] { let data = xorshift_bytes(len); for format in [None, Some("msgpack".to_string())] { let (stored, requested) = bytes_requested(|| storage.store(&data, format).unwrap()); From 0fbfe2f59bba434e2c3949828aadde22d511acf8 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sat, 7 Nov 2026 13:33:05 +1100 Subject: [PATCH 3/4] docs(byte-storage): state the size cost of the literal block on real small values The bench corpus cannot show the cost below 128 B, because a slice that short sits inside one record. State the measured growth on small JSON and MessagePack values, padding and byte runs, its 117-byte bound, and that 128 is the literal envelope writer's own cap. ByteStorage::store and StorageEnvelope::new now say what happens under 128 B, and the private comments name the constant instead of its value. --- README.md | 13 +++++++------ src/byte_storage.rs | 37 ++++++++++++++++++++++++++----------- 2 files changed, 33 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 1fecf2f..8468d5a 100644 --- a/README.md +++ b/README.md @@ -471,12 +471,13 @@ pull request. **Give small values their own path.** Most cached values are small, and a fixed per-call cost dominates them. Below 128 B, `store` writes one literals-only LZ4 block instead of running the compressor, which zeroes its 8 KiB hash table on every call and then searches for matches: −80.8% -instructions on a 63 B store ([#128]), and −95.0% instructions and about −81% wall time on a -64 B store when the threshold rose from 64 B to 128 B ([#156]). The trade: compressible inputs -under 128 B are stored uncompressed and grow; the threshold sits where the bench corpus shows that -starting to cost (`LITERAL_BLOCK_THRESHOLD`'s rustdoc, [#156]). From 128 B to under 4 KiB, the -compressor reuses one hash table and one output scratch per thread ([#144]). A change that branches -on a size threshold measures each side of it, adding `make perf-ir` cases where the +instructions on a 63 B store ([#128]), and −95.1% on a 64 B store (10,551 → 515 Ir/op) when the +threshold rose from 64 B to 128 B ([#156]). The trade: compressible inputs under 128 B are stored +uncompressed and grow, by 16% to 84% for small JSON and MessagePack records and up to 4.25 times +for padding and byte runs, at most 117 B per value (`LITERAL_BLOCK_THRESHOLD`'s rustdoc). 128 B is +also the most the one-pass literal envelope writer allows. From 128 B to under 4 KiB, the +compressor reuses one hash table and one output scratch per thread ([#144]). A change that +branches on a size threshold measures each side of it, adding `make perf-ir` cases where the [size list](#instruction-counts) misses one, as [#128], [#144] and [#156] did. Per-thread reuse has its own traps, all from [#144]: diff --git a/src/byte_storage.rs b/src/byte_storage.rs index b521a0e..708e227 100644 --- a/src/byte_storage.rs +++ b/src/byte_storage.rs @@ -83,13 +83,21 @@ pub(crate) const LARGE_PAYLOAD: usize = 64 * 1024; /// are not canonical (protocol `spec/wire-format.md`, "Compressed-byte /// reproducibility"), and every reader decodes a literals-only block. /// -/// The threshold sits where that price starts. On the bench corpus (a stream -/// of MessagePack records about 130 bytes long, `benches/common`), a value -/// under 128 bytes rarely repeats anything LZ4 can match: its envelope was -/// 0.2% bigger on average as a literal block, while the store took about a -/// fifth of the encoder path's wall time from 64 to 127 bytes. From the second -/// record on, the encoder finds matches: the literal block would cost 4% at -/// 144 bytes and 28% at 256, so the encoder keeps those. +/// From 64 to 127 bytes the literal block takes about a fifth of the encoder +/// path's wall time, and a compressible value pays for it in size. Measured +/// against the encoder, small structured values grew 16% to 84% (a JSON +/// record, feature flags, a MessagePack list of maps, a repeated UUID), +/// padding and byte runs up to 4.25 times (127 bytes of one byte: a 36-byte +/// envelope became 153 bytes), and random bytes not at all. The cost is +/// bounded: at most 117 bytes per value, since the literal envelope of a +/// 127-byte input is 153 bytes. The bench corpus (MessagePack records about +/// 130 bytes long, `benches/common`) shows only +0.2% below 128 bytes, +/// because a slice that short sits inside one record; past one record it +/// shows +4% at 144 bytes and +28% at 256. +/// +/// 128 is also the most `encode_literal_envelope`'s fixed widths allow (the +/// `<= 0x80` assert below): a higher threshold needs a wider envelope writer +/// as well as a size decision, so do not tune it up on corpus figures alone. #[cfg(feature = "compression")] const LITERAL_BLOCK_THRESHOLD: usize = 128; @@ -220,7 +228,8 @@ fn compress_with_thread_table(data: &[u8]) -> Result, ByteStorageError> // `retrieve`, `retrieve_data` and `validate` decode into its private borrowing twin, // `borrowed::StorageEnvelope`, whose `decode_one_pass` hard-codes this field // order and these encodings. `store` encodes from another, `EnvelopeRef`, -// and writes the same bytes by hand for an input under 128 bytes with a format +// and writes the same bytes by hand for an input under +// `LITERAL_BLOCK_THRESHOLD` with a format // of at most 255 bytes (`encode_literal_envelope`, sized by // `envelope_encoded_len`). Change all of them together; // `literal_envelope_matches_rmp_serde`, `encoded_len_matches_rmp_serde` and @@ -245,6 +254,9 @@ pub struct StorageEnvelope { impl StorageEnvelope { /// Create new envelope with data compression and checksum. /// + /// An input under 128 bytes is not compressed: it is written as one + /// literals-only LZ4 block, the input plus 1 or 2 bytes. + /// /// Takes the input by shared slice: it is only compressed and hashed (both /// borrow), never retained, so there is no reason to own it. Avoids a /// full-payload copy (up to `MAX_UNCOMPRESSED_SIZE`) on the write path. @@ -328,7 +340,7 @@ impl StorageEnvelope { /// `data`'s LZ4 block and checksum, with the input and block size checks. /// `StorageEnvelope::new` and `store`'s general path call it; the -/// sub-128-byte envelope `encode_literal_envelope` writes is too small to need +/// sub-[`LITERAL_BLOCK_THRESHOLD`] envelope `encode_literal_envelope` writes is too small to need /// them. #[cfg(all(feature = "compression", feature = "checksum"))] fn compress_and_checksum(data: &[u8]) -> Result<(Vec, [u8; 8]), ByteStorageError> { @@ -462,7 +474,7 @@ fn envelope_encoded_len( /// far below every size cap, so it needs none of `compress_and_checksum`'s /// checks. /// -/// `None` for an input of 128 bytes or more, or a format string longer than +/// `None` for an input of [`LITERAL_BLOCK_THRESHOLD`] bytes or more, or a format string longer than /// 255 bytes: `store` then takes its general path, which writes the same /// bytes. #[cfg(all(feature = "compression", feature = "checksum", feature = "messagepack"))] @@ -602,7 +614,10 @@ impl ByteStorage { /// Store arbitrary bytes with compression and checksums /// - /// Returns serialized StorageEnvelope bytes + /// Returns serialized StorageEnvelope bytes. An input under 128 bytes is + /// not compressed: it is written as one literals-only LZ4 block, the input + /// plus 1 or 2 bytes, which is faster than running the compressor and + /// costs at most 117 bytes of envelope over it. #[cfg(all(feature = "compression", feature = "checksum", feature = "messagepack"))] pub fn store(&self, data: &[u8], format: Option) -> Result, ByteStorageError> { // Security: Check input size before processing From 31e7017d98d6cbe536c1429c8f6f9c09d2a448ca Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sat, 7 Nov 2026 13:40:36 +1100 Subject: [PATCH 4/4] docs(byte-storage): give the literal-block size figures as examples Envelope lengths vary by a few bytes with the checksum's MessagePack encoding, so a single 127-byte example is not an invariant: a sweep of every repeated-byte input from 64 to 127 B reached about 4.5 times and 151 to 158 bytes. The 117-byte increment is the sweep's maximum. --- README.md | 7 ++++--- src/byte_storage.rs | 10 ++++++---- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 8468d5a..632bfd5 100644 --- a/README.md +++ b/README.md @@ -473,9 +473,10 @@ dominates them. Below 128 B, `store` writes one literals-only LZ4 block instead compressor, which zeroes its 8 KiB hash table on every call and then searches for matches: −80.8% instructions on a 63 B store ([#128]), and −95.1% on a 64 B store (10,551 → 515 Ir/op) when the threshold rose from 64 B to 128 B ([#156]). The trade: compressible inputs under 128 B are stored -uncompressed and grow, by 16% to 84% for small JSON and MessagePack records and up to 4.25 times -for padding and byte runs, at most 117 B per value (`LITERAL_BLOCK_THRESHOLD`'s rustdoc). 128 B is -also the most the one-pass literal envelope writer allows. From 128 B to under 4 KiB, the +uncompressed and grow. In the examples measured, small JSON and MessagePack records grew 16% to +84% and padding and byte runs up to about 4.5 times; no repeated-byte input from 64 B to 127 B grew +by more than 117 B (`LITERAL_BLOCK_THRESHOLD`'s rustdoc). 128 B is also the most the one-pass +literal envelope writer allows. From 128 B to under 4 KiB, the compressor reuses one hash table and one output scratch per thread ([#144]). A change that branches on a size threshold measures each side of it, adding `make perf-ir` cases where the [size list](#instruction-counts) misses one, as [#128], [#144] and [#156] did. diff --git a/src/byte_storage.rs b/src/byte_storage.rs index 708e227..1cd34f2 100644 --- a/src/byte_storage.rs +++ b/src/byte_storage.rs @@ -87,10 +87,12 @@ pub(crate) const LARGE_PAYLOAD: usize = 64 * 1024; /// path's wall time, and a compressible value pays for it in size. Measured /// against the encoder, small structured values grew 16% to 84% (a JSON /// record, feature flags, a MessagePack list of maps, a repeated UUID), -/// padding and byte runs up to 4.25 times (127 bytes of one byte: a 36-byte -/// envelope became 153 bytes), and random bytes not at all. The cost is -/// bounded: at most 117 bytes per value, since the literal envelope of a -/// 127-byte input is 153 bytes. The bench corpus (MessagePack records about +/// padding and byte runs up to about 4.5 times, and random bytes not at all. +/// Envelope lengths vary by a few bytes with the checksum, whose bytes +/// MessagePack writes in 1 or 2 bytes each, so these are examples: 127 bytes of +/// one byte went from 34 to 36 bytes to 151 to 158. The growth itself is at +/// most 117 bytes per value, the most a sweep of every repeated-byte input +/// from 64 to 127 bytes reached. The bench corpus (MessagePack records about /// 130 bytes long, `benches/common`) shows only +0.2% below 128 bytes, /// because a slice that short sits inside one record; past one record it /// shows +4% at 144 bytes and +28% at 256.