feat(fastlanes): store a bit width per FastLanes chunk in BitPacked

[mhk197]

Summary

BitPacked now stores a bit width per 1024-element FastLanes chunk, and serializes under one of two wire formats following the model in docs/specs/editions.md (one in-memory array, one wire ID per revision):

• fastlanes.bitpacked (unchanged, frozen): emitted whenever every chunk shares one width. Metadata and packed bytes are identical to today, so old readers keep reading new files and new readers keep reading old ones.
• fastlanes.bitpacked_v2: emitted only when chunk widths differ. The width table travels as a u8 child array with one entry per chunk, so metadata stays bounded and a compressor can re-encode the table; the metadata is the original struct with bit_width set to the widest chunk. Old readers reject it as an unknown encoding. It is not in any edition yet, so the default writer cannot produce it; a follow-up adds the edition entry and compressor scheme.

Following docs/specs/editions.md, one ArrayPlugin owns both formats; the core VTable trait is unchanged.

Nothing about the default writer's output changes: BitPackingScheme still picks one global width, so every array it produces serializes as fastlanes.bitpacked. The btrblocks goldens are untouched.

Changes

vortex-fastlanes.

• BitPackedPlugin registers both wire IDs. It serializes uniform arrays through the encoding's own serializer, so that path is byte-identical to today, and differing widths as fastlanes.bitpacked_v2 with BitPackedV2Metadata { offset, patches } and the width table as the last child. It deserializes both IDs and enforces each one's contract. BitPackedPatchedPlugin forwards to it.
• The bare vtable serializer returns None for differing widths. The blanket ArrayPlugin impl labels its output with the vtable's own ID, fastlanes.bitpacked, so registering the vtable directly can only ever write valid v1 or fail, never v2 bytes under the v1 ID.
• ChunkWidths: one width per chunk plus cached byte offsets. BitPacked::try_new takes it; bit_width() now returns the widest chunk.
• bitpack_to_best_chunk_widths chooses each chunk's width from its own histogram, charging the exact packed block plus exceptions, and packs, gathers exceptions and picks widths in one pass over each chunk while it is in cache. The existing bitpack_to_best_bit_width / bitpack_encode API keeps producing uniform widths.
• Every kernel (slice, take, filter, fused compare, between, cast, is_constant, scalar_at, the FoR fused decode) indexes chunks through the width table; each differs from before by a handful of lines. Sequential decode walks the table with a running offset.
• BitPacked::with_width_table swaps in a re-encoded width table. CUDA decode accepts uniform arrays and rejects differing widths, since there is no per-chunk kernel yet.
• benches/bitpack_chunk_widths.rs sweeps u8..u64 over uniform, drifting, random, zero-heavy and spiky width patterns, with and without exceptions and nulls, comparing the two width policies on size, compress time and decompress time.

Benchmark (32 chunks per case, divan fastest, per-chunk relative to global width)

Bytes: uniform data unchanged (0 bytes of overhead, since it stays in the original format); drift 43–50% smaller; random widths 40–44%; zero-heavy 50%; spiky 16–58%.

Compress: -15% to +7% on cases without exceptions, 14–35% faster wherever exceptions exist, 35–49% faster on spiky data where per-chunk widths avoid the patches a global width needs; 19% slower on u8 spiky data with exceptions, where it takes patches for a 58% smaller array.

Decompress: within 2% on uniform data, 6–21% faster on drift, random and zero-heavy data, 25–75% faster on spiky data.

Tests

Per-chunk behaviour (roundtrips, slicing across chunk boundaries, take, filter, compare, between, cast, is_constant, conformance harnesses), fused-versus-multipass equivalence, a short array whose single packed block exceeds its raw size, and serde: uniform and empty arrays serialize byte-identically under the original ID with no width table, differing widths under the v2 ID with the table as a child, a re-encoded table survives a round trip, the table is validated for dtype, length and presence, each ID rejects the other's children, and the bare vtable serializer refuses differing widths.

Validation

• cargo nextest run -p vortex-fastlanes -p vortex-array -p vortex-btrblocks -p vortex-file -p vortex
• cargo test --doc -p vortex-fastlanes -p vortex-array
• cargo clippy --all-targets --all-features on vortex-fastlanes, vortex-btrblocks, vortex-file, vortex-cuda; cargo +nightly fmt --all

[codspeed-hq]

Merging this PR will degrade performance by 3.24%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 5 improved benchmarks
❌ 8 regressed benchmarks
✅ 2210 untouched benchmarks
🆕 160 new benchmarks
⏩ 164 skipped benchmarks¹
🗄️ 1 archived benchmark run²

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	bitpacked_compress_u32	36.8 µs	53.6 µs	-31.35%
❌	Simulation	random_i8[0.5]	66.8 µs	90.7 µs	-26.32%
❌	Simulation	compress_rd[f64, (2000, 0.01)]	169.3 µs	194.1 µs	-12.8%
❌	Simulation	compress_rd[f64, (2000, 0.1)]	169.2 µs	194.1 µs	-12.8%
❌	Simulation	compress_rd[f64, (2000, 0.0)]	169.2 µs	194 µs	-12.77%
❌	Simulation	compress_rd[f32, (2000, 0.01)]	138 µs	157.4 µs	-12.33%
❌	Simulation	compress_rd[f32, (2000, 0.0)]	138 µs	157.4 µs	-12.29%
❌	Simulation	compress_rd[f32, (2000, 0.1)]	138 µs	157.3 µs	-12.24%
⚡	Simulation	random_i8[0.8]	98.2 µs	68.8 µs	+42.6%
⚡	Simulation	decompress[u64, (4000, 1024)]	85.6 µs	70.4 µs	+21.71%
⚡	WallTime	arrow_checked_add_u32_avx512[16384]	21.3 µs	17.6 µs	+21.05%
⚡	WallTime	arrow_checked_add_u32_avx2[16384]	21.3 µs	17.6 µs	+20.76%
⚡	Simulation	allocate_drop_arrow[0]	456.9 ns	402.7 ns	+13.45%
🆕	Simulation	compress_v1[u16, drift]	N/A	288.8 µs	N/A
🆕	Simulation	compress_v1[u16, drift+exc1%]	N/A	691.2 µs	N/A
🆕	Simulation	compress_v1[u16, drift+null10%]	N/A	516.2 µs	N/A
🆕	Simulation	compress_v1[u16, random]	N/A	293.5 µs	N/A
🆕	Simulation	compress_v1[u16, spiky]	N/A	748.5 µs	N/A
🆕	Simulation	compress_v1[u16, spiky+exc1%]	N/A	754.7 µs	N/A
🆕	Simulation	compress_v1[u16, uniform]	N/A	265.5 µs	N/A
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing mk/bitpacked-v2 (dcd7e85) with develop (98a2c86)}

Footnotes

1. 164 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. 1 benchmark was run, but is now archived. If it was deleted in another branch, consider rebasing to remove it from the report. Instead if it was added back, click here to restore it. ↩