Skip to content

Reuse compression/decompression results in compression benchmarks - #9744

Open
robert3005 wants to merge 3 commits into
developfrom
rk/compressionbench
Open

Reuse compression/decompression results in compression benchmarks#9744
robert3005 wants to merge 3 commits into
developfrom
rk/compressionbench

Conversation

@robert3005

Copy link
Copy Markdown
Contributor

Avoid decompressing and reloading files multiple times in benchmark run

Signed-off-by: Robert Kruszewski <github@robertk.io>
@codspeed-hq

codspeed-hq Bot commented Sep 3, 2026

Copy link
Copy Markdown

Merging this PR will regress 1 benchmark

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 7 improved benchmarks
❌ 1 regressed benchmark
✅ 2191 untouched benchmarks
⏩ 206 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation random_i16[0.8] 74.2 µs 92.5 µs -19.76%
WallTime arrow_checked_add_u32_neon[16384] 20.5 µs 12.8 µs +60.35%
Simulation random_i8[0.5] 90.6 µs 67 µs +35.25%
Simulation decompress[u64, (4000, 1024)] 85.5 µs 70.3 µs +21.59%
WallTime arrow_checked_add_u32_avx2[16384] 21.4 µs 17.7 µs +20.98%
Simulation allocate_drop_arrow[0] 456.9 ns 402.7 ns +13.45%
WallTime mul_u32_nonnull_avx512 6.3 µs 5.6 µs +11.86%
Simulation allocate_drop_bytes[0] 520.2 ns 466 ns +11.62%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing rk/compressionbench (c7361f2) with develop (265b705)

Open in CodSpeed

Footnotes

  1. 206 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@robert3005 robert3005 added the changelog/chore A trivial change label Sep 3, 2026
Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005 robert3005 added the action/bench-compress Run only the compression benchmark on this PR label Sep 3, 2026
@github-actions github-actions Bot removed the action/bench-compress Run only the compression benchmark on this PR label Sep 3, 2026
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done fc2a30c compress-bench 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Benchmarks: Compression 📖

Commits: PR fc2a30c0 vs base 792cef74


vortex / vortex-file-compressed / ns (0.785x ✅, 14↑ 1↓)
name ns (PR / base / %diff)
compress time/Arade 579701694 / 1173457064 / -50.6% 🟢
compress time/Bimbo 3368331824 / 5062112092 / -33.5% 🟢
compress time/CMSprovider 1274139808 / 4026792967 / -68.4% 🟢
compress time/Euro2016 261920627 / 524481612 / -50.1% 🟢
compress time/Food 272120602 / 433961311 / -37.3% 🟢
compress time/HashTags 430240085 / 879075118 / -51.1% 🟢
compress time/TPC-H l_comment canonical 536217619 / 1194031130 / -55.1% 🟢
compress time/TPC-H l_comment chunked 553897452 / 1171588937 / -52.7% 🟢
compress time/taxi 290872780 / 557440930 / -47.8% 🟢
compress time/wide table cols=100 chunks=1 rows=1000 8170101 / 8107505 / +0.8%
compress time/wide table cols=100 chunks=50 rows=1000 8374346 / 8497328 / -1.4%
compress time/wide table cols=1000 chunks=1 rows=1000 81520838 / 84263736 / -3.3%
compress time/wide table cols=1000 chunks=50 rows=1000 81873222 / 84524961 / -3.1%
compress time/wide table cols=10000 chunks=1 rows=1000 963379320 / 980849956 / -1.8%
compress time/wide table cols=10000 chunks=50 rows=1000 973425600 / 972116618 / +0.1%
decompress time/Arade 25347792 / 26001352 / -2.5%
decompress time/Bimbo 87065661 / 91779375 / -5.1%
decompress time/CMSprovider 67780247 / 72839252 / -6.9%
decompress time/Euro2016 15494241 / 15555604 / -0.4%
decompress time/Food 8207551 / 7000742 / +17.2% 🔴
decompress time/HashTags 97033839 / 108364467 / -10.5% 🟢
decompress time/TPC-H l_comment canonical 33135708 / 34641468 / -4.3%
decompress time/TPC-H l_comment chunked 33329764 / 34924803 / -4.6%
decompress time/taxi 13854866 / 14132673 / -2.0%
decompress time/wide table cols=100 chunks=1 rows=1000 2082667 / 2342811 / -11.1% 🟢
decompress time/wide table cols=100 chunks=50 rows=1000 2114096 / 2413617 / -12.4% 🟢
decompress time/wide table cols=1000 chunks=1 rows=1000 20439682 / 22840839 / -10.5% 🟢
decompress time/wide table cols=1000 chunks=50 rows=1000 20484015 / 22587781 / -9.3%
decompress time/wide table cols=10000 chunks=1 rows=1000 241573830 / 251218578 / -3.8%
decompress time/wide table cols=10000 chunks=50 rows=1000 242671707 / 277275144 / -12.5% 🟢
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
vortex-file-compressed size/Arade 144460980 / 144460980 / +0.0%
vortex-file-compressed size/Bimbo 461842908 / 461842908 / +0.0%
vortex-file-compressed size/CMSprovider 415192460 / 415192460 / +0.0%
vortex-file-compressed size/Euro2016 157458700 / 157674404 / -0.1%
vortex-file-compressed size/Food 41581624 / 41581624 / +0.0%
vortex-file-compressed size/HashTags 177840324 / 177840364 / -0.0%
vortex-file-compressed size/TPC-H l_comment canonical 163453384 / 163453384 / +0.0%
vortex-file-compressed size/TPC-H l_comment chunked 163453384 / 163453384 / +0.0%
vortex-file-compressed size/taxi 51475172 / 51475172 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=1 rows=1000 932744 / 932744 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=50 rows=1000 932744 / 932744 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=1 rows=1000 9309944 / 9309944 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=50 rows=1000 9309944 / 9309944 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=1 rows=1000 93117944 / 93117944 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=50 rows=1000 93117944 / 93117944 / +0.0%
vortex / vortex-file-compressed / ratio (0.852x ✅, 13↑ 2↓)
name ratio (PR / base / %diff)
vortex:parquet-zstd ratio compress time/Arade 0.199628265 / 0.410925518 / -51.4% 🟢
vortex:parquet-zstd ratio compress time/Bimbo 0.238403401 / 0.365013892 / -34.7% 🟢
vortex:parquet-zstd ratio compress time/CMSprovider 0.166675711 / 0.524457567 / -68.2% 🟢
vortex:parquet-zstd ratio compress time/Euro2016 0.173700375 / 0.34686689 / -49.9% 🟢
vortex:parquet-zstd ratio compress time/Food 0.304926877 / 0.477408633 / -36.1% 🟢
vortex:parquet-zstd ratio compress time/HashTags 0.17691294 / 0.363981168 / -51.4% 🟢
vortex:parquet-zstd ratio compress time/TPC-H l_comment canonical 0.145452614 / 0.326458599 / -55.4% 🟢
vortex:parquet-zstd ratio compress time/TPC-H l_comment chunked 0.150944546 / 0.318213819 / -52.6% 🟢
vortex:parquet-zstd ratio compress time/taxi 0.214965132 / 0.415096165 / -48.2% 🟢
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=1 rows=1000 1.56639306 / 1.61315358 / -2.9%
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=50 rows=1000 1.56477156 / 1.68235004 / -7.0%
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=1 rows=1000 1.26579248 / 1.35020408 / -6.3%
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=50 rows=1000 1.2846255 / 1.34628438 / -4.6%
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=1 rows=1000 1.40032921 / 1.32391917 / +5.8%
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=50 rows=1000 1.45099059 / 1.30873318 / +10.9% 🔴
vortex:parquet-zstd ratio decompress time/Arade 0.0367201499 / 0.036888534 / -0.5%
vortex:parquet-zstd ratio decompress time/Bimbo 0.0449777681 / 0.0481243261 / -6.5%
vortex:parquet-zstd ratio decompress time/CMSprovider 0.035772469 / 0.0381031485 / -6.1%
vortex:parquet-zstd ratio decompress time/Euro2016 0.0375537993 / 0.0374453672 / +0.3%
vortex:parquet-zstd ratio decompress time/Food 0.0379757039 / 0.0316199793 / +20.1% 🔴
vortex:parquet-zstd ratio decompress time/HashTags 0.145754153 / 0.158832893 / -8.2%
vortex:parquet-zstd ratio decompress time/TPC-H l_comment canonical 0.0503065697 / 0.0522666296 / -3.8%
vortex:parquet-zstd ratio decompress time/TPC-H l_comment chunked 0.0505980729 / 0.0528514094 / -4.3%
vortex:parquet-zstd ratio decompress time/taxi 0.0508196459 / 0.0498121621 / +2.0%
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=1 rows=1000 0.861944706 / 1.01127252 / -14.8% 🟢
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=50 rows=1000 0.851079864 / 1.0406159 / -18.2% 🟢
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=1 rows=1000 0.75879367 / 0.857924442 / -11.6% 🟢
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=50 rows=1000 0.782106371 / 0.842788026 / -7.2%
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=1 rows=1000 0.827512398 / 0.842556898 / -1.8%
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=50 rows=1000 0.823159665 / 0.917023088 / -10.2% 🟢
vortex:parquet-zstd size/Arade 0.559895285 / 0.559895285 / -0.0%
vortex:parquet-zstd size/Bimbo 1.20109789 / 1.20109789 / -0.0%
vortex:parquet-zstd size/CMSprovider 1.1016338 / 1.1016338 / -0.0%
vortex:parquet-zstd size/Euro2016 1.28037012 / 1.28212411 / -0.1%
vortex:parquet-zstd size/Food 1.16476769 / 1.16476769 / -0.0%
vortex:parquet-zstd size/HashTags 1.33216131 / 1.33216161 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment canonical 1.03217481 / 1.03217481 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment chunked 1.03217481 / 1.03217481 / -0.0%
vortex:parquet-zstd size/taxi 0.931110481 / 0.931110481 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=1 rows=1000 1.00036465 / 1.00036465 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=50 rows=1000 1.00036465 / 1.00036465 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=1 rows=1000 0.998492064 / 0.998492064 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=50 rows=1000 0.998492064 / 0.998492064 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=1 rows=1000 0.998690905 / 0.998690905 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=50 rows=1000 0.998690905 / 0.998690905 / -0.0%
vortex / parquet / ns (0.998x ➖, 0↑ 0↓)
name ns (PR / base / %diff)
parquet_rs-zstd compress time/Arade 2903905885 / 2855644183 / +1.7%
parquet_rs-zstd compress time/Bimbo 14128707115 / 13868272425 / +1.9%
parquet_rs-zstd compress time/CMSprovider 7644424037 / 7678014817 / -0.4%
parquet_rs-zstd compress time/Euro2016 1507887519 / 1512054415 / -0.3%
parquet_rs-zstd compress time/Food 892412649 / 908993430 / -1.8%
parquet_rs-zstd compress time/HashTags 2431931121 / 2415166486 / +0.7%
parquet_rs-zstd compress time/TPC-H l_comment canonical 3686545095 / 3657526966 / +0.8%
parquet_rs-zstd compress time/TPC-H l_comment chunked 3669542676 / 3681766365 / -0.3%
parquet_rs-zstd compress time/taxi 1353116096 / 1342919971 / +0.8%
parquet_rs-zstd compress time/wide table cols=100 chunks=1 rows=1000 5215869 / 5025873 / +3.8%
parquet_rs-zstd compress time/wide table cols=100 chunks=50 rows=1000 5351801 / 5050868 / +6.0%
parquet_rs-zstd compress time/wide table cols=1000 chunks=1 rows=1000 64403004 / 62408148 / +3.2%
parquet_rs-zstd compress time/wide table cols=1000 chunks=50 rows=1000 63733144 / 62783883 / +1.5%
parquet_rs-zstd compress time/wide table cols=10000 chunks=1 rows=1000 687966309 / 740868461 / -7.1%
parquet_rs-zstd compress time/wide table cols=10000 chunks=50 rows=1000 670869686 / 742792065 / -9.7%
parquet_rs-zstd decompress time/Arade 690296529 / 704862709 / -2.1%
parquet_rs-zstd decompress time/Bimbo 1935748809 / 1907130602 / +1.5%
parquet_rs-zstd decompress time/CMSprovider 1894760103 / 1911633418 / -0.9%
parquet_rs-zstd decompress time/Euro2016 412587842 / 415421323 / -0.7%
parquet_rs-zstd decompress time/Food 216126369 / 221402485 / -2.4%
parquet_rs-zstd decompress time/HashTags 665736359 / 682254570 / -2.4%
parquet_rs-zstd decompress time/TPC-H l_comment canonical 658675561 / 662783659 / -0.6%
parquet_rs-zstd decompress time/TPC-H l_comment chunked 658716075 / 660811195 / -0.3%
parquet_rs-zstd decompress time/taxi 272628149 / 283719325 / -3.9%
parquet_rs-zstd decompress time/wide table cols=100 chunks=1 rows=1000 2416242 / 2316696 / +4.3%
parquet_rs-zstd decompress time/wide table cols=100 chunks=50 rows=1000 2484016 / 2319412 / +7.1%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=1 rows=1000 26937075 / 26623369 / +1.2%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=50 rows=1000 26190830 / 26801260 / -2.3%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=1 rows=1000 291927747 / 298162152 / -2.1%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=50 rows=1000 294805148 / 302364409 / -2.5%
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
parquet size/Arade 258014282 / 258014282 / +0.0%
parquet size/Bimbo 384517292 / 384517292 / +0.0%
parquet size/CMSprovider 376887911 / 376887911 / +0.0%
parquet size/Euro2016 122979049 / 122979049 / +0.0%
parquet size/Food 35699500 / 35699500 / +0.0%
parquet size/HashTags 133497590 / 133497590 / +0.0%
parquet size/TPC-H l_comment canonical 158358238 / 158358238 / +0.0%
parquet size/TPC-H l_comment chunked 158358238 / 158358238 / +0.0%
parquet size/taxi 55283635 / 55283635 / +0.0%
parquet size/wide table cols=100 chunks=1 rows=1000 932404 / 932404 / +0.0%
parquet size/wide table cols=100 chunks=50 rows=1000 932404 / 932404 / +0.0%
parquet size/wide table cols=1000 chunks=1 rows=1000 9324004 / 9324004 / +0.0%
parquet size/wide table cols=1000 chunks=50 rows=1000 9324004 / 9324004 / +0.0%
parquet size/wide table cols=10000 chunks=1 rows=1000 93240004 / 93240004 / +0.0%
parquet size/wide table cols=10000 chunks=50 rows=1000 93240004 / 93240004 / +0.0%
vortex / arrow-ipc / ns (0.961x ➖, 5↑ 0↓)
name ns (PR / base / %diff)
arrow-ipc compress time/Arade 208047778 / 270802755 / -23.2% 🟢
arrow-ipc compress time/Bimbo 1643796243 / 1768629459 / -7.1%
arrow-ipc compress time/CMSprovider 1144146513 / 1453431634 / -21.3% 🟢
arrow-ipc compress time/Euro2016 93500156 / 98501828 / -5.1%
arrow-ipc compress time/Food 70847283 / 74079297 / -4.4%
arrow-ipc compress time/HashTags 197482861 / 291491710 / -32.3% 🟢
arrow-ipc compress time/TPC-H l_comment canonical 2127988832 / 2129153939 / -0.1%
arrow-ipc compress time/TPC-H l_comment chunked 2087936411 / 2121778879 / -1.6%
arrow-ipc compress time/taxi 170132800 / 160615016 / +5.9%
arrow-ipc compress time/wide table cols=100 chunks=1 rows=1000 367177 / 414441 / -11.4% 🟢
arrow-ipc compress time/wide table cols=100 chunks=50 rows=1000 358674 / 407735 / -12.0% 🟢
arrow-ipc compress time/wide table cols=1000 chunks=1 rows=1000 4625366 / 4930121 / -6.2%
arrow-ipc compress time/wide table cols=1000 chunks=50 rows=1000 4682552 / 4946489 / -5.3%
arrow-ipc compress time/wide table cols=10000 chunks=1 rows=1000 56957150 / 56556134 / +0.7%
arrow-ipc compress time/wide table cols=10000 chunks=50 rows=1000 55506177 / 56649310 / -2.0%
arrow-ipc decompress time/Arade 187885060 / 185753923 / +1.1%
arrow-ipc decompress time/Bimbo 585683761 / 592638199 / -1.2%
arrow-ipc decompress time/CMSprovider 690048008 / 715741545 / -3.6%
arrow-ipc decompress time/Euro2016 95567731 / 96372573 / -0.8%
arrow-ipc decompress time/Food 64765104 / 65549547 / -1.2%
arrow-ipc decompress time/HashTags 208482325 / 209955386 / -0.7%
arrow-ipc decompress time/TPC-H l_comment canonical 1052150647 / 1033293616 / +1.8%
arrow-ipc decompress time/TPC-H l_comment chunked 1022848225 / 1046224757 / -2.2%
arrow-ipc decompress time/taxi 73650584 / 72221291 / +2.0%
arrow-ipc decompress time/wide table cols=100 chunks=1 rows=1000 407301 / 390724 / +4.2%
arrow-ipc decompress time/wide table cols=100 chunks=50 rows=1000 411560 / 395360 / +4.1%
arrow-ipc decompress time/wide table cols=1000 chunks=1 rows=1000 4226142 / 4140209 / +2.1%
arrow-ipc decompress time/wide table cols=1000 chunks=50 rows=1000 4376866 / 4065180 / +7.7%
arrow-ipc decompress time/wide table cols=10000 chunks=1 rows=1000 53350934 / 50926584 / +4.8%
arrow-ipc decompress time/wide table cols=10000 chunks=50 rows=1000 52714394 / 50569984 / +4.2%
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
arrow-ipc size/Arade 691765162 / 691765162 / +0.0%
arrow-ipc size/Bimbo 3279590114 / 3279590114 / +0.0%
arrow-ipc size/CMSprovider 2422259682 / 2422259682 / +0.0%
arrow-ipc size/Euro2016 360582666 / 360582666 / +0.0%
arrow-ipc size/Food 255782338 / 255782338 / +0.0%
arrow-ipc size/HashTags 704785810 / 704785810 / +0.0%
arrow-ipc size/TPC-H l_comment canonical 4852929378 / 4852929378 / +0.0%
arrow-ipc size/TPC-H l_comment chunked 4852929378 / 4852929378 / +0.0%
arrow-ipc size/taxi 482329690 / 482329690 / +0.0%
arrow-ipc size/wide table cols=100 chunks=1 rows=1000 1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=100 chunks=50 rows=1000 1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=1 rows=1000 12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=50 rows=1000 12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=1 rows=1000 125752506 / 125752506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=50 rows=1000 125752506 / 125752506 / +0.0%

Reusing one Vortex array across iterations let every run after the first
find its statistics already cached, so the timed region shrank to just the
encoding. Reset the input before each iteration so the measurement matches
a fresh conversion.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019CX6MXw5FFYnqQNdV7K4ci
/// The Vortex writer computes statistics inside the timed region and caches them on the
/// array, so reusing one array across iterations would let every run after the first skip
/// that work. Clearing the cache keeps each iteration's measurement comparable.
pub fn reset(&self) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

holding vortex in memory lets you skip stat computation when writing/compression so in order to time the full thing we have to get rid of them

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant