prototype vortex-velox integration

[lwwmanning]

Summary

Prototype bindings to enable Vortex to act as a Velox DWIO file format. Introduces a vortex-velox crate that, similarly to the vortex-duckdb crate, exposes all of the necessary native exporter infrastructure.

The adapter supports Velox read callbacks, projection and filter expressions, natural Vortex partitions, schema conversion, and batched export. Native export covers fixed-width values, strings, Boolean values, dates, decimals, nested values, Dictionary, and Constant arrays. Timestamps use a validated Arrow fallback.

The paired Velox proof-of-concept is available on spiraldb/velox:main.

This branch includes two prerequisite changes that are not yet on develop:

• #9682: Evaluate constant list membership
• #9723: Faster sparse bitpacked filter and take

This remains a prototype. There are several additional pieces of work that are likely worth upstreaming into mainline Vortex, that were discovered incidentally.

Benchmark results

The latest local run on a Macbook Pro (M4 Max) compared Vortex, Parquet, and Nimble. The TPC-H suite contains all 22 queries. The reader suite contains 23 direct-reader workloads.

The geometric score uses exp(mean(log(format runtime / fastest runtime))). Lower values are better. A score of 1.000× matches the fastest format on every workload.

Suite	Format	Runtime sum	Geometric normalized score	Wins
TPC-H, 22 queries	Parquet	192.669 ms	1.188×	3
	Nimble	322.033 ms	1.998×	0
	Vortex	157.611 ms	1.022×	19
Reader, 23 workloads	Parquet	95.671 ms	1.330×	6
	Nimble	198.927 ms	3.043×	2
	Vortex	72.330 ms	1.211×	15

For TPC-H, Vortex completed the full benchmark in 18.2% less total TPC-H time than Parquet, and 50.8% less total TPC-H time than Nimble.

Vortex used 24.4% less total reader time than Parquet, and 63.6% less total reader time than Nimble.

Note that all of these benchmark runs included facebookincubator/velox#18609, which enabled SIMD for Parquet & Nimble readers on aarch64.

🤖 Generated with Codex

[codspeed-hq]

Merging this PR will regress 2 benchmarks

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 9 improved benchmarks
❌ 2 regressed benchmarks
✅ 2170 untouched benchmarks
🆕 429 new benchmarks
⏩ 206 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	random_i8[0.5]	66.8 µs	90.5 µs	-26.18%
❌	Simulation	cold_misaligned[(16, 64)]	344.8 µs	388.6 µs	-11.25%
⚡	WallTime	arrow_checked_add_u32_neon[16384]	20.5 µs	12.7 µs	+60.82%
⚡	Simulation	random_i8[0.8]	98.2 µs	69 µs	+42.28%
⚡	WallTime	arrow_checked_add_u32_avx2[16384]	21.3 µs	17.6 µs	+20.72%
⚡	Simulation	take[duplicates/repeated/primitive/nonnull/chunks=16/indices=1000]	234 µs	199.6 µs	+17.25%
⚡	Simulation	compact_sliced[(1024, 90)]	1.5 µs	1.3 µs	+16.67%
⚡	Simulation	compact_sliced[(2048, 90)]	1.6 µs	1.4 µs	+15.95%
⚡	Simulation	allocate_drop_arrow[0]	456.9 ns	402.7 ns	+13.45%
⚡	WallTime	mul_u32_nonnull_avx512	6.3 µs	5.6 µs	+12.67%
⚡	Simulation	patched_take_10_contiguous	66.4 µs	59.9 µs	+10.69%
🆕	WallTime	packed_current_neon[direct_u64_m4_u64_w40_n65536]	N/A	52.7 µs	N/A
🆕	WallTime	packed_current_neon[direct_u8_m4_u8_w6_n65536]	N/A	9.3 µs	N/A
🆕	WallTime	packed_current_neon[short_direct_u32_m4_u32_w10_n1024]	N/A	1.2 µs	N/A
🆕	WallTime	current_neon[long_m1_n65536]	N/A	5.8 µs	N/A
🆕	WallTime	current_neon[long_m4_n65536]	N/A	10.8 µs	N/A
🆕	WallTime	current_neon[long_m1_n65536]	N/A	8.7 µs	N/A
🆕	WallTime	current_neon[long_m4_n65536]	N/A	20.7 µs	N/A
🆕	WallTime	current_neon[long_m1_n65536]	N/A	13.7 µs	N/A
🆕	WallTime	current_neon[long_m4_n65536]	N/A	35.8 µs	N/A
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing wm/velox-e2e-poc (1a8ab43) with develop (98a2c86)²}

Footnotes

1. 206 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on develop (86d4a36) during the generation of this report, so 98a2c86 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩