duckdb: user file path as footer cache - #9733
Conversation
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Merging this PR will improve performance by 16.95%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | decompress[u64, (4000, 1024)] |
85.7 µs | 70.4 µs | +21.81% |
| ⚡ | WallTime | arrow_checked_add_u32_avx2[16384] |
21.4 µs | 17.7 µs | +21.28% |
| ⚡ | Simulation | allocate_drop_arrow[0] |
456.9 ns | 402.7 ns | +13.45% |
| ⚡ | Simulation | allocate_drop_bytes[0] |
520.2 ns | 466 ns | +11.62% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing myrrc/duckdb-fast-footer-key (2ac8893) with develop (dab1684)
Footnotes
-
206 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Polar Signals Profiling ResultsLatest Run
Previous Runs (10)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.000x ➖, 1↑ 1↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.966x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.988x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.031x ➖, 1↑ 2↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.980x ➖, 3↑ 0↓)
duckdb / vortex-file-compressed / ns (0.982x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.988x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (0.977x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.988x ➖, 0↑ 0↓)
File Size Changes (100 files changed, -0.0% overall, 48↑ 52↓)
Totals:
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 3↑ 4↓)
duckdb / parquet / ns (1.000x ➖, 5↑ 4↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.019x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.961x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.044x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.100x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.875x ➖, 3↑ 3↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.992x ➖, 2↑ 1↓)
duckdb / parquet / ns (0.995x ➖, 2↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.048x ➖, 1↑ 2↓)
datafusion / parquet / ns (1.040x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.065x ➖, 0↑ 2↓)
duckdb / parquet / ns (1.051x ➖, 0↑ 1↓)
|
|
We've also talked with @joseph-isaacs about using expiring the footer if file's mtime changed. This is what duckdb does, but this logic is invisible to the consumer and isn't propagated to MultiFileReader. We can wire it up later. |
Benchmarks: String Encoding 📖Commits: PR vortex / vortex-file-compressed / ms (1.041x ➖, 0↑ 2↓)
vortex / vortex-file-compressed / % (1.000x ➖, 0↑ 0↓)
|
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Use raw duckdb path (user input) for opening the file (and saving the footer) and retrieving the footer. This is fine since duckdb uses the same raw string for footer cache. Unlike duckdb, we don't save any etag to check whether file has changed since, but this diff doesn't change the state.
Make Vortex file's
open_cachedget an explicit key which is probed beforesource.uri().Also, remove "ordered" flag while constructing a scan since it wasn't used internally anyway.
A couple of micro-optimizations on top of that:
most extension types are Temporal (which are backed by a Primitive) this
is a reasonable shortcut that's also useful for FileStatistics.
Main improvements are clickbench q0 and q6