Repository navigation
Add row dictionary slice benchmarks - #12041
Rich-T-kid wants to merge 1 commit into
Conversation
5afca4f to
1a5a052
Compare
|
@alamb could you stamp 🙇 |
1a5a052 to
6a8c160
Compare
|
run benchmark row_format |
|
🤖 Benchmark starting (GKE) | trigger Target: row_format Sharding factor: 1 (1 workers). Comparing bench-row-dictionary-slices-11901 (6a8c160) to b231cf1 (merge-base) diff Run configurationrun benchmark row_format
env:
BENCH_FILTER: convert_columns_prepared 8192 string_dictionary
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: row_format Comparing bench-row-dictionary-slices-11901 (6a8c160) to b231cf1 (merge-base) diff Run configurationrun benchmark row_format
env:
BENCH_FILTER: convert_columns_prepared 8192 string_dictionary
shards: 1
DetailsPer-runner informationrow_format — shard 1/1Node: gk3-benchmark-cluster-nap-o8oe52el-122737a1-l6bj Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench row_formatCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
Superseded by #12054, which combines this with the related benchmark coverage. |
# Which issue does this PR close? Part of apache#11885. Adds benchmark coverage for apache#11901 and apache#12008, and follows up on the Boolean filter coverage requested in [apache#11918](apache#11918 (comment)). This consolidates and supersedes apache#11980 and apache#12041. # Why is this change needed? The existing suites do not cover Boolean arrays as filter data or the cost of operating on small key ranges backed by oversized dictionaries. They also do not exercise the 16-values-per-selected-key boundary between dense and sparse dictionary mappings. # What changes are included in this PR? - Adds nullable and non-null Boolean filter cases across the existing selectivities. - Adds prepared `RowConverter` cases for rotating 8,192-key ranges backed by unique string dictionaries with 1M and 4M values. - Measures the full-value encoding cost from apache#11901 without including `RowConverter` construction. - Adds concat cases for 12 distinct dictionaries sliced to 682 keys, with 12, 16, and 24 values per selected key. - Straddles apache#12008's 16-values-per-selected-key cutoff so we can assess the threshold. # Are these changes tested? Built and ran each filtered benchmark locally. Benchmark bot runs: - [`filter_kernels`](apache#12054 (comment)) - [`row_format`](apache#12054 (comment)) - [`concatenate_kernel`](apache#12054 (comment)) # Are there any significant user-facing changes? No. This only adds benchmark coverage.
Adds benchmarks for #11901.
They convert rotating 8,192-key ranges from unique string dictionaries with 1M and 4M values, reusing a prepared
RowConverterand timing onlyconvert_columns. This exposes the current behavior where each small conversion re-encodes the full values array.