perf: build spark_size LargeList lengths from i64 offsets

docs/source/contributor-guide/expression-audits/collection_funcs.md

@@ -48,6 +48,6 @@
 - Spark 4.0.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.
 - Spark 4.1.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.
 - Both `ArrayType` and `MapType` inputs are `Compatible` and run natively; every other child type is `Unsupported`.
-- Performance (tuned 2026-07-10, PR [#4877](https://github.com/apache/datafusion-comet/pull/4877)): compute list row sizes from the offset buffer instead of allocating a sliced `ArrayRef` per row via `list_array.value(i)`, removing one heap allocation per row. ~94% faster. Benchmark: `benches/array_size.rs`.
+- Performance (tuned 2026-08-22, PRs [#4877](https://github.com/apache/datafusion-comet/pull/4877), [#5233](https://github.com/apache/datafusion-comet/pull/5233), [#5395](https://github.com/apache/datafusion-comet/pull/5395), [#5300](https://github.com/apache/datafusion-comet/pull/5300)): List and FixedSizeList compute row lengths via Arrow's `length` kernel plus an `into_parts` null → `-1` rewrite. LargeList and Map subtract adjacent offsets directly into Int32 (LargeList falls back to per-row `try_from` on span overflow). Up to ~12x faster on List / FixedSizeList, ~6.6x on LargeList, and ~6-13x on Map versus the per-row builder loop. Benchmark: `benches/array_size.rs`.
 
 [Spark Expression Support]: ../../user-guide/latest/expressions.md

docs/source/contributor-guide/optimizing_expressions.md

@@ -140,20 +140,20 @@ Recurring correctness traps:
 These are the patterns that have produced real speedups in merged and in-flight Comet PRs. Reach
 for the lightest one that fits.
 
-| Technique                                                          | What it replaces                                                                                      | Example                                                                                                                                                                               |
-| ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| **Vectorized Arrow kernels** (`unary`, `try_unary`, `binary`)      | `iter().map().collect()` over `Option`/`Result`                                                       | `spark_cast_int_to_int` (up to 100x): map the values buffer in one pass, carry the null buffer over                                                                                   |
-| **`unary_opt` for null-producing conversions**                     | A builder loop that appends null on overflow, or `unary` + sentinel + `null_if_overflow` (two passes) | int/float-to-decimal casts (28-50%): map overflow to null in one vectorized pass; for the ANSI throw-on-overflow variant, gate a rare element-wise rescan on an O(1) null-count check |
-| **Zero-copy borrow with `Cow`**                                    | Allocating a new `String`/`Vec` per row when most rows are unchanged                                  | `to_json` `escape_string` (2x): borrow when nothing needs escaping, bulk-copy unescaped byte runs                                                                                     |
-| **Zero-copy buffer reuse**                                         | Copying every value into a fresh builder                                                              | `cast_binary_to_string` default path (up to 7000x): reuse the binary array's buffers instead of copying                                                                               |
-| **Preallocate builders to known size**                             | Repeated buffer growth/reallocation                                                                   | `spark_unhex`: preallocate `BinaryBuilder` to the known output length                                                                                                                 |
-| **Compile-time lookup tables**                                     | Per-element range matches / branching                                                                 | `spark_unhex`: 256-entry hex table instead of per-digit range match                                                                                                                   |
-| **Cache compiled regex** (thread-local, keyed by the constant arg) | `Regex::new()` per row                                                                                | `parse_url` QUERY-with-key (50x): the key is constant across a batch                                                                                                                  |
-| **Read from the offset buffer directly**                           | `list_array.value(i)` allocating a sliced `ArrayRef` per row                                          | `spark_size` scalar path: `value_length` instead of materializing a sliced array; Map array path (~6-13x): `MapArray::offsets().windows(2)` diff, `length` kernel doesn't support Map |
-| **Reuse Arrow `length` kernel + cheap null rewrite**               | Per-row `value_length` / builder loop, or `zip` via `MutableArrayData`                                | `spark_size` (~12x on no-null List): `length` for List/LargeList/FixedSizeList, then `into_parts` + `set_indices` to patch null slots to `-1`                                         |
-| **Typed scans over flat values buffers + hash probe**              | A per-element Arrow `eq`/compute kernel that allocates per call                                       | `spark_arrays_overlap` (up to 18x): scan buffers directly, hash probe for large lists                                                                                                 |
-| **ASCII / byte-offset fast path**                                  | `chars().count()` and per-char UTF-8 decoding                                                         | `substring` (up to 10x), `spark_lpad` (2x): slice by byte offset when input is ASCII                                                                                                  |
-| **`memcpy` from a precomputed buffer**                             | Char-by-char `push` into a scratch `String`                                                           | `spark_lpad`: pad from a precomputed repeating pad buffer, write directly into the builder                                                                                            |
+| Technique                                                          | What it replaces                                                                                      | Example                                                                                                                                                                                                                                                                                        |
+| ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Vectorized Arrow kernels** (`unary`, `try_unary`, `binary`)      | `iter().map().collect()` over `Option`/`Result`                                                       | `spark_cast_int_to_int` (up to 100x): map the values buffer in one pass, carry the null buffer over                                                                                                                                                                                            |
+| **`unary_opt` for null-producing conversions**                     | A builder loop that appends null on overflow, or `unary` + sentinel + `null_if_overflow` (two passes) | int/float-to-decimal casts (28-50%): map overflow to null in one vectorized pass; for the ANSI throw-on-overflow variant, gate a rare element-wise rescan on an O(1) null-count check                                                                                                          |
+| **Zero-copy borrow with `Cow`**                                    | Allocating a new `String`/`Vec` per row when most rows are unchanged                                  | `to_json` `escape_string` (2x): borrow when nothing needs escaping, bulk-copy unescaped byte runs                                                                                                                                                                                              |
+| **Zero-copy buffer reuse**                                         | Copying every value into a fresh builder                                                              | `cast_binary_to_string` default path (up to 7000x): reuse the binary array's buffers instead of copying                                                                                                                                                                                        |
+| **Preallocate builders to known size**                             | Repeated buffer growth/reallocation                                                                   | `spark_unhex`: preallocate `BinaryBuilder` to the known output length                                                                                                                                                                                                                          |
+| **Compile-time lookup tables**                                     | Per-element range matches / branching                                                                 | `spark_unhex`: 256-entry hex table instead of per-digit range match                                                                                                                                                                                                                            |
+| **Cache compiled regex** (thread-local, keyed by the constant arg) | `Regex::new()` per row                                                                                | `parse_url` QUERY-with-key (50x): the key is constant across a batch                                                                                                                                                                                                                           |
+| **Read from the offset buffer directly**                           | `list_array.value(i)` allocating a sliced `ArrayRef` per row                                          | `spark_size` scalar path: `value_length` instead of materializing a sliced array; Map array path (~6-13x): `MapArray::offsets().windows(2)` diff, `length` kernel doesn't support Map; LargeList array path (~6.6x): `offsets().windows(2)` diff into Int32, skips `length`'s Int64→Int32 cast |
+| **Reuse Arrow `length` kernel + cheap null rewrite**               | Per-row `value_length` / builder loop, or `zip` via `MutableArrayData`                                | `spark_size` (~12x on no-null List): `length` for List/FixedSizeList, then `into_parts` + `set_indices` to patch null slots to `-1`                                                                                                                                                            |
+| **Typed scans over flat values buffers + hash probe**              | A per-element Arrow `eq`/compute kernel that allocates per call                                       | `spark_arrays_overlap` (up to 18x): scan buffers directly, hash probe for large lists                                                                                                                                                                                                          |
+| **ASCII / byte-offset fast path**                                  | `chars().count()` and per-char UTF-8 decoding                                                         | `substring` (up to 10x), `spark_lpad` (2x): slice by byte offset when input is ASCII                                                                                                                                                                                                           |
+| **`memcpy` from a precomputed buffer**                             | Char-by-char `push` into a scratch `String`                                                           | `spark_lpad`: pad from a precomputed repeating pad buffer, write directly into the builder                                                                                                                                                                                                     |
 
 Cross-cutting principles behind the table:
 

native/spark-expr/benches/array_size.rs

@@ -48,9 +48,8 @@ fn create_list_array(rows: usize, elems_per_row: usize, with_nulls: bool) -> Arr
 }
 
 /// Build a `LargeListArray` (i64 offsets) of `rows` lists with `elems_per_row`
-/// Int32 elements. Every 10th row is null. LargeList exercises the extra
-/// Int64->Int32 cast on top of the length kernel.
-fn create_large_list_array(rows: usize, elems_per_row: usize) -> ArrayRef {
+/// Int32 elements. When `with_nulls` is true every 10th row is null.
+fn create_large_list_array(rows: usize, elems_per_row: usize, with_nulls: bool) -> ArrayRef {
     let total = rows * elems_per_row;
     let values = Int32Array::from((0..total as i32).collect::<Vec<i32>>());
 
@@ -60,13 +59,14 @@ fn create_large_list_array(rows: usize, elems_per_row: usize) -> ArrayRef {
         offsets.push((i * elems_per_row) as i64);
     }
 
-    let nulls = NullBuffer::from((0..rows).map(|i| i % 10 != 0).collect::<Vec<bool>>());
+    let nulls =
+        with_nulls.then(|| NullBuffer::from((0..rows).map(|i| i % 10 != 0).collect::<Vec<bool>>()));
     let field = Arc::new(Field::new("item", DataType::Int32, true));
     Arc::new(LargeListArray::new(
         field,
         OffsetBuffer::new(offsets.into()),
         Arc::new(values),
-        Some(nulls),
+        nulls,
     ))
 }
 
@@ -145,11 +145,21 @@ fn criterion_benchmark(c: &mut Criterion) {
         &create_list_array(rows, 5, false),
     );
 
-    // LargeList: exposes the Int64 length -> Int32 cast (extra allocation) on top of
-    // the length kernel.
+    // LargeList: builds Int32 lengths from i64 offsets (see spark_size_large_list_from_offsets).
+    // Shapes mirror List coverage (short/long + 10% null, short + no nulls).
     bench(
         "spark_size: LargeList (10% null)",
-        &create_large_list_array(rows, 5),
+        &create_large_list_array(rows, 5, true),
+    );
+    bench(
+        "spark_size: LargeList of long arrays",
+        &create_large_list_array(rows, 64, true),
+    );
+    // No-null LargeList: what the kernel sees behind CometSize's isnotnull guard, or for a
+    // non-nullable child.
+    bench(
+        "spark_size: LargeList, no nulls",
+        &create_large_list_array(rows, 5, false),
     );
 
     // Map shapes: no length-kernel support for MapArray, so this path differences