Repository navigation
Bug triage results: 2026-11-09 #7574
Description
Bug triage pass for 2026-11-09.
- Issues processed: 78 (34 bugs, 44 enhancements)
- Bugs by priority: critical 5, high 7, medium 16, low 6
- Skipped: 3
Criteria: Bug Triage Guide.
Labels have already been applied and requires-triage removed. Please spot-check the calls and close this issue when satisfied. Many issues arrived already carrying a type (and, for bugs, a priority) label; those were kept as they were.
Bugs
priority:critical
- Iceberg
truncateon integers with widths above 2^30: native kernel and native writer compute different partition values (#7442)- Area labels:
area:expressions,area:writer - Rationale: Native kernel and native writer compute different Iceberg partition values for integer truncate widths above 2^30, a silent mismatch (rare widths; see escalations).
- Area labels:
- read_side_padding ignores the pad string for dictionary inputs in lpad and rpad (#7453)
- Area labels:
area:expressions - Rationale:
lpad/rpadon dictionary input pads with spaces instead of the pad string, a silent wrong result.
- Area labels:
- With ANSI mode on, Comet's filter and projection return rows where Spark raises
DIVIDE_BY_ZERO(#7480)- Area labels:
area:expressions - Rationale: Under ANSI, Comet returns rows where Spark raises
DIVIDE_BY_ZERO, so it can silently differ from Spark (see escalations).
- Area labels:
- Offset overflow - Native columnar-to-row conversion wraps row offsets past 2 GiB (#7494)
- Area labels:
area:ffi - Rationale: Already labeled
bugwith this priority; kept as is.
- Area labels:
- Spark window over a doubted input order gets a sort by its own keys only, and operators above it return wrong rows (#7553)
- Area labels: none
- Rationale: Operators above a Spark window return wrong rows silently, which is
priority:criticalstep 1 of the decision tree.
priority:high
- Native Parquet scan reads invalid UTF-8 in nested BINARY values as unvalidated strings on DataFusion 56 (#7469)
- Area labels:
area:scan - Rationale: Already labeled
bugwith this priority; kept as is.
- Area labels:
- Native timestamp casts fail or panic for years beyond chrono's range (#7471)
- Area labels:
area:expressions - Rationale: Native timestamp casts panic or raise for years beyond chrono's range, which is a native panic.
- Area labels:
- Offset overflow - Shuffle read panics expanding dictionary blocks from the JVM columnar shuffle (#7496)
- Area labels:
area:shuffle - Rationale: Already labeled
bugwith this priority; kept as is.
- Area labels:
- Offset overflow - JVM columnar shuffle cuts row-to-Arrow batches by row count only (#7497)
- Area labels:
area:shuffle - Rationale: Already labeled
bugwith this priority; kept as is.
- Area labels:
- Offset overflow - String kernels can produce more than 2 GiB per batch, and some reserve it up front (#7500)
- Area labels:
area:expressions - Rationale: Already labeled
bugwith this priority; kept as is.
- Area labels:
- Offset overflow - Comet's grouped accumulators build one array for all emitted groups (#7501)
- Area labels:
area:aggregation - Rationale: Already labeled
bugwith this priority; kept as is.
- Area labels:
GetArrayItemfails on a non-INT index:could not cast array of type Int64 ...(#7551)- Area labels:
area:expressions - Rationale: A valid query with a non-INT array index fails with a native exception on a user's real workload, which is major functional breakage on a supported path.
- Area labels:
priority:medium
- Native Iceberg scan with AQE DPP fails when the DPP build side is a storage-partitioned join (#7441)
- Area labels:
area:scan - Rationale: Assertion failure for one narrow plan shape (DPP over a storage-partitioned join) with the opt-in native Iceberg scan; the Spark scan is a workaround.
- Area labels:
- Native lpad and rpad fail when the length argument is a NULL scalar, such as a scalar subquery that returns no rows (#7452)
- Area labels:
area:expressions - Rationale:
lpad/rpadfail on a NULL scalar length such as an empty scalar subquery; rare input with a fallback workaround.
- Area labels:
- ANSI: comparisons evaluate the right operand on rows where the left one is NULL (#7468)
- Area labels:
area:expressions - Rationale: Under ANSI, comparisons evaluate the right operand where Spark skips it and raise false errors; visible failure.
- Area labels:
- Native sort evaluates later sort keys on rows Spark's ordering skips, raising false ANSI errors (#7470)
- Area labels: none
- Rationale: Under ANSI, native sort evaluates later keys on rows Spark skips and raises false errors; visible failure.
spark.comet.expression.<Name>.enabled=falsehas no effect onparse_url,to_jsonandjson_array_lengthon Spark 4.1 (#7478)- Area labels:
area:expressions - Rationale:
spark.comet.expression.<Name>.enabled=falsehas no effect for three expressions on Spark 4.1; results are correct and a workaround is documented in the issue.
- Area labels:
randandrandnwith an integer seed fall back to Spark without a fallback reason (#7479)- Area labels:
area:expressions - Rationale:
rand/randnwith an integer seed falls back to Spark with no reason recorded; a missing-support gap with a workaround.
- Area labels:
- CometShuffleManager treats Spark's own shuffles differently from SortShuffleManager (#7488)
- Area labels:
area:shuffle - Rationale:
CometShuffleManageradds a row copy and leaks per-shuffle state on the executor heap; no wrong results or crash.
- Area labels:
- Native Iceberg scan copies the whole scan's delete-file metadata into every task, causing executor OOM on large merge-on-read tables (#7490)
- Area labels:
area:scan - Rationale: Opt-in native Iceberg scan hits executor OOM on very large merge-on-read tables; a broken feature with the Spark scan as workaround.
- Area labels:
- Offset overflow - Explode copies columns past 2 GiB and never splits one long row (#7495)
- Area labels:
area:expressions - Rationale: Already labeled
bugwith this priority; kept as is.
- Area labels:
- Offset overflow - Codegen dispatcher writes each batch's output into one Arrow vector (#7498)
- Area labels:
area:expressions,area:udf - Rationale: Already labeled
bugwith this priority; kept as is.
- Area labels:
- Offset overflow - Native Parquet scan expands flat dictionary columns and reads twice Spark's batch rows (#7499)
- Area labels:
area:scan - Rationale: Already labeled
bugwith this priority; kept as is.
- Area labels:
- Offset overflow - Spark-to-Arrow conversion cuts batches by row count only (#7502)
- Area labels:
area:scan - Rationale: Already labeled
bugwith this priority; kept as is.
- Area labels:
- Native TopK evaluates sort keys for a single-row input, raising false ANSI errors (#7527)
- Area labels: none
- Rationale: Under ANSI, a single-row TopK raises a false error that Spark does not; visible failure, limited to ANSI mode.
- ANY_VALUE memory regression in Comet 1.2.0 (#7541)
- Area labels:
area:aggregation - Rationale: Comet 1.2.0 fails
ANY_VALUE(array<string>)with an oversized memory reservation; disabling native aggregates is a workaround (see escalations).
- Area labels:
- Backport ANSI skipped-operand fix for arithmetic and concat to branch-1.2 (#7542)
- Area labels:
area:expressions - Rationale:
branch-1.2raises falseCAST_INVALID_INPUTerrors for queries Spark answers; the failure is visible and Spark fallback is a workaround.
- Area labels:
- [Bug] hll_sketch_agg, hll_union and hll_union_agg write compact sketch bytes where Spark writes updatable ones (#7548)
- Area labels:
area:aggregation,area:expressions - Rationale: Sketch bytes differ from Spark, but the expressions are gated behind
allowIncompatible, so users opt in knowingly; a broken feature with a workaround.
- Area labels:
priority:low
- test: no test on main reaches the folded UnsafeArrayData literal fix from feat: convert OneRowRelation to Arrow by default #6666 (#7458)
- Area labels:
area:expressions - Rationale: No test reaches the feat: convert OneRowRelation to Arrow by default #6666 fix, so a regression would go unnoticed; test-only.
- Area labels:
CometBenchmarkBase.runExpressionBenchmarkchecks the initial AQE plan, not the executed plan (#7481)- Area labels: none
- Rationale: Benchmark tooling checks the wrong plan; test/tooling only.
- Offset overflow - Lower-risk sites (opt-in features, unrealistic sizes, upstream panics) (#7503)
- Area labels: none
- Rationale: Already labeled
bugwith this priority; kept as is.
- [Bug] hll_* errors surface as Comet errors instead of HLL_INVALID_INPUT_SKETCH_BUFFER / HLL_UNION_DIFFERENT_LG_K (#7549)
- Area labels:
area:expressions - Rationale: Both engines fail on the same inputs; only the error class and SQLSTATE differ, which is cosmetic.
- Area labels:
- ParquetWriterExec's derived Debug prints object store options, which can hold secret keys (#7558)
- Area labels:
area:writer - Rationale: Derived
Debugwould print object store secrets, but Comet does not format the operator with{:?}today, so nothing leaks yet; low (see escalations).
- Area labels:
- Native Iceberg errors print data and delete file paths with their credentials (#7559)
- Area labels: none
- Rationale: Error messages can print credentials embedded in Iceberg file paths, but locations rarely carry credentials and nothing is wrong with results, so low (see escalations).
Enhancements
- Comet Native Executors: let a fragment's partial aggregate stop aggregating, including AVG (#7406)
- Area labels:
area:aggregation - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
- Comet Native Executors: choose partition counts for pipelined mode (#7407)
- Area labels: none
- Rationale: Already labeled
enhancement; kept as is.
- Comet Native Executors: broadcast a small build side in pipelined mode (#7408)
- Area labels: none
- Rationale: Already labeled
enhancement; kept as is.
- Comet Native Executors: TPC-H q17 in pipelined mode takes 277 s where Ballista takes 154 s (#7409)
- Area labels: none
- Rationale: Already labeled
enhancement; kept as is.
- Create 1.3.0 Release (Dec 2026) (#7410)
- Area labels: none
- Rationale: Already labeled
enhancement; kept as is.
- Enable the Comet in-memory cache (
InMemoryTableScan) by default (#7425)- Area labels: none
- Rationale: Already labeled
enhancement; kept as is.
- Enable
LocalTableScanconversion by default (#7426)- Area labels: none
- Rationale: Already labeled
enhancement; kept as is.
- Support Iceberg's
MergeRowsExecon Spark 3.4 and 3.5 (#7427)- Area labels: none
- Rationale: Already labeled
enhancement; kept as is.
- Native Iceberg writes for CTAS / RTAS on Spark 3.4 (#7428)
- Area labels:
area:writer - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
- Native JSON file scan (#7429)
- Area labels:
area:scan - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
- Native CSV scan for V1
FileSourceScanExec(#7430)- Area labels:
area:scan - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
- Native scans for Iceberg metadata tables (#7431)
- Area labels:
area:scan - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
- Support
from_protobuf(ProtobufDataToCatalyst) (#7432)- Area labels:
area:expressions - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
- Support
input_file_name()andinput_file_block_*with native scans (#7433)- Area labels:
area:scan - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
- Window
min/maxover string and binary input (#7434)- Area labels:
area:aggregation - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
percentilewith an array of percentages or a frequency argument (#7435)- Area labels:
area:aggregation - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
- Native text file writes (#7436)
- Area labels:
area:writer - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
- Native JSON file writes (#7437)
- Area labels:
area:writer - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
- Native CSV file writes (#7438)
- Area labels:
area:writer - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
approx_percentile/percentile_approxon non-numeric input (#7439)- Area labels:
area:aggregation - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
max_by/min_bywith variable-length value or ordering types (#7440)- Area labels:
area:aggregation - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
- Comet's TPC-DS speedup is smaller on Spark 4.2.0 than on Spark 4.1.3; investigate and improve Spark 4.2 acceleration (#7451)
- Area labels: none
- Rationale: Investigation to improve Spark 4.2 acceleration; Comet results are correct and the gap is mostly a faster Spark baseline.
- Pair the spark-4.2 profile with Delta 4.4 and run the Delta contrib tests on it (#7455)
- Area labels:
area:ci - Rationale: Request to pair the
spark-4.2profile with Delta 4.4 and run the Delta contrib tests on it.
- Area labels:
- perf: native Parquet scans make two S3 requests for a small file and fetch large merged ranges with one GET (#7459)
- Area labels:
area:scan - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
- Native Parquet write falls back on int96RebaseModeInWrite=LEGACY even when timestamps are not written as INT96 (#7463)
- Area labels:
area:writer - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
- Comet Native Executors: report each stage's Comet block metrics to Ballista (#7465)
- Area labels: none
- Rationale: Already labeled
enhancement; kept as is.
- Reduce per-partition overhead for ROW_NUMBER windows (#7466)
- Area labels: none
- Rationale: Already labeled
enhancement; kept as is.
- perf: file open and footer reads dominate native Parquet scan time on TPC-DS q54 (#7467)
- Area labels:
area:scan - Rationale: Performance work on native Parquet scan open time; nothing is incorrect.
- Area labels:
- perf: tasks that share a decoded broadcast each reserve the whole build side (#7472)
- Area labels: none
- Rationale: Already labeled
enhancement; kept as is.
- Native hash shuffle gathers rows one at a time even when a whole batch goes to one partition (#7474)
- Area labels:
area:shuffle - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
- Native CartesianProductExec: multi-partition inputs and spilling (#7476)
- Area labels: none
- Rationale: Already labeled
enhancement; kept as is.
- perf: broadcasts with dictionary-encoded columns are neither coalesced nor shared (#7486)
- Area labels: none
- Rationale: Already labeled
enhancement; kept as is.
- perf: probe tasks wait for the shared broadcast decode, and every wave of tasks decodes again (#7487)
- Area labels: none
- Rationale: Already labeled
enhancement; kept as is.
- Support Iceberg 1.12.0 and run the Iceberg suites on Spark 4.2 (#7493)
- Area labels:
area:Iceberg,area:ci - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
- Research: relate Spark 4.3 Arrow cache serializer to Comet in-memory cache (#7517)
- Area labels: none
- Rationale: Already labeled
enhancement; kept as is.
- [EPIC] Support Spark 4.3 (#7519)
- Area labels: none
- Rationale: Already labeled
enhancement; kept as is.
- Native Iceberg scan builds every partition's plan data on the driver up front (#7523)
- Area labels:
area:scan - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
- Support a literal str argument in native lpad/rpad (#7525)
- Area labels:
area:expressions - Rationale: Request to take the native path for a literal
strargument instead of codegen dispatch; results are already correct.
- Area labels:
- Native lambdas: count a widening cast as unable to raise under ANSI mode (#7535)
- Area labels:
area:expressions - Rationale: Already labeled
enhancement; kept as is.
- Area labels:
- Comet Native Executors: run a query in Spark when Comet is turned off (#7557)
- Area labels: none
- Rationale: Already labeled
enhancement; kept as is.
- Support Delta FileSizeHistogramAgg (#7570)
- Area labels:
area:aggregation - Rationale: Request to support a new Delta aggregate natively.
- Area labels:
- Support Delta DeletedRecordCountsHistogramAgg (#7571)
- Area labels:
area:aggregation - Rationale: Request to support a new Delta aggregate natively.
- Area labels:
- Support Delta BitmapAggregator with Portable encoding (#7572)
- Area labels:
area:aggregation - Rationale: Request to support a new Delta aggregate natively.
- Area labels:
- Support Delta InterleaveBits for one or two INT inputs (#7573)
- Area labels:
area:expressions - Rationale: Request to support a new Delta expression natively.
- Area labels:
Escalations to consider
- Native Iceberg errors print data and delete file paths with their credentials (#7559)
- The guide lists security vulnerabilities as
priority:critical. Labeledpriority:lowbecause the reporter calls the risk low and the path has to carry credentials, but a reviewer may want to raise it. #7558 is the same class (no leak today).
- The guide lists security vulnerabilities as
- ANY_VALUE memory regression in Comet 1.2.0 (#7541)
- A regression from 1.1.0 that fails the query;
priority:mediumhas a workaround, but escalate topriority:highif other users report it.
- A regression from 1.1.0 that fails the query;
- With ANSI mode on, Comet's filter and projection return rows where Spark raises
DIVIDE_BY_ZERO(#7480)- Labeled
priority:criticalper decision-tree step 1 (output differs from Spark with no error), but it only affects queries that Spark itself fails under ANSI; a reviewer may lower it.
- Labeled
- Iceberg
truncateon integers with widths above 2^30: native kernel and native writer compute different partition values (#7442)- Labeled
priority:criticalper step 1 (wrong partition values), but widths above 2^30 are unusual; a reviewer may lower it.
- Labeled
- Spark window over a doubted input order gets a sort by its own keys only, and operators above it return wrong rows (#7553)
- Silent wrong rows on main; no area label in the guide fits.
Skipped — needs more info
- Bug triage results: 2026-11-02 (#7403)
- A previous triage summary issue, not a bug or enhancement.
requires-triagewas left in place; a reviewer can close it.
- A previous triage summary issue, not a bug or enhancement.
- Bug triage results: 2026-10-26 (#7230)
- A previous triage summary issue, not a bug or enhancement.
requires-triagewas left in place; a reviewer can close it.
- A previous triage summary issue, not a bug or enhancement.
- TPC-DS q54/q68: local SF100 results and distributed reproduction limits (#6946)
- A reproduction report that states it does not claim Investigate TPC-DS q54 and q68 slowdowns in the 1.1.0 benchmark on Spark 4.2 #6418 is resolved; it is not clearly a bug or an enhancement.
Activity
andygrove commentedon Nov 9, 2026
Note on #7494 (native columnar-to-row conversion wraps row offsets past 2 GiB,
priority:critical): the native columnar-to-row implementation is being removed in #7505, which closes #7494, so the priority is not a concern once that PR merges.