You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native TopK evaluates sort keys for a single-row input, raising false ANSI errors #7527
With spark.sql.ansi.enabled=true, a TopK (ORDER BY ... LIMIT n) whose whole input is a single row fails in Comet with ARITHMETIC_OVERFLOW when its first sort key overflows on that row, while Spark returns the row.
Spark's TakeOrderedAndProjectExec goes through takeOrdered, which uses Guava's Ordering.leastOf (TopKSelector) per partition and a BoundedPriorityQueue to merge. Both store the first element without a comparison, and sorting a single element compares nothing, so Spark's lazily generated ordering never evaluates any sort key when the input has one row. Comet's native TopK evaluates every sort key over the whole batch.
With two or more rows Spark compares and evaluates the first key for every row, so both engines fail. This is pre-existing and independent of #7457 , which falls back when a sort key after the first can fail; it was raised in review there.
Steps to reproduce
SETspark.sql.ansi.enabled=true;
SETspark.sql.adaptive.enabled=false;
CREATETABLEone_row(a int, b int) USING parquet;
INSERT INTO one_row VALUES (1, 2147483647);
SELECT a, b FROM one_row ORDER BY b +1LIMIT10;
Spark returns 1, 2147483647. Comet fails with [ARITHMETIC_OVERFLOW] integer overflow (checked on Spark 4.1 with a CometSqlFileTestSuite fixture, current main plus #7457).
Expected behavior
Same result as Spark. Since it only affects a single-row input, it may be enough to document it rather than fall back for every TopK with a fallible first key.
Activity
Address review: allow float arithmetic and struct fields, exclude dat…
Address review: allow float arithmetic and struct fields, exclude dat…