You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Restore native row-group pruning for FLOAT and DOUBLE range filters in Iceberg scans #7588
#7156 stops passing <, <=, > and >= residuals on FLOAT and DOUBLE columns to iceberg-rust. At the pinned revision (b154603), these can drop rows that Spark returns:
The row filter compares floats with arrow_ord::cmp, which uses the IEEE 754 total order (f32::total_cmp). That order puts negative NaNs below every number. Spark puts every NaN above all numbers.
PageIndexEvaluator::visit_inequality checks only the literal for NaN, not the page bounds. Page-index pruning may have the same gap. I haven't checked this against a real file.
Results stay correct. Iceberg's planning still skips data files by their bounds. The filter above the scan applies the predicate. But for these filters the native scan now reads every row group and page of the files that remain. The same holds for any AND that contains one of them, since partial pushdown is still open in #5883.
Make the row filter order NaNs as Spark does. Otherwise, push these residuals for pruning only and leave the row filtering to the filter above the scan.
Remove the LT | LT_EQ | GT | GT_EQ case for FloatType and DoubleType in CometIcebergNativeScan.icebergExprToProto. Remove the matching line in the Iceberg guide too.
Zero literals stay out of pushdown either way. The native reader does not treat -0.0 as equal to 0.0 (#7136).
Tests should compare the native scan's results with Spark's. Use one file with negative and positive NaNs, both zeros and ordinary values. Use small pages so that page pruning has something to skip.
Activity
fix: treat timestamp columns as promoted when Iceberg schema history …