Skip to content

Restore native row-group pruning for FLOAT and DOUBLE range filters in Iceberg scans #7588

Open

Description

@rich7420

What is the problem the feature request solves?

#7156 stops passing <, <=, > and >= residuals on FLOAT and DOUBLE columns to iceberg-rust. At the pinned revision (b154603), these can drop rows that Spark returns:

Results stay correct. Iceberg's planning still skips data files by their bounds. The filter above the scan applies the predicate. But for these filters the native scan now reads every row group and page of the files that remain. The same holds for any AND that contains one of them, since partial pushdown is still open in #5883.

Describe the potential solution

  1. Bump the iceberg-rust pin past fix(expr): treat NaN row group bounds as might-match for inequalities iceberg-rust#3502.
  2. Check page-index pruning against NaN page bounds. If it has the same gap as fix: [df52] route timestamp timezone mismatches through spark_parquet_convert #3494, fix it upstream.
  3. Make the row filter order NaNs as Spark does. Otherwise, push these residuals for pruning only and leave the row filtering to the filter above the scan.
  4. Remove the LT | LT_EQ | GT | GT_EQ case for FloatType and DoubleType in CometIcebergNativeScan.icebergExprToProto. Remove the matching line in the Iceberg guide too.

Zero literals stay out of pushdown either way. The native reader does not treat -0.0 as equal to 0.0 (#7136).

Tests should compare the native scan's results with Spark's. Use one file with negative and positive NaNs, both zeros and ordinary values. Use small pages so that page pruning has something to skip.

Additional context

Requested in the review of #7156: #7156 (review)

Willingness to contribute

I can contribute a fix for this bug independently

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions