Repository navigation
Conversation
Perseus
force-pushed
the
anirudh.singh/iceberg-dyn-filtering-3
branch
from
November 9, 2026 21:11
6bfdfaf to
52659af
Compare
DataFusion offers each scan the filters above it, including the dynamic filters that joins, top-N sorts and aggregates fill in while the query runs. Iceberg scans didn't keep them. Keeping them is the first step toward skipping files with them. Both scans now keep every pushed filter, ANDed into one predicate. They answer that they don't apply them, so DataFusion still checks every row. EXPLAIN shows the dynamic ones on the scan as dynamic_filters=, and the codecs carry all of them to workers. With iceberg.dynamic_filtering_enabled on, scans also report the filters through apply_expressions. Joins and aggregates fill in a dynamic filter only when a scan reports it, and the coordinator sends updates to the workers holding those scans. The setting is off by default, and the plan carries it to workers. Enable it only once every worker supports it: an older worker can fail a query whose scan reports a dynamic filter.
…ltering is on With iceberg.dynamic_filtering_enabled off, scans aren't consumers of the dynamic filters pushed into them, so those filters stay empty. Plans no longer print them in that case.
Perseus
force-pushed
the
anirudh.singh/iceberg-dyn-filtering-3
branch
from
November 10, 2026 10:22
52659af to
ef70cff
Compare
A join filter can list thousands of keys, which filled the plan. An IN list of more than 5 values now prints as its count, for example `pickup_date@0 IN ([7 elements])`. Only the printed plan changes.
Every IN list now prints as its count, so a plan stays short however many keys a join filter lists. The rewrite moves into fmt_dynamic_filters, and the plan test for long lists goes away.
Removes the comment about decoding filters with the plan's converter, since that is the usual way to decode them. The lazy codec test builds its filter with DataFusion's binary and col helpers.
Adds a translator from a DataFusion physical filter to an iceberg predicate. It handles comparisons with a value, IN lists, IS NULL, LIKE prefixes, isnan, starts_with, AND, OR, NOT and CASE. It also handles the filters hash joins build while the query runs: IN lists, struct(a, b) IN (...) for joins on several columns, and the CASE a partitioned join builds with one branch per partition. The predicate matches every row the filter keeps, and can match more, so the filter still runs on every row. An AND can drop a side it can't convert. A NOT is converted only when its inside was converted fully. Floats and times aren't converted, and neither are LIKE prefixes with non-ASCII text. iceberg-rust handles these in ways that could skip matching rows or fail the read. Nothing calls it yet. Partition skipping will use it next.
Perseus
force-pushed
the
anirudh.singh/iceberg-dyn-filtering-3
branch
from
November 11, 2026 14:35
77858c6 to
c59f43e
Compare
…ters The converter only depends on DataFusion and iceberg, so it belongs with the shared helpers. The partition skipper in common/ no longer imports from table_provider/.
The partition skipper now reports files_pruned_by_partition, which shows how many files it checked and how many it kept.
Runs joins and top-N queries over identity, truncate, month, bucket and day partitions, with dynamic filtering on and off, and checks they return the same rows. The cases use keys on partition edges, keys in no partition, and nulls. A snapshot shows how many files each case kept.
Explain the two forms of CASE, what LIKE patterns convert, and why only ASCII prefixes are converted.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
This is part of a stack of PRs that implement dynamic filtering for Iceberg.
The previous PRs can be found here - #850 , #855
Please review this PR starting at dbd4c77
Relevant changes will also be ported to datafusion-iceberg.
Summary
This PR
PhysicalExpr->Iceberg::PredicatetranslationThe translator changes have also been raised in
datafusion-iceberghere - apache/datafusion-iceberg#106. If that merges after this PR, I will raise a follow-up to delete the code and use the public methods exposed there.Some of the code in this PR duplicates functionality from
iceberg-rustdue to those APIs not being public. I will raise PRs/issues in the repo to make them public so that the code can be deleted from here eventually'Notes