Add support for Equality Deletes on DeleteFileIndex

[rambleraptor]

Part of #3270

Rationale for this change

This adds support for getting equality deletes in the DeleteFileIndex.

I'm very purposefully ignoring them in _read_all_delete_files because they will crash.

Are these changes tested?

I made some equality deletes by-hand and had PyIceberg read them to see the indexes. Worked as expected. If you know a way to create equality deletes, I can test those as well.

Are there any user-facing changes?

• Adds support for equality deletes in DeleteFileIndex

[ndrluis]

@rambleraptor I think we should add a regression test for schema evolution here. This pruning path assumes the current table type for an equality field is the same type that was used when the data file and equality delete were written, which is not always true after a legal promotion like int -> long. In that case, historical manifests still contain 4-byte int bounds, and decoding them with the current LongType can fail in DeleteFileIndex.for_data_file(...).

For reference, Iceberg Java had to address the same schema-evolution issue in apache/iceberg#15268, where the fix was to avoid assuming the current schema is always the right one for equality-delete field resolution.

[rambleraptor]

Thanks @ndrluis for the suggestion. I opened #3293 to handle the type promotion case for the historical manifests. It looks like we're not handling that properly and I don't want to pollute this PR too much.

[rambleraptor]

@geruh @kevinjqliu @Fokko please take a look when you can!

[rambleraptor]

I've successfully tried this out with Flink (thanks @Fokko for the tip!) and it's working as I expect it to. Is it worth checking in the files created by Flink?

[geruh]

Nice, thanks for opening @rambleraptor!!! Left some comments below.

Also, +1 to add to the flink testing and I believe there were talks about this being added to the TCK! While working on #2255, we tested all delete file combinations with flink.

[rambleraptor]

@geruh thanks for the review!

Could not agree more on the Flink testing! I'll leave that for a follow-up PR if that's alright, since we haven't stood up Flink testing yet. I don't want to pollute this PR too much

[rambleraptor]

@geruh thanks so much for your review! ptal

[rambleraptor]

@kevinjqliu @geruh can I get some eyes on this? I'm hoping it's in good shape + we can push forward on this work

[arjun-panchmatia-mechademy]

+1. We're unable to read iceberg tables which are synced via CDC at all because equality deletes immediately throws a NotImpl error.

[rambleraptor]

@Fokko can you get this merged and then we can finish the rest of the equality deletes implementation?

[daviddallakyan2005]

Checked this out locally. make test PYTEST_ARGS="-v -k equality" gave 157 passed.

@arjun-panchmatia-mechademy the CDC failure is still present on this branch, which is expected. plan_files still raises before the new index is used:

ValueError: PyIceberg does not yet support equality deletes: https://github.com/apache/iceberg/issues/6568

REST planning raises NotImplementedError the same way. I am taking the PyArrow reads geruh outlined on #3270.

One gap in the index itself: _eq_applies_to_data_file looks up equality field types on the current schema. int to long is fine after #3293, but the case Java actually fixed in apache/iceberg#15268 (equality deletes on a column later dropped) still crashes:

ValueError: Could not find field with id: 1

GitHub currently marks this PR CONFLICTING (146 commits behind main). I am rebasing the index onto main and adding apply on read plus that dropped field test. I will credit @rambleraptor for the index work.

[daviddallakyan2005]

#3805 rebases this index onto current main and adds PyArrow apply on read for #3270. Please close this PR if that combined path is preferred. Thank you for the original work.

[rambleraptor]

Hey @daviddallakyan2005 thanks for your notes! I went ahead and rebased this PR. I've got a personal preference for getting this merged in and then applying the pyarrow work on top of it.

If (big if!) we need to revert anything, it'll be on the PyArrow side and not on the index side most likely.

[daviddallakyan2005]

Thanks. I will follow that split.

One index gap is still present after this PR. _eq_applies_to_data_file looks up equality field types on the current schema. If an equality delete is on a column that was later dropped, for_data_file raises ValueError: Could not find field with id: .... Java handled that in apache/iceberg#15268.

I will land that fix plus the PyArrow apply-on-read work as a follow-up as soon as this merges. #3805 is that follow-up. I will rebase it onto this once it lands.