feat(writer): add PositionDeleteFileWriter - #2986
Open
laskoviymishka wants to merge 2 commits into
Open
Conversation
laskoviymishka
force-pushed
the
feat/position-delete-writer
branch
2 times, most recently
from
August 11, 2026 19:13
054dc33 to
6f76750
Compare
laskoviymishka
marked this pull request as ready for review
August 12, 2026 11:11
anoopj
reviewed
Aug 13, 2026
anoopj
left a comment
Member
There was a problem hiding this comment.
I took a pass. Overall, the implementation looks mostly good to me. Some feedback:
Contributor
Author
|
Thanks for the review! Pushed a follow-up commit addressing all four:
Also merged latest |
anoopj
approved these changes
Aug 13, 2026
anoopj
left a comment
Member
There was a problem hiding this comment.
LGTM. There is a CI failure though:
error: `unparseable` should be `unparsable`
╭▸ ./crates/iceberg/src/writer/base_writer/position_delete_writer.rs:593:50
│
593 │ async fn test_position_delete_writer_rejects_unparseable_field_id() -> Result<()> {
It's a bit weird that the checker thinks unparseable is a typo. (both are correct forms)
Add PositionDeleteFileWriter and its builder under writer/base_writer. It writes a position delete file with the two required columns file_path (string, field id 2147483546) and pos (long, 2147483545) and sets DataContentType::PositionDeletes on close. position_delete_schema() returns the canonical Iceberg schema, built from the existing metadata_columns field definitions. write() checks that a batch has exactly those two required, non-nullable, correctly typed columns before writing, and rejects a closed writer. close() propagates the partition key and leaves sort_order_id null. Position delete files are a v2 construct: v3 replaces them with deletion vectors and forbids adding new ones, so callers must not route v3 writes here. This base writer has no format-version gate by design; that gating belongs at the transaction/commit layer. The writer also does not sort its input, so callers must supply rows sorted by (file_path, pos) until a higher-level writer enforces it. Setting referenced_data_file and a higher-level DeltaWriter are follow-ups. Tested: schema shape, a parquet round trip for single and multiple writes, partition propagation, that the reserved field ids survive into the written parquet schema, and the validation cases (wrong column count, wrong/missing/ unparsable field ids, wrong types including LargeUtf8, nullable columns, and writes after close). Refs apache#340.
laskoviymishka
force-pushed
the
feat/position-delete-writer
branch
from
August 24, 2026 12:15
487d1bc to
68689b4
Compare
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #340.
What changes are included in this PR?
This adds
PositionDeleteFileWriterunderwriter/base_writer, next to the dataand equality-delete writers. We already had a writer for equality deletes but
nothing that writes position delete files, so I modelled this one on
equality_delete_writer.rs.It writes the two required columns,
file_path(string, field id 2147483546) andpos(long, 2147483545), and sets the content type toPositionDeleteson close.position_delete_schema()/position_delete_arrow_schema()hand back thecanonical schema, built from the field definitions already in
metadata_columnsso the reserved ids aren't duplicated here.
write()checks the batch is exactly those two required, non-null, correctlytyped columns before handing it to the parquet writer, so a bad batch fails with a
clear message instead of a confusing error further down.
close()carries thepartition key onto the data file and leaves
sort_order_idnull.One thing worth calling out: this writer doesn't sort its input. Position deletes
have to be sorted by
file_paththenpos, and for now that's on the caller (thewrite()docs say so). I'd rather land the plain writer first and put sorting ontop of it, so a sorting writer,
referenced_data_file, and a higher-levelDeltaWriter(#2218) are follow-ups instead of being crammed in here.Are these changes tested?
Yes, 12 unit tests: the schema shape, a parquet round trip for single and multiple
writes (write it, read it back, compare), partition and spec-id propagation on
close, and the rejection cases: wrong column count, wrong or missing field ids on
either column, wrong types (including LargeUtf8, which is what a lot of Arrow
producers give you), nullable columns, and writing after close.
cargo test -p iceberg,clippy, andfmtare all clean.AI assistance disclosure
Drafted with help from Claude Code, then reviewed and tested by me before opening.