Repository navigation
feat: read version-range Lance changes as lazy DataFrames - #89
Open
hello-peter-tang wants to merge 1 commit into
Open
hello-peter-tang wants to merge 1 commit into
hello-peter-tang wants to merge 1 commit into
Conversation
Add `read_lance_changes`, a scan source over PyLance's delta readers with separate inserted, updated, and deleted-row-id modes. Both endpoints are opened and pinned at planning time, and the Arrow reader is constructed and consumed on scan workers one batch at a time. Deleted rows report only `_rowid`s that were live at the begin version and are absent at the end version, so inserts-then-deletes and compaction-only rewrites do not appear as deletions. Stable row ids are required at both endpoints unless the range is empty. Namespace addressing reuses the existing open-context reconstruction. Closes daft-engine#64
Author
|
Heads-up: the |
Author
|
Same as my note on #88: CI is still parked in Versus |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
daft_lance.read_lance_changes, a lazy Daft scan source over PyLance'sversion-range delta readers. Closes #64.
"inserted","updated", and"deleted"(deleted row ids), eachproducing a DataFrame with the upstream result schema: dataset columns plus
_rowid/_row_created_at_version/_row_last_updated_at_versionforinserted and updated rows,
_rowidonly for deleted rows.is pinned in the worker open kwargs, so a commit after planning cannot change
either endpoint or the result.
RecordBatchReaderare constructed and consumed onscan workers, yielding one batch at a time rather than collecting on the
driver.
begin_version >= 1,begin_version <= end_version,existing endpoints (the begin manifest is opened too), and stable row ids at
both endpoints unless the range is empty.
version, so insert-then-delete and compaction-only rewrites are not reported
as deletions.
table_id+namespace_impl+namespace_properties)reuses the existing worker open-context reconstruction.
Tests
tests/io/lance/test_lance_delta.py(50 tests) covers:deletion, and empty results, including explicit schema checks.
begin-live − end-live, insert-then-delete,delete-then-reinsert, and compaction cases.
stable-row-id prerequisites; unknown modes.
factory generator), plus a commit after planning leaving results unchanged.
DAFT_ANALYTICS_ENABLED=0 uv run pytest tests/ -q→ 615 passed, 10 skipped,2 xfailed, 2 xpassed.
ruff format --check,ruff check, andmypy daft_lance/are clean.