Skip to content

feat: read version-range Lance changes as lazy DataFrames - #89

Open
hello-peter-tang wants to merge 1 commit into
daft-engine:mainfrom
hello-peter-tang:feat/64-lance-version-range-changes
Open

hello-peter-tang wants to merge 1 commit into
daft-engine:mainfrom
hello-peter-tang:feat/64-lance-version-range-changes

Conversation

@hello-peter-tang

Copy link
Copy Markdown

Summary

Adds daft_lance.read_lance_changes, a lazy Daft scan source over PyLance's
version-range delta readers. Closes #64.

  • Modes: "inserted", "updated", and "deleted" (deleted row ids), each
    producing a DataFrame with the upstream result schema: dataset columns plus
    _rowid / _row_created_at_version / _row_last_updated_at_version for
    inserted and updated rows, _rowid only for deleted rows.
  • Both endpoints are opened and validated on the driver, and the end snapshot
    is pinned in the worker open kwargs, so a commit after planning cannot change
    either endpoint or the result.
  • The dataset and Arrow RecordBatchReader are constructed and consumed on
    scan workers, yielding one batch at a time rather than collecting on the
    driver.
  • Explicit validation: begin_version >= 1, begin_version <= end_version,
    existing endpoints (the begin manifest is opened too), and stable row ids at
    both endpoints unless the range is empty.
  • Deleted results are rows live at the begin version and absent at the end
    version, so insert-then-delete and compaction-only rewrites are not reported
    as deletions.
  • Namespace addressing (table_id + namespace_impl + namespace_properties)
    reuses the existing worker open-context reconstruction.

Tests

tests/io/lance/test_lance_delta.py (50 tests) covers:

  • Parity with PyLance across append, repeated update, row and whole-fragment
    deletion, and empty results, including explicit schema checks.
  • Deleted ids against begin-live − end-live, insert-then-delete,
    delete-then-reinsert, and compaction cases.
  • Equal, reversed, missing, and cleaned-up endpoints; non-integer inputs;
    stable-row-id prerequisites; unknown modes.
  • Worker-side execution and incremental batch consumption (unit test on the
    factory generator), plus a commit after planning leaving results unchanged.
  • Dir-namespace context reconstruction.

DAFT_ANALYTICS_ENABLED=0 uv run pytest tests/ -q → 615 passed, 10 skipped,
2 xfailed, 2 xpassed. ruff format --check, ruff check, and mypy daft_lance/ are clean.

Add `read_lance_changes`, a scan source over PyLance's delta readers with
separate inserted, updated, and deleted-row-id modes. Both endpoints are
opened and pinned at planning time, and the Arrow reader is constructed and
consumed on scan workers one batch at a time. Deleted rows report only
`_rowid`s that were live at the begin version and are absent at the end
version, so inserts-then-deletes and compaction-only rewrites do not appear
as deletions. Stable row ids are required at both endpoints unless the range
is empty. Namespace addressing reuses the existing open-context
reconstruction.

Closes daft-engine#64
@hello-peter-tang

Copy link
Copy Markdown
Author

Heads-up: the PR Test Suite / PR Checks run for this branch is sitting in action_required, so the PR shows no CI results at all. GitHub is holding fork runs until a maintainer approves them ("Approve and run"). @maintainers, could someone approve the pending run so this gets tested?

@hello-peter-tang

Copy link
Copy Markdown
Author

Same as my note on #88: CI is still parked in action_required, so I ran the checks locally on the PR head 6c778d9:

ruff check daft_lance tests            -> All checks passed!
ruff format --check daft_lance tests   -> 51 files already formatted
mypy --strict --ignore-missing-imports daft_lance
                                       -> Success: no issues found in 16 source files
pytest tests/ -q                       -> 602 passed, 10 skipped, 2 xfailed, 2 xpassed in 30.92s

Versus main on the same setup (552 passed, 10 skipped, 2 xfailed, 2 xpassed) this branch adds 50 passing tests, matching the new test_lance_delta.py cases. Environment: Linux x86_64, Python 3.11.6, deps from the committed uv.lock, DAFT_ANALYTICS_ENABLED=0. The 3.10/3.12/3.13 matrix and the Ray job still need a real CI run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose version-range Lance changes as lazy Daft DataFrames

1 participant