Skip to content

SNOW-4483962: Support change-tracking reads (CHANGES clause) and END - #4505

Open
sfc-gh-yixie wants to merge 1 commit into
mainfrom
yixie-SNOW-4483962-changes-clause
Open

sfc-gh-yixie wants to merge 1 commit into
mainfrom
yixie-SNOW-4483962-changes-clause

Conversation

@sfc-gh-yixie

Copy link
Copy Markdown
Collaborator
  1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

    Fixes SNOW-4483962

  2. Fill out the following pre-review checklist:

    • I am adding a new automated test(s) to verify correctness of my new code
      • If this test skips Local Testing mode, I'm requesting review from snowflakedb/local-testing
    • I am adding new logging messages
    • I am adding a new telemetry message
    • I am adding new credentials
    • I am adding a new dependency
    • If this is a new feature/behavior, I'm adding the Local Testing parity changes.
    • I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: Thread-safe Developer Guidelines
    • If adding any arguments to public Snowpark APIs or creating new public Snowpark APIs, I acknowledge that I have ensured my changes include AST support. Follow the link for more information: AST Support Guidelines
  3. Please describe how your code solves the related issue.

    This PR adds Snowflake change-tracking reads, the CHANGES clause, plus the END bound of the change interval, by extending the existing time-travel arguments.

    session.table("orders", changes="default", time_travel_mode="at",
                  statement=qid_start, end_statement=qid_end)
    # FROM orders CHANGES (INFORMATION => DEFAULT) AT (STATEMENT => '…') END (STATEMENT => '…')
    • New keyword arguments on Session.table, DataFrameReader.table and Table:

      • changes ("default" or "append_only", case-insensitive)
      • end_statement, end_offset, end_timestamp, end_timestamp_type

      The same names work as DataFrameReader options, e.g. session.read.option("changes", "append_only").

    • Validation in TimeTravelConfig is kept minimal:

      • changes requires a time-travel mode and exactly one AT/BEFORE argument. Without them Snowflake silently returns no rows and no columns.
      • END arguments require changes, and only one may be given.
      • end_offset is converted to an int, so a string from an option can't add arbitrary SQL.

      Snowflake checks everything else; for example, it accepts BEFORE (TIMESTAMP => …) with CHANGES, although its docs only list STATEMENT. END timestamps are normalized the same way as the start timestamp, using a helper shared by both.

    • Metadata columns: new constants METADATA_ACTION, METADATA_ISUPDATE and METADATA_ROW_ID in snowflake.snowpark.column for the extra columns of a change-tracking read.

    • Local testing: a CHANGES read raises NotImplementedError. The mock does not model time travel, and without this check it would silently return the plain table.

    AST: not included in this PR (checkbox intentionally unticked). The Table and ReadTable AST messages are generated from the monorepo IR (Snowpark/ast/dataframe.ir and dataframe-io.ir). Recording the new arguments therefore needs an IR change there first. The new arguments are not recorded in the AST yet, the same as the existing Iceberg time-travel options. Code comments mark where they would go, and the IR and AST work will follow separately.

    Testing:

    • Unit tests (tests/unit/test_utils.py): SQL generation for both changes values, each END kind (including tz-aware datetimes and quote escaping), and every validation error.
    • Live integration tests (tests/integ/test_dataframe.py): on a change-tracking table with an insert, an update and a delete, they check the exact rows, including METADATA$ACTION/METADATA$ISUPDATE. They cover DEFAULT and APPEND_ONLY, AT and BEFORE, with and without END, the reader-option path, copy.copy, and the SQL simplifier on and off.
    • Local testing: a test asserts NotImplementedError. The local-testing runs of test_dataframe.py/test_function.py, tests/mock and the existing time-travel and AST tests still pass.

🤖 Generated with Claude Code

Add `changes` ("default" / "append_only") and `end_statement`, `end_offset`,
`end_timestamp`, `end_timestamp_type` to Session.table, DataFrameReader.table
(also as reader options) and Table. They emit
`CHANGES (INFORMATION => ...) AT|BEFORE (...) [END (...)]` on top of the
existing time travel clause. Validation: `changes` requires AT/BEFORE, END
requires `changes`, at most one END argument.

Add METADATA_ACTION, METADATA_ISUPDATE and METADATA_ROW_ID column constants.
Local testing raises NotImplementedError for CHANGES reads instead of silently
returning the plain table. The new parameters are not encoded in the AST yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.16393% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.98%. Comparing base (aad22f1) to head (5f7181e).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
src/snowflake/snowpark/_internal/utils.py 89.65% 4 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4505      +/-   ##
==========================================
- Coverage   95.47%   94.98%   -0.49%     
==========================================
  Files         179      179              
  Lines       46927    46979      +52     
  Branches     8110     8119       +9     
==========================================
- Hits        44803    44625     -178     
- Misses       1332     1513     +181     
- Partials      792      841      +49     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

local testing Local Testing issues/PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants