Skip to content

SNOW-4432455: Make the DataFrame lineage trace a per-session debug feature - #4487

Merged
sfc-gh-yuwang merged 5 commits into
mainfrom
yuwang-SNOW-4432455-lineage-session
Nov 11, 2026
Merged

sfc-gh-yuwang merged 5 commits into
mainfrom
yuwang-SNOW-4432455-lineage-session

Conversation

@sfc-gh-yuwang

@sfc-gh-yuwang sfc-gh-yuwang commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #4485 (SNOW-4432458, SQL error line mapping per session) and targets main: until #4484 and #4485 merge, the diff also contains their commits. This PR's own commits are the SNOW-4432455 ones.

Summary

  • Registers dataframe_trace_on_error as a trace debug feature: session.debug.enable_dataframe_trace_on_error() (or set(dataframe_trace_on_error=True)) adds the lineage of a failed dataframe to the errors of that session.
  • wrap_exception reads it from the session of the failed plan; is_tracing_enabled(session) reads only the session; tracing_flags_on() (the publicapi fast path, which has no session) checks the count of tracing sessions.
  • configure_development_features now sets all three features in every active session and as the defaults of new ones. The context module keeps no debug flag: removes context._enable_dataframe_trace_on_error.
  • CHANGELOG: adds dataframe_trace_on_error to the features of the Session.debug entry.

With this PR, session.debug.set(eager_schema_validation=True, dataframe_trace_on_error=True, trace_sql_errors_to_dataframe=True) or the per-feature session.debug.enable_<feature>() / disable_<feature>() configure one session, and configure_development_features(...) stays the shortcut for every session.

Test plan

  • New test_lineage_trace_is_per_session: two sessions in one process; only the session with the feature on adds the lineage to the error.
  • New unit test_tracing_flags_on_counts_the_tracing_sessions. The publicapi unit tests turn the trace on through a session instead of the global.
  • Unit: no new failures compared with main.
  • SFCTEST0, Python 3.11: SQL error, CTE, query line interval, test_df_trace_*.py, debug trace, context, session, eager validation and CTE suites: 921 passed, 1 skipped, 1 xpassed.
  • Python 3.10: CTE, test_df_trace_*.py, debug trace, eager validation and context suites: 622 passed, 1 skipped.

Generated with Snowflake CoCo

@sfc-gh-yuwang
sfc-gh-yuwang force-pushed the yuwang-SNOW-4432458-line-mapping-session branch from c62932f to 020e198 Compare November 9, 2026 18:16
@sfc-gh-yuwang
sfc-gh-yuwang force-pushed the yuwang-SNOW-4432455-lineage-session branch from 6524d44 to 88d69c0 Compare November 9, 2026 18:16
@sfc-gh-yuwang
sfc-gh-yuwang marked this pull request as ready for review November 9, 2026 18:16
@sfc-gh-yuwang
sfc-gh-yuwang requested a review from a team as a code owner November 9, 2026 18:16
@sfc-gh-yuwang
sfc-gh-yuwang requested review from sfc-gh-aling, sfc-gh-jzeng and sfc-gh-yixie and removed request for a team November 9, 2026 18:16
@sfc-gh-yuwang
sfc-gh-yuwang force-pushed the yuwang-SNOW-4432458-line-mapping-session branch from 020e198 to a2090c7 Compare November 10, 2026 18:05
@sfc-gh-yuwang
sfc-gh-yuwang force-pushed the yuwang-SNOW-4432455-lineage-session branch from 88d69c0 to 27e9bf3 Compare November 10, 2026 18:05
@codecov-commenter

codecov-commenter commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.71429% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.47%. Comparing base (aad22f1) to head (40c1fb5).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...lake/snowpark/_internal/analyzer/snowflake_plan.py 71.42% 0 Missing and 4 partials ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4487   +/-   ##
=======================================
  Coverage   95.47%   95.47%           
=======================================
  Files         179      179           
  Lines       46927    46922    -5     
  Branches     8110     8106    -4     
=======================================
- Hits        44803    44800    -3     
+ Misses       1332     1331    -1     
+ Partials      792      791    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sfc-gh-yuwang
sfc-gh-yuwang force-pushed the yuwang-SNOW-4432458-line-mapping-session branch from a2090c7 to 5a347f9 Compare November 10, 2026 19:42
@sfc-gh-yuwang
sfc-gh-yuwang force-pushed the yuwang-SNOW-4432455-lineage-session branch from 27e9bf3 to 388ab40 Compare November 10, 2026 19:42
@sfc-gh-yuwang
sfc-gh-yuwang force-pushed the yuwang-SNOW-4432458-line-mapping-session branch from 5a347f9 to 882bfdf Compare November 10, 2026 22:15
@sfc-gh-yuwang
sfc-gh-yuwang force-pushed the yuwang-SNOW-4432455-lineage-session branch from 388ab40 to 10db98f Compare November 10, 2026 22:15
@sfc-gh-yuwang
sfc-gh-yuwang force-pushed the yuwang-SNOW-4432458-line-mapping-session branch from 882bfdf to cde2afd Compare November 10, 2026 22:57
@sfc-gh-yuwang
sfc-gh-yuwang force-pushed the yuwang-SNOW-4432455-lineage-session branch from 10db98f to 16924c9 Compare November 10, 2026 22:57
@sfc-gh-yuwang
sfc-gh-yuwang force-pushed the yuwang-SNOW-4432458-line-mapping-session branch from cde2afd to 09e040f Compare November 11, 2026 03:15
@sfc-gh-yuwang
sfc-gh-yuwang force-pushed the yuwang-SNOW-4432455-lineage-session branch from 16924c9 to abb3a9e Compare November 11, 2026 03:15
@sfc-gh-yuwang
sfc-gh-yuwang changed the base branch from yuwang-SNOW-4432458-line-mapping-session to main November 11, 2026 03:22
sfc-gh-yuwang and others added 5 commits November 10, 2026 21:38
…ature

session.debug.enable(dataframe_trace_on_error=True) adds the lineage of a failed dataframe to the errors of one session; configure_development_features sets every feature through the sessions, and the context module keeps no debug flag. tracing_flags_on() counts the sessions that record the trace.

Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
….debug

Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
…e failed query has a plan

So pyright knows the plan is not None where it is used; the behavior is unchanged.

Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
@sfc-gh-yuwang
sfc-gh-yuwang force-pushed the yuwang-SNOW-4432455-lineage-session branch from abb3a9e to 40c1fb5 Compare November 11, 2026 05:44
@sfc-gh-yuwang
sfc-gh-yuwang merged commit d5e1f8d into main Nov 11, 2026
27 of 30 checks passed
@sfc-gh-yuwang
sfc-gh-yuwang deleted the yuwang-SNOW-4432455-lineage-session branch November 11, 2026 06:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants