Skip to content

[Spanner Change Stream] Fix Change Streams PostgreSQL metadata queries - #40755

Open
jiangzzhu wants to merge 1 commit into
apache:masterfrom
jiangzzhu:fix-changestream-pg-and-schema-bugs
Open

jiangzzhu wants to merge 1 commit into
apache:masterfrom
jiangzzhu:fix-changestream-pg-and-schema-bugs

Conversation

@jiangzzhu

@jiangzzhu jiangzzhu commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR fixes two PostgreSQL metadata query issues in the Spanner Change Streams connector (PartitionMetadataDao):

  1. PartitionMetadataDao.tableExists() on PostgreSQL-dialect databases:

    • Previously, tableExists() hardcoded WHERE t.table_catalog = '' AND t.table_schema = ''. In PostgreSQL-dialect Spanner databases, table_catalog is the database name and user tables reside in table_schema = 'public', causing tableExists() to always return false.
    • When InitializeDoFn runs more than once (e.g., on retry or pipeline restart with an existing metadata table), createPartitionMetadataTable() (CREATE TABLE IF NOT EXISTS) succeeds as a no-op, and createFakeParentPartition() subsequently fails with ALREADY_EXISTS: Row [Parent0] in table Metadata_... already exists.
    • Updated tableExists() to check t.table_schema = 'public' when this.isPostgres() is true (consistent with findAllTableIndexes()).
  2. PartitionMetadataDao.getPartitionsMatchingState() on PostgreSQL-dialect databases:

    • Updated the PostgreSQL branch of getPartitionsMatchingState() to use bound query parameters (WHERE "State" = $1 AND "PartitionToken" = ANY ($2)) instead of string concatenation, and to omit the "PartitionToken" filter when composedPartitionTokens is empty (matching the GoogleSQL branch behavior).

Testing

  • Added unit tests in PartitionMetadataDaoTest covering both GoogleSQL and PostgreSQL dialects.

@github-actions github-actions Bot added java io gcp spanner size/m M: 50-249 lines changed labels Oct 23, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR resolves three important bugs in Spanner Change Streams (SpannerIO.readChangeStream()): fixing PartitionMetadataDao.tableExists() for PostgreSQL-dialect databases by checking table_schema = 'public', parameterizing getPartitionsMatchingState() and short-circuiting empty token updates in updateToScheduled(), and adding support for schema-qualified change stream and TVF names across SpannerIO and ChangeStreamDao. The changes are well-structured, adhere to existing SpannerIO patterns, and include comprehensive unit tests for both GoogleSQL and PostgreSQL dialects.

🔍 General Feedback

  • Switching PartitionMetadataDao.getPartitionsMatchingState() in the PostgreSQL branch from string concatenation to bound query parameters ($1 and ANY ($2)) is a great improvement for both query plan caching and safety.
  • The schema-qualification handling across fetchPartitionMode, checkTvfExistence, and ChangeStreamDao.changeStreamQuery cleanly accounts for the differences between GoogleSQL (<schema>.READ_<name>) and PostgreSQL ("spanner"."read_json_<name>" / "spanner"."read_proto_bytes_<name>").

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@jiangzzhu
jiangzzhu marked this pull request as draft October 23, 2026 19:43
@jiangzzhu
jiangzzhu force-pushed the fix-changestream-pg-and-schema-bugs branch from 72bf8ca to 77dbaa6 Compare October 23, 2026 21:04
@jiangzzhu jiangzzhu changed the title [SpannerIO] Fix Change Streams PostgreSQL metadata queries and schema-qualified change streams [SpannerIO] Fix Change Streams PostgreSQL metadata queries Oct 23, 2026
@jiangzzhu
jiangzzhu marked this pull request as ready for review October 23, 2026 21:25

@chenxuesdu chenxuesdu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

Copy link
Copy Markdown
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@jiangzzhu jiangzzhu changed the title [SpannerIO] Fix Change Streams PostgreSQL metadata queries [Spanner Change Streams] Fix Change Streams PostgreSQL metadata queries Oct 23, 2026
@jiangzzhu jiangzzhu changed the title [Spanner Change Streams] Fix Change Streams PostgreSQL metadata queries [SpannerIO] Fix Change Streams PostgreSQL metadata queries Oct 23, 2026
@jiangzzhu
jiangzzhu force-pushed the fix-changestream-pg-and-schema-bugs branch from 77dbaa6 to 4ece3c1 Compare October 23, 2026 22:43
@jiangzzhu jiangzzhu changed the title [SpannerIO] Fix Change Streams PostgreSQL metadata queries [Spanner Change Stream] Fix Change Streams PostgreSQL metadata queries Oct 23, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @nielm for label spanner.
Backup reviewer: chamikaramj (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants