Skip to content

feat(sink): fail fast on incompatible postgres target tables - #26678

Draft
yuhao-su wants to merge 1 commit into
yuhao/pg-sink-shared-batching-log-sinkerfrom
yuhao/pg-sink-fail-fast-probe
Draft

feat(sink): fail fast on incompatible postgres target tables#26678
yuhao-su wants to merge 1 commit into
yuhao/pg-sink-shared-batching-log-sinkerfrom
yuhao/pg-sink-fail-fast-probe

Conversation

@yuhao-su

@yuhao-su yuhao-su commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Stacked on #26677 (which is stacked on #26671).

Statements are prepared lazily since #26671, so an incompatible target table — a unique constraint dropped after CREATE SINK, a role missing a DML privilege, a read-only endpoint — only surfaced at the first flush, with data already flowing through the sink. The postgres writer constructor now fails fast:

  • Read-only targets are rejected (pg_is_in_recovery() / transaction_read_only = on, e.g. an RDS/Aurora reader endpoint). This needs its own check because EXPLAIN is exempt from PostgreSQL's read-only enforcement.
  • A single-row write — and, for upsert sinks, a single-row delete — is planned via EXPLAIN with NULL binds. Planning resolves the ON CONFLICT arbiter index and runs executor-start privilege checks; both are invisible to prepare, which performs parse analysis only (verified at the protocol level: PREPARE of a delete succeeds without the DELETE privilege, EXPLAIN of it fails; PREPARE of an upsert succeeds on a table without a matching unique index, EXPLAIN fails with SQLSTATE 42P10). Nothing is executed: EXPLAIN without ANALYZE writes no rows.
  • The single-row statements are prepared and cached: parse-level fail-fast, and size 1 is a genuine tail size of the power-of-two batch path.

The probe's bind arity now comes from a shared params_per_tuple(kind) (also used by prepare_batches), with a unit test pinning it to the $n placeholder count of each statement kind.

Documented limit: executor-init-only checks (e.g. a DEFERRABLE arbiter) are not exercised by EXPLAIN and still surface at the first flush.

Verification

  • Unit tests (10, including the new placeholder-arity test) and clippy clean.
  • Probe mechanics verified against PostgreSQL 18.4 at the protocol level (extended-protocol EXPLAIN with NULL binds plans and executes nothing; arbiter shown in the plan; privilege and read-only behaviors as described above).
  • e2e_test/sink/postgres_sink.slt (append-only + upsert + batching sections) passes on a local cluster with the probes active at every writer startup.
  • Read-only fail-fast end to end: a sink created against a database with default_transaction_read_only = on fails writer construction with the new error and retries with backoff; after ALTER DATABASE ... SET default_transaction_read_only = off, the sink recovers on its own and drains the backlog (row verified present in the target table).
  • Fail-fast end to end (manual, earlier on this stack): swapping the target table's pk to a different column and triggering RECOVER makes writer construction fail with failed to plan the upsert statement against the target table: ... there is no unique or exclusion constraint matching the ON CONFLICT specification before any row is written; restoring the pk and recovering resumes the sink and drains the backlog.

Known follow-ups (separate PRs)

  • Run the same EXPLAIN probe in validate() as well, so mismatches knowable at DDL time fail CREATE SINK instead of sink startup.
  • A condeferrable check in validate() to close the DEFERRABLE-arbiter gap at DDL time.

Checklist

  • I have written necessary rustdoc comments.
  • I have added necessary unit tests and integration tests.
  • I have added test labels as necessary.
  • My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.

Documentation

  • My PR needs documentation updates.

🤖 Generated with Claude Code

Statements are prepared lazily since the multi-row batching change, so an
incompatible target table (constraint dropped after CREATE SINK, missing DML
privilege, read-only endpoint) only surfaced at the first flush, with data
already flowing. The writer constructor now:

- rejects read-only targets (hot standby / transaction_read_only=on), which
  EXPLAIN alone cannot catch;
- EXPLAINs a single-row write and, for upsert sinks, a single-row delete:
  planning resolves the ON CONFLICT arbiter and runs executor-start privilege
  checks, both invisible to prepare;
- pre-warms the size-1 statements, a genuine tail size of the batch path.

Known limit documented on the probe: executor-init-only checks (e.g.
DEFERRABLE arbiters) still surface at the first flush.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@yuhao-su yuhao-su added type/feature Type: New feature. ci/run-e2e-postgres-sink-tests Run Postgres sink e2e tests in your PR. labels Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/run-e2e-postgres-sink-tests Run Postgres sink e2e tests in your PR. type/feature Type: New feature.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant