Skip to content

Add PostgreSQL partial-column inserts with composable CTE support - #827

Merged
LaurentRDC merged 13 commits into
haskell-beam:masterfrom
kushagarr:cte_support_postgres
Sep 28, 2026
Merged

LaurentRDC merged 13 commits into
haskell-beam:masterfrom
kushagarr:cte_support_postgres

Conversation

@kushagarr

@kushagarr kushagarr commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds PostgreSQL-specific partial-column inserts that preserve ON CONFLICT support and compose cleanly with RETURNING and data-modifying CTEs.

Motivation

The existing PostgreSQL insert helpers support insertValues, insertExpressions, and insertFrom, but require values for every table column. This prevents CTE inserts from omitting generated, identity, or database-defaulted columns.

Beam’s backend-independent insertOnly supports partial-column inserts, but cannot express PostgreSQL’s ON CONFLICT behavior. Beam also provides backend-independent conflict handling through BeamHasInsertOnConflict, whose insertOnConflict accepts full-table values. pgInsertOnly adds partial-target selection with conflict handling.

For query-backed inserts, default_ cannot substitute for omitted columns: PostgreSQL rejects SELECT DEFAULT, source.name FROM source, while INSERT INTO users (name) SELECT source.name FROM source lets the omitted column use its default.

Changes

  • Add pgInsertOnly for partial-column PostgreSQL inserts with PgInsertOnConflict.
  • Add cteInsertCommand to lift any SqlInsert Postgres table into a side-effect-only CTE.
  • Add cteInsertCommandReturning to attach a RETURNING projection and expose the inserted rows as a reusable CTE.
  • Refactor the existing full-row insert helpers through the same implementation.
  • Preserve the existing insert, insertReturning, cteInsert, and cteInsertReturning APIs.
  • Document the new composition model and record the APIs for beam-postgres-0.6.4.0.
  • Allow the integration suite to use an externally managed PostgreSQL instance through BEAM_POSTGRES_TEST_CONNSTR.

Design

Partial-column selection is treated as a property of insert construction, while conflict handling, RETURNING, and CTE placement remain independently composable:

command =
  Pg.pgInsertOnly table selectedFields values onConflict

runInsert command
returning command resultProjection
Pg.cteInsertCommand command
Pg.cteInsertCommandReturning command resultProjection

pgInsertOnly delegates target-column projection and base SQL generation to Beam’s existing insertOnly implementation, then adds PostgreSQL’s conflict clause. This avoids maintaining a second insert renderer and keeps column ordering, quoting, empty-input behavior, and type checking aligned with the backend-independent implementation.

The command-level CTE adapters also avoid introducing narrowly specialized partial-insert CTE APIs.

Compatibility

This is additive and does not require changes to existing user code. Existing PostgreSQL insert and CTE helpers retain their signatures and behavior.

cteInsertCommandReturning returns Nothing only for SqlInsertNoRows. A valid ON CONFLICT DO NOTHING command still returns a reusable relation, which may contain no rows after execution.

For both command adapters, SqlInsert does not carry the target table’s database type, so that type is not tied to the enclosing PgWith block.

Validation

  • Full beam-postgres test suite: 149 tests passed against PostgreSQL 18.4 with GHC 9.6.7.
  • Added rendering and integration coverage for:
    • selected-column ordering;
    • insertData and insertFrom;
    • generated identities and database defaults;
    • ON CONFLICT DO NOTHING;
    • ON CONFLICT DO UPDATE and excluded values;
    • SQL parameter ordering;
    • side-effect-only and returning CTEs;
    • empty inserts and zero-column projections;
    • compatibility between existing helpers and the composable command path.
  • Added negative type tests for invalid CTE placement and mismatched insert projections.
  • Haddock generation completed successfully.
  • git diff --check passes.

kushagarr added 12 commits July 11, 2026 05:19
- Introduced `pgWithSyntax` to prefix PostgreSQL statements with CTEs.
- Added new types for data-modifying CTEs: `PgDataModifyingCommonTableExpressionSyntax` and `PgCommonTableExpressionSyntax`.
- Implemented type classes for handling CTEs in SQL queries, ensuring correct placement of data-modifying CTEs.
- Created unit and integration tests for rendering and type safety of CTEs, including negative tests for invalid placements.
- Updated documentation to reflect new CTE capabilities and usage examples.
- Modified test setup to use PostgreSQL version 18.4 for better compatibility.
- Updated CTE handling in tests to reflect PostgreSQL's behavior with zero-column CTEs and side-effect-only CTEs.
- Added new tests for materialization execution and rendering, ensuring compliance with PostgreSQL 12+ features.
- Refined error handling for invalid CTE placements, particularly for side-effect-only operations.
- Improved documentation on PostgreSQL-specific CTE usage, including materialization options and nested CTEs.
- Adjusted dependencies in the SQLite library for compatibility.
- Ensured reproducibility of CI runs by pinning PostgreSQL server version.
@kushagarr

Copy link
Copy Markdown
Contributor Author

@LaurentRDC could you check this please.

@LaurentRDC

Copy link
Copy Markdown
Member

I haven't forgotten about this; I'm just swamped at work right now

Comment thread beam-postgres/Database/Beam/Postgres/Full.hs
@LaurentRDC
LaurentRDC marked this pull request as ready for review September 21, 2026 19:35
Comment thread beam-postgres/Database/Beam/Postgres/Full.hs Outdated
@LaurentRDC

Copy link
Copy Markdown
Member

Looks good to me overall.
I've marked this ready-for-review because I thought that was needed to run CI, but that's not true. So let me know if you plan on making further changes

@kushagarr

kushagarr commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the review — pushed a finishing pass, and I'm not planning further changes.

Conflict-support docs (your comment on pgInsertOnly): added the reference to
BeamHasInsertOnConflict in both the Haddock and the user guide, with the distinction
made explicit — insertOnConflict accepts full-table SqlInsertValues, so it can't
express a partial target.

Renamed the new CTE adapters: pgCteInsert → cteInsertCommand and
pgCteInsertReturning → cteInsertCommandReturning. What separates them from
cteInsert/cteInsertReturning isn't that they're Postgres-specific — it's that they take an already-built SqlInsert instead of table + values + conflict action. The pg prefix didn't convey that; Command does. It also keeps
pgCteInsert available in case the existing names are ever normalised.

Documented a limitation: SqlInsert doesn't carry the target table's database type, so
for the command-level adapters that type isn't tied to the enclosing PgWith block. Noted
on both functions and in the user guide. The existing cteInsert/cteInsertReturning
wrappers keep their signatures and their db linkage.

Version: corrected 0.6.3.1 → 0.6.4.0. This release only adds bindings. Updated the
cabal version, the ChangeLog and the @since annotations to match.

Existing exports and behaviour are unchanged.

If it looks good to you we can merge it and make a new release please @LaurentRDC

@LaurentRDC LaurentRDC left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your patience! I will cut a new release today

@LaurentRDC
LaurentRDC merged commit da95b85 into haskell-beam:master Sep 28, 2026
13 checks passed
@LaurentRDC

Copy link
Copy Markdown
Member

Released on Hackage

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants