Add INSERT ... SELECT and ClickHouse insert settings

CHANGELOG.md

@@ -16,6 +16,11 @@
 - Add `onConflictDoNothing` and `onConflictDoUpdate` to an insert (Postgres), with Drizzle's
   options: `target` (columns or `{ constraint }`), `targetWhere`, `set` (a record, or a callback
   over the existing row and `excluded`) and `where`. Add `DialectClauses.onConflict`, optional.
+- Add `select(query)` to an insert: `INSERT ... SELECT` from a query or union, its row checked
+  against the table at the type level. Tenant scope follows the read and where the written
+  tenant comes from.
+- Add `settings(record)` to an insert (ClickHouse): `INSERT ... SETTINGS name = value`. Add
+  `DialectClauses.insertSettings`, optional.
 - Add `CompiledQuery.kind` (`"select"` or `"insert"`); `rawCompiledQuery` takes it as an option.
 - Add `ParamStyle.maxParameters`; Postgres sets 65535, and a statement over it fails to compile.
 - Add `@maple-dev/effect-orm/database`, opt-in (see `docs/database.md`): a `Database` over the

design/writes.md

@@ -1,7 +1,8 @@
 # Writes: INSERT
 
-Status: phases 1 to 3 built (`values`, `returning`, `onConflictDoNothing` /
-`onConflictDoUpdate`, see `docs/inserts.md`); phase 4 not started. Section 11 lists where the build differs from the plan. UPDATE and DELETE come later and
+Status: phases 1 to 4 built (`values`, `returning`, `onConflictDoNothing` /
+`onConflictDoUpdate`, `select`, `settings`; see `docs/inserts.md`). `encodeInsertRows` (§7) is
+not built: no consumer has asked for it. Section 11 lists where the build differs from the plan. UPDATE and DELETE come later and
 will reuse what this note sets up (the write-statement state, the `RETURNING` path, value
 encoding).
 
@@ -250,3 +251,10 @@ note, reusing `kind: "write"`, the returning path and the `set` record type from
   `.onConflict(target).doUpdate(...)` in §6: Maple's 68 call sites then move over with renames.
   `$` in `set` and `where` is qualified with the table name (`"counters"."count"`), because an
   unqualified column there is ambiguous with `excluded`.
+- **`INSERT ... SELECT` checks types with its own `InsertSelectMisfits` / `InsertSelectMissing`**
+  rather than moving `MisfitColumns` out of `schema/define.ts`: an insert also has to exclude
+  computed columns and require the ones without a default, which a materialized view does not.
+  Its tenant scope is the SELECT's for an untenanted target (the read), and for a tenant target
+  single-tenant only when the read is and each row takes its tenant from a source tenant column
+  or the same param.
+- **Settings are a dialect clause**, `DialectClauses.insertSettings`, like the others.

docs/inserts.md

@@ -90,6 +90,37 @@ the codec rejects fails to compile with a `QueryBuilderError` that names the row
   several rows is bound once. A statement over 65535 bound values (Postgres's limit) fails to
   compile instead of being split: send fewer rows per statement.
 
+## Insert ... select
+
+`select(query)` inserts the rows a query (or a `unionAll`) selects, instead of `values`. Each
+selected alias names the column it goes into, so select under the target's column names:
+
+```ts
+const Spans = CH.table("spans", { OrgId: CH.string, Name: CH.string, Ms: CH.uint64 }, { tenantColumn: "OrgId" })
+const Daily = CH.table("daily", { OrgId: CH.string, Name: CH.string, Total: CH.uint64 }, { tenantColumn: "OrgId" })
+
+CH.insertInto(Daily).select(
+	CH.from(Spans)
+		.select(($) => ({ OrgId: $.OrgId, Name: $.Name, Total: CH.sum($.Ms) }))
+		.where(($) => [$.OrgId.eq(CH.param.string("orgId"))])
+		.groupBy("OrgId", "Name"),
+)
+// INSERT INTO daily (OrgId, Name, Total)
+// SELECT ... FROM spans WHERE spans.OrgId = 'o1' GROUP BY OrgId, Name
+```
+
+The selected row is checked against the table: selecting a column the table does not have (or
+a computed one), selecting a value of another type, or leaving out a required column is a type
+error naming the columns (`targetCannotTake`, `missingColumns`). A nullable result, such as a
+Postgres `sum`, does not fit a NOT NULL column; wrap it in `coalesce`.
+
+The column list is the aliases in select order, which is the order the SELECT writes them, so
+ClickHouse and Postgres agree. `returning` and `onConflict*` work with `select` as with
+`values`. `select` and `values` replace each other.
+
+A long `INSERT ... SELECT` on ClickHouse (a backfill over a big table) can outlast an HTTP
+timeout; run those in slices.
+
 ## Returning
 
 On Postgres, `returning` adds a RETURNING list and `Database.run` returns the inserted rows,
@@ -149,12 +180,31 @@ RETURNING "key" AS "key", "count" AS "count"
 - Calling either again replaces the clause. ClickHouse has no `ON CONFLICT` (deduplicate with a
   `ReplacingMergeTree` instead), so compiling one for it is a `QueryBuilderDefect`.
 
+## Settings
+
+On ClickHouse, `settings` adds a `SETTINGS` clause to the insert, before `VALUES` or the
+`SELECT`:
+
+```ts
+CH.insertInto(Daily).values(rows).settings({ async_insert: 1, wait_for_async_insert: 1 })
+// INSERT INTO daily (OrgId, Name, Total) SETTINGS async_insert = 1, wait_for_async_insert = 1
+// VALUES ...
+```
+
+Names must be plain identifiers; values (strings, numbers, booleans) are written as literals.
+Calling it again replaces them. Postgres has no insert settings, so compiling one with
+`settings` for it is a `QueryBuilderDefect`.
+
 ## Tenant scope
 
 An insert has a `tenantScope` like a query, worked out the same way. On a table with a
 `tenantColumn`, it is `"single-tenant"` when every row gives that column the same value or the
 same param, and `"cross-tenant"` when rows differ or a row uses another expression. An
-`onConflictDoUpdate` that sets the tenant column counts as one more row. A table
+`onConflictDoUpdate` that sets the tenant column counts as one more row.
+
+An `INSERT ... SELECT` into a tenant table is `"single-tenant"` when the SELECT is, and each row
+takes its tenant from a tenant column of the source or from the same param that pins the SELECT.
+Into a table without a tenant column, the insert has the SELECT's scope: what it reads. A table
 without a tenant column gives `"untenanted"`.
 
 ## Failures
@@ -170,6 +220,7 @@ without a tenant column gives `"untenanted"`.
 | `returning` for a dialect without RETURNING       | `QueryBuilderDefect`                     |
 | `onConflictDoUpdate` setting no or unknown columns | `QueryBuilderError` `InvalidArguments`  |
 | `onConflict*` without ON CONFLICT, a bad target   | `QueryBuilderDefect`                     |
+| `settings` without insert settings, a bad name    | `QueryBuilderDefect`                     |
 
 _(Backed by `src/ch/insert.test.ts`, `src/database/database.test.ts` and
 `tests/database.clickhouse.test.ts`.)_

docs/reference.md

@@ -62,7 +62,7 @@ Note `/sql` exports a `compile` (fragment → string) distinct from the root `co
 | `fromQuery` | `(query, alias) => CHQuery`               |
 | `fromUnion` | `(union, alias) => CHQuery`               |
 | `unionAll`  | `(...queries) => CHUnionQuery`            |
-| `insertInto` | `(table) => CHInsert`; `.values(row \| rows)` sets its rows, `.returning(...)` the RETURNING list, `.onConflictDoNothing(options?)` / `.onConflictDoUpdate(options)` the ON CONFLICT clause (Postgres). See [Inserting rows](./inserts.md) |
+| `insertInto` | `(table) => CHInsert`; `.values(row \| rows)` or `.select(query)` sets its rows, `.settings(record)` ClickHouse `SETTINGS`, `.returning(...)` the RETURNING list, `.onConflictDoNothing(options?)` / `.onConflictDoUpdate(options)` the ON CONFLICT clause (Postgres). See [Inserting rows](./inserts.md) |
 
 ### `CHQuery` methods
 
@@ -279,7 +279,7 @@ Types: `WindowSpec`, `CompiledWindowSpec`, `WindowFrameBound`, `WindowRowsFrame`
 
 **Everything else** — `Table`, `TableOptions`, `Expr`, `ColumnRef`, `Condition`, `Comparable`
 (what a value of a type may be compared against), `MapValueOf`, `Subquery`, `ParamMarker`,
-`ParamKind`, `CHQuery`, `CHUnionQuery`, `CHInsert`, `InsertRow`, `InsertRowOf`, `InsertValue`, `ConflictTarget`, `ConflictSet`, `OnConflictDoNothing`, `OnConflictDoUpdate`, `ColumnAccessor`, `JoinedColumnAccessor`,
+`ParamKind`, `CHQuery`, `CHUnionQuery`, `CHInsert`, `InsertRow`, `InsertRowOf`, `InsertValue`, `InsertSelectMisfits`, `InsertSelectMissing`, `InsertSettingValue`, `ConflictTarget`, `ConflictSet`, `OnConflictDoNothing`, `OnConflictDoUpdate`, `ColumnAccessor`, `JoinedColumnAccessor`,
 `JoinOnCallback`, `CompiledQuery`, `CompiledQueryInput`, `CompiledQueryRowSchema`, `RowSchemaMismatch`, `TenantScope`, `Dialect`, `DialectClauses`, `DialectTransactions`, `IsolationLevel`, `TransactionSettings`, `ParamStyle`, `FnResult`,
 `WindowFunnelMode`, `WindowSpec`, `WindowRowsFrame`, `WindowFrameBound`,
 `WindowOrderDirection`, `CompiledWindowSpec`.