Add CH.sql templates inside expressions, and Db.sql join/raw/empty

.gitignore

@@ -1,4 +1,4 @@
-node_modules/
+node_modules
 dist/
 coverage/
 *.tgz

CHANGELOG.md

@@ -2,6 +2,11 @@
 
 ## Unreleased
 
+- Add `CH.sql`: SQL templates inside expressions and conditions. `CH.sql(type)\`…\`` is a typed
+  `Expr`, ``CH.sql`…` `` an untyped one, `CH.sql.cond` a `Condition`; with `sql.ident`, `sql.raw`
+  and `sql.join`. Interpolated columns and params render as SQL and placeholders, a builder
+  query as a subquery compiled with the outer one, and a plain value as an escaped literal.
+- Add `Db.sql.join`, `Db.sql.raw` and `Db.sql.empty` to statement templates.
 - Add `isNull()`, `isNotNull()`, `between()` and `notBetween()` on every expression, and
   variadic `CH.and(...)` / `CH.or(...)` that skip `undefined` and write one flat group.
 - Add `distinct()` and `distinctOn(...aliases)` to queries, on both dialects.

design/gap-review.md

@@ -26,7 +26,7 @@ builder; **P1** commonly used; **P2** niche.
 | --- | --- | --- |
 | ~~UPDATE builder: SET values and expressions, WHERE, RETURNING~~ (built) | ~120 | M |
 | ~~DELETE builder: WHERE, RETURNING~~ (built) | ~79 | S |
-| A typed, value-binding `sql` template usable inside expressions; `sql.join` / `raw` / `empty` on `Db.sql` | ~163 | M |
+| ~~A typed `sql` template usable inside expressions; `sql.join` / `raw` / `empty` on `Db.sql`~~ (built: `CH.sql`; plain values are literals, params are bound) | ~163 | M |
 | Postgres column types: `timestamptz` as `Date`, `timestamp`, `date`, `interval`, `varchar(n)`, serial / identity | 226 timestamp columns | S |
 | ~~DISTINCT (and DISTINCT ON)~~ (built) | ~10 | S |
 | ~~`FOR UPDATE` / `FOR SHARE` / `SKIP LOCKED` / `NOWAIT`~~ (built) | 7 | S |

docs/database.md

@@ -122,6 +122,19 @@ const claimed = yield* Db.query(
 )  // ReadonlyArray<{ org_id: string; family: string }>
 ```
 
+`sql.join(values, separator?)` binds one value per item (or splices a `sql` item), joined by
+`sql\`, \`` unless you pass another separator; `sql.raw(text)` splices text you control; and
+`sql.empty` writes nothing, for an optional part. A `join` of no values fails when the statement
+renders, since `IN ()` is not SQL. Templates, identifiers and raw text are recognised by identity,
+so an object parsed from request JSON is bound as a value, never spliced:
+
+```ts
+Db.sql`SELECT * FROM t WHERE id IN (${Db.sql.join(ids)})${archived ? Db.sql` AND archived` : Db.sql.empty}`
+```
+
+For SQL inside a builder query rather than a whole statement, use
+[`CH.sql`](./extending.md#chsql--sql-templates-inside-a-query).
+
 `query` and `execute` also take a plain `{ sql, parameters }` for SQL you have as text.
 
 `FromSqlClientOptions`:

docs/extending.md

@@ -152,9 +152,56 @@ literal for all three.
 
 _(Backed by `src/ch/literal.test.ts > param.of`.)_
 
+## `CH.sql` — SQL templates inside a query
+
+For SQL the builder has no syntax for — a cast, an operator, a Postgres function — write a
+template. It is an expression (or, with `.cond`, a condition), so it goes anywhere the builder
+takes one: a select, a `where`, a join's ON, an UPDATE's SET.
+
+```ts
+CH.from(Keys)
+	.select(($) => ({
+		txid: CH.sql(PG.text)`pg_current_xact_id()::xid::text`,
+		next: CH.sql(PG.int8)`${$.uses} + ${1}`,
+	}))
+	.where(($) => [CH.sql.cond`${$.meta} @> ${CH.param.string("filter")}::jsonb`])
+// SELECT (pg_current_xact_id()::xid::text) AS "txid", ("keys"."uses" + 1) AS "next" …
+// WHERE ("keys"."meta" @> $1::jsonb)
+```
+
+Each `${value}` renders as the rest of the builder renders it:
+
+| Value | Renders as |
+| --- | --- |
+| a column, expression, or another template | its SQL |
+| a `param.*` | a placeholder: bound on Postgres, a literal on ClickHouse |
+| a builder query | `(subquery)`, compiled with the outer query, its tenant scope counted |
+| a string, number, boolean, `Date`, `DateTime.Utc`, `null` | the dialect's escaped literal |
+| `CH.sql.ident(name)` | the name quoted by the dialect; plain names only, dotted for `schema.table` |
+| `CH.sql.raw(text)` | the text as-is — never from input |
+| `CH.sql.join(values, separator?)` | each value rendered, joined by `", "` or `separator`; not parenthesized, so it fits `IN (${…})`; an empty list fails the compile |
+
+A template is written in parentheses, so `CH.sql.cond\`a OR b\`` in a `where` list stays one
+operand instead of swallowing the conditions it is AND-joined with. A negative number (or a
+param ClickHouse inlines as one) is parenthesized too, so `10-${n}` cannot become the comment
+`10--1`.
+
+`sql.raw` and `sql.ident` values are recognised by identity, not by their fields, so an object
+parsed from request JSON can never pass for one. An array or object has no literal the template
+could write without its SQL type, so it fails the compile with a `QueryBuilderError`; pass it as
+`param.of(type, name)` instead. A `unionAll` cannot be interpolated; select from it with
+`fromUnion` and interpolate that. `CH.sql(type)`
+declares the result type, which decodes the value when it is selected; a bare ``CH.sql`…` ``
+has none and costs the query its row schema, as `untypedExpr` does. A template condition is not
+evidence of tenant scope; being parenthesized, it cannot cancel the evidence of the conditions
+beside it either.
+
+_(Backed by `src/ch/sql-template.test.ts` and `src/database/database.test.ts`.)_
+
 ## Raw escape hatches
 
-`rawExpr` and `rawCond` take a SQL string as-is. `rawExpr` still requires the column type its
+`rawExpr` and `rawCond` take a SQL string as-is; prefer `CH.sql`, which renders values and
+params instead of taking text. `rawExpr` still requires the column type its
 SQL produces, so the row it lands in can still be decoded:
 
 ```ts

docs/reference.md

@@ -118,6 +118,7 @@ time; see [Params and compilation](./params-and-compilation.md#what-each-kind-ac
 | Export                    | Purpose                                                    |
 | ------------------------- | ---------------------------------------------------------- |
 | `lit(value)`              | Literal `Expr` from a `string` or `number`                 |
+| `sql(type)\`…\`` / `sql\`…\`` / `sql.cond\`…\`` | A template `Expr` (typed or untyped) or `Condition`; `sql.ident`, `sql.raw`, `sql.join`. See [Extending](./extending.md#chsql--sql-templates-inside-a-query). Types `SqlTag`, `SqlTemplateValue`, `SqlRaw`, `SqlIdent` |
 | `rawExpr(sql, type)`      | Unescaped `Expr` from SQL text, with a declared type       |
 | `untypedExpr<T>(sql)`     | Unescaped `Expr` with no type — costs the row schema       |
 | `rawCond(sql)`            | Unescaped `Condition` from SQL text                        |

src/ch/index.ts

@@ -76,6 +76,9 @@ export {
 	dynamicColumn,
 } from "./expr"
 
+// SQL templates inside expressions and conditions.
+export { sql, type SqlIdent, type SqlRaw, type SqlTag, type SqlTemplateValue } from "./sql-template"
+
 // Subquery conditions. These accept a `CHQuery` as well as raw SQL, so they
 // supersede the string-only `exists`/`inSubquery` still exported from `./expr`
 // for direct subpath importers.

src/ch/sql-template.test.ts

@@ -0,0 +1,139 @@
+import { describe, expect, it } from "@effect/vitest"
+import { Effect } from "effect"
+import * as CH from "./index"
+import * as PG from "../postgres"
+import * as T from "./types"
+import { QueryBuilderError } from "./errors"
+
+const Keys = CH.table("keys", { id: PG.uuid, org: PG.text, meta: PG.jsonb(), uses: PG.int8 }, { tenantColumn: "org" })
+const Events = CH.table("events", { OrgId: CH.string, Count: CH.uint64, Name: CH.string }, { tenantColumn: "OrgId" })
+
+describe("CH.sql", () => {
+	it("renders columns, params and plain values per dialect, parenthesized, with a typed row schema", () => {
+		const q = CH.from(Keys)
+			.select(($) => ({ txid: CH.sql(PG.text)`pg_current_xact_id()::xid::text`, next: CH.sql(PG.int8)`${$.uses} + ${1}` }))
+			.where(($) => [
+				$.org.eq(CH.param.string("org")),
+				CH.sql.cond`${$.meta} @> ${CH.param.string("filter")}::jsonb`,
+				CH.sql.cond`${$.id} <> ${"it's"}`,
+			])
+		const compiled = PG.compileUnsafe(q, { org: "o", filter: '{"a":1}' })
+		expect(compiled.sql).toContain('(pg_current_xact_id()::xid::text) AS "txid"')
+		expect(compiled.sql).toContain('("keys"."uses" + 1) AS "next"')
+		expect(compiled.sql).toContain(`("keys"."meta" @> $2::jsonb)`)
+		expect(compiled.sql).toContain(`("keys"."id" <> 'it''s')`)
+		expect(compiled.parameters).toEqual(["o", '{"a":1}'])
+		expect(compiled.rowSchemaSource).toBe("derived")
+		expect(compiled.tenantScope).toBe("single-tenant")
+
+		const ch = CH.compileUnsafe(
+			CH.from(Events).select(($) => ({ n: CH.sql(T.uint64)`${$.Count} * ${2}` })).where(($) => [CH.sql.cond`${$.Name} = ${"it's"}`]),
+		)
+		expect(ch.sql).toContain("(events.Count * 2) AS n")
+		expect(ch.sql).toContain("WHERE (events.Name = 'it\\'s')")
+	})
+
+	it("a template OR cannot swallow the conditions it is AND-joined with", () => {
+		const compiled = PG.compileUnsafe(
+			CH.from(Keys)
+				.select("id")
+				.where(($) => [CH.sql.cond`${$.uses} = 1 OR ${$.uses} = 2`, $.org.eq(CH.param.string("org"))]),
+			{ org: "o" },
+		)
+		expect(compiled.sql).toMatch(/WHERE \("keys"\."uses" = 1 OR "keys"\."uses" = 2\)\s+AND "keys"\."org" = \$1/)
+		const anded = CH.compileUnsafe(
+			CH.from(Events).select("Name").where(($) => [CH.and(CH.sql.cond`${$.Count} = 1 OR ${$.Count} = 2`, $.OrgId.eq("o"))]),
+		)
+		expect(anded.sql).toContain("((events.Count = 1 OR events.Count = 2) AND events.OrgId = 'o')")
+	})
+
+	it("an untyped template costs the row schema and names the alias", () => {
+		const compiled = PG.compileUnsafe(CH.from(Keys).select(() => ({ now: CH.sql`now()` })))
+		expect(compiled.rowSchemaSource).toBe("none")
+		expect(compiled.untypedColumns).toEqual(["now"])
+	})
+
+	it("raw, ident and join, nested templates, and a subquery compiled with the outer query", () => {
+		const Other = CH.table("other", { org: PG.text, id: PG.uuid }, { tenantColumn: "org" })
+		const compiled = PG.compileUnsafe(
+			CH.from(Keys)
+				.select("id")
+				.where(($) => [
+					CH.sql.cond`${CH.sql.ident("keys.org")} IN (${CH.sql.join(["a", "b", CH.param.string("c")])})`,
+					CH.sql.cond`${$.id} IN ${CH.from(Other).select("id").where(($o) => [$o.org.eq(CH.param.string("c"))])}`,
+					CH.sql.cond`${CH.sql`length(${$.org})`} > ${CH.sql.raw("2")}`,
+				]),
+			{ c: "z" },
+		)
+		expect(compiled.sql).toContain(`("keys"."org" IN ('a', 'b', $1))`)
+		expect(compiled.sql).toMatch(/\("keys"\."id" IN \(SELECT[\s\S]*"other"\."org" = \$1\)\)/)
+		expect(compiled.sql).toContain(`((length("keys"."org")) > 2)`)
+		expect(compiled.parameters).toEqual(["z"])
+		expect(compiled.tenantScope).toBe("cross-tenant")
+	})
+
+	it("a subquery in a template counts toward tenant scope", () => {
+		const Other = CH.table("other", { org: PG.text, id: PG.uuid }, { tenantColumn: "org" })
+		const scope = (inner: CH.CHQuery<any, any, any, any>) =>
+			PG.compileUnsafe(
+				CH.from(Keys).select("id").where(($) => [$.org.eq(CH.param.string("org")), CH.sql.cond`${$.id} IN ${inner}`]),
+				{ org: "o" },
+			).tenantScope
+		expect(scope(CH.from(Other).select("id"))).toBe("cross-tenant")
+		expect(scope(CH.from(Other).select("id").where(($) => [$.org.eq(CH.param.string("org"))]))).toBe("single-tenant")
+	})
+
+	it("a negative number never follows a `-` as a comment", () => {
+		const n = -1
+		const pg = PG.compileUnsafe(CH.from(Keys).select("id").where(($) => [CH.sql.cond`${$.uses} > 10-${n}`]))
+		expect(pg.sql).toContain(`("keys"."uses" > 10-(-1))`)
+		expect(PG.compileUnsafe(CH.from(Keys).select("id").where(() => [CH.sql.cond`x > 10-${-5n}`])).sql).toContain("10-(-5)")
+		// ClickHouse inlines params, so an inlined negative param is parenthesized too.
+		const ch = CH.compileUnsafe(
+			CH.from(Events).select("Name").where(($) => [CH.sql.cond`${$.Count} > 10-${CH.param.int("n")}`]),
+			{ n: -1 },
+		)
+		expect(ch.sql).toContain("(events.Count > 10-(-1))")
+		expect(ch.sql).not.toContain("--")
+	})
+
+	it.effect("objects parsed from JSON cannot pass for raw SQL, an identifier, or a date", () =>
+		Effect.gen(function* () {
+			const body = JSON.parse(
+				'{"raw":{"_tag":"@maple-dev/effect-orm/SqlRaw","sql":"1 OR 1=1"},"ident":{"_tag":"@maple-dev/effect-orm/SqlIdent","name":"password"},"utc":{"_tag":"Utc"}}',
+			)
+			for (const forged of [body.raw, body.ident, body.utc]) {
+				const error = yield* Effect.flip(PG.compile(CH.from(Keys).select("id").where(($) => [CH.sql.cond`${$.id} = ${forged}`])))
+				expect(error).toBeInstanceOf(QueryBuilderError)
+				expect(error.code).toBe("InvalidLiteral")
+			}
+		}),
+	)
+
+	it.effect("a value with no literal form, a bad ident, a union, an empty join, or a missing param fails the compile", () =>
+		Effect.gen(function* () {
+			const fails = (cond: CH.Condition, params: Record<string, unknown> = {}) =>
+				Effect.flip(PG.compile(CH.from(Keys).select("id").where(() => [cond]), params))
+			const array = yield* fails(CH.sql.cond`x = ANY(${["a"] as any})`)
+			expect(array).toBeInstanceOf(QueryBuilderError)
+			expect(array.message).toContain("typed param")
+			expect((yield* fails(CH.sql.cond`${CH.sql.ident("a; drop")} = 1`)).message).toContain("not a plain identifier")
+			expect((yield* fails(CH.sql.cond`x = ${CH.param.string("missing")}`)).code).toBe("UnresolvedParam")
+			expect((yield* fails(CH.sql.cond`x IN (${CH.sql.join([])})`)).message).toContain("no values to join")
+			expect((yield* fails(CH.sql.cond`x = ${new Date(Number.NaN)}`)).message).toContain("invalid Date")
+			const one = CH.from(Keys).select("id")
+			expect((yield* fails(CH.sql.cond`x IN ${CH.unionAll(one, one) as any}`)).message).toContain("fromUnion")
+		}),
+	)
+
+	it("works in a write's SET and WHERE", () => {
+		const compiled = PG.compileUnsafe(
+			CH.update(Keys)
+				.set(($) => ({ meta: CH.sql(PG.jsonb())`${$.meta} || ${CH.param.string("patch")}::jsonb` }))
+				.where(($) => [CH.sql.cond`${$.id} = ${CH.param.string("id")}::uuid`]),
+			{ patch: "{}", id: "k" },
+		)
+		expect(compiled.sql).toBe('UPDATE "keys" SET "meta" = ("meta" || $1::jsonb)\nWHERE ("id" = $2::uuid)')
+		expect(compiled.parameters).toEqual(["{}", "k"])
+	})
+})