Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,13 @@

## Unreleased

- Add `isNull()`, `isNotNull()`, `between()` and `notBetween()` on every expression, and
variadic `CH.and(...)` / `CH.or(...)` that skip `undefined` and write one flat group.
- Add `distinct()` and `distinctOn(...aliases)` to queries, on both dialects.
- Add Postgres row locks: `forUpdate`, `forNoKeyUpdate`, `forShare`, `forKeyShare`, with
`skipLocked`, `noWait` and `of` (`LockOptions`). Add `DialectClauses.locking`; ClickHouse
refuses them. `SqlQuery` gains `distinct`, `distinctOn` and `lock`.
- `compile(query)` no longer needs a params argument when the query has no params.
- Add `update(table).set(...).where(...)` and `deleteFrom(table).where(...)` (see
`docs/updates-and-deletes.md`), with `returning` on Postgres and `settings` on ClickHouse,
where they compile to an `ALTER TABLE ... UPDATE` mutation and a lightweight `DELETE`. A write
Expand Down
6 changes: 3 additions & 3 deletions design/gap-review.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,10 @@ builder; **P1** commonly used; **P2** niche.
| ~~DELETE builder: WHERE, RETURNING~~ (built) | ~79 | S |
| A typed, value-binding `sql` template usable inside expressions; `sql.join` / `raw` / `empty` on `Db.sql` | ~163 | M |
| Postgres column types: `timestamptz` as `Date`, `timestamp`, `date`, `interval`, `varchar(n)`, serial / identity | 226 timestamp columns | S |
| DISTINCT (and DISTINCT ON) | ~10 | S |
| `FOR UPDATE` / `FOR SHARE` / `SKIP LOCKED` / `NOWAIT` | 7 | S |
| ~~DISTINCT (and DISTINCT ON)~~ (built) | ~10 | S |
| ~~`FOR UPDATE` / `FOR SHARE` / `SKIP LOCKED` / `NOWAIT`~~ (built) | 7 | S |
| jsonb and array operators (`@>`, `->`, `?`, `&&`, `ANY`) | ~12 | M |
| `isNull` / `isNotNull` / `between`; variadic `and` / `or` that skip `undefined` | everywhere | S |
| ~~`isNull` / `isNotNull` / `between`; variadic `and` / `or` that skip `undefined`~~ (built) | everywhere | S |
| Constraint error helpers (unique, foreign key, not null); keep ClickHouse's numeric error codes, which `sqlStateOf` drops today | all upserts | S |
| Tenant-scope enforcement in `Database`, opt in, with an explicit cross-tenant entry point | safety | S |
| Postgres `defineTable` (indexes, unique, FKs), Postgres migrations, a drizzle-kit importer | 68 tables, 90 indexes, 47 unique, 75 folders | L; can wait, drizzle-kit can keep migrating |
Expand Down
30 changes: 19 additions & 11 deletions docs/expressions.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,24 +19,18 @@ $.Timestamp.gte(new Date(...)) // Timestamp >= '2026-01-01 00:00:00'

### Testing for NULL

`.eq(null)` emits `= NULL`; it does not test whether a value is missing. Use
`isNull` (or `isNotNull` for present values), declared with `defineCondFn`:
`.eq(null)` emits `= NULL`; it does not test whether a value is missing. Use `.isNull()` (or
`.isNotNull()` for present values), which write `IS NULL` and work on every dialect:

```ts title="null-filter.ts"
import * as CH from "@maple-dev/effect-orm"
import * as T from "@maple-dev/effect-orm/types"

const Notes = CH.table("notes", { Note: T.nullable(T.string) })
const isNull = CH.defineCondFn<[CH.Expr<string | null>]>("isNull")
export const compiled = CH.compileUnsafe(
CH.from(Notes).select("Note").where(($) => [isNull($.Note)]),
{},
)
console.log(compiled.sql) // SELECT Note AS Note FROM notes WHERE isNull(Note)
export const compiled = CH.compileUnsafe(CH.from(Notes).select("Note").where(($) => [$.Note.isNull()]))
console.log(compiled.sql) // SELECT Note AS Note FROM notes WHERE Note IS NULL
```

See [ClickHouse NULL predicates](https://clickhouse.com/docs/reference/functions/regular-functions/functions-for-nulls#isNull).

### Invalid literals

A value the column cannot hold fails while the SQL is being built:
Expand All @@ -62,6 +56,8 @@ Every `Expr<T>` carries:
| `.gt(x)` / `.gte(x)` | `> x` / `>= x` |
| `.lt(x)` / `.lte(x)` | `< x` / `<= x` |
| `.in_(...xs)` / `.notIn(...xs)` | `IN (…)` / `NOT IN (…)` |
| `.between(a, b)` / `.notBetween(a, b)` | `BETWEEN a AND b` / `NOT BETWEEN a AND b` |
| `.isNull()` / `.isNotNull()` | `IS NULL` / `IS NOT NULL` |

Each accepts a raw value or another `Expr<T>`. String literals are escaped; booleans emit as
`1` / `0`.
Expand All @@ -86,8 +82,20 @@ Each accepts a raw value or another `Expr<T>`. String literals are escaped; bool
`.and()` / `.or()` parenthesise their result, so precedence is explicit. `CH.not(condition)` wraps
in `NOT (…)` and is available from the root and `/expr` subpath.

`CH.and(...)` and `CH.or(...)` take any number of conditions, skip `undefined` ones, and write
one flat group. With none left they return `undefined`, which `where` skips, so optional
filters combine without special cases:

```ts
.where(($) => [
$.OrgId.eq("org_123"),
CH.or(CH.when(name, (n) => $.Name.eq(n)), CH.when(minMs, (ms) => $.Ms.gte(ms))),
])
// both given -> … AND (Name = 'checkout' OR Ms >= 100); neither -> only the OrgId test
```

The `where` array is AND-joined. [Tenant scoping](./tenant-scoping.md) preserves evidence
through both separate entries and `.and()`; `.or()` discards it.
through both separate entries, `.and()` and `CH.and()`; `.or()` and `CH.or()` discard it.

_(Backed by `docs/expressions.md > Combining conditions with and/or`.)_

Expand Down
34 changes: 34 additions & 0 deletions docs/queries.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,19 @@ Untyped callers receive `QueryBuilderDefect`; use a tuple for each sort key.

_(Backed by `docs/queries.md > orderBy takes tuples` and `> orderBy rejects a bare string`.)_

## `distinct` / `distinctOn`

```ts
.select("ServiceName").distinct()
// SELECT DISTINCT ServiceName …

.select(($) => ({ org: $.OrgId, id: $.Id })).distinctOn("org").orderBy(["org", "asc"], ["id", "desc"])
// SELECT DISTINCT ON (org) … — the newest row per org
```

`distinctOn` takes selected aliases and keeps the first row of each group in ORDER BY order;
Postgres wants those keys to lead the ORDER BY. Both ClickHouse and Postgres support it.

## `limit` / `offset`

```ts
Expand All @@ -128,6 +141,27 @@ your request boundary, and enforce an application maximum. Use a stable `orderBy
Accepts `"JSON"` or `"JSONEachRow"`. Most clients set the format themselves; use this only
when you are sending raw SQL somewhere that does not.

## Row locks

On Postgres, `forUpdate`, `forNoKeyUpdate`, `forShare` and `forKeyShare` add a locking clause
after LIMIT. Each takes `{ skipLocked?, noWait?, of? }`. The usual job-queue claim:

```ts
CH.from(Jobs)
.select("id")
.where(($) => [$.state.eq("queued")])
.orderBy(["id", "asc"])
.limit(1)
.forUpdate({ skipLocked: true })
// … LIMIT 1 FOR UPDATE SKIP LOCKED
```

A lock lasts until the transaction ends, so run the query inside `Database.transaction`. These
are a `QueryBuilderDefect`, refused before anything is sent: `skipLocked` and `noWait` together;
a qualified name in `of` (use the alias or `jobs`, not `public.jobs`); a lock on a query with
DISTINCT, GROUP BY or HAVING, or on a `unionAll` branch, which Postgres refuses; and any lock on
ClickHouse, which has no row locks.

## `withCTE`

See [Unions and CTEs](./unions-and-ctes.md#ctes).
Expand Down
8 changes: 6 additions & 2 deletions docs/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,8 @@ Note `/sql` exports a `compile` (fragment → string) distinct from the root `co
| `orderBy(...[col, dir])` | **Tuples**, not two strings |
| `limit(n)` / `offset(n)` | Rounded before emission |
| `format(fmt)` | `"JSON"` \| `"JSONEachRow"` |
| `distinct()` / `distinctOn(...aliases)` | `SELECT DISTINCT` / `SELECT DISTINCT ON (…)` |
| `forUpdate` / `forNoKeyUpdate` / `forShare` / `forKeyShare` | Postgres row locks; options `LockOptions` (`skipLocked`, `noWait`, `of`) |
| `innerJoin` / `leftJoin` / `crossJoin` | `(table, alias, on?)` |
| `innerJoinQuery` / `leftJoinQuery` / `crossJoinQuery` | `(query, alias, on?)` |
| `withCTE(name, query)` / `withCTE(name, sql, options?)` | Typed query derives scope; SQL form can declare `options.tenantScope` |
Expand All @@ -89,7 +91,7 @@ Note `/sql` exports a `compile` (fragment → string) distinct from the root `co

| Export | Signature |
| -------------------- | ------------------------------------------------------------------------------------ |
| `compile` | `(query, params, options?) => Effect<CompiledQuery<Output>, QueryBuilderError>`; also `(insert, params?, options?)`, whose options (`InsertCompileOptions`) are only `dialect` |
| `compile` | `(query, params?, options?) => Effect<CompiledQuery<Output>, QueryBuilderError>`; also `(insert, params?, options?)`, whose options (`InsertCompileOptions`) are only `dialect` |
| `compileUnsafe` | The same, returning `CompiledQuery<Output>` and throwing instead |
| `compileUnion` | `(union, params, options?) => Effect<CompiledQuery<Output>, QueryBuilderError>` |
| `compileUnionUnsafe` | The same, throwing instead |
Expand Down Expand Up @@ -125,13 +127,15 @@ time; see [Params and compilation](./params-and-compilation.md#what-each-kind-ac
| `inExprList(expr, exprs)` | Same for expression lists |
| `notInList(expr, values)` | `expr NOT IN ('a', 'b')` |
| `not(condition)` | `NOT (…)` |
| `and(...conds)` / `or(...conds)` | One flat `(… AND …)` / `(… OR …)`; skips `undefined`, returns `undefined` when none are left |
| `dynamicColumn(name, t?)` | An `Expr` from a runtime column name — a `GROUP BY` alias |
| `exists(q)` | `EXISTS (…)` from a query or pre-compiled SQL |
| `inSubquery(expr, q)` | `expr IN (…)` from a query or pre-compiled SQL |
| `notInSubquery(expr, q)` | `expr NOT IN (…)`; note the NULL semantics |
| `outerRef<T>(name)` | Reference an outer column in a correlated subquery |

`Expr<T>` methods: `eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `in_`, `notIn`, `like`, `notLike`,
`Expr<T>` methods: `eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `in_`, `notIn`, `between`, `notBetween`,
`isNull`, `isNotNull`, `like`, `notLike`,
`ilike` (string-only), and `add`, `sub`, `mul`, `div`, `mod` (number-only, **no parentheses**). `div` and `mod` decode
as `number | null` — ClickHouse sends `inf`/`nan` as JSON `null` — except by a numeric literal of
magnitude ≥ 1 (`Quotient<L, R>`), which keeps the dividend's nullability; use
Expand Down
2 changes: 1 addition & 1 deletion scripts/check-doc-examples.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ assert.equal(typedFunction.compiled.rowSchemaSource, "derived")
assert.deepEqual(await Effect.runPromise(typedFunction.compiled.decodeRows([{ name: "checkout", durationMs: "42" }])), [{ name: "checkout", durationMs: 42 }])
await assert.rejects(() => Effect.runPromise(typedFunction.compiled.decodeRows([{ name: 42, durationMs: "oops" }])))
const nullFilter = await import("./null-filter")
assert.match(sql(nullFilter.compiled), /WHERE isNull\\(notes.Note\\)/)
assert.match(sql(nullFilter.compiled), /WHERE notes.Note IS NULL/)
assert.deepEqual(await Effect.runPromise(nullFilter.compiled.decodeRows([{ Note: null }])), [{ Note: null }])
const escapedSql = await import("./escaped-sql")
const fragments = await import("@maple-dev/effect-orm/sql")
Expand Down
61 changes: 57 additions & 4 deletions src/ch/compile.ts
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,31 @@ export class CompiledQueryEncodeError extends Schema.TaggedError<CompiledQueryEn
},
) {}

/** `FOR UPDATE SKIP LOCKED` and the like, refused where the dialect has no locking. */
const lockClause = (lock: import("./query").LockClause | undefined): string | undefined => {
if (lock === undefined) return undefined
const dialect = currentDialect()
if (dialect.clauses.locking !== true) {
throw new QueryBuilderDefect({
message: `CHQuery: FOR ${lock.strength} has no meaning for the ${dialect.name} dialect, which has no row locks`,
})
}
if (lock.skipLocked === true && lock.noWait === true) {
throw new QueryBuilderDefect({ message: "CHQuery: a lock takes skipLocked or noWait, not both" })
}
// Postgres takes only unqualified names here, so `public.jobs` is refused, not quoted.
for (const name of lock.of ?? []) {
if (name.includes(".")) {
throw new QueryBuilderDefect({
message: `CHQuery: FOR ${lock.strength} OF ${JSON.stringify(name)}: name the table by its alias or unqualified name`,
})
}
}
const of = lock.of !== undefined && lock.of.length > 0 ? ` OF ${lock.of.map(quoteIdent).join(", ")}` : ""
const wait = lock.skipLocked === true ? " SKIP LOCKED" : lock.noWait === true ? " NOWAIT" : ""
return `FOR ${lock.strength}${of}${wait}`
}

/** `orderBy` takes `[column, direction]` tuples. A bare string is the natural
* mistake (`.orderBy("count", "desc")`), and it is invisible without types:
* destructuring a string yields its first two characters, so `"count"` used to
Expand Down Expand Up @@ -531,11 +556,12 @@ export function compileCH<
Output extends Record<string, any>,
Joins extends Record<string, ColumnDefs>,
Route extends string | undefined,
Params extends Record<string, any>,
Params extends Record<string, any> = {},
Decoded extends Output = Output,
>(
query: CHQuery<Cols, Output, Joins, Route>,
params: Params,
/** Values for the query's `param.*` markers. Optional when it has none. */
params?: Params,
options?: {
skipFormat?: boolean
rowSchema?: CompiledQueryRowSchema<Decoded>
Expand Down Expand Up @@ -578,11 +604,12 @@ export function compileCHUnsafe<
Output extends Record<string, any>,
Joins extends Record<string, ColumnDefs>,
Route extends string | undefined,
Params extends Record<string, any>,
Params extends Record<string, any> = {},
Decoded extends Output = Output,
>(
query: CHQuery<Cols, Output, Joins, Route>,
params: Params,
/** Values for the query's `param.*` markers. Optional when it has none. */
params?: Params,
options?: {
skipFormat?: boolean
rowSchema?: CompiledQueryRowSchema<Decoded>
Expand Down Expand Up @@ -827,6 +854,29 @@ function compileInner<
})

const sqlQuery: SqlQuery = {
distinct: state.distinct !== undefined,
distinctOn: Array.isArray(state.distinct)
? (state.distinct.length === 0
? (() => {
throw new QueryBuilderDefect({ message: "CHQuery: distinctOn() needs at least one key" })
})()
: state.distinct
).map((key: string) => {
if (!(options?.selectKeys ?? keys).includes(key)) {
throw new QueryBuilderDefect({ message: `CHQuery: distinctOn(${JSON.stringify(key)}) is not a selected alias` })
}
return raw(quoteIdent(key))
})
: undefined,
lock: (() => {
// Postgres refuses a lock on rows that are no longer table rows; say so here.
if (state.lock !== undefined && (state.distinct !== undefined || state.groupByKeys.length > 0 || state.havingFn !== undefined)) {
throw new QueryBuilderDefect({
message: `CHQuery: FOR ${state.lock.strength} cannot lock rows of a query with DISTINCT, GROUP BY or HAVING`,
})
}
return lockClause(state.lock)
})(),
select: selectFragments,
from: fromFragment,
joins,
Expand Down Expand Up @@ -1137,6 +1187,9 @@ function compileUnionInner<Output extends Record<string, any>, Params extends Re
const first = state.queries[0]
if (first === undefined) throw new QueryBuilderDefect({ message: "unionAll requires at least one query" })
const selectKeys = Object.keys(selectExprsOf(first) ?? {})
if (state.queries.some((q) => q._state.lock !== undefined)) {
throw new QueryBuilderDefect({ message: "unionAll: a branch cannot take a row lock; lock in a query over the union instead" })
}
const subQueries = state.queries.map((q) =>
compileInner(q, params, { skipFormat: true, deferParams, nested: true, selectKeys, enclosingCtes }),
)
Expand Down
3 changes: 3 additions & 0 deletions src/ch/dialect.ts
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,9 @@ export interface DialectClauses {
/** `SETTINGS` on an INSERT, UPDATE or DELETE (ClickHouse). Absent means
* no: a write with `.settings()` fails to compile for the dialect. */
readonly writeSettings?: boolean
/** Row locking on a SELECT (`FOR UPDATE`, `FOR SHARE`, `SKIP LOCKED`). Absent
* means no: a query with `.forUpdate()` and the like fails to compile. */
readonly locking?: boolean
/** UPDATE is written `ALTER TABLE t UPDATE ... WHERE ...`, a ClickHouse
* mutation, rather than `UPDATE t SET ... WHERE ...`. */
readonly alterTableUpdate?: boolean
Expand Down
51 changes: 51 additions & 0 deletions src/ch/expr.ts
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,22 @@ export interface Expr<TSType> {
notLike(this: Expr<string>, pattern: string): Condition
ilike(this: Expr<string>, pattern: string): Condition

// NULL and ranges
/** `expr IS NULL`. */
isNull(): Condition
/** `expr IS NOT NULL`. */
isNotNull(): Condition
/** `expr BETWEEN low AND high`, both ends included. */
between(
low: Comparable<Widen<TSType>> | Expr<TSType> | Expr<Widen<TSType>>,
high: Comparable<Widen<TSType>> | Expr<TSType> | Expr<Widen<TSType>>,
): Condition
/** `expr NOT BETWEEN low AND high`. */
notBetween(
low: Comparable<Widen<TSType>> | Expr<TSType> | Expr<Widen<TSType>>,
high: Comparable<Widen<TSType>> | Expr<TSType> | Expr<Widen<TSType>>,
): Condition

// IN / NOT IN
in_(...values: Array<Comparable<Widen<TSType>>>): Condition
notIn(...values: Array<Comparable<Widen<TSType>>>): Condition
Expand Down Expand Up @@ -248,6 +264,13 @@ export function makeExpr<T>(
lt: (other) => makeCond(lazy(() => `${compile(fragment)} < ${compile(operand(other))}`)),
lte: (other) => makeCond(lazy(() => `${compile(fragment)} <= ${compile(operand(other))}`)),

isNull: () => makeCond(lazy(() => `${compile(fragment)} IS NULL`)),
isNotNull: () => makeCond(lazy(() => `${compile(fragment)} IS NOT NULL`)),
between: (low, high) =>
makeCond(lazy(() => `${compile(fragment)} BETWEEN ${compile(operand(low))} AND ${compile(operand(high))}`)),
notBetween: (low, high) =>
makeCond(lazy(() => `${compile(fragment)} NOT BETWEEN ${compile(operand(low))} AND ${compile(operand(high))}`)),

like: (pattern: string) => makeCond(lazy(() => `${compile(fragment)} LIKE ${compile(str(pattern))}`)),
notLike: (pattern: string) => makeCond(lazy(() => `${compile(fragment)} NOT LIKE ${compile(str(pattern))}`)),
ilike: (pattern: string) => makeCond(lazy(() => `${compile(fragment)} ILIKE ${compile(str(pattern))}`)),
Expand Down Expand Up @@ -443,6 +466,34 @@ export function notInList(expr: Expr<string>, values: readonly string[]): Condit
return makeCond(lazy(() => `${compile(expr.toFragment())} NOT IN (${escaped()})`))
}

/**
* Conditions AND-joined, an `undefined` one skipped: `and(a, when(x, f), b)`.
* With none left it is `undefined`, which a `where` list skips in turn. Tenant
* evidence carries through, as with `.and`.
*/
export function and(...conditions: ReadonlyArray<Condition>): Condition
export function and(...conditions: ReadonlyArray<Condition | undefined>): Condition | undefined
export function and(...conditions: ReadonlyArray<Condition | undefined>): Condition | undefined {
const present = conditions.filter((c): c is Condition => c !== undefined)
if (present.length <= 1) return present[0]
return markTenantPredicate(
makeCond(lazy(() => `(${present.map((c) => compile(c.toFragment())).join(" AND ")})`)),
present.flatMap((c) => tenantPredicatesOf(c)),
)
}

/**
* Conditions OR-joined, an `undefined` one skipped. With none left it is
* `undefined`. An OR proves no tenant, so it carries no tenant evidence.
*/
export function or(...conditions: ReadonlyArray<Condition>): Condition
export function or(...conditions: ReadonlyArray<Condition | undefined>): Condition | undefined
export function or(...conditions: ReadonlyArray<Condition | undefined>): Condition | undefined {
const present = conditions.filter((c): c is Condition => c !== undefined)
if (present.length <= 1) return present[0]
return makeCond(lazy(() => `(${present.map((c) => compile(c.toFragment())).join(" OR ")})`))
}

/** Wrap a condition in NOT (...). */
export function not(condition: Condition): Condition {
return makeCond(lazy(() => `NOT (${compile(condition.toFragment())})`))
Expand Down
Loading
Loading