Repository navigation
feat(review): incremental and snapshot semantics that fit any dbt project #1725
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: feat/review-contracts-dialect
Are you sure you want to change the base?
Changes from all commits
5963e30
fb9e10e
3de5a64
2d8ca8b
7a61dd8
9625cfd
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -277,6 +277,7 @@ riskTierPathTokens: # OFF by default — see "Risk-tier path tokens" b | |||||||||
| pii: [ssn, email, phone] # or supply your own case-insensitive tokens | ||||||||||
| allowForceTier: true # set false in gate CI: --force-tier is then rejected | ||||||||||
| checkContractColumns: false # opt-in: new columns missing from an enforced contract (CI that doesn't run `dbt build`) | ||||||||||
| incrementalSliceVars: [] # opt-in: vars that pick one slice per incremental run (e.g. [tenant]) | ||||||||||
| consumerColumnLists: [] # OFF by default — see "Consumers outside dbt" below | ||||||||||
| rubric: | ||||||||||
| blockOn: [lineage_breakage, contract_violation, pii_exposure, semantic_change] | ||||||||||
|
|
@@ -499,6 +500,31 @@ The deterministic rules group into four families: | |||||||||
| - **dbt unit tests.** A unit test whose `given` inputs are all `rows: []` is | ||||||||||
| reported: it either never tests the model's logic, or expects rows the model has nothing to | ||||||||||
| build from. | ||||||||||
| - **Incremental and snapshot semantics.** Config is read from the manifest | ||||||||||
| (project, yml and inline, as dbt merges them) with the head's inline | ||||||||||
| `{{ config() }}` on top. Formatting-only edits never count as a change. | ||||||||||
| - an existing incremental model gains a column, or its `is_incremental()` | ||||||||||
| filter changes, and the PR doesn't say how existing rows are handled | ||||||||||
| (`--full-refresh`, a backfill, or forward-only) — a suggestion; | ||||||||||
| - `partition_by` changes on an incremental model (needs a full refresh); | ||||||||||
| - `on_schema_change='fail'` and the change adds or removes columns, so the | ||||||||||
| first incremental run after merge fails; | ||||||||||
| - a snapshot's `check_cols` or `strategy` changes (`.sql` or dbt 1.9+ yml | ||||||||||
| snapshots), which can version every row on the next run; | ||||||||||
| - `row_number()` dedup ordered only by a load timestamp (`loaded_at`, | ||||||||||
| `ingestion_timestamp`) — a suggestion; loader sync columns such as | ||||||||||
| `_fivetran_synced` are not flagged. | ||||||||||
| - **Opt-in: per-slice incremental runs.** For projects that run one slice per | ||||||||||
| invocation (a tenant or region chosen by a var), list those vars: | ||||||||||
| `incrementalSliceVars: [tenant]`. A `delete+insert` or `merge` model filtered | ||||||||||
| by one of them must include the filtered column in `unique_key`, or one | ||||||||||
| slice's run deletes or overwrites another's rows. | ||||||||||
|
Comment on lines
+520
to
+521
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: The detector recognizes a listed Prompt for AI agents
Suggested change
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Follow-up: nit, tracked. The docs will say slice-var detection covers |
||||||||||
| - **Adapter-dependent advice** fires only where the adapter table establishes | ||||||||||
| the behavior. On Trino, `on_table_exists='replace'` with an enforced contract | ||||||||||
| runs `create or replace` then `insert`, and `drop` removes the table during | ||||||||||
| the build, so readers outside dbt can see an empty or missing table (a | ||||||||||
| suggestion). `unique_key` and strategy changes are covered by the engine's | ||||||||||
| `DBT011`/`DBT012`. | ||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: Prompt for AI agents
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not changing. These are altimate-core |
||||||||||
| - **dbt config / Jinja (`DBT0xx`).** A core **minijinja** parser reads | ||||||||||
| `{{ config(...) }}` and the model body: incremental model without an | ||||||||||
| `is_incremental()` guard, `merge`/`delete+insert` without a `unique_key`, | ||||||||||
|
|
||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -11,6 +11,13 @@ | |
| export interface AdapterCapability { | ||
| /** `int / int` returns an integer, dropping the fraction (7 / 2 = 3). */ | ||
| integerDivisionTruncates: boolean | ||
| /** | ||
| * A `table` model with an enforced contract and `on_table_exists='replace'` | ||
| * is replaced atomically. False = `create or replace` then `insert` as two | ||
| * statements (readers see an empty table), and `on_table_exists='drop'` | ||
| * leaves no table during the build. Undefined = not established. | ||
| */ | ||
| contractTableReplaceAtomic?: boolean | ||
| /** Documentation backing the entries above. */ | ||
| sources: string[] | ||
| } | ||
|
|
@@ -26,7 +33,14 @@ export const ADAPTER_CAPABILITIES: Record<string, AdapterCapability> = { | |
| }, | ||
| trino: { | ||
| integerDivisionTruncates: true, | ||
| sources: ["https://trino.io/docs/current/functions/math.html"], | ||
| contractTableReplaceAtomic: false, | ||
| sources: [ | ||
| "https://trino.io/docs/current/functions/math.html", | ||
| // create_table_as with an enforced contract: `create or replace table` then `insert`; | ||
| // on_table_exists='drop' drops the table before building it. | ||
| "https://github.com/starburstdata/dbt-trino/blob/master/dbt/include/trino/macros/adapters.sql", | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P3: These source links point at the Prompt for AI agents
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Follow-up: nit, tracked. We'll pin the dbt-trino links. |
||
| "https://github.com/starburstdata/dbt-trino/blob/master/dbt/include/trino/macros/materializations/table.sql", | ||
| ], | ||
| }, | ||
| athena: { | ||
| // Athena SQL runs on Trino. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -50,6 +50,12 @@ export const ReviewConfig = z.object({ | |
| consumerColumnLists: z | ||
| .array(z.object({ files: z.string(), pattern: z.string(), model: z.string().optional() })) | ||
| .default([]), | ||
| /** | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. SUGGESTION: Inserted JSDoc now documents the wrong field This block landed between the existing
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Follow-up: nit, tracked. The JSDoc will move back onto |
||
| * Opt-in: vars that select the slice one incremental run processes (e.g. | ||
| * `tenant`, `region`). A `delete+insert`/`merge` model filtered by one of | ||
| * them must include the filtered column in `unique_key`. | ||
| */ | ||
| incrementalSliceVars: z.array(z.string()).default([]), | ||
|
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
|
||
| /** Allow `--force-tier` (a debug bypass of the risk classifier). Set false | ||
| * in gate CI so a run cannot skip checks by forcing a lower tier. */ | ||
| allowForceTier: z.boolean().default(true), | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -249,6 +249,41 @@ const detectDedupTie: Detector = (c) => { | |
| }) | ||
| } | ||
|
|
||
| // 9b. Dedup ordered ONLY by a load/ingestion timestamp. Rows loaded in one batch | ||
| // share that value, so which version survives is arbitrary (SCD2 versions ranked | ||
| // by `ingestion_timestamp` alone). A business column (`updated_at`) stays | ||
| // silent, and so do loader sync columns (`_fivetran_synced`, `_airbyte_*`): | ||
| // ordering by them is the standard staging idiom and ties are rare per key. | ||
| const LOAD_TS_RE = | ||
| /^(_?(ingest(ion|ed)?|load(ed)?|etl|extract(ed)?|batch)(_at|_ts|_time|_timestamp|_date|_dt)|_?(ingestion|load|etl|batch)_timestamp|_loaded_at)$/i | ||
| const detectDedupLoadTimestamp: Detector = (c) => { | ||
| for (const line of c.added) { | ||
| const s = stripComments(line) | ||
| const m = /row_number\s*\(\s*\)\s*over\s*\(([^)]*)\)/i.exec(s) | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: This detector only matches a Prompt for AI agents
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed later in #1726 ( There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: Prompt for AI agents
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Follow-up: low priority, tracked. Dedup detection will require |
||
| if (!m) continue | ||
| const order = /order\s+by\s+(.+)$/i.exec(m[1])?.[1] | ||
| if (!order) continue | ||
| const keys = order.split(",").map((k) => k.trim().replace(/\s+(asc|desc)\b.*$/i, "").replace(/\s+nulls\s+(first|last)$/i, "")) | ||
| if (keys.length !== 1) continue | ||
| const col = keys[0].split(".").pop()!.replace(/["`]/g, "") | ||
| if (!LOAD_TS_RE.test(col)) continue | ||
| return makeFinding({ | ||
| severity: "suggestion", | ||
| category: "dedup", | ||
| title: `${c.model}: dedup ordered only by load timestamp \`${col}\``, | ||
| body: | ||
| `\`row_number()\` is ordered only by \`${col}\`. Rows loaded in the same batch share it, so which version survives is ` + | ||
| "arbitrary and can change between runs. Order by the source's own change time first and add a unique tie-breaker.", | ||
| file: c.file.path, | ||
| model: c.model, | ||
| confidence: "high", | ||
| evidence: { tool: "dbt-patterns", result: { rule: "dedup_load_timestamp_only", line: line.trim() } }, | ||
| ruleKey: "dedup:load-timestamp-only", | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: This load-timestamp dedup finding overlaps with core lint L039, which already flags timestamp-only ROW_NUMBER dedup. Add a Prompt for AI agents
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not changing: the bundled core 0.7.0 has no L039 lint code, so we couldn't confirm an overlap. |
||
| }) | ||
| } | ||
| return null | ||
| } | ||
|
|
||
| // 11. Function/cast wrapped around a column in an added WHERE — defeats partition pruning. | ||
| const detectPartitionFunction: Detector = (c) => { | ||
| const hit = c.added.find((l) => { | ||
|
|
@@ -735,6 +770,7 @@ const MODEL_DETECTORS: Detector[] = [ | |
| detectFanout, | ||
| detectNotIn, | ||
| detectDedupTie, | ||
| detectDedupLoadTimestamp, | ||
| detectPartitionFunction, | ||
| detectCountDistinct, | ||
| detectPiiIntoMart, | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.