Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions docs/docs/usage/dbt-pr-review.md
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,7 @@ riskTierPathTokens: # OFF by default — see "Risk-tier path tokens" b
pii: [ssn, email, phone] # or supply your own case-insensitive tokens
allowForceTier: true # set false in gate CI: --force-tier is then rejected
checkContractColumns: false # opt-in: new columns missing from an enforced contract (CI that doesn't run `dbt build`)
incrementalSliceVars: [] # opt-in: vars that pick one slice per incremental run (e.g. [tenant])
consumerColumnLists: [] # OFF by default — see "Consumers outside dbt" below
rubric:
blockOn: [lineage_breakage, contract_violation, pii_exposure, semantic_change]
Expand Down Expand Up @@ -499,6 +500,31 @@ The deterministic rules group into four families:
- **dbt unit tests.** A unit test whose `given` inputs are all `rows: []` is
reported: it either never tests the model's logic, or expects rows the model has nothing to
build from.
- **Incremental and snapshot semantics.** Config is read from the manifest
(project, yml and inline, as dbt merges them) with the head's inline
`{{ config() }}` on top. Formatting-only edits never count as a change.
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
- an existing incremental model gains a column, or its `is_incremental()`
filter changes, and the PR doesn't say how existing rows are handled
(`--full-refresh`, a backfill, or forward-only) — a suggestion;
- `partition_by` changes on an incremental model (needs a full refresh);
- `on_schema_change='fail'` and the change adds or removes columns, so the
first incremental run after merge fails;
- a snapshot's `check_cols` or `strategy` changes (`.sql` or dbt 1.9+ yml
snapshots), which can version every row on the next run;
- `row_number()` dedup ordered only by a load timestamp (`loaded_at`,
`ingestion_timestamp`) — a suggestion; loader sync columns such as
`_fivetran_synced` are not flagged.
- **Opt-in: per-slice incremental runs.** For projects that run one slice per
invocation (a tenant or region chosen by a var), list those vars:
`incrementalSliceVars: [tenant]`. A `delete+insert` or `merge` model filtered
by one of them must include the filtered column in `unique_key`, or one
slice's run deletes or overwrites another's rows.
Comment on lines +520 to +521

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The detector recognizes a listed var() only on the right side of = or IN; range and reversed comparisons are missed. Limit the documentation to supported filters or extend detection.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/docs/usage/dbt-pr-review.md, line 507:

<comment>The detector recognizes a listed `var()` only on the right side of `=` or `IN`; range and reversed comparisons are missed. Limit the documentation to supported filters or extend detection.</comment>

<file context>
@@ -486,6 +487,31 @@ The deterministic rules group into four families:
+- **Opt-in: per-slice incremental runs.** For projects that run one slice per
+  invocation (a tenant or region chosen by a var), list those vars:
+  `incrementalSliceVars: [tenant]`. A `delete+insert` or `merge` model filtered
+  by one of them must include the filtered column in `unique_key`, or one
+  slice's run deletes or overwrites another's rows.
+- **Adapter-dependent advice** fires only where the adapter table establishes
</file context>
Suggested change
by one of them must include the filtered column in `unique_key`, or one
slice's run deletes or overwrites another's rows.
by one of them in an equality or `IN` filter must include the filtered column in `unique_key`, or one
slice's run deletes or overwrites another's rows.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: nit, tracked. The docs will say slice-var detection covers = and IN.

- **Adapter-dependent advice** fires only where the adapter table establishes
the behavior. On Trino, `on_table_exists='replace'` with an enforced contract
runs `create or replace` then `insert`, and `drop` removes the table during
the build, so readers outside dbt can see an empty or missing table (a
suggestion). `unique_key` and strategy changes are covered by the engine's
`DBT011`/`DBT012`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: DBT011/DBT012 don't appear anywhere in the repo, so this sentence points readers at rule IDs they can't find. Name the rule that actually covers unique_key and strategy changes, or remove the reference.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/docs/usage/dbt-pr-review.md, line 514:

<comment>`DBT011`/`DBT012` don't appear anywhere in the repo, so this sentence points readers at rule IDs they can't find. Name the rule that actually covers `unique_key` and strategy changes, or remove the reference.</comment>

<file context>
@@ -486,6 +487,31 @@ The deterministic rules group into four families:
+  runs `create or replace` then `insert`, and `drop` removes the table during
+  the build, so readers outside dbt can see an empty or missing table (a
+  suggestion). `unique_key` and strategy changes are covered by the engine's
+  `DBT011`/`DBT012`.
 - **dbt config / Jinja (`DBT0xx`).** A core **minijinja** parser reads
   `{{ config(...) }}` and the model body: incremental model without an
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing. These are altimate-core dbt_config_diff rule ids, and the bundled core 0.7.0 ships DBT011 unique_key_change and DBT012 incremental_strategy_change.

- **dbt config / Jinja (`DBT0xx`).** A core **minijinja** parser reads
`{{ config(...) }}` and the model body: incremental model without an
`is_incremental()` guard, `merge`/`delete+insert` without a `unique_key`,
Expand Down
16 changes: 15 additions & 1 deletion packages/opencode/src/altimate/review/adapter-capabilities.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,13 @@
export interface AdapterCapability {
/** `int / int` returns an integer, dropping the fraction (7 / 2 = 3). */
integerDivisionTruncates: boolean
/**
* A `table` model with an enforced contract and `on_table_exists='replace'`
* is replaced atomically. False = `create or replace` then `insert` as two
* statements (readers see an empty table), and `on_table_exists='drop'`
* leaves no table during the build. Undefined = not established.
*/
contractTableReplaceAtomic?: boolean
/** Documentation backing the entries above. */
sources: string[]
}
Expand All @@ -26,7 +33,14 @@ export const ADAPTER_CAPABILITIES: Record<string, AdapterCapability> = {
},
trino: {
integerDivisionTruncates: true,
sources: ["https://trino.io/docs/current/functions/math.html"],
contractTableReplaceAtomic: false,
sources: [
"https://trino.io/docs/current/functions/math.html",
// create_table_as with an enforced contract: `create or replace table` then `insert`;
// on_table_exists='drop' drops the table before building it.
"https://github.com/starburstdata/dbt-trino/blob/master/dbt/include/trino/macros/adapters.sql",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: These source links point at the master branch of dbt-trino, which moves over time. Pin them to a release tag or commit SHA so the cited behavior stays verifiable.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/adapter-capabilities.ts, line 41:

<comment>These source links point at the `master` branch of dbt-trino, which moves over time. Pin them to a release tag or commit SHA so the cited behavior stays verifiable.</comment>

<file context>
@@ -26,7 +33,14 @@ export const ADAPTER_CAPABILITIES: Record<string, AdapterCapability> = {
+      "https://trino.io/docs/current/functions/math.html",
+      // create_table_as with an enforced contract: `create or replace table` then `insert`;
+      // on_table_exists='drop' drops the table before building it.
+      "https://github.com/starburstdata/dbt-trino/blob/master/dbt/include/trino/macros/adapters.sql",
+      "https://github.com/starburstdata/dbt-trino/blob/master/dbt/include/trino/macros/materializations/table.sql",
+    ],
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: nit, tracked. We'll pin the dbt-trino links.

"https://github.com/starburstdata/dbt-trino/blob/master/dbt/include/trino/macros/materializations/table.sql",
],
},
athena: {
// Athena SQL runs on Trino.
Expand Down
6 changes: 6 additions & 0 deletions packages/opencode/src/altimate/review/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,12 @@ export const ReviewConfig = z.object({
consumerColumnLists: z
.array(z.object({ files: z.string(), pattern: z.string(), model: z.string().optional() }))
.default([]),
/**

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Inserted JSDoc now documents the wrong field

This block landed between the existing consumerColumnLists doc comment (lines 44-49) and the field it documents, so consumerColumnLists is left undocumented and its "Opt-in: explicit column lists outside dbt…" text now reads as if it describes incrementalSliceVars. Move incrementalSliceVars above that comment, or move the consumerColumnLists comment back beside its field.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: nit, tracked. The JSDoc will move back onto incrementalSliceVars.

* Opt-in: vars that select the slice one incremental run processes (e.g.
* `tenant`, `region`). A `delete+insert`/`merge` model filtered by one of
* them must include the filtered column in `unique_key`.
*/
incrementalSliceVars: z.array(z.string()).default([]),
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
/** Allow `--force-tier` (a debug bypass of the risk classifier). Set false
* in gate CI so a run cannot skip checks by forcing a lower tier. */
allowForceTier: z.boolean().default(true),
Expand Down
36 changes: 36 additions & 0 deletions packages/opencode/src/altimate/review/dbt-patterns.ts
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,41 @@ const detectDedupTie: Detector = (c) => {
})
}

// 9b. Dedup ordered ONLY by a load/ingestion timestamp. Rows loaded in one batch
// share that value, so which version survives is arbitrary (SCD2 versions ranked
// by `ingestion_timestamp` alone). A business column (`updated_at`) stays
// silent, and so do loader sync columns (`_fivetran_synced`, `_airbyte_*`):
// ordering by them is the standard staging idiom and ties are rare per key.
const LOAD_TS_RE =
/^(_?(ingest(ion|ed)?|load(ed)?|etl|extract(ed)?|batch)(_at|_ts|_time|_timestamp|_date|_dt)|_?(ingestion|load|etl|batch)_timestamp|_loaded_at)$/i
const detectDedupLoadTimestamp: Detector = (c) => {
for (const line of c.added) {
const s = stripComments(line)
const m = /row_number\s*\(\s*\)\s*over\s*\(([^)]*)\)/i.exec(s)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This detector only matches a row_number() over (...) whose window fits on one added line, so it misses the usual multi-line dbt dedup layout and any window whose PARTITION BY calls a function. Match against the joined added block instead, and use a regex that allows one level of nested parentheses.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/dbt-patterns.ts, line 262:

<comment>This detector only matches a `row_number() over (...)` whose window fits on one added line, so it misses the usual multi-line dbt dedup layout and any window whose PARTITION BY calls a function. Match against the joined added block instead, and use a regex that allows one level of nested parentheses.</comment>

<file context>
@@ -249,6 +249,41 @@ const detectDedupTie: Detector = (c) => {
+const detectDedupLoadTimestamp: Detector = (c) => {
+  for (const line of c.added) {
+    const s = stripComments(line)
+    const m = /row_number\s*\(\s*\)\s*over\s*\(([^)]*)\)/i.exec(s)
+    if (!m) continue
+    const order = /order\s+by\s+(.+)$/i.exec(m[1])?.[1]
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed later in #1726 (dedupWindows reads the whole file).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: ROW_NUMBER can merely number rows; this emits a dedup finding even when rn is only selected and no rows are filtered, as in the new test. Require evidence that the rank is used to retain one row before reporting that a version survives.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/dbt-patterns.ts, line 262:

<comment>`ROW_NUMBER` can merely number rows; this emits a dedup finding even when `rn` is only selected and no rows are filtered, as in the new test. Require evidence that the rank is used to retain one row before reporting that a version survives.</comment>

<file context>
@@ -249,6 +249,41 @@ const detectDedupTie: Detector = (c) => {
+const detectDedupLoadTimestamp: Detector = (c) => {
+  for (const line of c.added) {
+    const s = stripComments(line)
+    const m = /row_number\s*\(\s*\)\s*over\s*\(([^)]*)\)/i.exec(s)
+    if (!m) continue
+    const order = /order\s+by\s+(.+)$/i.exec(m[1])?.[1]
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. Dedup detection will require rn = 1 or qualify.

if (!m) continue
const order = /order\s+by\s+(.+)$/i.exec(m[1])?.[1]
if (!order) continue
const keys = order.split(",").map((k) => k.trim().replace(/\s+(asc|desc)\b.*$/i, "").replace(/\s+nulls\s+(first|last)$/i, ""))
if (keys.length !== 1) continue
const col = keys[0].split(".").pop()!.replace(/["`]/g, "")
if (!LOAD_TS_RE.test(col)) continue
return makeFinding({
severity: "suggestion",
category: "dedup",
title: `${c.model}: dedup ordered only by load timestamp \`${col}\``,
body:
`\`row_number()\` is ordered only by \`${col}\`. Rows loaded in the same batch share it, so which version survives is ` +
"arbitrary and can change between runs. Order by the source's own change time first and add a unique tie-breaker.",
file: c.file.path,
model: c.model,
confidence: "high",
evidence: { tool: "dbt-patterns", result: { rule: "dedup_load_timestamp_only", line: line.trim() } },
ruleKey: "dedup:load-timestamp-only",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This load-timestamp dedup finding overlaps with core lint L039, which already flags timestamp-only ROW_NUMBER dedup. Add a dedup:load-timestamp-only entry to CORE_AST_COVERED in orchestrate.ts, or drop the QUALIFY form here, so one issue is not posted twice.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/dbt-patterns.ts, line 281:

<comment>This load-timestamp dedup finding overlaps with core lint L039, which already flags timestamp-only ROW_NUMBER dedup. Add a `dedup:load-timestamp-only` entry to `CORE_AST_COVERED` in orchestrate.ts, or drop the QUALIFY form here, so one issue is not posted twice.</comment>

<file context>
@@ -249,6 +249,41 @@ const detectDedupTie: Detector = (c) => {
+      model: c.model,
+      confidence: "high",
+      evidence: { tool: "dbt-patterns", result: { rule: "dedup_load_timestamp_only", line: line.trim() } },
+      ruleKey: "dedup:load-timestamp-only",
+    })
+  }
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing: the bundled core 0.7.0 has no L039 lint code, so we couldn't confirm an overlap.

})
}
return null
}

// 11. Function/cast wrapped around a column in an added WHERE — defeats partition pruning.
const detectPartitionFunction: Detector = (c) => {
const hit = c.added.find((l) => {
Expand Down Expand Up @@ -735,6 +770,7 @@ const MODEL_DETECTORS: Detector[] = [
detectFanout,
detectNotIn,
detectDedupTie,
detectDedupLoadTimestamp,
detectPartitionFunction,
detectCountDistinct,
detectPiiIntoMart,
Expand Down
Loading
Loading