Skip to content

feat(review): incremental and snapshot semantics that fit any dbt project - #1725

Open
RaghwendraSingh wants to merge 6 commits into
feat/review-contracts-dialectfrom
feat/review-incremental-semantics
Open

RaghwendraSingh wants to merge 6 commits into
feat/review-contracts-dialectfrom
feat/review-incremental-semantics

Conversation

@RaghwendraSingh

@RaghwendraSingh RaghwendraSingh commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Changes since review: Review feedback commit: Spark: pack gating, quote-aware comment stripping, effective-config (yml/project) changes not reported as removals, and lineage-less column sets treated as unknown. The earlier dialect-pack commit is reworded to match its contents.

Issue for this PR

Part of #1792
Stacked on #1724 (← #1723 ← … ← #1719); merge in that order. Draft: the new rules are not yet measured on a public-repo benchmark.

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

dbt incremental and snapshot semantics. Rules follow dbt-core behavior, which is the same on every adapter. The one adapter-dependent rule reads the capability table and stays silent where the behavior isn't established. Config comes from the manifest (project, yml and inline, as dbt merges them), with the head's inline {{ config() }} on top. Every finding is advisory and never blocks.

Rules

Rule Fires when Notes
History plan (suggestion) An existing incremental model gains a column (old rows get NULL) or its is_incremental() filter changes, and the PR doesn't say how existing rows are handled Tokens are compared, so formatting never counts
partition_by change Changed on an incremental model Needs a full refresh
on_schema_change='fail' Columns are added or removed The first incremental run after merge fails
Snapshot tracking change check_cols or strategy changes on an existing snapshot .sql and dbt 1.9+ yml snapshots
Load-timestamp dedup (suggestion) row_number() ordered only by loaded_at / ingestion_timestamp Loader sync columns (_fivetran_synced, _airbyte_*, _sdc_*) are not flagged
Per-slice unique_key (opt-in: incrementalSliceVars) A delete+insert/merge model filtered by a listed var (e.g. a tenant) lacks that column in unique_key Otherwise one slice's run deletes or overwrites another's rows
Trino non-atomic rebuild (suggestion) on_table_exists='replace' with an enforced contract, or drop replace + contract runs create or replace then insert; drop removes the table during the build. Checked against the dbt-trino 1.10.6 source

unique_key and strategy changes are left to the engine's DBT011/DBT012.

Also in this PR

  • Dialect-pack rules (BigQuery: …, Snowflake: …, Redshift: …, Databricks: …) fire only on their own engine once the project's engine is known. A DuckDB project was getting BigQuery partition advice.
  • Column-set rules now see literal and function-only columns (current_date as load_date). Column lineage only reports columns that read a source column. select-list.ts adds the final SELECT's own names: an item counts only when it's aliased after a complete expression or is a bare column, and * defers to lineage.

Changed after an adversarial generality review, before opening. The goal was rules that fit any dbt project, not one monorepo:

  • the history rule no longer fires on every incremental edit
  • the slice rule is opt-in
  • a Trino/Athena microbatch-concurrency rule was removed: dbt-trino's concurrency default is changing, and the table format can't be known from the adapter
  • loader sync columns are no longer treated as load timestamps

How did you verify your code works?

  • New tests: test/altimate/review-incremental-semantics.test.ts covers the generic rules on two adapters, formatting-only edits, the opt-ins, yml snapshots and orchestrator wiring. test/altimate/review-select-list.test.ts covers the select-list names.
  • Suites: 495 tests across the review files pass, bun typecheck is clean, and the real-world corpus is unchanged (15/15 caught, 0/5 false positives).
  • Local end-to-end run. A real dbt 1.10 DuckDB project, released 0.12.10 vs this stack. A snapshot check_cols change and adding a column under on_schema_change='fail' are reported; 0.12.10 reports neither. A model reading a tenant var with no incrementalSliceVars configured stays quiet.
  • Replay of 7 real PRs from a private dbt monorepo. 0 off-diff findings, no earlier catch lost. One new true positive: a contracted mart on Trino with project-level on_table_exists: replace.

Cost side effects

None. No model calls or warehouse queries.

Screenshots / recordings

N/A (CLI).

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

🤖 Generated with Claude Code


Summary by cubic

Adds dbt incremental and snapshot semantics to PR review as advisory findings that fit any dbt project, following dbt-core behavior that is the same on every adapter. Config is read from the manifest (project, yml, inline, as dbt merges them) with the head's inline {{ config() }} on top; an inline value moved to yml with the same value isn't a change, and formatting-only edits never count.

Rules fire when an incremental model gains a column or its is_incremental() filter changes without a stated plan for existing rows, partition_by changes, on_schema_change='fail' meets added or removed columns, a snapshot's check_cols/strategy or query changes, or dedup orders only by a load timestamp. Per-slice runs are opt-in via incrementalSliceVars, and each must include the slice column in unique_key. On Trino, on_table_exists='replace' with an enforced contract and drop get a suggestion because readers can see an empty or missing table. Adapter-dependent advice reads the capability table and stays silent where behavior isn't established, dialect-pack rules (including Spark:) fire only on their own engine, and column-set rules now see literal and function-only columns (current_date as load_date) by adding the final SELECT's own names — taken from a set operation's first branch — to column lineage; when lineage returns nothing for one side, the column set stays unknown rather than reporting every expression as removed. Also fixes a false "table not found" warning against a manifest-only schema (category → kind).

Written for commit 9625cfd. Summary will update on new commits.

View guided diff Turn on auto-fix

@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 8f94278c-37df-4c98-b984-0a336ad3a930

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown

👋 This PR was automatically closed by our quality checks.

Common reasons:

  • New GitHub account with limited contribution history
  • PR description doesn't meet our guidelines
  • Contribution appears to be AI-generated without meaningful review

If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you.

// History diverges from new rows when the incremental filter changes or a
// column is added (old rows get NULL). Formatting-only edits never count,
// and `on_schema_change='fail'` has its own finding below.
if (!isNew && input.oldSql && cfg.onSchemaChange !== "fail" && bodyTokens(input.oldSql) !== bodyTokens(input.newSql) &&

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: on_schema_change='fail' also hides is_incremental() filter changes

The cfg.onSchemaChange !== "fail" guard skips this whole block before filterChanged is computed, and the on_schema_change finding below only fires when columnDelta() reports added/removed columns. So a model with on_schema_change='fail' whose PR changes only the incremental filter (no column change) gets no finding from either rule — the exact history divergence this rule targets. Only defer when the deferral actually applies, e.g. if ((cfg.onSchemaChange !== "fail" || added.length) && (filterChanged || added.length)).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7a61dd8. Under on_schema_change='fail' only column changes go to the fail finding, and a change to the is_incremental() filter alone is reported as needing a history plan again. There is a test for it.

const prev = before.get(name)
if (!prev) continue
const changed = ["check_cols", "strategy"].filter(
(k) => prev[k] !== undefined && JSON.stringify(prev[k]) !== JSON.stringify(cfg[k]),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Semantics-preserving check_cols edits are reported as a change

JSON.stringify compares arrays positionally, so reordering check_cols ([status, amount] -> [amount, status]) reports a check_cols change even though the snapshot tracks the same set. Normalize before comparing (sort the array values) here and in the norm() comparison in snapshotFindings (line 297) so a reorder stays silent.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. We'll sort check_cols before comparing.

out.push(
finding({ file, model: name }, "snapshot-tracking-change", "warning", "materialization",
`snapshot ${changed.map((k) => `\`${k}\``).join(" and ")} changed`,
"Changing what an existing snapshot compares can version every row on the next run (a column new to the snapshot compares " +

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Duplicated snapshot-change body and key list

The check_cols/strategy key list and the whole "Changing what an existing snapshot compares…" body are copied verbatim from snapshotFindings (lines 297-303). Extract a single helper (e.g. snapshotTrackingFinding(file, model, before, after)) so the SQL and yml snapshot paths cannot drift in wording or thresholds.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: nit, tracked. We'll dedupe the snapshot key list and body.

* whose first capture group is the list, `model` the model it mirrors
* (omitted = files that mention the model by name).
*/
/**

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Inserted JSDoc now documents the wrong field

This block landed between the existing consumerColumnLists doc comment (lines 44-49) and the field it documents, so consumerColumnLists is left undocumented and its "Opt-in: explicit column lists outside dbt…" text now reads as if it describes incrementalSliceVars. Move incrementalSliceVars above that comment, or move the consumerColumnLists comment back beside its field.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: nit, tracked. The JSDoc will move back onto incrementalSliceVars.

import { capabilityFor } from "./adapter-capabilities"

/**
* dbt incremental, microbatch and snapshot semantics. These rules follow

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Module doc still mentions microbatch

The microbatch rule and its capability entry were removed in this PR, so "incremental, microbatch and snapshot semantics" no longer matches what the file implements. Drop "microbatch".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: nit, tracked. We'll drop "microbatch" from the module doc.

@kilo-code-bot

kilo-code-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Code Review Summary

Status: 8 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 7
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/incremental-semantics.ts 131 The new manifest-based suppression trusts an on-disk manifest that may be stale (staleManifest is never passed in), so a PR that moves partition_by/snapshot check_cols/strategy to yml and changes the value is silently reported as unchanged

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/incremental-semantics.ts 129 Inline-vs-inline equality uses raw norm: reordered dict keys report a change, while whitespace inside quoted values ('order date' vs 'orderdate') is treated as equal
packages/opencode/src/altimate/review/incremental-semantics.ts 181 Whitespace is collapsed across string literals before the literal-preserving split, so 'on hold' → 'on hold' is invisible to the history/snapshot-logic rules
packages/opencode/src/altimate/review/incremental-semantics.ts 394 (carried) JSON.stringify compares check_cols positionally, so reordering the same columns is reported as a tracking change
packages/opencode/src/altimate/review/incremental-semantics.ts 400 (carried) Snapshot-change key list and body still duplicated between snapshotFindings and snapshotYmlFindings
packages/opencode/src/altimate/review/select-list.ts 71 First-branch UNION fix misses a parenthesized or FROM-less first branch, so a later branch's names are returned
packages/opencode/src/altimate/review/select-list.ts 20 The quote scanner does not handle backslash-escaped (or $$/[]-quoted) literals, so it can swallow the rest of the query
packages/opencode/src/altimate/review/orchestrate.ts 651 Empty lineage is treated as unknown and as failure, so literal-only models lose the column deltas this feature was added for
Files Reviewed (10 files in the incremental diff)
  • docs/docs/usage/dbt-pr-review.md - 0 issues
  • packages/opencode/src/altimate/review/config.ts - 0 issues
  • packages/opencode/src/altimate/review/incremental-semantics.ts - 5 issues
  • packages/opencode/src/altimate/review/orchestrate.ts - 1 issue
  • packages/opencode/src/altimate/review/rule-catalog.ts - 0 issues
  • packages/opencode/src/altimate/review/runner.ts - 0 issues
  • packages/opencode/src/altimate/review/select-list.ts - 2 issues
  • packages/opencode/test/altimate/review-contracts-dialect.test.ts - 0 issues
  • packages/opencode/test/altimate/review-incremental-semantics.test.ts - 0 issues
  • packages/opencode/test/altimate/review-select-list.test.ts - 0 issues

Fix these issues in Kilo Cloud

Previous Review Summary (commit fe31dc6)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit fe31dc6)

Status: 5 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 4
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/incremental-semantics.ts 220 on_schema_change='fail' skips the history block before filterChanged is evaluated, and the on_schema_change finding only fires on column changes — a filter-only change on such a model is reported by neither rule

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/incremental-semantics.ts 337 JSON.stringify compares check_cols positionally, so reordering the same columns is reported as a tracking change
packages/opencode/src/altimate/review/incremental-semantics.ts 343 Snapshot-change key list and body duplicated from snapshotFindings (lines 297-303)
packages/opencode/src/altimate/review/config.ts 50 New JSDoc inserted between the consumerColumnLists comment and its field, leaving that field undocumented
packages/opencode/src/altimate/review/incremental-semantics.ts 7 Module header still lists "microbatch" after the microbatch rule and capability entry were removed
Files Reviewed (13 files)
  • docs/docs/usage/dbt-pr-review.md - 0 issues
  • packages/opencode/src/altimate/review/adapter-capabilities.ts - 0 issues
  • packages/opencode/src/altimate/review/config.ts - 1 issue
  • packages/opencode/src/altimate/review/dbt-patterns.ts - 0 issues
  • packages/opencode/src/altimate/review/incremental-semantics.ts - 4 issues
  • packages/opencode/src/altimate/review/orchestrate.ts - 0 issues
  • packages/opencode/src/altimate/review/rule-catalog.ts - 0 issues
  • packages/opencode/src/altimate/review/runner.ts - 0 issues
  • packages/opencode/src/altimate/review/select-list.ts - 0 issues
  • packages/opencode/test/altimate/review-contracts-dialect.test.ts - 0 issues
  • packages/opencode/test/altimate/review-incremental-semantics.test.ts - 0 issues
  • packages/opencode/test/altimate/review-runner.test.ts - 0 issues
  • packages/opencode/test/altimate/review-select-list.test.ts - 0 issues

Fix these issues in Kilo Cloud


Reviewed by deepseek-v4.1-flash-1017 · Input: 158.1K · Output: 42.3K · Cached: 2.6M

Review guidance: REVIEW.md from base branch feat/review-contracts-dialect

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

13 issues found across 13 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/opencode/src/altimate/review/adapter-capabilities.ts">

<violation number="1" location="packages/opencode/src/altimate/review/adapter-capabilities.ts:41">
P3: These source links point at the `master` branch of dbt-trino, which moves over time. Pin them to a release tag or commit SHA so the cited behavior stays verifiable.</violation>
</file>

<file name="docs/docs/usage/dbt-pr-review.md">

<violation number="1" location="docs/docs/usage/dbt-pr-review.md:507">
P2: The detector recognizes a listed `var()` only on the right side of `=` or `IN`; range and reversed comparisons are missed. Limit the documentation to supported filters or extend detection.</violation>

<violation number="2" location="docs/docs/usage/dbt-pr-review.md:514">
P2: `DBT011`/`DBT012` don't appear anywhere in the repo, so this sentence points readers at rule IDs they can't find. Name the rule that actually covers `unique_key` and strategy changes, or remove the reference.</violation>
</file>

<file name="packages/opencode/src/altimate/review/incremental-semantics.ts">

<violation number="1" location="packages/opencode/src/altimate/review/incremental-semantics.ts:145">
P2: The non-greedy match stops at the first `endif`, not the one closing `is_incremental()`. With a nested Jinja `if`, changes after the inner block are omitted from `incrementalFilter` and can suppress the history finding; balance the Jinja blocks.</violation>

<violation number="2" location="packages/opencode/src/altimate/review/incremental-semantics.ts:236">
P2: This skips the case where `partition_by` is added to an existing incremental model. The existing table keeps no partitioning, so that case should warn too. Remove the old-value guard and rely on `configChanged`.</violation>
</file>

<file name="packages/opencode/src/altimate/review/dbt-patterns.ts">

<violation number="1" location="packages/opencode/src/altimate/review/dbt-patterns.ts:262">
P2: This detector only matches a `row_number() over (...)` whose window fits on one added line, so it misses the usual multi-line dbt dedup layout and any window whose PARTITION BY calls a function. Match against the joined added block instead, and use a regex that allows one level of nested parentheses.</violation>

<violation number="2" location="packages/opencode/src/altimate/review/dbt-patterns.ts:262">
P2: `ROW_NUMBER` can merely number rows; this emits a dedup finding even when `rn` is only selected and no rows are filtered, as in the new test. Require evidence that the rank is used to retain one row before reporting that a version survives.</violation>

<violation number="3" location="packages/opencode/src/altimate/review/dbt-patterns.ts:281">
P2: This load-timestamp dedup finding overlaps with core lint L039, which already flags timestamp-only ROW_NUMBER dedup. Add a `dedup:load-timestamp-only` entry to `CORE_AST_COVERED` in orchestrate.ts, or drop the QUALIFY form here, so one issue is not posted twice.</violation>
</file>

<file name="packages/opencode/src/altimate/review/rule-catalog.ts">

<violation number="1" location="packages/opencode/src/altimate/review/rule-catalog.ts:171">
P2: Gating `rs-getdate` to Redshift drops the idempotency warning for other engines that support `sysdate()`, such as Snowflake. The rule's own comment says this concern is kept across engines, so keep the rule engine-agnostic by removing the `Redshift:` prefix from its title or excluding it from the engine filter.</violation>
</file>

<file name="packages/opencode/src/altimate/review/orchestrate.ts">

<violation number="1" location="packages/opencode/src/altimate/review/orchestrate.ts:641">
P2: `outputColumns` merges in `selectListNames`, which reads only the last top-level SELECT. For UNION queries the output names come from the first branch, so a last-branch alias change can show up as a removed or added column. The consumer lanes then emit a false `removed column(s)` warning. Skip the merge when the SQL has a set operator, or read the first branch instead, and add a UNION case to the select-list tests.</violation>
</file>

<file name="packages/opencode/src/altimate/review/select-list.ts">

<violation number="1" location="packages/opencode/src/altimate/review/select-list.ts:42">
P2: The parser scans delimiters without respecting double-quoted identifiers: `select 1 as "from" from t` ends the list inside the alias. `blankComments` also strips `--` inside string literals, losing aliased outputs; make comment and keyword scanning quote-aware.

(Based on your team's feedback about quoted identifiers in SELECT scans.)</violation>

<violation number="2" location="packages/opencode/src/altimate/review/select-list.ts:53">
P2: Returning `null` on `*` discards every other aliased expression, even when lineage expands the wildcard. For `select *, 1 as new_col from t`, `outputColumns` keeps the lineage columns but loses `new_col`, so column-delta rules miss it; retain known aliases separately from wildcard uncertainty.</violation>
</file>

<file name="packages/opencode/src/altimate/review/runner.ts">

<violation number="1" location="packages/opencode/src/altimate/review/runner.ts:449">
P2: If packages contain models with the same basename, this returns whichever manifest node appears first. The caller passes only `modelNameFromPath` (the basename), so the rules can use another model's materialization or `unique_key`; resolve the node by path or `unique_id`.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

runs `create or replace` then `insert`, and `drop` removes the table during
the build, so readers outside dbt can see an empty or missing table (a
suggestion). `unique_key` and strategy changes are covered by the engine's
`DBT011`/`DBT012`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: DBT011/DBT012 don't appear anywhere in the repo, so this sentence points readers at rule IDs they can't find. Name the rule that actually covers unique_key and strategy changes, or remove the reference.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/docs/usage/dbt-pr-review.md, line 514:

<comment>`DBT011`/`DBT012` don't appear anywhere in the repo, so this sentence points readers at rule IDs they can't find. Name the rule that actually covers `unique_key` and strategy changes, or remove the reference.</comment>

<file context>
@@ -486,6 +487,31 @@ The deterministic rules group into four families:
+  runs `create or replace` then `insert`, and `drop` removes the table during
+  the build, so readers outside dbt can see an empty or missing table (a
+  suggestion). `unique_key` and strategy changes are covered by the engine's
+  `DBT011`/`DBT012`.
 - **dbt config / Jinja (`DBT0xx`).** A core **minijinja** parser reads
   `{{ config(...) }}` and the model body: incremental model without an
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing. These are altimate-core dbt_config_diff rule ids, and the bundled core 0.7.0 ships DBT011 unique_key_change and DBT012 incremental_strategy_change.

}
}

if (configChanged("partition_by") && inlineOld.partition_by !== undefined)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This skips the case where partition_by is added to an existing incremental model. The existing table keeps no partitioning, so that case should warn too. Remove the old-value guard and rely on configChanged.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/incremental-semantics.ts, line 236:

<comment>This skips the case where `partition_by` is added to an existing incremental model. The existing table keeps no partitioning, so that case should warn too. Remove the old-value guard and rely on `configChanged`.</comment>

<file context>
@@ -0,0 +1,348 @@
+      }
+    }
+
+    if (configChanged("partition_by") && inlineOld.partition_by !== undefined)
+      out.push(
+        finding(input, "incremental-partition-change", "warning", "materialization", "`partition_by` changed on an incremental model",
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. We'll report a newly added partition_by.

model: c.model,
confidence: "high",
evidence: { tool: "dbt-patterns", result: { rule: "dedup_load_timestamp_only", line: line.trim() } },
ruleKey: "dedup:load-timestamp-only",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This load-timestamp dedup finding overlaps with core lint L039, which already flags timestamp-only ROW_NUMBER dedup. Add a dedup:load-timestamp-only entry to CORE_AST_COVERED in orchestrate.ts, or drop the QUALIFY form here, so one issue is not posted twice.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/dbt-patterns.ts, line 281:

<comment>This load-timestamp dedup finding overlaps with core lint L039, which already flags timestamp-only ROW_NUMBER dedup. Add a `dedup:load-timestamp-only` entry to `CORE_AST_COVERED` in orchestrate.ts, or drop the QUALIFY form here, so one issue is not posted twice.</comment>

<file context>
@@ -249,6 +249,41 @@ const detectDedupTie: Detector = (c) => {
+      model: c.model,
+      confidence: "high",
+      evidence: { tool: "dbt-patterns", result: { rule: "dedup_load_timestamp_only", line: line.trim() } },
+      ruleKey: "dedup:load-timestamp-only",
+    })
+  }
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing: the bundled core 0.7.0 has no L039 lint code, so we couldn't confirm an overlap.

pos += piece.length + 1
const item = piece.replace(/^\s*(distinct|all)\s+/i, "").trim()
if (!item) continue
if (/(^|\.)\*$/.test(item)) return null

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Returning null on * discards every other aliased expression, even when lineage expands the wildcard. For select *, 1 as new_col from t, outputColumns keeps the lineage columns but loses new_col, so column-delta rules miss it; retain known aliases separately from wildcard uncertainty.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/select-list.ts, line 53:

<comment>Returning `null` on `*` discards every other aliased expression, even when lineage expands the wildcard. For `select *, 1 as new_col from t`, `outputColumns` keeps the lineage columns but loses `new_col`, so column-delta rules miss it; retain known aliases separately from wildcard uncertainty.</comment>

<file context>
@@ -0,0 +1,67 @@
+    pos += piece.length + 1
+    const item = piece.replace(/^\s*(distinct|all)\s+/i, "").trim()
+    if (!item) continue
+    if (/(^|\.)\*$/.test(item)) return null
+    const rawItem = raw.trim().replace(/^(distinct|all)\s+/i, "")
+    const bare = /^(?:[A-Za-z_][A-Za-z0-9_$]*\.)*("[^"]+"|[A-Za-z_][A-Za-z0-9_$]*)$/.exec(rawItem)
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. We'll keep known aliases next to a *.

try {
const raw = loadRawManifest(opts.manifestPath)
const hit = Object.entries<any>(raw?.nodes ?? {}).find(
([id, n]) => (id.startsWith("model.") || id.startsWith("snapshot.")) && n?.name === name,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: If packages contain models with the same basename, this returns whichever manifest node appears first. The caller passes only modelNameFromPath (the basename), so the rules can use another model's materialization or unique_key; resolve the node by path or unique_id.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/runner.ts, line 449:

<comment>If packages contain models with the same basename, this returns whichever manifest node appears first. The caller passes only `modelNameFromPath` (the basename), so the rules can use another model's materialization or `unique_key`; resolve the node by path or `unique_id`.</comment>

<file context>
@@ -442,6 +442,19 @@ export function createDispatcherRunner(opts: DispatcherRunnerOptions): ReviewRun
+      try {
+        const raw = loadRawManifest(opts.manifestPath)
+        const hit = Object.entries<any>(raw?.nodes ?? {}).find(
+          ([id, n]) => (id.startsWith("model.") || id.startsWith("snapshot.")) && n?.name === name,
+        )
+        return hit?.[1]?.config as Record<string, unknown> | undefined
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. We'll look up by unique_id. Same as #1724 r4626975851.

"https://trino.io/docs/current/functions/math.html",
// create_table_as with an enforced contract: `create or replace table` then `insert`;
// on_table_exists='drop' drops the table before building it.
"https://github.com/starburstdata/dbt-trino/blob/master/dbt/include/trino/macros/adapters.sql",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: These source links point at the master branch of dbt-trino, which moves over time. Pin them to a release tag or commit SHA so the cited behavior stays verifiable.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/adapter-capabilities.ts, line 41:

<comment>These source links point at the `master` branch of dbt-trino, which moves over time. Pin them to a release tag or commit SHA so the cited behavior stays verifiable.</comment>

<file context>
@@ -26,7 +33,14 @@ export const ADAPTER_CAPABILITIES: Record<string, AdapterCapability> = {
+      "https://trino.io/docs/current/functions/math.html",
+      // create_table_as with an enforced contract: `create or replace table` then `insert`;
+      // on_table_exists='drop' drops the table before building it.
+      "https://github.com/starburstdata/dbt-trino/blob/master/dbt/include/trino/macros/adapters.sql",
+      "https://github.com/starburstdata/dbt-trino/blob/master/dbt/include/trino/macros/materializations/table.sql",
+    ],
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: nit, tracked. We'll pin the dbt-trino links.

@hkaltai hkaltai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Panel: Claude, Codex, and lead reviewer. No CRITICAL or MAJOR findings; all findings are advisory-level correctness gaps. Counts: CRITICAL 0, MAJOR 0, MINOR 6 (inline), NIT 9.

The new rules are well scoped: adapter-specific behaviour is capability-gated, the slice rule is opt-in, formatting-only edits are filtered, and findings cannot block a merge. The main accuracy gaps are (1) Spark: dialect-pack rules remain ungated, (2) change detection compares only inline config, so moving config to yml produces false positives, and (3) the select-list reader mishandles UNION branches and -- inside string literals. Tests could not be fully run in the review environment (missing dependencies); the select-list tests pass. Missing tests: UNION output names, -- in literals, inline-to-yml config moves, Spark rules under another dialect.

Nits

  • packages/opencode/src/altimate/review/config.ts:44-55: incrementalSliceVars and its JSDoc sit between consumerColumnLists's JSDoc and the field, leaving that JSDoc dangling. Move the new field after consumerColumnLists.
  • incremental-semantics.ts:7 and orchestrate.ts:785: comments say "microbatch" but no microbatch rule remains. Drop the word.
  • incremental-semantics.ts:239: "The existing table keeps its old partitioning" understates it; on BigQuery/Spark the next run can fail. Suggest "...the next incremental run fails or keeps the old partitioning, depending on the adapter."
  • incremental-semantics.ts:283-285: both branches of the hasExposures ternary say nearly the same thing; simplify.
  • orchestrate.ts:639: comment "* in the final SELECT: keep lineage's own answer" sits above code with no separate branch; reword or remove.
  • contractTableReplaceAtomic also governs on_table_exists='drop', which has nothing to do with contracts; consider tableRebuildAtomic.
  • ticks is defined in both orchestrate.ts:666 and incremental-semantics.ts:204; share one.
  • incrementalSemanticsLane calls columnDelta again; two consumer lanes call it too, so lineage can run up to three times per model. Memoize per model context.
  • Also consider: per-slice unique_key rule requires an explicit incremental_strategy; with the adapter default (merge/delete+insert) it is silent (opt-in, so a gap not a bug). Normalizing SQL lowercases string literal values, so 'paid' -> 'PAID' produces no history-plan finding.


/** The engine a dialect-pack rule is about (from its `Engine:` title prefix), if any. */
export function ruleEngine(rule: Rule): string | undefined {
const m = /^(BigQuery|Snowflake|Redshift|Databricks):/.exec(rule.title)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Dialect gating skips the Spark: rules

const m = /^(BigQuery|Snowflake|Redshift|Databricks):/.exec(rule.title)

Three rules are titled Spark: ... (collect_list/collect_set ordering, CACHE/UNCACHE TABLE, to_date without format) and Spark/Databricks: explode() has a different prefix; none match, so ruleEngine returns undefined and they fire on every engine. On a Snowflake or Postgres project, to_date(order_date) still triggers "Spark: to_date/to_timestamp without format", which is the exact problem this gating is meant to fix.

Fix: add Spark (and Spark/Databricks) to the regex and map them to databricks, or store the engine explicitly on each pack rule instead of parsing the title. Add a test for a Spark rule under a non-Databricks dialect.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks. Fixed in fix(review): address review feedback: Spark: gating, quote-aware comments, effective-config changes, lineage-less column sets: Spark: and Spark/Databricks: titles map to databricks (a spark dialect counts as Databricks), so to_date(order_date) no longer fires on Snowflake or Postgres. Tested under Snowflake, Postgres, Databricks, Spark and an unknown engine.

const flat = flatten(sql)
const selects = [...flat.matchAll(/\bselect\b/gi)]
if (!selects.length) return null
const start = selects[selects.length - 1].index! + "select".length

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] selectListNames reads the last branch of a UNION

const start = selects[selects.length - 1].index! + "select".length

For select id, amount as revenue from a union all select id, total from b, output columns are named by the first branch, but this returns the last branch's names (id, total). outputColumns merges them with lineage, so columnDelta can report a column added or removed that the model never exposes. Appending union all select id, total from c yields a bogus "total is new" history-plan suggestion and consumer-lane warnings.

Fix: return null when the flattened text has a top-level union|except|intersect, or take the select list from the first branch.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Fixed in fix(review): address review comments: filter-only change under on_schema_change='fail', UNION output names: a top-level set operation takes its names from the first branch. Tested.


/** Text with comments blanked, same length. */
function blankComments(sql: string): string {
return sql.replace(/--[^\n]*/g, (m) => " ".repeat(m.length)).replace(/\/\*[\s\S]*?\*\//g, (m) => " ".repeat(m.length))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Comment stripping runs before string handling

return sql.replace(/--[^\n]*/g, (m) => " ".repeat(m.length))...

A -- inside a string literal truncates the rest of the line: select id, 'x--y' as tag, z from t yields ["id"], so tag and z are missed (and lineage-reported columns can then look removed). modelBody in incremental-semantics.ts has the same pattern, so an edit after 'a--b' on the same line is invisible to the history rules.

Fix: use one scanner that skips quoted strings before recognising comments (closing() already tracks quote state).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, thanks. Fixed in the new commit: one quote-aware blankComments scanner skips strings and quoted identifiers before recognising -- and /* */. selectListNames and modelBody both use it, so select id, 'x--y' as tag, z gives all three names and an edit after 'a--b' counts. modelBody also keeps string literals' case now, so 'paid' → 'PAID' is a change (your last nit).

}
}

if (configChanged("partition_by") && inlineOld.partition_by !== undefined)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] partition_by change compares only inline config

if (configChanged("partition_by") && inlineOld.partition_by !== undefined)

Moving partition_by from inline config() into schema.yml or dbt_project.yml leaves the effective value unchanged, but old-inline vs new-inline differs, so the author gets a "run --full-refresh" warning that is not needed. Conversely, a real change made in yml, or a new inline override of an inherited value, is never seen. Verified by reading configChanged and the inlineOld guard.

Fix: compare the old effective value (manifest base overlaid by old inline) against the new effective value, or at least stay silent when the new inline value is missing and the manifest value matches the old inline one.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Fixed in the new commit, partly. An inline value that is removed while the manifest carries the same value (moved to yml or dbt_project.yml) is no longer reported, and a removal with no yml value still is. A new inline override of an inherited value stays silent: without the base revision's effective config we can't tell whether the value changed. A real change made only in yml isn't seen by this rule yet. Both are tracked follow-ups.

const before = inlineConfig(input.oldSql)
const after = inlineConfig(input.newSql)
const out: Finding[] = []
const changed = ["check_cols", "strategy"].filter((k) => before[k] !== undefined && norm(before[k]) !== norm(after[k]))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Snapshot tracking change has the same inline-only comparison

const changed = ["check_cols", "strategy"].filter((k) => before[k] !== undefined && norm(before[k]) !== norm(after[k]))

Moving strategy or check_cols out of the inline config into yml reports a tracking change although the effective config is identical; adding an inline override to a snapshot that inherited its config is not reported.

Fix: compare effective tracking configs on both revisions.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Fixed in the new commit the same way: snapshots now receive the manifest config, and moving check_cols/strategy to yml with the same value is not a tracking change. Tested both ways.

if (cols.some((c) => c === "*" || c.endsWith(".*"))) return new Set()
return new Set(cols)
// `*` in the final SELECT: keep lineage's own answer (it expands stars it can resolve).
const named = selectListNames(sql)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Select-list names fill in when lineage returned nothing

return new Set([...cols, ...(named ?? [])])

Previously an empty lineage result gave an empty set and columnDelta reported unknown. Now select-list names fill in, skipping unaliased expressions. If lineage fails on only one revision, every unaliased expression column looks removed, feeding the consumer-contract and consumer-list warnings.

Fix: add select-list names only when lineage returned columns, or pass a flag saying whether lineage succeeded.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, thanks. Fixed in the new commit: when lineage returns no columns for a side, that side's set is unknown and select-list names only add to lineage's columns. There's a test where lineage fails on the base only and no column looks removed.

RaghwendraSingh and others added 6 commits November 20, 2026 12:13
…apter-dependent rules read from the capability table

- `incremental-semantics.ts`: config from the manifest (dbt-merged) overlaid by the head's inline `{{ config() }}`; a new `modelConfig` runner method
- incremental model or snapshot logic changed with no stated plan for history (`--full-refresh`, backfill, forward-only) in the PR
- `partition_by` changed on an incremental model
- `on_schema_change='fail'` with added or removed output columns
- `delete+insert`/`merge` whose `unique_key` lacks the column a `var()` filter slices runs by (one tenant's run deleting another's rows)
- snapshot `check_cols`/`strategy` change, and snapshot query change without a plan
- `row_number()` dedup ordered only by a load timestamp
- capability table: Trino/Athena concurrent Iceberg commits conflict (microbatch without `concurrent_batches=false`); Trino `on_table_exists='replace'` + enforced contract is create-then-insert and `drop` removes the table during the build (warning when the model has consumers); sources cite dbt-trino and Iceberg docs
- all findings are advisory (catalog origin); unchanged real-world corpus: 15/15 caught, 0/5 false positives

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found by the local end-to-end run on a real DuckDB dbt project:

- BigQuery/Snowflake/Redshift/Databricks pack rules (titled `Engine: …`) fire
  only on that engine once the project's engine is known; an unknown engine
  keeps today's behavior. `ruleEngine()` maps a rule's title prefix to its
  engine.
- Tests: a DuckDB or Snowflake project gets no BigQuery pack advice; a
  BigQuery project, or an unknown engine, keeps it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ory rule, opt-in slice vars, no microbatch guess, yml snapshots

From an adversarial generality review:

- the history rule fires only when an incremental model gains a column or its `is_incremental()` filter changes, compares tokens (formatting never counts), defers to the `on_schema_change='fail'` finding, and is a suggestion
- the slice-column `unique_key` rule is opt-in: only vars listed in `incrementalSliceVars` count (a `reporting_currency` var filter is not a slice)
- the Trino/Athena microbatch rule and its capability entry are removed: dbt-trino's concurrency default is changing and the connector/table format isn't knowable from the adapter
- the non-atomic Trino rebuild is always a suggestion and speaks to readers outside dbt, not downstream dbt models
- load-timestamp dedup ignores loader sync columns (`_fivetran_synced`, `_airbyte_*`, `_sdc_*`) and is a suggestion
- snapshot `check_cols`/`strategy` changes are also caught in dbt 1.9+ yml snapshots; snapshot query changes compare tokens

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found by the local end-to-end run: column lineage only reports columns that read a source column, so `current_date as load_date` or `1 as one` were invisible and adding one under `on_schema_change='fail'` went unreported. Output columns now add the final SELECT's own names (`select-list.ts`: an item counts only when aliased after a complete expression or a bare column; `*` defers to lineage).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hema_change='fail'`, UNION output names

- `incremental-semantics.ts`: under `on_schema_change='fail'` only column
  changes are deferred to the fail finding; a change to the `is_incremental()`
  filter alone is reported as needing a history plan again
- `select-list.ts`: a top-level set operation (`UNION`/`INTERSECT`/`EXCEPT`/
  `MINUS`) takes its output names from the first branch, so an alias change in
  a later branch no longer reads as added or removed columns
- tests for both

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mments, effective-config changes, lineage-less column sets

- Dialect packs: `Spark:` and `Spark/Databricks:` rules are gated like the
  other packs and apply to Databricks/Spark only once the engine is known.
- `blankComments` skips quoted strings and identifiers before recognising
  `--` and `/* */`, so `'x--y'` no longer truncates the select list;
  `modelBody` uses it too and keeps string literals' case, so an edit after
  `'a--b'` or `'paid'` -> `'PAID'` counts as a change.
- `partition_by` and snapshot `check_cols`/`strategy`: an inline value removed
  while the manifest carries the same value (moved to yml or
  `dbt_project.yml`) is not reported; snapshots now receive the manifest
  config.
- `outputColumns`: when lineage returns nothing for a side, the column set is
  unknown; select-list names only add to lineage's columns.
- Nits: `partition_by` message says the next run can fail; no "microbatch" in
  comments; `consumerColumnLists` JSDoc back above its field; one exposures
  sentence.
- Tests: Spark rules under other dialects, `--` in literals, literal case,
  inline-to-yml moves for `partition_by` and `check_cols`, one-sided lineage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@RaghwendraSingh
RaghwendraSingh force-pushed the feat/review-contracts-dialect branch from 1522b5a to 9fa035e Compare November 20, 2026 06:53
@RaghwendraSingh
RaghwendraSingh force-pushed the feat/review-incremental-semantics branch from fe31dc6 to 9625cfd Compare November 20, 2026 06:53
@github-actions

Copy link
Copy Markdown

👋 This PR was automatically closed by our quality checks.

Common reasons:

  • New GitHub account with limited contribution history
  • PR description doesn't meet our guidelines
  • Contribution appears to be AI-generated without meaningful review

If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you.

2 similar comments
@github-actions

Copy link
Copy Markdown

👋 This PR was automatically closed by our quality checks.

Common reasons:

  • New GitHub account with limited contribution history
  • PR description doesn't meet our guidelines
  • Contribution appears to be AI-generated without meaningful review

If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you.

@github-actions

Copy link
Copy Markdown

👋 This PR was automatically closed by our quality checks.

Common reasons:

  • New GitHub account with limited contribution history
  • PR description doesn't meet our guidelines
  • Contribution appears to be AI-generated without meaningful review

If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you.

@RaghwendraSingh

Copy link
Copy Markdown
Contributor Author

@hkaltai, re your review:

Thanks for the approval and the detailed notes. One commit on top:

  • Fixed: Spark rule gating, quote-aware comment stripping (select list and model body), partition_by and snapshot moves to yml, and lineage-less column sets. The UNION first-branch fix was already in.
  • Nits taken: incrementalSliceVars moved below consumerColumnLists's JSDoc; "microbatch" dropped from the comments; the partition_by message says the next run can fail or keep the old partitioning; one exposures sentence; the misleading * comment reworded; string-literal case kept.
  • Deferred: renaming contractTableReplaceAtomic, sharing ticks, memoising columnDelta per model, and the slice rule under the adapter-default strategy.
  • Tests added: Spark rules under other dialects, -- in literals, inline-to-yml moves for partition_by and check_cols, one-sided lineage. UNION names were already tested.
  • Note: the earlier commit "no false 'table not found' …" now only carries the dialect-pack gating, so it is reworded to fix(review): dialect packs only on their own engine. Its runner fix moved into feat(review): consumers outside dbt, engine-aware integer division, opt-in contract and column-list checks #1724.

const n = after[key]
if (o === undefined || norm(o) === norm(n)) return false
const m = manifest?.[key]
if (n === undefined && m !== undefined && m !== null && canonValue(inlineValue(o)) === canonValue(m)) return false

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: Manifest-based suppression can hide a real config change when the manifest is stale

manifestConfig is read from the on-disk head manifest, which this review already knows may be stale (OrchestrateInput.staleManifest from run.ts::detectStaleManifest), and the lane never passes that flag in. If a PR both moves partition_by (or a snapshot's check_cols/strategy) from inline config into yml and changes the value, a manifest compiled before the yml edit still carries the old inline value, so this returns false and the change is never reported. Thread staleManifest into the lane and skip the suppression when it is set (or compare against a base-revision manifest).

): boolean {
const o = before[key]
const n = after[key]
if (o === undefined || norm(o) === norm(n)) return false

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Compare canonical values, not raw text, for inline-vs-inline changes

norm only removes whitespace, normalizes quotes and lowercases, so reordering the same dict keys ({'field': 'd', 'granularity': 'day'} vs {'granularity': 'day', 'field': 'd'}) is reported as a change, while it strips whitespace inside quoted values so 'order date' and 'orderdate' are treated as equal. Use the same canonicalization as the manifest comparison.

Suggested change
if (o === undefined || norm(o) === norm(n)) return false
if (o === undefined || (n !== undefined && canonValue(inlineValue(o)) === canonValue(inlineValue(n)))) return false

}
// Lowercase everything but string literals: `'paid'` → `'PAID'` is a change.
return s
.replace(/\s+/g, " ")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Whitespace inside string literals is collapsed before the literal-preserving split

.replace(/\s+/g, " ") runs over the whole SQL, so where status = 'on hold' and 'on hold' produce the same modelBody (and neither the incremental-history nor the snapshot-logic rule fires), even though the literal value changed. Split on the string literals first and collapse whitespace only in the non-literal parts. Double-quoted literals are also still lowercased, since only '…' is preserved.

const flat = flatten(sql)
const selects = [...flat.matchAll(/\bselect\b/gi)]
if (!selects.length) return null
const head = SET_OP.test(flat) ? selects[0] : selects[selects.length - 1]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: A parenthesized or FROM-less first branch still yields a later branch's names

flatten blanks everything inside parentheses, so for (select a as x from t) union all select b as y from u the first top-level select is the second branch and this returns ["y"] instead of the first-branch name ["x"]. When the first branch has no from, fromRel likewise crosses the set operator: select 1 as a union all select 2 as b returns ["b"]. Return null when the set operator precedes the first top-level select, and bound the first branch's list at the set operator.

if (c === "'" || c === '"' || c === "`") {
// A doubled quote inside a string is an escaped quote.
let j = i + 1
while (j < sql.length && !(sql[j] === c && sql[j + 1] !== c)) j += sql[j] === c ? 2 : 1

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Backslash-escaped quotes desync the scanner

This loop only treats doubled quotes ('') as an escape. In dialects that accept \' (MySQL, Snowflake, Redshift, Postgres E'…'), and for BigQuery '''…''' strings, the escaped quote is taken as the closing one, a later quote is treated as a new opener and the rest of the query (including real -- comments) is swallowed: select 'O\'Brien' as name, y -- note, junk yields a bogus junk column, and select 'O\'--Brien' as name, y from t blanks everything after --. Treat \ escapes (and $$…$$ / [...] quoting) as opaque while scanning.

// Lineage returned nothing (failed, or could not analyse this side): the
// set is unknown. Filling it from the select list alone would make every
// unaliased expression look removed when only one side's lineage failed.
if (!cols.length) return new Set()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Empty lineage means unknown and legitimately empty, losing the literal-only case

columnLineage returns [] both on failure and when a model's output reads no source column. For a literal-only model (select 'x' as kind, 1 as one gaining current_date as load_date), both sides now return empty, so columnDelta reports nothing and selectListNames never runs — the case this feature was added for. Have the runner signal failure (null) separately so a legitimate empty result still falls through to the select-list names.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants