Repository navigation
fix(review): precision and new rules from a replay of human-caught defects on public dbt PRs - #1739
RaghwendraSingh wants to merge 13 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
|
Thanks for your contribution! This PR doesn't have a linked issue. All PRs must reference an existing issue. Please:
See CONTRIBUTING.md for details. |
full receipts (1 session)
orchestrator ·
|
| subagent | cost |
|---|---|
| You are judging whether an automated dbt PR reviewer caught real defects that h… | 461,048 tokens |
| You are judging whether an automated dbt PR reviewer caught real defects that h… | 290,426 tokens |
| You are a senior analytics engineer judging the quality of comments an automate… | 2,870,788 tokens |
| You are a senior analytics engineer judging the quality of comments an automate… | 388,026 tokens |
| You are judging whether an automated dbt PR reviewer caught real defects that h… | 918,952 tokens |
| You are a senior analytics engineer judging the quality of comments an automate… | 705,099 tokens |
| You are a senior analytics engineer judging the quality of comments an automate… | 3,051,691 tokens |
| Read the judging instructions in /private/tmp/claude-501/-Users-raghwendrasingh… | 1,297,174 tokens |
| You are judging a dbt PR-review benchmark, blind. Two versions of a code-review… | 497,477 tokens |
| You are judging whether an automated dbt PR reviewer caught real defects that h… | 348,137 tokens |
| You are judging whether an automated dbt PR reviewer caught real defects that h… | 400,555 tokens |
| You are labeling real dbt PR review comments for a benchmark. Label by READING… | 894,538 tokens |
| Read the judging instructions in /private/tmp/claude-501/-Users-raghwendrasingh… | 887,356 tokens |
| You are a senior analytics engineer judging the quality of comments an automate… | 1,348,124 tokens |
| You are a senior analytics engineer judging the quality of comments an automate… | 2,519,859 tokens |
| You are a senior analytics engineer judging the quality of comments an automate… | 3,052,334 tokens |
| You are judging whether an automated dbt PR reviewer caught real defects that h… | 561,451 tokens |
| You are auditing feature coverage for a dbt PR-review tool. For each real revie… | 1,085,864 tokens |
| Score the Phase 2 (AI verifier) decision run for the dbt PR-review benchmark. S… | 12,115,853 tokens |
| 111 more subagents | 281,050,584 unpriced tokens |
Generated by aireceipts
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
There was a problem hiding this comment.
27 issues found across 21 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/opencode/test/altimate/review-run-stale.test.ts">
<violation number="1" location="packages/opencode/test/altimate/review-run-stale.test.ts:68">
P3: These tests leave every temporary Git repository behind after completion, causing `/tmp` accumulation across repeated local and CI runs. Register cleanup with `afterEach`/`finally`, or use the test fixture with `await using` so each directory is disposed automatically.</violation>
</file>
<file name="docs/docs/usage/dbt-pr-review.md">
<violation number="1" location="docs/docs/usage/dbt-pr-review.md:586">
P2: The AI lane does not see every changed dbt file: deleted files are filtered out, and `analysis`/other classified dbt files are never added to its input. Say “eligible non-deleted files in these supported roles” or expand the implementation to cover all changed dbt kinds.</violation>
</file>
<file name="packages/opencode/src/altimate/review/runner.ts">
<violation number="1" location="packages/opencode/src/altimate/review/runner.ts:454">
P2: This requires `attached_node`, but older dbt manifests identify the test owner only through `depends_on.nodes`. Reuse that fallback or uniqueness tests disappear from both new key-based review lanes.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/runner.ts:457">
P2: `declaredUniqueKeys` misses standard dbt `unique` tests because `column_name` is nested under `test_metadata.kwargs`, so the new join and incremental-key checks silently skip models with ordinary uniqueness tests. Read `kwargs.column_name` like the existing manifest helper does.</violation>
</file>
<file name="packages/opencode/src/altimate/review/dbt-patterns.ts">
<violation number="1" location="packages/opencode/src/altimate/review/dbt-patterns.ts:318">
P2: Qualified columns written directly after a Jinja relation are no longer detected, so common `date({{ ref('events') }}.event_date)` filters evade the partition-pruning rule. Allow the identifier matcher to start after a qualification dot.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/dbt-patterns.ts:710">
P2: This prefix-based association can report an unchanged whole-table window when the PR adds another window with the same `row_number() over (` prefix. Associate the finding with the actual added window span rather than matching duplicate prefixes.</violation>
<violation number="3" location="packages/opencode/src/altimate/review/dbt-patterns.ts:863">
P2: Add `mysql` to `ADAPTERS`; otherwise MySQL-specific macros produce portability noise and project adapter discovery misses `mysql__...` variants.</violation>
<violation number="4" location="packages/opencode/src/altimate/review/dbt-patterns.ts:896">
P2: A default on another `var('window', ...)` call suppresses the warning for `var('window')`, but dbt does not make that per-call default global. Check each invocation independently, or only suppress when the project defines the variable.</violation>
<violation number="5" location="packages/opencode/src/altimate/review/dbt-patterns.ts:908">
P2: Distinct missing variables in one macro are deduplicated into a single finding because their `ruleKey` omits the variable name. Include `name` in the key so every compile-breaking variable is reported.</violation>
<violation number="6" location="packages/opencode/src/altimate/review/dbt-patterns.ts:1535">
P2: `movedTo` assumes any matching test on a newly tested column represents a column rename, so it hides genuine removals when a guardrail is intentionally reassigned. Only suppress the finding when a rename is established; otherwise retain the old-column removal.</violation>
</file>
<file name="packages/opencode/src/altimate/review/orchestrate.ts">
<violation number="1" location="packages/opencode/src/altimate/review/orchestrate.ts:1854">
P2: This reports missing targets for commented-out refs because the new lane passes raw added lines to `missingRefFindings`. Ignore SQL/Jinja comments before matching, otherwise a harmless comment can produce a warning that dbt cannot parse the model.</violation>
</file>
<file name="packages/opencode/src/altimate/review/run.ts">
<violation number="1" location="packages/opencode/src/altimate/review/run.ts:572">
P2: `projectResourceNames` treats every matching file as a dbt resource, so `ref('foo')` is accepted when only `tests/foo.sql`, `macros/foo.sql`, or an arbitrary CSV exists. Restrict both scans to configured model, seed, and snapshot paths, including package layouts.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/run.ts:574">
P2: `projectResourceNames` searches only `*.yml`, although dbt file handling supports `.yaml`; references to YAML-only snapshots in that extension produce false `ref-target-missing` findings. Search both extensions.</violation>
</file>
<file name="packages/opencode/src/altimate/review/project-rules.ts">
<violation number="1" location="packages/opencode/src/altimate/review/project-rules.ts:165">
P2: Valid Jinja `with` blocks currently produce a `jinja-unbalanced` finding because `with` is missing from `BLOCK_OPEN`. Add the supported Jinja block tags before treating their `end*` tags as errors.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/project-rules.ts:329">
P2: Inherited `incremental_strategy` values are ignored even though `manifestConfig` is provided to fill omitted file config. Fall back to `manifestConfig.incremental_strategy` here so inherited merge and microbatch models are checked.</violation>
<violation number="3" location="packages/opencode/src/altimate/review/project-rules.ts:331">
P2: Check that `cfg.unique_key` is a meaningful value before reporting it. Raw `None`, `null`, empty-string, and empty-list values are truthy strings here but do not configure a dbt unique key.</violation>
<violation number="4" location="packages/opencode/src/altimate/review/project-rules.ts:365">
P2: This rule reports a missing partition merge key without checking `incremental_predicates`, even though those predicates can prune the destination partitions and are offered as the fix. Suppress the finding when the configured predicates constrain the partition column.</violation>
</file>
<file name="packages/opencode/src/altimate/review/rule-catalog.ts">
<violation number="1" location="packages/opencode/src/altimate/review/rule-catalog.ts:2365">
P2: This whole-file UNION rule misses the common branch-edit case because its relevance regex requires `UNION` in the changed lines. Add an `added` condition for the changed `select *` branch and retain the whole-file check.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/rule-catalog.ts:2477">
P2: The relevance regex misses multiline CASE additions: `case` and `when` are commonly written on separate lines, so the structural check never runs for a newly added unbalanced CASE. Anchor relevance to an added `CASE` or `WHEN` token instead of requiring `case when` on one line.</violation>
</file>
<file name="packages/opencode/src/altimate/review/incremental-semantics.ts">
<violation number="1" location="packages/opencode/src/altimate/review/incremental-semantics.ts:143">
P2: `CLAUSE_END` treats `left` and `right` as clause starts even when they are SQL functions. A change from `left(name, 3)` to `left(name, 4)` can be reduced to the same `where` token and bypass the history check.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/incremental-semantics.ts:152">
P2: `predicates` returns a `where`/`having` branch before applying `CLAUSE_END`, so trailing `GROUP BY` or `ORDER BY` edits are reported as filter changes. Restrict this early return to bare `and`/`or` fragments.</violation>
<violation number="3" location="packages/opencode/src/altimate/review/incremental-semantics.ts:154">
P2: `predicates` documents `on` clauses but never scans them, and it also omits `qualify`; join-only or qualify-only incremental changes therefore bypass the history-plan check.</violation>
<violation number="4" location="packages/opencode/src/altimate/review/incremental-semantics.ts:171">
P2: `split` treats a nested Jinja `{% else %}` as the guard's branch separator. Changes in the nested else arm are dropped from `incrementalFilter`, so the history-plan check can miss a real incremental change.</violation>
</file>
<file name="packages/opencode/src/altimate/review/ai-review.ts">
<violation number="1" location="packages/opencode/src/altimate/review/ai-review.ts:39">
P3: The new `source_yml` label is unreachable because `classifyDbtFile` never produces that kind; source declaration YAML is labeled as `schema_yml` instead. Add source-YAML classification before the generic schema branch, or remove this unused role so the AI receives an accurate file label.</violation>
</file>
<file name="packages/opencode/src/altimate/review/format.ts">
<violation number="1" location="packages/opencode/src/altimate/review/format.ts:42">
P3: `GROUP_MIN` is documented as a model threshold, but the grouping check counts findings. Two occurrences for one model now collapse into one bullet and hide their separate details; require `names.length >= GROUP_MIN` or otherwise require distinct model/file names.</violation>
</file>
<file name="packages/opencode/src/altimate/review/finding.ts">
<violation number="1" location="packages/opencode/src/altimate/review/finding.ts:211">
P3: `ruleTitle` does not normalize model or file labels containing spaces or non-ASCII characters, so equivalent findings fail to merge or group. Strip the exact model/file prefix instead of restricting the prefix to `\w`, dots, slashes, and hyphens.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/finding.ts:229">
P3: This delimiter-joined grouping key is ambiguous when a file or title contains `|`, allowing unrelated findings to merge. Serialize the tuple with `JSON.stringify` or use a collision-safe key.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| 3. **AI reviewer lane (layer 3).** When a model is configured, the LLM reviewer | ||
| runs **after** the deterministic lanes, grounded in their findings, and adds | ||
| advisory contextual comments (intent, naming, business logic, test coverage). | ||
| It sees every changed dbt file, not only models: macros, singular data tests, |
There was a problem hiding this comment.
P2: The AI lane does not see every changed dbt file: deleted files are filtered out, and analysis/other classified dbt files are never added to its input. Say “eligible non-deleted files in these supported roles” or expand the implementation to cover all changed dbt kinds.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/docs/usage/dbt-pr-review.md, line 586:
<comment>The AI lane does not see every changed dbt file: deleted files are filtered out, and `analysis`/other classified dbt files are never added to its input. Say “eligible non-deleted files in these supported roles” or expand the implementation to cover all changed dbt kinds.</comment>
<file context>
@@ -579,6 +583,9 @@ across 12 dialects all plan; anything still unrecognized stays *undecidable*, ne
3. **AI reviewer lane (layer 3).** When a model is configured, the LLM reviewer
runs **after** the deterministic lanes, grounded in their findings, and adds
advisory contextual comments (intent, naming, business logic, test coverage).
+ It sees every changed dbt file, not only models: macros, singular data tests,
+ snapshots, schema/source yml, seeds (diff only) and project config, each
+ labeled with its role, with models first under the 20-file cap.
</file context>
| It sees every changed dbt file, not only models: macros, singular data tests, | |
| It sees eligible non-deleted dbt files, not only models: macros, singular data tests, |
| if (!uid) return [] | ||
| const keys: string[][] = [] | ||
| for (const [, n] of nodes) { | ||
| if (n?.resource_type !== "test" || n.attached_node !== uid) continue |
There was a problem hiding this comment.
P2: This requires attached_node, but older dbt manifests identify the test owner only through depends_on.nodes. Reuse that fallback or uniqueness tests disappear from both new key-based review lanes.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/runner.ts, line 454:
<comment>This requires `attached_node`, but older dbt manifests identify the test owner only through `depends_on.nodes`. Reuse that fallback or uniqueness tests disappear from both new key-based review lanes.</comment>
<file context>
@@ -442,6 +442,29 @@ export function createDispatcherRunner(opts: DispatcherRunnerOptions): ReviewRun
+ if (!uid) return []
+ const keys: string[][] = []
+ for (const [, n] of nodes) {
+ if (n?.resource_type !== "test" || n.attached_node !== uid) continue
+ const name = String(n.test_metadata?.name ?? "")
+ const kw = n.test_metadata?.kwargs ?? {}
</file context>
| if (n?.resource_type !== "test" || n.attached_node !== uid) continue | |
| if (n?.resource_type !== "test" || (n.attached_node || (n.depends_on?.nodes ?? []).find((id: string) => id.startsWith("model."))) !== uid) continue |
| * default when `x` isn't set in dbt_project.yml (in a package, every project that | ||
| * doesn't set it fails to compile). | ||
| */ | ||
| export const ADAPTERS = ["bigquery", "snowflake", "databricks", "spark", "redshift", "postgres", "trino", "athena", "duckdb", "exasol", "synapse", "sqlserver", "fabric", "oracle", "clickhouse", "presto"] |
There was a problem hiding this comment.
P2: Add mysql to ADAPTERS; otherwise MySQL-specific macros produce portability noise and project adapter discovery misses mysql__... variants.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/dbt-patterns.ts, line 863:
<comment>Add `mysql` to `ADAPTERS`; otherwise MySQL-specific macros produce portability noise and project adapter discovery misses `mysql__...` variants.</comment>
<file context>
@@ -824,6 +855,64 @@ const MODEL_DETECTORS: Detector[] = [
+ * default when `x` isn't set in dbt_project.yml (in a package, every project that
+ * doesn't set it fails to compile).
+ */
+export const ADAPTERS = ["bigquery", "snowflake", "databricks", "spark", "redshift", "postgres", "trino", "athena", "duckdb", "exasol", "synapse", "sqlserver", "fabric", "oracle", "clickhouse", "presto"]
+
+/** The adapter a macro file is specific to, from its directory or its macros' `<adapter>__` prefix. */
</file context>
| export const ADAPTERS = ["bigquery", "snowflake", "databricks", "spark", "redshift", "postgres", "trino", "athena", "duckdb", "exasol", "synapse", "sqlserver", "fabric", "oracle", "clickhouse", "presto"] | |
| export const ADAPTERS = ["bigquery", "snowflake", "databricks", "spark", "redshift", "postgres", "trino", "athena", "mysql", "duckdb", "exasol", "synapse", "sqlserver", "fabric", "oracle", "clickhouse", "presto"] |
| test: "singular data test", | ||
| snapshot: "snapshot", | ||
| schema_yml: "schema yml (tests, contracts, descriptions)", | ||
| source_yml: "sources yml", |
There was a problem hiding this comment.
P3: The new source_yml label is unreachable because classifyDbtFile never produces that kind; source declaration YAML is labeled as schema_yml instead. Add source-YAML classification before the generic schema branch, or remove this unused role so the AI receives an accurate file label.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-review.ts, line 39:
<comment>The new `source_yml` label is unreachable because `classifyDbtFile` never produces that kind; source declaration YAML is labeled as `schema_yml` instead. Add source-YAML classification before the generic schema branch, or remove this unused role so the AI receives an accurate file label.</comment>
<file context>
@@ -19,13 +19,45 @@ const MAX_FILES = 20
+ test: "singular data test",
+ snapshot: "snapshot",
+ schema_yml: "schema yml (tests, contracts, descriptions)",
+ source_yml: "sources yml",
+ seed: "seed CSV",
+ project_config: "project config",
</file context>
| export const SUMMARY_BUDGET_CHARS = 5_000 | ||
| /** The same rule on this many models or more collapses into one bullet. */ | ||
| export const GROUP_MIN = 3 | ||
| export const GROUP_MIN = 2 |
There was a problem hiding this comment.
P3: GROUP_MIN is documented as a model threshold, but the grouping check counts findings. Two occurrences for one model now collapse into one bullet and hide their separate details; require names.length >= GROUP_MIN or otherwise require distinct model/file names.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/format.ts, line 42:
<comment>`GROUP_MIN` is documented as a model threshold, but the grouping check counts findings. Two occurrences for one model now collapse into one bullet and hide their separate details; require `names.length >= GROUP_MIN` or otherwise require distinct model/file names.</comment>
<file context>
@@ -39,7 +39,7 @@ export function verdictHeadline(env: VerdictEnvelope): string {
export const SUMMARY_BUDGET_CHARS = 5_000
/** The same rule on this many models or more collapses into one bullet. */
-export const GROUP_MIN = 3
+export const GROUP_MIN = 2
export interface RenderOptions {
</file context>
| out.push(f) | ||
| continue | ||
| } | ||
| const key = [f.file, f.category, f.severity, !!f.degraded, ruleTitle(f)].join("|") |
There was a problem hiding this comment.
P3: This delimiter-joined grouping key is ambiguous when a file or title contains |, allowing unrelated findings to merge. Serialize the tuple with JSON.stringify or use a collision-safe key.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/finding.ts, line 229:
<comment>This delimiter-joined grouping key is ambiguous when a file or title contains `|`, allowing unrelated findings to merge. Serialize the tuple with `JSON.stringify` or use a collision-safe key.</comment>
<file context>
@@ -203,3 +203,46 @@ export function dedupe(findings: Finding[]): Finding[] {
+ out.push(f)
+ continue
+ }
+ const key = [f.file, f.category, f.severity, !!f.degraded, ruleTitle(f)].join("|")
+ const g = groups.get(key)
+ if (g) g.push(f)
</file context>
| const key = [f.file, f.category, f.severity, !!f.degraded, ruleTitle(f)].join("|") | |
| const key = JSON.stringify([f.file, f.category, f.severity, !!f.degraded, ruleTitle(f)]) |
| * names blanked, so the same rule on different models/columns shares a key. */ | ||
| export function ruleTitle(f: Pick<Finding, "title">): string { | ||
| return f.title | ||
| .replace(/^[\w.\-/]+: /, "") |
There was a problem hiding this comment.
P3: ruleTitle does not normalize model or file labels containing spaces or non-ASCII characters, so equivalent findings fail to merge or group. Strip the exact model/file prefix instead of restricting the prefix to \w, dots, slashes, and hyphens.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/finding.ts, line 211:
<comment>`ruleTitle` does not normalize model or file labels containing spaces or non-ASCII characters, so equivalent findings fail to merge or group. Strip the exact model/file prefix instead of restricting the prefix to `\w`, dots, slashes, and hyphens.</comment>
<file context>
@@ -203,3 +203,46 @@ export function dedupe(findings: Finding[]): Finding[] {
+ * names blanked, so the same rule on different models/columns shares a key. */
+export function ruleTitle(f: Pick<Finding, "title">): string {
+ return f.title
+ .replace(/^[\w.\-/]+: /, "")
+ .replace(/ \(\+\d+ more\)$/, "")
+ .replace(/`[^`]*`/g, "`…`")
</file context>
| })) | ||
| // Macros, data tests, snapshots, yml and seeds carry defects no model diff shows. | ||
| // Seeds go as a diff only: the full CSV is data, not logic. | ||
| const supporting = reviewable.filter((f) => AI_SUPPORTING_KINDS.has(f.kind) && f.status !== "deleted") |
There was a problem hiding this comment.
WARNING: project_config includes profiles.yml, and its full content is sent to the model provider
classifyDbtFile maps profiles, packages and dbt_project all to kind: "project_config" (diff-filter.ts:122), so this filter picks up a committed profiles.yml and the getContent(...) just below passes its whole text to the advisory AI lane as sql. That lane runs by default in the lite/full tiers (TIER_LANES), and a committed profiles file routinely carries password / token / private_key / client_secret or a key-file path, so warehouse credentials egress to a third-party model whenever the PR touches it. Seeds are correctly diff-only here; profiles.yml should be excluded from the payload entirely (or credential keys redacted) while dbt_project.yml / packages.yml continue to be sent.
| { added: /\bcase\s+when\b[^;]*$/i, unless: /\bend\b|\{\{|\{%/i }, | ||
| "The model has more `CASE` than `END` keywords, which is a syntax error.", | ||
| // Whole-file balance, not per line: a CASE whose END sits on a later line is fine. | ||
| // Jinja branches can emit different CASE/END counts per branch, so they abstain. |
There was a problem hiding this comment.
WARNING: this abstention disables the CASE/END check on almost every real model
JINJA_BRANCH matches any {% if %}, {% for %}, {% elif %} or {% else %} in the whole file, so caseDepth(s) === -1 && !JINJA_BRANCH.test(s) can only fire on a model with no Jinja control flow at all. Every incremental model has {% if is_incremental() %}, so an added CASE with no END — the exact syntax error this rule targets — is never reported there. The previous per-line version did fire in that case because it only inspected the added line. Consider abstaining only when a Jinja tag actually wraps or emits CASE/END (or compare the balance per branch) rather than on the presence of any block anywhere in the file.
| const col = String(c?.name ?? "").toLowerCase() | ||
| if (!nulls.has(col)) continue | ||
| const failing = [...testNames(c.data_tests), ...testNames(c.tests)].filter((t) => /(^|\.)(not_null|relationships)$/.test(t)) | ||
| if (!failing.length || !new RegExp(`\\b${col}\\b|not_null|relationships`, "i").test(added)) continue |
There was a problem hiding this comment.
SUGGESTION: the not_null|relationships alternatives make this "added in this diff" gate vacuous
The regex is an OR tested against the joined added text, so any added line containing not_null or relationships — i.e. almost every schema.yml test edit — satisfies it for every literal-NULL column in the file. A PR that adds a not_null test to column B will therefore also report a pre-existing test on column A, which defeats anchoring this finding to the change that introduced it. Test only an escaped \b<col>\b, or resolve the added lines per model/column.
| (p) => p && typeof p.git === "string" && (p.revision === undefined || !PINNED_REVISION.test(String(p.revision))), | ||
| ) | ||
| // Only entries this PR touched: a long-standing branch pin is not this change's decision. | ||
| const touched = branchPinned.filter((p) => added.includes(p.git) || (p.revision !== undefined && added.includes(String(p.revision)))) |
There was a problem hiding this comment.
SUGGESTION: added.includes(...) matches a short revision anywhere in the diff
added is the whole concatenated added text, so added.includes(String(p.revision)) is true for a pre-existing revision: main / dev / v2 pin as soon as any unrelated added line happens to contain that substring — the long-standing pin is then reported as this PR's decision, which is what this filter exists to prevent. Check the individual addedLines(file.diff) for the git URL or an exact revision: value instead of substring-matching the joined blob.
| const oldVars = varNames(oldContent) | ||
| const newVars = varNames(newContent) | ||
| // A key still named as a string literal is usually still read (handed to a macro that reads it). | ||
| const gone = [...oldVars].filter((v) => !newVars.has(v) && !new RegExp(`['"]${v}['"]`).test(newContent)) |
There was a problem hiding this comment.
SUGGESTION: unescaped new RegExp built from file content can throw and sink the file's whole lane
v comes from var('…') in the old file, so a name with an unbalanced regex metacharacter (var('a(b'), var('a[b'), or a trailing backslash) makes new RegExp throw SyntaxError. jinjaFindings is chained with modelConfigFindings in a single task (orchestrate.ts:1766-1771), so the throw marks the model unverified and drops that whole lane's findings for the file. detectMacroPatterns and dbtConfigLane already escape content-derived names into an esc variable for exactly this hazard — reuse that idiom here (it also stops names containing | / ^ from silently over-matching and suppressing real findings).
Code Review SummaryStatus: 5 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (21 files)
Fix these issues in Kilo Cloud Previous Review Summary (commit 502f835)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit 502f835)Status: 5 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (21 files)
Reviewed by deepseek-v4.1-flash-1017 · Input: 166.6K · Output: 18.2K · Cached: 1.2M Review guidance: REVIEW.md from base branch |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
1 similar comment
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
There was a problem hiding this comment.
12 issues found across 7 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/opencode/src/altimate/review/rule-catalog.ts">
<violation number="1" location="packages/opencode/src/altimate/review/rule-catalog.ts:2266">
P2: This matcher misses common filter layouts such as one-line `JOIN ... ON ...` and predicates placed on the line after `WHERE`, so the rule silently loses intended NULL-row warnings. Match clause context across the changed SQL or carry the active clause across added lines while retaining diff anchoring.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/rule-catalog.ts:2267">
P3: The CASE guard suppresses valid filter comparisons whenever the same line contains an unrelated CASE expression. Apply the guard to the matched comparison or distinguish CASE predicates structurally instead of rejecting the whole line.</violation>
</file>
<file name="packages/opencode/src/altimate/review/orchestrate.ts">
<violation number="1" location="packages/opencode/src/altimate/review/orchestrate.ts:1165">
P2: A materialization switch expressed through `config(**cfg)` is incorrectly treated as unchanged: both raw SQLs have no literal `materialized=`, so `undefined === undefined` suppresses the new unguarded-incremental finding. Normalize both sides with `inlineConfigSplats` and treat unknown materializations as different.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/orchestrate.ts:1175">
P2: Do not suppress a `unique_key_change` merely because the base model was microbatch; a microbatch-to-merge transition makes `unique_key` active again. Gate this exception on the head strategy or explicitly handle the strategy transition.</violation>
<violation number="3" location="packages/opencode/src/altimate/review/orchestrate.ts:1175">
P2: `changesMergeGrain` misses microbatch and partition settings supplied through `config(**cfg)`, because these checks inspect raw Jinja while the linter receives expanded config. Normalize `newSql` and `oldSql` with `inlineConfigSplats` before the microbatch and partition checks.</violation>
</file>
<file name="packages/opencode/src/altimate/review/dbt-patterns.ts">
<violation number="1" location="packages/opencode/src/altimate/review/dbt-patterns.ts:169">
P2: This matcher still reports `SUM(DISTINCT ...)` and `AVG(DISTINCT ...)` as fan-out-sensitive, although duplicate join rows cannot inflate those aggregates. Exclude `DISTINCT` for all three aggregate families to avoid reintroducing the false positives this rule was tightened to remove.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/dbt-patterns.ts:169">
P2: The fan-out detector ignores row-counting aggregates such as `COUNT_IF`. Include `COUNT_IF` (and equivalent count-with-filter forms) when deciding whether a join can inflate grouped metrics.</violation>
<violation number="3" location="packages/opencode/src/altimate/review/dbt-patterns.ts:210">
P2: The self-join scanner is not quote-aware, so parentheses inside SQL string literals can hide a real self-join. Mask quoted literals before counting nesting.</violation>
<violation number="4" location="packages/opencode/src/altimate/review/dbt-patterns.ts:688">
P2: This emits a CASE-without-ELSE finding when a diff adds only the CASE argument under an unchanged `max(`, `count(`, or `coalesce(` line. Use the surrounding `newSql` context when classifying the CASE instead of relying solely on added lines.</violation>
<violation number="5" location="packages/opencode/src/altimate/review/dbt-patterns.ts:736">
P2: `detectSelfJoin` treats parentheses inside SQL string literals as scope syntax. Ignore quoted strings while scanning scopes, or a literal like `'('` suppresses a real self-join finding.</violation>
<violation number="6" location="packages/opencode/src/altimate/review/dbt-patterns.ts:739">
P2: `detectSelfJoin` misses self-joins inside nested SELECTs. Recurse into nested SELECT scopes instead of masking them without separately analyzing their joins.</violation>
<violation number="7" location="packages/opencode/src/altimate/review/dbt-patterns.ts:787">
P2: The tightened BETWEEN pattern misses quoted timestamp columns such as `"created_at"`. Allow quoted identifiers while retaining the timestamp-name restriction.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| { added: /\b[\w.]+\s*(<>|!=)\s*'/i }, | ||
| // Only a filter drops rows: in a CASE or a select-list expression `<>` yields NULL, not fewer rows. | ||
| { | ||
| added: /(^\s*(where|and|or|on|having)\b|\bwhere\b).*\b[\w.]+\s*(<>|!=)\s*'/i, |
There was a problem hiding this comment.
P2: This matcher misses common filter layouts such as one-line JOIN ... ON ... and predicates placed on the line after WHERE, so the rule silently loses intended NULL-row warnings. Match clause context across the changed SQL or carry the active clause across added lines while retaining diff anchoring.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/rule-catalog.ts, line 2266:
<comment>This matcher misses common filter layouts such as one-line `JOIN ... ON ...` and predicates placed on the line after `WHERE`, so the rule silently loses intended NULL-row warnings. Match clause context across the changed SQL or carry the active clause across added lines while retaining diff anchoring.</comment>
<file context>
@@ -2261,7 +2261,11 @@ const MISC: Rule[] = [
- { added: /\b[\w.]+\s*(<>|!=)\s*'/i },
+ // Only a filter drops rows: in a CASE or a select-list expression `<>` yields NULL, not fewer rows.
+ {
+ added: /(^\s*(where|and|or|on|having)\b|\bwhere\b).*\b[\w.]+\s*(<>|!=)\s*'/i,
+ unless: /\b(case|when|then|iff?)\b/i,
+ },
</file context>
| * model's partition columns (a coarser column derived from the row, added for pruning). | ||
| */ | ||
| function changesMergeGrain(message: string, newSql: string, oldSql: string | undefined): boolean { | ||
| if (MICROBATCH.test(newSql) || (oldSql && MICROBATCH.test(oldSql))) return false |
There was a problem hiding this comment.
P2: Do not suppress a unique_key_change merely because the base model was microbatch; a microbatch-to-merge transition makes unique_key active again. Gate this exception on the head strategy or explicitly handle the strategy transition.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/orchestrate.ts, line 1175:
<comment>Do not suppress a `unique_key_change` merely because the base model was microbatch; a microbatch-to-merge transition makes `unique_key` active again. Gate this exception on the head strategy or explicitly handle the strategy transition.</comment>
<file context>
@@ -1151,6 +1150,40 @@ export function inlineConfigSplats(sql: string): string {
+ * model's partition columns (a coarser column derived from the row, added for pruning).
+ */
+function changesMergeGrain(message: string, newSql: string, oldSql: string | undefined): boolean {
+ if (MICROBATCH.test(newSql) || (oldSql && MICROBATCH.test(oldSql))) return false
+ const m = /\[([^\]]*)\]\s*→\s*\[([^\]]*)\]/.exec(message)
+ if (!m) return true
</file context>
| if (MICROBATCH.test(newSql) || (oldSql && MICROBATCH.test(oldSql))) return false | |
| if (MICROBATCH.test(newSql)) return false |
| }) | ||
| } | ||
|
|
||
| const INFLATING_AGGREGATE = /\b(sum|avg)\s*\(|\bcount\s*\(\s*(?!distinct\b)/i |
There was a problem hiding this comment.
P2: This matcher still reports SUM(DISTINCT ...) and AVG(DISTINCT ...) as fan-out-sensitive, although duplicate join rows cannot inflate those aggregates. Exclude DISTINCT for all three aggregate families to avoid reintroducing the false positives this rule was tightened to remove.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/dbt-patterns.ts, line 169:
<comment>This matcher still reports `SUM(DISTINCT ...)` and `AVG(DISTINCT ...)` as fan-out-sensitive, although duplicate join rows cannot inflate those aggregates. Exclude `DISTINCT` for all three aggregate families to avoid reintroducing the false positives this rule was tightened to remove.</comment>
<file context>
@@ -166,10 +166,63 @@ const detectCrossJoin: Detector = (c) => {
})
}
+const INFLATING_AGGREGATE = /\b(sum|avg)\s*\(|\bcount\s*\(\s*(?!distinct\b)/i
+const NULL_CONSUMING_CALL =
+ /\b(sum|count|avg|min|max|any_value|array_agg|listagg|string_agg|count_if|bool_or|bool_and|coalesce|ifnull|nvl|nullif)\s*\(\s*(distinct\s+)?$/i
</file context>
| const INFLATING_AGGREGATE = /\b(sum|avg)\s*\(|\bcount\s*\(\s*(?!distinct\b)/i | |
| const INFLATING_AGGREGATE = /\b(?:sum|avg)\s*\(\s*(?!distinct\b)|\bcount\s*\(\s*(?!distinct\b)/i |
| // Mask nested parentheses so a subquery's FROM counts as its own scope. | ||
| let depth = 0 | ||
| let flat = "" | ||
| for (const ch of scope) { |
There was a problem hiding this comment.
P2: The self-join scanner is not quote-aware, so parentheses inside SQL string literals can hide a real self-join. Mask quoted literals before counting nesting.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/dbt-patterns.ts, line 210:
<comment>The self-join scanner is not quote-aware, so parentheses inside SQL string literals can hide a real self-join. Mask quoted literals before counting nesting.</comment>
<file context>
@@ -166,10 +166,63 @@ const detectCrossJoin: Detector = (c) => {
+ // Mask nested parentheses so a subquery's FROM counts as its own scope.
+ let depth = 0
+ let flat = ""
+ for (const ch of scope) {
+ if (ch === "(") depth++
+ if (depth === 0) flat += ch
</file context>
| // call: the `max(case when k = 'x' then v end)` pivot, `count(case when … then 1 end)`, | ||
| // `coalesce(case … end, fallback)`. Only a bare CASE leaks NULLs unintentionally. | ||
| const bare = [...joined.matchAll(/\bcase\b[\s\S]*?\bend\b/gi)].find( | ||
| (m) => !/\belse\b/i.test(m[0]) && !NULL_CONSUMING_CALL.test(joined.slice(0, m.index)), |
There was a problem hiding this comment.
P2: This emits a CASE-without-ELSE finding when a diff adds only the CASE argument under an unchanged max(, count(, or coalesce( line. Use the surrounding newSql context when classifying the CASE instead of relying solely on added lines.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/dbt-patterns.ts, line 688:
<comment>This emits a CASE-without-ELSE finding when a diff adds only the CASE argument under an unchanged `max(`, `count(`, or `coalesce(` line. Use the surrounding `newSql` context when classifying the CASE instead of relying solely on added lines.</comment>
<file context>
@@ -628,8 +681,13 @@ const detectHavingNoGroupBy: Detector = (c) => {
+ // call: the `max(case when k = 'x' then v end)` pivot, `count(case when … then 1 end)`,
+ // `coalesce(case … end, fallback)`. Only a bare CASE leaks NULLs unintentionally.
+ const bare = [...joined.matchAll(/\bcase\b[\s\S]*?\bend\b/gi)].find(
+ (m) => !/\belse\b/i.test(m[0]) && !NULL_CONSUMING_CALL.test(joined.slice(0, m.index)),
+ )
+ if (!bare) return null
</file context>
| ) | ||
| const dup = refs.find((r, i) => refs.indexOf(r) !== i) | ||
| if (!dup) return null | ||
| if (!selectScopes(sql).some(scopeJoinsItself)) return null |
There was a problem hiding this comment.
P2: detectSelfJoin misses self-joins inside nested SELECTs. Recurse into nested SELECT scopes instead of masking them without separately analyzing their joins.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/dbt-patterns.ts, line 739:
<comment>`detectSelfJoin` misses self-joins inside nested SELECTs. Recurse into nested SELECT scopes instead of masking them without separately analyzing their joins.</comment>
<file context>
@@ -673,11 +731,12 @@ const detectNaturalJoin: Detector = (c) => {
)
- const dup = refs.find((r, i) => refs.indexOf(r) !== i)
- if (!dup) return null
+ if (!selectScopes(sql).some(scopeJoinsItself)) return null
return pattern(
c,
</file context>
| * model's partition columns (a coarser column derived from the row, added for pruning). | ||
| */ | ||
| function changesMergeGrain(message: string, newSql: string, oldSql: string | undefined): boolean { | ||
| if (MICROBATCH.test(newSql) || (oldSql && MICROBATCH.test(oldSql))) return false |
There was a problem hiding this comment.
P2: changesMergeGrain misses microbatch and partition settings supplied through config(**cfg), because these checks inspect raw Jinja while the linter receives expanded config. Normalize newSql and oldSql with inlineConfigSplats before the microbatch and partition checks.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/orchestrate.ts, line 1175:
<comment>`changesMergeGrain` misses microbatch and partition settings supplied through `config(**cfg)`, because these checks inspect raw Jinja while the linter receives expanded config. Normalize `newSql` and `oldSql` with `inlineConfigSplats` before the microbatch and partition checks.</comment>
<file context>
@@ -1151,6 +1150,40 @@ export function inlineConfigSplats(sql: string): string {
+ * model's partition columns (a coarser column derived from the row, added for pruning).
+ */
+function changesMergeGrain(message: string, newSql: string, oldSql: string | undefined): boolean {
+ if (MICROBATCH.test(newSql) || (oldSql && MICROBATCH.test(oldSql))) return false
+ const m = /\[([^\]]*)\]\s*→\s*\[([^\]]*)\]/.exec(message)
+ if (!m) return true
</file context>
| const outsideJinja = newSql.replace(/\{\{[\s\S]*?\}\}|\{%[\s\S]*?%\}|\{#[\s\S]*?#\}/g, " ") | ||
| if (!/\bselect\b/i.test(outsideJinja)) return false | ||
| if (oldSql === undefined) return true | ||
| const sameMaterialization = MATERIALIZED.exec(oldSql)?.[1]?.toLowerCase() === MATERIALIZED.exec(newSql)?.[1]?.toLowerCase() |
There was a problem hiding this comment.
P2: A materialization switch expressed through config(**cfg) is incorrectly treated as unchanged: both raw SQLs have no literal materialized=, so undefined === undefined suppresses the new unguarded-incremental finding. Normalize both sides with inlineConfigSplats and treat unknown materializations as different.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/orchestrate.ts, line 1165:
<comment>A materialization switch expressed through `config(**cfg)` is incorrectly treated as unchanged: both raw SQLs have no literal `materialized=`, so `undefined === undefined` suppresses the new unguarded-incremental finding. Normalize both sides with `inlineConfigSplats` and treat unknown materializations as different.</comment>
<file context>
@@ -1151,6 +1150,40 @@ export function inlineConfigSplats(sql: string): string {
+ const outsideJinja = newSql.replace(/\{\{[\s\S]*?\}\}|\{%[\s\S]*?%\}|\{#[\s\S]*?#\}/g, " ")
+ if (!/\bselect\b/i.test(outsideJinja)) return false
+ if (oldSql === undefined) return true
+ const sameMaterialization = MATERIALIZED.exec(oldSql)?.[1]?.toLowerCase() === MATERIALIZED.exec(newSql)?.[1]?.toLowerCase()
+ return !(sameMaterialization && !/\bis_incremental\s*\(/.test(oldSql))
+}
</file context>
| const sameMaterialization = MATERIALIZED.exec(oldSql)?.[1]?.toLowerCase() === MATERIALIZED.exec(newSql)?.[1]?.toLowerCase() | |
| const oldMaterialization = MATERIALIZED.exec(inlineConfigSplats(oldSql))?.[1]?.toLowerCase() | |
| const newMaterialization = MATERIALIZED.exec(inlineConfigSplats(newSql))?.[1]?.toLowerCase() | |
| const sameMaterialization = oldMaterialization !== undefined && oldMaterialization === newMaterialization |
| // Only a filter drops rows: in a CASE or a select-list expression `<>` yields NULL, not fewer rows. | ||
| { | ||
| added: /(^\s*(where|and|or|on|having)\b|\bwhere\b).*\b[\w.]+\s*(<>|!=)\s*'/i, | ||
| unless: /\b(case|when|then|iff?)\b/i, |
There was a problem hiding this comment.
P3: The CASE guard suppresses valid filter comparisons whenever the same line contains an unrelated CASE expression. Apply the guard to the matched comparison or distinguish CASE predicates structurally instead of rejecting the whole line.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/rule-catalog.ts, line 2267:
<comment>The CASE guard suppresses valid filter comparisons whenever the same line contains an unrelated CASE expression. Apply the guard to the matched comparison or distinguish CASE predicates structurally instead of rejecting the whole line.</comment>
<file context>
@@ -2261,7 +2261,11 @@ const MISC: Rule[] = [
+ // Only a filter drops rows: in a CASE or a select-list expression `<>` yields NULL, not fewer rows.
+ {
+ added: /(^\s*(where|and|or|on|having)\b|\bwhere\b).*\b[\w.]+\s*(<>|!=)\s*'/i,
+ unless: /\b(case|when|then|iff?)\b/i,
+ },
{ add: "where status <> 'cancelled'" },
</file context>
hkaltai
left a comment
There was a problem hiding this comment.
Review panel: Claude, Codex, and lead reviewer (all 22 changed files covered; the review suite passes: ~490-504 tests).
Verdict: request changes. 5 MAJOR, 3 MINOR inline (more MINOR-level items listed below), plus nits.
Consensus (Claude and Codex both, lead reproduced): gate softening from merge-before-verdict, profiles.yml sent to the AI lane. Reproduced by lead: versioned refs false positive, working-tree missing-ref false positive, NULL-column false positive. Claude-only and reproduced: the test-removal movedTo suppression.
Counts: CRITICAL 0, MAJOR 5, MINOR 3 inline + 9 in the list below, NIT 4+.
Direction and most of the precision work are good: structural detectors, :(top) pathspecs, paired fire/silent tests.
Further MINOR items and Nits
project-rules.ts:165BLOCK_OPEN lackswith/block/autoescape, so{% with a = 1 %}...{% endwith %}reports an unmatchedendwith(MINOR-level false positive; add the tags or ignore untrackedend*).project-rules.ts:176any=marks a{% set %}as inline, so{% set q | replace("x","a=b") %}..{% endset %}is flagged; detect=outside strings.incremental-semantics.ts:154predicate extraction omitsONdespite its doc, so an incremental-only join predicate change (>to>=) is not seen.rule-catalog.ts:435duplicate-WHEN normalisation lowercases string literals ('A'vs'a'flagged duplicate).rule-catalog.ts:407stripping qualifiers conflatesa.idandb.idin the repeated-partition-column rule.rule-catalog.ts:542ignore-nulls-inside-callhasworksOn: ["bigquery"]; DuckDB acceptsIGNORE NULLSin window functions.rule-catalog.tshardcoded-year-filterregex has no filter context (case when fiscal_year = 2024matches); anchor on where/and/on/having.orchestrate.ts:1275uniqueKeyMismatchLaneis not scoped to the diff, so a pre-existing mismatch re-fires on any PR touching the model.project-rules.tsvar-renamed-no-fallbacktreats any removed var plus any added var as a rename;incremental-filter-created-onlybody wording is wrong when onlyunique_keyheld.- Orphaned doc comments:
orchestrate.ts:~1084(dbt CONFIG lint abovedefinedVars),~1229(abovejoinKeyUniquenessLaneinstead ofuniqueKeyMismatchLane),project-rules.ts:~475,dbt-patterns.ts:~918. - Naming:
projectResourceNamesandliteralNullColumnsoverstate certainty;AiReviewFile.sqlnow holds yml/CSV (content);FEASIBLEin rule-catalog is unclear; package-pin/version rules use categorymaterialization. - Missing tests: verdict after merging, profiles exclusion, versioned refs, working-tree names, null-column UNION case.
| // project that runs on one known engine. Core's dialect-aware check (a | ||
| // function missing on THIS engine) is a real error and is kept. | ||
| if (input.suppressPortability) out = out.filter((f) => !isSuppressiblePortability(f, dialect, input.portabilityTargets)) | ||
| out = mergeRepeats(anchorToDiff(out, input.changedFiles)) |
There was a problem hiding this comment.
[MAJOR] Merging repeats before the verdict softens the gate
out = mergeRepeats(anchorToDiff(out, input.changedFiles))
cleanFindings feeds both rendering and computeIdealVerdict/counts. computeIdealVerdict returns REQUEST_CHANGES at warningPatternThreshold (default 3) warnings, but merging folds same-rule warnings in one file into one. Repro: three high-confidence altimate_core.check warnings in one model that differ only in the backticked column give REQUEST_CHANGES unmerged and COMMENT merged, so a PR that blocked in gate mode now passes. Merging also keeps only the first body, so later instances lose their evidence.
Fix: compute verdict and counts from the unmerged list and merge only for presentation (or carry an instance count into the verdict). Add a verdict test.
| import { dbtProjectFindings, jinjaFindings, joinTargets, missingRefFindings, modelConfigFindings, packagesFindings, testsOnNullColumns } from "./project-rules" | ||
| import { type AiLaneStatus, type AiReviewFile, type AiReviewInput, type AiReviewResult } from "./ai-review" | ||
|
|
||
| const AI_SUPPORTING_KINDS = new Set(["macro", "test", "snapshot", "schema_yml", "source_yml", "seed", "project_config"]) |
There was a problem hiding this comment.
[MAJOR] profiles.yml (and other project_config) is sent in full to the AI model
const AI_SUPPORTING_KINDS = new Set([..., "seed", "project_config"])
project_config includes profiles.yml (diff-filter.ts:122). The AI lane now calls getContent(path, "new") and sends the whole file plus its diff. A repo with a committed profiles.yml containing password:/token:/keyfile_json ships those credentials to the model provider whenever a PR touches it. Reproduced with a sentinel password reaching the AI callback unchanged.
Fix: exclude profiles.yml from AI inputs; ideally send only diffs for project_config and redact secret-like keys. Add a test.
| } | ||
| const oldCols = cols(oldSet) | ||
| const newCols = cols(newSet) | ||
| const movedTo = (model: string, column: string, testField: string) => |
There was a problem hiding this comment.
[MAJOR] Test-removal 'moved' heuristic hides real removals
const movedTo = (model: string, column: string, testField: string) =>
!!column && !newCols.get(model)?.has(column) && ...
oldCols/newCols are derived from the set of test occurrences, not from columns declared in the yml, so 'column gone' really means 'column has no test any more'. Repro: customer_id keeps its yml entry but loses not_null, and the same PR adds discount_code: [not_null]. The removal is suppressed (result []); without the new column it is reported.
Fix: take column existence from parsed columns[].name; treat as moved only when the old column is absent from the yml (ideally exactly one new column took the test).
| const refs = new Set<string>() | ||
| for (const l of addedLines(file.diff)) | ||
| for (const m of l.matchAll(/\bref\s*\(\s*['"]([^'"]+)['"]\s*(?:,\s*v(?:ersion)?\s*=\s*[^)]*)?\)/g)) refs.add(m[1]) | ||
| const missing = [...refs].filter((r) => !names.has(r.toLowerCase()) && !names.has(r.toLowerCase().replace(/_v\d+$/, ""))) |
There was a problem hiding this comment.
[MAJOR] Versioned models reported as missing refs
const missing = [...refs].filter((r) => !names.has(r.toLowerCase()) && !names.has(r.toLowerCase().replace(/_v\d+$/, "")))
The _v\d+ strip is applied to the ref name, but versioned models live in dim_customers_v1.sql/_v2.sql with no unversioned file. With names {dim_customers_v1, dim_customers_v2}, both ref('dim_customers') and ref('dim_customers', v=2) warn 'points at no model' (reproduced), claiming the project fails to parse. The existing test passes only because its fixture includes the unversioned name.
Fix: when building names, also add each name with _v\d+ stripped; read versions[].defined_in and yml model names. Test with only _v1/_v2 files.
| dbtRoot: string, | ||
| ): Promise<Set<string> | undefined> { | ||
| if (projectRootRel === undefined) return undefined | ||
| const files = await listFilesAt(cwd, head) |
There was a problem hiding this comment.
[MAJOR] Resource names ignore uncommitted files in working-tree reviews
const files = await listFilesAt(cwd, head)
When head is undefined the review diffs the working tree, but resource names come from the tree at HEAD, so a new, uncommitted model is invisible. Repro: staged models/new_model.sql gives names ["a"]; any changed model with ref('new_model') gets a warning that the project fails to parse.
Fix: for head === undefined use git ls-files --cached --others --exclude-standard, or add basenames of non-deleted changed files. The YAML-snapshot and projectAdapters lookups have the same HEAD-only read.
| for (const c of m.columns) { | ||
| const col = String(c?.name ?? "").toLowerCase() | ||
| if (!nulls.has(col)) continue | ||
| const failing = [...testNames(c.data_tests), ...testNames(c.tests)].filter((t) => /(^|\.)(not_null|relationships)$/.test(t)) |
There was a problem hiding this comment.
[MINOR] testsOnNullColumns false positives
const failing = [...].filter((t) => /(^|\.)(not_null|relationships)$/.test(t))
(a) relationships ignores NULLs in dbt, so 'fails on every row' is wrong for it. (b) literalNullColumns scans the whole model, so a null as discount in one UNION branch or CTE marks the column always-NULL even if other branches fill it (repro: select id, discount from a union all select id, null as discount from b). (c) The added-line check \bcol\b|not_null|relationships fires on any added not_null anywhere in the file.
Fix: drop relationships; flag only columns NULL in the final SELECT and abstain with UNION/later references; scope the added-line check to the column's block.
| // coerce, a function it doesn't know, or syntax it can't parse is a gap in the engine, | ||
| // not an error in the model. Unresolved columns and tables are kept. | ||
| const msg = String(issue.message ?? "") | ||
| if (rule === "validate" && (ENGINE_LIMIT_MESSAGE.test(msg) || /^syntax error/i.test(msg))) continue |
There was a problem hiding this comment.
[MINOR] All 'Syntax error' validate results are dropped
if (rule === "validate" && (ENGINE_LIMIT_MESSAGE.test(msg) || /^syntax error/i.test(msg))) continue
This hides genuine syntax regressions (e.g. select (id from t) for raw SQL where dbt compile did not run, and the neighbouring comment says unresolved items are kept. If it is a deliberate precision trade-off, restrict it to Jinja-templated/compiled-failure cases or surface it as an unverified-analysis suggestion.
| * test, one per join onto the same kind of key) into a single finding that | ||
| * lists every instance. Critical findings stay separate: each one can block. | ||
| */ | ||
| export function mergeRepeats(findings: Finding[]): Finding[] { |
There was a problem hiding this comment.
[MINOR] Merged finding id is unstable and project-rule fingerprints collide
export function mergeRepeats(findings: Finding[]): Finding[] {
A merged finding keeps first.id (the lowest-sorted instance); if that instance is fixed later the id changes and the inline comment is re-posted. Separately, project-rules findings in one file share one fingerprint (e.g. two missing refs x and y), so dedupe() drops one before merging.
Fix: derive the merged id from the group key, and include the affected resource/macro/column in project-rule fingerprints.
- `var_no_default`: skip vars defined in `dbt_project.yml` (top level or project-scoped) or defaulted elsewhere in the same model - `nested-subqueries-deep`, `nested-case`: count real nesting with a paren / CASE stack instead of counting occurrences (sibling CTEs, two CASEs) - `case-when-no-end`: check the whole model, abstain on Jinja branches - `select-star-in-cte-final`: fire only when the final SELECT * reads straight from another relation, not from a CTE - partition-pruning: only flag a function that wraps a column, not a literal, a var or the clock - schema.yml test removal: tests moved to a renamed column are not removals - incremental filter change: compare predicates of the `is_incremental()` branch only, not select-list edits or the else branch - grain not_null finding is a suggestion with accurate wording Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- monorepo: per-file hunk diffs came back empty when the dbt project sits in a
subdirectory (`--name-status` paths are repo-root relative, the pathspec was
resolved against the project dir), so every diff-anchored rule and the risk
tier saw 0 changed lines; anchor pathspecs with `:(top)`
- join rules (`join-on-cast`, `join-or-condition`) match each JOIN's whole ON
clause, so a predicate on a continuation line counts; `join-on-cast` also
covers `date()`/`timestamp()`/`lower()` and similar around a column
- `config(**cfg)` with a literal `{% set cfg = {...} %}` dict is inlined before
the config lint, so incremental/materialization rules see it
- changed macros are reviewed: catalog rules marked `macros` and `var()` read
with no default
- a column removed from a final-layer model (marts, reporting, dimensional,
extracts) with no declared consumers is a suggestion
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eclared uniqueness test - `runner.ts`: `declaredUniqueKeys(model)` reads `unique` / `unique_combination_of_columns` tests attached to the model in the manifest - `orchestrate.ts`: `uniqueKeyMismatchLane` warns when an incremental model's `unique_key` matches none of its declared uniqueness keys - tests for match, mismatch, case-insensitivity and non-incremental models Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er a precision sample Measured on 423 replayed public dbt PRs: the modified-model fan-out finding was 1,717 of 7,820 findings (22%) and matched none of the 391 real reviewer-found defects. - `orchestrate.ts`: a modified model's downstream count no longer emits a finding; it already drives risk tiering and appears in the tier reasons. Deleted or renamed models with consumers still report the break. - `rule-catalog.ts` `join-or-condition`: skip an OR whose disjunct is an IS [NOT] NULL guard (open-ended interval joins, null-safe keys). A sample of 8 hits had 6 such guards. The body now names the duplicate-row risk the human reviewers raised. - `orchestrate.ts`: intermediate/staging models nested under a final-layer folder are not treated as final-layer for the removed-column suggestion. - tests for each Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e advisory AI lane The AI lane only received changed model SQL. In a labeled sample of 206 defects that only judgment can catch, 104 were in other files (82 macros, 15 yml, 7 singular tests), and a macro-only PR skipped the lane with "no changed model SQL to review". - `orchestrate.ts`: changed macros, singular tests, snapshots, schema/source yml, seeds and project config join the lane's input with their current content (seeds: diff only) - `ai-review.ts`: each file carries its dbt `kind`; `selectAiFiles` puts models first, then macros, tests, snapshots, yml, seeds, config under the 20-file cap; headers name the role and content is fenced by type; skip reason is now "no changed dbt files to review" - tests for the inputs, ordering, cap and prompt layout; docs note what the lane sees Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hape On the 377-PR compile-mode replay the check fired 41 times across 27 models; most were a declared key plus a partition column (`block_month`, `effective_date`). - silent when the merge key is a declared key plus only date/partition columns - `suggestion` when the merge key is wider than a declared key (duplicates only if an extra column changes for an existing row); `warning` when it is narrower or a different key - tests for each shape Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ublic dbt PR replay Ranked by volume on 413 compile-mode replays and checked against which findings matched any of 391 real reviewer-found defects: - engine check: injection rules (`union_injection`, `comment_injection`, `unbalanced_quote`) are skipped in models; `validate` errors from the engine's planner (planning/execution/not implemented) are skipped and parser syntax errors are suggestions; unresolved columns stay - undecidable equivalence is a suggestion to data-diff instead of a warning (426 hits, 1 partial match) - `union-select-star` needs a UNION branch that is `select *` - `order-by-in-cte` needs an ORDER BY at the CTE's top level that is not a top-N - `unqualified-column-multi-table` needs a changed bare-column line and a joining SELECT block with a bare column - `ref-double-quotes` removed: quoting style only - window without PARTITION BY reads the whole OVER clause, so a PARTITION BY on the next line counts - new join before an aggregate is a suggestion (1 partial match in 254) - tests for each shape Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Written from the defects in a 391-defect replay corpus that a deterministic rule can catch. On their own commits (no AI, no manifest), 50 of the 59 targeted defects now produce the matching finding; the join-key, missing-ref and table-`unique_key` rules need a manifest or installed packages and are measured on the full replay. New module `project-rules.ts` (files outside the SELECT): - packages.yml: a dependency this PR points at a git branch instead of a release/tag/sha - dbt_project.yml: version going down, a duplicate grantee, newly disabled resources - Jinja: unbalanced blocks (stray `endif`, missing `endmacro`, `macro x` without `()`), macro parameters moved position, a var() key renamed without the old key as fallback, code its own comment marks temporary, naive `datetime.utcnow()` - model config: `unique_key` on a table/view, BigQuery microbatch `batch_size` vs partition granularity, partition column missing from the merge key, an incremental filter that never constrains the partition column, a creation-time filter on a model whose rows change - `ref()` to a name no model/seed/snapshot carries (repo + installed packages; suggests a near match) - a new join whose key no uniqueness test on the joined model covers (manifest) - a not_null/relationships test on a column the model sets to a literal NULL Catalog: LEFT JOIN filtered in WHERE, case-folding one side of a join, null-safe comparison replaced by `=`, window ORDER BY repeating the partition key, duplicate CASE WHEN, QUALIFY / in-call IGNORE NULLS / MIN_BY-MAX_BY portability, async ClickHouse mutations, language-dependent DATENAME, hardcoded year filters. Also: portability advice is skipped in an adapter's own macro variant when that adapter supports the construct; the config dict-splat inliner handles nested dicts; `run.ts` lists resource names at head for the ref check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Join-key rule: only a LEFT JOIN onto a model with a declared uniqueness test whose key the join doesn't cover (an SCD2 dimension without `is_current`, a natural key instead of the declared one). Inner joins, models with no declared key, range / `OR` / Jinja-generated ON conditions are skipped; keys inside `cast(...)` / function wrappers are read. - Portability: a project whose own macros carry adapter variants (`postgres__x`, `snowflake__x`, ...) is judged against each of those adapters, so `min_by` in a package that supports Postgres is reported even when this checkout runs on DuckDB. `presto__` counts as Trino. - Missing `ref()`: snapshots defined only in YAML (dbt 1.9+) count as targets. - `var()` rename: skipped when the old key is still passed as a string literal (handed to a macro that reads it). - `grepMatchesAt` git helper; paths are repo-root relative. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the summary - `mergeRepeats`: the same rule several times in one file (one finding per column of a grain test, one per join onto the same kind of key) becomes one finding whose body lists each instance. Critical findings stay separate. - `ruleTitle` moves to `finding.ts` and strips any leading `<model or file>: `, so yml-based findings group across files too. - The summary groups a rule on two or more models into one bullet (was three). On the public dbt PR replay this takes findings per PR from 17.0 to 15.9 and summary bullets per PR from 9.1 to 8.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lane output The AI prompt lists the top 40 deterministic findings as "do not repeat". It received them before dedupe, rubric exclusions, off-engine portability suppression and repeat merging, so dropped advice and per-column repeats could fill those slots. - `cleanFindings` in `orchestrate.ts`: dedupe, exclusions, portability, diff anchoring, `mergeRepeats`, most severe first. The AI lane gets this list; the final findings are the same list after the severity threshold. - test: suppressed portability advice never reaches the AI, and its grounding matches what the reader sees Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… actually risky A blind judge of 150 sampled findings marked these rules wrong most often on public dbt PRs: - `new join before an aggregate`: only SUM, AVG and plain COUNT can inflate; MIN/MAX and `COUNT(DISTINCT)` can't - `CASE without ELSE`: a CASE inside an aggregate or `coalesce`/`nullif` (pivots, conditional counts) means NULL on purpose - `self-join`: one relation read twice in the same FROM … JOIN chain, not the same ref in two CTEs or UNION branches - `BETWEEN on a timestamp`: needs a timestamp-named column; on a DATE column BETWEEN is the right range - `<>/!= excludes NULL rows`: only in a filter (WHERE/AND/OR/ON/HAVING), not in a CASE or the select list - `validate`: a manifest schema lists only documented columns, so "column not found" is dropped unless the schema came from a catalog (as "table not found" already was); parser syntax errors on compiled models are dropped too - `incremental no guard`: skipped when a macro generates the model's SQL, or when the model was already unguarded before the PR - `unique key change`: skipped for microbatch, and when the only added key columns are the model's partition columns Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lane, rule precision, lazy native load - Never send `profiles.yml`/`profiles.yaml` (warehouse credentials) to the AI lane, diff or content - `mergeRepeats`: list each same-titled instance with its own message and line instead of dropping them - `projectResourceNames`: in working-tree mode include index and untracked files, so a new uncommitted model is refable - `testsOnNullColumns`: flag only `not_null` (dbt `relationships` skips NULL keys), and only when an added line falls inside that column's YAML node - `left-join-filtered-in-where`: require an added line to reference the LEFT JOIN's right side - `case-when-no-end`: abstain only when a CASE/END sits inside a Jinja `if`/`for` block, not on any Jinja branch in the file - Move `isExistenceError` to the dependency-free `native/existence-error.ts` so `runner.ts` no longer loads the NAPI binding eagerly Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7da6850 to
e77d121
Compare
3d72154 to
bb6a1fe
Compare
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
1 similar comment
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
Thanks for updating your PR! It now meets our contributing guidelines. 👍 |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
Issue for this PR
Closes #1792
Type of change
What does this PR do?
Stacked on #1726. This PR covers precision fixes and new rules for the no-AI review lanes, measured by replaying public dbt PRs that had defects found by human reviewers.
How it was measured
Corpus: 391 defects that human reviewers caught on public OSS dbt PRs, replayed with
altimate reviewin compile mode on about 377–423 PRs.A judge labelled each defect caught, partial or missed against the findings.
Each defect was also tagged by what can reach it:
Results (no AI lane)
What changed
False positives and precision
var_no_default, nesting counts,case-when-no-end,select-star-in-cte-final, partition pruning, test removal on renamed columns, and incremental filter changes.union-select-star,order-by-in-cte,unqualified-column-multi-tableandjoin-or-conditionref-double-quotesis removedunique_keyvs declared uniqueness test check is graded by key shape.Rules that should have fired but didn't
:(top).config(**cfg)is inlined before the config lint.New rules (
project-rules.tsplus additions to the catalog)packages.yml: a dependency pinned to a branch.dbt_project.yml: the version going down, a duplicate grantee, or resources newly disabled.var()renamed without the old key as a fallback, andutcnow().unique_keyon a table or viewbatch_sizevs the partition granularityref()to a name that doesn't exist. YAML snapshots and installed packages count, and a near match is suggested.is_current.postgres__x,snowflake__x, ...).PR comment
AI lane input
How did you verify your code works?
tscis clean, apart from existingtui/errors.Cost side effects
No new model calls, warehouse queries or cloud resources; the new rules are deterministic and local. When the AI lane is enabled, it now also receives changed macros, singular tests, snapshots, schema/source yml, seeds and project config, so its input tokens grow with the number of changed non-model files, within the existing file-count and per-file character caps. The number of AI calls is unchanged.
Screenshots / recordings
Not a UI change.
Checklist
🤖 Generated with Claude Code
Summary by cubic
Fixes false positives and adds rules to the no-AI review lanes, measured by replaying 391 defects human reviewers caught on public dbt PRs. The rule set now catches 51 of the 53 defects it targets, and the modified-model fan-out finding (22% of all findings, matched none of the defects) is removed.
Precision and noise
var_no_default, nesting counts,case-when-no-end,select-star-in-cte-final, partition pruning, test removal on renamed columns, and incremental filter changes.BETWEENonly on timestamp columns,<>only in filters, and skipped incremental-guard and unique-key changes when a macro generates the SQL or only partition columns were added.ref-double-quotesis removed.config(**cfg)is inlined, changed macros are reviewed, and pathspecs are anchored with:(top)so monorepo hunk diffs aren't empty.unique_keythat matches none of the model's declared uniqueness tests is graded by key shape; a column removed from a final-layer model with no declared consumers becomes a suggestion.New rules
packages.yml: dependency pinned to a branch.dbt_project.yml: version going down, duplicate grantee, newly disabled resources.var()without a fallback,utcnow().unique_keyon a table/view, BigQuery microbatchbatch_size, partition column missing from the merge key, an incremental filter that never filters on the partition column.ref()to a nonexistent name; a LEFT JOIN onto a model whose declared unique key the join doesn't cover; portability judged against every adapter the project's own macros target.Written for commit bb6a1fe. Summary will update on new commits.