Repository navigation
feat(review): opt-in test-gap check and focused AI batches; verifier reads every changed file - #1772
RaghwendraSingh wants to merge 7 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
| const others = [...new Set([...newContent.matchAll(DOC_REF)].map((m) => m[1]))].filter((n) => !missing.includes(n)).slice(0, 5) | ||
| let resolves = false | ||
| for (const n of others) if ((resolves = await docExists(n))) break | ||
| if (!others.length && projectHasDocs) resolves = await projectHasDocs() |
There was a problem hiding this comment.
WARNING: projectHasDocs() cannot tell a project docs block from one that resolves in an installed package
docExists only greps the dbt project's own paths (grepProject pathspecs come from the dbt_project.yml model/macro/seed/snapshot/test dirs, run.ts:431-438), so a doc('name') defined by an installed package already counts as "missing". This new fallback then treats any docs block anywhere in the project as proof that the name resolves, so for a file whose only doc() is the new one (typically a newly added schema yml) a package-provided doc is reported as "dbt fails to parse" — exactly the package case the guard above was added to avoid. The finding body also says "the other doc() references in this file resolve there", which is false on this branch (there are none). Consider also searching installed-package .md files before concluding the name is missing, or keep the previous "another doc() in this file resolves" requirement for names that could come from a package.
| // The project defines docs blocks of its own (any `{% docs x %}` in its .md files). | ||
| let docsAny: Promise<boolean> | undefined | ||
| const projectHasDocs = () => | ||
| (docsAny ??= (input.grepProject?.("docs", "*.md") ?? Promise.resolve([])) |
There was a problem hiding this comment.
SUGGESTION: the generic docs pattern can exhaust the search cap before reaching a docs block
grepFixedAt stops after 50 total hits (git.ts:194-210; -m 50 per file), and docs shows up in ordinary markdown prose, so those hits can be used up before the file holding {% docs ... %} is scanned. projectHasDocs then returns false and the new doc-block-missing finding stays silent — the recall gap this change targets. A block-specific fixed string (e.g. separate searches for {% docs and {%- docs) would keep unrelated prose from diluting the result.
| const TEST_GAP_FILES_MAX = 40 | ||
|
|
||
| /** The PR's other changed paths, appended to a batch message for the test-gap instruction. */ | ||
| export function testGapFileList(message: string, batch: AiReviewFile[], prFiles: string[] = []): string { |
There was a problem hiding this comment.
SUGGESTION: three copies of the same "other PR paths" renderer
This function, the focused-batch ## OTHER FILES CHANGED IN THIS PR list in buildBatchMessage (lines 777-782), and buildVerifyMessage's "Files this PR changes" list (ai-verify.ts:386-402) each implement slice(0, 40) + "… and N more" with their own 40 constant (TEST_GAP_FILES_MAX, FOCUS_OTHER_FILES, PR_FILES_MAX). With ALTIMATE_REVIEW_AI_FOCUSED=1 and ALTIMATE_REVIEW_AI_TEST_GAP=1, the same batch message carries both lists and repeats every other path in the prompt. A single shared helper (and cap) would remove the duplication and the double listing.
| * be dropped, or null to keep it. Centralizing this here (vs prompt text) is | ||
| * what keeps the false-positive rate down deterministically. | ||
| */ | ||
| /** Findings that break `dbt parse` for the whole project, so a dev/sandbox/scratch folder doesn't excuse them. */ |
There was a problem hiding this comment.
SUGGESTION: the exclusionReason JSDoc now documents this constant
The /** Apply the rubric's exclusion predicates ... */ block above now attaches to PROJECT_BREAKING_RULES, so exclusionReason loses its doc and editors/TS show the wrong description for it. Move the constant above the JSDoc, or give the constant its own comment.
Code Review SummaryStatus: 5 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (17 files)
Fix these issues in Kilo Cloud Previous Review Summary (commit 1acd5d8)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit 1acd5d8)Status: 4 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (16 files)
Reviewed by deepseek-v4.1-flash-1017 · Input: 223K · Output: 38.8K · Cached: 4.3M Review guidance: REVIEW.md from base branch |
There was a problem hiding this comment.
10 issues found across 16 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/opencode/src/altimate/review/run.ts">
<violation number="1" location="packages/opencode/src/altimate/review/run.ts:440">
P2: `prFiles` includes deleted files, so a PR that removes a test file still shows that test under "PR's other changed paths" and the test-gap check may treat it as coverage. Filter out `status === "deleted"` before mapping to paths.</violation>
</file>
<file name="packages/opencode/test/altimate/review-ai-test-gap.test.ts">
<violation number="1" location="packages/opencode/test/altimate/review-ai-test-gap.test.ts:29">
P3: This test depends on the real native `altimate_core.review_ai_prompt` call, but it never stubs `Dispatcher.call`. If the native core is unavailable, `runAiReview` returns early and `LLM.stream` is never called, so the test fails on an undefined `off`. Stub it with `spyOn(Dispatcher, "call").mockResolvedValue({ success: true, data: { prompt: "CORE" } } as any)`, as the replay test does.</violation>
</file>
<file name="packages/opencode/src/altimate/review/ai-review.ts">
<violation number="1" location="packages/opencode/src/altimate/review/ai-review.ts:117">
P2: For PRs with more than 40 other paths, later test-file names disappear from the reviewer’s prompt, so an unselected test can be missed and trigger a false test-gap comment. Preserve test-like paths beyond the cap or let the reviewer enumerate all PR paths.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/ai-review.ts:579">
P2: With `ALTIMATE_REVIEW_AI_MAX_BATCHES` between 0 and 1 (e.g. 0.5), `Array.from({ length: maxBatches })` creates no bins and the `reduce` throws, and `runAiReview` rejects instead of returning a status. Clamp the bin count to a positive integer.</violation>
</file>
<file name="packages/opencode/src/altimate/review/dbt-patterns.ts">
<violation number="1" location="packages/opencode/src/altimate/review/dbt-patterns.ts:933">
P2: A `join ... using (...)` has no ON clause, but this change does not step over it, so the join chain still breaks there and a redundant predicate on the next join is missed. Let the bare-join match also consume a `USING (...)` list.</violation>
</file>
<file name="packages/opencode/src/altimate/review/ai-verify.ts">
<violation number="1" location="packages/opencode/src/altimate/review/ai-verify.ts:395">
P2: The verifier cannot discover changed files after the first 40 paths, so a test added at path 41+ can be missed by the test-gap check; add a bounded file-list tool or otherwise expose all paths to the verifier.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/ai-verify.ts:783">
P2: The verifier's read allowlist now includes every PR file, including paths the user excluded with `exclude` or rubric `excludeGlobs` and out-of-project files. `prFiles` comes from the unfiltered `collected` list in run.ts, so those contents can reach the model provider. Build `prFiles` from the reviewable set after exclusions, or filter the allowlist with the same exclusion rules.</violation>
</file>
<file name="packages/opencode/src/altimate/review/orchestrate.ts">
<violation number="1" location="packages/opencode/src/altimate/review/orchestrate.ts:2287">
P2: The generic `docs` search can exhaust `grepFixedAt`’s 50-hit limit on Markdown prose before reaching a docs block, causing `projectHasDocs()` to suppress missing-`doc()` findings. Use a targeted, complete docs-block search instead.</violation>
</file>
<file name="packages/opencode/src/altimate/review/rule-catalog.ts">
<violation number="1" location="packages/opencode/src/altimate/review/rule-catalog.ts:775">
P3: The lookbehind exempts any seam character preceded by a backslash, so a literal ending in `\\n` (escaped backslash plus `n`) also escapes detection even though rendered SQL still ends in a word character. Check backslash parity before suppressing the match.</violation>
</file>
<file name="packages/opencode/src/altimate/review/rubric.ts">
<violation number="1" location="packages/opencode/src/altimate/review/rubric.ts:79">
P3: Move `PROJECT_BREAKING_RULES` above the `exclusionReason` JSDoc or give the constant its own comment so the predicate documentation remains attached to the function.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| const specs = glob ? [`${root}${glob}`] : dirs.map((d) => path.posix.normalize(root + d)) | ||
| return grepFixedAt(gitRoot, opts.head, pattern, specs.filter((d) => !d.startsWith(".."))) | ||
| }, | ||
| prFiles: collected.map((f) => f.path), |
There was a problem hiding this comment.
P2: prFiles includes deleted files, so a PR that removes a test file still shows that test under "PR's other changed paths" and the test-gap check may treat it as coverage. Filter out status === "deleted" before mapping to paths.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/run.ts, line 440:
<comment>`prFiles` includes deleted files, so a PR that removes a test file still shows that test under "PR's other changed paths" and the test-gap check may treat it as coverage. Filter out `status === "deleted"` before mapping to paths.</comment>
<file context>
@@ -437,6 +437,7 @@ export async function reviewPullRequest(opts: ReviewPullRequestOptions): Promise
const specs = glob ? [`${root}${glob}`] : dirs.map((d) => path.posix.normalize(root + d))
return grepFixedAt(gitRoot, opts.head, pattern, specs.filter((d) => !d.startsWith("..")))
},
+ prFiles: collected.map((f) => f.path),
prTitle: opts.prTitle,
prBody: opts.prBody,
</file context>
| prFiles: collected.map((f) => f.path), | |
| prFiles: collected.filter((f) => f.status !== "deleted").map((f) => f.path), |
| } | ||
| if (batches.length <= maxBatches) return batches | ||
| const size = (c: AiReviewFile[]) => c.reduce((n, f) => n + fileSection(f, budget.maxFileChars, budget.coverage).length, 0) | ||
| const bins = Array.from({ length: maxBatches }, () => ({ chars: 0, chunks: [] as number[] })) |
There was a problem hiding this comment.
P2: With ALTIMATE_REVIEW_AI_MAX_BATCHES between 0 and 1 (e.g. 0.5), Array.from({ length: maxBatches }) creates no bins and the reduce throws, and runAiReview rejects instead of returning a status. Clamp the bin count to a positive integer.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-review.ts, line 579:
<comment>With `ALTIMATE_REVIEW_AI_MAX_BATCHES` between 0 and 1 (e.g. 0.5), `Array.from({ length: maxBatches })` creates no bins and the `reduce` throws, and `runAiReview` rejects instead of returning a status. Clamp the bin count to a positive integer.</comment>
<file context>
@@ -431,6 +479,126 @@ export function batchAiFiles(files: AiReviewFile[], budget: AiInputBudget): AiRe
+ }
+ if (batches.length <= maxBatches) return batches
+ const size = (c: AiReviewFile[]) => c.reduce((n, f) => n + fileSection(f, budget.maxFileChars, budget.coverage).length, 0)
+ const bins = Array.from({ length: maxBatches }, () => ({ chars: 0, chunks: [] as number[] }))
+ chunks
+ .map((c, i) => ({ i, chars: size(c) }))
</file context>
| const bins = Array.from({ length: maxBatches }, () => ({ chars: 0, chunks: [] as number[] })) | |
| const bins = Array.from({ length: Math.max(1, Math.floor(maxBatches)) }, () => ({ chars: 0, chunks: [] as number[] })) |
| if (!tail) { | ||
| joins.push({ preds: [], start: m.index!, end: i, inner }) | ||
| // A join with no ON (`cross join params as p`): step over its alias so the chain continues. | ||
| const bare = /^\s+(?:as\s+)?(?!(?:natural|inner|left|right|full|cross|outer|join|lateral|on|using|where|group|order|qualify|having|union|limit|window|select|from)\b)\w+\b/i.exec(s.slice(i)) |
There was a problem hiding this comment.
P2: A join ... using (...) has no ON clause, but this change does not step over it, so the join chain still breaks there and a redundant predicate on the next join is missed. Let the bare-join match also consume a USING (...) list.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/dbt-patterns.ts, line 933:
<comment>A `join ... using (...)` has no ON clause, but this change does not step over it, so the join chain still breaks there and a redundant predicate on the next join is missed. Let the bare-join match also consume a `USING (...)` list.</comment>
<file context>
@@ -929,7 +929,9 @@ export function redundantJoinPredicates(sql: string): string[] {
if (!tail) {
- joins.push({ preds: [], start: m.index!, end: i, inner })
+ // A join with no ON (`cross join params as p`): step over its alias so the chain continues.
+ const bare = /^\s+(?:as\s+)?(?!(?:natural|inner|left|right|full|cross|outer|join|lateral|on|using|where|group|order|qualify|having|union|limit|window|select|from)\b)\w+\b/i.exec(s.slice(i))
+ joins.push({ preds: [], start: m.index!, end: i + (bare?.[0].length ?? 0), inner })
continue
</file context>
| const bare = /^\s+(?:as\s+)?(?!(?:natural|inner|left|right|full|cross|outer|join|lateral|on|using|where|group|order|qualify|having|union|limit|window|select|from)\b)\w+\b/i.exec(s.slice(i)) | |
| const bare = /^\s+(?:as\s+)?(?!(?:natural|inner|left|right|full|cross|outer|join|lateral|on|using|where|group|order|qualify|having|union|limit|window|select|from)\b)\w+\b|^\s+using\s*\([^)]*\)/i.exec(s.slice(i)) |
| ...ev.addedLines.keys(), | ||
| ...graph.parents.map((p) => p.path), | ||
| ...graph.downstream.map((d) => d.path), | ||
| ...(ev.prFiles ?? []), |
There was a problem hiding this comment.
P2: The verifier's read allowlist now includes every PR file, including paths the user excluded with exclude or rubric excludeGlobs and out-of-project files. prFiles comes from the unfiltered collected list in run.ts, so those contents can reach the model provider. Build prFiles from the reviewable set after exclusions, or filter the allowlist with the same exclusion rules.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-verify.ts, line 783:
<comment>The verifier's read allowlist now includes every PR file, including paths the user excluded with `exclude` or rubric `excludeGlobs` and out-of-project files. `prFiles` comes from the unfiltered `collected` list in run.ts, so those contents can reach the model provider. Build `prFiles` from the reviewable set after exclusions, or filter the allowlist with the same exclusion rules.</comment>
<file context>
@@ -763,6 +780,7 @@ export function verifierTools(
...ev.addedLines.keys(),
...graph.parents.map((p) => p.path),
...graph.downstream.map((d) => d.path),
+ ...(ev.prFiles ?? []),
]),
]
</file context>
| const inBatch = new Set(batch.map((f) => f.path)) | ||
| const others = prFiles.filter((p) => !inBatch.has(p)) | ||
| if (!others.length) return message | ||
| const shown = others.slice(0, TEST_GAP_FILES_MAX) |
There was a problem hiding this comment.
P2: For PRs with more than 40 other paths, later test-file names disappear from the reviewer’s prompt, so an unselected test can be missed and trigger a false test-gap comment. Preserve test-like paths beyond the cap or let the reviewer enumerate all PR paths.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-review.ts, line 117:
<comment>For PRs with more than 40 other paths, later test-file names disappear from the reviewer’s prompt, so an unselected test can be missed and trigger a false test-gap comment. Preserve test-like paths beyond the cap or let the reviewer enumerate all PR paths.</comment>
<file context>
@@ -76,6 +96,34 @@ export function aiDryRunEnabled(env: Record<string, string | undefined> = proces
+ const inBatch = new Set(batch.map((f) => f.path))
+ const others = prFiles.filter((p) => !inBatch.has(p))
+ if (!others.length) return message
+ const shown = others.slice(0, TEST_GAP_FILES_MAX)
+ return [
+ message,
</file context>
| const parts = [`PR: ${(prTitle ?? "(untitled)").slice(0, 200)}`] | ||
| // A test-coverage claim is checked against the PR's own test files, which may sit outside the dbt project. | ||
| if (prFiles.length && candidates.some((c) => c.finding.category === "test_coverage")) { | ||
| const shown = prFiles.slice(0, PR_FILES_MAX) |
There was a problem hiding this comment.
P2: The verifier cannot discover changed files after the first 40 paths, so a test added at path 41+ can be missed by the test-gap check; add a bounded file-list tool or otherwise expose all paths to the verifier.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-verify.ts, line 395:
<comment>The verifier cannot discover changed files after the first 40 paths, so a test added at path 41+ can be missed by the test-gap check; add a bounded file-list tool or otherwise expose all paths to the verifier.</comment>
<file context>
@@ -378,12 +380,27 @@ export function hunkAround(diff: string | undefined, line?: number): string {
+ const parts = [`PR: ${(prTitle ?? "(untitled)").slice(0, 200)}`]
+ // A test-coverage claim is checked against the PR's own test files, which may sit outside the dbt project.
+ if (prFiles.length && candidates.some((c) => c.finding.category === "test_coverage")) {
+ const shown = prFiles.slice(0, PR_FILES_MAX)
+ parts.push(
+ "",
</file context>
| const seen: any[] = [] | ||
| spyOn(Provider, "defaultModel").mockResolvedValue({ providerID: "fake", modelID: "m" } as any) | ||
| spyOn(Provider, "getModel").mockResolvedValue(FAKE_MODEL as any) | ||
| spyOn(LLM, "stream").mockImplementation(async (input: any) => { |
There was a problem hiding this comment.
P3: This test depends on the real native altimate_core.review_ai_prompt call, but it never stubs Dispatcher.call. If the native core is unavailable, runAiReview returns early and LLM.stream is never called, so the test fails on an undefined off. Stub it with spyOn(Dispatcher, "call").mockResolvedValue({ success: true, data: { prompt: "CORE" } } as any), as the replay test does.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/test/altimate/review-ai-test-gap.test.ts, line 29:
<comment>This test depends on the real native `altimate_core.review_ai_prompt` call, but it never stubs `Dispatcher.call`. If the native core is unavailable, `runAiReview` returns early and `LLM.stream` is never called, so the test fails on an undefined `off`. Stub it with `spyOn(Dispatcher, "call").mockResolvedValue({ success: true, data: { prompt: "CORE" } } as any)`, as the replay test does.</comment>
<file context>
@@ -0,0 +1,58 @@
+ const seen: any[] = []
+ spyOn(Provider, "defaultModel").mockResolvedValue({ providerID: "fake", modelID: "m" } as any)
+ spyOn(Provider, "getModel").mockResolvedValue(FAKE_MODEL as any)
+ spyOn(LLM, "stream").mockImplementation(async (input: any) => {
+ seen.push(input)
+ return {
</file context>
| // Single- or double-quoted. | ||
| added: | ||
| // An escape (`\n`) at the seam is not a word. | ||
| /'[^'\n]*\s[^'\n]*(?<!\\)[A-Za-z0-9]'\s*~\s*[A-Za-z_][\w.]*|[\w)]\s*~\s*'[A-Za-z0-9][^'\n]*\s[^'\n]*'|"[^"\n]*\s[^"\n]*(?<!\\)[A-Za-z0-9]"\s*~\s*[A-Za-z_][\w.]*|[\w)]\s*~\s*"[A-Za-z0-9][^"\n]*\s[^"\n]*"/, |
There was a problem hiding this comment.
P3: The lookbehind exempts any seam character preceded by a backslash, so a literal ending in \\n (escaped backslash plus n) also escapes detection even though rendered SQL still ends in a word character. Check backslash parity before suppressing the match.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/rule-catalog.ts, line 775:
<comment>The lookbehind exempts any seam character preceded by a backslash, so a literal ending in `\\n` (escaped backslash plus `n`) also escapes detection even though rendered SQL still ends in a word character. Check backslash parity before suppressing the match.</comment>
<file context>
@@ -769,7 +769,10 @@ const CORRECTNESS: Rule[] = [
+ // Single- or double-quoted.
+ added:
+ // An escape (`\n`) at the seam is not a word.
+ /'[^'\n]*\s[^'\n]*(?<!\\)[A-Za-z0-9]'\s*~\s*[A-Za-z_][\w.]*|[\w)]\s*~\s*'[A-Za-z0-9][^'\n]*\s[^'\n]*'|"[^"\n]*\s[^"\n]*(?<!\\)[A-Za-z0-9]"\s*~\s*[A-Za-z_][\w.]*|[\w)]\s*~\s*"[A-Za-z0-9][^"\n]*\s[^"\n]*"/,
macros: true,
},
</file context>
| * what keeps the false-positive rate down deterministically. | ||
| */ | ||
| /** Findings that break `dbt parse` for the whole project, so a dev/sandbox/scratch folder doesn't excuse them. */ | ||
| const PROJECT_BREAKING_RULES = new Set(["dbt-project-rules:yaml-parse-error", "dbt-project-rules:doc-block-missing"]) |
There was a problem hiding this comment.
P3: Move PROJECT_BREAKING_RULES above the exclusionReason JSDoc or give the constant its own comment so the predicate documentation remains attached to the function.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/rubric.ts, line 79:
<comment>Move `PROJECT_BREAKING_RULES` above the `exclusionReason` JSDoc or give the constant its own comment so the predicate documentation remains attached to the function.</comment>
<file context>
@@ -75,12 +75,15 @@ export function blockingCategories(rubric: Rubric): Set<string> {
* what keeps the false-positive rate down deterministically.
*/
+/** Findings that break `dbt parse` for the whole project, so a dev/sandbox/scratch folder doesn't excuse them. */
+const PROJECT_BREAKING_RULES = new Set(["dbt-project-rules:yaml-parse-error", "dbt-project-rules:doc-block-missing"])
+
export function exclusionReason(finding: Finding, rubric: Rubric): string | null {
</file context>
`ALTIMATE_REVIEW_AI_FOCUSED=1` reviews PRs with at least 7 selected files
in several smaller calls of related files instead of one large call.
- `groupAiFiles`: connected components over macro dispatch families
(`default__x`, `snowflake__x`, `adapter.dispatch('x'`), `ref()` edges and
yml `- name:` declarations, with a same-directory fallback for files that
have no edge; components and members keep selection order
- `packFocusedBatches`: chunks of at most 5 files, never split across calls;
above 4 calls the chunks are balanced by prompt size into exactly 4
- each focused call carries only its own files' grounding and lists the other
changed files it must not comment on; `ai.input[].batch` records the call
- findings from all calls are deduplicated by file, line and normalized title
before the verifier
- knobs: `ALTIMATE_REVIEW_AI_FOCUS_MIN_FILES`, `ALTIMATE_REVIEW_AI_FOCUS_BATCH_FILES`,
`ALTIMATE_REVIEW_AI_MAX_BATCHES`; with the switch unset batches and messages
are unchanged (coverage-only messages pinned by a new snapshot)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ules - `doc-block-missing`: a yml whose only `doc()` is the new one now fires when the project keeps docs blocks of its own (before, it needed another `doc()` in the same file to resolve) - `yaml-parse-error`: anchor on the added line at or before the parser's error line, or the first added line, when the error line is missing or past the edit (an unterminated quote) - `jinja-concat-missing-space`: match double-quoted strings too, and ignore an escape such as `\n` at the seam Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`redundant-join-predicate` reset the chain at `cross join params as p` because the alias was left in the gap before the next join. Step over the alias of a join that has no ON clause so predicates of earlier inner joins are still tracked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… folders `skipNonProdModels` dropped every finding under `dev/`, `sandbox/` or `scratch/`. A yml that no longer parses or a `doc()` with no docs block breaks `dbt parse` for the whole project wherever the file lives, so `yaml-parse-error` and `doc-block-missing` are exempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A test-coverage claim on a dbt file was discarded as unverifiable when the PR's tests lived outside the dbt project: the verifier could only read the reviewed dbt files and their graph neighbours. - `AiVerifyEvidence.prFiles`: every changed path in the PR, including tests and CI outside the dbt project; `read_file` may open them - the verifier message lists those files when a candidate is a `test_coverage` claim (other batches are unchanged) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ALTIMATE_REVIEW_AI_TEST_GAP=1` appends one instruction to the core reviewer prompt: for a changed macro, materialization or generic test whose runtime behaviour changes, say whether the PR adds a test that runs it, and report one `test_coverage` comment when none does. Each batch message also lists the PR's other changed paths (tests and CI outside the dbt project included), so the model can tell whether a test exists. Unset, the system prompt and messages are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cs blocks - `prFiles` (the AI test-gap list and the verifier's read allowlist) now drops paths matching the user's exclude globs and build output / vendored paths (new `isExcludedPath` in `diff-filter.ts`); out-of-project PR files stay readable by design. - `doc-block-missing`: `doc()` names defined in installed packages' `.md` files (`dbt_packages/`, new `packageDocNames` in `run.ts`) count as defined. The no-other-`doc()` fallback now also requires the packages' docs blocks to be known (no packages declared, or installed), so a package-provided `doc()` is not reported as a parse failure. The finding body no longer claims other `doc()` references resolve when that path did not check them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b07b80d to
bf6e861
Compare
1acd5d8 to
31cb5c4
Compare
|
Thanks for updating your PR! It now meets our contributing guidelines. 👍 |
| const isDev = /(^|\/)(dev|sandbox|scratch)\//.test(finding.file) | ||
|
|
||
| if (ex.skipNonProdModels && isDev) return "non-prod model (dev/sandbox/scratch)" | ||
| if (ex.skipNonProdModels && isDev && !PROJECT_BREAKING_RULES.has(finding.rule ?? "")) return "non-prod model (dev/sandbox/scratch)" |
There was a problem hiding this comment.
WARNING: ref-target-missing is still dropped in dev/sandbox/scratch folders even though it breaks dbt parse
This exemption only keeps yaml-parse-error and doc-block-missing, but missingRefFindings emits dbt-project-rules:ref-target-missing, whose own body says "so dbt fails to parse the project" (project-rules.ts:828). skipNonProdModels defaults to true, so a ref('typo') added under models/dev/, sandbox/ or scratch/ is still filtered out here — the exact case the new constant is meant to stop. Add dbt-project-rules:ref-target-missing to the set (and jinja-unbalanced, which breaks compilation of every dependent model, if that is in scope for this fix).
| else batches.push([...c]) | ||
| } | ||
| if (batches.length <= maxBatches) return batches | ||
| const size = (c: AiReviewFile[]) => c.reduce((n, f) => n + fileSection(f, budget.maxFileChars, budget.coverage).length, 0) |
There was a problem hiding this comment.
WARNING: focused batching ignores the per-call batchChars budget
packFocusedBatches only uses sizes to balance chunks; it never splits a batch when the sections exceed budget.batchChars. batchAiFiles enforces that cap at line 475, and its default under ALTIMATE_REVIEW_AI_COVERAGE=1 is COVERAGE_BATCH_CHARS (120k) precisely so one call does not overrun the context window. With both flags on, whole chunks are packed into bins, so a bin can hold two 5-file chunks (~220k chars at the default ~22k/file), and a smaller ALTIMATE_REVIEW_AI_MAX_BATCHES makes it worse. Split bin contents against budget.batchChars the way batchAiFiles does, or otherwise clamp each call's section size.
| if (!files) return undefined | ||
| const prefix = projectRootRel ? projectRootRel + "/" : "" | ||
| const names = new Set<string>() | ||
| if (!files.includes(prefix + "packages.yml") && !files.includes(prefix + "dependencies.yml")) return names |
There was a problem hiding this comment.
SUGGESTION: a working-tree review treats an uncommitted packages.yml as "no packages"
listFilesAt(cwd, head) falls back to git ls-tree HEAD when head is undefined, so a newly added/uncommitted packages.yml is not in files and this returns an empty set. Callers treat an empty set as authoritative (projectHasDocs then assumes the packages' blocks are known), while dbt_packages below is read from the working tree, so a doc() supplied by an installed package is reported missing. Mirror projectResourceNames, which merges listWorkingTreeFiles when head === undefined (run.ts:655), or return undefined here instead of an empty set.
| for (let i = 0; i < batches.length; i += BATCH_CONCURRENCY) { | ||
| const slice = batches.slice(i, i + BATCH_CONCURRENCY) | ||
| results.push(...(await Promise.all(slice.map((b, j) => reviewBatch(plan.messages[i + j], b, system, model))))) | ||
| results.push(...(await Promise.all(slice.map((b, j) => reviewBatch(testGap ? testGapFileList(plan.messages[i + j], b, input.prFiles) : plan.messages[i + j], b, system, model))))) |
There was a problem hiding this comment.
SUGGESTION: appending the test-gap file list at send time breaks the planned message
buildBatchMessage ends with "Return ONLY the JSON array per the output contract.", so the appended ## OTHER FILES THIS PR CHANGES section becomes the last thing the model reads and the contract is no longer final. It also means plan.messages — documented as the messages runAiReview sends exactly (line 818) and what the dry run reports — differs from what is actually sent when ALTIMATE_REVIEW_AI_TEST_GAP=1. Build the list inside buildBatchMessage (before the contract line) so planning and sending stay identical.
| expect(seam('{% set q = "select " ~ cols ~ "from " ~ rel %}')).toBe(true) | ||
| expect(seam('{% set q = "select " ~ cols ~ " from " ~ rel %}')).toBe(false) | ||
| expect(seam('{{ elementary.debug_log("column_snapshot_query - \\n" ~ column_snapshot_query) }}')).toBe(false) | ||
| expect(seam("{{ log('query: \\n' ~ q) }}")).toBe(false) |
There was a problem hiding this comment.
SUGGESTION: this case cannot fail, so the new single-quote escape guard is untested
fires() applies the rule's unless clause (rule-catalog.ts:799), and log( matches it, so the assertion is false whether or not (?<!\\) exists in the single-quoted alternative. Use a non-log/print context (a {% set %} or the dbt.datediff(...) call used above) to exercise the new lookbehind; line 749 already covers the log() exclusion.
Issue for this PR
Part of #1792
Type of change
What does this PR do?
Stacked on #1771. Adds two opt-in AI-lane options, lets the verifier read every changed file, and fixes the rule gaps the planted-bug suite found. Defaults are unchanged.
What changed
ALTIMATE_REVIEW_AI_TEST_GAP=1: when a changed macro, materialization or generic test changes runtime behaviour, the AI says whether the PR adds a test that runs it, as onetest_coveragecomment. Each batch also gets the PR's other changed paths.ALTIMATE_REVIEW_AI_FOCUSED=1: on PRs with 7 or more files, related files are reviewed together in batches of up to 5. Related means a macro with its dispatch variants, a test with the model it refs, and a yml with the models it declares. Grounding is filtered per batch, findings are deduplicated, and there are at most 4 batches. No paid measurement yet, so it stays opt-in.Results
Test-gap check
Planted-bug suite on repos never used for tuning: 236 of 238 caught, 1 false alarm on 237 no-defect twins.
How did you verify your code works?
bun test test/altimate/review: 748 pass.bunx tsc --noEmit: clean.test/altimate/review-ai-test-gap.test.ts, plus focused-batch cases inreview-ai-inputs.test.ts.Cost side effects
None by default; both options are opt-in.
ALTIMATE_REVIEW_AI_TEST_GAP=1measured at about 1.02x AI-lane tokens.ALTIMATE_REVIEW_AI_FOCUSED=1splits PRs with 7 or more files into at most 4 batches, so up to 4 AI calls per review instead of one (not yet measured, hence opt-in). Letting the verifier read every changed file only applies when the verifier is enabled and stays within its 12k-token cap. No warehouse queries or cloud resources.Screenshots / recordings
Not a UI change.
Checklist
🤖 Generated with Claude Code