Skip to content

feat(review): opt-in AI input coverage, hunk excerpts and a dbt manifest context card - #1771

Open
RaghwendraSingh wants to merge 5 commits into
feat/review-recall-rulesfrom
feat/review-ai-input-coverage
Open

RaghwendraSingh wants to merge 5 commits into
feat/review-recall-rulesfrom
feat/review-ai-input-coverage

Conversation

@RaghwendraSingh

@RaghwendraSingh RaghwendraSingh commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Issue for this PR

Part of #1792

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Stacked on #1770. Gives the AI lane better input on large PRs. All of it is opt-in: with the flags unset, prompts are byte-identical to before (snapshot test).

  • ALTIMATE_REVIEW_AI_COVERAGE=1:
    • Per-kind file quotas (models 12, macros 8, tests 4, yml/config 6), so models can't crowd out supporting files.
    • Hunk-centred excerpts instead of cutting at the first 6,000 characters.
    • Files not sent are listed in the coverage ledger.
  • ALTIMATE_REVIEW_AI_CONTEXT_CARD=1: a short dbt context card per file from the manifest. For models: materialization, unique key, parents, children, exposures, declared columns with their tests, siblings. For macros: callers and dispatch variants.
  • ALTIMATE_REVIEW_AI_DRYRUN=1: builds the AI input and records its size without calling the model, for cost estimates.

Results (blind Claude judge)

Set Without With coverage + card Tokens
21 tuning-repo PRs where the input changes (39 defects) 6 caught 10 caught 1.32×
25 repos never used for tuning (129 snapshots, 188 defects), with the test-gap check from the next PR (config C vs A) 43 caught, 44% useful, 1.2% wrong 47 caught, 48% useful, 0.9% wrong 1.25×
  • The fresh-repo gain is +4 catches (95% CI −4 to +12). That leans better but is within noise, so these stay opt-in.

How did you verify your code works?

  • bun test test/altimate/review: 725 pass.
  • bunx tsc --noEmit: clean.
  • New cases in test/altimate/review-ai-inputs.test.ts: quotas, redistribution, hunk windows, ledger, card rendering, default prompt snapshot.

Cost side effects

None by default; every option is opt-in and the default prompts are byte-identical. With ALTIMATE_REVIEW_AI_COVERAGE=1 and ALTIMATE_REVIEW_AI_CONTEXT_CARD=1 the AI lane's input grows with the PR's changed files (bounded by the per-kind quotas), measured at 1.25-1.32x AI-lane tokens; the number of AI calls is unchanged. The context card is built from the local manifest (no warehouse queries). ALTIMATE_REVIEW_AI_DRYRUN=1 makes no model call and records the input size for cost estimates.

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

Changes since review: c99cc33753 addresses the second-round review feedback: dry run evaluated before replay, raw-line windows only when context lines confirm them, full-diff file budget capped at maxFileChars, project-first model/macro card resolution, optional ai.unsent quota note.

🤖 Generated with Claude Code


Summary by cubic

Adds opt-in flags that give the AI reviewer better input on large PRs while leaving the default prompt byte-identical (pinned by a snapshot test).

What changed

  • ALTIMATE_REVIEW_AI_COVERAGE=1 sends per-kind file quotas (models 12, macros 8, tests 4, yml/config 6), most-changed files first, with unused quota redistributed; quotas are overridable via ALTIMATE_REVIEW_AI_QUOTA_*, and a quota of 0 disables that bucket.
  • The diff section gets its own 16,000-char budget (ALTIMATE_REVIEW_AI_COVERAGE_DIFF_CHARS) that keeps hunks whole; over-budget files send ±20-line windows around changed lines with omission markers, and batches are capped at 120,000 chars per call (ALTIMATE_REVIEW_AI_BATCH_CHARS). When a whole diff fits, the current-file section shrinks to a 2,000-char head.
  • For compiled SQL whose line numbers drift after Jinja expansion, windows anchor on the added lines' text instead of line numbers; when the windows would be approximate, only the file head is sent with a marker.
  • Files over the quotas are reported in the coverage notes and ai.unsent rather than in the "Not reviewed" ledger, since the deterministic lanes still checked them.
  • ALTIMATE_REVIEW_AI_CONTEXT_CARD=1 adds a short dbt context card per file from the manifest: materialization and keys, parents/children/exposures, declared columns with tests, and macro callers and variants; yml and singular tests get the card of the model they test.
  • ALTIMATE_REVIEW_AI_DRYRUN=1 plans the AI input and records its size without calling the model, for cost estimates; it runs before replay so a dry run with a replay envelope makes no verifier call.
  • Per-file ai.input metadata (chars, truncated, hunks sent/total, card) is recorded on the verdict envelope under these flags. The coverage ledger marks exactly the files the AI lane sends.

In a blind evaluation across 25 repos never used for tuning, coverage plus the context card caught 47 vs 43 defects with fewer wrong findings at about 1.25× the tokens; the gain is within noise, so the flags stay opt-in.

Migration

  • No changes required; the flags default to off.

Written for commit 31b384c. Summary will update on new commits.

View guided diff Turn on auto-fix

@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: d1c13b70-311a-4995-9c19-e58da09c747b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

9 issues found across 9 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/opencode/test/altimate/fixtures/review-ai-prompt-input.ts">

<violation number="1" location="packages/opencode/test/altimate/fixtures/review-ai-prompt-input.ts:12">
P3: The hunk header counts in `bigDiff` don't match the hunk body: the old side has 3 lines and the new side has `size + 2`, but the header says `size + 2` and `size + 4`. Use the real counts so this fixture is a valid unified diff for any future hunk-window test.</violation>
</file>

<file name="packages/opencode/src/altimate/review/ai-review.ts">

<violation number="1" location="packages/opencode/src/altimate/review/ai-review.ts:391">
P2: A diff between `coverage.diffChars` and `maxChars` bypasses `excerptDiff`, so a lower `ALTIMATE_REVIEW_AI_COVERAGE_DIFF_CHARS` override is ignored. Compare against the effective limit, such as `Math.min(maxChars, coverage.diffChars)`.</violation>

<violation number="2" location="packages/opencode/src/altimate/review/ai-review.ts:421">
P3: Batch sizing counts every card, but `buildBatchMessage` omits cards after the 6,000-character per-call cap. Card-heavy inputs can therefore split into unnecessary model calls; apply the same cumulative card cap when estimating batches.</violation>

<violation number="3" location="packages/opencode/src/altimate/review/ai-review.ts:649">
P2: `planAiInput` now runs outside the `try` in `runAiReview`, so an exception while rendering the prompt escapes the lane. The doc comment promises this function never throws. Move `const plan = planAiInput(input)` inside the `try`, or wrap the planning in its own guard that returns an `error` status.</violation>
</file>

<file name="packages/opencode/src/altimate/review/runner.ts">

<violation number="1" location="packages/opencode/src/altimate/review/runner.ts:570">
P2: Column tests disappear from the card for manifests without `attached_node`; fall back to `depends_on.nodes` and `test_metadata.kwargs.column_name` when those fields are absent.</violation>

<violation number="2" location="packages/opencode/src/altimate/review/runner.ts:603">
P3: `text()` keeps only `partition_by.field` and drops `granularity`. Include the granularity in the rendered value, since the card is meant to show partitioning facts the diff does not.</violation>

<violation number="3" location="packages/opencode/src/altimate/review/runner.ts:639">
P2: These `.slice()` caps run before rendering, so the card's `+N more` marker can never show for callers, columns or siblings. A macro with 40 callers reads as having 10. Return the full lists and let `renderContextCard`'s `list()` cap them, or keep the true totals.</violation>
</file>

<file name="packages/opencode/src/altimate/review/ai-context-card.ts">

<violation number="1" location="packages/opencode/src/altimate/review/ai-context-card.ts:95">
P2: Seed files never get a context card. `MODEL_KINDS` leaves out `seed`, although `ModelCard` and `modelCard()` are documented to cover seeds. Add `seed` to the set, and drop the `kind === "model"` filter in the yml/test branch if seed parents should be carded there too.</violation>

<violation number="2" location="packages/opencode/src/altimate/review/ai-context-card.ts:123">
P2: Schema and test changes attached to seed or snapshot nodes lose their related context card because this filter accepts only models. Include the resource kinds supported by `modelCard`.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

const replay = aiReplayPath()
if (replay) return runAiReplay(input, replay)
const startedAt = Date.now()
const plan = planAiInput(input)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: planAiInput now runs outside the try in runAiReview, so an exception while rendering the prompt escapes the lane. The doc comment promises this function never throws. Move const plan = planAiInput(input) inside the try, or wrap the planning in its own guard that returns an error status.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-review.ts, line 649:

<comment>`planAiInput` now runs outside the `try` in `runAiReview`, so an exception while rendering the prompt escapes the lane. The doc comment promises this function never throws. Move `const plan = planAiInput(input)` inside the `try`, or wrap the planning in its own guard that returns an `error` status.</comment>

<file context>
@@ -263,13 +646,20 @@ export async function runAiReview(input: AiReviewInput): Promise<AiReviewResult>
   const replay = aiReplayPath()
   if (replay) return runAiReplay(input, replay)
   const startedAt = Date.now()
+  const plan = planAiInput(input)
+  const { files, budget } = plan
+  const dryRun = aiDryRunEnabled()
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing: the only caller wraps the lane in try/catch and records state: "error" (orchestrate.ts ~2498), so nothing escapes. Only the "never throws" doc wording is loose.

.filter(([id, n]) => id.startsWith("model.") && asArray<string>(n?.depends_on?.macros).includes(uid))
.map(([, n]) => String(n.name))
.sort((a, b) => Number(changed.has(b)) - Number(changed.has(a)))
.slice(0, 10)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: These .slice() caps run before rendering, so the card's +N more marker can never show for callers, columns or siblings. A macro with 40 callers reads as having 10. Return the full lists and let renderContextCard's list() cap them, or keep the true totals.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/runner.ts, line 639:

<comment>These `.slice()` caps run before rendering, so the card's `+N more` marker can never show for callers, columns or siblings. A macro with 40 callers reads as having 10. Return the full lists and let `renderContextCard`'s `list()` cap them, or keep the true totals.</comment>

<file context>
@@ -531,6 +532,129 @@ export function createDispatcherRunner(opts: DispatcherRunnerOptions): ReviewRun
+          .filter(([id, n]) => id.startsWith("model.") && asArray<string>(n?.depends_on?.macros).includes(uid))
+          .map(([, n]) => String(n.name))
+          .sort((a, b) => Number(changed.has(b)) - Number(changed.has(a)))
+          .slice(0, 10)
+        const macroName = String(macro.name)
+        const sep = macroName.indexOf("__")
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. We'll pass the totals so the card can say how many it left out.

return text.slice(0, maxChars - 1) + "…"
}

const MODEL_KINDS = new Set(["model_sql", "python_model", "snapshot"])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Seed files never get a context card. MODEL_KINDS leaves out seed, although ModelCard and modelCard() are documented to cover seeds. Add seed to the set, and drop the kind === "model" filter in the yml/test branch if seed parents should be carded there too.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-context-card.ts, line 95:

<comment>Seed files never get a context card. `MODEL_KINDS` leaves out `seed`, although `ModelCard` and `modelCard()` are documented to cover seeds. Add `seed` to the set, and drop the `kind === "model"` filter in the yml/test branch if seed parents should be carded there too.</comment>

<file context>
@@ -0,0 +1,134 @@
+  return text.slice(0, maxChars - 1) + "…"
+}
+
+const MODEL_KINDS = new Set(["model_sql", "python_model", "snapshot"])
+const DECLARING_KINDS = new Set(["schema_yml", "source_yml", "test"])
+
</file context>
Suggested change
const MODEL_KINDS = new Set(["model_sql", "python_model", "snapshot"])
const MODEL_KINDS = new Set(["model_sql", "python_model", "snapshot", "seed"])

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. We'll either add seed or drop the unreachable seed lookup.

const out: string[] = []
for (let h = 0; h < hunks; h++) {
const start = 10 + h * 100
out.push(`@@ -${start},${size + 2} +${start},${size + 4} @@`)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The hunk header counts in bigDiff don't match the hunk body: the old side has 3 lines and the new side has size + 2, but the header says size + 2 and size + 4. Use the real counts so this fixture is a valid unified diff for any future hunk-window test.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/test/altimate/fixtures/review-ai-prompt-input.ts, line 12:

<comment>The hunk header counts in `bigDiff` don't match the hunk body: the old side has 3 lines and the new side has `size + 2`, but the header says `size + 2` and `size + 4`. Use the real counts so this fixture is a valid unified diff for any future hunk-window test.</comment>

<file context>
@@ -0,0 +1,64 @@
+  const out: string[] = []
+  for (let h = 0; h < hunks; h++) {
+    const start = 10 + h * 100
+    out.push(`@@ -${start},${size + 2} +${start},${size + 4} @@`)
+    out.push(` select ${h} as ctx_before`)
+    for (let i = 0; i < size; i++) out.push(`+  , coalesce(col_${h}_${i}, 'unknown_value_${i}') as renamed_column_${h}_${i}`)
</file context>
Suggested change
out.push(`@@ -${start},${size + 2} +${start},${size + 4} @@`)
out.push(`@@ -${start},3 +${start},${size + 2} @@`)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing: the parser doesn't use the counts, and the fixture feeds a byte-identical snapshot that has to stay as it is.

materialized: text(config.materialized),
uniqueKey: text(config.unique_key),
incrementalStrategy: text(config.incremental_strategy),
partitionBy: text(config.partition_by),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: text() keeps only partition_by.field and drops granularity. Include the granularity in the rendered value, since the card is meant to show partitioning facts the diff does not.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/runner.ts, line 603:

<comment>`text()` keeps only `partition_by.field` and drops `granularity`. Include the granularity in the rendered value, since the card is meant to show partitioning facts the diff does not.</comment>

<file context>
@@ -531,6 +532,129 @@ export function createDispatcherRunner(opts: DispatcherRunnerOptions): ReviewRun
+          materialized: text(config.materialized),
+          uniqueKey: text(config.unique_key),
+          incrementalStrategy: text(config.incremental_strategy),
+          partitionBy: text(config.partition_by),
+          clusterBy: text(config.cluster_by),
+          contractEnforced: config.contract?.enforced === true || node.contract?.enforced === true,
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing for now: the card shows the partition column. Granularity is a nice-to-have, and we may add it with the next card revision.

let size = 0
for (const f of files) {
const n = fileSection(f, budget.maxFileChars).length
const n = fileSection(f, budget.maxFileChars, budget.coverage).length

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: Batch sizing counts every card, but buildBatchMessage omits cards after the 6,000-character per-call cap. Card-heavy inputs can therefore split into unnecessary model calls; apply the same cumulative card cap when estimating batches.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-review.ts, line 421:

<comment>Batch sizing counts every card, but `buildBatchMessage` omits cards after the 6,000-character per-call cap. Card-heavy inputs can therefore split into unnecessary model calls; apply the same cumulative card cap when estimating batches.</comment>

<file context>
@@ -105,7 +418,7 @@ export function batchAiFiles(files: AiReviewFile[], budget: AiInputBudget): AiRe
   let size = 0
   for (const f of files) {
-    const n = fileSection(f, budget.maxFileChars).length
+    const n = fileSection(f, budget.maxFileChars, budget.coverage).length
     if (cur.length && size + n > budget.batchChars) {
       batches.push(cur)
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. Sizing over-counts cards, so batches come out smaller than needed, never larger.

/** Head-side lines of context kept around each changed line in a file excerpt. */
const EXCERPT_CONTEXT_LINES = 20
/** Characters of a file's diff section under coverage (hunks beyond it are listed as omitted). */
const COVERAGE_DIFF_CHARS = 16_000

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: Coverage mode can build a single ~660k-char call with no per-call cap

With ALTIMATE_REVIEW_AI_COVERAGE=1 the diff section can be COVERAGE_DIFF_CHARS (16,000) per file plus up to 6,000 of current file, while the new quotas allow 30 files (12 models + 8 macros + 4 tests + 6 yml) and batchChars still defaults to Infinity (line 57), so batchAiFiles keeps them all in one batch. One call can reach ~30 x 22k = 660k chars (~165k tokens) versus <= 20 x 12k = 240k before. That overruns a 128k-token model's context, reviewBatch throws, every batch fails and the lane reports error with no findings - on exactly the large PRs this mode targets. Consider defaulting batchChars to a bounded value when coverage is on, or scaling the per-file budgets against a per-call total.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in bf6e861: with ALTIMATE_REVIEW_AI_COVERAGE=1, batchChars now defaults to 120k chars (about 30k tokens), so 30 quota files split across calls instead of one ~660k-char call. ALTIMATE_REVIEW_AI_BATCH_CHARS still overrides it, and with coverage off the default stays Infinity (the prompt snapshot test is unchanged).

const BUCKET_ORDER = [...new Set(KIND_ORDER.map((k) => BUCKET_OF[k]))]

/** Added and removed lines of a unified diff (file headers before the first hunk excluded). */
export function changedLineCount(diff: string | undefined): number {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Duplicates the existing countChangedLines

diff-filter.ts:329 already exports countChangedLines (imported from the same module for walkDiffLines) for this exact job, and it correctly ignores +++/--- file headers. This copy counts +++ b/... and --- a/... as changed lines whenever the diff has no @@ hunk (first === -1, so Math.max(first, 0) === 0), which inflates the most-changed ordering used by selectAiFilesByQuota for header-less diffs. Reuse countChangedLines(f.diff) here instead.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point; we'll reuse countChangedLines in a follow-up cleanup. Behaviour is the same for git diffs, which always carry @@ headers.

if (!kept.length) return { text: truncate(content, maxChars), truncated: true }
// Fill what is left with the file head, before the first window.
let head = 0
while (head + 1 < kept[0][0] && used + size(1, head + 1) + MARKER <= maxChars) head++

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Quadratic head fill

size(1, head + 1) re-slices and re-sums lines 1..head+1 on every iteration, so extending the head over H lines costs O(H^2) with a fresh array each step (H can approach maxChars for files with many short or blank lines, and this runs once per file under coverage). Accumulate the running length instead, e.g. keep headChars and add lines[head].length + 1 per step, then test used + headChars + MARKER <= maxChars.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing now: it's bounded by the per-file char budget (a few hundred lines). We'll switch it to a running sum if profiling ever shows it.

return text.slice(0, maxChars - 1) + "…"
}

const MODEL_KINDS = new Set(["model_sql", "python_model", "snapshot"])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Changed seeds never get a context card

MODEL_KINDS omits "seed" and seeds are not in DECLARING_KINDS, so cardsFor returns [] for every seed file. But runner.modelCard explicitly resolves seed. nodes (runner.ts:543) and ModelCard is documented as facts about "one model, snapshot or seed", so that branch is unreachable through this feature. Either add "seed" to MODEL_KINDS (a changed seed is a legitimate review target) or drop the seed lookup from modelCard.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same root cause as r4626982915; tracked there.

@kilo-code-bot

kilo-code-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Code Review Summary

Status: 8 Issues Found | Recommendation: Address before merge

The branch was rebased onto a newer base; the PR's own files are unchanged since the last review at c99cc337, so every finding below was re-verified at HEAD 31b384c1. Each already has an active inline thread on the same line, so no new inline comments were posted.

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 6
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/format.ts 292 The quota note claims "the deterministic checks still ran" on every file in ai.unsent, but no deterministic lane touches seeds (.csv files are never touch()ed), so an out-of-quota seed is announced as checked while the same summary renders the ledger entry no checks run for seed files at the full tier for that path.
packages/opencode/src/altimate/review/ai-review.ts 61 quota() still validates with Number(), so " ", "0x0" and "0e0" all become 0 and silently disable a bucket, while "1e3" becomes 1000; require a trimmed decimal integer, consistent with the documented "invalid values keep the default".

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/ai-review.ts 286 locateAddedLines returns undefined when no added line is eligible and the context lines are also Jinja/short, so compiled SQL silently falls back to raw line numbers without the "windows would be approximate" marker.
packages/opencode/src/altimate/review/ai-review.ts 353 excerptFile breaks out of the window loop on the first window that does not fit, dropping later, smaller windows that would still fit; use continue so changed lines keep their SQL window.
packages/opencode/src/altimate/review/ai-review.ts 358 The excerpt head fill recomputes size(1, head + 1) per iteration, making it O(H²) per file under coverage.
packages/opencode/src/altimate/review/ai-review.ts 684 When a quota excludes every sendable file, the lane reports no changed dbt files to review even though plan.unsent lists real changes; the check also precedes the dry-run reason.
packages/opencode/src/altimate/review/ai-review.ts 154 changedLineCount duplicates countChangedLines in diff-filter.ts and counts +++/--- on hunk-less diffs.
packages/opencode/src/altimate/review/orchestrate.ts 2600 touch() re-implements the selection branch owned by planAiInput, so the ledger can silently drift from what the lane actually sends; extract one shared selector.
Files Reviewed (10 files)
  • packages/opencode/src/altimate/review/ai-context-card.ts - no new issues (seed cards and patch_path resolution re-verified)
  • packages/opencode/src/altimate/review/ai-review.ts - 6 issues
  • packages/opencode/src/altimate/review/format.ts - 1 issue
  • packages/opencode/src/altimate/review/orchestrate.ts - 1 issue
  • packages/opencode/src/altimate/review/run.ts - no issues
  • packages/opencode/src/altimate/review/runner.ts - no new issues (path/project card resolution re-verified)
  • packages/opencode/src/altimate/review/verdict.ts - no issues
  • packages/opencode/test/altimate/fixtures/review-ai-prompt-default.txt - no issues
  • packages/opencode/test/altimate/fixtures/review-ai-prompt-input.ts - no issues
  • packages/opencode/test/altimate/review-ai-inputs.test.ts - no issues

Fix these issues in Kilo Cloud

Previous Review Summaries (3 snapshots, latest commit c99cc33)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit c99cc33)

Status: 7 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 5
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/format.ts 292 The new quota note claims "the deterministic checks still ran" on every file in ai.unsent, but no deterministic lane touches seeds .csv files are never touch()ed), so an out-of-quota seed is announced as checked while the same summary renders the ledger entry no checks run for seed files at the full tier for that path.
packages/opencode/src/altimate/review/ai-review.ts 61 quota() validates with Number() instead of a strict integer, so " ", "0x0" and "0e0" all become 0 and silently disable a bucket, while "1e3" becomes 1000; inconsistent with num() and with the documented "invalid values keep the default".

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/ai-review.ts 353 excerptFile breaks out of the window loop on the first window that does not fit, dropping later, smaller windows that would still fit; use continue so changed lines keep their SQL window.
packages/opencode/src/altimate/review/ai-review.ts 684 When a quota excludes every sendable file, the lane reports no changed dbt files to review (and preempts the dry-run reason) even though ai.unsent lists real changes.
packages/opencode/src/altimate/review/orchestrate.ts 2594 touch() re-implements the selection branch owned by planAiInput, so the ledger can silently drift from what the lane actually sends; extract one shared selector.
packages/opencode/src/altimate/review/ai-review.ts 154 changedLineCount duplicates countChangedLines in diff-filter.ts and counts +++/--- on hunk-less diffs. Previously raised; deferred by the author to a follow-up.
packages/opencode/src/altimate/review/ai-review.ts 358 The excerpt head fill recomputes size(1, head + 1) per iteration, making it O(H²) per file under coverage. Previously raised; author declined to change it now.
Files Reviewed (10 files)
  • packages/opencode/src/altimate/review/ai-context-card.ts - no new issues (seed cards fixed; tested-seed/snapshot parents acknowledged as a follow-up)
  • packages/opencode/src/altimate/review/ai-review.ts - 4 issues (2 carried)
  • packages/opencode/src/altimate/review/format.ts - 1 issue
  • packages/opencode/src/altimate/review/orchestrate.ts - 1 issue
  • packages/opencode/src/altimate/review/run.ts - no issues
  • packages/opencode/src/altimate/review/runner.ts - no new issues (path/project resolution re-verified)
  • packages/opencode/src/altimate/review/verdict.ts - no issues
  • packages/opencode/test/altimate/fixtures/review-ai-prompt-default.txt - no issues
  • packages/opencode/test/altimate/fixtures/review-ai-prompt-input.ts - no issues
  • packages/opencode/test/altimate/review-ai-inputs.test.ts - no issues

The previous percent-escaped ai.input note is resolved: coverage now bounds each call at 120,000 chars and the truncated flag uses the budget actually applied. Cost side effects are documented in the PR body (opt-in, 30-file cap, bounded batches, ≤1.32x tokens).

Fix these issues in Kilo Cloud

Previous review (commit bf6e861)

Status: 3 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 0
SUGGESTION 3
Issue Details (click to expand)

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/ai-review.ts 149 changedLineCount duplicates the existing countChangedLines in diff-filter.ts:378 and counts +++/--- headers for header-less diffs, inflating the most-changed ordering used by selectAiFilesByQuota.
packages/opencode/src/altimate/review/ai-review.ts 343 Head fill recomputes size(1, head + 1) on every iteration (fresh slice + re-sum), so extending the head is O(H²) per file under coverage; keep a running total.
packages/opencode/src/altimate/review/ai-context-card.ts 95 Seeds are in neither MODEL_KINDS nor DECLARING_KINDS, so cardsFor returns [] for every seed while modelCard resolves seed. nodes — a changed seed never gets a card.
Files Reviewed (9 files)
  • packages/opencode/src/altimate/review/ai-context-card.ts - 1 issue
  • packages/opencode/src/altimate/review/ai-review.ts - 2 issues
  • packages/opencode/src/altimate/review/orchestrate.ts - no issues
  • packages/opencode/src/altimate/review/run.ts - no issues
  • packages/opencode/src/altimate/review/runner.ts - no issues
  • packages/opencode/src/altimate/review/verdict.ts - no issues
  • packages/opencode/test/altimate/fixtures/review-ai-prompt-default.txt - no issues
  • packages/opencode/test/altimate/fixtures/review-ai-prompt-input.ts - no issues
  • packages/opencode/test/altimate/review-ai-inputs.test.ts - no issues

Note: the per-call size bound raised earlier is now fixed (batchChars defaults to 120,000 under coverage). The PR description still omits the repository's required "Cost side effects" section; when coverage is enabled the lane sends up to 30 files and splits them into several model calls, so that section should be filled in.

Fix these issues in Kilo Cloud

Previous review (commit b07b80d)

Status: 4 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 3
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/ai-review.ts 30 Coverage lets one call reach ~660k chars (~165k tokens): 16k diff + 6k file per file, 30-file quotas, batchChars still Infinity. Overruns 128k-context models and fails the lane on the large PRs the mode targets.

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/ai-review.ts 144 changedLineCount duplicates countChangedLines in diff-filter.ts and counts +++/--- headers as changed lines for header-less diffs.
packages/opencode/src/altimate/review/ai-review.ts 338 Head fill recomputes size(1, head + 1) each iteration: O(H²) with a fresh slice per step; keep a running total.
packages/opencode/src/altimate/review/ai-context-card.ts 95 Seeds are in neither MODEL_KINDS nor DECLARING_KINDS, so modelCard's seed. branch (runner.ts:543) is unreachable and changed seeds get no card.

Note: the PR description omits the repository's required "Cost side effects" section (the automated issue-compliance check already flagged this). The lane is opt-in and bounded by file quotas and character budgets, but it does raise per-review token spend ~1.25x when enabled, so that section should be filled in.

Fix these issues in Kilo Cloud

Files Reviewed (9 files)
  • packages/opencode/src/altimate/review/ai-context-card.ts - 1 issue
  • packages/opencode/src/altimate/review/ai-review.ts - 3 issues
  • packages/opencode/src/altimate/review/orchestrate.ts - no issues
  • packages/opencode/src/altimate/review/run.ts - no issues
  • packages/opencode/src/altimate/review/runner.ts - no issues
  • packages/opencode/src/altimate/review/verdict.ts - no issues
  • packages/opencode/test/altimate/fixtures/review-ai-prompt-default.txt - no issues
  • packages/opencode/test/altimate/fixtures/review-ai-prompt-input.ts - no issues
  • packages/opencode/test/altimate/review-ai-inputs.test.ts - no issues

Reviewed by deepseek-v4.1-flash-1017 · Input: 166.9K · Output: 30.2K · Cached: 2.7M

Review guidance: REVIEW.md from base branch feat/review-recall-rules

@hkaltai hkaltai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: approve, with minor follow-ups. Panel: Claude, Codex and lead reviewer, covering all 9 changed files.

The core promise holds: with the flags unset the prompt is byte-identical to the base (confirmed independently by two reviewers against the snapshot), and the 31 tests in review-ai-inputs.test.ts pass. All findings concern behavior behind the opt-in flags and advisory context quality, so none is CRITICAL or MAJOR.

Counts: CRITICAL 0, MAJOR 0, MINOR 8 (inline), NIT 10.

Most worthwhile follow-ups: bound the coverage-mode call size (batchChars default), make compiled-SQL windowing robust when added lines are Jinja-only, and resolve model/macro cards by project and path rather than by first name match.

Nits

  • ai-review.ts:51 num requires n > 0: ALTIMATE_REVIEW_AI_QUOTA_TEST=0 silently becomes 4, so a bucket cannot be disabled. Quoted: return v && Number.isFinite(n) && n > 0 ? n : d. Fix: separate parser allowing 0 for quotas.
  • excerptFile window excerpt (ai-review.ts:327 area): when the first context window exceeds the budget, the beginning is kept and the changed anchor can be dropped (350-char lines, change at line 100, 6,000 budget). Fix: reserve space for changed lines first, then expand context.
  • runner.ts:640-649: a macro name containing __ splits at the first __, so an unrelated get_x__y is listed as a variant of y. Fix: only treat default/adapter prefixes or dispatchers as variants.
  • ai-review.ts:240: preamble longer than budget emits an empty "(hunk truncated)" but reports hunksSent: 1. Fix: record 0 when maxChars - used <= 0.
  • ai-review.ts:313: excerptFile can exceed maxChars by about 40 chars (trailing marker not reserved when no head window). Fix: reserve it.
  • ai-context-card.ts MODEL_KINDS lacks seed although the ModelCard doc says "model, snapshot or seed". Fix: add seed or fix the doc.
  • Comment at ai-review.ts:563 says cards stop once the call holds 6,000 chars, but a too-large card is skipped and later smaller ones are still added. Reword.
  • planAiInput doc says "Pure apart from reading env", but env is a parameter.
  • Naming: AiReviewFile.card (string) vs AiInputRecord.card (boolean); fileChars in renderFile is a budget (fileBudget); CoverageArg = boolean | AiCoverageRender | undefined is a test-only tri-state; local const MARKER = 40 could be module-level MARKER_RESERVE.
  • Missing tests: dry-run+replay+verify, duplicate model names across packages, macro basename/path collisions, YAML cards via the real runner (not a mocked graphParents), Jinja-only changes on compiled SQL, bounded size under coverage, and rendering of quota-unsent files in the "Not reviewed" block.

@@ -263,13 +646,20 @@ export async function runAiReview(input: AiReviewInput): Promise<AiReviewResult>
const replay = aiReplayPath()
if (replay) return runAiReplay(input, replay)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Replay returns before the dry-run check

if (replay) return runAiReplay(input, replay)

With ALTIMATE_REVIEW_AI_DRYRUN=1, ALTIMATE_REVIEW_AI_REPLAY=<envelope> and ALTIMATE_REVIEW_AI_VERIFY=1, replay returns before dry-run is evaluated, so a stored envelope with AI findings still reaches the verifier (a model call) and no ai.input is recorded. Replay is an eval-only path, hence MINOR, but it breaks the no-model-call promise of dry-run. Fix: evaluate dry-run (and plan the input) before dispatching replay, and add a regression test. (Codex)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good catch. In c99cc33 ("dry run before replay, ...") runAiReview plans the input and checks dry run before dispatching replay, so ALTIMATE_REVIEW_AI_DRYRUN=1 with a replay envelope and the verifier switch returns skipped with ai.input recorded and makes no model call. A regression test covers the three switches together.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Fixed — ai-review.ts:684-687 now returns the dry-run result before runAiReplay; test 'dry run is honoured before replay' covers DRYRUN+REPLAY+VERIFY. Suite: 42 pass.

* lines (8+ non-space chars, no Jinja) are found by text, and "unlocated" otherwise.
*/
function locateAddedLines(lines: string[], rows: Array<{ line: number; text: string; kind: string }>): number[] | "unlocated" | undefined {
const eligible = rows.filter((r) => r.kind === "+" && !JINJA.test(r.text) && r.text.replace(/\s/g, "").length >= 8)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Compiled-SQL windows fall back to raw line numbers when no added line is eligible

const eligible = rows.filter((r) => r.kind === "+" && !JINJA.test(r.text) && r.text.replace(/\s/g, "").length >= 8)

When every added line is Jinja or under 8 non-space chars ({{ ref(..) }} swaps, {% if is_incremental() %}, , id), eligible is empty and the function returns undefined, which excerptFile treats as "raw line numbers match". On compiled SQL where macro expansion shifted lines, windows are then centred on the wrong lines and the unapproximated marker is not emitted. The diff itself is still sent, so this degrades excerpt quality only (opt-in flag). Fix: let the caller say whether f.sql is compiled, or also check context rows; return "unlocated" if nothing can be verified. Add a Jinja-only test on compiled SQL. (Claude)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. In c99cc33, when no added line is eligible (Jinja-only or short changes), locateAddedLines checks the eligible context lines: raw line numbers are used only when those match, otherwise the result is "unlocated" and the head goes out with the "approximate" marker. There is a new test with a {{ dbt_utils.star(...) }} swap on compiled SQL, plus the matching raw-file case, which still gets its window.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Partially fixed — locateAddedLines (ai-review.ts:280-287) now falls back to eligible context lines, with a test. But when the context lines are also Jinja/short (e.g. {% if is_incremental() %} around one added line) it still returns undefined and trusts raw line numbers on compiled SQL; see the new inline comment.

maxFiles: num(env.ALTIMATE_REVIEW_AI_MAX_FILES, MAX_FILES),
maxFileChars: num(env.ALTIMATE_REVIEW_AI_MAX_FILE_CHARS, MAX_DIFF_CHARS),
batchChars: num(env.ALTIMATE_REVIEW_AI_BATCH_CHARS, Infinity),
coverage:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Coverage mode enlarges the single call with no bound

batchChars: num(env.ALTIMATE_REVIEW_AI_BATCH_CHARS, Infinity),

Coverage raises the file cap to 30 and adds a per-diff budget, while batchChars stays Infinity. A 40-large-model PR was measured at about 242k chars by default vs about 658k with coverage, which can exceed context or the 300 s timeout and return no AI findings at all. The unbounded single call is pre-existing, but coverage makes it markedly worse. Fix: default batchChars to a finite value when coverage is on, or cap total chars and push overflow into unsent. (Claude)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was fixed in a3c5f07 ("bound coverage per-call size"), which landed after the commit you reviewed. With coverage on, batchChars now defaults to 120k characters instead of Infinity, so a 30-file coverage run is split across calls. ALTIMATE_REVIEW_AI_BATCH_CHARS still overrides it, and the default without coverage is unchanged. The test "coverage bounds the per-call size by default" covers the 30 x 16k case.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Fixed — ai-review.ts:37,67: batchChars defaults to 120,000 under coverage (Infinity otherwise); test asserts the default and that 30 files split into several calls.

const nodes = Object.entries<any>(raw.nodes ?? {})
const isModel = (id: string) => id.startsWith("model.")
const hit =
nodes.find(([id, n]) => isModel(id) && n?.name === name) ??

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] modelCard picks the first same-named model across packages

nodes.find(([id, n]) => isModel(id) && n?.name === name) ??

A dependency package model with the same name listed earlier in the manifest supplies materialization, keys, columns and graph facts for the changed project model (reproduced by both external reviewers). The card is presented as authoritative. Fix: prefer metadata.project_name (as macroCard does) and/or match by path or unique id. (Codex, Claude)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, confirmed. modelCard now takes the changed file's path. Since c99cc33 it resolves in this order: the node whose original_file_path is that file within metadata.project_name, then a same-named model, snapshot or seed in the project, then any package. The test has a dependency-package orders listed first, and the card now comes from the project's model.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Fixed — runner.ts:572-590 resolves by changed path, then this project, then any package; test puts a pkg model first and expects the project one.

const byName = macros.filter(([, m]) => m?.name === name)
const path = cardOpts?.path
const hit =
byName.find(([, m]) => m.package_name === project) ??

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] macroCard resolves by file basename before path, and not within the project

byName.find(([, m]) => m.package_name === project) ??
          byName[0] ??

The lookup name is the file basename, not necessarily the macro defined in it. macros/helpers.sql defining foo can get the card of an unrelated helpers macro, or a package macro (e.g. dbt_utils.date_spine) with its callers and variants. The path fallback is also not restricted to the project package. Fix: match by path within the project first, then name within the project, then any package. (Codex, Claude)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c99cc33 in the order you suggested: first the macros defined in the changed file within the project (preferring the one named like the file), then the name within the project, then any package. macros/helpers.sql defining foo now gets foo's card instead of a package helpers macro, and a project date_spine wins over dbt_utils.date_spine. Both cases are tested.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Fixed — runner.ts:671-678 prefers macros defined in the changed file within this project before name lookup; helpers -> foo assertion covers it. (A nested-path suffix collision remains possible but is rare; nit.)

return macro ? [{ name: macro.name, macro }] : []
}
if (!DECLARING_KINDS.has(kind)) return []
const parents = (await runner.graphParents?.([f.path])) ?? []

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] YAML declaration cards depend on test-node parents, not patch_path

const parents = (await runner.graphParents?.([f.path])) ?? []

graphParents links a YAML file to nodes via tests, while models are tied to their YAML by patch_path. A schema YAML with no generic tests gets no card even though the manifest has its models' config; removing the last test removes the card. The orchestrator test mocks graphParents, which hides this. Fix: resolve declarations through patch_path. (Codex, Claude)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point; the mocked graphParents hid this. c99cc33 adds a runner method, patchedModels(path), which returns the project's models, snapshots and seeds whose patch_path is the yml. A yml's cards are now the union of those and the tested models (still at most 3). The new test uses the real dispatcher runner with no test nodes, and the schema yml still gets its model's card.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Fixed — new patchedModels (runner.ts:554-569) uses patch_path, unioned with test parents in ai-context-card.ts:123-129; test runs against the real manifest reader without test nodes.

: fileChars < maxChars
? truncate(f.sql, fileChars)
: excerptFile(f.sql, f.diff, maxChars).text
if (f.sql.length > fileChars) stats = { ...stats, truncated: true }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Ledger reports truncated:false when MAX_FILE_CHARS < 2000

if (f.sql.length > fileChars) stats = { ...stats, truncated: true }

With ALTIMATE_REVIEW_AI_MAX_FILE_CHARS=1000 and 1,500-char SQL, fileChars is widened to 2000 but excerptFile(..., maxChars) still truncates, so the record says truncated:false while the prompt contains (truncated). Fix: compare against the budget actually applied (Math.min(maxChars, ...)). (Claude)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracing this. In c99cc33 the full-diff file budget is Math.min(COVERAGE_FULL_DIFF_FILE_CHARS, maxChars), so the truncated check uses the budget that was actually applied. A test with ALTIMATE_REVIEW_AI_MAX_FILE_CHARS=500 checks that the prompt shows (truncated) and the record says truncated: true.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Fixed — ai-review.ts:414 bounds the file head by Math.min(COVERAGE_FULL_DIFF_FILE_CHARS, maxChars); the ledger now reports truncated:true for the small-budget fixture.

...(manifestMissing && !anyManifest
? modelFiles.map((f) => ({ path: f.path, reason: "no dbt manifest: lineage, config and graph checks skipped" }))
: []),
...aiUnsent.map((p) => ({ path: p, reason: "not sent to the AI (file quota)" })),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Quota-omitted files listed under Not reviewed although engine lanes checked them

...aiUnsent.map((p) => ({ path: p, reason: "not sent to the AI (file quota)" })),

The ledger means "every changed file the review did not look at", but these files went through the deterministic lanes; on a large PR dozens of engine-checked files show up in that block. The default maxFiles cap records nothing, so behaviour is inconsistent. Fix: report as a separate note or field. (Claude)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, the ledger was the wrong place for them. Since c99cc33 these paths stay on the AI lane status (ai.unsent, now in the envelope schema) and appear as one summary note: "N file(s) were not sent to the advisory AI review (file quota); the deterministic checks still ran on them: ...". They no longer appear under "Not reviewed", and the orchestrator test checks both the note and the ledger.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Fixed — quota-omitted files are no longer in the Not reviewed ledger; they are carried as ai.unsent (verdict.ts:243) and shown as a separate AI-lane note (format.ts:285-294), with a test. The note wording is inaccurate for seeds (new inline comment).

@github-actions

Copy link
Copy Markdown

Thanks for updating your PR! It now meets our contributing guidelines. 👍

@github-actions

Copy link
Copy Markdown

👋 This PR was automatically closed by our quality checks.

Common reasons:

  • New GitHub account with limited contribution history
  • PR description doesn't meet our guidelines
  • Contribution appears to be AI-generated without meaningful review

If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you.

@RaghwendraSingh
RaghwendraSingh force-pushed the feat/review-recall-rules branch from 936b24a to b234d43 Compare November 20, 2026 08:22
@RaghwendraSingh
RaghwendraSingh force-pushed the feat/review-ai-input-coverage branch from bf6e861 to c99cc33 Compare November 20, 2026 08:22
@github-actions

Copy link
Copy Markdown

👋 This PR was automatically closed by our quality checks.

Common reasons:

  • New GitHub account with limited contribution history
  • PR description doesn't meet our guidelines
  • Contribution appears to be AI-generated without meaningful review

If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you.

2 similar comments
@github-actions

Copy link
Copy Markdown

👋 This PR was automatically closed by our quality checks.

Common reasons:

  • New GitHub account with limited contribution history
  • PR description doesn't meet our guidelines
  • Contribution appears to be AI-generated without meaningful review

If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you.

@github-actions

Copy link
Copy Markdown

👋 This PR was automatically closed by our quality checks.

Common reasons:

  • New GitHub account with limited contribution history
  • PR description doesn't meet our guidelines
  • Contribution appears to be AI-generated without meaningful review

If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you.

@RaghwendraSingh

Copy link
Copy Markdown
Contributor Author

Thanks for the approval and the detailed pass. All eight MINORs are addressed. The batchChars bound was already in a3c5f07; the other seven are fixed in c99cc33, each with a test.

From the nits, c99cc33 also covers:

  • quotas accept 0 to disable a bucket, and no spare quota flows into a disabled bucket;
  • MODEL_KINDS includes seed, matching the ModelCard doc;
  • the card-limit comment and the planAiInput purity comment are reworded.

Several of your missing tests are now added: dry-run with replay and verify, YAML cards through the real runner, Jinja-only changes on compiled SQL, and the quota note rendering.

Left as follow-ups:

  • the excerptFile anchor-first budget and the ~40-char overshoot;
  • the empty-preamble hunksSent count;
  • variant detection for x__y macros;
  • the naming nits.

These only affect excerpt quality behind the opt-in flags, and I preferred to keep this commit focused.

const head = aiUnsent.slice(0, 5).map(codeSpan).join(", ")
const more = aiUnsent.length > 5 ? ` and ${aiUnsent.length - 5} more` : ""
notes.push(
`${aiUnsent.length} file(s) were not sent to the advisory AI review (file quota); the deterministic checks still ran on them: ${head}${more}.`,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: The note asserts the deterministic checks ran on every quota-skipped file, but seed files never reach a deterministic lane. orchestrate.ts touches models (1919), snapshots (2215, 2283), macros (2248), tests/project config (2283), schema yml (2390) and modified dbt_project.yml (2421); no lane touches a .csv seed, and only AI-sent files are touched at 2594. With ALTIMATE_REVIEW_AI_COVERAGE=1, one out-of-quota seed (e.g. ALTIMATE_REVIEW_AI_QUOTA_YML=0, or 40 models + 10 seeds where the spare quota is absorbed by models) is reported here as "deterministic checks still ran", while coverageNotes also renders the ledger entry no checks run for seed files at the full tier (orchestrate.ts:2659-2661) for the same path - the same summary contradicts itself. Use the listed set computed at line 264: make this claim only for aiUnsent paths that are not in the ledger, or say that files without deterministic coverage are listed under "Not reviewed".

// A quota of 0 disables that bucket.
const quota = (v: string | undefined, d: number) => {
const n = Number(v)
return v && Number.isInteger(n) && n >= 0 ? n : d

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: quota() validates with Number() instead of a strict integer, so a whitespace or non-decimal value silently becomes a quota of 0 and disables the bucket: Number(" "), Number("0x0") and Number("0e0") are all 0, v is truthy, and n >= 0 accepts it. ALTIMATE_REVIEW_AI_QUOTA_MODEL=" " in a workflow then sends no model at all, while the neighbouring num() falls back to the default for the same input and the test states invalid values keep the default (-1 does). Conversely "1e3" is accepted as 1000. Require a trimmed decimal integer:

Suggested change
return v && Number.isInteger(n) && n >= 0 ? n : d
return typeof v === "string" && /^\d+$/.test(v.trim()) && Number.isSafeInteger(Number(v)) ? Number(v) : d

used += size(s, end) + MARKER
}
}
break

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: One window that does not fit stops the loop, so later windows are dropped while budget is left. Windows are sorted but their sizes vary with line length (and the last one is clipped at EOF): with 80-char lines and maxChars=6000, anchors at 100, 600 and 990 give windows costing about 3.4k, 3.4k and 2.6k - the loop breaks on the second (6.7k > 6.0k) even though the first and third fit in 5.95k, and the head fill below then spends the remainder above the first window. The SQL excerpt therefore omits the changed line at 990 unless the diff excerpt still carries that hunk (it is capped at diffChars). Let later windows that still fit be kept:

Suggested change
break
continue

const budget = aiInputBudget()
const files = selectAiFiles(input.files, budget.maxFiles)
if (!files.length) return done([], { state: "skipped", reason: "no changed dbt files to review" })
if (!files.length && (dryRun || !replay)) return done([], { state: "skipped", reason: "no changed dbt files to review" })

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: files is the quota-selected list, not the changed-file list, so when a quota excludes every sendable file the lane reports no changed dbt files to review even though files did change and are then listed as quota-skipped in the same summary. With ALTIMATE_REVIEW_AI_COVERAGE=1 ALTIMATE_REVIEW_AI_QUOTA_MACRO=0 and only macros/to_cents.sql changed, ai.unsent holds that file while the reason says there were no changed files. The check also precedes the dry-run branch (686), so a dry run with empty quotas reports the same false reason instead of the dry-run one. Choose the reason from plan.unsent.length and keep the dry-run reason first.

// Only the files the AI lane will send are reached (its file cap, or the
// per-kind quotas in coverage mode); the rest stay in the ledger.
const aiBudget = aiInputBudget()
touch(aiBudget.coverage ? selectAiFilesByQuota(aiFiles, aiBudget.coverage.quotas).sent : selectAiFiles(aiFiles, aiBudget.maxFiles))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: This repeats the selection branch owned by planAiInput (ai-review.ts:651-656: coverage means selectAiFilesByQuota(...).sent, otherwise selectAiFiles(..., maxFiles)). The ledger's accuracy depends on this selection matching exactly what the lane sends; if either copy changes, a file is silently marked reviewed here without being sent, or the reverse - the same drift this change fixes for the quota case. Extract one exported selector (e.g. selectAiInputFiles(files, budget) returning { files, unsent }) and call it from both places instead of maintaining the branch twice.

@hkaltai hkaltai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: approve. Panel: Claude, Codex and lead reviewer, re-review of c99cc33 (previously approved at b07b80d). review-ai-inputs.test.ts passes (42), the full review suite passes (827 per one reviewer) and typecheck is clean. With the flags unset the prompt is still byte-identical (snapshot, also checked against the base by a reviewer).

Counts: CRITICAL 0, MAJOR 0, MINOR 2 new (inline), NIT 6 new.

Status of prior MINOR findings (replied in-thread)

Finding Status
Replay before dry run Fixed
Compiled-SQL windows, no eligible added line Partially fixed (see new inline comment)
batchChars Infinity under coverage Fixed
modelCard first match across packages Fixed
macroCard basename before path Fixed
YAML cards rely on test parents Fixed (singular tests on seed/snapshot parents still model-only; nit)
Ledger truncated:false Fixed
Quota-omitted files under Not reviewed Fixed (note wording issue, new inline comment)

Prior nits: fixed: quota 0 parsing, seed in MODEL_KINDS, card-cap comment, "pure" doc. Not fixed (still nits/non-blocking): first oversized excerpt window can drop its anchor (ai-review.ts:342-351), __ macro-prefix variants (runner.ts:688-695), empty truncated hunk counted as sent (ai-review.ts:249-256), trailing marker not reserved (ai-review.ts:333), naming items.

New nits

  • ai-review.ts:672 const plan = planAiInput(input) sits outside the try in runAiReview (and now before replay), so a render failure breaks the "never throws" contract; the orchestrator catches it. Fix: wrap planning or plan after the replay branch.
  • ai-review.ts:684 if (!files.length && (dryRun || !replay)) return done([], { state: "skipped", reason: "no changed dbt files to review" }): when every quota is 0 all files are unsent but the reason says no changed files. Fix: distinct reason when plan.unsent is non-empty.
  • ai-review.ts:59 the quota parser maps " " (whitespace) to 0 and so disables a bucket. Fix: accept only trimmed decimal integer text.
  • runner.ts:44-46 pathMatches suffix match: macros/nested/macros/helpers.sql vs macros/helpers.sql can resolve to the wrong macro (inFile[0]). Fix: prefer exact project-relative path, then the most specific suffix.
  • ai-context-card.ts:127 singular-test parents still filter to kind === "model", so seed/snapshot parents get no card. Fix: include them.
  • ai-context-card.ts:115 card is labelled with f.model rather than the resolved node's name when a path match picks a differently named snapshot. Fix: use the resolved name.
  • runner.ts:625,643,686 .slice() caps run before rendering, so the card's +N more never shows (11 callers read as 10). Fix: return full lists and cap in the renderer.
  • Comment hygiene: COVERAGE_BATCH_CHARS doc hard-codes "30 files ... ~660k" arithmetic that will drift; excerptFile docstring does not mention the context-line fallback.
  • Naming unchanged: AiReviewFile.card (string) vs AiInputRecord.card (boolean); fileChars is a budget; CoverageArg tri-state test-only; local MARKER; isOther -> isSnapshotOrSeed.

const head = aiUnsent.slice(0, 5).map(codeSpan).join(", ")
const more = aiUnsent.length > 5 ? ` and ${aiUnsent.length - 5} more` : ""
notes.push(
`${aiUnsent.length} file(s) were not sent to the advisory AI review (file quota); the deterministic checks still ran on them: ${head}${more}.`,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] AI-unsent note claims deterministic checks ran on seeds, which they did not

`${aiUnsent.length} file(s) were not sent to the advisory AI review (file quota); the deterministic checks still ran on them: ${head}${more}.`

Seed files are only touched by the AI lane. With coverage on, a yml quota of 1 and two changed seeds, the summary says the deterministic checks still ran on them: seeds/b.csv while the ledger says no checks run for seed files at the full tier. Two reviewers reproduced this; the new note is false for that case.

Fix: list only unsent paths that were touched by a deterministic lane, or drop the 'deterministic checks still ran' clause; add a test with a quota-omitted seed.

const eligible = rows.filter((r) => r.kind === "+" && isEligible(r))
if (!eligible.length) {
const context = rows.filter((r) => r.kind !== "+" && isEligible(r))
return context.length && !context.every(matchesRaw) ? "unlocated" : undefined

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Compiled-SQL windows still trust raw line numbers when context is also Jinja/short

return context.length && !context.every(matchesRaw) ? "unlocated" : undefined

When the added lines are ineligible and the context lines are too (e.g. {% if is_incremental() %} / {% endif %} around one added line, or short lines like select, from t), context is empty, so this returns undefined and the raw window is used. On compiled SQL where the line sits ~300 lines lower, the excerpt contains neither the changed text nor the approximate marker. Reproduced by two reviewers. This is the remaining part of the original finding.

Fix: pass an isCompiled flag and return "unlocated" when compiled content differs from raw and neither added nor context lines can be checked; add a test for the no-eligible-context case.

RaghwendraSingh and others added 5 commits November 20, 2026 16:12
…ext card

All switches are off by default; with them unset the AI prompt is
byte-identical to before (pinned by a snapshot captured before the change).

- `ALTIMATE_REVIEW_AI_COVERAGE=1`: per-kind file quotas (models 12, macros 8,
  tests 4, yml/config 6; `ALTIMATE_REVIEW_AI_QUOTA_{MODEL,MACRO,TEST,YML}`),
  most-changed files first, unused quota redistributed up to the total.
  Over-budget diffs keep whole hunks and name omitted hunks by line range;
  over-budget files send +/-20-line windows around changed lines plus the file
  head, with `… (lines a–b omitted)` markers. Files left out go to
  `scope.notReviewed` as "not sent to the AI (file quota)".
- `ai.input` on the envelope records per file what was sent (chars, truncated,
  hunks sent/total, card) under coverage, context cards or dry run.
- `ALTIMATE_REVIEW_AI_DRYRUN=1`: the lane plans its input, records `ai.input`
  and returns `skipped` without calling a model. `planAiInput` is the pure
  planner `runAiReview` now sends from.
- `ALTIMATE_REVIEW_AI_CONTEXT_CARD=1` (needs a manifest): a `dbt context (from
  manifest):` block under each file header, from new non-throwing runner
  methods `modelCard` and `macroCard`; yml and singular tests get the cards of
  the models they test. At most 600 chars per file and 6,000 per call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… SQL

Under `ALTIMATE_REVIEW_AI_COVERAGE=1`, a model's current file is often
dbt-compiled SQL, whose line numbers drift from the raw diff once Jinja
expands. When the added lines are not at their raw line numbers, `excerptFile`
now finds each added line (8+ non-space chars, no Jinja) by its trimmed text,
scanning forward, and centres the windows there. If fewer than half are found
it keeps the head cut with a marker saying windows would be approximate. Raw
content keeps raw line numbers; the default prompt is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…header

Under `ALTIMATE_REVIEW_AI_COVERAGE=1` only (the default prompt is unchanged):

- The diff section gets its own budget, `ALTIMATE_REVIEW_AI_COVERAGE_DIFF_CHARS`
  (default 16,000), instead of the 6,000-char file budget. Hunks stay whole
  and in order; only hunks past that budget are listed as omitted. Multi-hunk
  macro diffs that sent 7 of 20 hunks now go whole.
- When a long diff (over the file budget) is sent with every hunk, the
  current-file section shrinks to a 2,000-char head; otherwise the file keeps
  its +/-20-line windows within the file budget.
- Fix: an excerpted diff replaced the whole file section (header, card and
  current file) because the excerpt's `text` overwrote the section text.
  `ai.input` chars, hunksSent and hunksTotal now describe what is really sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `aiInputBudget`: with `ALTIMATE_REVIEW_AI_COVERAGE=1`, `batchChars` now defaults
  to 120k chars instead of `Infinity`, so 30 quota files with 16k diffs no longer
  build one ~660k-char call. `ALTIMATE_REVIEW_AI_BATCH_CHARS` still overrides it;
  with coverage off the default stays `Infinity` (prompt snapshot unchanged).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…olution, compiled windows, quota note

- `runAiReview`: dry run is evaluated (and the input planned) before replay, so
  `ALTIMATE_REVIEW_AI_DRYRUN=1` with a replay envelope makes no verifier call and
  records `ai.input`.
- `excerptFile`: when no added line is eligible (Jinja-only or short changes),
  context lines decide whether raw line numbers hold; otherwise the head is sent
  with the "approximate" marker instead of windows on the wrong lines.
- `renderFile`: the full-diff file budget is capped at `maxFileChars`, so the
  `truncated` record matches what the prompt carries.
- `modelCard`: resolves the node defined in the changed file within the project,
  then by name in the project, then any package. `macroCard`: a macro defined in
  the changed file within the project first (the one named like the file), then
  by name in the project, then any package.
- Context cards: new `patchedModels` resolves a yml's models through
  `patch_path`, so a schema yml without generic tests still gets cards; seeds get
  their own card as the `ModelCard` doc says.
- Files over the coverage quotas are reported as an AI-lane note
  (`ai.unsent` in the envelope) instead of entries in "Not reviewed": the
  deterministic lanes still checked them.
- Coverage quotas accept 0 to disable a bucket (no spare quota flows into it).
- Comment fixes: card batch limit wording, `planAiInput` purity.
- Restack: the coverage ledger marks reached exactly the files the AI lane sends (per-kind
  quotas in coverage mode, the file cap otherwise), with a test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@RaghwendraSingh
RaghwendraSingh force-pushed the feat/review-recall-rules branch from b234d43 to 3d47982 Compare November 20, 2026 11:37
@RaghwendraSingh
RaghwendraSingh force-pushed the feat/review-ai-input-coverage branch from c99cc33 to 31b384c Compare November 20, 2026 11:37
@github-actions

Copy link
Copy Markdown

👋 This PR was automatically closed by our quality checks.

Common reasons:

  • New GitHub account with limited contribution history
  • PR description doesn't meet our guidelines
  • Contribution appears to be AI-generated without meaningful review

If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you.

1 similar comment
@github-actions

Copy link
Copy Markdown

👋 This PR was automatically closed by our quality checks.

Common reasons:

  • New GitHub account with limited contribution history
  • PR description doesn't meet our guidelines
  • Contribution appears to be AI-generated without meaningful review

If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants