Skip to content

feat(review): opt-in AI verifier that confirms or refutes each AI comment from dbt evidence - #1769

Open
RaghwendraSingh wants to merge 14 commits into
feat/review-posting-policyfrom
feat/review-ai-verifier
Open

RaghwendraSingh wants to merge 14 commits into
feat/review-posting-policyfrom
feat/review-ai-verifier

Conversation

@RaghwendraSingh

@RaghwendraSingh RaghwendraSingh commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Issue for this PR

Part of #1792

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Stacked on #1758. Adds an opt-in verifier for the AI review lane (ALTIMATE_REVIEW_AI_VERIFY=1). After the AI reviewer drafts its comments, one more bounded model pass tries to confirm or refute each one from dbt evidence. Comments it can't support are moved to the summary or dropped. Off by default: with the flag unset, the review output is unchanged.

Results (replay of public dbt PRs, blind Claude judge)

Verifier on vs off, same runs (50 PRs, 59 reviewer-found defects, tuning repos)

AI comments per PR Useful Wrong
Verifier off 2.48 45% 1%
Verifier on 1.32 53% 0%
  • On 26 PRs from repos never used for tuning: useful 38% → 59%, wrong 1 → 0.
  • The first version dropped 3 real catches by discarding cross-file claims it could not check. Those claims now go to the summary instead of being dropped (12829821cd), so catches are not lost.

Repos never used for any tuning (25 repos, 101 PRs, one snapshot each, 153 defects)

Build Defects caught AI comments useful / wrong
Released v0.12.11 with its AI 24 34% / 3.6%
This stack, AI + verifier + rules 35 43% / 1.3%

Difference +11 catches (95% CI +1 to +22, PR-level bootstrap). The verifier costs about 1.4× the tokens of the AI review alone.

What changed

  • ai-verify.ts:
    • Deterministic features: evidence line valid, targets unchanged code, restates an engine finding, claims another file, style only, low confidence, hedged data-diff.
    • The verifier prompt and a tolerant parser.
    • A bounded tool loop: read_file, compiled_sql, model_columns, column_reach, model_config, downstream, equivalence, grain, grep_project, pr_description. At most 3 rounds and 12k tokens.
    • Table-driven triage into keep / summary / discard.
  • Fails open: on an error, timeout or missing verdict, the original findings stay.
  • Verifier decisions, tokens and dropped findings are recorded in the envelope and telemetry.
  • ALTIMATE_REVIEW_AI_REPLAY=<envelope> replays stored generator findings so only the verifier is re-run.
  • Runner gains columnReach, and the stream reader keeps tool parts.

How did you verify your code works?

  • bun test test/altimate/review: 667 pass.
  • bunx tsc --noEmit: clean.
  • New: test/altimate/review-ai-verify.test.ts (features, every triage row and knob, tool loop limits, allowlist, fail-open, parser).

Cost side effects

None by default: the verifier is opt-in (ALTIMATE_REVIEW_AI_VERIFY=1). When enabled it adds one bounded model pass per review (at most 3 tool rounds and 12k tokens), measured at about 1.4x the AI-lane tokens of the AI review alone. Its tools read local project files, the manifest and the in-process engine; no warehouse queries. Verifier tokens are recorded in the envelope and telemetry.

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

Changes since review: a74e400684 addresses the second-round review feedback: verifier summary demotion checked before config tiers, replay fails open with no verifier model, evidence-less refuted judged unverifiable, realistic tool-round budget (12k cap documented as soft), one deadline over graph and tool calls.

🤖 Generated with Claude Code


Summary by cubic

Adds an opt-in, dbt-grounded verifier for the AI review lane. With ALTIMATE_REVIEW_AI_VERIFY=1, one bounded model pass after the AI reviewer tries to confirm or refute each comment from dbt evidence (compiled SQL, DAG lineage, project grep); comments it can't support move to the summary or are dropped. With the flag unset the review output is unchanged. The verifier fails open: any error, timeout, or exhausted budget leaves the findings it didn't review untouched.

Results (blind judge on public dbt PRs)

AI comments per PR (% useful / % wrong)
Verifier off 2.48 (45% / 1%)
Verifier on 1.32 (53% / 0%)

On 26 PRs from repos never used for tuning, useful went 38% → 59% and wrong went 1 → 0. Cross-file and speculative claims the verifier can't check go to the summary instead of being dropped, so real catches aren't lost. The verifier costs about 1.4× the tokens of the AI review alone.

What changed

  • Deterministic features per finding (evidence line valid, unchanged code, engine grounding, cross-file claim, style only, low confidence) and a tolerant verdict parser feeding table-driven triage into keep / summary / discard.
  • A bounded loop of 10 read-only tools, at most 3 rounds and 12k tokens, with a forced answer when the budget is spent.
  • Verifier decisions and tokens are recorded in the envelope and telemetry; discarded findings keep their line and body, so one run doubles as its no-verifier baseline, and retriage() re-runs triage offline with hypothetical knobs and no model call.
  • ALTIMATE_REVIEW_AI_REPLAY=<envelope> replays stored generator findings so verifier settings can be compared without regenerating.
  • grepProject and columnReach are wired into the runner; failed searches, timeouts, and unknown models report as "no evidence" rather than engine failures.
  • Coverage: 667 tests pass, including new suites for features, every triage row and knob, tool loop limits, fail-open, and replay.

Written for commit 08af665. Summary will update on new commits.

View guided diff Turn on auto-fix

@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: bf5d10cb-940a-4060-9df0-13ad1a136067

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8 issues found across 16 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/opencode/src/altimate/telemetry/index.ts">

<violation number="1" location="packages/opencode/src/altimate/telemetry/index.ts:1625">
P2: The verifier's USD cost is recorded in the envelope but never reaches telemetry. Add an `ai_verify_cost` field here and map it from `env.ai?.verifier?.cost` in review/telemetry.ts, so verifier spend is measurable alongside its tokens.</violation>
</file>

<file name="packages/opencode/src/altimate/review/git.ts">

<violation number="1" location="packages/opencode/src/altimate/review/git.ts:207">
P3: A valid filename like `models/a:123:b.sql` is split at the filename’s `:123:` and returned with a false path and line. Parse an unambiguous path/line format instead.</violation>
</file>

<file name="packages/opencode/src/altimate/review/ai-verify.ts">

<violation number="1" location="packages/opencode/src/altimate/review/ai-verify.ts:198">
P2: A finding with no usable line and no quoted or backticked identifier is classified as targeting unchanged code, so triage discards it as `unchanged_code` unless the model confirmed it. An empty identifier list should not count as evidence of unchanged code. Require at least one identifier before marking it unchanged.</violation>

<violation number="2" location="packages/opencode/src/altimate/review/ai-verify.ts:1003">
P2: When a batch's share cannot cover even its direct-answer estimate, `force` still sends that call; repeated across batches, verifier usage can exceed the 12k PR budget. Skip that batch as budget-exhausted when the answer itself cannot fit, leaving its findings unreviewed.</violation>
</file>

<file name="packages/opencode/src/altimate/llm/stream-text.ts">

<violation number="1" location="packages/opencode/src/altimate/llm/stream-text.ts:32">
P3: `readStreamStep` adds a new error-propagation path and an `invalid` passthrough, but no test covers either. Add cases to stream-text.test.ts for a mid-stream `error` part (must throw) and for a tool call with `invalid: true`.</violation>
</file>

<file name="packages/opencode/src/altimate/review/verdict.ts">

<violation number="1" location="packages/opencode/src/altimate/review/verdict.ts:237">
P3: The `dropped` schema accepts any decision and any length, but the producer only records discards and caps the list at 20. Narrow `decision` to `z.literal("discard")` and add `.max(20)` to the array so the schema states the invariant.</violation>
</file>

<file name="packages/opencode/src/altimate/review/ai-review.ts">

<violation number="1" location="packages/opencode/src/altimate/review/ai-review.ts:312">
P2: With the verifier on, lane-level `tokens`, `cost` and `calls` cover only the generator, so `ai_cost`/`ai_tokens_total`/`ai_calls` in telemetry undercount total AI spend. Either add the verifier's usage into these totals or document that they exclude it, and update the `tokens`/`calls` JSDoc to match.</violation>
</file>

<file name="packages/opencode/src/altimate/review/ai-replay.ts">

<violation number="1" location="packages/opencode/src/altimate/review/ai-replay.ts:121">
P3: Discarded findings are rebuilt from the envelope's `body`, which `applyTriage` clamps to 600 characters. The replayed body is truncated, and the verifier and output see it that way without any `replayIncomplete` flag. Note the truncation in the replay status, or stop clamping for replayed records.</violation>
</file>

Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

View guided diff | Turn on auto-fix | Re-trigger cubic

.filter((l) => l.kind === "+" || added.has(l.line))
.map((l) => l.text.toLowerCase())
const ids = quotedNames(text)
targetsUnchangedCode = !ids.some((id) => addedText.some((t) => t.includes(id)))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: A finding with no usable line and no quoted or backticked identifier is classified as targeting unchanged code, so triage discards it as unchanged_code unless the model confirmed it. An empty identifier list should not count as evidence of unchanged code. Require at least one identifier before marking it unchanged.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-verify.ts, line 198:

<comment>A finding with no usable line and no quoted or backticked identifier is classified as targeting unchanged code, so triage discards it as `unchanged_code` unless the model confirmed it. An empty identifier list should not count as evidence of unchanged code. Require at least one identifier before marking it unchanged.</comment>

<file context>
@@ -0,0 +1,1230 @@
+      .filter((l) => l.kind === "+" || added.has(l.line))
+      .map((l) => l.text.toLowerCase())
+    const ids = quotedNames(text)
+    targetsUnchangedCode = !ids.some((id) => addedText.some((t) => t.includes(id)))
+  }
+
</file context>
Suggested change
targetsUnchangedCode = !ids.some((id) => addedText.some((t) => t.includes(id)))
targetsUnchangedCode = ids.length > 0 && !ids.some((id) => addedText.some((t) => t.includes(id)))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. This matches the stated policy, but requiring ids.length > 0 is the safer reading; queued.

ai_verify_tool_calls?: number
/** AI findings the verifier kept (including those moved to the summary) and discarded. */
ai_verify_kept?: number
ai_verify_discarded?: number

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The verifier's USD cost is recorded in the envelope but never reaches telemetry. Add an ai_verify_cost field here and map it from env.ai?.verifier?.cost in review/telemetry.ts, so verifier spend is measurable alongside its tokens.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/telemetry/index.ts, line 1625:

<comment>The verifier's USD cost is recorded in the envelope but never reaches telemetry. Add an `ai_verify_cost` field here and map it from `env.ai?.verifier?.cost` in review/telemetry.ts, so verifier spend is measurable alongside its tokens.</comment>

<file context>
@@ -1613,6 +1613,16 @@ export namespace Telemetry {
+        ai_verify_tool_calls?: number
+        /** AI findings the verifier kept (including those moved to the summary) and discarded. */
+        ai_verify_kept?: number
+        ai_verify_discarded?: number
         /** Where the review base came from (flag, github_event, default_branch, …). */
         base_source?: string
</file context>
Suggested change
ai_verify_discarded?: number
ai_verify_discarded?: number
/** Estimated USD cost of the verifier from the model's price table. */
ai_verify_cost?: number

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. Tokens are already reported; we'll add ai_verify_cost.

findings = v.findings
verifier = v.status
}
return done(findings, { state: "ok", model: modelLabel, ...(reason ? { reason } : {}), ...spent, ...(verifier ? { verifier } : {}) })

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: With the verifier on, lane-level tokens, cost and calls cover only the generator, so ai_cost/ai_tokens_total/ai_calls in telemetry undercount total AI spend. Either add the verifier's usage into these totals or document that they exclude it, and update the tokens/calls JSDoc to match.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-review.ts, line 312:

<comment>With the verifier on, lane-level `tokens`, `cost` and `calls` cover only the generator, so `ai_cost`/`ai_tokens_total`/`ai_calls` in telemetry undercount total AI spend. Either add the verifier's usage into these totals or document that they exclude it, and update the `tokens`/`calls` JSDoc to match.</comment>

<file context>
@@ -281,15 +294,22 @@ export async function runAiReview(input: AiReviewInput): Promise<AiReviewResult>
+      findings = v.findings
+      verifier = v.status
+    }
+    return done(findings, { state: "ok", model: modelLabel, ...(reason ? { reason } : {}), ...spent, ...(verifier ? { verifier } : {}) })
   } catch (err) {
     log.error("ai review failed", { error: err })
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing: the verifier is reported separately as ai_verify_tokens_* so lane and verifier spend stay distinguishable.

const roundOutput = Math.min(VERIFY_MAX_CALLS_PER_ROUND * TOOL_OUTPUT_MAX, ctx.maxToolChars - toolChars)
const roomForRound = tokens + nextCall() + nextCall(roundOutput) <= ctx.maxTokens
const overBudget = !roomForRound || toolChars > ctx.maxToolChars
const force = rounds >= ctx.maxRounds || overBudget

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: When a batch's share cannot cover even its direct-answer estimate, force still sends that call; repeated across batches, verifier usage can exceed the 12k PR budget. Skip that batch as budget-exhausted when the answer itself cannot fit, leaving its findings unreviewed.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-verify.ts, line 1003:

<comment>When a batch's share cannot cover even its direct-answer estimate, `force` still sends that call; repeated across batches, verifier usage can exceed the 12k PR budget. Skip that batch as budget-exhausted when the answer itself cannot fit, leaving its findings unreviewed.</comment>

<file context>
@@ -0,0 +1,1230 @@
+      const roundOutput = Math.min(VERIFY_MAX_CALLS_PER_ROUND * TOOL_OUTPUT_MAX, ctx.maxToolChars - toolChars)
+      const roomForRound = tokens + nextCall() + nextCall(roundOutput) <= ctx.maxTokens
+      const overBudget = !roomForRound || toolChars > ctx.maxToolChars
+      const force = rounds >= ctx.maxRounds || overBudget
+      if (overBudget && rounds < ctx.maxRounds) out.state = "budget"
+      if (force && rounds > 0) messages.push({ role: "user", content: ANSWER_NOW })
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. We'll skip the batch as budget-exhausted.

* by the SDK; the caller runs them and appends the results. Like
* `readStreamText`, a stream error is thrown rather than lost.
*/
export async function readStreamStep(stream: {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: readStreamStep adds a new error-propagation path and an invalid passthrough, but no test covers either. Add cases to stream-text.test.ts for a mid-stream error part (must throw) and for a tool call with invalid: true.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/llm/stream-text.ts, line 32:

<comment>`readStreamStep` adds a new error-propagation path and an `invalid` passthrough, but no test covers either. Add cases to stream-text.test.ts for a mid-stream `error` part (must throw) and for a tool call with `invalid: true`.</comment>

<file context>
@@ -12,3 +12,42 @@ export async function readStreamText(stream: {
+ * by the SDK; the caller runs them and appends the results. Like
+ * `readStreamText`, a stream error is thrown rather than lost.
+ */
+export async function readStreamStep(stream: {
+  fullStream: AsyncIterable<{ type: string; error?: unknown }>
+  readonly text: PromiseLike<string>
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing: nit. It is exercised through the verifier tests.

summary: z.number().int().nonnegative().optional(),
unreviewed: z.number().int().nonnegative().optional(),
/** Discarded findings (at most 20), for auditing the verifier's precision. */
dropped: z

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The dropped schema accepts any decision and any length, but the producer only records discards and caps the list at 20. Narrow decision to z.literal("discard") and add .max(20) to the array so the schema states the invariant.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/verdict.ts, line 237:

<comment>The `dropped` schema accepts any decision and any length, but the producer only records discards and caps the list at 20. Narrow `decision` to `z.literal("discard")` and add `.max(20)` to the array so the schema states the invariant.</comment>

<file context>
@@ -201,18 +210,80 @@ export const VerdictEnvelope = z.object({
+          summary: z.number().int().nonnegative().optional(),
+          unreviewed: z.number().int().nonnegative().optional(),
+          /** Discarded findings (at most 20), for auditing the verifier's precision. */
+          dropped: z
+            .array(
+              z.object({
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing: nit (verdict.ts:237, schema narrowing).

severity: d.severity === "warning" ? "warning" : "suggestion",
category: category.data,
title: d.title,
body: d.body,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: Discarded findings are rebuilt from the envelope's body, which applyTriage clamps to 600 characters. The replayed body is truncated, and the verifier and output see it that way without any replayIncomplete flag. Note the truncation in the replay status, or stop clamping for replayed records.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-replay.ts, line 121:

<comment>Discarded findings are rebuilt from the envelope's `body`, which `applyTriage` clamps to 600 characters. The replayed body is truncated, and the verifier and output see it that way without any `replayIncomplete` flag. Note the truncation in the replay status, or stop clamping for replayed records.</comment>

<file context>
@@ -0,0 +1,203 @@
+        severity: d.severity === "warning" ? "warning" : "suggestion",
+        category: category.data,
+        title: d.title,
+        body: d.body,
+        file: d.file,
+        model: modelNameFromPath(d.file),
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, low priority (ai-replay.ts:121, clamped bodies not flagged). Leaving this for a follow-up PR rather than widening this one.

const hits: Array<{ path: string; line: number; text: string }> = []
for (const row of out.split("\n")) {
const rest = ref && row.startsWith(ref + ":") ? row.slice(ref.length + 1) : row
const m = /^(.+?):(\d+):(.*)$/.exec(rest)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: A valid filename like models/a:123:b.sql is split at the filename’s :123: and returned with a false path and line. Parse an unambiguous path/line format instead.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/git.ts, line 207:

<comment>A valid filename like `models/a:123:b.sql` is split at the filename’s `:123:` and returned with a false path and line. Parse an unambiguous path/line format instead.</comment>

<file context>
@@ -178,6 +178,40 @@ export async function grepMatchesAt(cwd: string, ref: string | undefined, patter
+  const hits: Array<{ path: string; line: number; text: string }> = []
+  for (const row of out.split("\n")) {
+    const rest = ref && row.startsWith(ref + ":") ? row.slice(ref.length + 1) : row
+    const m = /^(.+?):(\d+):(.*)$/.exec(rest)
+    if (!m) continue
+    hits.push({ path: m[1], line: Number(m[2]), text: m[3].slice(0, 300) })
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing: nit. dbt project file names with :<digits>: aren't a realistic case.

compiledSql: async (p, side) => {
const ctx = ctxByPath.get(p)
const engine = side === "new" ? ctx?.engineNewSql : ctx?.engineOldSql
return engine ?? (await getCompiled?.(ctx?.file.oldPath && side === "old" ? ctx.file.oldPath : p, side))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: compiled_sql falls back to on-disk compiled artifacts for a side the PR does not have

engineNewSql/engineOldSql are deliberately undefined for a deleted head or an added base (see lines 1844-1856, which never call getCompiled for status === "deleted" or without an old side). This new closure ignores that: ctx exists for a deleted model with engineNewSql === undefined, so it calls getCompiled(p, "new"), which reads target/compiled/<project>/<path> from disk with no ref or status check (compiled.ts:108-121). dbt compile does not purge stale artifacts, so compiled_sql(path, "new") — and equivalence() built on it — can show the deleted model's old SQL as head, letting the verifier confirm a claim about code that no longer exists while read_file(path, "new") correctly reports it missing.

Return engine when ctxByPath has the path (even when it is undefined) and use getCompiled only for parents/downstream models outside that map.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. When the engine context has the path, we'll return its value even if it is undefined.

const near = (r: { head?: number }, i: number) => {
if (typeof line !== "number") return i < 2 * HUNK_RADIUS
// A removed row or hunk header sits beside the head row that follows it.
const at = r.head ?? rows.slice(i).find((x) => x.head !== undefined)?.head ?? right

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: hunkAround anchors removed rows to the next hunk and drops blank context rows

rows.slice(i).find(...) scans the whole accumulated row list, not the current hunk. A hunk that ends in deletions has no following head row of its own, so its - rows anchor to the next hunk's first head line: they are omitted from the excerpt for a finding in that hunk (the deleted code is missing exactly where the finding points) and included in a distant hunk's excerpt as if adjacent. Separately, row === "" (a blank context line that lost its leading space) is skipped entirely, so right is not incremented and every later row in the excerpt is numbered one too low; inHunk is also never reset for unrecognized rows.

walkDiffRows (diff-filter.ts:349-372) already numbers +, context, - and blank rows correctly — reuse it and mark hunk-header rows separately instead of re-deriving the walk here. The verifier decides confirm/refute from this excerpt, so a missing deletion or shifted numbers can refute a valid comment.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. We'll reuse walkDiffRows.

const dirs = layout
? [...layout.dirs.model, ...layout.dirs.macro, ...layout.dirs.seed, ...layout.dirs.snapshot, ...layout.dirs.test]
: ["models", "macros", "seeds", "snapshots", "tests"]
const specs = glob ? [`${root}${glob}`] : dirs.map((d) => path.posix.normalize(root + d))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: A repo-relative glob is prefixed with the project root a second time

The verifier is shown repo-relative paths (file: ${f.file} in buildVerifyMessage, and grep_project results are repo-relative), so in a monorepo the model naturally passes glob: "dbt/models/**". root + glob then becomes dbt/dbt/models/**, git matches nothing, and the tool returns the literal "no matches" — which the verifier can read as proof of absence and use to refute a correct cross-file claim.

Normalize the glob, only add projectRootRel when it is not already present, and reject any .. segment (dbt/../secrets/** currently passes the startsWith("..") filter). Keep the resulting spec inside projectRootRel.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f3c5e86: the new projectGrepSpecs doesn't prefix a glob that already carries the project root and rejects .. segments. A failed search is now reported as failed, too. Tests added.

return { decision: "keep", rule: "plausible_local" }
if (m.verdict === "unverifiable") return { decision: "discard", rule: "unverifiable" }
if (m.value === "informational") return { decision: "summary", rule: "informational" }
return { decision: "keep", rule: "verified" }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: A refuted verdict with no cited evidence is recorded as verified

Row 2 only discards a refutation that carries evidence (m.evidence.trim()), so an answer like {"verdict":"refuted","evidence":""} falls through every later rule and reaches this unconditional return. applyTriage then writes verification: { decision: "keep", rule: "verified", verdict: "refuted" } into the signed envelope and into decisions[], which is contradictory audit data for offline re-triage. (No Verified: suffix is appended because that also needs evidence, so this is a labeling defect rather than posted text.)

Make the last row conditional on m.verdict === "confirmed" and use a distinct rule such as refuted_unsupported otherwise.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. We'll record refuted_unsupported.

return s.length > TOOL_OUTPUT_MAX ? s.slice(0, TOOL_OUTPUT_MAX) + "\n… (truncated)" : s
}

function safeReason(err: unknown): string {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Duplicate helpers instead of reusing the AI lane's

safeReason here is byte-for-byte the same as ai-review.ts:190; positiveNumber duplicates aiInputBudget's num (ai-review.ts:37-40); and BATCH_CONCURRENCY is declared in both modules. Export the existing helpers and import them so key/bearer-token masking and env parsing cannot drift between the two lanes. backticked() in this file is also a strict subset of quotedNames() and can be dropped.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing in this PR: nit. We'll consolidate in a cleanup.

}
let body = f.body
const suffix = verification?.rule === "verified" && verification.evidence ? `\n\nVerified: ${verification.evidence}` : ""
if (suffix && body.endsWith(suffix)) body = body.slice(0, -suffix.length)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: unverify only strips a Verified: suffix that is the tail of the body, which mergeRepeats breaks

applyTriage appends the suffix, but cleanFindings then runs mergeRepeats (finding.ts:291-295), which rewrites the first finding of a group to ${first.body}\n\nSame rule, also here:…. If a verified finding wins that tie-break, the stored body is …Verified: <evidence>\n\nSame rule, also here:…, so body.endsWith(suffix) is false and replay keeps a Verified: claim the new verifier never made (it is fed f.body by buildVerifyMessage/extractFeatures, and is posted if re-kept). The merged (+N more) title is likewise not undone.

Remove the last occurrence of the suffix instead of requiring a tail match, and strip the (+N more) marker when rebuilding the generator's finding.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, low priority (ai-replay.ts:156, unverify suffix). Leaving this for a follow-up PR rather than widening this one.

@kilo-code-bot

kilo-code-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Code Review Summary

Status: 14 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 4
SUGGESTION 10
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/orchestrate.ts 2499 The verifier shares the deterministic lanes' runner, so a verifier-only engine failure (e.g. the new column_reach tool) is added to engineErrors after the AI lane and turns an APPROVE into COMMENT, contradicting the verifier's fail-open contract.
packages/opencode/src/altimate/review/ai-verify.ts 923 grep_project accepts any glob inside the dbt project (any tracked repo file when the project root is the repo root), so profiles.yml/.env-style credentials the AI lane deliberately withholds can be pulled into the verifier prompt.
packages/opencode/src/altimate/review/ai-verify.ts 818 read_file maps every undefined from getContent (git timeout/bad ref, escaping symlink) to "does not exist on that side", giving the verifier a false refutation it treats as evidence.
packages/opencode/src/altimate/review/orchestrate.ts 2508 compiled_sql falls back to the on-disk target/compiled artifact for a side the PR does not have, so a deleted model's stale head SQL can be shown to the verifier (and used by the equivalence lane).

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/ai-review.ts 320 A fail-closed verifier error/timeout returns no findings but the lane still reports state: "ok", and the PR comment never reads the nested verifier status, so the dropped AI findings are invisible.
packages/opencode/src/altimate/telemetry/index.ts 1623 The JSDoc says ai_verify_kept includes summary findings, but applyTriage counts summary separately and telemetry emits kept/discarded only, so kept+discarded undercounts reviewed findings.
packages/opencode/src/altimate/llm/stream-text.ts 51 On a stream error, readStreamStep throws before awaiting finishReason/response; unlike .text these SDK promises are not guarded, so a rejection becomes an unhandled rejection.
packages/opencode/src/altimate/review/ai-replay.ts 138 Verifier decisions use pre-merge ids while envelope findings use synthetic merged ids, so any merged AI repeat is reported missing and sets replayIncomplete on every replay.
packages/opencode/src/altimate/review/ai-verify.ts 1024 roundOutput goes negative once the tool-output budget is spent, so nextCall(roundOutput) subtracts estimated input tokens; only the sibling toolChars > maxToolChars term still forces the answer. Clamp it at 0.
packages/opencode/src/altimate/review/git.ts 223 With no ref, grep_project searches tracked files only, so a new untracked file is invisible and the search reads as "no matches"; grepMatchesAt already passes --untracked for the working tree.
packages/opencode/src/altimate/review/ai-verify.ts 373 hunkAround re-implements diff numbering and anchors removed rows (and hunk headers) to the next head row anywhere later in the diff; walkDiffRows already numbers every row and carries the hunk id needed to stay inside its own hunk.
packages/opencode/src/altimate/review/ai-verify.ts 758 safeReason duplicates the AI lane's exported helper but omits redactSecrets; positiveNumber and BATCH_CONCURRENCY are also duplicated, and backticked() is a subset of quotedNames().
packages/opencode/src/altimate/review/ai-replay.ts 159 unverify only strips a Verified: suffix at the end of the body; mergeRepeats appends after it, so replay keeps stale Verified: text and the (+N more) title.
packages/opencode/src/altimate/review/git.ts 257 projectGrepSpecs keeps the trailing slash on root, so glob equal to the project root ("dbt") becomes the pathspec dbt/dbt and never searches the project.
Files Reviewed (16 files)
  • packages/opencode/src/altimate/llm/stream-text.ts - 1 issue
  • packages/opencode/src/altimate/review/ai-replay.ts - 2 issues
  • packages/opencode/src/altimate/review/ai-review.ts - 1 issue
  • packages/opencode/src/altimate/review/ai-verify.ts - 5 issues
  • packages/opencode/src/altimate/review/finding.ts - clean
  • packages/opencode/src/altimate/review/format.ts - clean
  • packages/opencode/src/altimate/review/git.ts - 2 issues
  • packages/opencode/src/altimate/review/orchestrate.ts - 2 issues
  • packages/opencode/src/altimate/review/run.ts - clean
  • packages/opencode/src/altimate/review/runner.ts - clean
  • packages/opencode/src/altimate/review/telemetry.ts - clean
  • packages/opencode/src/altimate/review/verdict.ts - clean
  • packages/opencode/src/altimate/telemetry/index.ts - 1 issue
  • packages/opencode/test/altimate/review-ai-inputs.test.ts - clean
  • packages/opencode/test/altimate/review-ai-replay.test.ts - clean
  • packages/opencode/test/altimate/review-ai-verify.test.ts - clean
Previous Review Summaries (3 snapshots, latest commit a74e400)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit a74e400)

Status: 7 Issues Found | Recommendation: Address before merge

Incremental review of a74e400 (the follow-up to f3c5e86). The fixes for verifier demotion, replay fail-open (findings kept, verifier: skipped), refuted-without-evidence, the now-soft token budget, the timeout covering evidence tools, the columnReach unknown-model guard, and grep failure-versus-empty all check out, and re-checking them turned up no new correctness defect. The issues below remain open: one prior warning, two new suggestions, and three prior suggestions the author has deferred.

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 6
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/orchestrate.ts 2502 compiled_sql falls through to the on-disk target/compiled artifact for a side the PR does not have, so a deleted model's stale head SQL can be shown to the verifier (and used by the equivalence lane).

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/ai-verify.ts 1024 roundOutput goes negative once the tool-output budget is spent, so nextCall(roundOutput) subtracts estimated input tokens; only the sibling toolChars > maxToolChars term still forces the answer. Clamp it at 0.
packages/opencode/src/altimate/review/git.ts 223 With no ref, grep_project searches tracked files only, so a new untracked file is invisible and the search reads as "no matches"; grepMatchesAt already passes --untracked for the working tree.
packages/opencode/src/altimate/review/ai-verify.ts 373 hunkAround re-implements diff numbering and anchors removed rows (and hunk headers) to the next head row anywhere later in the diff; walkDiffRows already numbers every row and carries the hunk id needed to stay inside its own hunk.
packages/opencode/src/altimate/review/ai-verify.ts 758 safeReason duplicates the AI lane's exported helper but omits redactSecrets; positiveNumber and BATCH_CONCURRENCY are also duplicated, and backticked() is a subset of quotedNames().
packages/opencode/src/altimate/review/ai-replay.ts 159 unverify only strips a Verified: suffix at the end of the body; mergeRepeats appends after it, so replay keeps stale Verified: text and the (+N more) title.
packages/opencode/src/altimate/review/git.ts 257 projectGrepSpecs keeps the trailing slash on root, so glob equal to the project root ("dbt") becomes the pathspec dbt/dbt and never searches the project.
Files Reviewed (16 files)
  • packages/opencode/src/altimate/llm/stream-text.ts - clean
  • packages/opencode/src/altimate/review/ai-replay.ts - 1 issue
  • packages/opencode/src/altimate/review/ai-review.ts - clean
  • packages/opencode/src/altimate/review/ai-verify.ts - 3 issues
  • packages/opencode/src/altimate/review/finding.ts - clean
  • packages/opencode/src/altimate/review/format.ts - clean
  • packages/opencode/src/altimate/review/git.ts - 2 issues
  • packages/opencode/src/altimate/review/orchestrate.ts - 1 issue
  • packages/opencode/src/altimate/review/run.ts - clean
  • packages/opencode/src/altimate/review/runner.ts - clean
  • packages/opencode/src/altimate/review/telemetry.ts - clean
  • packages/opencode/src/altimate/review/verdict.ts - clean
  • packages/opencode/src/altimate/telemetry/index.ts - clean
  • packages/opencode/test/altimate/review-ai-inputs.test.ts - clean
  • packages/opencode/test/altimate/review-ai-replay.test.ts - clean
  • packages/opencode/test/altimate/review-ai-verify.test.ts - clean

Previous review (commit f3c5e86)

Status: 6 Issues Found | Recommendation: Address before merge

Incremental review of f3c5e86 (the grep_project / column_reach follow-up), with prior findings re-checked against current HEAD. The grep_project failure-vs-empty fix, the projectGrepSpecs double-prefix/.. fix, and the unknown-model columnReach handling all check out; the issues below remain open.

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 4
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/orchestrate.ts 2484 compiled_sql falls back to on-disk target/compiled artifacts for a side the PR does not have, so a deleted model's stale head SQL can be shown to the verifier (and used by equivalence).
packages/opencode/src/altimate/review/ai-verify.ts 370 hunkAround anchors removed rows to the next hunk and drops blank context rows, shifting head-side line numbers in the excerpt the verifier decides from. walkDiffRows already does this correctly.

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/ai-verify.ts 738 The verifier's safeReason duplicates the AI lane's exported helper but omits redactSecrets, so its error strings mask less; positiveNumber and BATCH_CONCURRENCY are also duplicated, and backticked() is a subset of quotedNames().
packages/opencode/src/altimate/review/ai-verify.ts 473 A refuted verdict with no cited evidence falls through to keep/verified, recording a self-contradictory verification in the envelope.
packages/opencode/src/altimate/review/ai-replay.ts 156 unverify only strips a Verified: suffix at the end of the body; mergeRepeats appends after it, so replay keeps stale Verified: text and the (+N more) title.
packages/opencode/src/altimate/review/git.ts 253 projectGrepSpecs keeps the trailing slash on root, so glob equal to the project root ("dbt") becomes the pathspec dbt/dbt and never searches the project.
Files Reviewed (7 files)
  • packages/opencode/src/altimate/review/git.ts - 1 issue
  • packages/opencode/src/altimate/review/run.ts - clean
  • packages/opencode/src/altimate/review/runner.ts - clean
  • packages/opencode/src/altimate/review/ai-verify.ts - 3 issues
  • packages/opencode/src/altimate/review/ai-replay.ts - 1 issue
  • packages/opencode/src/altimate/review/orchestrate.ts - 1 issue
  • packages/opencode/test/altimate/review-ai-verify.test.ts - clean

Fix these issues in Kilo Cloud

Previous review (commit 8aba994)

Status: 6 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 3
SUGGESTION 3
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/orchestrate.ts 2390 compiled_sql falls back to on-disk target/compiled artifacts for a side the PR does not have, so a deleted model's stale head SQL can be shown to the verifier (and used by equivalence).
packages/opencode/src/altimate/review/ai-verify.ts 369 hunkAround anchors removed rows to the next hunk and drops blank context rows, shifting head-side line numbers in the excerpt the verifier decides from. walkDiffRows already does this correctly.
packages/opencode/src/altimate/review/run.ts 436 A repo-relative glob is prefixed with the project root a second time (dbt/dbt/...), returning a false no matches and ignoring .. segments inside the glob.

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/ai-verify.ts 472 A refuted verdict with no cited evidence falls through to keep/verified, recording a self-contradictory verification in the envelope.
packages/opencode/src/altimate/review/ai-verify.ts 737 safeReason, positiveNumber and BATCH_CONCURRENCY duplicate the AI lane's helpers; backticked() duplicates quotedNames().
packages/opencode/src/altimate/review/ai-replay.ts 156 unverify only strips a Verified: suffix at the end of the body; mergeRepeats appends after it, so replay keeps stale Verified: text and the (+N more) title.
Files Reviewed (16 files)
  • packages/opencode/src/altimate/llm/stream-text.ts - clean
  • packages/opencode/src/altimate/review/ai-replay.ts - 1 issue
  • packages/opencode/src/altimate/review/ai-review.ts - clean
  • packages/opencode/src/altimate/review/ai-verify.ts - 3 issues
  • packages/opencode/src/altimate/review/finding.ts - clean
  • packages/opencode/src/altimate/review/format.ts - clean (one demotion hole already covered by an existing comment)
  • packages/opencode/src/altimate/review/git.ts - clean (already covered by existing comments)
  • packages/opencode/src/altimate/review/orchestrate.ts - 1 issue
  • packages/opencode/src/altimate/review/run.ts - 1 issue
  • packages/opencode/src/altimate/review/runner.ts - clean (already covered by an existing comment)
  • packages/opencode/src/altimate/review/telemetry.ts - clean
  • packages/opencode/src/altimate/review/verdict.ts - clean
  • packages/opencode/src/altimate/telemetry/index.ts - clean
  • packages/opencode/test/altimate/review-ai-inputs.test.ts - clean
  • packages/opencode/test/altimate/review-ai-replay.test.ts - clean
  • packages/opencode/test/altimate/review-ai-verify.test.ts - clean

Fix these issues in Kilo Cloud


Reviewed by deepseek-v4.1-flash-1017 · Input: 159.7K · Output: 41.4K · Cached: 4.1M

Review guidance: REVIEW.md from base branch feat/review-posting-policy

@hkaltai hkaltai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: approve with minor changes. Panel: Claude, Codex, and a lead pass, all covering every changed file (reviewed against the stack base only). No CRITICAL or MAJOR defects; flag-off behaviour is unchanged and the verifier fails open as described. bun test test/altimate/review passes (667).

Severity Count
CRITICAL 0
MAJOR 0
MINOR 6 (inline)
NIT 6

Consensus (2+ reviewers): config-rule override of summary demotion, replay dropping findings on model failure. Single-reviewer findings were verified against the code. A Codex-rated MAJOR (token budget exceeded) was downgraded to MINOR: it fails safe and degrades to the summary, but the 12k bound is not hard. Missing tests worth adding: grepFixedAt / grepProject wiring, new review_run telemetry fields, and the regressions for the findings above.

Nits

  • telemetry/index.ts:1623 /** AI findings the verifier kept (including those moved to the summary) and discarded. */ - summary decisions are counted separately and summary/unreviewed are not emitted. Fix the comment or emit ai_verify_summary / ai_verify_unreviewed.
  • ai-verify.ts:1224 log.error("ai verify failed", ...) path returns findings unchanged even with ALTIMATE_REVIEW_AI_VERIFY_FAIL=closed. Apply failMode in the catch or document that closed mode is per-verdict only.
  • ai-verify.ts:369 rows.slice(i).find(...) is O(n^2) per candidate on large diffs. Precompute next head line in one backward pass.
  • ai-verify.ts:88, :447-449 experiment notes ("opt-in until the decision run says otherwise", "in the decision run about half of these discards were judged valuable") will go stale; move to the PR/design doc. :3 ("One extra, bounded model call") and :637 ("may spend before it must answer") overstate a hard bound.
  • ai-replay.ts:78-82 "as the generator built them" overstates fidelity: discarded bodies are clamped to 600 chars and kept findings have been through anchor/clean post-processing. Persist raw generator findings or document, and strip preExisting/evidenceLine in unverify.
  • Naming: ts/TsFeatures (saved as envelope key decisions[].ts) reads as timestamp/TypeScript; rename to features. kept vs summary is ambiguous in status/telemetry. DroppedFinding.decision is always "discard". Generic exported Decision/Verification names. orchestrate.ts:162-167 graphParents doc says project-relative but paths are repo-relative.

Also worth noting: verifier tokens/cost are not added to ai.cost/ai.tokens (only under ai.verifier), and ALTIMATE_REVIEW_AI_REPLAY is honoured in production runAiReview.

* the PR added, at or above the severity floor. */
export function inlineEligible(f: Finding, posting: PostingPolicy): boolean {
if (f.tier !== "inline" && findingOrigin(f) !== "ai") return false
if (f.tier !== "inline" && (findingOrigin(f) !== "ai" || verifierDemoted(f))) return false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Verifier demotion can be overridden by a config rule tier

if (f.tier !== "inline" && (findingOrigin(f) !== "ai" || verifierDemoted(f))) return false

verifierDemoted is only consulted when tier !== "inline", and cleanFindings re-tiers every finding from ruleTier(f.rule, config.rules) after the AI lane, overwriting the summary tier set by triage. With a rule such as ai-review/*: inline in the review config, an unverifiable cross-file finding the verifier moved to the summary becomes inline-eligible again and is posted inline. Reproduced independently by two reviewers.

Fix: check verifierDemoted(f) first (return false), or keep summary in cleanFindings when verification.decision === "summary". Add a regression test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thank you. inlineEligible now checks verifierDemoted(f) before anything else, so an ai-review/*: inline rule can no longer bring back a finding the verifier moved to the summary. The test in review-ai-verify.test.ts covers the re-tiered case. Fixed in "fix(review): address review feedback: verifier demotion, …" (a74e400684).

model = await Provider.getModel(chosen.providerID, chosen.modelID)
} catch (err) {
const reason = `no usable model: ${err instanceof Error ? err.message : String(err)}`.slice(0, 200)
return done([], { ...lane.status, state: "skipped", reason })

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Replay drops all replayed findings when no verifier model resolves

return done([], { ...lane.status, state: "skipped", reason })

The generator findings are already rebuilt at this point, but are discarded on a model-lookup failure, contradicting the module's fail-open design. Scenario: ALTIMATE_REVIEW_AI_REPLAY=env.json ALTIMATE_REVIEW_AI_VERIFY=1 with an unavailable or misnamed --ai-model yields zero AI findings, silently skewing an A/B comparison.

Fix: return lane.findings unchanged with verifier: { state: "skipped", reason }, and add a test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, that broke the fail-open promise and would skew an A/B run. runAiReplay now returns the replayed generator findings unchanged and records verifier: { state: "skipped", reason: "no usable model: …" }. The new replay test mocks a failing Provider.getModel (same commit).

? { decision: "keep", rule: "unreviewed" }
: { decision: "discard", rule: "unreviewed" }
if (ts.restatesGrounding) return { decision: "discard", rule: "duplicate_of_engine" }
if (m.verdict === "refuted" && m.evidence.trim()) return { decision: "discard", rule: "refuted" }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Refuted verdict without evidence is kept as "verified" while unverifiable is discarded

if (m.verdict === "refuted" && m.evidence.trim()) return { decision: "discard", rule: "refuted" }

A refuted verdict with blank evidence skips this row and every unverifiable row, and falls through to keep / "verified". For a finding with evidenceLineValid: false, unverifiable is discarded but refuted with empty evidence is kept and labelled verified, so the model saying the claim is wrong makes it more likely to survive.

Fix: normalise refuted-without-evidence to unverifiable before the remaining rows (or give it its own rule name so it is never labelled verified), and test the invalid-line case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right that this inverted the intent. triage now treats a refuted verdict with blank evidence as unverifiable before the remaining rows run. It is never labelled verified, and with evidenceLineValid: false it is discarded as unverifiable. The triage test asserts both the invalid-line and cross-file cases (same commit).

Math.ceil((requestChars(ctx.fixedChars, messages) + ANSWER_NOW.length + extra) / charsPerToken) +
VERIFY_OUTPUT_RESERVE
const roundOutput = Math.min(VERIFY_MAX_CALLS_PER_ROUND * TOOL_OUTPUT_MAX, ctx.maxToolChars - toolChars)
const roomForRound = tokens + nextCall() + nextCall(roundOutput) <= ctx.maxTokens

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Token budget is a planning target, not a bound; tool rounds rarely run

const roomForRound = tokens + nextCall() + nextCall(roundOutput) <= ctx.maxTokens

Two related effects. (1) The forced answer is always sent, even when a batch's minimum request exceeds its budget share, and the output reserve is not enforced; with 40 findings (5 batches) an in-memory harness spent ~26.7k tokens against the 12k default, so the documented 12k bound can be exceeded. (2) roundOutput assumes the worst case (6 x 2,500 chars) and maxTokens is split across batches, so with 8 findings (one batch) the first call is already toolChoice: "none" and with 9 findings both calls are; the tool loop effectively never runs on mid-size PRs and the state reports budget. Fails safe (cross-file claims go to the summary), but limits both the cost bound and the verifier's usefulness.

Fix: plan with a realistic round cost (e.g. TOOL_OUTPUT_MAX * 2) or give each batch a floor allowing one round; skip batches whose minimum request cannot fit (keep findings unchanged); document the cap as soft otherwise.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both points taken. A round is now planned at two full tool outputs (ROUND_OUTPUT_PLAN), not the six-call worst case. A new test shows a batch of 8 at an 8k budget now gets a tool round where it was forced before. The forced answer is still always sent, so instead of adding a hard stop, VERIFY_MAX_TOKENS, verifyBatch and the module header now document the cap as soft. Whether to raise the 12k default is a tuning call for the next benchmark run, so it is not changed here.

if (ctx.abort.aborted) throw ctx.abort.reason ?? new Error("aborted")
if (force || step.finishReason !== "tool-calls" || !step.toolCalls.length) {
out.verdicts = parseVerifierOutput(step.text, ids)
if (!out.verdicts.size && out.state === "ok") {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Unparsable forced answer is reported as budget, not error

if (!out.verdicts.size && out.state === "ok") {

When the answer was forced early (state: "budget") and the model returns nothing parsable (e.g. []), the batch keeps budget with no error, although every finding is unreviewed. Reproduced with 8 findings and an LLM reply of "[]". Telemetry ai_verify_state then hides verifier failures.

Fix: set error/reason whenever verdicts.size === 0, regardless of the earlier state.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. A batch that returns no parsable verdicts is now error with "the verifier returned no parsable verdicts", even when the answer was forced early. Telemetry ai_verify_state no longer hides it. The budget test now includes your 8-finding "[]" case (same commit).

toolChoice: force ? "none" : "auto",
abort: ctx.abort,
}
const step = await ctx.llm(req)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Timeout cannot interrupt awaited evidence tools

const step = await ctx.llm(req)

The timeout aborts model requests only; await runToolCall(...) in the same loop (and graph gathering before the timer starts) is not raced against the abort signal. One reviewer reproduced a pending evidence read keeping the verifier unresolved after a 10 ms timeout until the read was released. Local git/file reads rarely stall, so impact is limited, but it weakens the fail-open-on-timeout guarantee.

Fix: apply one deadline across graph gathering, tools and model calls (race pending work against the abort signal and stop scheduling after expiry).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reproducing this. The timer now starts before graph gathering. gatherGraph and every runToolCall are raced against the same abort signal through a small untilAborted helper, and an abort there reports timeout rather than error. The new test uses a readFile that never settles and checks that the run ends at the deadline with findings unchanged (same commit).

@github-actions

Copy link
Copy Markdown

Thanks for updating your PR! It now meets our contributing guidelines. 👍

if (glob) {
const g = glob.trim().replace(/^\.?\/+/, "")
if (!g || g.split("/").includes("..")) return undefined
return [root && !g.startsWith(root) ? root + g : g]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: A glob equal to the project root is prefixed a second time

root keeps its trailing slash, so glob: "dbt" (the project root itself, a natural way to ask for the whole project) fails the g.startsWith(root) test and becomes the pathspec dbt/dbt. That directory cannot exist, so the search covers nothing and comes back as no matches/failed instead of searching dbt/ — the same false "evidence of absence" this commit set out to remove. "dbt/" and "dbt/**" work, only the bare root does not.

Suggested change
return [root && !g.startsWith(root) ? root + g : g]
return [root && g !== root.slice(0, -1) && !g.startsWith(root) ? root + g : g]

@RaghwendraSingh
RaghwendraSingh force-pushed the feat/review-posting-policy branch from 4733424 to e7a80ad Compare November 20, 2026 08:22
@RaghwendraSingh

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough pass, Harish. All six MINORs are addressed in fix(review): address review feedback: verifier demotion, replay fail-open, refuted verdicts, budget and timeout (a74e400684), each with a regression test. From the nits: the ai_verify_kept telemetry comment now says what is emitted. ALTIMATE_REVIEW_AI_VERIFY_FAIL=closed now also applies in the catch path. The next-head-line lookup is a single backward pass. The stale "decision run" notes are gone, and the header and VERIFY_MAX_TOKENS docs no longer claim a hard bound. Replay now strips preExisting/evidenceLine and says its fidelity is approximate. Not changed in this PR: the naming nits (ts/TsFeatures, Decision/Verification), because decisions[].ts is an envelope key that stored envelopes and the replay schema already read. Also unchanged: folding verifier tokens into ai.cost, since they are kept separate under ai.verifier on purpose for the A/B comparison. Happy to take either in a follow-up.

VERIFY_OUTPUT_RESERVE
// A typical round returns about two full tool outputs; planning for the
// worst case (every call at the cap) kept the tool loop from ever running.
const roundOutput = Math.min(ROUND_OUTPUT_PLAN, ctx.maxToolChars - toolChars)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: roundOutput can go negative once the tool-output budget is spent

After toolChars exceeds ctx.maxToolChars, ctx.maxToolChars - toolChars is negative, so nextCall(roundOutput) subtracts estimated input tokens and roomForRound can look affordable. Only the sibling toolChars > ctx.maxToolChars term still forces the answer, so the planner's own estimate is wrong in exactly the state it exists to detect.

Suggested change
const roundOutput = Math.min(ROUND_OUTPUT_PLAN, ctx.maxToolChars - toolChars)
const roundOutput = Math.min(ROUND_OUTPUT_PLAN, Math.max(0, ctx.maxToolChars - toolChars))

const near = (r: { head?: number }, i: number) => {
if (typeof line !== "number") return i < 2 * HUNK_RADIUS
// A removed row or hunk header sits beside the head row that follows it.
const at = r.head ?? nextHead[i]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Reuse walkDiffRows instead of re-deriving the walk

walkDiffRows (diff-filter.ts:398) already yields +/-/context rows with head line numbers, treats a blank context row ("") as a context line, resets inHunk on an unrecognized row, and tags each row with its hunk. This pass keeps none of that: a removed row or hunk header is anchored to the next head row anywhere later in the diff, so for a diff whose hunk ends in deletions (-U0, or a caller-supplied diff) the deleted code is omitted from its own hunk's excerpt and appears in a distant one. Anchoring with the hunk field fixes that and removes the duplicate numbering logic.

// No pathspec would search the whole repository.
if (!pathspecs.length) return undefined
const maxHits = opts.maxHits ?? 50
const args = ["grep", "-n", "-I", "-F", "--full-name", "-m", String(maxHits), "-e", pattern, ...(ref ? [ref] : []), "--"]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: With no ref, grep_project cannot see untracked files

git grep without a revision searches the working tree's tracked files only, while grepMatchesAt in the same module deliberately passes --untracked for the working-tree case. A new untracked model/macro/seed is therefore invisible and the tool answers "no matches" — the false evidence of absence this PR set out to remove.

Suggested change
const args = ["grep", "-n", "-I", "-F", "--full-name", "-m", String(maxHits), "-e", pattern, ...(ref ? [ref] : []), "--"]
const args = ["grep", "-n", "-I", "-F", "--full-name", ...(ref ? [] : ["--untracked"]), "-m", String(maxHits), "-e", pattern, ...(ref ? [ref] : []), "--"]

RaghwendraSingh and others added 10 commits November 20, 2026 16:12
Adds the skeleton of a dbt-grounded verifier for the advisory AI lane,
behind `ALTIMATE_REVIEW_AI_VERIFY=1`. With the variable unset nothing
changes: the lane gets no `evidence` and the envelope is identical.

- `ai-verify.ts`: `AiVerifyEvidence`, `VerifierStatus`, `aiVerifyEnabled`
  and a `runAiVerify` that returns the findings unchanged for now
- `runAiReview` runs the verifier after the review batches when given
  `evidence` and the variable is set; `AiLaneStatus.verifier` reports it
- `runReview` builds the evidence when `aiVerify` is set: engine SQL,
  numbered file slices, added lines, PR text and a project grep
- `run.ts` passes `aiVerify` and a `git grep -F` backed `grepProject`
  (5 s timeout, 50 hits) over the project's dbt directories
- `ReviewRunner.columnReach` wraps `columnReach` without throwing
- `readStreamStep` reads one AI SDK step keeping its tool calls and
  response messages
- envelope `ai.verifier` and `review_run` fields `ai_verify_state`,
  `ai_verify_tokens_total`, `ai_verify_tool_calls`, `ai_verify_kept`,
  `ai_verify_discarded`

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`extractFeatures` computes, without a model, the facts the verifier's
triage table reads for each AI finding:

- `evidenceLineValid`: the finding's line is on the diff (null without one)
- `targetsUnchangedCode`: a diff line the PR did not add, a `preExisting`
  tag, or no usable line and nothing the finding names on an added line
- `restatesGrounding`: title-word Jaccard of at least 0.5 with an engine
  finding on the same file, or the same category and column
- `claimsOtherFile`: a `ref()` to another model, a `source()`, or a
  backticked name of a dbt parent or downstream model
- `styleOnly`, `lowConfidence`, `hedgedDataDiff`

AI findings reach the verifier before `anchorToDiff` tags `preExisting`,
so a commentable line outside the added set counts as unchanged code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `VERIFY_SYSTEM_PROMPT`: an adversarial verifier that tries to refute
  each candidate from its hunk and tool results, treats the diff, SQL, PR
  text and tool output as untrusted data, and answers with one JSON
  element per candidate (`claim_type`, `verdict`, `value`,
  `requires_assumed_input`, `evidence`, `tools_used`)
- `buildVerifyMessage`: the PR title and, per candidate, its id, file,
  line, category, severity, title, body and the diff rows within 15 lines
  of it (`hunkAround`, at most 2,000 chars); never the whole file or the
  engine findings the reviewer saw
- `parseVerifierOutput`: strips `<think>` blocks and fences, takes the
  first balanced JSON array, ignores unknown ids and coerces bad enums
  (`verdict` to unverifiable, `claim_type` to code_fact, `value` to
  actionable); ids it does not return stay unreviewed

The prompt lives in TypeScript for the decision run and can move to core
later.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`triage` decides each AI finding from its deterministic features and the
verifier's verdict, first matching row wins: grounding duplicate,
refuted with evidence, unverifiable cross-file claim, speculation or an
assumed input, style or nit, unchanged code, plausible local claim
(kept), unverifiable, informational (summary), verified (kept). Without
a verdict the fail mode decides: open keeps the finding unchanged.

`applyTriage` records `evidence.result.verification` on reviewed kept
findings (the evidence tool stays `ai-review`), appends `Verified: ...`
to verified bodies, never raises severity, and lists at most 20 discards.
`inlineEligible` no longer posts an AI finding inline once the verifier
moved it to the summary.

Knobs for experiments: `ALTIMATE_REVIEW_AI_VERIFY_DROP_NITS=0`,
`ALTIMATE_REVIEW_AI_VERIFY_KEEP_LOCAL=0`, `ALTIMATE_REVIEW_AI_VERIFY_FAIL=closed`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`runAiVerify` now verifies the AI lane's findings: deterministic
features, then one model conversation per batch of at most 8 findings
(kept together by file), then triage. It never throws; whatever it could
not review comes back unchanged.

- Ten read-only tools, declared with `tool()` and no `execute`; the loop
  runs the calls itself and appends `tool-result` messages:
  `read_file` (changed files, their dbt parents and direct downstream
  models only), `compiled_sql`, `model_columns`, `column_reach`,
  `model_config`, `downstream`, `equivalence`, `grain`, `grep_project`,
  `pr_description`. Every result is a string of at most 2,500 chars;
  failures read `not available: ...`
- `LLM.stream` runs one step, so the loop drives the rounds: up to 3 tool
  rounds and 6 calls per round, then a forced answer with
  `toolChoice: "none"`; the answer is forced early once the batch's
  share of 20,000 tokens (`ALTIMATE_REVIEW_AI_VERIFY_MAX_TOKENS`) or
  16,000 tool-output chars is spent (state `budget`)
- one `AbortController` per PR with `aiTimeoutMs` of the prompt size
  (state `timeout`); a model error or an unparsable answer leaves that
  batch's findings unreviewed
- `harnessVerifierLlm` makes the call through `LLM.stream` at
  temperature 0 and reports usage with `streamUsage`; tests inject a
  scripted `VerifierLlm`
- status: turns, tool calls, tokens, cost, reviewed, kept, discarded,
  summary, unreviewed and up to 20 dropped findings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The verifier's default budget drops from 20,000 to 12,000 tokens per PR
(`ALTIMATE_REVIEW_AI_VERIFY_MAX_TOKENS` still overrides it), and the
forced final answer now counts against it. Before each call the loop
plans the rest of the batch: another tool round is taken only if that
call, plus the forced answer after it with a full round of tool output,
fits the batch's share. Otherwise it answers now with
`toolChoice: "none"`, from the first call on if need be. Call sizes are
estimated from the characters sent (system prompt, tool declarations,
messages) plus a 1,000-token output reserve, calibrated by the usage
the provider reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every verifier call repeats the system prompt and the tool declarations,
so they are kept to the minimum that stays unambiguous: shorter tool and
`side` descriptions, and `read_file` line bounds and the `grep_project`
pattern length checked in code rather than in the JSON schema (which
emitted `exclusiveMinimum`/`maximum` noise). Tool declarations go from
3,408 to 2,950 chars; the fixed per-call size from 4,832 to 4,374.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ai.verifier.decisions[]` now records every AI finding the verifier saw,
uncapped: its deterministic features, the model's verdict (claim type,
verdict, value, assumed input, tools used, and whether it gave
evidence; absent when unreviewed), and the decision and rule triage
applied. `dropped[]` is unchanged.

`retriage(decisions, knobs)` re-runs triage over those records without
a model call, so one run's envelopes answer what `dropNits`,
`keepUnverifiedLocal` and `failMode` would have changed. It also takes a
hypothetical `claimsOtherFileAsCrossFile` knob that sends findings
naming another model or source through the cross-file rule; live triage
still does not use `claimsOtherFile`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ards

`ai.verifier.dropped[]` now carries each discard's `startLine` and body
(clamped to 600 chars), so a verifier run can be scored both with and
without its discards. That makes the no-verifier baseline a free
by-product of the same run instead of a separate one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… budget

When a batch's prompt left no room for a tool round, the first call was
sent with `toolChoice: "none"` but no instruction to answer. The model
then returned about 18 output tokens and no parsable verdicts, so every candidate in the batch went unreviewed.
In the decision run this hit 10 PRs (dev and heldout) with large batches. The first
message now ends with an explicit answer-now line in that case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RaghwendraSingh and others added 4 commits November 20, 2026 16:12
…the summary

The verifier discarded these. In the 50-PR dev decision run, 10 of 21
`unverifiable_cross_file` discards and 9 of 19 `speculative` discards
were judged valuable, and every catch the verifier lost came from the
first rule. They now stay in the review summary, never inline.

Offline re-triage of the same run (no model calls), dev: catches 18 vs
15 with discarding (18 without the verifier); verified AI findings
(outside the summary) unchanged at 0.84/PR, 69% valuable, 0% wrong; remaining
discards 2 of 18 valuable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…verifier

`ALTIMATE_REVIEW_AI_REPLAY=<earlier review --json envelope>` skips the
generator call in `runAiReview` and rebuilds its findings from that
envelope, so verifier settings can be compared for verifier tokens only.

- New `ai-replay.ts`: kept AI findings lose the verifier's
  `evidence.result.verification`, the `Verified:` body suffix, and the
  tier/origin set after generation; `ai.verifier.dropped` entries are
  rebuilt as `Finding`s with the stored `id` and `rule`. Order follows
  `ai.verifier.decisions`. A `dropped` list shorter than the discard
  count is warned on stderr and reported as `replayIncomplete`.
- The AI lane status copies the stored generator `tokens`/`calls`/`cost`
  and adds `replayed: true` (envelope schema and `ai_replayed` telemetry).
  An unreadable envelope is an error on stderr and `state: "error"`; the
  generator is never called as a fallback.
- `ALTIMATE_REVIEW_AI_VERIFY_MAX_TOOL_CHARS` and
  `ALTIMATE_REVIEW_AI_VERIFY_MAX_ROUNDS` override the verifier budgets;
  the prompt states the effective round limit. Defaults unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…project false evidence

- `runner.ts`: `columnReach` answers an unknown model (or no manifest) with
  `undefined` instead of recording an engine failure, so a verifier tool
  argument can no longer land in `engineErrors` and downgrade the verdict.
- `git.ts`: `grepFixedAt` returns `undefined` when git fails or times out with
  nothing found, and for an empty pathspec list (which would search the whole
  repo); only exit 1 (no match) returns `[]`.
- `ai-verify.ts`: `grep_project` reports a failed search as failed ("not
  evidence of absence") rather than "no matches".
- `git.ts`/`run.ts`: new `projectGrepSpecs` builds the search pathspecs; a glob
  that already carries the project prefix (the verifier sees repo-relative
  paths) is not prefixed again, and a glob with `..` segments is rejected.
- Tests for each.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…open, refuted verdicts, budget and timeout

- `inlineEligible`: the verifier's summary demotion is checked first, so a
  config rule that re-tiers AI findings to `inline` cannot post it inline.
- `runAiReplay`: when no verifier model resolves, the replayed generator
  findings are returned unchanged with `verifier.state: "skipped"`.
- `triage`: a `refuted` verdict without evidence is judged as `unverifiable`,
  so it is never labelled `verified`.
- Budget: plan a tool round at two full tool outputs instead of the worst
  case, so mid-size batches can use tools; document the token cap as soft.
- A forced answer with no parsable verdicts is reported as `error`, not
  `budget`.
- One deadline covers graph gathering and evidence tool calls as well as
  model calls; an abort there reports `timeout`.
- An unexpected verifier failure honours `ALTIMATE_REVIEW_AI_VERIFY_FAIL=closed`.
- Replay `unverify` also strips `preExisting`/`evidenceLine`; doc notes the
  approximate fidelity. Next-head-line lookup is one backward pass.
- Comment fixes: telemetry `ai_verify_kept`, stale experiment notes, and
  overstated budget bounds.
- Tests for each change.
- Restack: `inlineEligible` keeps the parent branch's AI-defaults-to-inline tiering with the
  verifier demotion checked first; the demotion test stamps the kept finding `inline`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@RaghwendraSingh
RaghwendraSingh force-pushed the feat/review-posting-policy branch from e7a80ad to 2dcef4b Compare November 20, 2026 11:37
dialect,
files: aiFiles,
projectRoot: input.layout?.root,
runner: input.runner,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: Verifier-only engine failures land in the shared failure ledger and can change the verdict

The verifier gets the same runner instance as the deterministic lanes (runner: input.runner), and every runner method that throws records into that instance's failure map (runner.ts:127-135). engineErrors is assembled after the AI lane (orchestrate.ts:2543), and any entry makes reviewIncomplete() true, turning an APPROVE into COMMENT (verdict.ts:394-414). column_reach is a verifier-only tool (ai-verify.ts:850-866) whose engine call can fail on models no deterministic lane touched, so an opt-in advisory pass can now downgrade the review, contradicting the verifier's fail-open contract. Snapshot runner.failures() before the AI lane and exclude verifier-only increments, or give the verifier a runner view that does not record failures.

if (!p) return scope
return (
(await ev.readFile(p, (side as "old" | "new") ?? "new", start as number, end as number)) ??
NA(`${p} does not exist on that side`)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: An unreadable file is reported to the verifier as missing

readFile is documented to return undefined "when unreadable" (ai-verify.ts:32), and makeContentResolver maps every git failure (bad ref, timeout, git show error) and every safeReadInside refusal (escaping symlink) to undefined (git.ts:129-152). This fallback then tells the model ${p} does not exist on that side, which the prompt accepts as refutation evidence ("a tool result ... contradicts it"). A transient git failure or a blocked symlink can therefore refute a valid comment. Return a distinct "could not read" answer (or a discriminated result) and only claim absence when the side is known to lack the file.

async run({ pattern, glob }) {
if (!ev.grepProject) return NA("project search unavailable")
if (String(pattern).length < 2) return NA("pattern needs at least 2 characters")
const hits = await ev.grepProject(String(pattern), glob as string | undefined)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: grep_project can read credential files the AI lane deliberately withholds

The tool accepts any glob; projectGrepSpecs only rejects .. segments and re-roots the path (git.ts:252-261). With the dbt project at the repo root (rootRel === "") a call such as glob: "profiles.yml" or glob: ".env" searches tracked repo files, and grepFixedAt forwards up to 50 matching lines into the prompt. orchestrate.ts:51-52 and :2481-2482 explicitly keep profiles.ya?ml out of the AI lane because it holds warehouse credentials, so this bypasses that boundary (a malicious diff can also steer the model's choice of pattern/glob). Filter credential paths and restrict the glob to the project's dbt dirs/kinds before searching.

if (input.evidence && aiVerifyEnabled()) {
const v = await runAiVerify({ findings, grounding: input.grounding, evidence: input.evidence, model })
findings = v.findings
verifier = v.status

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: A fail-closed verifier failure is invisible in the report

When runAiVerify hits an error/timeout with ALTIMATE_REVIEW_AI_VERIFY_FAIL=closed, it returns no findings and state: "error"/"timeout", but this line only stores that status and done(...) still reports the lane as state: "ok". coverageNotes (format.ts:278-284) reads only env.ai.state/env.ai.reason, so the PR comment shows a clean AI lane while every AI finding disappeared. Fold the verifier's non-ok state/reason into the lane status (or render it in the comment) so the loss is stated.

/** Verifier tokens: input (including cached) plus output, across batches and tool rounds. */
ai_verify_tokens_total?: number
ai_verify_tool_calls?: number
/** AI findings the verifier kept and discarded; `kept` includes those moved to the summary (summary and unreviewed counts are not emitted). */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: The comment says ai_verify_kept includes summary findings, but the producer does not

applyTriage increments kept only for decision === "keep" and counts summary separately (ai-verify.ts:613-614), while telemetry emits only ai_verify_kept/ai_verify_discarded (telemetry.ts:129-130). So kept + discarded undercounts reviewed findings by the summary count, the opposite of what this JSDoc promises. Either add the summary findings to the emitted kept or state that it excludes them.

}
})(),
})
const [finishReason, response] = await Promise.all([stream.finishReason, stream.response])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: finishReason/response are left unobserved when the stream errors

readStreamText throws on an error part (or a rejected .text), so line 51 never runs. .text is explicitly guarded against an unhandled rejection at line 9 for exactly this reason, but the two new SDK promises are not: if they reject when the stream failed, the process gets an unhandled rejection. Observe them before rethrowing (Promise.allSettled or void p.catch(() => {})), as readStreamText does for .text.

if ((v?.discarded ?? 0) > listed)
warnings.push(`the verifier discarded ${v!.discarded} findings but the envelope lists ${listed}`)
const have = new Set(out.map((f) => f.id))
const missing = (v?.decisions ?? []).filter((d) => !have.has(d.id)).length

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: A merged repeat is reported as a missing decision on every replay

decisions is recorded inside runAiVerify on the generator's ids, but env.findings is written after cleanFindings merges repeats (orchestrate.ts:2456, finding.ts:305-311) and the merged finding gets a synthetic f_<hash> id. have therefore never contains the merged group's original ids, so any envelope with a merged AI repeat reports "N findings the verifier decided are not in the envelope" and sets replayIncomplete even though every finding is represented. Treat a merged id as covering its group, or record decisions after merging.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants