Skip to content

feat(review): opt-in AI verifier that confirms or refutes each AI comment from dbt evidence - #1769

Open
RaghwendraSingh wants to merge 13 commits into
feat/review-posting-policyfrom
feat/review-ai-verifier
Open

RaghwendraSingh wants to merge 13 commits into
feat/review-posting-policyfrom
feat/review-ai-verifier

Conversation

@RaghwendraSingh

@RaghwendraSingh RaghwendraSingh commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Issue for this PR

Part of #1792

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Stacked on #1758. Adds an opt-in verifier for the AI review lane (ALTIMATE_REVIEW_AI_VERIFY=1). After the AI reviewer drafts its comments, one more bounded model pass tries to confirm or refute each one from dbt evidence. Comments it can't support are moved to the summary or dropped. Off by default: with the flag unset, the review output is unchanged.

Results (replay of public dbt PRs, blind Claude judge)

Verifier on vs off, same runs (50 PRs, 59 reviewer-found defects, tuning repos)

AI comments per PR Useful Wrong
Verifier off 2.48 45% 1%
Verifier on 1.32 53% 0%
  • On 26 PRs from repos never used for tuning: useful 38% → 59%, wrong 1 → 0.
  • The first version dropped 3 real catches by discarding cross-file claims it could not check. Those claims now go to the summary instead of being dropped (3b2c31e2dd), so catches are not lost.

Repos never used for any tuning (25 repos, 101 PRs, one snapshot each, 153 defects)

Build Defects caught AI comments useful / wrong
Released v0.12.11 with its AI 24 34% / 3.6%
This stack, AI + verifier + rules 35 43% / 1.3%

Difference +11 catches (95% CI +1 to +22, PR-level bootstrap). The verifier costs about 1.4× the tokens of the AI review alone.

What changed

  • ai-verify.ts:
    • Deterministic features: evidence line valid, targets unchanged code, restates an engine finding, claims another file, style only, low confidence, hedged data-diff.
    • The verifier prompt and a tolerant parser.
    • A bounded tool loop: read_file, compiled_sql, model_columns, column_reach, model_config, downstream, equivalence, grain, grep_project, pr_description. At most 3 rounds and 12k tokens.
    • Table-driven triage into keep / summary / discard.
  • Fails open: on an error, timeout or missing verdict, the original findings stay.
  • Verifier decisions, tokens and dropped findings are recorded in the envelope and telemetry.
  • ALTIMATE_REVIEW_AI_REPLAY=<envelope> replays stored generator findings so only the verifier is re-run.
  • Runner gains columnReach, and the stream reader keeps tool parts.

How did you verify your code works?

  • bun test test/altimate/review: 667 pass.
  • bunx tsc --noEmit: clean.
  • New: test/altimate/review-ai-verify.test.ts (features, every triage row and knob, tool loop limits, allowlist, fail-open, parser).

Cost side effects

None by default: the verifier is opt-in (ALTIMATE_REVIEW_AI_VERIFY=1). When enabled it adds one bounded model pass per review (at most 3 tool rounds and 12k tokens), measured at about 1.4x the AI-lane tokens of the AI review alone. Its tools read local project files, the manifest and the in-process engine; no warehouse queries. Verifier tokens are recorded in the envelope and telemetry.

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

🤖 Generated with Claude Code


Summary by cubic

Adds an opt-in, dbt-grounded verifier for the AI review lane. With ALTIMATE_REVIEW_AI_VERIFY=1, one bounded model pass after the AI reviewer tries to confirm or refute each comment from dbt evidence (compiled SQL, DAG lineage, project grep); comments it can't support move to the summary or are dropped. With the flag unset the review output is unchanged. The verifier fails open: any error, timeout, or exhausted budget leaves the findings it didn't review untouched.

Results (blind judge on public dbt PRs)

AI comments per PR (% useful / % wrong)
Verifier off 2.48 (45% / 1%)
Verifier on 1.32 (53% / 0%)

On 26 PRs from repos never used for tuning, useful went 38% → 59% and wrong went 1 → 0. Cross-file claims the verifier can't check go to the summary instead of being dropped, so real catches aren't lost. The verifier costs about 1.4× the tokens of the AI review alone.

What changed

  • Deterministic features per finding (evidence line valid, unchanged code, engine grounding, cross-file claim, style only, low confidence), a tolerant verdict parser, and table-driven triage into keep / summary / discard.
  • A bounded tool loop of 10 read-only tools, at most 3 rounds and 12k tokens.
  • Verifier decisions, tokens, and dropped findings are recorded in the envelope and telemetry; retriage() re-runs triage offline without a model call.
  • ALTIMATE_REVIEW_AI_REPLAY=<envelope> replays stored generator findings so verifier settings can be compared without regenerating.
  • grepProject and columnReach are wired into the runner for evidence gathering.
  • Coverage: 667 tests pass, including new suites for features, every triage row and knob, tool loop limits, fail-open, and replay.

Written for commit f3c5e86. Summary will update on new commits.

View guided diff Turn on auto-fix

@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 5a4ac470-d21b-46d6-898a-0004c652c769

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

11 issues found across 16 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/opencode/src/altimate/telemetry/index.ts">

<violation number="1" location="packages/opencode/src/altimate/telemetry/index.ts:1625">
P2: The verifier's USD cost is recorded in the envelope but never reaches telemetry. Add an `ai_verify_cost` field here and map it from `env.ai?.verifier?.cost` in review/telemetry.ts, so verifier spend is measurable alongside its tokens.</violation>
</file>

<file name="packages/opencode/src/altimate/review/ai-replay.ts">

<violation number="1" location="packages/opencode/src/altimate/review/ai-replay.ts:121">
P3: Discarded findings are rebuilt from the envelope's `body`, which `applyTriage` clamps to 600 characters. The replayed body is truncated, and the verifier and output see it that way without any `replayIncomplete` flag. Note the truncation in the replay status, or stop clamping for replayed records.</violation>

<violation number="2" location="packages/opencode/src/altimate/review/ai-replay.ts:196">
P1: When replay cannot resolve a verifier model, this branch drops every stored AI finding even though verification never ran. Return `findings` so model-resolution failures fail open like other verifier errors.</violation>
</file>

<file name="packages/opencode/src/altimate/review/format.ts">

<violation number="1" location="packages/opencode/src/altimate/review/format.ts:390">
P2: A configured `ai-review:*: inline` rule can overwrite the verifier’s `summary` tier, and this condition then bypasses `verifierDemoted`. Check verifier demotion unconditionally before allowing inline-tier findings.</violation>
</file>

<file name="packages/opencode/src/altimate/review/git.ts">

<violation number="1" location="packages/opencode/src/altimate/review/git.ts:207">
P3: A valid filename like `models/a:123:b.sql` is split at the filename’s `:123:` and returned with a false path and line. Parse an unambiguous path/line format instead.</violation>
</file>

<file name="packages/opencode/src/altimate/review/ai-verify.ts">

<violation number="1" location="packages/opencode/src/altimate/review/ai-verify.ts:198">
P2: A finding with no usable line and no quoted or backticked identifier is classified as targeting unchanged code, so triage discards it as `unchanged_code` unless the model confirmed it. An empty identifier list should not count as evidence of unchanged code. Require at least one identifier before marking it unchanged.</violation>

<violation number="2" location="packages/opencode/src/altimate/review/ai-verify.ts:1003">
P2: When a batch's share cannot cover even its direct-answer estimate, `force` still sends that call; repeated across batches, verifier usage can exceed the 12k PR budget. Skip that batch as budget-exhausted when the answer itself cannot fit, leaving its findings unreviewed.</violation>
</file>

<file name="packages/opencode/src/altimate/review/runner.ts">

<violation number="1" location="packages/opencode/src/altimate/review/runner.ts:471">
P2: `columnReach` records an expected "model not found" as an engine failure via `fail()`. That failure lands in `engineErrors`, which downgrades an APPROVE to COMMENT and marks the run as not completed. Return `undefined` for not-found without calling `fail`, as `modelConsumers` does, and reserve `fail` for real errors such as an unreadable manifest.</violation>
</file>

<file name="packages/opencode/src/altimate/llm/stream-text.ts">

<violation number="1" location="packages/opencode/src/altimate/llm/stream-text.ts:32">
P3: `readStreamStep` adds a new error-propagation path and an `invalid` passthrough, but no test covers either. Add cases to stream-text.test.ts for a mid-stream `error` part (must throw) and for a tool call with `invalid: true`.</violation>
</file>

<file name="packages/opencode/src/altimate/review/verdict.ts">

<violation number="1" location="packages/opencode/src/altimate/review/verdict.ts:237">
P3: The `dropped` schema accepts any decision and any length, but the producer only records discards and caps the list at 20. Narrow `decision` to `z.literal("discard")` and add `.max(20)` to the array so the schema states the invariant.</violation>
</file>

<file name="packages/opencode/src/altimate/review/ai-review.ts">

<violation number="1" location="packages/opencode/src/altimate/review/ai-review.ts:312">
P2: With the verifier on, lane-level `tokens`, `cost` and `calls` cover only the generator, so `ai_cost`/`ai_tokens_total`/`ai_calls` in telemetry undercount total AI spend. Either add the verifier's usage into these totals or document that they exclude it, and update the `tokens`/`calls` JSDoc to match.</violation>
</file>

Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

View guided diff | Turn on auto-fix | Re-trigger cubic

model = await Provider.getModel(chosen.providerID, chosen.modelID)
} catch (err) {
const reason = `no usable model: ${err instanceof Error ? err.message : String(err)}`.slice(0, 200)
return done([], { ...lane.status, state: "skipped", reason })

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: When replay cannot resolve a verifier model, this branch drops every stored AI finding even though verification never ran. Return findings so model-resolution failures fail open like other verifier errors.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-replay.ts, line 196:

<comment>When replay cannot resolve a verifier model, this branch drops every stored AI finding even though verification never ran. Return `findings` so model-resolution failures fail open like other verifier errors.</comment>

<file context>
@@ -0,0 +1,203 @@
+      model = await Provider.getModel(chosen.providerID, chosen.modelID)
+    } catch (err) {
+      const reason = `no usable model: ${err instanceof Error ? err.message : String(err)}`.slice(0, 200)
+      return done([], { ...lane.status, state: "skipped", reason })
+    }
+    const v = await runAiVerify({ findings, grounding: input.grounding, evidence: input.evidence, model })
</file context>

try {
return columnReach(opts.manifestPath, model, column, dialect)
} catch (err) {
fail("columnReach", err)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: columnReach records an expected "model not found" as an engine failure via fail(). That failure lands in engineErrors, which downgrades an APPROVE to COMMENT and marks the run as not completed. Return undefined for not-found without calling fail, as modelConsumers does, and reserve fail for real errors such as an unreadable manifest.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/runner.ts, line 471:

<comment>`columnReach` records an expected "model not found" as an engine failure via `fail()`. That failure lands in `engineErrors`, which downgrades an APPROVE to COMMENT and marks the run as not completed. Return `undefined` for not-found without calling `fail`, as `modelConsumers` does, and reserve `fail` for real errors such as an unreadable manifest.</comment>

<file context>
@@ -463,6 +464,15 @@ export function createDispatcherRunner(opts: DispatcherRunnerOptions): ReviewRun
+      try {
+        return columnReach(opts.manifestPath, model, column, dialect)
+      } catch (err) {
+        fail("columnReach", err)
+        return undefined
+      }
</file context>

.filter((l) => l.kind === "+" || added.has(l.line))
.map((l) => l.text.toLowerCase())
const ids = quotedNames(text)
targetsUnchangedCode = !ids.some((id) => addedText.some((t) => t.includes(id)))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: A finding with no usable line and no quoted or backticked identifier is classified as targeting unchanged code, so triage discards it as unchanged_code unless the model confirmed it. An empty identifier list should not count as evidence of unchanged code. Require at least one identifier before marking it unchanged.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-verify.ts, line 198:

<comment>A finding with no usable line and no quoted or backticked identifier is classified as targeting unchanged code, so triage discards it as `unchanged_code` unless the model confirmed it. An empty identifier list should not count as evidence of unchanged code. Require at least one identifier before marking it unchanged.</comment>

<file context>
@@ -0,0 +1,1230 @@
+      .filter((l) => l.kind === "+" || added.has(l.line))
+      .map((l) => l.text.toLowerCase())
+    const ids = quotedNames(text)
+    targetsUnchangedCode = !ids.some((id) => addedText.some((t) => t.includes(id)))
+  }
+
</file context>
Suggested change
targetsUnchangedCode = !ids.some((id) => addedText.some((t) => t.includes(id)))
targetsUnchangedCode = ids.length > 0 && !ids.some((id) => addedText.some((t) => t.includes(id)))

ai_verify_tool_calls?: number
/** AI findings the verifier kept (including those moved to the summary) and discarded. */
ai_verify_kept?: number
ai_verify_discarded?: number

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The verifier's USD cost is recorded in the envelope but never reaches telemetry. Add an ai_verify_cost field here and map it from env.ai?.verifier?.cost in review/telemetry.ts, so verifier spend is measurable alongside its tokens.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/telemetry/index.ts, line 1625:

<comment>The verifier's USD cost is recorded in the envelope but never reaches telemetry. Add an `ai_verify_cost` field here and map it from `env.ai?.verifier?.cost` in review/telemetry.ts, so verifier spend is measurable alongside its tokens.</comment>

<file context>
@@ -1613,6 +1613,16 @@ export namespace Telemetry {
+        ai_verify_tool_calls?: number
+        /** AI findings the verifier kept (including those moved to the summary) and discarded. */
+        ai_verify_kept?: number
+        ai_verify_discarded?: number
         /** Where the review base came from (flag, github_event, default_branch, …). */
         base_source?: string
</file context>
Suggested change
ai_verify_discarded?: number
ai_verify_discarded?: number
/** Estimated USD cost of the verifier from the model's price table. */
ai_verify_cost?: number

findings = v.findings
verifier = v.status
}
return done(findings, { state: "ok", model: modelLabel, ...(reason ? { reason } : {}), ...spent, ...(verifier ? { verifier } : {}) })

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: With the verifier on, lane-level tokens, cost and calls cover only the generator, so ai_cost/ai_tokens_total/ai_calls in telemetry undercount total AI spend. Either add the verifier's usage into these totals or document that they exclude it, and update the tokens/calls JSDoc to match.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-review.ts, line 312:

<comment>With the verifier on, lane-level `tokens`, `cost` and `calls` cover only the generator, so `ai_cost`/`ai_tokens_total`/`ai_calls` in telemetry undercount total AI spend. Either add the verifier's usage into these totals or document that they exclude it, and update the `tokens`/`calls` JSDoc to match.</comment>

<file context>
@@ -281,15 +294,22 @@ export async function runAiReview(input: AiReviewInput): Promise<AiReviewResult>
+      findings = v.findings
+      verifier = v.status
+    }
+    return done(findings, { state: "ok", model: modelLabel, ...(reason ? { reason } : {}), ...spent, ...(verifier ? { verifier } : {}) })
   } catch (err) {
     log.error("ai review failed", { error: err })
</file context>

const roundOutput = Math.min(VERIFY_MAX_CALLS_PER_ROUND * TOOL_OUTPUT_MAX, ctx.maxToolChars - toolChars)
const roomForRound = tokens + nextCall() + nextCall(roundOutput) <= ctx.maxTokens
const overBudget = !roomForRound || toolChars > ctx.maxToolChars
const force = rounds >= ctx.maxRounds || overBudget

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: When a batch's share cannot cover even its direct-answer estimate, force still sends that call; repeated across batches, verifier usage can exceed the 12k PR budget. Skip that batch as budget-exhausted when the answer itself cannot fit, leaving its findings unreviewed.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-verify.ts, line 1003:

<comment>When a batch's share cannot cover even its direct-answer estimate, `force` still sends that call; repeated across batches, verifier usage can exceed the 12k PR budget. Skip that batch as budget-exhausted when the answer itself cannot fit, leaving its findings unreviewed.</comment>

<file context>
@@ -0,0 +1,1230 @@
+      const roundOutput = Math.min(VERIFY_MAX_CALLS_PER_ROUND * TOOL_OUTPUT_MAX, ctx.maxToolChars - toolChars)
+      const roomForRound = tokens + nextCall() + nextCall(roundOutput) <= ctx.maxTokens
+      const overBudget = !roomForRound || toolChars > ctx.maxToolChars
+      const force = rounds >= ctx.maxRounds || overBudget
+      if (overBudget && rounds < ctx.maxRounds) out.state = "budget"
+      if (force && rounds > 0) messages.push({ role: "user", content: ANSWER_NOW })
</file context>

* by the SDK; the caller runs them and appends the results. Like
* `readStreamText`, a stream error is thrown rather than lost.
*/
export async function readStreamStep(stream: {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: readStreamStep adds a new error-propagation path and an invalid passthrough, but no test covers either. Add cases to stream-text.test.ts for a mid-stream error part (must throw) and for a tool call with invalid: true.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/llm/stream-text.ts, line 32:

<comment>`readStreamStep` adds a new error-propagation path and an `invalid` passthrough, but no test covers either. Add cases to stream-text.test.ts for a mid-stream `error` part (must throw) and for a tool call with `invalid: true`.</comment>

<file context>
@@ -12,3 +12,42 @@ export async function readStreamText(stream: {
+ * by the SDK; the caller runs them and appends the results. Like
+ * `readStreamText`, a stream error is thrown rather than lost.
+ */
+export async function readStreamStep(stream: {
+  fullStream: AsyncIterable<{ type: string; error?: unknown }>
+  readonly text: PromiseLike<string>
</file context>

summary: z.number().int().nonnegative().optional(),
unreviewed: z.number().int().nonnegative().optional(),
/** Discarded findings (at most 20), for auditing the verifier's precision. */
dropped: z

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The dropped schema accepts any decision and any length, but the producer only records discards and caps the list at 20. Narrow decision to z.literal("discard") and add .max(20) to the array so the schema states the invariant.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/verdict.ts, line 237:

<comment>The `dropped` schema accepts any decision and any length, but the producer only records discards and caps the list at 20. Narrow `decision` to `z.literal("discard")` and add `.max(20)` to the array so the schema states the invariant.</comment>

<file context>
@@ -201,18 +210,80 @@ export const VerdictEnvelope = z.object({
+          summary: z.number().int().nonnegative().optional(),
+          unreviewed: z.number().int().nonnegative().optional(),
+          /** Discarded findings (at most 20), for auditing the verifier's precision. */
+          dropped: z
+            .array(
+              z.object({
</file context>

severity: d.severity === "warning" ? "warning" : "suggestion",
category: category.data,
title: d.title,
body: d.body,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: Discarded findings are rebuilt from the envelope's body, which applyTriage clamps to 600 characters. The replayed body is truncated, and the verifier and output see it that way without any replayIncomplete flag. Note the truncation in the replay status, or stop clamping for replayed records.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-replay.ts, line 121:

<comment>Discarded findings are rebuilt from the envelope's `body`, which `applyTriage` clamps to 600 characters. The replayed body is truncated, and the verifier and output see it that way without any `replayIncomplete` flag. Note the truncation in the replay status, or stop clamping for replayed records.</comment>

<file context>
@@ -0,0 +1,203 @@
+        severity: d.severity === "warning" ? "warning" : "suggestion",
+        category: category.data,
+        title: d.title,
+        body: d.body,
+        file: d.file,
+        model: modelNameFromPath(d.file),
</file context>

const hits: Array<{ path: string; line: number; text: string }> = []
for (const row of out.split("\n")) {
const rest = ref && row.startsWith(ref + ":") ? row.slice(ref.length + 1) : row
const m = /^(.+?):(\d+):(.*)$/.exec(rest)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: A valid filename like models/a:123:b.sql is split at the filename’s :123: and returned with a false path and line. Parse an unambiguous path/line format instead.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/git.ts, line 207:

<comment>A valid filename like `models/a:123:b.sql` is split at the filename’s `:123:` and returned with a false path and line. Parse an unambiguous path/line format instead.</comment>

<file context>
@@ -178,6 +178,40 @@ export async function grepMatchesAt(cwd: string, ref: string | undefined, patter
+  const hits: Array<{ path: string; line: number; text: string }> = []
+  for (const row of out.split("\n")) {
+    const rest = ref && row.startsWith(ref + ":") ? row.slice(ref.length + 1) : row
+    const m = /^(.+?):(\d+):(.*)$/.exec(rest)
+    if (!m) continue
+    hits.push({ path: m[1], line: Number(m[2]), text: m[3].slice(0, 300) })
</file context>

compiledSql: async (p, side) => {
const ctx = ctxByPath.get(p)
const engine = side === "new" ? ctx?.engineNewSql : ctx?.engineOldSql
return engine ?? (await getCompiled?.(ctx?.file.oldPath && side === "old" ? ctx.file.oldPath : p, side))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: compiled_sql falls back to on-disk compiled artifacts for a side the PR does not have

engineNewSql/engineOldSql are deliberately undefined for a deleted head or an added base (see lines 1844-1856, which never call getCompiled for status === "deleted" or without an old side). This new closure ignores that: ctx exists for a deleted model with engineNewSql === undefined, so it calls getCompiled(p, "new"), which reads target/compiled/<project>/<path> from disk with no ref or status check (compiled.ts:108-121). dbt compile does not purge stale artifacts, so compiled_sql(path, "new") — and equivalence() built on it — can show the deleted model's old SQL as head, letting the verifier confirm a claim about code that no longer exists while read_file(path, "new") correctly reports it missing.

Return engine when ctxByPath has the path (even when it is undefined) and use getCompiled only for parents/downstream models outside that map.

const near = (r: { head?: number }, i: number) => {
if (typeof line !== "number") return i < 2 * HUNK_RADIUS
// A removed row or hunk header sits beside the head row that follows it.
const at = r.head ?? rows.slice(i).find((x) => x.head !== undefined)?.head ?? right

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: hunkAround anchors removed rows to the next hunk and drops blank context rows

rows.slice(i).find(...) scans the whole accumulated row list, not the current hunk. A hunk that ends in deletions has no following head row of its own, so its - rows anchor to the next hunk's first head line: they are omitted from the excerpt for a finding in that hunk (the deleted code is missing exactly where the finding points) and included in a distant hunk's excerpt as if adjacent. Separately, row === "" (a blank context line that lost its leading space) is skipped entirely, so right is not incremented and every later row in the excerpt is numbered one too low; inHunk is also never reset for unrecognized rows.

walkDiffRows (diff-filter.ts:349-372) already numbers +, context, - and blank rows correctly — reuse it and mark hunk-header rows separately instead of re-deriving the walk here. The verifier decides confirm/refute from this excerpt, so a missing deletion or shifted numbers can refute a valid comment.

const dirs = layout
? [...layout.dirs.model, ...layout.dirs.macro, ...layout.dirs.seed, ...layout.dirs.snapshot, ...layout.dirs.test]
: ["models", "macros", "seeds", "snapshots", "tests"]
const specs = glob ? [`${root}${glob}`] : dirs.map((d) => path.posix.normalize(root + d))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: A repo-relative glob is prefixed with the project root a second time

The verifier is shown repo-relative paths (file: ${f.file} in buildVerifyMessage, and grep_project results are repo-relative), so in a monorepo the model naturally passes glob: "dbt/models/**". root + glob then becomes dbt/dbt/models/**, git matches nothing, and the tool returns the literal "no matches" — which the verifier can read as proof of absence and use to refute a correct cross-file claim.

Normalize the glob, only add projectRootRel when it is not already present, and reject any .. segment (dbt/../secrets/** currently passes the startsWith("..") filter). Keep the resulting spec inside projectRootRel.

return { decision: "keep", rule: "plausible_local" }
if (m.verdict === "unverifiable") return { decision: "discard", rule: "unverifiable" }
if (m.value === "informational") return { decision: "summary", rule: "informational" }
return { decision: "keep", rule: "verified" }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: A refuted verdict with no cited evidence is recorded as verified

Row 2 only discards a refutation that carries evidence (m.evidence.trim()), so an answer like {"verdict":"refuted","evidence":""} falls through every later rule and reaches this unconditional return. applyTriage then writes verification: { decision: "keep", rule: "verified", verdict: "refuted" } into the signed envelope and into decisions[], which is contradictory audit data for offline re-triage. (No Verified: suffix is appended because that also needs evidence, so this is a labeling defect rather than posted text.)

Make the last row conditional on m.verdict === "confirmed" and use a distinct rule such as refuted_unsupported otherwise.

return s.length > TOOL_OUTPUT_MAX ? s.slice(0, TOOL_OUTPUT_MAX) + "\n… (truncated)" : s
}

function safeReason(err: unknown): string {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Duplicate helpers instead of reusing the AI lane's

safeReason here is byte-for-byte the same as ai-review.ts:190; positiveNumber duplicates aiInputBudget's num (ai-review.ts:37-40); and BATCH_CONCURRENCY is declared in both modules. Export the existing helpers and import them so key/bearer-token masking and env parsing cannot drift between the two lanes. backticked() in this file is also a strict subset of quotedNames() and can be dropped.

}
let body = f.body
const suffix = verification?.rule === "verified" && verification.evidence ? `\n\nVerified: ${verification.evidence}` : ""
if (suffix && body.endsWith(suffix)) body = body.slice(0, -suffix.length)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: unverify only strips a Verified: suffix that is the tail of the body, which mergeRepeats breaks

applyTriage appends the suffix, but cleanFindings then runs mergeRepeats (finding.ts:291-295), which rewrites the first finding of a group to ${first.body}\n\nSame rule, also here:…. If a verified finding wins that tie-break, the stored body is …Verified: <evidence>\n\nSame rule, also here:…, so body.endsWith(suffix) is false and replay keeps a Verified: claim the new verifier never made (it is fed f.body by buildVerifyMessage/extractFeatures, and is posted if re-kept). The merged (+N more) title is likewise not undone.

Remove the last occurrence of the suffix instead of requiring a tail match, and strip the (+N more) marker when rebuilding the generator's finding.

@kilo-code-bot

kilo-code-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Code Review Summary

Status: 6 Issues Found | Recommendation: Address before merge

Incremental review of f3c5e86 (the grep_project / column_reach follow-up), with prior findings re-checked against current HEAD. The grep_project failure-vs-empty fix, the projectGrepSpecs double-prefix/.. fix, and the unknown-model columnReach handling all check out; the issues below remain open.

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 4
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/orchestrate.ts 2484 compiled_sql falls back to on-disk target/compiled artifacts for a side the PR does not have, so a deleted model's stale head SQL can be shown to the verifier (and used by equivalence).
packages/opencode/src/altimate/review/ai-verify.ts 370 hunkAround anchors removed rows to the next hunk and drops blank context rows, shifting head-side line numbers in the excerpt the verifier decides from. walkDiffRows already does this correctly.

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/ai-verify.ts 738 The verifier's safeReason duplicates the AI lane's exported helper but omits redactSecrets, so its error strings mask less; positiveNumber and BATCH_CONCURRENCY are also duplicated, and backticked() is a subset of quotedNames().
packages/opencode/src/altimate/review/ai-verify.ts 473 A refuted verdict with no cited evidence falls through to keep/verified, recording a self-contradictory verification in the envelope.
packages/opencode/src/altimate/review/ai-replay.ts 156 unverify only strips a Verified: suffix at the end of the body; mergeRepeats appends after it, so replay keeps stale Verified: text and the (+N more) title.
packages/opencode/src/altimate/review/git.ts 253 projectGrepSpecs keeps the trailing slash on root, so glob equal to the project root ("dbt") becomes the pathspec dbt/dbt and never searches the project.
Files Reviewed (7 files)
  • packages/opencode/src/altimate/review/git.ts - 1 issue
  • packages/opencode/src/altimate/review/run.ts - clean
  • packages/opencode/src/altimate/review/runner.ts - clean
  • packages/opencode/src/altimate/review/ai-verify.ts - 3 issues
  • packages/opencode/src/altimate/review/ai-replay.ts - 1 issue
  • packages/opencode/src/altimate/review/orchestrate.ts - 1 issue
  • packages/opencode/test/altimate/review-ai-verify.test.ts - clean

Fix these issues in Kilo Cloud

Previous Review Summary (commit 8aba994)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 8aba994)

Status: 6 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 3
SUGGESTION 3
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/orchestrate.ts 2390 compiled_sql falls back to on-disk target/compiled artifacts for a side the PR does not have, so a deleted model's stale head SQL can be shown to the verifier (and used by equivalence).
packages/opencode/src/altimate/review/ai-verify.ts 369 hunkAround anchors removed rows to the next hunk and drops blank context rows, shifting head-side line numbers in the excerpt the verifier decides from. walkDiffRows already does this correctly.
packages/opencode/src/altimate/review/run.ts 436 A repo-relative glob is prefixed with the project root a second time (dbt/dbt/...), returning a false no matches and ignoring .. segments inside the glob.

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/ai-verify.ts 472 A refuted verdict with no cited evidence falls through to keep/verified, recording a self-contradictory verification in the envelope.
packages/opencode/src/altimate/review/ai-verify.ts 737 safeReason, positiveNumber and BATCH_CONCURRENCY duplicate the AI lane's helpers; backticked() duplicates quotedNames().
packages/opencode/src/altimate/review/ai-replay.ts 156 unverify only strips a Verified: suffix at the end of the body; mergeRepeats appends after it, so replay keeps stale Verified: text and the (+N more) title.
Files Reviewed (16 files)
  • packages/opencode/src/altimate/llm/stream-text.ts - clean
  • packages/opencode/src/altimate/review/ai-replay.ts - 1 issue
  • packages/opencode/src/altimate/review/ai-review.ts - clean
  • packages/opencode/src/altimate/review/ai-verify.ts - 3 issues
  • packages/opencode/src/altimate/review/finding.ts - clean
  • packages/opencode/src/altimate/review/format.ts - clean (one demotion hole already covered by an existing comment)
  • packages/opencode/src/altimate/review/git.ts - clean (already covered by existing comments)
  • packages/opencode/src/altimate/review/orchestrate.ts - 1 issue
  • packages/opencode/src/altimate/review/run.ts - 1 issue
  • packages/opencode/src/altimate/review/runner.ts - clean (already covered by an existing comment)
  • packages/opencode/src/altimate/review/telemetry.ts - clean
  • packages/opencode/src/altimate/review/verdict.ts - clean
  • packages/opencode/src/altimate/telemetry/index.ts - clean
  • packages/opencode/test/altimate/review-ai-inputs.test.ts - clean
  • packages/opencode/test/altimate/review-ai-replay.test.ts - clean
  • packages/opencode/test/altimate/review-ai-verify.test.ts - clean

Fix these issues in Kilo Cloud


Reviewed by deepseek-v4.1-flash-1017 · Input: 176K · Output: 22.3K · Cached: 1.5M

Review guidance: REVIEW.md from base branch feat/review-posting-policy

@hkaltai hkaltai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: approve with minor changes. Panel: Claude, Codex, and a lead pass, all covering every changed file (reviewed against the stack base only). No CRITICAL or MAJOR defects; flag-off behaviour is unchanged and the verifier fails open as described. bun test test/altimate/review passes (667).

Severity Count
CRITICAL 0
MAJOR 0
MINOR 6 (inline)
NIT 6

Consensus (2+ reviewers): config-rule override of summary demotion, replay dropping findings on model failure. Single-reviewer findings were verified against the code. A Codex-rated MAJOR (token budget exceeded) was downgraded to MINOR: it fails safe and degrades to the summary, but the 12k bound is not hard. Missing tests worth adding: grepFixedAt / grepProject wiring, new review_run telemetry fields, and the regressions for the findings above.

Nits

  • telemetry/index.ts:1623 /** AI findings the verifier kept (including those moved to the summary) and discarded. */ - summary decisions are counted separately and summary/unreviewed are not emitted. Fix the comment or emit ai_verify_summary / ai_verify_unreviewed.
  • ai-verify.ts:1224 log.error("ai verify failed", ...) path returns findings unchanged even with ALTIMATE_REVIEW_AI_VERIFY_FAIL=closed. Apply failMode in the catch or document that closed mode is per-verdict only.
  • ai-verify.ts:369 rows.slice(i).find(...) is O(n^2) per candidate on large diffs. Precompute next head line in one backward pass.
  • ai-verify.ts:88, :447-449 experiment notes ("opt-in until the decision run says otherwise", "in the decision run about half of these discards were judged valuable") will go stale; move to the PR/design doc. :3 ("One extra, bounded model call") and :637 ("may spend before it must answer") overstate a hard bound.
  • ai-replay.ts:78-82 "as the generator built them" overstates fidelity: discarded bodies are clamped to 600 chars and kept findings have been through anchor/clean post-processing. Persist raw generator findings or document, and strip preExisting/evidenceLine in unverify.
  • Naming: ts/TsFeatures (saved as envelope key decisions[].ts) reads as timestamp/TypeScript; rename to features. kept vs summary is ambiguous in status/telemetry. DroppedFinding.decision is always "discard". Generic exported Decision/Verification names. orchestrate.ts:162-167 graphParents doc says project-relative but paths are repo-relative.

Also worth noting: verifier tokens/cost are not added to ai.cost/ai.tokens (only under ai.verifier), and ALTIMATE_REVIEW_AI_REPLAY is honoured in production runAiReview.

* the PR added, at or above the severity floor. */
export function inlineEligible(f: Finding, posting: PostingPolicy): boolean {
if (f.tier !== "inline" && findingOrigin(f) !== "ai") return false
if (f.tier !== "inline" && (findingOrigin(f) !== "ai" || verifierDemoted(f))) return false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Verifier demotion can be overridden by a config rule tier

if (f.tier !== "inline" && (findingOrigin(f) !== "ai" || verifierDemoted(f))) return false

verifierDemoted is only consulted when tier !== "inline", and cleanFindings re-tiers every finding from ruleTier(f.rule, config.rules) after the AI lane, overwriting the summary tier set by triage. With a rule such as ai-review/*: inline in the review config, an unverifiable cross-file finding the verifier moved to the summary becomes inline-eligible again and is posted inline. Reproduced independently by two reviewers.

Fix: check verifierDemoted(f) first (return false), or keep summary in cleanFindings when verification.decision === "summary". Add a regression test.

model = await Provider.getModel(chosen.providerID, chosen.modelID)
} catch (err) {
const reason = `no usable model: ${err instanceof Error ? err.message : String(err)}`.slice(0, 200)
return done([], { ...lane.status, state: "skipped", reason })

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Replay drops all replayed findings when no verifier model resolves

return done([], { ...lane.status, state: "skipped", reason })

The generator findings are already rebuilt at this point, but are discarded on a model-lookup failure, contradicting the module's fail-open design. Scenario: ALTIMATE_REVIEW_AI_REPLAY=env.json ALTIMATE_REVIEW_AI_VERIFY=1 with an unavailable or misnamed --ai-model yields zero AI findings, silently skewing an A/B comparison.

Fix: return lane.findings unchanged with verifier: { state: "skipped", reason }, and add a test.

? { decision: "keep", rule: "unreviewed" }
: { decision: "discard", rule: "unreviewed" }
if (ts.restatesGrounding) return { decision: "discard", rule: "duplicate_of_engine" }
if (m.verdict === "refuted" && m.evidence.trim()) return { decision: "discard", rule: "refuted" }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Refuted verdict without evidence is kept as "verified" while unverifiable is discarded

if (m.verdict === "refuted" && m.evidence.trim()) return { decision: "discard", rule: "refuted" }

A refuted verdict with blank evidence skips this row and every unverifiable row, and falls through to keep / "verified". For a finding with evidenceLineValid: false, unverifiable is discarded but refuted with empty evidence is kept and labelled verified, so the model saying the claim is wrong makes it more likely to survive.

Fix: normalise refuted-without-evidence to unverifiable before the remaining rows (or give it its own rule name so it is never labelled verified), and test the invalid-line case.

Math.ceil((requestChars(ctx.fixedChars, messages) + ANSWER_NOW.length + extra) / charsPerToken) +
VERIFY_OUTPUT_RESERVE
const roundOutput = Math.min(VERIFY_MAX_CALLS_PER_ROUND * TOOL_OUTPUT_MAX, ctx.maxToolChars - toolChars)
const roomForRound = tokens + nextCall() + nextCall(roundOutput) <= ctx.maxTokens

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Token budget is a planning target, not a bound; tool rounds rarely run

const roomForRound = tokens + nextCall() + nextCall(roundOutput) <= ctx.maxTokens

Two related effects. (1) The forced answer is always sent, even when a batch's minimum request exceeds its budget share, and the output reserve is not enforced; with 40 findings (5 batches) an in-memory harness spent ~26.7k tokens against the 12k default, so the documented 12k bound can be exceeded. (2) roundOutput assumes the worst case (6 x 2,500 chars) and maxTokens is split across batches, so with 8 findings (one batch) the first call is already toolChoice: "none" and with 9 findings both calls are; the tool loop effectively never runs on mid-size PRs and the state reports budget. Fails safe (cross-file claims go to the summary), but limits both the cost bound and the verifier's usefulness.

Fix: plan with a realistic round cost (e.g. TOOL_OUTPUT_MAX * 2) or give each batch a floor allowing one round; skip batches whose minimum request cannot fit (keep findings unchanged); document the cap as soft otherwise.

if (ctx.abort.aborted) throw ctx.abort.reason ?? new Error("aborted")
if (force || step.finishReason !== "tool-calls" || !step.toolCalls.length) {
out.verdicts = parseVerifierOutput(step.text, ids)
if (!out.verdicts.size && out.state === "ok") {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Unparsable forced answer is reported as budget, not error

if (!out.verdicts.size && out.state === "ok") {

When the answer was forced early (state: "budget") and the model returns nothing parsable (e.g. []), the batch keeps budget with no error, although every finding is unreviewed. Reproduced with 8 findings and an LLM reply of "[]". Telemetry ai_verify_state then hides verifier failures.

Fix: set error/reason whenever verdicts.size === 0, regardless of the earlier state.

toolChoice: force ? "none" : "auto",
abort: ctx.abort,
}
const step = await ctx.llm(req)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Timeout cannot interrupt awaited evidence tools

const step = await ctx.llm(req)

The timeout aborts model requests only; await runToolCall(...) in the same loop (and graph gathering before the timer starts) is not raced against the abort signal. One reviewer reproduced a pending evidence read keeping the verifier unresolved after a 10 ms timeout until the read was released. Local git/file reads rarely stall, so impact is limited, but it weakens the fail-open-on-timeout guarantee.

Fix: apply one deadline across graph gathering, tools and model calls (race pending work against the abort signal and stop scheduling after expiry).

RaghwendraSingh and others added 13 commits November 20, 2026 12:18
Adds the skeleton of a dbt-grounded verifier for the advisory AI lane,
behind `ALTIMATE_REVIEW_AI_VERIFY=1`. With the variable unset nothing
changes: the lane gets no `evidence` and the envelope is identical.

- `ai-verify.ts`: `AiVerifyEvidence`, `VerifierStatus`, `aiVerifyEnabled`
  and a `runAiVerify` that returns the findings unchanged for now
- `runAiReview` runs the verifier after the review batches when given
  `evidence` and the variable is set; `AiLaneStatus.verifier` reports it
- `runReview` builds the evidence when `aiVerify` is set: engine SQL,
  numbered file slices, added lines, PR text and a project grep
- `run.ts` passes `aiVerify` and a `git grep -F` backed `grepProject`
  (5 s timeout, 50 hits) over the project's dbt directories
- `ReviewRunner.columnReach` wraps `columnReach` without throwing
- `readStreamStep` reads one AI SDK step keeping its tool calls and
  response messages
- envelope `ai.verifier` and `review_run` fields `ai_verify_state`,
  `ai_verify_tokens_total`, `ai_verify_tool_calls`, `ai_verify_kept`,
  `ai_verify_discarded`

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`extractFeatures` computes, without a model, the facts the verifier's
triage table reads for each AI finding:

- `evidenceLineValid`: the finding's line is on the diff (null without one)
- `targetsUnchangedCode`: a diff line the PR did not add, a `preExisting`
  tag, or no usable line and nothing the finding names on an added line
- `restatesGrounding`: title-word Jaccard of at least 0.5 with an engine
  finding on the same file, or the same category and column
- `claimsOtherFile`: a `ref()` to another model, a `source()`, or a
  backticked name of a dbt parent or downstream model
- `styleOnly`, `lowConfidence`, `hedgedDataDiff`

AI findings reach the verifier before `anchorToDiff` tags `preExisting`,
so a commentable line outside the added set counts as unchanged code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `VERIFY_SYSTEM_PROMPT`: an adversarial verifier that tries to refute
  each candidate from its hunk and tool results, treats the diff, SQL, PR
  text and tool output as untrusted data, and answers with one JSON
  element per candidate (`claim_type`, `verdict`, `value`,
  `requires_assumed_input`, `evidence`, `tools_used`)
- `buildVerifyMessage`: the PR title and, per candidate, its id, file,
  line, category, severity, title, body and the diff rows within 15 lines
  of it (`hunkAround`, at most 2,000 chars); never the whole file or the
  engine findings the reviewer saw
- `parseVerifierOutput`: strips `<think>` blocks and fences, takes the
  first balanced JSON array, ignores unknown ids and coerces bad enums
  (`verdict` to unverifiable, `claim_type` to code_fact, `value` to
  actionable); ids it does not return stay unreviewed

The prompt lives in TypeScript for the decision run and can move to core
later.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`triage` decides each AI finding from its deterministic features and the
verifier's verdict, first matching row wins: grounding duplicate,
refuted with evidence, unverifiable cross-file claim, speculation or an
assumed input, style or nit, unchanged code, plausible local claim
(kept), unverifiable, informational (summary), verified (kept). Without
a verdict the fail mode decides: open keeps the finding unchanged.

`applyTriage` records `evidence.result.verification` on reviewed kept
findings (the evidence tool stays `ai-review`), appends `Verified: ...`
to verified bodies, never raises severity, and lists at most 20 discards.
`inlineEligible` no longer posts an AI finding inline once the verifier
moved it to the summary.

Knobs for experiments: `ALTIMATE_REVIEW_AI_VERIFY_DROP_NITS=0`,
`ALTIMATE_REVIEW_AI_VERIFY_KEEP_LOCAL=0`, `ALTIMATE_REVIEW_AI_VERIFY_FAIL=closed`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`runAiVerify` now verifies the AI lane's findings: deterministic
features, then one model conversation per batch of at most 8 findings
(kept together by file), then triage. It never throws; whatever it could
not review comes back unchanged.

- Ten read-only tools, declared with `tool()` and no `execute`; the loop
  runs the calls itself and appends `tool-result` messages:
  `read_file` (changed files, their dbt parents and direct downstream
  models only), `compiled_sql`, `model_columns`, `column_reach`,
  `model_config`, `downstream`, `equivalence`, `grain`, `grep_project`,
  `pr_description`. Every result is a string of at most 2,500 chars;
  failures read `not available: ...`
- `LLM.stream` runs one step, so the loop drives the rounds: up to 3 tool
  rounds and 6 calls per round, then a forced answer with
  `toolChoice: "none"`; the answer is forced early once the batch's
  share of 20,000 tokens (`ALTIMATE_REVIEW_AI_VERIFY_MAX_TOKENS`) or
  16,000 tool-output chars is spent (state `budget`)
- one `AbortController` per PR with `aiTimeoutMs` of the prompt size
  (state `timeout`); a model error or an unparsable answer leaves that
  batch's findings unreviewed
- `harnessVerifierLlm` makes the call through `LLM.stream` at
  temperature 0 and reports usage with `streamUsage`; tests inject a
  scripted `VerifierLlm`
- status: turns, tool calls, tokens, cost, reviewed, kept, discarded,
  summary, unreviewed and up to 20 dropped findings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The verifier's default budget drops from 20,000 to 12,000 tokens per PR
(`ALTIMATE_REVIEW_AI_VERIFY_MAX_TOKENS` still overrides it), and the
forced final answer now counts against it. Before each call the loop
plans the rest of the batch: another tool round is taken only if that
call, plus the forced answer after it with a full round of tool output,
fits the batch's share. Otherwise it answers now with
`toolChoice: "none"`, from the first call on if need be. Call sizes are
estimated from the characters sent (system prompt, tool declarations,
messages) plus a 1,000-token output reserve, calibrated by the usage
the provider reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every verifier call repeats the system prompt and the tool declarations,
so they are kept to the minimum that stays unambiguous: shorter tool and
`side` descriptions, and `read_file` line bounds and the `grep_project`
pattern length checked in code rather than in the JSON schema (which
emitted `exclusiveMinimum`/`maximum` noise). Tool declarations go from
3,408 to 2,950 chars; the fixed per-call size from 4,832 to 4,374.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ai.verifier.decisions[]` now records every AI finding the verifier saw,
uncapped: its deterministic features, the model's verdict (claim type,
verdict, value, assumed input, tools used, and whether it gave
evidence; absent when unreviewed), and the decision and rule triage
applied. `dropped[]` is unchanged.

`retriage(decisions, knobs)` re-runs triage over those records without
a model call, so one run's envelopes answer what `dropNits`,
`keepUnverifiedLocal` and `failMode` would have changed. It also takes a
hypothetical `claimsOtherFileAsCrossFile` knob that sends findings
naming another model or source through the cross-file rule; live triage
still does not use `claimsOtherFile`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ards

`ai.verifier.dropped[]` now carries each discard's `startLine` and body
(clamped to 600 chars), so a verifier run can be scored both with and
without its discards. That makes the no-verifier baseline a free
by-product of the same run instead of a separate one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… budget

When a batch's prompt left no room for a tool round, the first call was
sent with `toolChoice: "none"` but no instruction to answer. The model
then returned about 18 output tokens and no parsable verdicts, so every candidate in the batch went unreviewed.
In the decision run this hit 10 PRs (dev and heldout) with large batches. The first
message now ends with an explicit answer-now line in that case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the summary

The verifier discarded these. In the 50-PR dev decision run, 10 of 21
`unverifiable_cross_file` discards and 9 of 19 `speculative` discards
were judged valuable, and every catch the verifier lost came from the
first rule. They now stay in the review summary, never inline.

Offline re-triage of the same run (no model calls), dev: catches 18 vs
15 with discarding (18 without the verifier); verified AI findings
(outside the summary) unchanged at 0.84/PR, 69% valuable, 0% wrong; remaining
discards 2 of 18 valuable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…verifier

`ALTIMATE_REVIEW_AI_REPLAY=<earlier review --json envelope>` skips the
generator call in `runAiReview` and rebuilds its findings from that
envelope, so verifier settings can be compared for verifier tokens only.

- New `ai-replay.ts`: kept AI findings lose the verifier's
  `evidence.result.verification`, the `Verified:` body suffix, and the
  tier/origin set after generation; `ai.verifier.dropped` entries are
  rebuilt as `Finding`s with the stored `id` and `rule`. Order follows
  `ai.verifier.decisions`. A `dropped` list shorter than the discard
  count is warned on stderr and reported as `replayIncomplete`.
- The AI lane status copies the stored generator `tokens`/`calls`/`cost`
  and adds `replayed: true` (envelope schema and `ai_replayed` telemetry).
  An unreadable envelope is an error on stderr and `state: "error"`; the
  generator is never called as a fallback.
- `ALTIMATE_REVIEW_AI_VERIFY_MAX_TOOL_CHARS` and
  `ALTIMATE_REVIEW_AI_VERIFY_MAX_ROUNDS` override the verifier budgets;
  the prompt states the effective round limit. Defaults unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…project false evidence

- `runner.ts`: `columnReach` answers an unknown model (or no manifest) with
  `undefined` instead of recording an engine failure, so a verifier tool
  argument can no longer land in `engineErrors` and downgrade the verdict.
- `git.ts`: `grepFixedAt` returns `undefined` when git fails or times out with
  nothing found, and for an empty pathspec list (which would search the whole
  repo); only exit 1 (no match) returns `[]`.
- `ai-verify.ts`: `grep_project` reports a failed search as failed ("not
  evidence of absence") rather than "no matches".
- `git.ts`/`run.ts`: new `projectGrepSpecs` builds the search pathspecs; a glob
  that already carries the project prefix (the verifier sees repo-relative
  paths) is not prefixed again, and a glob with `..` segments is rejected.
- Tests for each.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Thanks for updating your PR! It now meets our contributing guidelines. 👍

if (glob) {
const g = glob.trim().replace(/^\.?\/+/, "")
if (!g || g.split("/").includes("..")) return undefined
return [root && !g.startsWith(root) ? root + g : g]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: A glob equal to the project root is prefixed a second time

root keeps its trailing slash, so glob: "dbt" (the project root itself, a natural way to ask for the whole project) fails the g.startsWith(root) test and becomes the pathspec dbt/dbt. That directory cannot exist, so the search covers nothing and comes back as no matches/failed instead of searching dbt/ — the same false "evidence of absence" this commit set out to remove. "dbt/" and "dbt/**" work, only the bare root does not.

Suggested change
return [root && !g.startsWith(root) ? root + g : g]
return [root && g !== root.slice(0, -1) && !g.startsWith(root) ? root + g : g]

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants