Skip to content

fix(review): an honest PR comment: real lint-only banner, AI lane status, PR intent, project_dir - #1720

Open
RaghwendraSingh wants to merge 3 commits into
fix/review-scope-difffrom
fix/review-ci-experience
Open

RaghwendraSingh wants to merge 3 commits into
fix/review-scope-difffrom
fix/review-ci-experience

Conversation

@RaghwendraSingh

@RaghwendraSingh RaghwendraSingh commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Changes since review: Review feedback commit: AI-lane parse failures and model errors are reported explicitly, the lane reason is credential-masked, unverified models get one note per reason, and the docs are updated.

Issue for this PR

Closes #1792
Stacked on #1719; please review and merge that one first. Takes ideas from the closed #1241, not its code.

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

The review comment could misstate what it checked:

  • Lint-only banner: a single undecidable finding set degraded, so every manifest-backed run said "Lint-only run — no dbt manifest". Now it shows only when no changed model resolved against a manifest. Undecidable findings, models without compiled SQL, and files outside the project each get their own line.
  • Silent AI lane: the lane returned [] on every failure, so an expired key, a timeout, or a missing model looked like "no AI findings". It now reports ok | skipped | timeout | error with a reason (credentials masked), the model, and duration. These land in a new signed ai field and in the comment. The timeout now scales with prompt size (60 s + 1 s per 2K chars, capped at 5 min) instead of a fixed 60 s. --ai-model provider/model picks the model explicitly.
  • No PR intent: prTitle/prBody existed but nothing passed them. The CLI now reads them from the GitHub pull_request event, or from GitLab's CI_MERGE_REQUEST_TITLE/_DESCRIPTION.
  • Action: new project_dir input, mapped to --cwd, for monorepos.
  • Docs: a table of what each input unlocks (fetch depth, manifest, compiled SQL, target-base/, catalog.json) and a snippet for compiling the base.
  • Telemetry: review_run gains undecided, unverified_models, ai_state, ai_duration_ms, base_source, out_of_scope_files.

Open question, deliberately not changed: --no-ai still auto-registers the free tier, which is a network call. A release test asserts that on purpose, so the decision is the maintainers'. Changing it is a one-line guard.

How did you verify your code works?

  • New test/altimate/review-ci-honesty.test.ts (13 tests):
    • banner with and without a manifest
    • AI lane: disabled, outside the tier, timeout (signature still verifies), throwing, ok with model and selector, legacy array return
    • timeout scaling
    • PR context from GitHub and GitLab env, and its forwarding to the lane
  • 1,859 tests across the review files and release gates pass. bun typecheck is clean.

Cost side effects

None. The AI lane's timeout ceiling can rise from 60 s to 5 min on very large prompts, but the number of model calls doesn't change (still at most one per review).

Screenshots / recordings

N/A (CLI).

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

🤖 Generated with Claude Code


Summary by cubic

Makes the dbt PR review comment candid about what it checked. Previously a single undecidable finding labeled any manifest-backed run as "Lint-only", the advisory AI lane returned [] on every failure, and no PR intent reached the AI check.

Review honesty

  • The lint-only banner now shows only when no changed model resolved against a manifest; undecidable findings, models without compiled SQL, and out-of-scope files each get their own line.
  • The AI lane reports ok | skipped | timeout | error with a reason, model, and duration on a new signed ai envelope field and in the comment; it never fails silently.
  • Lane reasons are masked with the shared redactSecrets masker (basic auth, bearer tokens, URL userinfo, key=... secrets) before reaching the public comment, and the lane never throws.
  • A core parse failure (success: false) is flagged as error, and an unresolvable explicit --ai-model errors while a missing default only skips.
  • The AI timeout scales with prompt size (60 s + 1 s per 2K chars, capped at 5 min); --ai-model provider/model selects the model explicitly.
  • The CLI reads the PR title and body from GitHub's pull_request event or GitLab's CI_MERGE_REQUEST_TITLE/_DESCRIPTION for the intent check.

Inputs, docs, and telemetry

  • New project_dir input maps to --cwd for monorepos; manifest_path resolves relative to it.
  • review_run telemetry gains undecided, unverified_models, ai_state, ai_duration_ms, base_source, and out_of_scope_files.
  • Docs now list what each input unlocks and how to compile the base into target-base/, including a project_dir form.

Deliberately unchanged: --no-ai still auto-registers the free tier, which is a network call; a release test asserts that, so the decision is the maintainers'. Model call count is unchanged (still at most one per review); only the timeout ceiling rises, to 5 minutes on very large prompts.

Written for commit e8a893e. Summary will update on new commits.

View guided diff Turn on auto-fix

@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1797aeea-755a-42f2-b344-45676944ff42

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown

Thanks for your contribution!

This PR doesn't have a linked issue. All PRs must reference an existing issue.

Please:

  1. Open an issue describing the bug/feature (if one doesn't exist)
  2. Add Fixes #<number> or Closes #<number> to this PR description

See CONTRIBUTING.md for details.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6 issues found across 11 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/opencode/src/altimate/telemetry/index.ts">

<violation number="1" location="packages/opencode/src/altimate/telemetry/index.ts:1606">
P3: `ai_state` is typed as `string`, but the AI lane only produces `ok | skipped | timeout | error`. Narrow the type to that union so the telemetry contract matches the envelope schema and typos are caught at compile time.</violation>
</file>

<file name="packages/opencode/test/altimate/review-ci-honesty.test.ts">

<violation number="1" location="packages/opencode/test/altimate/review-ci-honesty.test.ts:157">
P3: This test creates a temp directory and never removes it, so every run leaves a `review-event-*` directory behind. Remove it in a `finally` block (`fs.rm(dir, { recursive: true, force: true })`) or use the repo's tmpdir fixture with auto cleanup.</violation>
</file>

<file name="packages/opencode/src/altimate/review/ai-review.ts">

<violation number="1" location="packages/opencode/src/altimate/review/ai-review.ts:66">
P2: `safeReason` still exposes credential formats that do not use the four dashed prefixes, so provider failures can disclose secrets in PR comments. Reuse the repository's credential/error redactor or mask credential-labeled values and credential-bearing URLs before truncating the reason.

(Based on your team's feedback about masking credential-stem error values.) [a5c5a10f-6a4a-4e42-8c2e-f8e01b48683d]</violation>
</file>

<file name="packages/opencode/src/altimate/review/orchestrate.ts">

<violation number="1" location="packages/opencode/src/altimate/review/orchestrate.ts:1418">
P2: Empty-scope reviews now emit an AI `skipped` status and coverage note even though there is no review scope or AI lane to skip. Guard the AI-status block on a non-empty review scope so empty reviews leave `ai` undefined.</violation>
</file>

<file name="github/review/action.yml">

<violation number="1" location="github/review/action.yml:189">
P2: A misspelled `project_dir` silently broadens a monorepo review to every changed file. Validate that the directory contains `dbt_project.yml`/`.yaml` before adding `--cwd`, or make the review fail closed when project discovery fails.</violation>
</file>

<file name="packages/opencode/src/altimate/review/format.ts">

<violation number="1" location="packages/opencode/src/altimate/review/format.ts:144">
P2: The PR comment does not report the AI lane’s available model and duration: successful runs show only the model, while failed or timed-out runs show only state and reason. Include model and duration in the rendered AI status.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread docs/docs/usage/dbt-pr-review.md Resolved
return raw
.replace(/\s+/g, " ")
.replace(/(bearer\s+)[^\s]+/gi, "$1***")
.replace(/\b(sk|pk|key|token)[-_][A-Za-z0-9_-]{8,}/gi, "***")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: safeReason still exposes credential formats that do not use the four dashed prefixes, so provider failures can disclose secrets in PR comments. Reuse the repository's credential/error redactor or mask credential-labeled values and credential-bearing URLs before truncating the reason.

(Based on your team's feedback about masking credential-stem error values.) [a5c5a10f-6a4a-4e42-8c2e-f8e01b48683d]

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-review.ts, line 66:

<comment>`safeReason` still exposes credential formats that do not use the four dashed prefixes, so provider failures can disclose secrets in PR comments. Reuse the repository's credential/error redactor or mask credential-labeled values and credential-bearing URLs before truncating the reason.

(Based on your team's feedback about masking credential-stem error values.) [a5c5a10f-6a4a-4e42-8c2e-f8e01b48683d]</comment>

<file context>
@@ -31,6 +32,39 @@ export interface AiReviewInput {
+  return raw
+    .replace(/\s+/g, " ")
+    .replace(/(bearer\s+)[^\s]+/gi, "$1***")
+    .replace(/\b(sk|pk|key|token)[-_][A-Za-z0-9_-]{8,}/gi, "***")
+    .slice(0, 200)
 }
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4d9a88a. safeReason now runs the shared redactSecrets masker (Authorization headers, Basic/Bearer values, key=value secrets, URL userinfo, provider-prefixed tokens) before collapsing whitespace. Tests added.

// headless-without-model) — the review degrades to deterministic-only.
if (lanes.has("ai_review") && input.aiReview) {
let aiStatus: AiLaneStatus | undefined
if (!lanes.has("ai_review")) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Empty-scope reviews now emit an AI skipped status and coverage note even though there is no review scope or AI lane to skip. Guard the AI-status block on a non-empty review scope so empty reviews leave ai undefined.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/orchestrate.ts, line 1418:

<comment>Empty-scope reviews now emit an AI `skipped` status and coverage note even though there is no review scope or AI lane to skip. Guard the AI-status block on a non-empty review scope so empty reviews leave `ai` undefined.</comment>

<file context>
@@ -1410,7 +1414,12 @@ export async function runReview(input: OrchestrateInput): Promise<VerdictEnvelop
   // headless-without-model) — the review degrades to deterministic-only.
-  if (lanes.has("ai_review") && input.aiReview) {
+  let aiStatus: AiLaneStatus | undefined
+  if (!lanes.has("ai_review")) {
+    aiStatus = { state: "skipped", reason: `not part of the ${tier} tier` }
+  } else if (!input.aiReview) {
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing: the note is accurate ("no changed model SQL to review") and tells the reader why the lane didn't run.

Comment thread github/review/action.yml
ALTIMATE_REVIEW_SIGNING_KEY: ${{ inputs.signing_key }}
run: |
args=(--mode "$IN_MODE" --manifest "$IN_MANIFEST" --severity "$IN_SEVERITY")
[[ -n "$IN_PROJECT_DIR" && "$IN_PROJECT_DIR" != "." ]] && args+=(--cwd "$IN_PROJECT_DIR")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: A misspelled project_dir silently broadens a monorepo review to every changed file. Validate that the directory contains dbt_project.yml/.yaml before adding --cwd, or make the review fail closed when project discovery fails.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At github/review/action.yml, line 189:

<comment>A misspelled `project_dir` silently broadens a monorepo review to every changed file. Validate that the directory contains `dbt_project.yml`/`.yaml` before adding `--cwd`, or make the review fail closed when project discovery fails.</comment>

<file context>
@@ -181,6 +186,7 @@ runs:
         ALTIMATE_REVIEW_SIGNING_KEY: ${{ inputs.signing_key }}
       run: |
         args=(--mode "$IN_MODE" --manifest "$IN_MANIFEST" --severity "$IN_SEVERITY")
+        [[ -n "$IN_PROJECT_DIR" && "$IN_PROJECT_DIR" != "." ]] && args+=(--cwd "$IN_PROJECT_DIR")
         [[ -n "$IN_BASE" ]] && args+=(--base "$IN_BASE")
         [[ -n "$IN_HEAD" ]] && args+=(--head "$IN_HEAD")
</file context>
Suggested change
[[ -n "$IN_PROJECT_DIR" && "$IN_PROJECT_DIR" != "." ]] && args+=(--cwd "$IN_PROJECT_DIR")
if [[ -n "$IN_PROJECT_DIR" && "$IN_PROJECT_DIR" != "." ]]; then
project_path="${GITHUB_WORKSPACE:-$PWD}/$IN_PROJECT_DIR"
if [[ ! -f "$project_path/dbt_project.yml" && ! -f "$project_path/dbt_project.yaml" ]]; then
echo "::error::project_dir must contain dbt_project.yml or dbt_project.yaml"
exit 1
fi
args+=(--cwd "$IN_PROJECT_DIR")
fi

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. The action should check that dbt_project.y(a)ml exists before --cwd.

notes.push(`${env.scope.outOfScopeFiles} changed file(s) outside the dbt project were not reviewed.`)
if (env.ai && env.ai.state !== "ok") {
const what = { skipped: "skipped", timeout: "timed out", error: "failed" }[env.ai.state]
notes.push(`Advisory AI review ${what}${env.ai.reason ? `: ${env.ai.reason}` : ""}.`)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The PR comment does not report the AI lane’s available model and duration: successful runs show only the model, while failed or timed-out runs show only state and reason. Include model and duration in the rendered AI status.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/format.ts, line 144:

<comment>The PR comment does not report the AI lane’s available model and duration: successful runs show only the model, while failed or timed-out runs show only state and reason. Include model and duration in the rendered AI status.</comment>

<file context>
@@ -110,11 +111,41 @@ export function renderSummary(env: VerdictEnvelope): string {
+    notes.push(`${env.scope.outOfScopeFiles} changed file(s) outside the dbt project were not reviewed.`)
+  if (env.ai && env.ai.state !== "ok") {
+    const what = { skipped: "skipped", timeout: "timed out", error: "failed" }[env.ai.state]
+    notes.push(`Advisory AI review ${what}${env.ai.reason ? `: ${env.ai.reason}` : ""}.`)
+  }
+  return notes.length ? [...notes.map((n) => `> ℹ️ ${n}`), ""] : []
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing: the model is already shown in the footer; duration is in the envelope and telemetry and isn't useful in the comment.

/** Models the engine could not verify (no compiled SQL). */
unverified_models?: number
/** Advisory AI lane outcome: ok | skipped | timeout | error. */
ai_state?: string

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: ai_state is typed as string, but the AI lane only produces ok | skipped | timeout | error. Narrow the type to that union so the telemetry contract matches the envelope schema and typos are caught at compile time.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/telemetry/index.ts, line 1606:

<comment>`ai_state` is typed as `string`, but the AI lane only produces `ok | skipped | timeout | error`. Narrow the type to that union so the telemetry contract matches the envelope schema and typos are caught at compile time.</comment>

<file context>
@@ -1594,10 +1594,20 @@ export namespace Telemetry {
+        /** Models the engine could not verify (no compiled SQL). */
+        unverified_models?: number
+        /** Advisory AI lane outcome: ok | skipped | timeout | error. */
+        ai_state?: string
+        ai_duration_ms?: number
+        /** Where the review base came from (flag, github_event, default_branch, …). */
</file context>
Suggested change
ai_state?: string
ai_state?: "ok" | "skipped" | "timeout" | "error"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing: telemetry fields are kept as strings for schema stability. The values come from the AiLaneState union at the source.


describe("PR title and description reach the intent check", () => {
test("GitHub: read from the pull_request event", async () => {
const dir = await fs.mkdtemp(path.join(tmpdir(), "review-event-"))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This test creates a temp directory and never removes it, so every run leaves a review-event-* directory behind. Remove it in a finally block (fs.rm(dir, { recursive: true, force: true })) or use the repo's tmpdir fixture with auto cleanup.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/test/altimate/review-ci-honesty.test.ts, line 157:

<comment>This test creates a temp directory and never removes it, so every run leaves a `review-event-*` directory behind. Remove it in a `finally` block (`fs.rm(dir, { recursive: true, force: true })`) or use the repo's tmpdir fixture with auto cleanup.</comment>

<file context>
@@ -0,0 +1,188 @@
+
+describe("PR title and description reach the intent check", () => {
+  test("GitHub: read from the pull_request event", async () => {
+    const dir = await fs.mkdtemp(path.join(tmpdir(), "review-event-"))
+    const ev = path.join(dir, "event.json")
+    await fs.writeFile(ev, JSON.stringify({ pull_request: { title: "Add fct_orders", body: "## Spec\n- grain: order_id" } }))
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing in this PR: test-hygiene nit, to be cleaned up with the fixture.

if (env.summary.degraded) {
lines.push(
"> ⚙️ **Lint-only run** — no dbt manifest/warehouse was available, so lineage, equivalence and",
"> ⚙️ **Lint-only run** — no changed model resolved against a dbt manifest, so lineage, equivalence and",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: The banner now asserts a cause that is only one of the two ways summary.degraded becomes true.

orchestrate.ts:1120 sets runDegraded = modelFiles.length > 0 ? !anyManifest : reviewable.length === 0, so an empty reviewable set also renders this banner. That happens for a PR whose only changed files are non-dbt (.md, .json, … are dropped by filterChangedFiles → shouldReview), and there the advice "Wire manifest_path" cannot change anything. The comment then misstates why checks were skipped, which is the class of problem this PR is fixing. Branch the wording on the actual cause, or give the empty-reviewable case its own note.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed later in #1721 (runDegraded = modelFiles.length > 0 && !anyManifest).

const head = unverified.slice(0, 5).map((u) => `\`${u.path}\``).join(", ")
const more = unverified.length > 5 ? ` and ${unverified.length - 5} more` : ""
notes.push(
`${unverified.length} model(s) not verified by the engine — ${unverified[0].reason}: ${head}${more}. ` +

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Only the first model's reason is reported for the whole set.

unverified entries carry two distinct reasons (orchestrate.ts:1108 head-side vs 1110 base-side), so a PR that mixes gaps gets every path labelled with unverified[0].reason. With unverified[0] a base-side entry, the line even tells the user to run dbt compile for models that were only missing target-base. Group by reason or use a neutral phrase ("no compiled SQL for the head or base").

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed later in #1723 (grouped by reason).

notes.push(`${env.scope.outOfScopeFiles} changed file(s) outside the dbt project were not reviewed.`)
if (env.ai && env.ai.state !== "ok") {
const what = { skipped: "skipped", timeout: "timed out", error: "failed" }[env.ai.state]
notes.push(`Advisory AI review ${what}${env.ai.reason ? `: ${env.ai.reason}` : ""}.`)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: env.ai.reason is interpolated into a > ℹ️ blockquote verbatim.

safeReason in ai-review.ts collapses whitespace, but the orchestrate catch path does not (see the comment there), so a multi-line error message breaks out of the blockquote and can inject arbitrary markdown into the posted PR comment. The footer at lines 114-115 has the same shape: env.scope.baseRef and env.ai.model are placed in single-backtick spans without the fence-length logic this file already applies to tierReasons (lines 64-74, added for exactly this reason). Reuse that escaping for these values.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

safeReason already collapses whitespace for lane-produced reasons. The remaining catch-path reason and backtick escaping are tracked as a low-priority follow-up.

if (lanes.has("ai_review") && input.aiReview) {
let aiStatus: AiLaneStatus | undefined
if (!lanes.has("ai_review")) {
aiStatus = { state: "skipped", reason: `not part of the ${tier} tier` }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: This reason can be false.

lanes is input.config.reviewers.length ? input.config.reviewers : TIER_LANES[tier] (line 1175), so the branch also fires when a user restricts lanes with reviewers: in .altimate/review.yml. A full-tier run with reviewers: ["sql_quality"] then reports "not part of the full tier" in the PR comment, which is a misstatement of the same kind the PR set out to remove. Distinguish "excluded by the reviewers config" from "not part of the tier".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. The reason should branch on config.reviewers.length when reviewers: excludes ai_review.

// A failed AI lane must never fail the review.
} catch (err) {
// A failed AI lane must never fail the review — but it must say so.
aiStatus = { state: "error", reason: String((err as Error)?.message ?? err).slice(0, 200) }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: This reason reaches the signed envelope and the PR comment unmasked.

Unlike safeReason in ai-review.ts, it is not whitespace-collapsed, not credential-masked, and only sliced to 200 chars — while AiLaneStatus.reason is documented as "never contains credentials". runAiReview does not throw today, so this is the fallback for any injected lane, but the contract should hold on every path: reuse the masking helper (or sanitize the message here) rather than embedding the raw error.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked (same as the cubic thread on this line).

- name: Compile the base into target-base/
run: |
git worktree add ../base "origin/${{ github.base_ref }}"
(cd ../base && dbt deps && dbt compile --target-path "$GITHUB_WORKSPACE/target-base")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: This snippet does not work in the monorepo layout the paragraph below documents.

It writes target-base/ to the repository root and runs dbt compile from the worktree root. The resolver reads <dbt root>/target-base/compiled/<project>/… (compiled.ts:117, with opts.cwd = dbtRoot from run.ts:363), and dbt compile needs dbt_project.yml, so with project_dir: dbt both steps fail or land where the review never looks — equivalence and column-breakage silently stay off. Add a monorepo variant, e.g. (cd ../base/dbt && dbt deps && dbt compile --target-path "$GITHUB_WORKSPACE/dbt/target-base"), and note the path must sit next to the project's manifest.json.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked (same as the cubic docs thread).


describe("PR title and description reach the intent check", () => {
test("GitHub: read from the pull_request event", async () => {
const dir = await fs.mkdtemp(path.join(tmpdir(), "review-event-"))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: This temp directory is never removed.

test/AGENTS.md ("Temporary Directory Fixture") asks tests to use tmpdir() / tmpdirScoped() from fixture/fixture.ts so cleanup is automatic; a raw fs.mkdtemp leaks one directory per run under the parallel suite. Use await using tmp = await tmpdir({ init: ... }) and write event.json inside tmp.path.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing in this PR: test-hygiene nit.

@kilo-code-bot

kilo-code-bot Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Code Review Summary

Status: 5 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 3

Fix these issues in Kilo Cloud

Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/test/altimate/review-ci-honesty.test.ts 205 New test embeds Basic YWxpY2U6czNjcmV0UGFzcw== (base64 of alice:s3cretPass); GitGuardian reports it as a hardcoded secret, and the incident stays open until the header is built at runtime.
packages/opencode/src/altimate/review/format.ts 45 Carried forward. The lint-only banner asserts "no changed model resolved against a dbt manifest", but summary.degraded is also true when reviewable.length === 0 (a docs-only PR), where the manifest_path advice cannot help.

SUGGESTION

File Line Issue
packages/opencode/src/cli/cmd/review.ts 69 New --ai-model flag is missing from the CLI options table in docs/docs/usage/dbt-pr-review.md.
packages/opencode/src/altimate/review/orchestrate.ts 1419 Carried forward. "not part of the ${tier} tier" also fires when the reviewers: config excluded ai_review, so the posted reason can be false.
packages/opencode/test/altimate/review-ci-honesty.test.ts 168 Carried forward. The fs.mkdtemp temp dir is never cleaned up; test/AGENTS.md asks for the tmpdir() fixture.
Files Reviewed (12 files)
  • docs/docs/reference/telemetry.md - no issues
  • docs/docs/usage/dbt-pr-review.md - no issues (base-compile merge-base and monorepo guidance now present)
  • github/review/action.yml - no issues
  • packages/opencode/src/altimate/review/ai-review.ts - no issues (in-band parse failure, explicit-model error, masked catch reason verified)
  • packages/opencode/src/altimate/review/format.ts - 1 issue
  • packages/opencode/src/altimate/review/orchestrate.ts - 1 issue
  • packages/opencode/src/altimate/review/run.ts - no issues
  • packages/opencode/src/altimate/review/telemetry.ts - no issues
  • packages/opencode/src/altimate/review/verdict.ts - no issues
  • packages/opencode/src/altimate/telemetry/index.ts - no issues
  • packages/opencode/src/cli/cmd/review.ts - 1 issue
  • packages/opencode/test/altimate/review-ci-honesty.test.ts - 2 issues

Verified on e8a893ec: a core parse failure (success: false or a non-array findings) now reports error instead of an empty ok; an unresolvable explicit --ai-model is error while a missing default stays skipped; both lane-reason paths (runAiReview and the orchestrate catch) run through safeReason → redactSecrets; ai is covered by the envelope signature; and the AI lane still makes at most one model call per review. The unverified-model note is now grouped per reason, matching the two reason strings in orchestrate.ts:1108-1110. Typecheck/tests were not re-run in this read-only review.

Previous Review Summary (commit 1fda92a)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 1fda92a)

Status: 7 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 6

Fix these issues in Kilo Cloud

Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/format.ts 45 Lint-only banner asserts "no changed model resolved against a dbt manifest", but summary.degraded is also true when reviewable.length === 0 (e.g. a docs-only PR). The "wire manifest_path" advice cannot help there, so the comment misstates why checks were skipped.

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/format.ts 136 Unverified-model note uses only unverified[0].reason for every path; head-side and base-side gaps are conflated.
packages/opencode/src/altimate/review/format.ts 144 env.ai.reason is interpolated into the PR comment verbatim; footer scope.baseRef/ai.model in single-backtick spans lack the fence-length escaping already used for tierReasons.
packages/opencode/src/altimate/review/orchestrate.ts 1419 "not part of the ${tier} tier" also fires when reviewers: config excluded ai_review, so the posted reason can be false.
packages/opencode/src/altimate/review/orchestrate.ts 1444 Catch-path reason is raw (no masking, no whitespace collapse) although AiLaneStatus.reason is documented credential-free and is posted publicly.
docs/docs/usage/dbt-pr-review.md 218 Base-compile snippet writes target-base/ to the repo root and compiles from the worktree root, so it does not work in the project_dir monorepo layout this same PR documents.
packages/opencode/test/altimate/review-ci-honesty.test.ts 157 fs.mkdtemp temp dir is never cleaned up; test/AGENTS.md asks for the tmpdir() fixture.
Files Reviewed (11 files)
  • docs/docs/usage/dbt-pr-review.md - 1 issue
  • github/review/action.yml - no issues
  • packages/opencode/src/altimate/review/ai-review.ts - no issues
  • packages/opencode/src/altimate/review/format.ts - 3 issues
  • packages/opencode/src/altimate/review/orchestrate.ts - 2 issues
  • packages/opencode/src/altimate/review/run.ts - no issues
  • packages/opencode/src/altimate/review/telemetry.ts - no issues
  • packages/opencode/src/altimate/review/verdict.ts - no issues
  • packages/opencode/src/altimate/telemetry/index.ts - no issues
  • packages/opencode/src/cli/cmd/review.ts - no issues
  • packages/opencode/test/altimate/review-ci-honesty.test.ts - 1 issue

Verified on 1fda92a8: prContextFromEnv reads the right CI variables, the new ai field is covered by the signature, and the AI lane still makes at most one model call per review (no cost side effect). --cwd resolution for project_dir matches how manifest_path is resolved. Typecheck/CI gates were not re-run in this read-only review.


Reviewed by deepseek-v4.1-flash-1017 · Input: 144.2K · Output: 42.2K · Cached: 4M

Review guidance: REVIEW.md from base branch fix/review-scope-diff

@dev-punia-altimate dev-punia-altimate left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Code Review — OpenCodeReview (Gemini) — 3 finding(s)

  • 3 anchored to a line (posted inline when the comment stream is on)
  • 0 without a line anchor
All findings (full text)

1. packages/opencode/src/altimate/review/run.ts (L382-L385)

[🔵 LOW] The checklist prohibits the use of nested ternary expressions. Please refactor this to avoid nesting, perhaps by computing the disabled reason in a separate variable before using it here.

Suggested change:

    cliVersion: opts.cliVersion ?? Installation.VERSION,
    aiReview: opts.noAi || config.ai === false ? undefined : runAiReview,
    aiDisabledReason: (function() {
      if (opts.noAi) return "disabled by --no-ai"
      if (config.ai === false) return "disabled by `ai: false` in .altimate/review.yml"
      return undefined
    })(),
    aiModel: opts.aiModel,

2. packages/opencode/src/altimate/review/ai-review.ts (L124-L129)

[🔴 HIGH] Moving buildUserMessage outside the try...catch block introduces the risk of unhandled exceptions. If buildUserMessage throws an error (e.g., due to an unexpected value or formatting error), it will crash the function, violating the JSDoc contract that "a review must not crash because the AI layer is unavailable".

Consider moving the message generation and timeout setup inside the try block to ensure all potential errors are caught and a proper failure status is returned.

Suggested change:

  const controller = new AbortController()
  let timeout: ReturnType<typeof setTimeout> | undefined
  let timeoutMs = 0
  let modelLabel: string | undefined
  try {
    const message = buildUserMessage({ ...input, files })
    timeoutMs = aiTimeoutMs(message.length)
    timeout = setTimeout(() => controller.abort(), timeoutMs)

3. packages/opencode/src/altimate/review/ai-review.ts (L61-L68)

[🟠 MEDIUM] While this effectively sanitizes bearer tokens and typical API key patterns, it may not cover other sensitive credentials occasionally found in network errors, such as Basic <base64> auth headers or credentials embedded within URLs (e.g., https://user:pass@host/...).

Consider expanding the regex to cover these patterns for more robust redaction.

Suggested change:

function safeReason(err: unknown): string {
  const raw = err instanceof Error ? err.message : String(err)
  return raw
    .replace(/\s+/g, " ")
    .replace(/(bearer\s+|basic\s+)[^\s]+/gi, "$1***")
    .replace(/:\/\/[^/]*@[^/]+/gi, "://***@")
    .replace(/\b(sk|pk|key|token)[-_][A-Za-z0-9_-]{8,}/gi, "***")
    .slice(0, 200)
}

Comment on lines 382 to +385
cliVersion: opts.cliVersion ?? Installation.VERSION,
aiReview: opts.noAi || config.ai === false ? undefined : runAiReview,
aiDisabledReason: opts.noAi ? "disabled by --no-ai" : config.ai === false ? "disabled by `ai: false` in .altimate/review.yml" : undefined,
aiModel: opts.aiModel,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🔵 LOW] The checklist prohibits the use of nested ternary expressions. Please refactor this to avoid nesting, perhaps by computing the disabled reason in a separate variable before using it here.

Suggested change:

Suggested change
cliVersion: opts.cliVersion ?? Installation.VERSION,
aiReview: opts.noAi || config.ai === false ? undefined : runAiReview,
aiDisabledReason: opts.noAi ? "disabled by --no-ai" : config.ai === false ? "disabled by `ai: false` in .altimate/review.yml" : undefined,
aiModel: opts.aiModel,
cliVersion: opts.cliVersion ?? Installation.VERSION,
aiReview: opts.noAi || config.ai === false ? undefined : runAiReview,
aiDisabledReason: (function() {
if (opts.noAi) return "disabled by --no-ai"
if (config.ai === false) return "disabled by `ai: false` in .altimate/review.yml"
return undefined
})(),
aiModel: opts.aiModel,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, agreed. Fixed in 4d9a88a: the AI-disabled reason is now computed in a small if/else if block before the runReview call.

Comment on lines 124 to 129
const message = buildUserMessage({ ...input, files })
const controller = new AbortController()
const timeout = setTimeout(() => controller.abort(), AI_TIMEOUT_MS)
const timeoutMs = aiTimeoutMs(message.length)
const timeout = setTimeout(() => controller.abort(), timeoutMs)
let modelLabel: string | undefined
try {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🔴 HIGH] Moving buildUserMessage outside the try...catch block introduces the risk of unhandled exceptions. If buildUserMessage throws an error (e.g., due to an unexpected value or formatting error), it will crash the function, violating the JSDoc contract that "a review must not crash because the AI layer is unavailable".

Consider moving the message generation and timeout setup inside the try block to ensure all potential errors are caught and a proper failure status is returned.

Suggested change:

Suggested change
const message = buildUserMessage({ ...input, files })
const controller = new AbortController()
const timeout = setTimeout(() => controller.abort(), AI_TIMEOUT_MS)
const timeoutMs = aiTimeoutMs(message.length)
const timeout = setTimeout(() => controller.abort(), timeoutMs)
let modelLabel: string | undefined
try {
const controller = new AbortController()
let timeout: ReturnType<typeof setTimeout> | undefined
let timeoutMs = 0
let modelLabel: string | undefined
try {
const message = buildUserMessage({ ...input, files })
timeoutMs = aiTimeoutMs(message.length)
timeout = setTimeout(() => controller.abort(), timeoutMs)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, fixed in 4d9a88a. Message building and the timeout setup now sit inside the try, so runAiReview keeps its never-throws contract. Before this, a throw wouldn't have crashed the review: the caller in orchestrate.ts catches it and records the lane as state: error. So the impact was a misreported lane status, not a crash, but the contract should hold locally.

Comment on lines 61 to 68
function safeReason(err: unknown): string {
const raw = err instanceof Error ? err.message : String(err)
return raw
.replace(/\s+/g, " ")
.replace(/(bearer\s+)[^\s]+/gi, "$1***")
.replace(/\b(sk|pk|key|token)[-_][A-Za-z0-9_-]{8,}/gi, "***")
.slice(0, 200)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🟠 MEDIUM] While this effectively sanitizes bearer tokens and typical API key patterns, it may not cover other sensitive credentials occasionally found in network errors, such as Basic <base64> auth headers or credentials embedded within URLs (e.g., https://user:pass@host/...).

Consider expanding the regex to cover these patterns for more robust redaction.

Suggested change:

Suggested change
function safeReason(err: unknown): string {
const raw = err instanceof Error ? err.message : String(err)
return raw
.replace(/\s+/g, " ")
.replace(/(bearer\s+)[^\s]+/gi, "$1***")
.replace(/\b(sk|pk|key|token)[-_][A-Za-z0-9_-]{8,}/gi, "***")
.slice(0, 200)
}
function safeReason(err: unknown): string {
const raw = err instanceof Error ? err.message : String(err)
return raw
.replace(/\s+/g, " ")
.replace(/(bearer\s+|basic\s+)[^\s]+/gi, "$1***")
.replace(/:\/\/[^/]*@[^/]+/gi, "://***@")
.replace(/\b(sk|pk|key|token)[-_][A-Za-z0-9_-]{8,}/gi, "***")
.slice(0, 200)
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this was a real gap. Fixed in 4d9a88a: safeReason now reuses the repo's redactSecrets (from installation), which masks Authorization: headers, Basic/Bearer values, user:pass@/token userinfo in URLs, api_key=...-style assignments and provider-prefixed tokens. Redaction runs before whitespace is collapsed, so a header match stops at its own line. New tests check that Basic, URL, key=value and bearer credentials are masked and that a plain error message is left alone.

@dev-punia-altimate

Copy link
Copy Markdown
Contributor

🤖 Code Review — OpenCodeReview (Gemini) — 3 finding(s)

  • 3 anchored to a line (posted inline when the comment stream is on)
  • 0 without a line anchor
All findings (full text)

1. packages/opencode/src/altimate/review/run.ts (L382-L385)

[🔵 LOW] The checklist prohibits the use of nested ternary expressions. Please refactor this to avoid nesting, perhaps by computing the disabled reason in a separate variable before using it here.

Suggested change:

    cliVersion: opts.cliVersion ?? Installation.VERSION,
    aiReview: opts.noAi || config.ai === false ? undefined : runAiReview,
    aiDisabledReason: (function() {
      if (opts.noAi) return "disabled by --no-ai"
      if (config.ai === false) return "disabled by `ai: false` in .altimate/review.yml"
      return undefined
    })(),
    aiModel: opts.aiModel,

2. packages/opencode/src/altimate/review/ai-review.ts (L124-L129)

[🔴 HIGH] Moving buildUserMessage outside the try...catch block introduces the risk of unhandled exceptions. If buildUserMessage throws an error (e.g., due to an unexpected value or formatting error), it will crash the function, violating the JSDoc contract that "a review must not crash because the AI layer is unavailable".

Consider moving the message generation and timeout setup inside the try block to ensure all potential errors are caught and a proper failure status is returned.

Suggested change:

  const controller = new AbortController()
  let timeout: ReturnType<typeof setTimeout> | undefined
  let timeoutMs = 0
  let modelLabel: string | undefined
  try {
    const message = buildUserMessage({ ...input, files })
    timeoutMs = aiTimeoutMs(message.length)
    timeout = setTimeout(() => controller.abort(), timeoutMs)

3. packages/opencode/src/altimate/review/ai-review.ts (L61-L68)

[🟠 MEDIUM] While this effectively sanitizes bearer tokens and typical API key patterns, it may not cover other sensitive credentials occasionally found in network errors, such as Basic <base64> auth headers or credentials embedded within URLs (e.g., https://user:pass@host/...).

Consider expanding the regex to cover these patterns for more robust redaction.

Suggested change:

function safeReason(err: unknown): string {
  const raw = err instanceof Error ? err.message : String(err)
  return raw
    .replace(/\s+/g, " ")
    .replace(/(bearer\s+|basic\s+)[^\s]+/gi, "$1***")
    .replace(/:\/\/[^/]*@[^/]+/gi, "://***@")
    .replace(/\b(sk|pk|key|token)[-_][A-Za-z0-9_-]{8,}/gi, "***")
    .slice(0, 200)
}

@hkaltai hkaltai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: approve, with minor follow-ups. Panel: Claude, Codex, lead. No CRITICAL or MAJOR findings. 0 CRITICAL, 0 MAJOR, 6 MINOR (inline), 6 NIT. Claude's run of the six review test files (172) and typecheck passed.

Findings above were each verified against the code. The verdict logic is unchanged by the banner fix (degraded does not feed the verdict), so this cannot flip approve/block.

[MINOR] (ai-review.ts:185, outside the diff hunks) Parser failure reported as state ok

const parsed = (((parseRes.data ?? {}) as Record<string, unknown>).findings as any[]) ?? []

The core parser can return { success: false, data: {}, error } instead of throwing. success is ignored and findings default to [], so a failed parse is reported as ok with no findings, which is exactly the silent failure this PR aims to remove.

Fix: check parseRes.success and that findings is an array; otherwise return state: "error" with a masked reason.

Also worth doing: docs/docs/reference/telemetry.md review_run row does not list the new fields (undecided, unverified_models, ai_state, ai_duration_ms, base_source, out_of_scope_files) and the changed meaning of degraded. Tests are missing for runAiReview itself (timeout with partial text, empty text, missing prompt, bad model), safeReason, and the project_dir to --cwd wiring.

Nits

  • orchestrate.ts:1418-1419 / format.ts AI note: every trivial-tier review now shows "Advisory AI review skipped: not part of the trivial tier"; consider keeping that in the envelope only.
  • verdict.ts:153 durationMs: z.number().int().nonnegative(): an injected lane returning a fractional duration makes VerdictEnvelope.parse throw; round it before building the envelope.
  • format.ts:114 (env.scope?.baseRef ? ` · base \`${env.scope.baseRef}\ ...) ``: branch names may contain backticks; escape them.
  • orchestrate.ts:1411-1416 comment still says the lane is "Skipped when no aiReview fn is injected ... deterministic-only"; update to mention aiStatus reporting.
  • ai-review.ts aiTimeoutMs doc says "capped at 5 minutes", but the prompt limits make the cap effectively unreachable (about 183 s); say so or lower the cap.
  • ai-review.ts:60 safeReason: name suggests validation; redactErrorForComment would be clearer.

return raw
.replace(/\s+/g, " ")
.replace(/(bearer\s+)[^\s]+/gi, "$1***")
.replace(/\b(sk|pk|key|token)[-_][A-Za-z0-9_-]{8,}/gi, "***")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Secret masking in safeReason is incomplete

.replace(/\b(sk|pk|key|token)[-_][A-Za-z0-9_-]{8,}/gi, "***")

Only bearer ... and sk|pk|key|token followed by -/_ are masked. ?key=AIza..., api_key=..., x-api-key: ..., ghp_..., AKIA... all pass through (confirmed by running the regex). The reason lands in a public PR comment and the signed envelope. Scenario: a provider error echoes a request URL containing ?key=... and it is posted on the PR.

Fix: add patterns for key=/token= query and header values, AIza..., gh[pousr]_..., AKIA..., JWTs, or mask any 24+ char key-like run; add unit tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Fixed in fix(review): address review comments: credential redaction in AI-lane reasons, never-throw AI lane. safeReason now runs the shared redactSecrets (key/token query and header values, provider-prefixed tokens, Basic/Bearer, URL userinfo) before collapsing whitespace, with tests.

const chosen = input.model ? Provider.parseModel(input.model) : await Provider.defaultModel()
model = await Provider.getModel(chosen.providerID, chosen.modelID)
} catch (err) {
return done([], { state: "skipped", reason: `no usable model: ${safeReason(err)}`, model: input.model })

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Explicit --ai-model that fails to resolve reports skipped

return done([], { state: "skipped", reason: `no usable model: ${safeReason(err)}`, model: input.model })

A typo or missing credentials for an explicitly requested model looks the same as --no-ai in the comment and in ai_state telemetry. Fix: use error when input.model was supplied; keep skipped for no resolvable default.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, thanks. Fixed in fix(review): address review feedback: AI parse failure, explicit model error, masked lane reason, per-reason unverified notes, docs: an explicitly requested model that can't be resolved is error, and only a missing default is skipped.

// A failed AI lane must never fail the review.
} catch (err) {
// A failed AI lane must never fail the review — but it must say so.
aiStatus = { state: "error", reason: String((err as Error)?.message ?? err).slice(0, 200) }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Thrown-lane reason is unmasked and multi-line

aiStatus = { state: "error", reason: String((err as Error)?.message ?? err).slice(0, 200) }

Bypasses safeReason, so credentials in the message are published and newlines can break the > quote. Only reachable for injected lanes, but it is the defence-in-depth path. Fix: export and reuse safeReason.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Fixed in the same commit: the catch in runReview goes through safeReason, so the reason is masked, on one line and bounded. There's a test with a multi-line bearer header.

const head = unverified.slice(0, 5).map((u) => `\`${u.path}\``).join(", ")
const more = unverified.length > 5 ? ` and ${unverified.length - 5} more` : ""
notes.push(
`${unverified.length} model(s) not verified by the engine — ${unverified[0].reason}: ${head}${more}. ` +

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Unverified note shows the first model's reason for all models

`${unverified.length} model(s) not verified by the engine — ${unverified[0].reason}: ${head}${more}. `

Reasons differ (no compiled SQL for head vs base, orchestrate.ts:1108-1110). If a.sql lacks the base compile and b.sql the head compile, the comment says both lack the base compile. Fix: group by reason, one line each.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Fixed in the same commit: unverified models are grouped by reason, one line each, with a test. #1723 extends the same grouping to failed files and abstentions.

Comment thread docs/docs/usage/dbt-pr-review.md Outdated
```yaml
- name: Compile the base into target-base/
run: |
git worktree add ../base "origin/${{ github.base_ref }}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Base-compile recipe: wrong baseline and monorepo path

git worktree add ../base "origin/${{ github.base_ref }}"

(1) This checks out the target-branch tip, but the review diffs against the merge-base with the PR head. If the target advanced, equivalence is compared against a different baseline (a formatting-only PR can get a false critical). Use $(git merge-base origin/<base> HEAD). (2) target-base/ is resolved under the dbt project root, so with project_dir the snippet compiles from and writes to the wrong folder; compile from ../base/<project_dir> with --target-path "$GITHUB_WORKSPACE/<project_dir>/target-base".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both points are right, thanks. Fixed in the same commit: the recipe checks out $(git merge-base origin/<base> HEAD) and shows the project_dir form (cd ../base/<project_dir>, --target-path "$GITHUB_WORKSPACE/<project_dir>/target-base"). dbt_prepare in #1726 does the same.

RaghwendraSingh and others added 3 commits November 20, 2026 12:10
…tus, PR intent, `project_dir`

- Lint-only now means no changed model resolved against a manifest; an
  undecidable finding no longer flips the whole run. Undecidable findings,
  unverified models, out-of-scope files and AI-lane problems each get their
  own line in the summary.
- The advisory AI lane reports `ok | skipped | timeout | error` with a
  reason, model and duration on the signed envelope (`ai`) and in the
  comment; it never fails silently. The timeout scales with prompt size
  (60 s + 1 s per 2K chars, capped at 5 min). `--ai-model provider/model`
  selects the model explicitly.
- The CLI passes the PR title and description to the intent check (GitHub
  `pull_request` event, GitLab `CI_MERGE_REQUEST_TITLE`/`_DESCRIPTION`).
- The action gains `project_dir` (→ `--cwd`) for monorepos.
- Docs: what each input unlocks (fetch depth, manifest, compiled SQL,
  `target-base/`, `catalog.json`) and how to compile the base.
- `review_run` telemetry adds `undecided`, `unverified_models`, `ai_state`,
  `ai_duration_ms`, `base_source`, `out_of_scope_files`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… reasons, never-throw AI lane

- `safeReason` now runs the shared `redactSecrets` masker before collapsing
  whitespace, so `Basic` auth values, `Authorization:` headers, `user:pass@`
  URL userinfo and `api_key=...` secrets no longer reach the public PR
  comment. Exported and covered by tests.
- `runAiReview` builds the user message and arms the timeout inside the `try`,
  keeping its never-throws contract.
- `run.ts`: the AI-disabled reason is computed before the call instead of in a
  nested ternary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l error, masked lane reason, per-reason unverified notes, docs

- `runAiReview`: core's in-band parse failure (`success: false`, or no
  findings array) is reported as `state: "error"`, never an "ok" run with no
  findings.
- `runAiReview`: an explicitly requested `--ai-model` that cannot be resolved
  is an `error`; only a missing default model is `skipped`.
- `runReview`: a thrown AI lane's reason goes through `safeReason` (masked,
  one line, bounded).
- `coverageNotes`: unverified models are grouped by reason, one line each, so
  a missing head compile and a missing base compile are not reported as the
  same thing.
- Docs: the base-compile recipe checks out the merge-base (what the review
  diffs against) and shows the `project_dir` form; the telemetry reference
  lists the new `review_run` fields and the narrower meaning of `degraded`.
- Tests: masked multi-line lane reason, per-reason unverified notes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gitguardian

gitguardian Bot commented Nov 20, 2026

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
40673273 Triggered Base64 Basic Authentication 4d9a88a packages/opencode/test/altimate/review-ci-honesty.test.ts View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 existing issue remains and 4 new issues found across 7 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="docs/docs/usage/dbt-pr-review.md">

<violation number="1" location="docs/docs/usage/dbt-pr-review.md:218">
P3: This `run:` step interpolates `${{ github.base_ref }}` directly into the shell script, which is the injection pattern `github/review/action.yml` avoids. Users copy this recipe, so pass the value through `env:` and reference `$BASE_REF` instead.</violation>

<violation number="2" location="docs/docs/usage/dbt-pr-review.md:218">
P2: This command computes the base from `HEAD`, but the preceding default `actions/checkout` on `pull_request` checks out GitHub’s synthetic merge commit; the merge-base therefore becomes the target branch tip, not the PR’s branch point. Check out `github.event.pull_request.head.sha` (and make the review use that same head), or pass that SHA explicitly here before compiling the base.</violation>
</file>

<file name="packages/opencode/src/altimate/review/ai-review.ts">

<violation number="1" location="packages/opencode/src/altimate/review/ai-review.ts:149">
P2: Only a missing default should be `skipped`. Any `Provider.defaultModel()` failure, including a config or provider-load error, is reported as `skipped`. Consider separating the default-model path so real failures report `error`.</violation>
</file>

<file name="docs/docs/reference/telemetry.md">

<violation number="1" location="docs/docs/reference/telemetry.md:76">
P2: This row documents a `degraded` condition that the review currently does not report: any parseable manifest makes `degraded` false, even when the changed model is absent from it. Track whether each changed model actually resolves against the manifest, and update the stale note below that still says undecidable findings set `degraded`.</violation>
</file>

Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.

View guided diff | Turn on auto-fix | Re-trigger cubic

// An explicitly requested model that cannot be resolved is an error, not
// the same as `--no-ai`; only a missing default is a skip.
return done([], {
state: input.model ? "error" : "skipped",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Only a missing default should be skipped. Any Provider.defaultModel() failure, including a config or provider-load error, is reported as skipped. Consider separating the default-model path so real failures report error.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-review.ts, line 149:

<comment>Only a missing default should be `skipped`. Any `Provider.defaultModel()` failure, including a config or provider-load error, is reported as `skipped`. Consider separating the default-model path so real failures report `error`.</comment>

<file context>
@@ -137,7 +143,13 @@ export async function runAiReview(input: AiReviewInput): Promise<AiReviewResult>
+      // An explicitly requested model that cannot be resolved is an error, not
+      // the same as `--no-ai`; only a missing default is a skip.
+      return done([], {
+        state: input.model ? "error" : "skipped",
+        reason: `no usable model: ${safeReason(err)}`,
+        model: input.model,
</file context>

| `first_prompt_sent` | The user's first typed message in an onboarding session, and the first typed message in a linked project or a session pinned to a workspace, once per workspace for each machine and Altimate instance. A slash command sent as a command, as the TUI and `run` send it, is excluded, so the hidden `/onboard-connect` submission does not count. The VS Code chat sends a command as its expanded text, so there a command counts. A subagent's prompt never counts. `origin` says which rule sent it: `onboarding` for an onboarding session's first typed prompt, bound or not, and `workspace` for the workspace's row. So a later prompt in an onboarding session, after its project was linked, is a `workspace` row. |
| `onboarding_abandoned` | The CLI exited during a first run without connecting. `last_stage` is the furthest point reached: `started`, `model_picker`, `provider_setup`, `altimate_base_confirm`, or `gateway_auth`. (`connected` is a funnel position but never a `last_stage` — reaching it means the run completed, which is not an abandonment.) Only emitted for a genuine first run — opening `/connect` as an existing user does not enter the funnel, and abandonment after setup completes is out of scope by definition. Emitted on the exit path under a bounded flush, so the measured rate is a lower bound — see [Delivery & Reliability](#delivery--reliability). |
| `review_run` | A dbt/SQL review completed or failed — `invocation` (`cli` for `altimate-code review`, `tool` for the `dbt_pr_review` tool), status, duration, and on success the verdict, the pre-gating verdict, mode, risk tier, and finding counts by severity and by category. No file paths, model or column names, finding titles or bodies, SQL, diff content, or repository/branch/PR names. |
| `review_run` | A dbt/SQL review completed or failed — `invocation` (`cli` for `altimate-code review`, `tool` for the `dbt_pr_review` tool), status, duration, and on success the verdict, the pre-gating verdict, mode, risk tier, finding counts by severity and by category, `degraded` (lint-only: no reviewable files, or no changed model resolved against a manifest; an undecidable finding alone no longer sets it), `undecided` (findings the engine could not decide), `unverified_models` (models without compiled SQL), `ai_state` and `ai_duration_ms` (the advisory AI lane's outcome), `base_source` (where the review base came from, e.g. `flag` or `default_branch`) and `out_of_scope_files` (changed files outside the dbt project). No file paths, model or column names, finding titles or bodies, SQL, diff content, or repository/branch/PR names. |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This row documents a degraded condition that the review currently does not report: any parseable manifest makes degraded false, even when the changed model is absent from it. Track whether each changed model actually resolves against the manifest, and update the stale note below that still says undecidable findings set degraded.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/docs/reference/telemetry.md, line 76:

<comment>This row documents a `degraded` condition that the review currently does not report: any parseable manifest makes `degraded` false, even when the changed model is absent from it. Track whether each changed model actually resolves against the manifest, and update the stale note below that still says undecidable findings set `degraded`.</comment>

<file context>
@@ -73,7 +73,7 @@ We collect the following categories of events:
 | `first_prompt_sent` | The user's first typed message in an onboarding session, and the first typed message in a linked project or a session pinned to a workspace, once per workspace for each machine and Altimate instance. A slash command sent as a command, as the TUI and `run` send it, is excluded, so the hidden `/onboard-connect` submission does not count. The VS Code chat sends a command as its expanded text, so there a command counts. A subagent's prompt never counts. `origin` says which rule sent it: `onboarding` for an onboarding session's first typed prompt, bound or not, and `workspace` for the workspace's row. So a later prompt in an onboarding session, after its project was linked, is a `workspace` row. |
 | `onboarding_abandoned` | The CLI exited during a first run without connecting. `last_stage` is the furthest point reached: `started`, `model_picker`, `provider_setup`, `altimate_base_confirm`, or `gateway_auth`. (`connected` is a funnel position but never a `last_stage` — reaching it means the run completed, which is not an abandonment.) Only emitted for a genuine first run — opening `/connect` as an existing user does not enter the funnel, and abandonment after setup completes is out of scope by definition. Emitted on the exit path under a bounded flush, so the measured rate is a lower bound — see [Delivery & Reliability](#delivery--reliability). |
-| `review_run` | A dbt/SQL review completed or failed — `invocation` (`cli` for `altimate-code review`, `tool` for the `dbt_pr_review` tool), status, duration, and on success the verdict, the pre-gating verdict, mode, risk tier, and finding counts by severity and by category. No file paths, model or column names, finding titles or bodies, SQL, diff content, or repository/branch/PR names. |
+| `review_run` | A dbt/SQL review completed or failed — `invocation` (`cli` for `altimate-code review`, `tool` for the `dbt_pr_review` tool), status, duration, and on success the verdict, the pre-gating verdict, mode, risk tier, finding counts by severity and by category, `degraded` (lint-only: no reviewable files, or no changed model resolved against a manifest; an undecidable finding alone no longer sets it), `undecided` (findings the engine could not decide), `unverified_models` (models without compiled SQL), `ai_state` and `ai_duration_ms` (the advisory AI lane's outcome), `base_source` (where the review base came from, e.g. `flag` or `default_branch`) and `out_of_scope_files` (changed files outside the dbt project). No file paths, model or column names, finding titles or bodies, SQL, diff content, or repository/branch/PR names. |
 | `review_post_outcome` | Whether a review was published to GitHub — `not_requested`, `not_attempted`, `target_unresolved`, `full`, `partial`, or `summary_failed`, plus duration. Emitted on the **CLI path only** — the `dbt_pr_review` tool completes reviews but never publishes, so a `review_run` with `invocation: tool` has no post event and that is not a failure. Within the CLI path there is exactly one per **completed** review: a review that failed emits `review_run: failed` and no post event, so absence there means the review failed rather than that an event was lost. `not_attempted` is publication requested but never reached (a bad `--output` path, a stdout write error). No repository, PR, or comment content. |
 | `rule_proposal_shown` | A proposed rule was raised: shown as a card, or stored for `learn proposals` when no attached client shows cards (`stored`), or raised again for the same wording (`repeat`) — what raised it (`remember`, `shown_rule`, `recurrence`), the kind of change, whether it is a "Save as" choice, linked/owner, the buttons offered, whether it weakens verification, evidence session count, whether it came from a correction, how many paths it applies to, and its length. The card id is random. No rule text, quotes, paths or globs. |
</file context>

- name: Compile the base into target-base/
run: |
# The review diffs against the merge-base, not the target branch tip.
git worktree add ../base "$(git merge-base "origin/${{ github.base_ref }}" HEAD)"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This command computes the base from HEAD, but the preceding default actions/checkout on pull_request checks out GitHub’s synthetic merge commit; the merge-base therefore becomes the target branch tip, not the PR’s branch point. Check out github.event.pull_request.head.sha (and make the review use that same head), or pass that SHA explicitly here before compiling the base.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/docs/usage/dbt-pr-review.md, line 218:

<comment>This command computes the base from `HEAD`, but the preceding default `actions/checkout` on `pull_request` checks out GitHub’s synthetic merge commit; the merge-base therefore becomes the target branch tip, not the PR’s branch point. Check out `github.event.pull_request.head.sha` (and make the review use that same head), or pass that SHA explicitly here before compiling the base.</comment>

<file context>
@@ -214,10 +214,14 @@ Compiling the base (optional, for equivalence and column-breakage):
         run: |
-          git worktree add ../base "origin/${{ github.base_ref }}"
+          # The review diffs against the merge-base, not the target branch tip.
+          git worktree add ../base "$(git merge-base "origin/${{ github.base_ref }}" HEAD)"
           (cd ../base && dbt deps && dbt compile --target-path "$GITHUB_WORKSPACE/target-base")

</file context>


</details>

- name: Compile the base into target-base/
run: |
# The review diffs against the merge-base, not the target branch tip.
git worktree add ../base "$(git merge-base "origin/${{ github.base_ref }}" HEAD)"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This run: step interpolates ${{ github.base_ref }} directly into the shell script, which is the injection pattern github/review/action.yml avoids. Users copy this recipe, so pass the value through env: and reference $BASE_REF instead.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/docs/usage/dbt-pr-review.md, line 218:

<comment>This `run:` step interpolates `${{ github.base_ref }}` directly into the shell script, which is the injection pattern `github/review/action.yml` avoids. Users copy this recipe, so pass the value through `env:` and reference `$BASE_REF` instead.</comment>

<file context>
@@ -214,10 +214,14 @@ Compiling the base (optional, for equivalence and column-breakage):
         run: |
-          git worktree add ../base "origin/${{ github.base_ref }}"
+          # The review diffs against the merge-base, not the target branch tip.
+          git worktree add ../base "$(git merge-base "origin/${{ github.base_ref }}" HEAD)"
           (cd ../base && dbt deps && dbt compile --target-path "$GITHUB_WORKSPACE/target-base")

</file context>


</details>

test("Basic auth, URL userinfo, key=value and bearer values are masked", () => {
const err = new Error(
"request to https://alice:s3cretPass@proxy.example.com/v1 failed\n" +
"Authorization: Basic YWxpY2U6czNjcmV0UGFzcw==\n" +

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: This literal trips the repository's secret scanner.

Basic YWxpY2U6czNjcmV0UGFzcw== is only base64 of alice:s3cretPass (repeated at line 210), but GitGuardian cannot know it is fake and has opened a Base64 Basic Authentication incident on this PR (commit 4d9a88a), so that status keeps reporting a secret on a public repo. Build the header at runtime instead, which keeps the same coverage without a scanner-shaped literal:

const basic = Buffer.from("alice:s3cretPass").toString("base64")
const err = new Error(
  "request to https://alice:s3cretPass@proxy.example.com/v1 failed\n" +
    `Authorization: Basic ${basic}\n` +
    "api_key=abcd1234efgh and Bearer eyJhbGciOiJIUzI1NiJ9.payload",
)

and keep the expect(out).not.toContain(basic) assertion.

choices: ["trivial", "lite", "full"] as const,
describe: "[EXPERIMENTAL / bench debug] override the tier classifier — verdict envelope carries tierForced: true",
})
.option("ai-model", {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: The new flag is missing from the CLI reference.

docs/docs/usage/dbt-pr-review.md documents every other flag in its Options table (lines 109-120, including --no-ai), but --ai-model is absent, so the only documented way to choose the advisory model is the PR description. Add a row such as | --ai-model <provider/model> | Explicit model for the advisory AI lane (default: the configured default model). |.

@RaghwendraSingh

Copy link
Copy Markdown
Contributor Author

@hkaltai, re your review:

Thanks for the approval. One commit on top addresses the follow-ups:

  • ai-review.ts:185: core's in-band success: false, or a missing findings array, now reports state: "error" instead of "ok, no findings".
  • All five inline MINORs are fixed (masking was already in; explicit-model error, masked thrown-lane reason, per-reason notes and the docs recipe are new).
  • docs/docs/reference/telemetry.md now lists undecided, unverified_models, ai_state, ai_duration_ms, base_source and out_of_scope_files, and the narrower meaning of degraded.
  • Tests added for the masked multi-line lane reason and per-reason notes. Unit tests for runAiReview itself (timeout, empty text, parse failure) need a Provider/LLM harness and are a tracked follow-up.
  • Nits I'm leaving for later: the trivial-tier AI note, durationMs rounding, backtick escaping in the base ref, the aiTimeoutMs doc, and the safeReason name.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants