Repository navigation
fix(review): an honest PR comment: real lint-only banner, AI lane status, PR intent, project_dir #1720
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: fix/review-scope-diff
Are you sure you want to change the base?
fix(review): an honest PR comment: real lint-only banner, AI lane status, PR intent, project_dir #1720
Changes from all commits
c4b72b4
4d9a88a
e8a893e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -199,6 +199,38 @@ jobs: | |
| # model_api_key: ${{ secrets.ANTHROPIC_API_KEY }} | ||
| ``` | ||
|
|
||
| ### What the review needs (and what each input unlocks) | ||
|
|
||
| | Input | How to produce it | Unlocks | | ||
| |---|---|---| | ||
| | Full git history of the PR's target branch | `actions/checkout` with `fetch-depth: 0` | Correct base: the diff starts at the merge-base with the PR's target branch. A target branch that is not fetched is an error, never a silent fallback to `main`. | | ||
| | `target/manifest.json` | `dbt compile` (or `dbt parse` for lint + lineage only) | Blast radius, lineage breakage, tiering. Without it the run is lint-only. | | ||
| | `target/compiled/…` | `dbt compile` | Engine checks on the head (lint, grade, PII, grain). | | ||
| | `target-base/compiled/…` | Compile the base ref into `target-base/` (below) | Equivalence proofs and column-breakage for modified models. | | ||
| | `target/catalog.json` | `dbt docs generate` | Complete warehouse columns for column-lineage breakage. | | ||
|
|
||
| Compiling the base (optional, for equivalence and column-breakage): | ||
|
|
||
| ```yaml | ||
| - name: Compile the base into target-base/ | ||
| run: | | ||
| # The review diffs against the merge-base, not the target branch tip. | ||
| git worktree add ../base "$(git merge-base "origin/${{ github.base_ref }}" HEAD)" | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: This command computes the base from Prompt for AI agents</file context> There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P3: This Prompt for AI agents</file context> |
||
| (cd ../base && dbt deps && dbt compile --target-path "$GITHUB_WORKSPACE/target-base") | ||
|
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. SUGGESTION: This snippet does not work in the monorepo layout the paragraph below documents. It writes
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Follow-up: low priority, tracked (same as the cubic docs thread). |
||
| ``` | ||
|
|
||
| With `project_dir`, compile from that folder and write next to its own | ||
| `target/`: `(cd ../base/<project_dir> && dbt deps && dbt compile --target-path "$GITHUB_WORKSPACE/<project_dir>/target-base")`. | ||
|
|
||
| **Monorepos:** set `project_dir` to the folder holding `dbt_project.yml`. Only | ||
| changes inside it are reviewed, `manifest_path` is resolved relative to it, and | ||
| changes elsewhere are counted in the comment. | ||
|
|
||
| **What the comment tells you it could not check:** findings the engine could not | ||
| decide, models without compiled SQL, files outside the project, and whether the | ||
| advisory AI lane ran, was skipped, timed out or failed (with the reason). The | ||
| same facts are in the envelope (`scope`, `ai`) and the `review_run` telemetry. | ||
|
|
||
| ### Model & credentials for the advisory lane | ||
|
|
||
| The deterministic engine (lineage, equivalence, PII, grade, lint — the only layer | ||
|
|
@@ -356,8 +388,9 @@ reviewer does **not** re-implement Jinja — it consumes dbt's own compiled outp | |
| from `target/compiled/<project>/…` (HEAD) and `target-base/compiled/…` (BASE), | ||
| produced by `dbt compile`. To enable full equivalence verdicts, compile both | ||
| the base and head refs in CI (the base into `target-base/`, the Recce | ||
| convention). Without compiled SQL the engine lanes fall back to raw and stay | ||
| *undecidable* (never fabricated) — the `dbt-patterns` lane still runs. | ||
| convention). A model without compiled SQL is never sent to the engine as raw | ||
| Jinja: it is listed as **not verified** (with the reason) in the comment and in | ||
| the envelope's `scope.unverified`, and the `dbt-patterns` lane still runs. | ||
|
|
||
| ## What it checks — deterministic rule catalog | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -9,8 +9,12 @@ inputs: | |||||||||||||||||||
| description: "comment (never blocks) | gate (blocks the merge on REQUEST_CHANGES)" | ||||||||||||||||||||
| required: false | ||||||||||||||||||||
| default: "comment" | ||||||||||||||||||||
| project_dir: | ||||||||||||||||||||
| description: "Directory of the dbt project (the one holding dbt_project.yml), relative to the repository root. Only changes inside it are reviewed — set it in a monorepo. `manifest_path` is resolved relative to it." | ||||||||||||||||||||
| required: false | ||||||||||||||||||||
| default: "." | ||||||||||||||||||||
| manifest_path: | ||||||||||||||||||||
| description: "Path to the compiled dbt manifest.json. Run `dbt compile` before this step for the full (non-degraded) verdict." | ||||||||||||||||||||
| description: "Path to the compiled dbt manifest.json, relative to `project_dir`. Run `dbt compile` before this step for the full (non-degraded) verdict; `dbt docs generate` (catalog.json) and a base compile into `target-base/` enable column-lineage breakage and equivalence proofs." | ||||||||||||||||||||
| required: false | ||||||||||||||||||||
| default: "target/manifest.json" | ||||||||||||||||||||
| severity_threshold: | ||||||||||||||||||||
|
|
@@ -170,6 +174,7 @@ runs: | |||||||||||||||||||
| # from `${{ }}` would be a shell-injection vector. Build an arg array instead. | ||||||||||||||||||||
| env: | ||||||||||||||||||||
| IN_MODE: ${{ inputs.mode }} | ||||||||||||||||||||
| IN_PROJECT_DIR: ${{ inputs.project_dir }} | ||||||||||||||||||||
| IN_MANIFEST: ${{ inputs.manifest_path }} | ||||||||||||||||||||
| IN_SEVERITY: ${{ inputs.severity_threshold }} | ||||||||||||||||||||
| IN_BASE: ${{ inputs.base }} | ||||||||||||||||||||
|
|
@@ -181,6 +186,7 @@ runs: | |||||||||||||||||||
| ALTIMATE_REVIEW_SIGNING_KEY: ${{ inputs.signing_key }} | ||||||||||||||||||||
| run: | | ||||||||||||||||||||
| args=(--mode "$IN_MODE" --manifest "$IN_MANIFEST" --severity "$IN_SEVERITY") | ||||||||||||||||||||
| [[ -n "$IN_PROJECT_DIR" && "$IN_PROJECT_DIR" != "." ]] && args+=(--cwd "$IN_PROJECT_DIR") | ||||||||||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: A misspelled Prompt for AI agents
Suggested change
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Follow-up: low priority, tracked. The action should check that |
||||||||||||||||||||
| [[ -n "$IN_BASE" ]] && args+=(--base "$IN_BASE") | ||||||||||||||||||||
| [[ -n "$IN_HEAD" ]] && args+=(--head "$IN_HEAD") | ||||||||||||||||||||
| [[ "$IN_POST" == "true" ]] && args+=(--post) | ||||||||||||||||||||
|
|
||||||||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,13 +5,15 @@ import { Agent } from "@/agent/agent" | |
| import { MessageV2 } from "@/session/message-v2" | ||
| import { MessageID, SessionID } from "@/session/schema" | ||
| import { Log } from "@/altimate/util/log" | ||
| import { redactSecrets } from "@/installation" | ||
| import { Dispatcher } from "../native" | ||
| import { type Finding, type ReviewCategory, type Severity, makeFinding } from "./finding" | ||
| import { readStreamText } from "../llm/stream-text" | ||
|
|
||
| const log = Log.create({ service: "ai-review" }) | ||
|
|
||
| const AI_TIMEOUT_MS = 60_000 | ||
| const AI_TIMEOUT_BASE_MS = 60_000 | ||
| const AI_TIMEOUT_MAX_MS = 300_000 | ||
| const MAX_DIFF_CHARS = 6_000 // per file, keep the prompt bounded | ||
| const MAX_FILES = 20 | ||
|
|
||
|
|
@@ -31,6 +33,42 @@ export interface AiReviewInput { | |
| grounding: Finding[] | ||
| prTitle?: string | ||
| prBody?: string | ||
| /** Explicit `provider/model`; omit to use the configured default model. */ | ||
| model?: string | ||
| } | ||
|
|
||
| /** What happened in the advisory lane, reported on the envelope and in the comment. */ | ||
| export type AiLaneState = "ok" | "skipped" | "timeout" | "error" | ||
| export interface AiLaneStatus { | ||
| state: AiLaneState | ||
| /** Why the lane was skipped or failed (never contains credentials). */ | ||
| reason?: string | ||
| /** `provider/model` the lane ran with. */ | ||
| model?: string | ||
| durationMs?: number | ||
| } | ||
| export interface AiReviewResult { | ||
| findings: Finding[] | ||
| status: AiLaneStatus | ||
| } | ||
|
|
||
| /** Timeout scaled to the prompt: 60 s plus 1 s per 2,000 characters, capped at 5 minutes. */ | ||
| export function aiTimeoutMs(promptChars: number): number { | ||
| return Math.min(AI_TIMEOUT_MAX_MS, AI_TIMEOUT_BASE_MS + Math.ceil(promptChars / 2_000) * 1_000) | ||
| } | ||
|
|
||
| /** Error text safe to show in a PR comment: one line, bounded, with | ||
| * credentials masked (authorization headers, `Basic`/`Bearer` values, | ||
| * `key=value` secrets, `user:pass@` URL userinfo, provider-prefixed tokens). | ||
| * Redaction runs before whitespace is collapsed so a header match stops at | ||
| * its own line. */ | ||
| export function safeReason(err: unknown): string { | ||
| const raw = err instanceof Error ? err.message : String(err) | ||
| return redactSecrets(raw) | ||
| .replace(/\s+/g, " ") | ||
| .replace(/(bearer\s+)[^\s]+/gi, "$1***") | ||
| .replace(/\b(sk|pk|key|token)[-_][A-Za-z0-9_-]{8,}/gi, "***") | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: (Based on your team's feedback about masking credential-stem error values.) [a5c5a10f-6a4a-4e42-8c2e-f8e01b48683d] Prompt for AI agents
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in 4d9a88a.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [MINOR] Secret masking in safeReason is incomplete Only Fix: add patterns for
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks. Fixed in |
||
| .slice(0, 200) | ||
| } | ||
|
|
||
| /** | ||
|
|
@@ -73,23 +111,47 @@ function buildUserMessage(input: AiReviewInput): string { | |
|
|
||
| /** | ||
| * Run the LLM reviewer lane. Returns advisory findings (severity ≤ warning, | ||
| * clamped by core), or [] if no model / core is available or the call fails — | ||
| * a review must never crash because the AI layer is unavailable. | ||
| * clamped by core) and a status saying whether the lane ran, was skipped, timed | ||
| * out or failed. Never throws — a review must not crash because the AI layer is | ||
| * unavailable — but never fails silently either: the status reaches the | ||
| * envelope and the PR comment. | ||
| */ | ||
| export async function runAiReview(input: AiReviewInput): Promise<Finding[]> { | ||
| export async function runAiReview(input: AiReviewInput): Promise<AiReviewResult> { | ||
| const startedAt = Date.now() | ||
| const done = (findings: Finding[], status: Omit<AiLaneStatus, "durationMs">): AiReviewResult => ({ | ||
| findings, | ||
| status: { ...status, durationMs: Date.now() - startedAt }, | ||
| }) | ||
| const files = input.files.filter((f) => f.status !== "deleted" && (f.diff || f.sql)) | ||
| if (!files.length) return [] | ||
| if (!files.length) return done([], { state: "skipped", reason: "no changed model SQL to review" }) | ||
|
|
||
| const controller = new AbortController() | ||
| const timeout = setTimeout(() => controller.abort(), AI_TIMEOUT_MS) | ||
| let timeoutMs = AI_TIMEOUT_BASE_MS | ||
| let timeout: ReturnType<typeof setTimeout> | undefined | ||
| let modelLabel: string | undefined | ||
| try { | ||
| const message = buildUserMessage({ ...input, files }) | ||
| timeoutMs = aiTimeoutMs(message.length) | ||
| timeout = setTimeout(() => controller.abort(), timeoutMs) | ||
| // Prompt comes from the compiled core, not this file. | ||
| const promptRes = await Dispatcher.call("altimate_core.review_ai_prompt", {}) | ||
| const system = ((promptRes.data ?? {}) as Record<string, unknown>).prompt as string | undefined | ||
| if (!system) return [] | ||
|
|
||
| const defaultModel = await Provider.defaultModel() | ||
| const model = await Provider.getModel(defaultModel.providerID, defaultModel.modelID) | ||
| if (!system) return done([], { state: "error", reason: "the core reviewer prompt is unavailable" }) | ||
|
|
||
| let model: Awaited<ReturnType<typeof Provider.getModel>> | ||
| try { | ||
| const chosen = input.model ? Provider.parseModel(input.model) : await Provider.defaultModel() | ||
| model = await Provider.getModel(chosen.providerID, chosen.modelID) | ||
| } catch (err) { | ||
| // An explicitly requested model that cannot be resolved is an error, not | ||
| // the same as `--no-ai`; only a missing default is a skip. | ||
| return done([], { | ||
| state: input.model ? "error" : "skipped", | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: Only a missing default should be Prompt for AI agents |
||
| reason: `no usable model: ${safeReason(err)}`, | ||
| model: input.model, | ||
| }) | ||
| } | ||
| modelLabel = `${model.providerID}/${model.id}` | ||
|
|
||
| const agent: Agent.Info = { | ||
| name: "dbt-ai-reviewer", | ||
|
|
@@ -119,18 +181,29 @@ export async function runAiReview(input: AiReviewInput): Promise<Finding[]> { | |
| abort: controller.signal, | ||
| sessionID: user.sessionID, | ||
| retries: 1, | ||
| messages: [{ role: "user", content: buildUserMessage({ ...input, files }) }], | ||
| messages: [{ role: "user", content: message }], | ||
| }) | ||
| const text = await readStreamText(stream) | ||
| if (!text) return [] | ||
| if (controller.signal.aborted) | ||
| return done([], { state: "timeout", reason: `no answer within ${Math.round(timeoutMs / 1000)} s`, model: modelLabel }) | ||
| if (!text) return done([], { state: "error", reason: "the model returned no text", model: modelLabel }) | ||
|
|
||
| // Parse + clamp in core (the prompt-injection-resistant, advisory-only | ||
| // contract). Returns already-validated, severity-clamped, file-checked items. | ||
| const parseRes = await Dispatcher.call("altimate_core.review_ai_parse", { | ||
| text, | ||
| valid_files: files.map((f) => f.path), | ||
| }) | ||
| const parsed = (((parseRes.data ?? {}) as Record<string, unknown>).findings as any[]) ?? [] | ||
| // The core parser reports failure in-band (`success: false`) rather than | ||
| // throwing; that is an error, never an "ok" run with no findings. | ||
| const parsedRaw = ((parseRes.data ?? {}) as Record<string, unknown>).findings | ||
| if (parseRes.success === false || !Array.isArray(parsedRaw)) | ||
| return done([], { | ||
| state: "error", | ||
| reason: `could not parse the model's answer: ${safeReason(parseRes.error ?? "no findings list")}`, | ||
| model: modelLabel, | ||
| }) | ||
| const parsed = parsedRaw as any[] | ||
| const byFile = new Map(files.map((f) => [f.path, f])) | ||
|
|
||
| const out: Finding[] = [] | ||
|
|
@@ -165,10 +238,12 @@ export async function runAiReview(input: AiReviewInput): Promise<Finding[]> { | |
| } | ||
| } | ||
| log.info("ai review complete", { findings: out.length }) | ||
| return out | ||
| return done(out, { state: "ok", model: modelLabel }) | ||
| } catch (err) { | ||
| log.error("ai review failed", { error: err }) | ||
| return [] | ||
| if (controller.signal.aborted) | ||
| return done([], { state: "timeout", reason: `no answer within ${Math.round(timeoutMs / 1000)} s`, model: modelLabel }) | ||
| return done([], { state: "error", reason: safeReason(err), model: modelLabel }) | ||
| } finally { | ||
| clearTimeout(timeout) | ||
| } | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
P2: This row documents a
degradedcondition that the review currently does not report: any parseable manifest makesdegradedfalse, even when the changed model is absent from it. Track whether each changed model actually resolves against the manifest, and update the stale note below that still says undecidable findings setdegraded.Prompt for AI agents