feat(pr-review): score confidence on 1 to 10 so clean reviews read safe

apps/ai/src/chat/prompts.ts

@@ -316,14 +316,22 @@ Write for an engineer who has ten seconds before they look at the diff. Short, p
 - \`checked\` is at most three bullets, each under 20 words, on risks you examined and ruled out, with the evidence you read or ran, never a claim from the description restated: "New query filters \`OrgId\` (\`queries/keys.ts:41\`)". Never generic ("reviewed for security issues").
 
 ## Confidence
-The review leads with one number, 1 to 5, how safe the change is to merge. It is computed from your findings (their severity and count), \`tests\`, \`risk\` and the observability coverage, so set those honestly:
+The review leads with one number, 1 to 10, how safe the change is to merge: 9 to 10 safe, 7 to 8 likely safe, 5 to 6 needs attention, 3 to 4 risky, 1 to 2 do not merge. It is computed from your findings (their severity and count), \`tests\`, \`risk\` and the observability coverage, so set those honestly:
 - \`tests\`: covered (tests exercise the behavior the change adds or alters), partial (some of it), missing (a behavior change no test exercises), not_needed (docs, copy, config, pure refactors under existing tests, generated files).
 - \`risk\`: high when the change touches auth, tenancy or org scoping, data migrations or deletes, billing, concurrency, or a public API or SDK contract; medium for shared code with many callers or a user-visible behavior change; low for everything contained.
-- \`confidence\` is optional and can only lower the number, by one point: set it when something specific you read makes the change riskier than those signals say. Never lower it because you cannot run the code, reach a live service or see production; no reviewer can, and that is not a property of this change.
+- \`confidence\` is your own number on the same 1 to 10 scale, and it must match what your confidenceReason says. It can lower the computed number by up to two points or raise it by one; findings cap it either way. Lower it when something specific you read makes the change riskier than the signals say, or when production diffs went unread (one point for a few, two for whole areas). Raise it only when you verified the risky part end to end (ran it, or read every caller and a test that fails without the change). Never lower it because you cannot run the code, reach a live service or see production; no reviewer can, and that is not a property of this change. Omit it when the signals already say what you would.
 - \`confidenceReason\` is one sentence, under 25 words, naming what decides the number or the one place that needs a careful look: "The new branch in \`listKeys\` changes tenant scoping and has no test". Never "Some risk remains". For a clean, contained change, say what makes it safe.
 
+How past reviews should have read:
+- 10: copy-only change across 11 files; the locale file keeps its exact key set. No findings, tests not needed, risk low.
+- 9: \`Effect.forEach\` chunking of bulk writes, pinned by a 3,100-row test. No findings, tests covered, risk medium: the medium risk alone is not a reason to lower it.
+- 8: tracer fix pinned by a real-error test, but one await path has no test of its own. No findings, tests partial, risk medium.
+- 7: an auth-path gate with tests, where one unguarded call now rests on a patch applying. Lowered one point from 8 for that specific reason.
+- 6: two shared-primitive warnings confirmed, and most per-view migration diffs unread. Lowered for the unread areas.
+- 4: a new endpoint with no Server span and a token in a query string. A warning plus unobservable work.
+
 ## Producing the review
-Call \`submit_review\` once with: resolved (the handles of earlier findings this head fixes, when the kickoff listed any; a fix you did not read is not resolved), verdict (clean | issues | not_applicable), tests, risk, confidence (only to lower it), confidenceReason, summary, keyChanges, checked, coverage (observability only: one row per unit of production work the diff adds, with unit, kind, instrumented and evidence; empty when it adds none; build tooling, tests and scripts are not units), findings (path, line, endLine, category, checkId for observability, severity, title, body, suggestion, replacement; only the ones you did not already save, since saved findings are added for you), telemetryDismissals (only for a removed name you found still emitted at the head: name, path, line). The review IS the submit_review call; prose instead of it is discarded.
+Call \`submit_review\` once with: resolved (the handles of earlier findings this head fixes, when the kickoff listed any; a fix you did not read is not resolved), verdict (clean | issues | not_applicable), tests, risk, confidence (1 to 10, only when it differs from the signals), confidenceReason, summary, keyChanges, checked, coverage (observability only: one row per unit of production work the diff adds, with unit, kind, instrumented and evidence; empty when it adds none; build tooling, tests and scripts are not units), findings (path, line, endLine, category, checkId for observability, severity, title, body, suggestion, replacement; only the ones you did not already save, since saved findings are added for you), telemetryDismissals (only for a removed name you found still emitted at the head: name, path, line). The review IS the submit_review call; prose instead of it is discarded.
 
 ## After the review
 If someone asks a follow-up in this session, answer with the same tools and the evidence you already gathered.

apps/web/src/components/code-review/code-review-analytics.tsx

@@ -452,7 +452,7 @@ function SecondaryStats({ current }: { current: CodeReviewTotals }) {
 	const stats = [
 		{
 			label: "Avg confidence",
-			value: current.avgConfidence === null ? "–" : `${current.avgConfidence.toFixed(1)}/5`,
+			value: current.avgConfidence === null ? "–" : `${current.avgConfidence.toFixed(1)}/10`,
 			hint: "How safe the reviewer judged each change to merge",
 		},
 		{

apps/web/src/components/code-review/review-detail-sheet.tsx

@@ -145,7 +145,7 @@ function ReviewDetailContent({
 					<Stat label="Outcome" value={outcome.label} tone={outcome.tone} />
 					<Stat
 						label="Confidence"
-						value={review.confidence === null ? "–" : `${review.confidence}/5`}
+						value={review.confidence === null ? "–" : `${review.confidence}/10`}
 					/>
 					<Stat label="Quality" value={review.score === null ? "–" : `${review.score}/100`} />
 					<Stat label="Issues" value={formatCount(review.findings)} />

apps/web/src/routes/code-review/pull-requests.tsx

@@ -280,7 +280,7 @@ function ReviewTable({
 									) : (
 										<>
 											{review.confidence}
-											<span className="text-muted-foreground">/5</span>
+											<span className="text-muted-foreground">/10</span>
 										</>
 									)}
 								</TableCell>

docs/pr-review-agent-plan.md

@@ -101,18 +101,18 @@ needs work (50+) or poor. A single warning scores 90 but grades good, so the hea
 gap excellent. The score is computed from the findings rather than asked of the model, so the same
 gaps always score the same, and it is stored in `pr_reviews.score`.
 
-The headline is now a confidence level from 1 to 5 (`confidencePrReview`, same file), from "do not
+The headline is now a confidence level from 1 to 10 (`confidencePrReview`, same file), from "do not
 merge" to "safe to merge". It starts from the quality score, deducts for missing tests, a high-risk
-area and unobservable new work, and is capped by findings. The quality score is shown beneath it.
+area and unobservable new work in whole points, and is capped by findings. The quality score is shown beneath it.
 
 ### Publishing to GitHub
 
 `PrReviewService.submitReview` stores the report, then calls
 `VcsProviderClient.publishPullRequestReview`, which posts, in order:
 
-1. **A check run** named `Maple / review` on the head SHA, titled `Confidence <n>/5 · <verdict>`.
+1. **A check run** named `Maple / review` on the head SHA, titled `Confidence <n>/10 · <verdict>`.
    It concludes `success` only when the review finished, left nothing to address and did not
-   rate confidence 3 or lower; otherwise `neutral`. It concludes `failure` only when the repository enables
+   rate confidence 6 or lower; otherwise `neutral`. It concludes `failure` only when the repository enables
    `blockOnContractBreaks` and an undismissed contract break is open (see "Production telemetry").
    An installation that has not granted
    `checks: write` answers 403, and the review is posted without the check run. A rate-limited

packages/backend/src/services/pr-review/PrReviewService.test.ts

@@ -826,7 +826,7 @@ describe("PrReviewService.submitReview", () => {
 			assert.include(publication.summaryComment.body, "### Still open from earlier reviews")
 			assert.include(publication.summaryComment.body, "~~F1 · off by one~~")
 			// Two warnings (quality 80) and an unobservable route.
-			assert.equal(publication.title, "Confidence 2/5 · 2 issues to address")
+			assert.equal(publication.title, "Confidence 5/10 · 2 issues to address")
 			assert.deepEqual(resolvedThreads, ["T1:Fixed in `2222222`."])
 			const stored = Option.getOrThrow(yield* reviews.getReview(orgId, second.reviewId!))
 			assert.deepEqual(
@@ -1285,7 +1285,7 @@ describe("buildPublication", () => {
 		assert.equal(publication.conclusion, "neutral")
 		assert.include(publication.reviewBody ?? "", "1 inline note")
 		// Quality 88 reads 4, less half a point for the unobservable route.
-		assert.equal(publication.title, "Confidence 3/5 · 1 issue to address")
+		assert.equal(publication.title, "Confidence 6/10 · 1 issue to address")
 	})
 
 	it("always writes the summary comment, even with nothing to say inline", () => {
@@ -1303,7 +1303,7 @@ describe("buildPublication", () => {
 		assert.isTrue(publication.summaryComment.body.startsWith(prReviewCommentMarker(UNKNOWN_REVIEW)))
 		assert.include(
 			publication.summaryComment.body,
-			"## Maple review\n\n🟢 **Confidence 4/5** · likely safe to merge\n<sub>quality 100/100 · no findings · 0/1 new units observable</sub>",
+			"## Maple review\n\n🟢 **Confidence 9/10** · safe to merge\n<sub>quality 100/100 · no findings · 0/1 new units observable</sub>",
 		)
 	})
 
@@ -1357,7 +1357,7 @@ describe("buildPublication", () => {
 			repositoryUrl: `${REPO_URL}/`,
 		})
 		assert.include(comment, `(${REPO_URL}/blob/${HEAD}/src/a%20b.ts#L4-L6)`)
-		assert.include(comment, "🔴 **Confidence 2/5** · risky as written")
+		assert.include(comment, "🔴 **Confidence 4/10** · risky as written")
 		assert.include(comment, "<sub>quality 75/100 · 1 critical · 0/1 new units observable</sub>")
 		assert.include(comment, "Observability coverage: 0 of 1 changes observable")
 		assert.include(comment, "<details><summary>🔴 <b>Critical</b> · gap</summary>")
@@ -1442,7 +1442,7 @@ describe("buildPublication", () => {
 					coverage: [],
 					tests: "covered",
 					risk: "low",
-					confidence: 5,
+					confidence: 10,
 					confidenceReason: "Small change, verified end to end.",
 				}),
 				partial,
@@ -1451,18 +1451,18 @@ describe("buildPublication", () => {
 			})
 		assert.include(
 			render([]),
-			"🟢 **Confidence 5/5** · safe to merge\nSmall change, verified end to end.\n<sub>quality 100/100 · no findings · tests covered · risk low</sub>",
+			"🟢 **Confidence 10/10** · safe to merge\nSmall change, verified end to end.\n<sub>quality 100/100 · no findings · tests covered · risk low</sub>",
 		)
 		const critical = render([
 			{ path: "a.ts", line: 1, category: "correctness", severity: "critical", title: "t", body: "b" },
 		])
 		assert.include(
 			critical,
-			"🔴 **Confidence 2/5** · risky as written\nHeld at 2 because a critical finding is open.",
+			"🔴 **Confidence 4/10** · risky as written\nHeld at 4 because a critical finding is open.",
 		)
 		assert.notInclude(critical, "verified end to end")
 		const partial = render([], true)
-		assert.include(partial, "🟡 **Confidence 3/5** · needs attention\n<sub>")
+		assert.include(partial, "🟡 **Confidence 6/10** · needs attention\n<sub>")
 		// The early end is the warning; a reason saying so again is left out.
 		assert.notInclude(partial, "Held at")
 		assert.include(partial, "ended early")
@@ -1476,14 +1476,17 @@ describe("buildPublication", () => {
 				headSha: HEAD,
 				partial,
 				repositoryUrl: REPO_URL,
+				// Signals read 7: an unobservable unit, partial tests and a medium-risk area.
 				report: new PrReviewReport({
 					...report([]),
+					tests: "partial",
+					risk: "medium",
 					...(confidence === undefined ? undefined : { confidence }),
 				}),
 			}).conclusion
-		assert.equal(conclusion(5), "success")
-		assert.equal(conclusion(4), "success")
-		assert.equal(conclusion(3), "neutral")
+		assert.equal(conclusion(undefined), "success")
+		assert.equal(conclusion(8), "success")
+		assert.equal(conclusion(6), "neutral")
 		assert.equal(conclusion(undefined, true), "neutral")
 	})
 

packages/backend/src/services/pr-review/PrReviewService.ts

@@ -34,11 +34,13 @@ import {
 	PrReviewTelemetry,
 	openContractBreaks,
 	PR_REVIEW_CONFIDENCE_LABEL,
+	PR_REVIEW_CONFIDENCE_MAX,
 	PR_REVIEW_FAILURE_COPY,
 	DEFAULT_REVIEWER_MENTION,
 	type PrReviewFailureReason,
 	prReviewFailureReason,
 	confidencePrReview,
+	prReviewConfidenceTone,
 	scorePrReview,
 	type PullRequestCheckAnnotation,
 	type PullRequestEventJob,
@@ -529,7 +531,8 @@ const SEVERITY_ALERT = {
 } as const satisfies Record<PrReviewFinding["severity"], string>
 
 /** Green when safe, amber when it needs a look, red when risky. */
-const confidenceMark = (confidence: number) => (confidence >= 4 ? "🟢" : confidence === 3 ? "🟡" : "🔴")
+const CONFIDENCE_MARK = { safe: "🟢", attention: "🟡", risky: "🔴" } as const
+const confidenceMark = (confidence: number) => CONFIDENCE_MARK[prReviewConfidenceTone(confidence)]
 
 /** `observability · SPAN-03`, or the bare category for every other lens. */
 const categoryLabel = (finding: { readonly category: string; readonly checkId?: string }): string =>
@@ -622,7 +625,7 @@ export const renderReviewMarkdown = (input: ReviewMarkdownInput & { readonly hea
 		lines.push("**Nothing to review**", "")
 	} else {
 		lines.push(
-			`${confidenceMark(confidence.confidence)} **Confidence ${confidence.confidence}/5** · ${PR_REVIEW_CONFIDENCE_LABEL[confidence.confidence]}`,
+			`${confidenceMark(confidence.confidence)} **Confidence ${confidence.confidence}/${PR_REVIEW_CONFIDENCE_MAX}** · ${PR_REVIEW_CONFIDENCE_LABEL[confidence.confidence]}`,
 		)
 		// An early end is already the warning below; the reason would only say it again.
 		if (confidence.reason !== undefined && confidence.cappedBy !== "partial")
@@ -877,7 +880,7 @@ export const buildPublication = (input: {
 		checkName: PR_REVIEW_CHECK_NAME,
 		// Fixed order, like the scorecard: a check list scans down one column.
 		title: [
-			`Confidence ${confidence === undefined ? "–" : `${confidence.confidence}/5`}`,
+			`Confidence ${confidence === undefined ? "–" : `${confidence.confidence}/${PR_REVIEW_CONFIDENCE_MAX}`}`,
 			verdictTitle(report, carried),
 		].join(" · "),
 		summary: renderCheckSummary(markdown),
@@ -886,7 +889,7 @@ export const buildPublication = (input: {
 		// finished, found nothing to address and is confident the change is safe.
 		conclusion: blocking
 			? "failure"
-			: hasIssues || input.partial || (confidence !== undefined && confidence.confidence <= 3)
+			: hasIssues || input.partial || (confidence !== undefined && prReviewConfidenceTone(confidence.confidence) !== "safe")
 				? "neutral"
 				: "success",
 		annotations,

packages/db/drizzle/20261006231211_pr_review_confidence_ten/migration.sql

@@ -0,0 +1,18 @@
+-- PR review confidence moved from 1-5 to 1-10: double stored values so history reads on one scale.
+-- Only values still on the old scale are touched, and a stored "Held at N" cap reason doubles with them.
+UPDATE "pr_reviews"
+SET "report_json" = CASE
+	WHEN "report_json"->>'confidenceReason' ~ '^Held at [1-5] '
+		THEN jsonb_set(
+			"report_json",
+			'{confidenceReason}',
+			to_jsonb(regexp_replace(
+				"report_json"->>'confidenceReason',
+				'^Held at [1-5] ',
+				'Held at ' || (substring("report_json"->>'confidenceReason' from '^Held at ([1-5]) ')::int * 2) || ' '
+			))
+		)
+	ELSE "report_json"
+END || jsonb_build_object('confidence', ("report_json"->>'confidence')::int * 2)
+WHERE jsonb_typeof("report_json"->'confidence') = 'number'
+	AND ("report_json"->>'confidence')::numeric BETWEEN 1 AND 5;