Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
1f8e33b
Add Pull Request Review Links and Automated Bot Comments
LinceMathew Oct 12, 2026
1c88ca6
Add risk-based auto-approve policy engine
shrsv Oct 13, 2026
0e397a0
Add hover help to every auto-approve / risk-policy control
shrsv Oct 13, 2026
eddaff9
Stop risk score from hiding the LLM classification gate's verdict
shrsv Oct 13, 2026
5cd909e
Weight risk scoring by the full finding taxonomy, not just security
shrsv Oct 13, 2026
7ddf910
Append Review Links to Pull Request Descriptions
LinceMathew Oct 13, 2026
36b620a
Fix Auto Approve Policies missing from mega menu and Ctrl-K search fo…
shrsv Oct 13, 2026
875c28f
Surface blast radius as the top, highest-weighted risk signal
shrsv Oct 13, 2026
98ac352
Prove and explain the risk-based auto-approve engine, with a regenera…
shrsv Oct 14, 2026
daaf8a0
Add an Auto Approve Reports tab: score distribution, decisions, calib…
shrsv Oct 14, 2026
6791bcb
Add Multi-Provider Auto-Approval and Risk Management
LinceMathew Oct 15, 2026
2a5c726
Add a risk-policy engine six-pager and refresh the verification report
shrsv Oct 15, 2026
988cb48
Lead README with Auto-Approve, matching the /livereview site pitch
shrsv Oct 15, 2026
a12be24
Split README into Auto-Approve and Within-Diff Prioritization sections
shrsv Oct 15, 2026
0f7df18
Clarify auto-approve wording and surface the decision on the review page
LinceMathew Oct 15, 2026
248b60e
Remove local dump noise from db/schema.sql
LinceMathew Oct 15, 2026
f33295b
bug fixes
LinceMathew Oct 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .git-hooks-binary-allowlist
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,6 @@ assets/screenshots/git-lrc/risk-scored-view-1.png
assets/screenshots/git-lrc/risk-scored-view-2.png
assets/screenshots/git-lrc/summary-deck.gif
scripts/blast-radius-demo-export/static/assets/gemini-icon.png

assets/screenshots/auto-approve/demo.mp4
assets/screenshots/auto-approve/poster.webp
4 changes: 4 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -381,6 +381,10 @@ SECURITY_GOVULN_PACKAGES := $(filter-out ./scripts ./cmd/onboarding-pdf,$(TEST_P
testall:
$(GOTEST) -count=1 $(TEST_PACKAGES)

.PHONY: regen-risk-report
regen-risk-report:
python3 scripts/regen_risk_report.py

.PHONY: security-govulncheck security-govulncheck-json security-osv security-gitleaks security-semgrep security-dependabot security-gh-secret-scanning security-triage

# Run Go vulnerability analysis for reachable vulnerabilities.
Expand Down
36 changes: 33 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,36 @@

# LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by **blast radius**: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
LiveReview is the org-wide risk-aware harness for AI-written code. It covers two different jobs: deciding which pull requests are safe to skip entirely, and prioritizing what's risky inside the ones that still need a human.

<a id="auto-approve"></a>
## Auto-Approve: Ship the Safe 50% Without a Human

**Auto-Approve Upto 50% of Your Pull Requests, Safely.** Risk scoring decides which MRs/PRs are safe enough to ship without a human, so your team only spends review time on risky and critical changes.

[![Watch the Auto-Approve demo](./assets/screenshots/auto-approve/poster.webp)](./assets/screenshots/auto-approve/demo.mp4)

<p align="center"><i>LiveReview auto-approving a low-risk change while routing a risky change to a human reviewer. Click to play.</i></p>

The risk score blends three signals:

- Semantic LLM analysis
- A custom ML model trained on your production risk signals
- Static graph-based codebase analysis

Inspired by Meta's RADAR paper: **50x** fewer production incidents, **3x** fewer rollbacks, **35%** faster reviews.

How it works:

1. **Track outcomes** - trace production incidents and rollbacks back to the commits and reviews behind them.
2. **Learn from the data** - combine those outcomes with LLM analysis, static checks, and ML signals into a risk score per change.
3. **Automate selectively** - auto-approve the lowest-risk 30-50% of changes. Choose a conservative, balanced, or aggressive policy. Risky or uncertain changes still go to a human.
4. **Keep improving** - track incidents, false approvals, and overrides, and tighten or loosen the policy as real data comes in.

<a id="blast-radius-scoring"></a>
## Within-Diff Prioritization: Blast Radius & Review Priority

For everything that still goes to a human, LiveReview scores every hunk of a diff by **blast radius**: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.

https://github.com/user-attachments/assets/b7663ad5-e792-4d24-8452-18bbb9b958a0

Expand Down Expand Up @@ -51,7 +80,7 @@ LiveReview organizes around five capabilities, each answering a question your te

| Pillar | Question it answers | What it covers |
|---|---|---|
| [**Review**](#org-wide-harness) | Can we get both speed and quality? | Multi-level review checkpoints (commit, push, MR/PR, CI/CD, scheduled), [Adaptive Reviews](#adaptive-reviews) |
| [**Review**](#org-wide-harness) | Can we get both speed and quality? | Multi-level review checkpoints (commit, push, MR/PR, CI/CD, scheduled), [Auto-Approve](#auto-approve), [Adaptive Reviews](#adaptive-reviews) |
| [**Understand**](#features) | Can an engineer quickly understand what changed and what it means? | Summary decks, Issue Navigator, PR summaries, [Cognitive Coverage / PR Quizzes](#cognitive-coverage) |
| [**Enforce**](#repository-rules) | Can we make our engineering standards and review policies actually happen? | [Repository Rules](#repository-rules), [CI/CD Gates](#cicd-gates), custom prompts, scheduled reviews |
| [**Improve**](#impact-report) | How are we doing, and where should engineering improve? | [Impact Report](#impact-report), [Livi / Data-Backed Decisions](#data-backed-decisions) |
Expand All @@ -66,7 +95,8 @@ The result: engineering becomes a business advantage. You stay ahead, your engin
| I want to... | Go to |
|---|---|
| Try LiveReview free in under 5 minutes | [Quick Start](#quick-start) |
| Understand Blast-Radius scoring | [above ↑](#livereview-risk-aware-ai-code-review-for-business-critical-systems) |
| Auto-approve low-risk pull requests safely | [Auto-Approve](#auto-approve) |
| Understand Blast-Radius scoring | [Within-Diff Prioritization](#blast-radius-scoring) |
| Enforce checks at commit / push / PR / CI | [Org-Wide Harness](#org-wide-harness) |
| Write a precise, custom rule for blocking a merge | [CI/CD Gates](#cicd-gates) |

Expand Down
Binary file added assets/screenshots/auto-approve/demo.mp4
Binary file not shown.
Binary file added assets/screenshots/auto-approve/poster.webp
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
-- migrate:up

-- Set once the "Start a review / View history" links comment is posted on a new PR/MR,
-- so duplicate webhook deliveries don't post it twice.
ALTER TABLE pull_requests ADD COLUMN links_comment_posted_at TIMESTAMPTZ;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: info

Using TIMESTAMPTZ for links_comment_posted_at is a good practice.


-- migrate:down

ALTER TABLE pull_requests DROP COLUMN IF EXISTS links_comment_posted_at;
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
-- migrate:up

-- Deleting a connector cascades to its repositories; review_commits rows blocked that.
-- Keep the review's commit history and just clear the repository link instead.
ALTER TABLE review_commits DROP CONSTRAINT review_commits_repository_id_fkey;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: warning

Changing the foreign key constraint to ON DELETE SET NULL represents a significant behavioral change. Please confirm that this desired data retention behavior is intended.

Suggestions:

  1. Confirm this ON DELETE SET NULL behavior is explicitly desired for review_commits.

ALTER TABLE review_commits ADD CONSTRAINT review_commits_repository_id_fkey
FOREIGN KEY (repository_id) REFERENCES repositories(id) ON DELETE SET NULL;

-- migrate:down

ALTER TABLE review_commits DROP CONSTRAINT review_commits_repository_id_fkey;
ALTER TABLE review_commits ADD CONSTRAINT review_commits_repository_id_fkey
FOREIGN KEY (repository_id) REFERENCES repositories(id);
168 changes: 168 additions & 0 deletions db/migrations/20261013090000_create_risk_policy_engine.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
-- migrate:up

-- Risk-based auto-approve policy engine. See docs/risk-policy-engine.md for the
-- full design. Replaces the placeholder "no critical findings" policy in
-- internal/jobqueue/pr_policy_worker.go with a layered funnel
-- (eligibility -> static jq gates -> trained risk score -> LLM classification -> decision).

-- 1. The seniority dial that gates eligibility: a plain integer owners bump per
-- user (no roles/tiers table). Default 1 (lowest).
ALTER TABLE users ADD COLUMN review_level INTEGER NOT NULL DEFAULT 1;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: warning

The review_level column has a default value of 1 on the users table. Please ensure that this default value aligns with the initial policy.

Suggestions:

  1. Verify DEFAULT 1 for review_level on existing users aligns with initial policy expectations.


-- 2. Pull-request author email, captured opportunistically during PR sync so the
-- identity map (below) can match an author to a LiveReview user by email as
-- well as by provider username. Often absent on GitHub (commit author email
-- may be used); nullable.
ALTER TABLE pull_requests ADD COLUMN author_email TEXT;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: warning

The author_email column is nullable on the pull_requests table. Please ensure that downstream logic gracefully handles NULL values.

Suggestions:

  1. Confirm all consumers of pull_requests.author_email handle NULL values correctly.


-- 3. User identities: the "alternative emails / handles" each org member can hold,
-- used to map a PR author (provider username or email) back to a LiveReview
-- user so eligibility uses their real review_level instead of the default.
-- Org-scoped; a member manages their own, owner/admin manage others'.
CREATE TABLE user_identities (
id BIGSERIAL PRIMARY KEY,
org_id BIGINT NOT NULL REFERENCES orgs(id),
user_id BIGINT NOT NULL REFERENCES users(id),
kind TEXT NOT NULL CHECK (kind IN ('email', 'username')),
value TEXT NOT NULL, -- normalized (lowercased) email or handle
created_by BIGINT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (org_id, kind, value)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: info

The UNIQUE (org_id, kind, value) constraint on the user_identities table is good for maintaining data integrity.

);

CREATE INDEX idx_user_identities_org_user ON user_identities (org_id, user_id);
CREATE INDEX idx_user_identities_lookup ON user_identities (org_id, kind, value);

-- 4. Per-org / per-repo auto-approve policy. repository_id NULL = org default.
-- 'inherit' is only valid on repo-scoped rows (explicit "use the org default").
CREATE TABLE auto_approve_policies (
id BIGSERIAL PRIMARY KEY,
org_id BIGINT NOT NULL REFERENCES orgs(id),
repository_id BIGINT REFERENCES repositories(id),
mode TEXT NOT NULL DEFAULT 'custom'
CHECK (mode IN ('inherit', 'custom', 'always_approve', 'never_approve')),
min_author_level INTEGER NOT NULL DEFAULT 1,
eligibility_whitelist_user_ids BIGINT[] NOT NULL DEFAULT '{}',
risk_threshold_percentile INTEGER NOT NULL DEFAULT 20,
require_llm_confidence INTEGER NOT NULL DEFAULT 8,
blocking_ruleset_ids BIGINT[] NOT NULL DEFAULT '{}',

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: info

The blocking_ruleset_ids array implies that a rulesets table either exists or will be created. Please ensure consistency in this regard.

Suggestions:

  1. Verify rulesets table definition and its relationship with blocking_ruleset_ids.

landing_delay_seconds INTEGER NOT NULL DEFAULT 0,
daily_approve_cap INTEGER,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: warning

The daily_approve_cap column is nullable. Please confirm if NULL implies no cap and document this behavior accordingly.

Suggestions:

  1. Explicitly document the meaning of a NULL daily_approve_cap value (e.g., no cap).

restrict_outcome_reporting_to_admins BOOLEAN NOT NULL DEFAULT false,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: info

The restrict_outcome_reporting_to_admins setting is a good security and privacy control.

force_approve_min_level INTEGER NOT NULL DEFAULT 3,
created_by BIGINT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE UNIQUE INDEX idx_auto_approve_policies_org_default ON auto_approve_policies (org_id) WHERE repository_id IS NULL;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: info

Unique indexes for organization-default and repository-specific policies are critical for ensuring correct policy lookup.

CREATE UNIQUE INDEX idx_auto_approve_policies_org_repo ON auto_approve_policies (org_id, repository_id) WHERE repository_id IS NOT NULL;
CREATE INDEX idx_auto_approve_policies_org ON auto_approve_policies (org_id);

-- 5. Append-only policy evaluation trace: one row per stage per review, so the
-- UI can render the full funnel (eligibility -> static -> risk -> llm -> final),
-- not just the final approve/reject.
CREATE TABLE review_policy_log (
id BIGSERIAL PRIMARY KEY,
review_id BIGINT NOT NULL REFERENCES reviews(id),
org_id BIGINT NOT NULL REFERENCES orgs(id),
stage TEXT NOT NULL,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: warning

The stage and verdict columns in the review_policy_log table could benefit from CHECK constraints or enum types to ensure consistency.

Suggestions:

  1. Add CHECK constraints for stage and verdict columns to enforce allowed values.
  2. Consider using PostgreSQL ENUM types for stage and verdict if values are fixed.

verdict TEXT NOT NULL,
detail JSONB NOT NULL DEFAULT '{}',
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_review_policy_log_review ON review_policy_log (review_id);
CREATE INDEX idx_review_policy_log_org ON review_policy_log (org_id, created_at DESC);

-- 6. Production-truth outcomes (revert / incident / confirmed-safe) that feed the
-- risk model's feedback loop. reason is mandatory. 'test' source is excluded
-- from training.
CREATE TABLE review_outcomes (
id BIGSERIAL PRIMARY KEY,
review_id BIGINT NOT NULL REFERENCES reviews(id),
org_id BIGINT NOT NULL REFERENCES orgs(id),
reported_by_user_id BIGINT,
outcome_type TEXT NOT NULL CHECK (outcome_type IN ('revert', 'incident', 'confirmed_safe')),
source TEXT NOT NULL CHECK (source IN ('webhook', 'manual', 'ci', 'test')),
reason TEXT NOT NULL,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: info

The reason column is mandatory for review_outcomes, which is good for accountability and auditability.

detail JSONB NOT NULL DEFAULT '{}',
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_review_outcomes_review ON review_outcomes (review_id);
CREATE INDEX idx_review_outcomes_org_created ON review_outcomes (org_id, created_at DESC);

-- 7. Human force-approve/override of a specific review's verdict, fully audited.
CREATE TABLE review_policy_overrides (
id BIGSERIAL PRIMARY KEY,
review_id BIGINT NOT NULL REFERENCES reviews(id),
org_id BIGINT NOT NULL REFERENCES orgs(id),
actor_user_id BIGINT NOT NULL REFERENCES users(id),
original_verdict TEXT NOT NULL,
forced_verdict TEXT NOT NULL,
reason TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_review_policy_overrides_review ON review_policy_overrides (review_id);
CREATE INDEX idx_review_policy_overrides_org ON review_policy_overrides (org_id, created_at DESC);

-- 8. Feature snapshots written at every policy evaluation: the training set once
-- review_outcomes rows attach to the same review_ids. Recomputed-after-the-fact
-- features would drift, so they're snapshotted at evaluation time.
CREATE TABLE risk_feature_snapshots (
id BIGSERIAL PRIMARY KEY,
review_id BIGINT NOT NULL REFERENCES reviews(id),
org_id BIGINT NOT NULL REFERENCES orgs(id),
model_version INTEGER NOT NULL,
features JSONB NOT NULL,
score DOUBLE PRECISION NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_risk_feature_snapshots_review ON risk_feature_snapshots (review_id);
CREATE INDEX idx_risk_feature_snapshots_org ON risk_feature_snapshots (org_id, created_at DESC);

-- 9. The trained risk model. Weights are a few dozen floats, so the row IS the
-- model (no blob store). org_id NULL = the shared seed/default every org
-- starts from. Append-only: any version can be reactivated as an instant rollback.
CREATE TABLE risk_model_versions (
id BIGSERIAL PRIMARY KEY,
org_id BIGINT REFERENCES orgs(id),
version INTEGER NOT NULL,
weights_json JSONB NOT NULL,
trained_at TIMESTAMPTZ NOT NULL DEFAULT now(),
training_sample_count INTEGER NOT NULL DEFAULT 0,
holdout_auc DOUBLE PRECISION,
update_kind TEXT NOT NULL CHECK (update_kind IN ('seed', 'online_update', 'batch_refit')),
is_active BOOLEAN NOT NULL DEFAULT false
);

CREATE INDEX idx_risk_model_versions_org_active ON risk_model_versions (org_id) WHERE is_active;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: info

The index on org_id and is_active for risk_model_versions is critical for the performance of active model lookups.


-- 10. Coarse, category-level admin weight overrides (blast_radius / security_findings /
-- critical_findings / diff_size): an instant multiplier on live scoring,
-- decoupled from the nightly retrain cadence.
CREATE TABLE risk_factor_overrides (
id BIGSERIAL PRIMARY KEY,
org_id BIGINT NOT NULL REFERENCES orgs(id),
feature_category TEXT NOT NULL,
multiplier DOUBLE PRECISION NOT NULL DEFAULT 1.0,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (org_id, feature_category)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: info

The UNIQUE (org_id, feature_category) constraint on risk_factor_overrides ensures that there is only one override per category per organization.

);

-- migrate:down

DROP TABLE IF EXISTS risk_factor_overrides;
DROP TABLE IF EXISTS risk_model_versions;
DROP TABLE IF EXISTS risk_feature_snapshots;
DROP TABLE IF EXISTS review_policy_overrides;
DROP TABLE IF EXISTS review_outcomes;
DROP TABLE IF EXISTS review_policy_log;
DROP TABLE IF EXISTS auto_approve_policies;
DROP TABLE IF EXISTS user_identities;
ALTER TABLE pull_requests DROP COLUMN IF EXISTS author_email;
ALTER TABLE users DROP COLUMN IF EXISTS review_level;
Loading
Loading