Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ All notable changes to opendrs. Versions follow [Semantic Versioning](https://se
## 0.5.0 - 2026-10-23

The verification pass (`llm.verify`) ships, measured against the field test's real corpus (49 PRs,
six repositories — [docs/experiments.md](docs/experiments.html#verifier-precision-study-on-the-field-test-corpus)).
six repositories — [docs/experiments.md](docs/experiments.md#verifier-precision-study-on-the-field-test-corpus)).
The first run (v1) recalled the true-useful facts and correctly handled the noisy
`coupling.missing` fires, but missed both `contract.dependency` WRONG fires and ran over the cost
bar: $0.0210/PR against a $0.02 bar, with a dead `confidence` field required on every verified
Expand All @@ -15,12 +15,12 @@ them before the verifier ever saw a candidate). Fixed at the root — the dead f
elimination case now scored correctly — the re-run (v2) passes on all three legs: recall 1/14
wrongly dropped, `coupling.missing` 14/18 handled, both WRONG fires accounted for, and
**$0.0192/PR**, under the cap. Full numbers, including the honest v1 FAIL, in
[docs/experiments.md](docs/experiments.html#verifier-precision-study-on-the-field-test-corpus).
[docs/experiments.md](docs/experiments.md#verifier-precision-study-on-the-field-test-corpus).

### Added

- Verification pass (`llm.verify`, Phase 2 of the verification-pass spec,
[docs/verification.md](docs/verification.html)): with `llm.verify.enabled: true`, one batched
[docs/verification.md](docs/verification.md)): with `llm.verify.enabled: true`, one batched
call to the judge's provider reads every captain fact, review-first hunk and check of the brief
with fresh context and keeps, drops or downgrades each one. A drop needs a named class (docs,
tests, fixtures, generated, formatting) and a cited line, else it is a downgrade; an uncited
Expand All @@ -33,21 +33,21 @@ wrongly dropped, `coupling.missing` 14/18 handled, both WRONG fires accounted fo
commit, `wrong`/`intentional`/`ignored` downgrade that fact on every later run without a model
call, and `useful` pins it against any verifier drop or downgrade.
- Pre-PR self-check (Phase 1 of the verification-pass spec,
[docs/specs/2026-10-20-verification-pass-design.md](docs/specs/2026-10-20-verification-pass-design.html)):
[docs/specs/2026-10-20-verification-pass-design.md](docs/specs/2026-10-20-verification-pass-design.md)):
`opendrs brief --self-check-template` and the MCP tool `self_check_items` list the ids a pre-PR
self-check should fix, explain, or (a real, unresolved problem the agent can't fix itself) flag as
`needs-fix`; the Action reads a `<!-- opendrs-self-check:v1 ... -->` block from the head commit's
own message, recomputes the brief fresh, and renders one advisory line next to the captain line
(`author agent self-checked: N facts addressed, M explained, K flagged as still needing a fix ·
not part of the review decision`). Advisory only — never read by the gate or the captain's
priority. See
[docs/llm-reviewers.md](docs/llm-reviewers.html#self-check-before-you-open-the-pr) and
[docs/github-action.md](docs/github-action.html#author-self-check-advisory).
[docs/llm-reviewers.md](docs/llm-reviewers.md#self-check-before-you-open-the-pr) and
[docs/github-action.md](docs/github-action.md#author-self-check-advisory).
- Two new captain facts from the ApacheJIT captain-fact-lift experiment: `change.concurrency`
(strong; a changed, non-comment line of non-test code uses a concurrency primitive — 2.04× bug
lift) and `change.safety_removed` (info; net deleted error-handling or guard lines per hunk —
1.79× lift, but fires often enough to stay informational). See
[docs/captain.md](docs/captain.html#4-the-facts).
[docs/captain.md](docs/captain.md#4-the-facts).
- The captain brief's layers detector (`layers.edge`, `layers.cycle`) and the drift detector's
"outside the original modules" check now use auto-detected module boundaries instead of a flat
path-segment split: package roots are found from `package.json` (JS/TS), `go.mod` directories
Expand Down Expand Up @@ -80,8 +80,8 @@ for the module-boundaries design.
either every hunk of the firing file removes lines with no identifier surviving in the partner's
base text, or the firing file is a test whose source partner is untouched (spec
§3.8 of
[docs/specs/2026-10-20-verification-pass-design.md](docs/specs/2026-10-20-verification-pass-design.html)).
See [docs/captain.md](docs/captain.html#4-the-facts).
[docs/specs/2026-10-20-verification-pass-design.md](docs/specs/2026-10-20-verification-pass-design.md)).
See [docs/captain.md](docs/captain.md#4-the-facts).
- A repo that already sets `captain.moduleDepth` explicitly in `.opendrs.yml` keeps its exact
current behaviour: setting it disables auto-detection repo-wide and module boundaries stay a
flat split on that many path segments from the repo root, unchanged.
Expand Down
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ default branch; see [how to auto-approve low-risk PRs](docs/auto-approve.md) and
flowchart LR
diff["git diff<br/>merge-base...head"] --> signals
hist["git history"] --> signals
graph["import graph"] --> signals
imports["import graph"] --> signals
llm["LLM judge<br/>(optional)"] --> signals
signals["Signals<br/>each with evidence<br/>and provenance"] --> model["Logistic model<br/>score = sigmoid(b + Σ w·x)"]
model --> tier["Tier<br/>green / yellow / red"]
Expand Down Expand Up @@ -127,6 +127,8 @@ editor nor the AI review comments showed. See [the captain brief](docs/captain.m
| Reviewer assignment | One of the top three experts is asked to review risky PRs | on in the Action | [Policy](docs/policy.md) |
| Hotspots | Rank files by risk at rest | run on demand | [CLI and MCP](docs/cli-and-mcp.md) |
| Review brief | Ranks hunks to read first, fixed check questions, safe-to-skim list, hard-rule reasons — recipes for claude-code-action, Bugbot, CodeRabbit (limited), Copilot and Codex (paste) | on | [LLM reviewers](docs/llm-reviewers.md), [CLI and MCP](docs/cli-and-mcp.md) |
| Pre-PR self-check | Lists the ids an author agent should fix or explain before opening the PR; opendrs reads the verdicts back from the head commit and adds one advisory line, never part of the gate | off | [Self-check recipe](docs/llm-reviewers.md#self-check-before-you-open-the-pr) |
| Verification pass | One batched LLM call re-checks every captain fact, review-first hunk and check with fresh context and keeps, drops or downgrades each one; never changes the score, tier or policy | off | [Verification pass](docs/verification.md) |
| Incident feedback | `opendrs feedback` records incidents for calibration | run on demand | [Calibration](docs/calibration.md) |

Every setting lives in `.opendrs.yml`, read from the base branch so a PR cannot loosen its own gate:
Expand Down Expand Up @@ -167,6 +169,7 @@ Flags, exit codes, MCP tools and the agent skill: [CLI, MCP server, Claude Code
| [LLM judge](docs/llm-judge.md) | turn on the optional LLM judge or the `requireSafe` gate |
| [Feeding the review brief to an LLM reviewer](docs/llm-reviewers.md) | get opendrs's findings in front of Claude Code, claude-code-action, Bugbot, CodeRabbit, Copilot or Codex |
| [The captain brief](docs/captain.md) | know what the final human reviewer sees: priority, reviewer, and the architecture and history facts |
| [Verification pass](docs/verification.md) | turn on an LLM re-check of the brief's facts before a human reads them |
| [Calibration and backtesting](docs/calibration.md) | fit the model to your repository and choose thresholds |
| [Benchmark results](docs/results.md) | see how opendrs ranks risk on public repositories |
| [Experiments](docs/experiments.md) | see what we tried and what did not work |
Expand Down
4 changes: 2 additions & 2 deletions docs/alternatives.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,11 @@ what it measures, read [what PR blast radius is](blast-radius.md); to set it up,
- **You've seen CodeRabbit's "Merge Risk" label and "Estimated code review effort":** they are
useful, but CodeRabbit does not publish how either is computed. opendrs's score is calibrated on
your own repository's history (`opendrs backtest`), every signal's contribution to it is shown in
the comment, and the [captain brief](captain.html) names the actual facts behind a verdict — a
the comment, and the [captain brief](captain.md) names the actual facts behind a verdict — a
new import edge, a broken contract, a file that usually changes with one this PR didn't — instead
of a label alone. It is also open source, with no service to host or account to create.
- **You want the single highest accuracy number, and can host a model:** DRS-OSS reports AUC 0.895
on ApacheJIT's official split; opendrs's calibrated model reports AUC 0.823 on the same benchmark
(a different split; see [Calibration and backtesting](calibration.html)). That gap is the
(a different split; see [Calibration and backtesting](calibration.md)). That gap is the
trade-off: DRS-OSS needs a fine-tuned Llama 3.1 8B classifier running behind a REST API you host
and keep up; opendrs runs as a CLI, GitHub Action or MCP server with no model service to deploy.
6 changes: 3 additions & 3 deletions docs/auto-approve.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ up a gate that approves the low-risk ones on its own and sends everything else t
> A PR whose risk score is below 0.3, that changes nothing on a blocked path, and whose AI review
> passed is approved and goes to the merge queue. Everything else waits for a reviewer.

The score comes from [opendrs](./), which measures each PR's [blast radius](blast-radius.html),
The score comes from [opendrs](./), which measures each PR's [blast radius](blast-radius.md),
file history, author experience, size and tests.

## 1. Score every PR without approving anything
Expand Down Expand Up @@ -57,7 +57,7 @@ npx @error-path/opendrs backtest # compare default and calibrated

The backtest reports what share of bug-introducing commits the score would have flagged at each
threshold. Commit the calibrated model through a normal reviewed PR. See
[Calibration and backtesting](calibration.html) for how to read the numbers.
[Calibration and backtesting](calibration.md) for how to read the numbers.

## 3. Turn on the branch rules approval depends on

Expand Down Expand Up @@ -129,4 +129,4 @@ Whatever the score:
- PRs from forks, drafts, and PRs where a reviewer requested changes;
- PRs where a signal could not be computed and that unknown could have changed the tier.

The full rule order and every option are in the [GitHub Action guide](github-action.html).
The full rule order and every option are in the [GitHub Action guide](github-action.md).
6 changes: 3 additions & 3 deletions docs/blast-radius.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ dependents count from the graph.

The dependents count drives `blast.dependents` and `blast.public_surface` in the score above, and
it also shows up in the AI-facing review brief's hunk reasons. It reaches the
[captain brief](captain.html) only once: as the "imported by N files" evidence on a
[captain brief](captain.md) only once: as the "imported by N files" evidence on a
`contract.api` fact, when a deleted public declaration has dependents. The captain brief does not
otherwise repeat a raw dependents count — that is editor trivia the captain can see in one click.

Expand All @@ -56,7 +56,7 @@ A logistic model turns the signals into a probability between 0 and 1. Each sign
is shown in the PR comment, so you can see why a PR scored what it did. The default weights are
priors; `opendrs calibrate` fits them to your repository's history. The full feature table and
its link to Meta's Diff Risk Score are in the
[signal reference](signals.html#metas-drs-features-and-their-opendrs-signals).
[signal reference](signals.md#metas-drs-features-and-their-opendrs-signals).

## Check a change's blast radius yourself

Expand All @@ -68,4 +68,4 @@ npx @error-path/opendrs explain src/auth.ts # one file's dependents, history an
Coding agents can call the same checks through the MCP server (`score_diff`,
`explain_file_risk`, `suggest_reviewers`) before they open a PR, and split a change whose blast
radius is large. To use the score as an approval gate, see
[Auto-approve low-risk pull requests](auto-approve.html).
[Auto-approve low-risk pull requests](auto-approve.md).
2 changes: 1 addition & 1 deletion docs/calibration.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ labelled positive.
close, and this benchmark is how to measure it.

Every experiment behind these numbers, including the ones that did not help, is in the
[experiment log](experiments.html).
[experiment log](experiments.md).

The full report, with per-project numbers, coverage and caveats, is in
[`bench/apachejit/results.md`](https://github.com/error-path/opendrs/blob/main/bench/apachejit/results.md); [`bench/apachejit/README.md`](https://github.com/error-path/opendrs/blob/main/bench/apachejit/README.md)
Expand Down
8 changes: 4 additions & 4 deletions docs/captain.md
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,7 @@ A quick confirm with no facts is one line and no block: `**Captain: quick confir
facts (not capped at five) is also in the "For the captain" section of the brief for AI reviewers,
and in the `captain` Action output described below.

With the [verification pass](verification.html) on, each fact in the comment and in the brief ends
With the [verification pass](verification.md) on, each fact in the comment and in the brief ends
with its verdict (`[verified · new]`, `[verified · pre-existing]`, `[downgraded by verifier: …]`,
`[unverified]`), a dropped fact becomes a `dropped by verifier` note in the brief, and the "why"
and the five facts come from the verified list. The priority does not: it is computed before the
Expand Down Expand Up @@ -232,7 +232,7 @@ captain:
```

As with the rest of `.opendrs.yml`, this is read from the base commit, so a PR cannot switch its
own captain brief off. See the [configuration reference](configuration.html) for the full file.
own captain brief off. See the [configuration reference](configuration.md) for the full file.

## 8. What is measured, and what is not

Expand All @@ -245,7 +245,7 @@ own captain brief off. See the [configuration reference](configuration.html) for
split only. That ApacheJIT lift measures whether the file *was buggy*, not whether a reviewer
would have cared; measured against real review comments and fixes instead (the outcome-labels
study, §4 above), it did not clear its own bar and was demoted to `info`-only with a narrower
trigger, as recorded there. Details: [Experiments](experiments.html).
trigger, as recorded there. Details: [Experiments](experiments.md).
- **Contracts and change risk** (`contract.dependency`, `contract.api`, `contract.schema`,
`change.concurrency`, `change.safety_removed`) cleared the same pre-registered bug-lift bar
(≥ 1.2×, ≥ 50 firing test commits) on ApacheJIT's test split (24,690 commits, pooled):
Expand All @@ -255,7 +255,7 @@ own captain brief off. See the [configuration reference](configuration.html) for
(firing on 21 % of test commits — too common to weight `strong` without pushing most PRs to
"careful review", hence `info`). The docs may say these predict bugs; the facts still ship
regardless of the number, as decisions a captain signs off on either way. Details:
[Experiments](experiments.html#captain-fact-lift-on-apachejit).
[Experiments](experiments.md#captain-fact-lift-on-apachejit).
- **Layers** (`layers.edge`, `layers.cycle`) is not shown to predict bugs: `layers.edge` lifted
only 1.12× on a 2,000-commit sample (19 firings), short of the bar. It still ships — a new import
edge is a decision a captain signs off on, not a claim that it causes incidents.
Expand Down
2 changes: 1 addition & 1 deletion docs/cli-and-mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ Every tool also takes `repoDir`, the absolute path to the git repository (any di
|---|---|---|
| `score_diff` | `base` (base ref to diff against; defaults to `origin/HEAD`, or `main` when that is not set), `head` (head ref to score; defaults to `HEAD`) | The Diff Risk Score (0-1, calibrated probability that the change causes an incident) for the changes between the two refs: the score, tier (green/yellow/red), the gate decision and its reasons, the top contributing signals with evidence, signals that could not be computed (unknowns), suggested reviewers, and the LLM judge's rationale when enabled. Use it before opening a pull request, or to decide whether a change should be split. |
| `review_brief` | `base`, `head` (as `score_diff`), `maxTokens` (override the token budget) | The review brief as Markdown text content and structured `ReviewBrief` JSON. Call it before reviewing a diff. |
| `self_check_items` | `base`, `head`, `maxTokens` (as `review_brief`) | The ids (captain facts, reviewFirst hunks, check questions) a pre-PR self-check should fix or explain, as `{ items: [{ id, file, line?, text }] }`. Write verdicts into the head commit's message as described in [docs/llm-reviewers.md](llm-reviewers.html#self-check-before-you-open-the-pr); advisory only, never part of the gate. |
| `self_check_items` | `base`, `head`, `maxTokens` (as `review_brief`) | The ids (captain facts, reviewFirst hunks, check questions) a pre-PR self-check should fix or explain, as `{ items: [{ id, file, line?, text }] }`. Write verdicts into the head commit's message as described in [docs/llm-reviewers.md](llm-reviewers.md#self-check-before-you-open-the-pr); advisory only, never part of the gate. |
| `explain_file_risk` | `path` (repo-relative file path with forward slashes, e.g. `src/auth/session.ts`) | The standing risk profile of one file at HEAD: commit and bug-fix history, number of authors, direct and transitive dependents (blast radius; null when the language is not supported), whether it is on a critical or blocked path, and its top authors. Use it before editing a file you do not know, or to explain a high `blast.dependents` contribution. |
| `list_hotspots` | `top` (how many files to return; defaults to 20) | The files tracked at HEAD ranked by code-at-rest risk: bug-fix commits, churn, dependents and critical paths in the history lookback window (authors shown, not scored), as `{ hotspots, warnings }`, with transitive dependents in the import graph (null when the language is not supported) and critical or blocked path flags. Use it to find where a codebase is most likely to break, or to pick files for reengineering or extra test coverage. |
| `suggest_reviewers` | `paths` (repo-relative file paths the change touches; at least one) | Up to three reviewers (author emails) for the files, ranked by how often they recently changed them in git history. Use it to pick human reviewers for a risky change. |
Expand Down
4 changes: 2 additions & 2 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ description: "Every key in .opendrs.yml with its default: thresholds, auto-appro
| `graph` | The import graph: on or off, depth and file cap |
| `model` | Where `opendrs calibrate` writes the fitted model; priors are used until it exists |
| `brief` | Token budget for the review brief's Markdown |
| `captain` | The [captain brief](captain.html): the module map for the layers detector (auto-detected by default, or a flat depth / literal globs), the change-coupling detector's thresholds, and extra schema/contract globs |
| `captain` | The [captain brief](captain.md): the module map for the layers detector (auto-detected by default, or a flat depth / literal globs), the change-coupling detector's thresholds, and extra schema/contract globs |
| `content` | JS regexes matched against each added line (see [What opendrs refuses](#what-opendrs-refuses)) |
| `freezes` | Windows in which auto-approval is suspended or held to a stricter green (see [What opendrs refuses](#what-opendrs-refuses)) |
| `sources` | Per-source policy for agent, bot and human PRs; every setting only tightens the gate |
Expand Down Expand Up @@ -201,7 +201,7 @@ The default `history.fixPattern` counts any commit subject matching
`history.prior_fixes`. On a repository that uses Conventional Commits heavily, this also counts
routine housekeeping — `fix(deps): update dependencies`, `chore(deps)`, `fix(ci)`, `fix(docs)`,
`fix: typo` — as if it were a real bug fix. The [score-saturation
experiment](experiments.html#score-saturation-fixpattern-and-incident-granularity-on-apachejit)
experiment](experiments.md#score-saturation-fixpattern-and-incident-granularity-on-apachejit)
measured this on `vitejs/vite`: excluding those commits dropped the share matched by
`fixPattern` by 21.6 %. It was not significant pooled across ApacheJIT's mostly pre-Conventional-
Commits Java projects, so it is not a new default — but it is worth setting per repo if you use the
Expand Down
Loading
Loading