From 2c537cc35a4353870dee96e236a8ef85c5f1370c Mon Sep 17 00:00:00 2001 From: Kirk Marple Date: Wed, 18 Nov 2026 16:32:17 -0800 Subject: [PATCH 1/2] Review: a not-met entry whose open finding the check resolves gets its comment; the placeholder points to the body (#638) reviewPost leaves out the not-met comment for an entry whose finding is already open, and now returns each such comment in `relied` with the ids of those findings. The "judge again" step runs after the review is posted; when the finding check resolves every finding an entry relied on, the review stage posts that entry's not-met comment in a review of its own (marked, so it is Pendulum's finding) before the answer list is read, so the answer pass takes it. No not-met entry is left with neither a comment nor an open thread. The placeholder line comment no longer says "no findings" when the review body lists an entry not met (assigned by a decision, or with its finding open): it says there are no new line findings and points to the body. Its marker is unchanged. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_017VqP6hg8ahWzteM9zqi8pU --- docs/PLAN.md | 2 +- plugin/scripts/lib/handoff.mjs | 26 ++++++++++++++++++++++---- plugin/scripts/review.mjs | 13 ++++++++++++- plugin/test/exit.test.mjs | 9 ++++++++- 4 files changed, 43 insertions(+), 7 deletions(-) diff --git a/docs/PLAN.md b/docs/PLAN.md index 58e7da41..173b7d26 100644 --- a/docs/PLAN.md +++ b/docs/PLAN.md @@ -233,7 +233,7 @@ Git LFS: in a repository whose `.gitattributes` uses LFS, every worker and revie Pendulum merges on its own (`auto_merge`, on by default); the gate is CI green on the integrated head, the reviewer's one pass and the merge queue's re-run. The owner reads merged diffs when they choose, and the bench's judge samples them. A defect the reviewer flagged and the worker rebutted stays visible on the merged PR; a repeated class becomes a reviewer-skill change scored on the replay set. -The reviewer is one model call per PR on the top model, and one more for each chunk of commits written after it (below), on a head whose validation is green or still running. A head without the validated marker (handed off by the worker, pushed or marked ready by a person) gets Pendulum's validation in the review stage. With required checks and a reviewer to run, the reviewer starts at the first reading that shows a check run on the head (CI has started) and no required check red, once the PR's mergeable state reads no conflict, and the wait goes on beside it: a head that conflicts, whose checks never start, or whose reading already shows a required check red gets no reviewer pass yet, but its fix passes first, as does every head when the reviewer does not run or validation is the commands alone, and a red that stays red Blocks before any review. A head handed off from the local route is not given its commands again on that first pass. A red the checks show while the reviewer runs, or after it (after triage's rerun and the base-failure check, as any red), gets no fix pass of its own: once the reviewer is done, the answer pass takes it with the reviewer's comments, its failures in the prompt and their whole logs in `$PENDULUM_IN/checks/`, and runs even when the review has no comment. A conflict, a check that never starts or a base failure ends the stage as before, ending a reviewer call still running, and its review is not posted, so the next review stage reviews the head it works on. The validated marker goes on the head the checks were green on; when the PR's head moved meanwhile (a person's push), the stage posts nothing and puts the review label back on for the new head. The reviewer's rerun and the foundation pass finish the review that started. The progress comment shows the overlap: the reviewer's line ends `beside the required checks on `, the pass's own lines meanwhile start `during the reviewer:`, and the checks' end is `required checks on green|red|settled while the reviewer runs`, or, when they outlast the reviewer, `waiting for required checks on after the reviewer`; the scorecard counts each minute once, the overlap as review beside CI (bench/README.md). It gets the Objective text, the diff and the PR body, and it posts review comments tied to lines, each naming the acceptance entry or repository rule it concerns, with assumptions that change behaviour flagged explicitly. It also judges every acceptance entry, met or not, with the line that shows it: Pendulum lists the issue's entries in the prompt, puts the judgments in the review body, names any entry left unjudged, and turns an entry judged not met into a comment the worker must answer. On a split part that leaves work for the next, the prompt says it is part N and not the last: an entry the PR body assigns to a later part is judged deferred and gets no comment, while one the body claims and the diff does not deliver is still a finding. The last part and a one-PR Objective are judged against every entry. Reason: otherwise every earlier part re-raises the entries later parts deliver, and the worker rebuts each one. An entry a maintainer's recorded decision assigns outside the PR (to the owner after the merge, to another issue) is judged not met with that decision's key, and gets no comment: Pendulum keeps the judgment only when the key is exactly one of the stage's trusted decisions (authors with write access), lists the entry in the review body as assigned by that decision, with its author and link, and never counts it met; any other key is a plain not met. Before each reviewer call Pendulum writes `$PENDULUM_IN/decisions.md` with every trusted decision, uncut and under its key, so the decisions the prompt cuts for length are there to read. An entry whose finding from an earlier review is still open on the PR gets no second comment. Reason: a decision the reviewer cannot read or cannot cite turns the entry into a finding every round, which no change on the branch can answer. A call that fails (a crash, a non-zero exit, a cap) or an answer that holds no JSON or judges none of the entries is a missed pass: it reruns once in a fresh session on every harness, on the fallback reviewer where the harness has one. A provider's refusal is not rerun, a subscription's usage limit included (below): the fallback model draws on the same subscription. When neither call produced a review, nothing is posted and the Objective Blocks with how each call ended: no placeholder stands in for a review that did not happen, and Pendulum never merges past the review. A rerun that answered but judged none of the entries is posted, and its placeholder says it is not a clean pass. Entries left unjudged are recorded on the review, not enforced: they do not hold the merge. It has no verdict and no refuse path: a reviewer that can stop a run turns every false positive into a hold. An Objective labelled `pendulum:foundation` (trusted as the harness and model labels are, and outside the plan's record, since it changes only the review) gets a second call after that one, the foundation pass: it checks each shared contract the diff adds or changes against its callers in the repository, and shared test fixtures against isolation, and returns line-tied comments only, which go up in the same review. It runs within the job's `max_usd` like any call, reruns once when it misses, and when it cannot run or never answers nothing is posted and the stage Blocks, as for the first pass. The review that includes it carries a second marker (``): a PR reviewed before the label was added gets the foundation pass alone at its next review stage, and a rebased PR keeps its review only when that marker is there. Reason: a defect in a contract later work builds on (a mapping that folds new error codes into an old one, a fixture that writes the developer's real state) passes a review of the diff alone, and every later Objective inherits it. The worker gets the comments once and addresses or rebuts each; Pendulum pushes, posts the replies, and validation runs again; rebutted findings stay visible on the merged PR. Each finding the answer pass replied to (Pendulum's reviewer's, a named reviewer's, code scanning's) then goes to the finding check (`lib/finding-check.mjs`), before another answer round is spent on it. A reply that says the finding is fixed and names a test that fails without the fix (replies.json's `fixed` and `test`: one of the validation commands exactly as listed, a single simple command, and the arguments that run only that test, at least one of them no option; the answer pass is offered `test` only when there are validation commands) has that test run through the stage's own validation path (`runValidation`: its sandbox and env), each argument shell-quoted, in the checkout itself: first on the PR's head, then, when the commit the finding was raised on (else the head the stage's review covers) is known green (Pendulum's validated marker on it, and no red the stage held there: a command that runs more than the test fails on a red commit for any red it had), on that commit's tree with the head's copy of each argument that names a file put back, after which the checkout is reset to the head and its HEAD and tracked files checked. Each run gets the stage's time left less a reserve for the exit, and none starts with under a minute of it. Failing without the fix and passing with it resolves the finding; failing on the head leaves it open (`test failed`) with no decision call. A test that passes both ways is no evidence, and one that cannot be run so is not checkable (a command that is no validation command or not one simple command, no test argument, a commit not known green, a dependency manifest or lockfile changed since that commit, a checkout not clean at the head, a run that timed out or changed tracked files, too little time): either reply goes to the decision model, its note kept for the Block. Only the named test is run: a reproduction written in a reply's prose is never run. For every other reply the decision model is asked whether the answer resolves it, `fixed` (the change shows it), `rebutted` (the reply names an acceptance entry or repository rule that permits the code), `decided` (a maintainer's recorded decision, not overruled by a later one, speaks to the very entry, rule, file or behaviour the finding concerns and settles it: it waives or changes what the finding asks, assigns that work outside the PR, such as to the owner after the merge, or permits the code; general approval such as "LGTM" or "ship it" settles nothing, and doubt is `unresolved`) or `unresolved`, given the finding (its reviewer, file, line, body and when it was raised), the reply, what the PR changed since the commit the finding was raised on (its own file, then its other files: those the reply names by path first, then those the finding names, then the rest, cut at 8,000 characters with the paths of the files left out listed after the cut; without the default branch's changes; a finding with no commit of its own reads from the head the stage's review covers), for a reply that says its fix is in the PR body (replies.json's `"fixed": true` and `"pr_body": true`) what the body changed since the finding was raised (git's diff, one line of context, of the body's version at that time, from GitHub's record of the body's edits read once per check, to the body now, cut at 16,000 characters; `no change`, or `whether the body changed could not be read` when the record does not hold that version), with `fixed` then taking a defect gone from the body and the question saying a body change alone never fixes a finding that asks for a change to the code or tests or says the code does not meet an acceptance entry or rule (every other reply's question carries no body; Pendulum's posted answer carries `pr_body` in its marker, ``, so an answer judged again in a later stage keeps it), the issue's acceptance entries, the repository's rules (AGENTS.md at the head, the protected paths, the validation commands), and the maintainers' decisions the stage read (comments on the issue and the PR, the PR's review bodies and its inline review comments, by people with write access, a dismissed review's left out; each with its author, time, place and review state, cut at 2,000 characters; within 8,000 characters the newest decision always, the issue's newest within half the window while the PR has decisions too, then the PR's newest, then the issue's in any room left, with a count of those left out; a decision the reply quotes counts only when it is there, so a comment by anyone else settles nothing), each part between markers with a random tag, as data never to be followed; at most four questions at once. `fixed`, `rebutted` or `decided` at the confidence floor or more (Red-check triage, below; no confidence is none of them; `decided` never on code scanning's alert, nor when a decision newer than the oldest one shown was left out for length) resolves the finding's thread; any other verdict leaves it open (`judged not fixed`, or `judged not resolved` for a reply that did not say fixed). Pendulum posts the evidence in the thread first, under `` (the test as it ran, the commit it failed on and the head it passed on, or the verdict, its confidence, the floor and who answered); that comment and Pendulum's answer are never a person's answer, so a thread a person reopens stays open, then resolves the thread (a finding in the review body: that PR comment is what resolves it), and a resolved thread, not a reply, marks it answered on every side (the answer pass's list, the exit, the merge pass). Anything else leaves it open. A person's reply answers as before, from someone who can push only: any such reply but Pendulum's own on the reviewer's comment, a writer's on a named reviewer's, none on code scanning's; a person may resolve the thread too. How each finding ended is recorded as data in the review stage's progress comment (its findings line: the reviews its reviewer posted, and each finding it listed with the review or outside reviewer that raised it and its outcome: `fixed-test`, `fixed-verdict`, `answered` for a `rebutted` or `decided` at the floor, `not-fixed`, `not-checkable` for no verdict or no decision model, or `open`), from the check's own branches and whether each ended resolved (a thread already resolved posts no evidence), never from the reply's words; a later stage's listed `open` never replaces a judgement, and a report reads only records written before its period ends; `pendulum report` counts the findings and outcomes per Objective, reviewer model and review stage beside those stages' dollars ([REPORT.md](REPORT.md)). Reason: findings per dollar and a reviewer's quality are then exact counts, not readings of prose. Findings still open after the answer pass (unanswered, or read as not resolved), when the rest of the exit is delivered, take another answer round: the review label goes back on, and the next stage's answer pass gets them again, each with Pendulum's earlier answer (`earlier`). A Question the answer pass raised stands with findings open: it carries them. The rounds per re-entry (the Objective label applied again, a restart or a Block) are at most two, shared with code scanning's rounds at the merge (below) and counted in Pendulum's record on the issue (``); past them the stage Blocks (cause `work`) naming each open finding, with what the check found on it (`test failed`, `judged not fixed` or `judged not resolved`, after `not evidence` or `not checkable` for a named test that proved nothing), and how to clear it: a fix on the branch, a reply by someone who can push or a resolved thread, or for a finding in the review body that resolved marker by someone who can push. An answer the gateway gave no verdict on (a failed call, a refused key, an answer outside the choices; once the key is refused no further question is asked in the stage) costs no answer round: it goes to no worker pass, the review label goes back on, and the next stage judges it again first (when that stage was started only for this, an answer it then reads as not resolved waits for the answer round its exit takes); the second stage in a row without a verdict Blocks naming the decider's error code (cause `setup` for a refused key, `provider-quota` for a rate limit or spent credits, else `outage`). With no decider (`decision_route` off, or `gateway` with no key) the decision model's part is off: a reply answers its finding as before unless a test it names proves the fix or fails on the head, the progress comment says `finding check: off ()` once per stage, and doctor's `finding check` line warns. The reviewer reads each chunk of work once. The answer pass's and the fix passes' commits are the review's answer and get no review of their own when they do only what the review's findings and the failing checks asked. The answer gate decides it: Pendulum asks the decision model red-check triage uses (the same key, `triage_model`, below) one question, `only_asked` or `more_than_asked`, giving it the diff they wrote since the head the review read, with the default branch's changes left out (the replay below), then each finding the answer pass answered (path, line, body) and the failure lines and output's end of each red the fix passes and the answer pass were given, at most 32 KiB in one question (`ANSWER_INPUT_BYTES`). A diff too large for one question is asked about in parts, each with the findings and failures and each within that size: whole files packed in order, and a file too large alone split into its hunks, packed in order the same way, each part with the file's header; four parts at a time, none started once one is held. Only every part covered covers them, and the first part held is named (`part of … ()`); a single hunk too large for a question is not judged, named in the hold, and no part is asked, even with no decider. Before each question, the first included, the stage's `max_usd` (in known dollars: a price still settling counts nothing) and its wall clock are read: once either is reached, that part and every later one are not judged, each named in the hold, and never covered. The question says a failing check asks only that it pass with the smallest change, so new behaviour or a feature is more than asked even when a check's output mentions it. The diff, the findings and the failures go as data (fenced, or JSON strings), with no worker secret in them. Pendulum records the head they leave in its own comment on the PR (``, edited in place), and the later of that record and its last review's `commit_id` is the head the review covers. Only `only_asked` at the confidence floor (Red-check triage, below) covers them. Anything else holds the PR: a lower or missing confidence, `more_than_asked`, a call that fails, times out or answers outside the two choices, no key, a hunk too large for a question, a diff over 16 MiB (not read whole), a file git diffs as binary (its numstat `-`, or `Binary files … differ`: a text file with one NUL byte is one, and the model would see none of it; no question is asked), or a diff that cannot be told apart. Then the record stays on the head the review read, marked as an answer's (`:answer` after the sha), the review label goes back on, and the next review stage's pass reads those commits. That pass's own answer is gated too: held again, the stage Blocks (cause `work`) naming why, the record staying on the head that pass read, and a person reviews those commits, so the gate never loops. The gate's answer goes to the job log (`answer gate (): → …`, with `(in parts)` after the confidence for a diff asked about in parts). The review goes up on the head the reviewer read (its `commit_id`), which is the head Pendulum's checkout was on when the reviewer started; when GitHub's head is no longer Pendulum's by then (someone else pushed), nothing is posted and the review label goes back on for the new head. A pass on new commits that finds nothing leaves the earlier reviews' open findings to the answer pass. A review stage whose head is that head, or only a rebase of it (the covered head's own change, replayed with `git merge-tree` onto the head's merge base with the default branch, gives exactly the head's tree: a clean rebase, GitHub's update-branch merge, Pendulum's merge of a default branch), runs no reviewer and goes on to validation and the merge. Commits written after it (the work stage's answer to a Question, a maintainer's push, a resolved conflict) get a reviewer pass before the merge, on the diff from that replay to the head, so neither the reviewed code nor the default branch's changes are in it; when the replay conflicts, the pass gets the whole PR. The prompt and the posted review name the head it follows on; the pass judges every acceptance entry as a first pass does, and a foundation Objective's foundation pass runs on the same diff. An answer pass that applied maintainers' decisions (a Question's answer, a comment no pass had read) in new commits does not merge: the stage covers the head before it (the head the review read, marked as an answer's, when the answer gate holds what the stage wrote before it), puts the review label back on, and the next review stage reviews those commits. Reason: the one-pass rule keeps the same code from being reviewed twice, and code written after the review was never reviewed. +The reviewer is one model call per PR on the top model, and one more for each chunk of commits written after it (below), on a head whose validation is green or still running. A head without the validated marker (handed off by the worker, pushed or marked ready by a person) gets Pendulum's validation in the review stage. With required checks and a reviewer to run, the reviewer starts at the first reading that shows a check run on the head (CI has started) and no required check red, once the PR's mergeable state reads no conflict, and the wait goes on beside it: a head that conflicts, whose checks never start, or whose reading already shows a required check red gets no reviewer pass yet, but its fix passes first, as does every head when the reviewer does not run or validation is the commands alone, and a red that stays red Blocks before any review. A head handed off from the local route is not given its commands again on that first pass. A red the checks show while the reviewer runs, or after it (after triage's rerun and the base-failure check, as any red), gets no fix pass of its own: once the reviewer is done, the answer pass takes it with the reviewer's comments, its failures in the prompt and their whole logs in `$PENDULUM_IN/checks/`, and runs even when the review has no comment. A conflict, a check that never starts or a base failure ends the stage as before, ending a reviewer call still running, and its review is not posted, so the next review stage reviews the head it works on. The validated marker goes on the head the checks were green on; when the PR's head moved meanwhile (a person's push), the stage posts nothing and puts the review label back on for the new head. The reviewer's rerun and the foundation pass finish the review that started. The progress comment shows the overlap: the reviewer's line ends `beside the required checks on `, the pass's own lines meanwhile start `during the reviewer:`, and the checks' end is `required checks on green|red|settled while the reviewer runs`, or, when they outlast the reviewer, `waiting for required checks on after the reviewer`; the scorecard counts each minute once, the overlap as review beside CI (bench/README.md). It gets the Objective text, the diff and the PR body, and it posts review comments tied to lines, each naming the acceptance entry or repository rule it concerns, with assumptions that change behaviour flagged explicitly. It also judges every acceptance entry, met or not, with the line that shows it: Pendulum lists the issue's entries in the prompt, puts the judgments in the review body, names any entry left unjudged, and turns an entry judged not met into a comment the worker must answer. On a split part that leaves work for the next, the prompt says it is part N and not the last: an entry the PR body assigns to a later part is judged deferred and gets no comment, while one the body claims and the diff does not deliver is still a finding. The last part and a one-PR Objective are judged against every entry. Reason: otherwise every earlier part re-raises the entries later parts deliver, and the worker rebuts each one. An entry a maintainer's recorded decision assigns outside the PR (to the owner after the merge, to another issue) is judged not met with that decision's key, and gets no comment: Pendulum keeps the judgment only when the key is exactly one of the stage's trusted decisions (authors with write access), lists the entry in the review body as assigned by that decision, with its author and link, and never counts it met; any other key is a plain not met. Before each reviewer call Pendulum writes `$PENDULUM_IN/decisions.md` with every trusted decision, uncut and under its key, so the decisions the prompt cuts for length are there to read. An entry whose finding from an earlier review is still open on the PR gets no second comment; when the stage's finding check then resolves that finding (judging again an answer the last stage got no verdict on, below), the entry's comment goes up after all, in a review of its own, so the answer pass takes it. A review with no line comment beside such an entry, or one a decision assigns, has a placeholder that says there are no new line findings and points to the body. Reason: a decision the reviewer cannot read or cannot cite turns the entry into a finding every round, which no change on the branch can answer. A call that fails (a crash, a non-zero exit, a cap) or an answer that holds no JSON or judges none of the entries is a missed pass: it reruns once in a fresh session on every harness, on the fallback reviewer where the harness has one. A provider's refusal is not rerun, a subscription's usage limit included (below): the fallback model draws on the same subscription. When neither call produced a review, nothing is posted and the Objective Blocks with how each call ended: no placeholder stands in for a review that did not happen, and Pendulum never merges past the review. A rerun that answered but judged none of the entries is posted, and its placeholder says it is not a clean pass. Entries left unjudged are recorded on the review, not enforced: they do not hold the merge. It has no verdict and no refuse path: a reviewer that can stop a run turns every false positive into a hold. An Objective labelled `pendulum:foundation` (trusted as the harness and model labels are, and outside the plan's record, since it changes only the review) gets a second call after that one, the foundation pass: it checks each shared contract the diff adds or changes against its callers in the repository, and shared test fixtures against isolation, and returns line-tied comments only, which go up in the same review. It runs within the job's `max_usd` like any call, reruns once when it misses, and when it cannot run or never answers nothing is posted and the stage Blocks, as for the first pass. The review that includes it carries a second marker (``): a PR reviewed before the label was added gets the foundation pass alone at its next review stage, and a rebased PR keeps its review only when that marker is there. Reason: a defect in a contract later work builds on (a mapping that folds new error codes into an old one, a fixture that writes the developer's real state) passes a review of the diff alone, and every later Objective inherits it. The worker gets the comments once and addresses or rebuts each; Pendulum pushes, posts the replies, and validation runs again; rebutted findings stay visible on the merged PR. Each finding the answer pass replied to (Pendulum's reviewer's, a named reviewer's, code scanning's) then goes to the finding check (`lib/finding-check.mjs`), before another answer round is spent on it. A reply that says the finding is fixed and names a test that fails without the fix (replies.json's `fixed` and `test`: one of the validation commands exactly as listed, a single simple command, and the arguments that run only that test, at least one of them no option; the answer pass is offered `test` only when there are validation commands) has that test run through the stage's own validation path (`runValidation`: its sandbox and env), each argument shell-quoted, in the checkout itself: first on the PR's head, then, when the commit the finding was raised on (else the head the stage's review covers) is known green (Pendulum's validated marker on it, and no red the stage held there: a command that runs more than the test fails on a red commit for any red it had), on that commit's tree with the head's copy of each argument that names a file put back, after which the checkout is reset to the head and its HEAD and tracked files checked. Each run gets the stage's time left less a reserve for the exit, and none starts with under a minute of it. Failing without the fix and passing with it resolves the finding; failing on the head leaves it open (`test failed`) with no decision call. A test that passes both ways is no evidence, and one that cannot be run so is not checkable (a command that is no validation command or not one simple command, no test argument, a commit not known green, a dependency manifest or lockfile changed since that commit, a checkout not clean at the head, a run that timed out or changed tracked files, too little time): either reply goes to the decision model, its note kept for the Block. Only the named test is run: a reproduction written in a reply's prose is never run. For every other reply the decision model is asked whether the answer resolves it, `fixed` (the change shows it), `rebutted` (the reply names an acceptance entry or repository rule that permits the code), `decided` (a maintainer's recorded decision, not overruled by a later one, speaks to the very entry, rule, file or behaviour the finding concerns and settles it: it waives or changes what the finding asks, assigns that work outside the PR, such as to the owner after the merge, or permits the code; general approval such as "LGTM" or "ship it" settles nothing, and doubt is `unresolved`) or `unresolved`, given the finding (its reviewer, file, line, body and when it was raised), the reply, what the PR changed since the commit the finding was raised on (its own file, then its other files: those the reply names by path first, then those the finding names, then the rest, cut at 8,000 characters with the paths of the files left out listed after the cut; without the default branch's changes; a finding with no commit of its own reads from the head the stage's review covers), for a reply that says its fix is in the PR body (replies.json's `"fixed": true` and `"pr_body": true`) what the body changed since the finding was raised (git's diff, one line of context, of the body's version at that time, from GitHub's record of the body's edits read once per check, to the body now, cut at 16,000 characters; `no change`, or `whether the body changed could not be read` when the record does not hold that version), with `fixed` then taking a defect gone from the body and the question saying a body change alone never fixes a finding that asks for a change to the code or tests or says the code does not meet an acceptance entry or rule (every other reply's question carries no body; Pendulum's posted answer carries `pr_body` in its marker, ``, so an answer judged again in a later stage keeps it), the issue's acceptance entries, the repository's rules (AGENTS.md at the head, the protected paths, the validation commands), and the maintainers' decisions the stage read (comments on the issue and the PR, the PR's review bodies and its inline review comments, by people with write access, a dismissed review's left out; each with its author, time, place and review state, cut at 2,000 characters; within 8,000 characters the newest decision always, the issue's newest within half the window while the PR has decisions too, then the PR's newest, then the issue's in any room left, with a count of those left out; a decision the reply quotes counts only when it is there, so a comment by anyone else settles nothing), each part between markers with a random tag, as data never to be followed; at most four questions at once. `fixed`, `rebutted` or `decided` at the confidence floor or more (Red-check triage, below; no confidence is none of them; `decided` never on code scanning's alert, nor when a decision newer than the oldest one shown was left out for length) resolves the finding's thread; any other verdict leaves it open (`judged not fixed`, or `judged not resolved` for a reply that did not say fixed). Pendulum posts the evidence in the thread first, under `` (the test as it ran, the commit it failed on and the head it passed on, or the verdict, its confidence, the floor and who answered); that comment and Pendulum's answer are never a person's answer, so a thread a person reopens stays open, then resolves the thread (a finding in the review body: that PR comment is what resolves it), and a resolved thread, not a reply, marks it answered on every side (the answer pass's list, the exit, the merge pass). Anything else leaves it open. A person's reply answers as before, from someone who can push only: any such reply but Pendulum's own on the reviewer's comment, a writer's on a named reviewer's, none on code scanning's; a person may resolve the thread too. How each finding ended is recorded as data in the review stage's progress comment (its findings line: the reviews its reviewer posted, and each finding it listed with the review or outside reviewer that raised it and its outcome: `fixed-test`, `fixed-verdict`, `answered` for a `rebutted` or `decided` at the floor, `not-fixed`, `not-checkable` for no verdict or no decision model, or `open`), from the check's own branches and whether each ended resolved (a thread already resolved posts no evidence), never from the reply's words; a later stage's listed `open` never replaces a judgement, and a report reads only records written before its period ends; `pendulum report` counts the findings and outcomes per Objective, reviewer model and review stage beside those stages' dollars ([REPORT.md](REPORT.md)). Reason: findings per dollar and a reviewer's quality are then exact counts, not readings of prose. Findings still open after the answer pass (unanswered, or read as not resolved), when the rest of the exit is delivered, take another answer round: the review label goes back on, and the next stage's answer pass gets them again, each with Pendulum's earlier answer (`earlier`). A Question the answer pass raised stands with findings open: it carries them. The rounds per re-entry (the Objective label applied again, a restart or a Block) are at most two, shared with code scanning's rounds at the merge (below) and counted in Pendulum's record on the issue (``); past them the stage Blocks (cause `work`) naming each open finding, with what the check found on it (`test failed`, `judged not fixed` or `judged not resolved`, after `not evidence` or `not checkable` for a named test that proved nothing), and how to clear it: a fix on the branch, a reply by someone who can push or a resolved thread, or for a finding in the review body that resolved marker by someone who can push. An answer the gateway gave no verdict on (a failed call, a refused key, an answer outside the choices; once the key is refused no further question is asked in the stage) costs no answer round: it goes to no worker pass, the review label goes back on, and the next stage judges it again first (when that stage was started only for this, an answer it then reads as not resolved waits for the answer round its exit takes); the second stage in a row without a verdict Blocks naming the decider's error code (cause `setup` for a refused key, `provider-quota` for a rate limit or spent credits, else `outage`). With no decider (`decision_route` off, or `gateway` with no key) the decision model's part is off: a reply answers its finding as before unless a test it names proves the fix or fails on the head, the progress comment says `finding check: off ()` once per stage, and doctor's `finding check` line warns. The reviewer reads each chunk of work once. The answer pass's and the fix passes' commits are the review's answer and get no review of their own when they do only what the review's findings and the failing checks asked. The answer gate decides it: Pendulum asks the decision model red-check triage uses (the same key, `triage_model`, below) one question, `only_asked` or `more_than_asked`, giving it the diff they wrote since the head the review read, with the default branch's changes left out (the replay below), then each finding the answer pass answered (path, line, body) and the failure lines and output's end of each red the fix passes and the answer pass were given, at most 32 KiB in one question (`ANSWER_INPUT_BYTES`). A diff too large for one question is asked about in parts, each with the findings and failures and each within that size: whole files packed in order, and a file too large alone split into its hunks, packed in order the same way, each part with the file's header; four parts at a time, none started once one is held. Only every part covered covers them, and the first part held is named (`part of … ()`); a single hunk too large for a question is not judged, named in the hold, and no part is asked, even with no decider. Before each question, the first included, the stage's `max_usd` (in known dollars: a price still settling counts nothing) and its wall clock are read: once either is reached, that part and every later one are not judged, each named in the hold, and never covered. The question says a failing check asks only that it pass with the smallest change, so new behaviour or a feature is more than asked even when a check's output mentions it. The diff, the findings and the failures go as data (fenced, or JSON strings), with no worker secret in them. Pendulum records the head they leave in its own comment on the PR (``, edited in place), and the later of that record and its last review's `commit_id` is the head the review covers. Only `only_asked` at the confidence floor (Red-check triage, below) covers them. Anything else holds the PR: a lower or missing confidence, `more_than_asked`, a call that fails, times out or answers outside the two choices, no key, a hunk too large for a question, a diff over 16 MiB (not read whole), a file git diffs as binary (its numstat `-`, or `Binary files … differ`: a text file with one NUL byte is one, and the model would see none of it; no question is asked), or a diff that cannot be told apart. Then the record stays on the head the review read, marked as an answer's (`:answer` after the sha), the review label goes back on, and the next review stage's pass reads those commits. That pass's own answer is gated too: held again, the stage Blocks (cause `work`) naming why, the record staying on the head that pass read, and a person reviews those commits, so the gate never loops. The gate's answer goes to the job log (`answer gate (): → …`, with `(in parts)` after the confidence for a diff asked about in parts). The review goes up on the head the reviewer read (its `commit_id`), which is the head Pendulum's checkout was on when the reviewer started; when GitHub's head is no longer Pendulum's by then (someone else pushed), nothing is posted and the review label goes back on for the new head. A pass on new commits that finds nothing leaves the earlier reviews' open findings to the answer pass. A review stage whose head is that head, or only a rebase of it (the covered head's own change, replayed with `git merge-tree` onto the head's merge base with the default branch, gives exactly the head's tree: a clean rebase, GitHub's update-branch merge, Pendulum's merge of a default branch), runs no reviewer and goes on to validation and the merge. Commits written after it (the work stage's answer to a Question, a maintainer's push, a resolved conflict) get a reviewer pass before the merge, on the diff from that replay to the head, so neither the reviewed code nor the default branch's changes are in it; when the replay conflicts, the pass gets the whole PR. The prompt and the posted review name the head it follows on; the pass judges every acceptance entry as a first pass does, and a foundation Objective's foundation pass runs on the same diff. An answer pass that applied maintainers' decisions (a Question's answer, a comment no pass had read) in new commits does not merge: the stage covers the head before it (the head the review read, marked as an answer's, when the answer gate holds what the stage wrote before it), puts the review label back on, and the next review stage reviews those commits. Reason: the one-pass rule keeps the same code from being reviewed twice, and code written after the review was never reviewed. A repository may also name third-party reviewers, reviewer Apps outside Pendulum, by login (`third_party_reviewers`, such as `greptile-apps[bot]`; `lib/third-party-review.mjs`). Their review is part of the PR's review. After Pendulum's reviewer posts, the review stage reads the PR's reviews once for each named reviewer's submitted review whose `commit_id` is the head Pendulum's reviewer read. While one is missing, the stage does not wait in place: it records the wait on the PR (a comment, marker ``, counted only when Pendulum or someone who can push wrote it, since a local loop writes as its developer; the bound the dispatcher, `serve` and `run` wake the review at is its `minutes`, the `third_party_wait_minutes` setting of the stage that wrote it (20 by default, taken as at most 24 hours whatever the record says), from GitHub's creation time of that comment; the review stage that resumes judges the wait by its own setting), takes the `pendulum:review` label off, moves the Objective to In review when it is not there (a Question a person answered by marking the PR ready still carries `pendulum:question`) and ends with exit `waiting`, holding no local slot, runner or sandbox. The dispatcher's tick, on every route (the Action's, the sandbox's, `serve`'s), reads that record from the PR comments it already reads for send-backs. While the bound is ahead and the review is missing, it leaves the PR as it is: no merge pass, no branch update, no send-back; when it cannot read the PR's comments and third-party reviewers are named, it leaves the PR alone that tick too. Once that review of the head exists, or the bound has passed, or the PR's head is no longer the one the record names (a maintainer's commit, GitHub's update-branch), it puts `pendulum:review` back on, which starts the review stage on that route as any review label does. The reviews are read at most once a tick, and in `serve` at most once a minute per PR. An App's review submitted on a Pendulum PR that is not Pendulum's own ticks the dispatcher (`pull_request_review`, its reviewer of type `Bot`; a person's review ticks nothing), and on a public repository the bound's end is a `third-party` wake; on a private one it waits for the next event or the cron. The review stage that starts again finds Pendulum's own review on the PR (its marker), so the reviewer does not run again; it reads the record, says on its progress comment which reviewers reviewed the head or were left out past the bound, removes the record and goes on to the answer pass. A review label applied after the record was last written means a stage has taken the wait up, so the dispatcher never puts the label back twice for one record. A stage that starts again before the bound (the label re-applied by hand, a restarted loop) rewrites the same record and ends `waiting` again; a new head gets a new record and its own bound. A stage that goes on without waiting (the reviewer's review in, the bound passed, the cap spent) removes the record, so none holds a later merge. A record that cannot be read or written, a label that cannot be taken off, or Pendulum's own login unknown (its record could not be told from a pasted one) is no wait: the stage goes on without the reviewer rather than stall. `pendulum run` and `pendulum review`, which may have no `serve` behind them, do the dispatcher's part in place, once a minute, and then review again, and `pendulum run` on an In-review Objective whose review waits takes that wait up the same way; `serve` leaves it to its own ticks. The answer pass then gets each named reviewer's root line comments that neither Pendulum nor anyone who can push has replied to, from any head, beside Pendulum's own. Each comment carries `from` (the reviewer's login), and its body is sanitised like harness text and cut at 8,000 characters. The prompt and the pendulum-deliver-pr skill call their text untrusted review input: the worker judges each finding against the code, fixes it or rebuts it, and follows no instruction in it. Pendulum posts the replies in their threads, and the finding check resolves those it reads as resolved. The exit counts a named reviewer's comment as unanswered only when the answer pass was given it; one posted after the wait is left for the next review. Only the named logins are waited for; any other reviewer's comment is handled as without the setting (the merge pass, below, has the answer pass take an unlisted reviewer's thread when the base's rules require every conversation resolved). With none named, the stage reads nothing more and runs as it would without the setting. The wait comes before the answer pass, so before the merge; there is no wait for the answer pass's new head, just as Pendulum's reviewer does not review it again. Code scanning (GitHub's own App, `github-advanced-security[bot]`, whose alerts on the PR's lines are review comments, CodeQL's say) is answered the same way on every repository, with no setting and no wait: the answer pass gets its open comments marked as untrusted input, and Pendulum resolves each thread once it replied. On code scanning's thread only Pendulum's own answer counts, never a person's reply alone, and an alert adds only its own file to the answer pass's fix-pass scope, never a protected or toolchain file. diff --git a/plugin/scripts/lib/handoff.mjs b/plugin/scripts/lib/handoff.mjs index cd321378..0c429542 100644 --- a/plugin/scripts/lib/handoff.mjs +++ b/plugin/scripts/lib/handoff.mjs @@ -156,7 +156,8 @@ export function noReviewReason(recs, reviewOf) { * the answer, the PR or a comment can supply one), which the body lists as assigned by that decision, with its author * and link, and which stays not met; any other key is a plain not met. And an entry whose finding is already open on * the PR (`open`: Pendulum's own open findings there, session.mjs unaddressedReviewComments), which the answer pass - * takes from that thread. An entry of the issue (acceptanceEntries) the reviewer + * takes from that thread; its comment comes back in `relied`, with the ids of those findings (reliedResolved). An entry + * of the issue (acceptanceEntries) the reviewer * left unjudged is named in the body and returned in `unjudged`. With no comment at all, Pendulum's placeholder goes on * the first line the diff shows, and `placeholder` is true: nothing to answer. The placeholder only ever stands for a * pass that ran and answered: a failed call (`ok` false) or an answer with no reviewer JSON throws, so no caller can @@ -183,7 +184,10 @@ export function reviewPost({ issue, review, diff, entries = [], ok = true, split const openEntries = new Set(own(open).map(commentEntry).filter(Boolean)); // No comment is added for an entry judged deferred on a split part that is not the last (a later part delivers it), // for one a trusted decision assigns elsewhere, or for one whose finding is already open on the PR. - const unmet = judged.filter((a) => !a.met && !(splitPart && a.deferred) && !assignedBy(a) && !commented.has(entryKey(a.entry)) && !openEntries.has(entryKey(a.entry))).map((a) => ({ path: a.path ?? undefined, line: a.line, body: `acceptance ${a.entry.replace(/^\s*acceptance\s+/i, "")}: not met.${a.evidence ? ` ${a.evidence}` : ""}` })); + const owed = judged.filter((a) => !a.met && !(splitPart && a.deferred) && !assignedBy(a) && !commented.has(entryKey(a.entry))); + const notMet = (a) => ({ path: a.path ?? undefined, line: a.line, body: `acceptance ${a.entry.replace(/^\s*acceptance\s+/i, "")}: not met.${a.evidence ? ` ${a.evidence}` : ""}` }); + const unmet = owed.filter((a) => !openEntries.has(entryKey(a.entry))).map(notMet); + const relied = owed.filter((a) => openEntries.has(entryKey(a.entry))).map((a) => ({ ids: own(open).filter((c) => commentEntry(c) === entryKey(a.entry)).map((c) => String(c.id)), comment: notMet(a) })); let comments = snapComments( [...raw, ...unmet].map((c) => ({ path: c.path, line: c.line, body: typeof c.body === "string" ? sanitize(c.body) : c.body })), diff, @@ -196,10 +200,13 @@ export function reviewPost({ issue, review, diff, entries = [], ok = true, split const first = Object.entries(diffLines(diff)).find(([, v]) => v.lines.size); placeholder = true; // A pass that judged no entry (reviewMissed) is not a clean pass: the placeholder says so. + // An entry not met with no comment (assigned by a decision, or its finding already open) is in the body. const said = review && judgedNone(review, entries) ? "the reviewer judged none of the acceptance entries; this is not a clean pass." - : "no findings; the pass ran and returned nothing line-tied."; + : judged.some((a) => !a.met && !(splitPart && a.deferred)) + ? "no new line findings; the review body lists the acceptance entries not met." + : "no findings; the pass ran and returned nothing line-tied."; comments = [{ path: first?.[0] ?? null, line: first ? (first[1].first ?? Math.min(...first[1].lines)) : null, side: "RIGHT", body: `${PLACEHOLDER_MARKER}\nPendulum reviewer: ${said}` }]; } const checked = "checked the shared contracts the diff changes against their callers"; @@ -213,9 +220,20 @@ export function reviewPost({ issue, review, diff, entries = [], ok = true, split lines.push("", judged.length ? "Acceptance as the reviewer judged it:" : "The reviewer judged no acceptance entry.", ...judged.slice(0, 60).map((a) => judgedLine(a, { splitPart, by: assignedBy(a), open: !a.met && !commented.has(entryKey(a.entry)) && openEntries.has(entryKey(a.entry)) }))); if (unjudged.length) lines.push("", `Not judged: ${unjudged.map((id) => sanitize(id)).join(", ")}.`); } - return { body: lines.join("\n"), comments, placeholder, unjudged }; + return { body: lines.join("\n"), comments, placeholder, unjudged, relied }; } +/** + * The not-met comments reviewPost left out for an open finding (`relied`) whose findings the finding check has since + * resolved, every one (`resolved`: ids as strings), sanitised and fitted to the diff: review.mjs posts them after all, + * so no entry judged not met is left with neither a comment nor an open finding. + */ +export const reliedResolved = (relied, resolved, diff) => + snapComments( + (relied || []).filter((r) => r.ids.length && r.ids.every((id) => resolved.has(id))).map((r) => ({ ...r.comment, body: sanitize(r.comment.body) })), + diff, + ); + /** * What Pendulum's reviews on a PR (`reviews`, GitHub's list) already cover: `reviewed`, one carries its review marker (the * first pass ran); `foundation`, one also carries the foundation marker (the foundation pass ran, with the first pass or diff --git a/plugin/scripts/review.mjs b/plugin/scripts/review.mjs index 7070052c..350551b0 100644 --- a/plugin/scripts/review.mjs +++ b/plugin/scripts/review.mjs @@ -36,7 +36,7 @@ import { accountGate, authorGate, choiceGate, planGate } from "./lib/objective-t import { stageStart } from "./lib/objective-cap.mjs"; import { ranWith } from "./lib/choice.mjs"; import { credentialChecks } from "./doctor.mjs"; -import { exitFromOut, handoverChecker, handoverPosted, isSplitPart, noReviewReason, openHandoff, partsRecord, postNote, postReplies, publishPr, reviewAnswer, reviewCallEnd, reviewCallTimeout, reviewMissed, reviewPost, reviewRerun, reviewRetry, reviewsCover, splitPartOf } from "./lib/handoff.mjs"; +import { exitFromOut, handoverChecker, handoverPosted, isSplitPart, noReviewReason, openHandoff, partsRecord, postNote, postReplies, publishPr, reliedResolved, reviewAnswer, reviewCallEnd, reviewCallTimeout, reviewMissed, reviewPost, reviewRerun, reviewRetry, reviewsCover, splitPartOf } from "./lib/handoff.mjs"; import { TRIAGE_MODEL, confidenceFloor, openTriage, triageKey } from "./lib/triage.mjs"; import { deciderKind, openDecider, settleDecisionCosts } from "./lib/decider.mjs"; import { flushDecisionSpend } from "./lib/decision-spend.mjs"; @@ -1916,6 +1916,8 @@ async function main() { const unjudged = []; // What the check found on each finding it left open (a failed test, a no, a test it could not run), for the exit (#516). const checkFound = []; + // The findings it resolved, by id. + const resolvedNow = new Set(); const findingRules = async (head) => { const agents = head ? await outbox.git(["show", `${head}:AGENTS.md`], { maxBytes: 64 * 1024 }).catch(() => null) : null; return [agents?.code === 0 ? `AGENTS.md:\n${agents.stdout}` : "", `Protected paths (no pass may change them): ${protectedList.join(", ")}`, inputs.validation.length ? `Validation commands: ${inputs.validation.join("; ")}` : ""].filter(Boolean).join("\n\n"); @@ -1931,6 +1933,7 @@ async function main() { const r = await checkAnswers({ repo, issue, prNumber: pr.number, findings, repliesText, check: findingCheck, runTest, ...findingReads({ git: outbox.git, base: outbox.base, head, repo, prNumber: pr.number, token }), since: reviewedAt, issueBody: issueData.body || "", rules: await findingRules(head), decisions, opts: token, progress }).catch((e) => (log(`finding check: ${String(e?.message || e).slice(0, 160)}; the findings stay open`), { unjudged: [], found: [] })); unjudged.push(...r.unjudged); checkFound.push(...(r.found || [])); + for (const id of r.resolved || []) resolvedNow.add(String(id)); }; // Answers the last stage's check got no verdict on are judged again first, with no worker pass for them (a gateway // failure costs no answer round); one still without a verdict waits for the exit (session.mjs openFindingsExit). @@ -1945,6 +1948,14 @@ async function main() { if (again.length) { await progress(`finding check: judging again ${again.length} answer(s) the last stage got no verdict on`, { step: "check", again: again.length }); await judgeAnswers(again, againReplies(again)); + // A not-met entry this stage's review left without a comment for an open finding the check has just resolved gets + // its comment after all, in a review of its own, so the answer pass takes it. + const raised = reliedResolved(post?.relied, resolvedNow, diff); + if (raised.length) { + const posted = await postReview(repo, pr.number, { body: `${marker(issue, "review")}\nPendulum reviewer: ${raised.length} comment(s) for the acceptance entries this stage's review judged not met beside an open finding, which the finding check has since resolved.`, comments: raised, commitId: read.head }, token); + await progress.findings({ pr: pr.number, reviews: [posted.review?.id] }); + log(`posted ${raised.length} not-met comment(s) whose open finding the finding check resolved`); + } } const held = new Set([...unjudged.map((u) => String(u.id)), ...(lastCheck.judgeOnly ? again.map((c) => String(c.id)) : [])]); const toAnswer = answerList(again.length ? await unaddressedReviewComments(repo, pr.number, reviewerLogin, { ...token, reviewIds: await pendulumReviewIds(), thirdParty, unlisted, decided }) : listedOpen, { placeholder, first: scope.first }).filter((c) => !held.has(String(c.id))); diff --git a/plugin/test/exit.test.mjs b/plugin/test/exit.test.mjs index a5dbf170..bc8eff10 100644 --- a/plugin/test/exit.test.mjs +++ b/plugin/test/exit.test.mjs @@ -6,7 +6,7 @@ import { test } from "node:test"; import { appJwt, normalizePem } from "../scripts/lib/app-token.mjs"; import { input, parseArgs, resolveInputs } from "../scripts/lib/inputs.mjs"; import { EXIT_PROMPT, acceptanceEntries, capReached, commandPaths, decideExit, deniedHostsFix, fenceDecidable, isHostName, diffLines, entryKey, fixScopeReason, fixScopeRefused, isPlaceholder, marker, namesPath, parseExitRecord, parseFindings, parseReview, protectedPaths, setupPaths, snapComments, pushRefusal, toolchainFile, workflowScopeRefused, workspaceDeclarations } from "../scripts/lib/logic.mjs"; -import { noReviewReason, reviewMissed, reviewAnswer, reviewCallEnd, reviewCallTimeout, reviewPost, reviewProduced, reviewRetry } from "../scripts/lib/handoff.mjs"; +import { noReviewReason, reliedResolved, reviewMissed, reviewAnswer, reviewCallEnd, reviewCallTimeout, reviewPost, reviewProduced, reviewRetry } from "../scripts/lib/handoff.mjs"; import { exitRecord, scopeRefusal } from "../scripts/lib/session.mjs"; const ok = { ok: true }; @@ -216,6 +216,13 @@ test("#634: an entry whose finding is already open on the PR gets no second not- assert.deepEqual(post.comments.map((c) => c.body), ["acceptance B: not met."]); }); +test("#638: a not-met entry left without a comment for an open finding gets it once the finding check resolves that finding", () => { + const diff = readFileSync(join(import.meta.dirname, "fixtures", "sample.diff"), "utf8"); + const review = { comments: [], acceptance: [{ entry: "A", met: false, path: "src/a.ts", line: 3, evidence: "" }] }; + const post = reviewPost({ issue: 7, review, diff, entries: [{ id: "A", text: "a" }], open: [{ id: 5, path: "src/a.ts", line: 3, body: "acceptance A: not met. the limit is gone" }] }); + assert.deepEqual(reliedResolved(post.relied, new Set(["5"]), diff).map((c) => c.body), ["acceptance A: not met."]); +}); + test("review: a reviewer call always has a wall-clock cap; past the stage's deadline none starts (#2660)", () => { // exec treats a timeout of 0 as no timer: a call started with nothing left would run until the job itself is cut. assert.equal(reviewCallTimeout(0), null); From c431e479296b9bfad96d7058921f5c4da9dc9639 Mon Sep 17 00:00:00 2001 From: Kirk Marple Date: Wed, 18 Nov 2026 16:41:12 -0800 Subject: [PATCH 2/2] Review: findings about to be judged again hold back no not-met comment (#638) Review of #643: the `relied` list lived only in the stage's memory, so a stage that ended between the review post and "judge again" (thirdPartyGate's `waiting` exit, a stop, an outage, a throw from the extra post) lost it, and a not-met entry was again left with neither a comment nor an open thread. Before reviewPost, the stage now reads the answer-rounds record and drops from the open findings every one whose answer the last stage's check got no verdict on (checkState `unjudged`): those are judged again after the review goes up and may be resolved then. Their entries get their not-met comment in the main review; when the check leaves the finding open, the entry has two open threads. With the record unread (not an outage), no open finding holds an entry back. The `relied` list, reliedResolved and the extra review are gone. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_017VqP6hg8ahWzteM9zqi8pU --- docs/PLAN.md | 2 +- plugin/scripts/lib/handoff.mjs | 21 +++------------------ plugin/scripts/review.mjs | 33 ++++++++++++++++++++------------- plugin/test/exit.test.mjs | 10 ++++++---- 4 files changed, 30 insertions(+), 36 deletions(-) diff --git a/docs/PLAN.md b/docs/PLAN.md index 173b7d26..a5cdeeeb 100644 --- a/docs/PLAN.md +++ b/docs/PLAN.md @@ -233,7 +233,7 @@ Git LFS: in a repository whose `.gitattributes` uses LFS, every worker and revie Pendulum merges on its own (`auto_merge`, on by default); the gate is CI green on the integrated head, the reviewer's one pass and the merge queue's re-run. The owner reads merged diffs when they choose, and the bench's judge samples them. A defect the reviewer flagged and the worker rebutted stays visible on the merged PR; a repeated class becomes a reviewer-skill change scored on the replay set. -The reviewer is one model call per PR on the top model, and one more for each chunk of commits written after it (below), on a head whose validation is green or still running. A head without the validated marker (handed off by the worker, pushed or marked ready by a person) gets Pendulum's validation in the review stage. With required checks and a reviewer to run, the reviewer starts at the first reading that shows a check run on the head (CI has started) and no required check red, once the PR's mergeable state reads no conflict, and the wait goes on beside it: a head that conflicts, whose checks never start, or whose reading already shows a required check red gets no reviewer pass yet, but its fix passes first, as does every head when the reviewer does not run or validation is the commands alone, and a red that stays red Blocks before any review. A head handed off from the local route is not given its commands again on that first pass. A red the checks show while the reviewer runs, or after it (after triage's rerun and the base-failure check, as any red), gets no fix pass of its own: once the reviewer is done, the answer pass takes it with the reviewer's comments, its failures in the prompt and their whole logs in `$PENDULUM_IN/checks/`, and runs even when the review has no comment. A conflict, a check that never starts or a base failure ends the stage as before, ending a reviewer call still running, and its review is not posted, so the next review stage reviews the head it works on. The validated marker goes on the head the checks were green on; when the PR's head moved meanwhile (a person's push), the stage posts nothing and puts the review label back on for the new head. The reviewer's rerun and the foundation pass finish the review that started. The progress comment shows the overlap: the reviewer's line ends `beside the required checks on `, the pass's own lines meanwhile start `during the reviewer:`, and the checks' end is `required checks on green|red|settled while the reviewer runs`, or, when they outlast the reviewer, `waiting for required checks on after the reviewer`; the scorecard counts each minute once, the overlap as review beside CI (bench/README.md). It gets the Objective text, the diff and the PR body, and it posts review comments tied to lines, each naming the acceptance entry or repository rule it concerns, with assumptions that change behaviour flagged explicitly. It also judges every acceptance entry, met or not, with the line that shows it: Pendulum lists the issue's entries in the prompt, puts the judgments in the review body, names any entry left unjudged, and turns an entry judged not met into a comment the worker must answer. On a split part that leaves work for the next, the prompt says it is part N and not the last: an entry the PR body assigns to a later part is judged deferred and gets no comment, while one the body claims and the diff does not deliver is still a finding. The last part and a one-PR Objective are judged against every entry. Reason: otherwise every earlier part re-raises the entries later parts deliver, and the worker rebuts each one. An entry a maintainer's recorded decision assigns outside the PR (to the owner after the merge, to another issue) is judged not met with that decision's key, and gets no comment: Pendulum keeps the judgment only when the key is exactly one of the stage's trusted decisions (authors with write access), lists the entry in the review body as assigned by that decision, with its author and link, and never counts it met; any other key is a plain not met. Before each reviewer call Pendulum writes `$PENDULUM_IN/decisions.md` with every trusted decision, uncut and under its key, so the decisions the prompt cuts for length are there to read. An entry whose finding from an earlier review is still open on the PR gets no second comment; when the stage's finding check then resolves that finding (judging again an answer the last stage got no verdict on, below), the entry's comment goes up after all, in a review of its own, so the answer pass takes it. A review with no line comment beside such an entry, or one a decision assigns, has a placeholder that says there are no new line findings and points to the body. Reason: a decision the reviewer cannot read or cannot cite turns the entry into a finding every round, which no change on the branch can answer. A call that fails (a crash, a non-zero exit, a cap) or an answer that holds no JSON or judges none of the entries is a missed pass: it reruns once in a fresh session on every harness, on the fallback reviewer where the harness has one. A provider's refusal is not rerun, a subscription's usage limit included (below): the fallback model draws on the same subscription. When neither call produced a review, nothing is posted and the Objective Blocks with how each call ended: no placeholder stands in for a review that did not happen, and Pendulum never merges past the review. A rerun that answered but judged none of the entries is posted, and its placeholder says it is not a clean pass. Entries left unjudged are recorded on the review, not enforced: they do not hold the merge. It has no verdict and no refuse path: a reviewer that can stop a run turns every false positive into a hold. An Objective labelled `pendulum:foundation` (trusted as the harness and model labels are, and outside the plan's record, since it changes only the review) gets a second call after that one, the foundation pass: it checks each shared contract the diff adds or changes against its callers in the repository, and shared test fixtures against isolation, and returns line-tied comments only, which go up in the same review. It runs within the job's `max_usd` like any call, reruns once when it misses, and when it cannot run or never answers nothing is posted and the stage Blocks, as for the first pass. The review that includes it carries a second marker (``): a PR reviewed before the label was added gets the foundation pass alone at its next review stage, and a rebased PR keeps its review only when that marker is there. Reason: a defect in a contract later work builds on (a mapping that folds new error codes into an old one, a fixture that writes the developer's real state) passes a review of the diff alone, and every later Objective inherits it. The worker gets the comments once and addresses or rebuts each; Pendulum pushes, posts the replies, and validation runs again; rebutted findings stay visible on the merged PR. Each finding the answer pass replied to (Pendulum's reviewer's, a named reviewer's, code scanning's) then goes to the finding check (`lib/finding-check.mjs`), before another answer round is spent on it. A reply that says the finding is fixed and names a test that fails without the fix (replies.json's `fixed` and `test`: one of the validation commands exactly as listed, a single simple command, and the arguments that run only that test, at least one of them no option; the answer pass is offered `test` only when there are validation commands) has that test run through the stage's own validation path (`runValidation`: its sandbox and env), each argument shell-quoted, in the checkout itself: first on the PR's head, then, when the commit the finding was raised on (else the head the stage's review covers) is known green (Pendulum's validated marker on it, and no red the stage held there: a command that runs more than the test fails on a red commit for any red it had), on that commit's tree with the head's copy of each argument that names a file put back, after which the checkout is reset to the head and its HEAD and tracked files checked. Each run gets the stage's time left less a reserve for the exit, and none starts with under a minute of it. Failing without the fix and passing with it resolves the finding; failing on the head leaves it open (`test failed`) with no decision call. A test that passes both ways is no evidence, and one that cannot be run so is not checkable (a command that is no validation command or not one simple command, no test argument, a commit not known green, a dependency manifest or lockfile changed since that commit, a checkout not clean at the head, a run that timed out or changed tracked files, too little time): either reply goes to the decision model, its note kept for the Block. Only the named test is run: a reproduction written in a reply's prose is never run. For every other reply the decision model is asked whether the answer resolves it, `fixed` (the change shows it), `rebutted` (the reply names an acceptance entry or repository rule that permits the code), `decided` (a maintainer's recorded decision, not overruled by a later one, speaks to the very entry, rule, file or behaviour the finding concerns and settles it: it waives or changes what the finding asks, assigns that work outside the PR, such as to the owner after the merge, or permits the code; general approval such as "LGTM" or "ship it" settles nothing, and doubt is `unresolved`) or `unresolved`, given the finding (its reviewer, file, line, body and when it was raised), the reply, what the PR changed since the commit the finding was raised on (its own file, then its other files: those the reply names by path first, then those the finding names, then the rest, cut at 8,000 characters with the paths of the files left out listed after the cut; without the default branch's changes; a finding with no commit of its own reads from the head the stage's review covers), for a reply that says its fix is in the PR body (replies.json's `"fixed": true` and `"pr_body": true`) what the body changed since the finding was raised (git's diff, one line of context, of the body's version at that time, from GitHub's record of the body's edits read once per check, to the body now, cut at 16,000 characters; `no change`, or `whether the body changed could not be read` when the record does not hold that version), with `fixed` then taking a defect gone from the body and the question saying a body change alone never fixes a finding that asks for a change to the code or tests or says the code does not meet an acceptance entry or rule (every other reply's question carries no body; Pendulum's posted answer carries `pr_body` in its marker, ``, so an answer judged again in a later stage keeps it), the issue's acceptance entries, the repository's rules (AGENTS.md at the head, the protected paths, the validation commands), and the maintainers' decisions the stage read (comments on the issue and the PR, the PR's review bodies and its inline review comments, by people with write access, a dismissed review's left out; each with its author, time, place and review state, cut at 2,000 characters; within 8,000 characters the newest decision always, the issue's newest within half the window while the PR has decisions too, then the PR's newest, then the issue's in any room left, with a count of those left out; a decision the reply quotes counts only when it is there, so a comment by anyone else settles nothing), each part between markers with a random tag, as data never to be followed; at most four questions at once. `fixed`, `rebutted` or `decided` at the confidence floor or more (Red-check triage, below; no confidence is none of them; `decided` never on code scanning's alert, nor when a decision newer than the oldest one shown was left out for length) resolves the finding's thread; any other verdict leaves it open (`judged not fixed`, or `judged not resolved` for a reply that did not say fixed). Pendulum posts the evidence in the thread first, under `` (the test as it ran, the commit it failed on and the head it passed on, or the verdict, its confidence, the floor and who answered); that comment and Pendulum's answer are never a person's answer, so a thread a person reopens stays open, then resolves the thread (a finding in the review body: that PR comment is what resolves it), and a resolved thread, not a reply, marks it answered on every side (the answer pass's list, the exit, the merge pass). Anything else leaves it open. A person's reply answers as before, from someone who can push only: any such reply but Pendulum's own on the reviewer's comment, a writer's on a named reviewer's, none on code scanning's; a person may resolve the thread too. How each finding ended is recorded as data in the review stage's progress comment (its findings line: the reviews its reviewer posted, and each finding it listed with the review or outside reviewer that raised it and its outcome: `fixed-test`, `fixed-verdict`, `answered` for a `rebutted` or `decided` at the floor, `not-fixed`, `not-checkable` for no verdict or no decision model, or `open`), from the check's own branches and whether each ended resolved (a thread already resolved posts no evidence), never from the reply's words; a later stage's listed `open` never replaces a judgement, and a report reads only records written before its period ends; `pendulum report` counts the findings and outcomes per Objective, reviewer model and review stage beside those stages' dollars ([REPORT.md](REPORT.md)). Reason: findings per dollar and a reviewer's quality are then exact counts, not readings of prose. Findings still open after the answer pass (unanswered, or read as not resolved), when the rest of the exit is delivered, take another answer round: the review label goes back on, and the next stage's answer pass gets them again, each with Pendulum's earlier answer (`earlier`). A Question the answer pass raised stands with findings open: it carries them. The rounds per re-entry (the Objective label applied again, a restart or a Block) are at most two, shared with code scanning's rounds at the merge (below) and counted in Pendulum's record on the issue (``); past them the stage Blocks (cause `work`) naming each open finding, with what the check found on it (`test failed`, `judged not fixed` or `judged not resolved`, after `not evidence` or `not checkable` for a named test that proved nothing), and how to clear it: a fix on the branch, a reply by someone who can push or a resolved thread, or for a finding in the review body that resolved marker by someone who can push. An answer the gateway gave no verdict on (a failed call, a refused key, an answer outside the choices; once the key is refused no further question is asked in the stage) costs no answer round: it goes to no worker pass, the review label goes back on, and the next stage judges it again first (when that stage was started only for this, an answer it then reads as not resolved waits for the answer round its exit takes); the second stage in a row without a verdict Blocks naming the decider's error code (cause `setup` for a refused key, `provider-quota` for a rate limit or spent credits, else `outage`). With no decider (`decision_route` off, or `gateway` with no key) the decision model's part is off: a reply answers its finding as before unless a test it names proves the fix or fails on the head, the progress comment says `finding check: off ()` once per stage, and doctor's `finding check` line warns. The reviewer reads each chunk of work once. The answer pass's and the fix passes' commits are the review's answer and get no review of their own when they do only what the review's findings and the failing checks asked. The answer gate decides it: Pendulum asks the decision model red-check triage uses (the same key, `triage_model`, below) one question, `only_asked` or `more_than_asked`, giving it the diff they wrote since the head the review read, with the default branch's changes left out (the replay below), then each finding the answer pass answered (path, line, body) and the failure lines and output's end of each red the fix passes and the answer pass were given, at most 32 KiB in one question (`ANSWER_INPUT_BYTES`). A diff too large for one question is asked about in parts, each with the findings and failures and each within that size: whole files packed in order, and a file too large alone split into its hunks, packed in order the same way, each part with the file's header; four parts at a time, none started once one is held. Only every part covered covers them, and the first part held is named (`part of … ()`); a single hunk too large for a question is not judged, named in the hold, and no part is asked, even with no decider. Before each question, the first included, the stage's `max_usd` (in known dollars: a price still settling counts nothing) and its wall clock are read: once either is reached, that part and every later one are not judged, each named in the hold, and never covered. The question says a failing check asks only that it pass with the smallest change, so new behaviour or a feature is more than asked even when a check's output mentions it. The diff, the findings and the failures go as data (fenced, or JSON strings), with no worker secret in them. Pendulum records the head they leave in its own comment on the PR (``, edited in place), and the later of that record and its last review's `commit_id` is the head the review covers. Only `only_asked` at the confidence floor (Red-check triage, below) covers them. Anything else holds the PR: a lower or missing confidence, `more_than_asked`, a call that fails, times out or answers outside the two choices, no key, a hunk too large for a question, a diff over 16 MiB (not read whole), a file git diffs as binary (its numstat `-`, or `Binary files … differ`: a text file with one NUL byte is one, and the model would see none of it; no question is asked), or a diff that cannot be told apart. Then the record stays on the head the review read, marked as an answer's (`:answer` after the sha), the review label goes back on, and the next review stage's pass reads those commits. That pass's own answer is gated too: held again, the stage Blocks (cause `work`) naming why, the record staying on the head that pass read, and a person reviews those commits, so the gate never loops. The gate's answer goes to the job log (`answer gate (): → …`, with `(in parts)` after the confidence for a diff asked about in parts). The review goes up on the head the reviewer read (its `commit_id`), which is the head Pendulum's checkout was on when the reviewer started; when GitHub's head is no longer Pendulum's by then (someone else pushed), nothing is posted and the review label goes back on for the new head. A pass on new commits that finds nothing leaves the earlier reviews' open findings to the answer pass. A review stage whose head is that head, or only a rebase of it (the covered head's own change, replayed with `git merge-tree` onto the head's merge base with the default branch, gives exactly the head's tree: a clean rebase, GitHub's update-branch merge, Pendulum's merge of a default branch), runs no reviewer and goes on to validation and the merge. Commits written after it (the work stage's answer to a Question, a maintainer's push, a resolved conflict) get a reviewer pass before the merge, on the diff from that replay to the head, so neither the reviewed code nor the default branch's changes are in it; when the replay conflicts, the pass gets the whole PR. The prompt and the posted review name the head it follows on; the pass judges every acceptance entry as a first pass does, and a foundation Objective's foundation pass runs on the same diff. An answer pass that applied maintainers' decisions (a Question's answer, a comment no pass had read) in new commits does not merge: the stage covers the head before it (the head the review read, marked as an answer's, when the answer gate holds what the stage wrote before it), puts the review label back on, and the next review stage reviews those commits. Reason: the one-pass rule keeps the same code from being reviewed twice, and code written after the review was never reviewed. +The reviewer is one model call per PR on the top model, and one more for each chunk of commits written after it (below), on a head whose validation is green or still running. A head without the validated marker (handed off by the worker, pushed or marked ready by a person) gets Pendulum's validation in the review stage. With required checks and a reviewer to run, the reviewer starts at the first reading that shows a check run on the head (CI has started) and no required check red, once the PR's mergeable state reads no conflict, and the wait goes on beside it: a head that conflicts, whose checks never start, or whose reading already shows a required check red gets no reviewer pass yet, but its fix passes first, as does every head when the reviewer does not run or validation is the commands alone, and a red that stays red Blocks before any review. A head handed off from the local route is not given its commands again on that first pass. A red the checks show while the reviewer runs, or after it (after triage's rerun and the base-failure check, as any red), gets no fix pass of its own: once the reviewer is done, the answer pass takes it with the reviewer's comments, its failures in the prompt and their whole logs in `$PENDULUM_IN/checks/`, and runs even when the review has no comment. A conflict, a check that never starts or a base failure ends the stage as before, ending a reviewer call still running, and its review is not posted, so the next review stage reviews the head it works on. The validated marker goes on the head the checks were green on; when the PR's head moved meanwhile (a person's push), the stage posts nothing and puts the review label back on for the new head. The reviewer's rerun and the foundation pass finish the review that started. The progress comment shows the overlap: the reviewer's line ends `beside the required checks on `, the pass's own lines meanwhile start `during the reviewer:`, and the checks' end is `required checks on green|red|settled while the reviewer runs`, or, when they outlast the reviewer, `waiting for required checks on after the reviewer`; the scorecard counts each minute once, the overlap as review beside CI (bench/README.md). It gets the Objective text, the diff and the PR body, and it posts review comments tied to lines, each naming the acceptance entry or repository rule it concerns, with assumptions that change behaviour flagged explicitly. It also judges every acceptance entry, met or not, with the line that shows it: Pendulum lists the issue's entries in the prompt, puts the judgments in the review body, names any entry left unjudged, and turns an entry judged not met into a comment the worker must answer. On a split part that leaves work for the next, the prompt says it is part N and not the last: an entry the PR body assigns to a later part is judged deferred and gets no comment, while one the body claims and the diff does not deliver is still a finding. The last part and a one-PR Objective are judged against every entry. Reason: otherwise every earlier part re-raises the entries later parts deliver, and the worker rebuts each one. An entry a maintainer's recorded decision assigns outside the PR (to the owner after the merge, to another issue) is judged not met with that decision's key, and gets no comment: Pendulum keeps the judgment only when the key is exactly one of the stage's trusted decisions (authors with write access), lists the entry in the review body as assigned by that decision, with its author and link, and never counts it met; any other key is a plain not met. Before each reviewer call Pendulum writes `$PENDULUM_IN/decisions.md` with every trusted decision, uncut and under its key, so the decisions the prompt cuts for length are there to read. An entry whose finding from an earlier review is still open on the PR gets no second comment, unless the last stage's finding check got no verdict on that finding's answer: the stage judges it again after the review goes up (below) and may resolve it, so the entry gets its comment in the review (two open threads for one entry when the check then leaves that finding open). A review with no line comment beside such an entry, or one a decision assigns, has a placeholder that says there are no new line findings and points to the body. Reason: a decision the reviewer cannot read or cannot cite turns the entry into a finding every round, which no change on the branch can answer. A call that fails (a crash, a non-zero exit, a cap) or an answer that holds no JSON or judges none of the entries is a missed pass: it reruns once in a fresh session on every harness, on the fallback reviewer where the harness has one. A provider's refusal is not rerun, a subscription's usage limit included (below): the fallback model draws on the same subscription. When neither call produced a review, nothing is posted and the Objective Blocks with how each call ended: no placeholder stands in for a review that did not happen, and Pendulum never merges past the review. A rerun that answered but judged none of the entries is posted, and its placeholder says it is not a clean pass. Entries left unjudged are recorded on the review, not enforced: they do not hold the merge. It has no verdict and no refuse path: a reviewer that can stop a run turns every false positive into a hold. An Objective labelled `pendulum:foundation` (trusted as the harness and model labels are, and outside the plan's record, since it changes only the review) gets a second call after that one, the foundation pass: it checks each shared contract the diff adds or changes against its callers in the repository, and shared test fixtures against isolation, and returns line-tied comments only, which go up in the same review. It runs within the job's `max_usd` like any call, reruns once when it misses, and when it cannot run or never answers nothing is posted and the stage Blocks, as for the first pass. The review that includes it carries a second marker (``): a PR reviewed before the label was added gets the foundation pass alone at its next review stage, and a rebased PR keeps its review only when that marker is there. Reason: a defect in a contract later work builds on (a mapping that folds new error codes into an old one, a fixture that writes the developer's real state) passes a review of the diff alone, and every later Objective inherits it. The worker gets the comments once and addresses or rebuts each; Pendulum pushes, posts the replies, and validation runs again; rebutted findings stay visible on the merged PR. Each finding the answer pass replied to (Pendulum's reviewer's, a named reviewer's, code scanning's) then goes to the finding check (`lib/finding-check.mjs`), before another answer round is spent on it. A reply that says the finding is fixed and names a test that fails without the fix (replies.json's `fixed` and `test`: one of the validation commands exactly as listed, a single simple command, and the arguments that run only that test, at least one of them no option; the answer pass is offered `test` only when there are validation commands) has that test run through the stage's own validation path (`runValidation`: its sandbox and env), each argument shell-quoted, in the checkout itself: first on the PR's head, then, when the commit the finding was raised on (else the head the stage's review covers) is known green (Pendulum's validated marker on it, and no red the stage held there: a command that runs more than the test fails on a red commit for any red it had), on that commit's tree with the head's copy of each argument that names a file put back, after which the checkout is reset to the head and its HEAD and tracked files checked. Each run gets the stage's time left less a reserve for the exit, and none starts with under a minute of it. Failing without the fix and passing with it resolves the finding; failing on the head leaves it open (`test failed`) with no decision call. A test that passes both ways is no evidence, and one that cannot be run so is not checkable (a command that is no validation command or not one simple command, no test argument, a commit not known green, a dependency manifest or lockfile changed since that commit, a checkout not clean at the head, a run that timed out or changed tracked files, too little time): either reply goes to the decision model, its note kept for the Block. Only the named test is run: a reproduction written in a reply's prose is never run. For every other reply the decision model is asked whether the answer resolves it, `fixed` (the change shows it), `rebutted` (the reply names an acceptance entry or repository rule that permits the code), `decided` (a maintainer's recorded decision, not overruled by a later one, speaks to the very entry, rule, file or behaviour the finding concerns and settles it: it waives or changes what the finding asks, assigns that work outside the PR, such as to the owner after the merge, or permits the code; general approval such as "LGTM" or "ship it" settles nothing, and doubt is `unresolved`) or `unresolved`, given the finding (its reviewer, file, line, body and when it was raised), the reply, what the PR changed since the commit the finding was raised on (its own file, then its other files: those the reply names by path first, then those the finding names, then the rest, cut at 8,000 characters with the paths of the files left out listed after the cut; without the default branch's changes; a finding with no commit of its own reads from the head the stage's review covers), for a reply that says its fix is in the PR body (replies.json's `"fixed": true` and `"pr_body": true`) what the body changed since the finding was raised (git's diff, one line of context, of the body's version at that time, from GitHub's record of the body's edits read once per check, to the body now, cut at 16,000 characters; `no change`, or `whether the body changed could not be read` when the record does not hold that version), with `fixed` then taking a defect gone from the body and the question saying a body change alone never fixes a finding that asks for a change to the code or tests or says the code does not meet an acceptance entry or rule (every other reply's question carries no body; Pendulum's posted answer carries `pr_body` in its marker, ``, so an answer judged again in a later stage keeps it), the issue's acceptance entries, the repository's rules (AGENTS.md at the head, the protected paths, the validation commands), and the maintainers' decisions the stage read (comments on the issue and the PR, the PR's review bodies and its inline review comments, by people with write access, a dismissed review's left out; each with its author, time, place and review state, cut at 2,000 characters; within 8,000 characters the newest decision always, the issue's newest within half the window while the PR has decisions too, then the PR's newest, then the issue's in any room left, with a count of those left out; a decision the reply quotes counts only when it is there, so a comment by anyone else settles nothing), each part between markers with a random tag, as data never to be followed; at most four questions at once. `fixed`, `rebutted` or `decided` at the confidence floor or more (Red-check triage, below; no confidence is none of them; `decided` never on code scanning's alert, nor when a decision newer than the oldest one shown was left out for length) resolves the finding's thread; any other verdict leaves it open (`judged not fixed`, or `judged not resolved` for a reply that did not say fixed). Pendulum posts the evidence in the thread first, under `` (the test as it ran, the commit it failed on and the head it passed on, or the verdict, its confidence, the floor and who answered); that comment and Pendulum's answer are never a person's answer, so a thread a person reopens stays open, then resolves the thread (a finding in the review body: that PR comment is what resolves it), and a resolved thread, not a reply, marks it answered on every side (the answer pass's list, the exit, the merge pass). Anything else leaves it open. A person's reply answers as before, from someone who can push only: any such reply but Pendulum's own on the reviewer's comment, a writer's on a named reviewer's, none on code scanning's; a person may resolve the thread too. How each finding ended is recorded as data in the review stage's progress comment (its findings line: the reviews its reviewer posted, and each finding it listed with the review or outside reviewer that raised it and its outcome: `fixed-test`, `fixed-verdict`, `answered` for a `rebutted` or `decided` at the floor, `not-fixed`, `not-checkable` for no verdict or no decision model, or `open`), from the check's own branches and whether each ended resolved (a thread already resolved posts no evidence), never from the reply's words; a later stage's listed `open` never replaces a judgement, and a report reads only records written before its period ends; `pendulum report` counts the findings and outcomes per Objective, reviewer model and review stage beside those stages' dollars ([REPORT.md](REPORT.md)). Reason: findings per dollar and a reviewer's quality are then exact counts, not readings of prose. Findings still open after the answer pass (unanswered, or read as not resolved), when the rest of the exit is delivered, take another answer round: the review label goes back on, and the next stage's answer pass gets them again, each with Pendulum's earlier answer (`earlier`). A Question the answer pass raised stands with findings open: it carries them. The rounds per re-entry (the Objective label applied again, a restart or a Block) are at most two, shared with code scanning's rounds at the merge (below) and counted in Pendulum's record on the issue (``); past them the stage Blocks (cause `work`) naming each open finding, with what the check found on it (`test failed`, `judged not fixed` or `judged not resolved`, after `not evidence` or `not checkable` for a named test that proved nothing), and how to clear it: a fix on the branch, a reply by someone who can push or a resolved thread, or for a finding in the review body that resolved marker by someone who can push. An answer the gateway gave no verdict on (a failed call, a refused key, an answer outside the choices; once the key is refused no further question is asked in the stage) costs no answer round: it goes to no worker pass, the review label goes back on, and the next stage judges it again first (when that stage was started only for this, an answer it then reads as not resolved waits for the answer round its exit takes); the second stage in a row without a verdict Blocks naming the decider's error code (cause `setup` for a refused key, `provider-quota` for a rate limit or spent credits, else `outage`). With no decider (`decision_route` off, or `gateway` with no key) the decision model's part is off: a reply answers its finding as before unless a test it names proves the fix or fails on the head, the progress comment says `finding check: off ()` once per stage, and doctor's `finding check` line warns. The reviewer reads each chunk of work once. The answer pass's and the fix passes' commits are the review's answer and get no review of their own when they do only what the review's findings and the failing checks asked. The answer gate decides it: Pendulum asks the decision model red-check triage uses (the same key, `triage_model`, below) one question, `only_asked` or `more_than_asked`, giving it the diff they wrote since the head the review read, with the default branch's changes left out (the replay below), then each finding the answer pass answered (path, line, body) and the failure lines and output's end of each red the fix passes and the answer pass were given, at most 32 KiB in one question (`ANSWER_INPUT_BYTES`). A diff too large for one question is asked about in parts, each with the findings and failures and each within that size: whole files packed in order, and a file too large alone split into its hunks, packed in order the same way, each part with the file's header; four parts at a time, none started once one is held. Only every part covered covers them, and the first part held is named (`part of … ()`); a single hunk too large for a question is not judged, named in the hold, and no part is asked, even with no decider. Before each question, the first included, the stage's `max_usd` (in known dollars: a price still settling counts nothing) and its wall clock are read: once either is reached, that part and every later one are not judged, each named in the hold, and never covered. The question says a failing check asks only that it pass with the smallest change, so new behaviour or a feature is more than asked even when a check's output mentions it. The diff, the findings and the failures go as data (fenced, or JSON strings), with no worker secret in them. Pendulum records the head they leave in its own comment on the PR (``, edited in place), and the later of that record and its last review's `commit_id` is the head the review covers. Only `only_asked` at the confidence floor (Red-check triage, below) covers them. Anything else holds the PR: a lower or missing confidence, `more_than_asked`, a call that fails, times out or answers outside the two choices, no key, a hunk too large for a question, a diff over 16 MiB (not read whole), a file git diffs as binary (its numstat `-`, or `Binary files … differ`: a text file with one NUL byte is one, and the model would see none of it; no question is asked), or a diff that cannot be told apart. Then the record stays on the head the review read, marked as an answer's (`:answer` after the sha), the review label goes back on, and the next review stage's pass reads those commits. That pass's own answer is gated too: held again, the stage Blocks (cause `work`) naming why, the record staying on the head that pass read, and a person reviews those commits, so the gate never loops. The gate's answer goes to the job log (`answer gate (): → …`, with `(in parts)` after the confidence for a diff asked about in parts). The review goes up on the head the reviewer read (its `commit_id`), which is the head Pendulum's checkout was on when the reviewer started; when GitHub's head is no longer Pendulum's by then (someone else pushed), nothing is posted and the review label goes back on for the new head. A pass on new commits that finds nothing leaves the earlier reviews' open findings to the answer pass. A review stage whose head is that head, or only a rebase of it (the covered head's own change, replayed with `git merge-tree` onto the head's merge base with the default branch, gives exactly the head's tree: a clean rebase, GitHub's update-branch merge, Pendulum's merge of a default branch), runs no reviewer and goes on to validation and the merge. Commits written after it (the work stage's answer to a Question, a maintainer's push, a resolved conflict) get a reviewer pass before the merge, on the diff from that replay to the head, so neither the reviewed code nor the default branch's changes are in it; when the replay conflicts, the pass gets the whole PR. The prompt and the posted review name the head it follows on; the pass judges every acceptance entry as a first pass does, and a foundation Objective's foundation pass runs on the same diff. An answer pass that applied maintainers' decisions (a Question's answer, a comment no pass had read) in new commits does not merge: the stage covers the head before it (the head the review read, marked as an answer's, when the answer gate holds what the stage wrote before it), puts the review label back on, and the next review stage reviews those commits. Reason: the one-pass rule keeps the same code from being reviewed twice, and code written after the review was never reviewed. A repository may also name third-party reviewers, reviewer Apps outside Pendulum, by login (`third_party_reviewers`, such as `greptile-apps[bot]`; `lib/third-party-review.mjs`). Their review is part of the PR's review. After Pendulum's reviewer posts, the review stage reads the PR's reviews once for each named reviewer's submitted review whose `commit_id` is the head Pendulum's reviewer read. While one is missing, the stage does not wait in place: it records the wait on the PR (a comment, marker ``, counted only when Pendulum or someone who can push wrote it, since a local loop writes as its developer; the bound the dispatcher, `serve` and `run` wake the review at is its `minutes`, the `third_party_wait_minutes` setting of the stage that wrote it (20 by default, taken as at most 24 hours whatever the record says), from GitHub's creation time of that comment; the review stage that resumes judges the wait by its own setting), takes the `pendulum:review` label off, moves the Objective to In review when it is not there (a Question a person answered by marking the PR ready still carries `pendulum:question`) and ends with exit `waiting`, holding no local slot, runner or sandbox. The dispatcher's tick, on every route (the Action's, the sandbox's, `serve`'s), reads that record from the PR comments it already reads for send-backs. While the bound is ahead and the review is missing, it leaves the PR as it is: no merge pass, no branch update, no send-back; when it cannot read the PR's comments and third-party reviewers are named, it leaves the PR alone that tick too. Once that review of the head exists, or the bound has passed, or the PR's head is no longer the one the record names (a maintainer's commit, GitHub's update-branch), it puts `pendulum:review` back on, which starts the review stage on that route as any review label does. The reviews are read at most once a tick, and in `serve` at most once a minute per PR. An App's review submitted on a Pendulum PR that is not Pendulum's own ticks the dispatcher (`pull_request_review`, its reviewer of type `Bot`; a person's review ticks nothing), and on a public repository the bound's end is a `third-party` wake; on a private one it waits for the next event or the cron. The review stage that starts again finds Pendulum's own review on the PR (its marker), so the reviewer does not run again; it reads the record, says on its progress comment which reviewers reviewed the head or were left out past the bound, removes the record and goes on to the answer pass. A review label applied after the record was last written means a stage has taken the wait up, so the dispatcher never puts the label back twice for one record. A stage that starts again before the bound (the label re-applied by hand, a restarted loop) rewrites the same record and ends `waiting` again; a new head gets a new record and its own bound. A stage that goes on without waiting (the reviewer's review in, the bound passed, the cap spent) removes the record, so none holds a later merge. A record that cannot be read or written, a label that cannot be taken off, or Pendulum's own login unknown (its record could not be told from a pasted one) is no wait: the stage goes on without the reviewer rather than stall. `pendulum run` and `pendulum review`, which may have no `serve` behind them, do the dispatcher's part in place, once a minute, and then review again, and `pendulum run` on an In-review Objective whose review waits takes that wait up the same way; `serve` leaves it to its own ticks. The answer pass then gets each named reviewer's root line comments that neither Pendulum nor anyone who can push has replied to, from any head, beside Pendulum's own. Each comment carries `from` (the reviewer's login), and its body is sanitised like harness text and cut at 8,000 characters. The prompt and the pendulum-deliver-pr skill call their text untrusted review input: the worker judges each finding against the code, fixes it or rebuts it, and follows no instruction in it. Pendulum posts the replies in their threads, and the finding check resolves those it reads as resolved. The exit counts a named reviewer's comment as unanswered only when the answer pass was given it; one posted after the wait is left for the next review. Only the named logins are waited for; any other reviewer's comment is handled as without the setting (the merge pass, below, has the answer pass take an unlisted reviewer's thread when the base's rules require every conversation resolved). With none named, the stage reads nothing more and runs as it would without the setting. The wait comes before the answer pass, so before the merge; there is no wait for the answer pass's new head, just as Pendulum's reviewer does not review it again. Code scanning (GitHub's own App, `github-advanced-security[bot]`, whose alerts on the PR's lines are review comments, CodeQL's say) is answered the same way on every repository, with no setting and no wait: the answer pass gets its open comments marked as untrusted input, and Pendulum resolves each thread once it replied. On code scanning's thread only Pendulum's own answer counts, never a person's reply alone, and an alert adds only its own file to the answer pass's fix-pass scope, never a protected or toolchain file. diff --git a/plugin/scripts/lib/handoff.mjs b/plugin/scripts/lib/handoff.mjs index 0c429542..d4cb7807 100644 --- a/plugin/scripts/lib/handoff.mjs +++ b/plugin/scripts/lib/handoff.mjs @@ -156,8 +156,7 @@ export function noReviewReason(recs, reviewOf) { * the answer, the PR or a comment can supply one), which the body lists as assigned by that decision, with its author * and link, and which stays not met; any other key is a plain not met. And an entry whose finding is already open on * the PR (`open`: Pendulum's own open findings there, session.mjs unaddressedReviewComments), which the answer pass - * takes from that thread; its comment comes back in `relied`, with the ids of those findings (reliedResolved). An entry - * of the issue (acceptanceEntries) the reviewer + * takes from that thread. An entry of the issue (acceptanceEntries) the reviewer * left unjudged is named in the body and returned in `unjudged`. With no comment at all, Pendulum's placeholder goes on * the first line the diff shows, and `placeholder` is true: nothing to answer. The placeholder only ever stands for a * pass that ran and answered: a failed call (`ok` false) or an answer with no reviewer JSON throws, so no caller can @@ -184,10 +183,7 @@ export function reviewPost({ issue, review, diff, entries = [], ok = true, split const openEntries = new Set(own(open).map(commentEntry).filter(Boolean)); // No comment is added for an entry judged deferred on a split part that is not the last (a later part delivers it), // for one a trusted decision assigns elsewhere, or for one whose finding is already open on the PR. - const owed = judged.filter((a) => !a.met && !(splitPart && a.deferred) && !assignedBy(a) && !commented.has(entryKey(a.entry))); - const notMet = (a) => ({ path: a.path ?? undefined, line: a.line, body: `acceptance ${a.entry.replace(/^\s*acceptance\s+/i, "")}: not met.${a.evidence ? ` ${a.evidence}` : ""}` }); - const unmet = owed.filter((a) => !openEntries.has(entryKey(a.entry))).map(notMet); - const relied = owed.filter((a) => openEntries.has(entryKey(a.entry))).map((a) => ({ ids: own(open).filter((c) => commentEntry(c) === entryKey(a.entry)).map((c) => String(c.id)), comment: notMet(a) })); + const unmet = judged.filter((a) => !a.met && !(splitPart && a.deferred) && !assignedBy(a) && !commented.has(entryKey(a.entry)) && !openEntries.has(entryKey(a.entry))).map((a) => ({ path: a.path ?? undefined, line: a.line, body: `acceptance ${a.entry.replace(/^\s*acceptance\s+/i, "")}: not met.${a.evidence ? ` ${a.evidence}` : ""}` })); let comments = snapComments( [...raw, ...unmet].map((c) => ({ path: c.path, line: c.line, body: typeof c.body === "string" ? sanitize(c.body) : c.body })), diff, @@ -220,20 +216,9 @@ export function reviewPost({ issue, review, diff, entries = [], ok = true, split lines.push("", judged.length ? "Acceptance as the reviewer judged it:" : "The reviewer judged no acceptance entry.", ...judged.slice(0, 60).map((a) => judgedLine(a, { splitPart, by: assignedBy(a), open: !a.met && !commented.has(entryKey(a.entry)) && openEntries.has(entryKey(a.entry)) }))); if (unjudged.length) lines.push("", `Not judged: ${unjudged.map((id) => sanitize(id)).join(", ")}.`); } - return { body: lines.join("\n"), comments, placeholder, unjudged, relied }; + return { body: lines.join("\n"), comments, placeholder, unjudged }; } -/** - * The not-met comments reviewPost left out for an open finding (`relied`) whose findings the finding check has since - * resolved, every one (`resolved`: ids as strings), sanitised and fitted to the diff: review.mjs posts them after all, - * so no entry judged not met is left with neither a comment nor an open finding. - */ -export const reliedResolved = (relied, resolved, diff) => - snapComments( - (relied || []).filter((r) => r.ids.length && r.ids.every((id) => resolved.has(id))).map((r) => ({ ...r.comment, body: sanitize(r.comment.body) })), - diff, - ); - /** * What Pendulum's reviews on a PR (`reviews`, GitHub's list) already cover: `reviewed`, one carries its review marker (the * first pass ran); `foundation`, one also carries the foundation marker (the foundation pass ran, with the first pass or diff --git a/plugin/scripts/review.mjs b/plugin/scripts/review.mjs index 350551b0..9aa2cb56 100644 --- a/plugin/scripts/review.mjs +++ b/plugin/scripts/review.mjs @@ -36,7 +36,7 @@ import { accountGate, authorGate, choiceGate, planGate } from "./lib/objective-t import { stageStart } from "./lib/objective-cap.mjs"; import { ranWith } from "./lib/choice.mjs"; import { credentialChecks } from "./doctor.mjs"; -import { exitFromOut, handoverChecker, handoverPosted, isSplitPart, noReviewReason, openHandoff, partsRecord, postNote, postReplies, publishPr, reliedResolved, reviewAnswer, reviewCallEnd, reviewCallTimeout, reviewMissed, reviewPost, reviewRerun, reviewRetry, reviewsCover, splitPartOf } from "./lib/handoff.mjs"; +import { exitFromOut, handoverChecker, handoverPosted, isSplitPart, noReviewReason, openHandoff, partsRecord, postNote, postReplies, publishPr, reviewAnswer, reviewCallEnd, reviewCallTimeout, reviewMissed, reviewPost, reviewRerun, reviewRetry, reviewsCover, splitPartOf } from "./lib/handoff.mjs"; import { TRIAGE_MODEL, confidenceFloor, openTriage, triageKey } from "./lib/triage.mjs"; import { deciderKind, openDecider, settleDecisionCosts } from "./lib/decider.mjs"; import { flushDecisionSpend } from "./lib/decision-spend.mjs"; @@ -486,6 +486,13 @@ export async function headMovedFrom({ repo, prNumber, own, token, before = null return now.head.sha; } +/** + * Pendulum's open findings (`open`) a not-met entry may rely on for its comment (handoff.mjs reviewPost `open`): those + * whose answer the last stage's check left without a verdict (`unjudged`, checkState) are judged again after the review + * goes up and may be resolved then, so they are left out and their entries get their comment (#638). + */ +export const settledOpen = (open, unjudged) => open.filter((c) => !unjudged.some((id) => String(id) === String(c.id))); + /** * The review comments the answer pass gets: every open one, but after a first review that found nothing (its * placeholder) only the third-party reviewers' and code scanning's (`from`). A later pass that found nothing leaves the @@ -1827,10 +1834,21 @@ async function main() { // Pendulum's own findings still open on the PR from an earlier review (never a third party's or a writer's comment): // an entry judged not met again gets no second comment beside its open one. Read only when an earlier review of // Pendulum's exists and this answer judges an entry not met; unread, each such entry gets its comment as before. + // One whose answer the last stage's check got no verdict on is judged again after this review goes up and may be + // resolved then, so it holds no entry back (settledOpen); with that record unread, none does. const openFindings = !alreadyReviewed && !scope.first && (reviewOf(answer).acceptance || []).some((a) => !a.met) ? await unaddressedReviewComments(repo, pr.number, reviewerLogin, { ...token, reviewIds: await pendulumReviewIds() }).then( - (open) => open.filter((c) => !c.from && !c.writer), + async (open) => { + const mine = open.filter((c) => !c.from && !c.writer); + if (!mine.length) return mine; + const rec = await readAnswerRounds(repo, { issue, prNumber: pr.number, objectiveLabel: inputs.objectiveLabel }, token).catch((e) => { + if (outageMet(e)) throw e; + log(`open findings: the answer-rounds record could not be read (${String(e?.message || e).slice(0, 120)})`); + return null; + }); + return rec ? settledOpen(mine, checkState(rec).unjudged) : []; + }, (e) => { // Unread because of an outage (network.mjs outageMet): the stage ends on it, as on any read. if (outageMet(e)) throw e; @@ -1916,8 +1934,6 @@ async function main() { const unjudged = []; // What the check found on each finding it left open (a failed test, a no, a test it could not run), for the exit (#516). const checkFound = []; - // The findings it resolved, by id. - const resolvedNow = new Set(); const findingRules = async (head) => { const agents = head ? await outbox.git(["show", `${head}:AGENTS.md`], { maxBytes: 64 * 1024 }).catch(() => null) : null; return [agents?.code === 0 ? `AGENTS.md:\n${agents.stdout}` : "", `Protected paths (no pass may change them): ${protectedList.join(", ")}`, inputs.validation.length ? `Validation commands: ${inputs.validation.join("; ")}` : ""].filter(Boolean).join("\n\n"); @@ -1933,7 +1949,6 @@ async function main() { const r = await checkAnswers({ repo, issue, prNumber: pr.number, findings, repliesText, check: findingCheck, runTest, ...findingReads({ git: outbox.git, base: outbox.base, head, repo, prNumber: pr.number, token }), since: reviewedAt, issueBody: issueData.body || "", rules: await findingRules(head), decisions, opts: token, progress }).catch((e) => (log(`finding check: ${String(e?.message || e).slice(0, 160)}; the findings stay open`), { unjudged: [], found: [] })); unjudged.push(...r.unjudged); checkFound.push(...(r.found || [])); - for (const id of r.resolved || []) resolvedNow.add(String(id)); }; // Answers the last stage's check got no verdict on are judged again first, with no worker pass for them (a gateway // failure costs no answer round); one still without a verdict waits for the exit (session.mjs openFindingsExit). @@ -1948,14 +1963,6 @@ async function main() { if (again.length) { await progress(`finding check: judging again ${again.length} answer(s) the last stage got no verdict on`, { step: "check", again: again.length }); await judgeAnswers(again, againReplies(again)); - // A not-met entry this stage's review left without a comment for an open finding the check has just resolved gets - // its comment after all, in a review of its own, so the answer pass takes it. - const raised = reliedResolved(post?.relied, resolvedNow, diff); - if (raised.length) { - const posted = await postReview(repo, pr.number, { body: `${marker(issue, "review")}\nPendulum reviewer: ${raised.length} comment(s) for the acceptance entries this stage's review judged not met beside an open finding, which the finding check has since resolved.`, comments: raised, commitId: read.head }, token); - await progress.findings({ pr: pr.number, reviews: [posted.review?.id] }); - log(`posted ${raised.length} not-met comment(s) whose open finding the finding check resolved`); - } } const held = new Set([...unjudged.map((u) => String(u.id)), ...(lastCheck.judgeOnly ? again.map((c) => String(c.id)) : [])]); const toAnswer = answerList(again.length ? await unaddressedReviewComments(repo, pr.number, reviewerLogin, { ...token, reviewIds: await pendulumReviewIds(), thirdParty, unlisted, decided }) : listedOpen, { placeholder, first: scope.first }).filter((c) => !held.has(String(c.id))); diff --git a/plugin/test/exit.test.mjs b/plugin/test/exit.test.mjs index bc8eff10..191a7939 100644 --- a/plugin/test/exit.test.mjs +++ b/plugin/test/exit.test.mjs @@ -6,8 +6,9 @@ import { test } from "node:test"; import { appJwt, normalizePem } from "../scripts/lib/app-token.mjs"; import { input, parseArgs, resolveInputs } from "../scripts/lib/inputs.mjs"; import { EXIT_PROMPT, acceptanceEntries, capReached, commandPaths, decideExit, deniedHostsFix, fenceDecidable, isHostName, diffLines, entryKey, fixScopeReason, fixScopeRefused, isPlaceholder, marker, namesPath, parseExitRecord, parseFindings, parseReview, protectedPaths, setupPaths, snapComments, pushRefusal, toolchainFile, workflowScopeRefused, workspaceDeclarations } from "../scripts/lib/logic.mjs"; -import { noReviewReason, reliedResolved, reviewMissed, reviewAnswer, reviewCallEnd, reviewCallTimeout, reviewPost, reviewProduced, reviewRetry } from "../scripts/lib/handoff.mjs"; +import { noReviewReason, reviewMissed, reviewAnswer, reviewCallEnd, reviewCallTimeout, reviewPost, reviewProduced, reviewRetry } from "../scripts/lib/handoff.mjs"; import { exitRecord, scopeRefusal } from "../scripts/lib/session.mjs"; +import { settledOpen } from "../scripts/review.mjs"; const ok = { ok: true }; const pr = (body = "## Acceptance\nok", draft = false) => ({ exists: true, draft, body }); @@ -216,11 +217,12 @@ test("#634: an entry whose finding is already open on the PR gets no second not- assert.deepEqual(post.comments.map((c) => c.body), ["acceptance B: not met."]); }); -test("#638: a not-met entry left without a comment for an open finding gets it once the finding check resolves that finding", () => { +test("#638: an open finding about to be judged again holds back no not-met comment", () => { const diff = readFileSync(join(import.meta.dirname, "fixtures", "sample.diff"), "utf8"); const review = { comments: [], acceptance: [{ entry: "A", met: false, path: "src/a.ts", line: 3, evidence: "" }] }; - const post = reviewPost({ issue: 7, review, diff, entries: [{ id: "A", text: "a" }], open: [{ id: 5, path: "src/a.ts", line: 3, body: "acceptance A: not met. the limit is gone" }] }); - assert.deepEqual(reliedResolved(post.relied, new Set(["5"]), diff).map((c) => c.body), ["acceptance A: not met."]); + const open = [{ id: 5, path: "src/a.ts", line: 3, body: "acceptance A: not met. the limit is gone" }]; + const post = reviewPost({ issue: 7, review, diff, entries: [{ id: "A", text: "a" }], open: settledOpen(open, [5]) }); + assert.deepEqual(post.comments.map((c) => c.body), ["acceptance A: not met."]); }); test("review: a reviewer call always has a wall-clock cap; past the stage's deadline none starts (#2660)", () => {