Repository navigation
Finish the code review article plan: noise ledger, second reads, replacement rounds, finding evals - #106
Merged
Conversation
… and harden replacement rounds and the scorer
jacekradko
marked this pull request as ready for review
October 22, 2026 01:51
…lier facts, evals score class and gate on a baseline
…, and gate evals on finished, scored walks
… the runner's mkdir
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Finishes the last four slices of the code review article plan in one PR, as asked. The noise ledger (
sessionctl.py stats) counts what every stored session raised and how the reviewer answered. Every flag now gets a fresh-context second read before it is written. A moved head starts a linked replacement that walks only what moved, carries the earlier calls, and refuses a flag the reviewer already dropped;historyis the part of DD-2520 that needed.evals/scores headless walks against two real misses from this repo and three known-clean PRs.Look hardest at the second read's demotion rule. The first eval run showed that demoting on "would a senior stop" demoted the one real defect in #83, so a flag is now demoted only when the PR did not introduce it or it is not real, and a doubt about worth rides on the confirmed flag. The other place worth a careful read is the dropped-finding match in
link-predecessor, which counts a finding as the same only when its line reads exactly the same and those words appear once in the file's diff.Testing
On 1316da5: the session store, sessionctl, scripts, skill, snapshot, evals and serve suites pass. The full local suite also fails two tests that fail the same way on main (DD-7843). A full
evals/run.pyrun on the final skill passes 3 of 5 cases with 1 miss (#76) and 2 false alarms; that is the committedevals/baseline.json. Walks vary between runs, and the runs, costs and the doubtful ai-kit #5 label are indocs/misses.md. The six flags stored before this change were replayed through the second read on scratch copies, and two fresh readers given only a screenshot of a round-2 page stated the recommendation, the decision owed, and what a confirmed second read means.Risk
Each flag now costs one subagent read, and a read that disputes wrongly hides a flag behind
UNVERIFIEDuntil the reviewer opens it. A bad anchor match would refuse a valid flag in a replacement round. The ledger and history read other sessions only through backup copies. Theevalsworkflow spends model budget on every PR that touches the skill once anANTHROPIC_API_KEYsecret is added, and does nothing until then.How to revert
Revert the squash commit. The new stored fields,
second_readon a beat andpredecessoron a session, are optional and older code carries them through untouched.