Skip to content

Finish the code review article plan: noise ledger, second reads, replacement rounds, finding evals - #106

Merged
jacekradko merged 14 commits into
mainfrom
jacek/article-finish
Oct 22, 2026
Merged

jacekradko merged 14 commits into
mainfrom
jacek/article-finish

Conversation

@jacekradko

@jacekradko jacekradko commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Finishes the last four slices of the code review article plan in one PR, as asked. The noise ledger (sessionctl.py stats) counts what every stored session raised and how the reviewer answered. Every flag now gets a fresh-context second read before it is written. A moved head starts a linked replacement that walks only what moved, carries the earlier calls, and refuses a flag the reviewer already dropped; history is the part of DD-2520 that needed. evals/ scores headless walks against two real misses from this repo and three known-clean PRs.

Look hardest at the second read's demotion rule. The first eval run showed that demoting on "would a senior stop" demoted the one real defect in #83, so a flag is now demoted only when the PR did not introduce it or it is not real, and a doubt about worth rides on the confirmed flag. The other place worth a careful read is the dropped-finding match in link-predecessor, which counts a finding as the same only when its line reads exactly the same and those words appear once in the file's diff.

Testing

On 1316da5: the session store, sessionctl, scripts, skill, snapshot, evals and serve suites pass. The full local suite also fails two tests that fail the same way on main (DD-7843). A full evals/run.py run on the final skill passes 3 of 5 cases with 1 miss (#76) and 2 false alarms; that is the committed evals/baseline.json. Walks vary between runs, and the runs, costs and the doubtful ai-kit #5 label are in docs/misses.md. The six flags stored before this change were replayed through the second read on scratch copies, and two fresh readers given only a screenshot of a round-2 page stated the recommendation, the decision owed, and what a confirmed second read means.

Risk

Each flag now costs one subagent read, and a read that disputes wrongly hides a flag behind UNVERIFIED until the reviewer opens it. A bad anchor match would refuse a valid flag in a replacement round. The ledger and history read other sessions only through backup copies. The evals workflow spends model budget on every PR that touches the skill once an ANTHROPIC_API_KEY secret is added, and does nothing until then.

How to revert

Revert the squash commit. The new stored fields, second_read on a beat and predecessor on a session, are optional and older code carries them through untouched.

@jacekradko
jacekradko marked this pull request as ready for review October 22, 2026 01:51
@jacekradko
jacekradko merged commit f08359b into main Oct 22, 2026
6 checks passed
@jacekradko
jacekradko deleted the jacek/article-finish branch October 22, 2026 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant