Repository navigation
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Problem
A guidance:rev:<sha> arm runs a revision of a repository's guidance against review_replay tasks built from PRs merged earlier. When the guidance was written or edited after a task's review, it can encode that review, and the arm then "catches" the thread because a rule was written from it. Nothing in a batch shows this.
A replay of IntelPython/dpnp#3099 (eight AGENTS.md files; 26 tasks from dpnp PRs merged 2026-09-01..10-28; guidance:none vs guidance:rev:d8fb283) ran into it:
- The cut looked clean. The guidance's first commit (
7781481, 14 Oct 18:45Z) was mined from reviews before it. So a split by review date looked like an out-of-sample test: 9 tasks reviewed after the cut, recall +0.111 against −0.025 for the 17 before. - It was not clean. The evaluated revision has later guidance commits that name three of those 9 tasks' PRs:
c9b533d"Update test guidance for the fixes in #3103";a9f79ed"Update CMake guidance for the extension helper in #3111";441690c, with a rule taken from #3098.
- And
441690ccame after every one of them. It was committed after the first review comment of all 9. - The batch can't show it.
task.jsonhas no review time, andmeta.jsonrecords the arm's SHA but not its guidance history. The split, and finding the leak, were done by hand.
Proposal
- Review time:
repo tasks replaywritesreviewed_atintotask.json: thecreatedAtof the earliest gold thread's first comment. The threads file already holds it. - Guidance history:
repo run, for aguidance:rev:<rev>arm, records the commits reachable from<rev>that touch the manifest's guidance globs, inmeta.jsonarm_rev.guidance_commits({sha, date, subject}, newest first, capped). - Report it:
repo compareflags the tasks such an arm may have seen.- By date: some guidance commit is dated after the task's
reviewed_at. - By name: a guidance commit's message names the task's
source_pr(#<n>). - The output says how many tasks each flag covers and names the named ones. The rest can be compared on their own (with a
repo compare --taskfilter, or an option for it).
- By date: some guidance commit is dated after the task's
- Arms with no history: a
patch:arm has none, so it reads "unknown".guidance:rawis the base's own guidance and is never flagged.
Open questions
- The date rule is strict. Here it flags all 26 tasks for
d8fb283: the last guidance commit came after every review. Is a per-file rule better, counting only commits to a guidance file the task's diff makes relevant? - Which date? Author date or committer date, given that a rebase moves the committer date.
- Content: should a commit that names no PR, but adds text matching a gold comment, be found too? That is a text-similarity question, likely out of scope.
Done when
The fixture tests cover two cases: a guidance commit that names a task's PR, and one dated after a task's review. repo compare flags both, and leaves the other tasks unflagged. The docs (contract §4.6 and §9) say what the flags mean.
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity
antonwolfy commentedon Nov 1, 2026
Proposed answers to the open questions, so a PR can start from them.
dir/AGENTS.mdapplies todir/and below. Whether that clears any task ford8fb283depends on which files the late commits touched. If the root file is among them, all 26 stay flagged, and that is the honest answer: a clean replay of that revision needs PRs reviewed after its last guidance commit.git commit --amendand a rebase keep the author date. The committer date is set when the commit is written, so flagging by it can only flag more tasks, never fewer. A rebase moving it later errs in the safe direction.guidance_commitsrecords both (authored,committed); the flag readscommitted.public_at, notreviewed_at. It holds the earliest time the task's answer was public: here thecreatedAtof the earliest gold thread's first comment (repos/replay/threads.pyalready reads it). Authoring tasks (repo tasks: authoring replay tasks from merged PRs #294) leak the same way and can carry the same field (the PR'screatedAtthere), sorepo compareflags both kinds by one rule.public_at. Soarm_revkeeps that commit per file, uncapped. The capped newest-firstguidance_commitslist serves the name flag, withtruncated: truewhen the cap cut it.--jsongives each task's flags, andrepo compare --task(feat(repo): compare records the start context, filters tasks, and flags inconclusive rows #296) already compares a subset. A dedicated option can follow if passing the ids by hand proves clumsy.Order: two PRs.
public_atintask.jsonfromrepo tasks replay, andguidance_commitsinmeta.jsonarm_rev(repos/workspace.py). No compare change.repo compare, after feat(repo): time to first useful result, with a rate in repo compare #332 and feat(repo): rule ledger: repo compare --ledger with per-rule keep/review/insufficient-evidence verdicts #336 merge.repos/compare.pyis at its 800-line cap, and feat(repo): time to first useful result, with a rate in repo compare #332 moves the per-task values into a newcompare_values.py.fix(repo): NUL-framed guidance history; count merges and link targets
fix(repo): pin git log config; count every link of a guidance chain
fix(repo): refuse a shallow guidance history; keep CR and UTC dates
feat(repo): record public_at and guidance:rev history for leak checks (…