feat(roadmap): an independent verifier refutes a plan before the draft pull request opens #3039

What happened

Plans are checked for form only. The review of 2026-11-01 used an independent agent per plan that tried to refute it, and it found the false premises, the broken task order and the dropped criteria that no gate caught.

What should happen

• After the planner and the gates pass, a second agent with a fresh context and the same read-only tools tries to refute the plan: paths, premise, scope against the criteria, task order, spec delta, verify commands.
• Findings go back to the planner for a repair round (bounded, see D1). A plan with a finding that survives the rounds opens no draft; the item is returned to New with the evidence, or marked status/blocked with the question for a person.

Acceptance criteria

• The verifier has no access to the planner's reasoning, only to the plan and the tree.
• On the golden set of stage_eval_cases, after the gates and the verification pass, the job finds at least 90 percent of the known defects of the severities blocker and major. This is the floor that docs/ROADMAP_WORKFLOW.md and test(roadmap): score the stage pipeline with a cloud model first #3207 state, and devops roadmap eval verify reports the recall against it.
• A repair round is bounded and its findings are in the PR comment.
• A plan with an open finding after the last repair round stays a draft with the label plan/needs-fix (fix(roadmap): the stage marks a plan ready although its own model review found a defect #3264), and the issue gets a comment with the evidence.

Related: tracker for this set (filed with it), #2908 (issue workflow), #2867 (draft PR evaluator), #3005 (task kind), #3010 to #3015 (plan gates), #2919 (stuck-item metrics), #2953 (Ollama parser).

Alternatives check of 2026-11-04: partly.
Tool: github/spec-kit (https://github.com/github/spec-kit), MIT, last activity 2026-11-03, maintained.

Proposed design

Problem statement

The verify pass of the roadmap checks a plan for form only. It reads the start of each cited file and it has no recall floor. A finding of task_order or verify_form is advice only, although the body asks for these checks. This item gives the verifier the read tools of the toolbox, the two missing return types, a trace of its reads, and an eval that reports recall against the floor that #3207 names. A finding that survives the repair rounds keeps the plan in draft with plan/needs-fix and comments its evidence on the issue.

Acceptance criteria

Requirement: Read-only verify tools

The verifier SHALL use the read-only tools of src/devops_cli/roadmap/stage_toolbox.py and MUST hold no write tool.

Scenario: Verify tool set holds read tools only

• WHEN A test builds the tool set of the verify pass
• THEN The tool set holds the read tools of the toolbox, no write tool, and no text of the planner.

Requirement: No planner reasoning

The verify request SHALL hold the plan and the read results and MUST hold no planner reasoning or intermediate message.

Scenario: Request holds no planner text

• WHEN The stage job builds the verify request for a plan
• THEN The request holds the plan document and the read results, and a test finds no planner message in it.

Requirement: Trace of the reads

A verify run SHALL record each read in its RunTrace.

Scenario: Trace lists each read

• WHEN The verifier reads a file during a verify run
• THEN The run trace lists the path of that read, and the comment of the staged item reports the reads.

Requirement: Recall against the floor

The verify eval SHALL report the recall of the verifier on the golden set of stage_eval_cases against the floor that #3207 names.

Scenario: Eval reports recall and floor

• WHEN devops roadmap eval verify runs on the golden set
• THEN The report states the recall and the floor, and a change to the pass states the score before and after.

Requirement: Task order and verify form gate

The verifier SHALL return task_order and verify_form findings, and an open finding of either type MUST keep the plan in draft.

Scenario: Open task order finding keeps the draft

• WHEN The verifier returns a task_order finding on a plan
• THEN The plan stays in draft with the label plan/needs-fix, and the change states the score of devops roadmap eval verify before and after.

Requirement: Evidence of a surviving finding

A plan with a finding that survives the repair rounds SHALL keep the label plan/needs-fix and the issue SHALL receive a comment with the evidence.

Scenario: Surviving finding comments its evidence

• WHEN The plan still has an open finding after the last repair round
• THEN The plan stays in draft, and the issue receives a comment with the verifier's findings.

Requirement: Verify request window

The verify request MUST fit the model window and leave room for the planner tools.

Scenario: Largest plan fits the window

• WHEN The stage job builds the verify request for the largest plan of the golden set
• THEN The tokens of the request plus the reserved room are at most the model window.

Existing Tool Considered

github/spec-kit (MIT, last activity 2026-11-03, maintained) covers a read-only analysis pass over the spec, the plan and the tasks, with a findings report. It lacks a bounded repair loop, a pipeline gate that keeps the draft pull request closed, a recall floor on a known-defect set, enforced isolation from the planner's reasoning, and a check of paths and verify commands against the tree. The rest builds on the verify pass of #3153 and #3264, so the written cost is the delta: about 300k tokens and two days to add the read toolbox, the two return types, the trace and the eval floor, and to keep and fix them.

Key questions

• What does the verifier check?: It checks the paths, the premise, the scope against the criteria, the task order, the spec delta and the verify commands. RETURN_TYPES holds missing_path, false_premise and ignores_existing_code. OPEN_TYPES adds spec_delta_wrong. [sources: src/devops_cli/roadmap/stage_verify.py]
• What is the A1 set, and which command scores it?: The A1 set is the golden set of stage_eval_cases. The command devops roadmap eval verify scores it. [sources: docs/ROADMAP_WORKFLOW.md:267]
• Which gates must pass before the verifier runs?: The verifier runs in the stage job after the plan gates, that is after check_proposal, and before the draft pull request opens. [sources: docs/ROADMAP_WORKFLOW.md, src/devops_cli/roadmap/stage_verify.py]
• Where in the pipeline does the verifier run?: In the model lane, in the stage job after the plan gates pass and before the draft pull request opens. [sources: CONTEXT.md, docs/ROADMAP_WORKFLOW.md]
• Which tools does the verifier get?: The verifier uses the read-only tools of src/devops_cli/roadmap/stage_toolbox.py. A test holds that the tool set has no write tool and no text of the planner. (best practice) [sources: src/devops_cli/roadmap/stage_toolbox.py]
  • Chosen: Reuse the read tools of the toolbox. The planner already uses them, and the project keeps and fixes one tool set.
  • Rejected: Build a separate verifier tool set. It is a second tool set to keep and fix, and it can drift from the planner tools.
• How does a verify run record what it read?: Each read goes into the RunTrace of the run. The comment of the staged item reports the reads and the tokens. (best practice) [sources: CONTEXT.md]
  • Chosen: Reuse RunTrace. The project has one trace format, and a person reads the reads and the tokens in one place.
  • Rejected: Write a separate verify log. It is a second format to keep and fix, and it adds no information.
• Which number is the recall floor, and where does it live?: The floor is the one that test(roadmap): score the stage pipeline with a cloud model first #3207 names: 90% recall of blocker and major defects, in docs/ROADMAP_WORKFLOW.md. This item refers to that floor and adds no second number. (best practice) [sources: docs/ROADMAP_WORKFLOW.md]
  • Rejected: 80% of all defects. The best model tested found 56% of the labeled defects and 68% of the major ones, so 80% of all is out of reach now, and the project would hold two floors.
  • Chosen: 90% of blocker and major defects, named in test(roadmap): score the stage pipeline with a cloud model first #3207. The project holds one floor, and the eval compares the verifier against it.
  • Rejected: No floor. The eval can then neither pass nor fail, and the item keeps no measure.
• Which return types does the verifier gate on?: task_order and verify_form join RETURN_TYPES and OPEN_TYPES. An open finding of either type keeps the plan in draft with plan/needs-fix. (best practice) [sources: src/devops_cli/roadmap/stage_verify.py]
  • Chosen: Add both types to the gating sets. The body asks for these checks, and an advice-only finding changes no plan.
  • Rejected: Keep both types as advice. The checks then gate nothing, and staged plans still need correction commits after the verify pass.
• What happens to a plan whose finding survives the repair rounds?: Since fix(roadmap): the stage marks a plan ready although its own model review found a defect #3264, an open finding keeps the plan in draft with the label plan/needs-fix. The issue receives a comment with the evidence. The rule of the body, that no pull request opens, no longer applies. (best practice) [sources: src/devops_cli/roadmap/stage_verify.py, docs/ROADMAP_WORKFLOW.md]
  • Chosen: Keep the draft with plan/needs-fix and comment the evidence. fix(roadmap): the stage marks a plan ready although its own model review found a defect #3264 already holds this rule, and a person reads the plan and the evidence in one place.
  • Rejected: Open no draft and return the item to New or mark it blocked. fix(roadmap): the stage marks a plan ready although its own model review found a defect #3264 supersedes it, and it hides the plan from review.
• How many repair rounds are allowed?: Two repair rounds, that is three verification attempts. (best practice) [sources: CONTEXT.md]
  • Chosen: Two repair rounds. The planner gets two chances to fix mechanical findings, and the loop stays bounded.
  • Rejected: One repair round. The planner gets one chance only.
  • Rejected: A token budget. A budget is harder to enforce and debug than a round count.
• Does the verify request fit the model window?: The verify request uses the same window limit as the planner request and leaves room for the planner tools. (best practice) [sources: CONTEXT.md]
  • Chosen: Apply the planner window limit with reserved room. One limit covers both requests, and the planner tools keep their room.
  • Rejected: Set no limit on the verify request. A large plan then overflows the window and the verify run fails.

Fit

Fits in one pull request.

Suspected block

None.

Dependencies

#3207