feat(roadmap): a failed verification command gets a repair round #3577

Problem or Motivation

Only a failed ruff, mypy or hook, and a missing tick, get a repair round today. A failed verification command (a lost or failing test) stops the step at once, though the agent has the failure text and could fix it. Examples: #2570, #2531, #2524, #2469. Split from #3053.

Proposed Solution & Architectural Design

• A verification command that exits 1 gives the agent one round with the class (test, types, lint), the evidence and a diagnosis hint.
• The turn may change the files of the step only, as the turn for a static check does.
• A failure of class plan_defect or premise_false still stops at once.
• Acceptance: a replay where the first attempt fails its verification command and the second passes ends done, and the trace shows both attempts.

Target Subsystem

src/devops_cli/roadmap/implement_step.py (_extra_ask, _exit_failure).

Existing Tool Considered

pydantic-ai gives ModelRetry for output validation. The step already has the turn for a static check; this change extends it to one more kind of failure: about 40000 tokens to build, 2000 each quarter to keep, 8000 to fix one miss, about 3 hours.

Source: #3053

Proposed design

Problem statement

A failed verification command stops the implement step at once. A lost test also stops the step with no repair turn. The agent has the failure text and can fix the failure. This change gives one repair round to a failed verification command and to a lost test. A plan defect and a false premise still stop at once.

Acceptance criteria

Requirement: Verification failure gets a repair turn

When a verification command that is not a negative search exits 1, the step SHALL give the agent one repair turn with the command's class, its masked output and a diagnosis hint.

Scenario: The repair turn makes the verification pass

• WHEN The first attempt fails a verification command with exit 1 and the repair turn makes the command pass.
• THEN The step ends done, and the trace holds one attempt for the failure and one attempt for the repair.

Requirement: Lost test gets a repair turn

When the step loses a test that existed at the base commit and the test is not recorded, the step SHALL give one repair turn that may restore or edit the test file.

Scenario: The repair turn restores a lost test

• WHEN A step removes a test from a test file that existed at the base commit, and the repair turn restores the test.
• THEN The step ends done, and the trace holds one attempt for the lost test and one attempt for the repair.

Requirement: Plan defect and false premise stop at once

A failure of class plan_defect or premise_false MUST stop the step with no repair turn.

Scenario: A plan defect gets no repair turn

• WHEN The step fails with class plan_defect or premise_false.
• THEN The step stops, and the trace records no repair attempt.

Requirement: Repair turn changes only step files

The repair turn for a failed verification command or a lost test MUST change only the files of the step and the change's tasks.md.

Scenario: The repair turn edits an outside file

• WHEN The repair turn changes a file that the step did not change.
• THEN The step stops, and the reason names the paths that the turn changed.

Existing Tool Considered

No alternatives check has run. pydantic-ai ModelRetry covers model output validation. It does not cover a failed shell verification command or a lost test, and it cannot limit the changed files to the step. The project extends the existing _extra_ask and _repair_rounds path. The written cost is about 40000 tokens to build, 2000 each quarter to keep, 8000 to fix one miss, and about 3 hours.

Key questions

• Should a failed verification command get a repair turn?: Yes. Extend _extra_ask and _repair_rounds so a verification command that exits 1 gets one bounded repair turn. (best practice) [sources: src/devops_cli/roadmap/implement_step.py:1496, src/devops_cli/roadmap/failure.py, tests/roadmap/test_roadmap_implement_step_repair.py, docs/ERRORS.md:83]
  • Chosen: Extend the existing repair rounds. The step already counts rounds and checks the changed files, so one new failure with repair paths reuses the path.
  • Rejected: Build a pydantic-ai ModelRetry loop. ModelRetry handles tool and output validation, not the exit status of a shell verification command.
  • Rejected: Keep the immediate stop. The agent has the failure text and can fix a failing test.
• Should a lost test also get a repair turn?: Yes. _exit_failure must carry the lost test files as repair_paths so _extra_ask gives the turn. (best practice) [sources: src/devops_cli/roadmap/implement_step.py, src/devops_cli/roadmap/implement_test_names.py, tests/roadmap/test_roadmap_implement_step_repair.py]
  • Chosen: Give a lost test a repair turn. The lost test is a failure that the agent can fix, and the reply records the case.
  • Rejected: Keep the current stop with class TESTS_LOST. _exit_failure returns no repair_paths, so the agent gets no turn.
• Which files may a repair turn change?: Only the files that the step changed, plus the change's tasks.md. (best practice) [sources: src/devops_cli/roadmap/implement_step.py, tests/roadmap/test_roadmap_implement_step_repair.py]
  • Chosen: Limit the turn to the step files. _foreign_change already compares the checkout with the tree before the turn.
  • Rejected: Allow the whole checkout. A repair turn could hide an unrelated edit or a stray file.
• Which failures must stop at once?: A failure of class plan_defect or premise_false stops at once with no repair turn. (best practice) [sources: src/devops_cli/roadmap/failure.py, src/devops_cli/roadmap/implement_step.py]
  • Chosen: Stop plan_defect and premise_false at once. No attempt can fix a wrong plan or a false premise.
  • Rejected: Give every failure a repair turn. A repair turn would spend tokens on a failure that it cannot fix.
• How does the step assign the class of a failed verification command?: The step reads the effective program of the command after uv run: pytest is test, mypy is types, and ruff is lint; another allowed command uses the verification class. (best practice) [sources: src/devops_cli/roadmap/implement_shell_policy.py, src/devops_cli/roadmap/implement_step.py, src/devops_cli/roadmap/failure.py]
  • Chosen: Class from the command program. The program is stable and the allowlist already admits it.
  • Rejected: Class from the command output. The output is model-visible and can name another tool.
  • Rejected: One generic verification class. The issue names test, types and lint as distinct classes.
• Should the change add a new repair-round setting?: No. Reuse roadmap.implement.repair_rounds, as the static-check turn does. (best practice) [sources: src/devops_cli/roadmap/implement_step.py, src/devops_cli/config/settings.py]
  • Chosen: Reuse the existing setting. The step already counts rounds and names the setting in a spent stop.
  • Rejected: Add a verification setting. The issue asks for one more failure kind, not a new budget.

Fit

Fits in one pull request.

Suspected block

None.

Dependencies

None.

Alternatives check of 2026-11-07: in-use.
The item already names this tool.
Tool: pydantic/pydantic-ai (https://github.com/pydantic/pydantic-ai), MIT, last activity 2026-11-07, maintained.