feat(roadmap): a repair attempt cannot repeat a failed tool call #3570

Problem or Motivation

The agent of a repair round can send the same failed tool call again. The issue #3053 asks that no attempt repeats a previous failed action unchanged. Nothing in the tool executor stops it today.

Proposed Solution & Architectural Design

• The step keeps a digest of each failed tool call: the tool name and its normalized arguments (argument_digest of implement_trace.py is the base).
• The executor refuses a call whose digest is in the set. It returns the earlier failure and its diagnosis hint to the model as a new turn.
• Acceptance: a stub model that repeats a failed call gets the refusal with the earlier failure; a changed call runs.

Target Subsystem

src/devops_cli/roadmap/implement_tool_log.py.

Existing Tool Considered

pydantic-ai ModelRetry returns a message but keeps no set of failed actions across rounds. The project builds the guard: about 40000 tokens to build, 2000 each quarter to keep, 8000 to fix one miss, about 2 hours.

Source: #3053

Proposed design

Problem statement

The agent of a repair round can send the same failed tool call again. Nothing in the step stops the repeat, so the step spends turns and tokens on a call that already failed. The step must keep the digest of each failed call. It must refuse a call whose digest it already holds and return the earlier reason to the model.

Acceptance criteria

Requirement: Failed call digest recorded

The step SHALL record the digest of every tool call that ends as a retry or as a failure.

Scenario: A retried call is remembered

• WHEN A stub model calls a tool whose handler raises ModelRetry.
• THEN The failed map of the ToolCallLog holds the digest that argument_digest gives for that call.

Requirement: Repeated failed call refused

The executor MUST refuse a tool call whose digest equals the digest of an earlier failed call of the same step.

Scenario: An unchanged repeat gets the earlier reason

• WHEN A stub model sends the same failing call in two replies.
• THEN The second call returns the refusal text with the earlier reason, and the tool handler has run once.

Requirement: Changed call runs

The executor SHALL run a tool call whose digest differs from every recorded failed digest.

Scenario: A changed argument runs

• WHEN The stub model changes one argument of the failed call.
• THEN The tool handler runs and the call returns its result.

Requirement: Successful call not recorded

The step MUST NOT record the digest of a tool call that returns a result.

Scenario: A repeated successful call runs

• WHEN The stub model sends again a call that returned a result.
• THEN The tool handler runs again and the call returns its result with no refusal.

Requirement: Refusal text is masked

The refusal text MUST carry the reason that the log masked for the earlier call.

Scenario: A path outside the checkout stays hidden

• WHEN The earlier retry names a path outside the checkout.
• THEN The refusal text shows the text of MESSAGES.roadmap.implement_tool_path_hidden in place of that path.

Requirement: Repeat counted and logged

The step SHALL count each refused repeat and write one line to the Job log with the tool name and the outcome repeat.

Scenario: The log line names the repeat

• WHEN The stub model repeats a failed call.
• THEN The Job log holds one line with the tool name and repeat, and the repeat counter is one.

Requirement: Repeat limit ends the run

The step SHALL end the run with UnexpectedModelBehavior when the count of refused repeats passes CONST_ROADMAP_IMPLEMENT_MAX_REPEATS.

Scenario: The model repeats until the run ends

• WHEN The stub model repeats one failed call more than CONST_ROADMAP_IMPLEMENT_MAX_REPEATS times.
• THEN The run ends with UnexpectedModelBehavior, and its message names CONST_ROADMAP_IMPLEMENT_MAX_REPEATS.

Requirement: Failed map is bounded

The failed map SHALL hold at most CONST_ROADMAP_IMPLEMENT_REMEMBERED_FAILURES digests and drop the oldest.

Scenario: The oldest digest is dropped

• WHEN The step records more failures than CONST_ROADMAP_IMPLEMENT_REMEMBERED_FAILURES.
• THEN The map holds the newest digests, and a repeat of the oldest failure runs.

Existing Tool Considered

No alternatives check has run. pydantic-ai ModelRetry returns a message to the model but keeps no set of failed calls across the turns of a step. Its Hooks capability is the place to add the guard, not the guard itself. The project builds the guard: about 40000 tokens to build, 2000 each quarter to keep, 8000 to fix one miss, and about 2 hours.

Key questions

• Which function gives the digest of a tool call?: argument_digest(call) in src/devops_cli/roadmap/implement_trace.py. It digests the tool name and the normalized arguments. ToolCallLog._execute already calls it for every call. [sources: src/devops_cli/roadmap/implement_trace.py, src/devops_cli/roadmap/implement_tool_log.py]
• Does one ToolCallLog cover all turns of a step?: Yes. The step builds one ToolCallLog, and its counters run over all turns of the step. A failed call of an earlier turn is therefore in the map of a later turn. [sources: src/devops_cli/roadmap/implement_tool_log.py]
• Where does the step keep the set of failed digests?: In ToolCallLog, as an ordered map from the digest to the masked reason of the failure. (best practice) [sources: src/devops_cli/roadmap/implement_tool_log.py]
  • Chosen: ToolCallLog in implement_tool_log.py. It is the one hook that sees every call and its outcome, and it already computes the digest of the call.
  • Rejected: A new module implement_repeat_guard.py. It would add a second hook on the same call and read the digest a second time.
  • Rejected: The agent run of agent_run.py. It sees no tool outcome and no digest.
• Which tool outcomes record a digest?: A retry and a failure. A call that returns a result records nothing. A shell refusal records nothing, because its own limit already bounds it (fix(roadmap): raise or configure implement tool retry budget #3271). A call whose arguments fail validation records nothing, because the executor hook never sees it. (best practice) [sources: src/devops_cli/roadmap/implement_tool_log.py]
  • Chosen: A retry and a failure. Both are calls that the model must not send again unchanged.
  • Rejected: Every outcome that is not OK, a shell refusal included. CONST_ROADMAP_IMPLEMENT_MAX_REFUSALS already bounds a repeated shell refusal (fix(roadmap): raise or configure implement tool retry budget #3271).
  • Rejected: Only a retry. A failed call is also a call that the model must not repeat unchanged.
• How does the executor refuse a repeated call?: It returns a refusal text as the tool result, and counts the repeat. The check runs before the started line of a command, so a refused call writes no started line. It does not raise ModelRetry. (best practice) [sources: src/devops_cli/roadmap/implement_shell_refusal.py, src/devops_cli/roadmap/implement_tool_log.py]
  • Chosen: Return a refusal result. ShellRefusal already returns a refusal as a result (fix(roadmap): raise or configure implement tool retry budget #3271), and a result reaches the model as a new turn without spending the retry budget of the tool.
  • Rejected: Raise ModelRetry. It spends the retry budget of the tool, so a few repeats end the run with UnexpectedModelBehavior before the step reports the repeats.
• What text does the refusal return?: The masked reason that the log kept for the digest, inside a new MESSAGES.roadmap text. The text names the tool and says to change the call. (best practice) [sources: src/devops_cli/roadmap/implement_tool_log.py]
  • Chosen: The masked reason of the earlier call. logged_reason already masks, escapes and cuts the reason for the Job log, so the model gets a safe diagnosis hint.
  • Rejected: The raw message of the exception. It can hold a path or a secret that the model chose.
  • Rejected: A fixed text with no reason. The model needs the earlier reason to change the call.
• How does the step bound the map and the repeats?: The map holds at most CONST_ROADMAP_IMPLEMENT_REMEMBERED_FAILURES digests and drops the oldest. The count of refused repeats passes CONST_ROADMAP_IMPLEMENT_MAX_REPEATS and ends the run with UnexpectedModelBehavior, as the refusal limit does. (best practice) [sources: src/devops_cli/roadmap/implement_tool_log.py]
  • Chosen: A bounded ordered map and a repeat limit. A long step keeps a fixed amount of memory, and a model that repeats for ever stops with a named cause.
  • Rejected: An unbounded map with no repeat limit. A long step keeps every failure, and the model can repeat until the run ends with no cause.
• Does a refused repeat count as a retry?: No. It has its own counter and its own log line, with the outcome repeat. (best practice) [sources: src/devops_cli/roadmap/implement_tool_log.py]
  • Chosen: A separate counter and the outcome repeat. The retry counter and the per-tool reasons then hold the failures that the model caused, not the guard's text.
  • Rejected: Count it in retried. The stop reason of the tool would show the guard's text in place of the failure that ended the run.

Fit

Fits in one pull request.

Suspected block

None.

Dependencies

None.

Alternatives check of 2026-11-07: in-use.
The item already names this tool.
Tool: pydantic/pydantic-ai (https://github.com/pydantic/pydantic-ai), MIT, last activity 2026-11-07, maintained.