feat(roadmap): an enterprise-grade plan and implement pipeline (tracker) #3027

What happened

On 2026-11-01 an evaluation of the 37 plans that the stage job made found 34 flawed plans and 1 to reject, and the implement job stopped on a wrong plan at group 1 on at least 12 items and on a scratch file on #2175. The Service planning and implementation functions work as a demo loop. They do not use the tools the project owns, do not measure their quality, give up at the first failure, lose work on a stop, and cannot run long tasks.

What should happen

The pipeline is usable, reliable and robust:

• Every change is testable and verifiable: behavior has a failing test first, other changes have a repeatable verify command (fix(roadmap): the stage job plans a failing test for groups that change no code with behavior #3005), and every run is scored against a golden set (A1, A2).
• Every useful project tool is available to the planner and the implementer, under a policy (B1, C1, F4).
• Long tasks are supported: checkpoints, resumption, progress-based deadlines (D2, E1, E2, E3).
• Agents try to fix a failure before they give up, in bounded rounds, with changes of strategy and of model tier, and may amend the plan (D1, D3, D4).
• Failures are classified, each class has one action, and recovery is proven by fault injection (F1, F5).
• A person sees what ran, what it cost and what waits (A3, F2, F3, G1, G2).

Acceptance criteria

• Every sub-issue of this tracker is closed.
• devops roadmap eval stage and devops roadmap eval implement print a score, and the scores are above a floor that the owner sets.
• The fault-injection suite passes in CI.
• No item stops on a class that has an automatic action.

Related: tracker for this set (filed with it), #2908 (issue workflow), #2867 (draft PR evaluator), #3005 (task kind), #3010 to #3015 (plan gates), #2919 (stuck-item metrics), #2953 (Ollama parser).

Alternatives check of 2026-11-04: partly.
Tool: OpenHands/OpenHands (https://github.com/OpenHands/OpenHands), MIT, last activity 2026-11-04, maintained.

Proposed design

Problem statement

The roadmap plan and implement pipeline is unreliable. On 2026-11-01, 34 of 37 plans were flawed. The implement job stopped at the first failure on at least 12 items and discarded the work. The pipeline does not use the project's tools, does not measure quality, and cannot run long tasks. This tracker covers 30 sub-issues that make the pipeline production-grade.

Acceptance criteria

Requirement: Eval score printed

The devops roadmap eval stage and devops roadmap eval implement commands SHALL print a score for each run against the golden set.

Scenario: Stage eval prints a score

• WHEN The stage eval command runs against a fixture golden set.
• THEN The command prints a numeric score and exits zero.

Requirement: Eval floor enforced

The eval commands SHALL exit non-zero when a run scores below the floor that the owner sets.

Scenario: Score below the floor

• WHEN A fixture run scores below the configured floor.
• THEN The command exits non-zero and prints the floor.

Requirement: Fault injection in CI

The fault-injection suite SHALL run in CI and SHALL fail the build when an injected fault is not recovered.

Scenario: Injected fault recovers

• WHEN CI runs the fault-injection suite.
• THEN Every injected fault recovers and the suite passes.

Requirement: No stop on automatic class

The implement job SHALL recover from every failure class that has an automatic action and SHALL NOT stop the item.

Scenario: Automatic action recovers

• WHEN A failure of a class with an automatic action occurs during implement.
• THEN The item continues without a stop.

Requirement: Tracker completion

Every sub-issue of this tracker SHALL be closed before the tracker is complete.

Scenario: Open sub-issue keeps the tracker open

• WHEN The tracker completion check runs with one linked sub-issue open.
• THEN The check reports the tracker as not complete.

Existing Tool Considered

The alternatives check of 2026-11-04 found a partial cover. OpenHands/OpenHands (https://github.com/OpenHands/OpenHands) is MIT and maintained, with last activity on 2026-11-04. It covers the implementer runtime, the sandbox, pause and resume, conversation persistence, and a benchmark harness. It lacks the roadmap plan stage that produces OpenSpec plans and applies the plan gates (#3011-#3019). It lacks the devops roadmap eval stage and eval implement commands, the project golden set, and the owner-set floor. It lacks the fault-injection suite (#3075), the failure taxonomy, bounded repair rounds, plan amendment, and the model ladder (#3053, #3057, #3060, #3065). It lacks the needs-input state, run console, dashboards, and alerts (#3069, #3076, #3077). The project builds these parts itself. The written cost is the tokens to build, keep and fix the code until it meets the acceptance criteria. It covers the planner, implementer, runner, service, eval and fault-injection code. It includes the elapsed time across the 30 sub-issues.

Key questions

• Is this item a tracker of sub-issues or one implementation?: It is a tracker. The 30 sub-issues deliver it, and each has one pull request. (best practice) [sources: CONTEXT.md]
  • Chosen: Tracker of sub-issues. CONTEXT.md says the sub-issues deliver an issue that has sub-issues.
  • Rejected: One implementation. One pull request cannot deliver 30 items that each need their own pull request.
• Should the pipeline adopt OpenHands/OpenHands for the implementer runtime?: No. OpenHands covers the implementer runtime and the benchmark harness only. The project keeps its Runner and Service and builds the roadmap parts that the tool lacks. (best practice) [sources: CONTEXT.md]
  • Rejected: Adopt OpenHands as the implementer runtime. It lacks the plan stage, the eval commands, the fault injection, the taxonomy, and the run console.
  • Chosen: Keep the project Runner and Service. CONTEXT.md specifies the Runner sandbox and the Service lanes for the roadmap jobs.
• What score floor must the eval commands exceed?: The owner sets the floor. The commands read it from configuration and fail when a score is below it. Recommended: set the floor from the first measured baseline on the golden set, then raise it as the pipeline improves. (a person decides)

Open questions

• What score floor must the eval commands exceed? (a person decides)

Fit

Needs splitting into 30 items:

1. feat(roadmap): the stage job probes the premise at the staging commit and returns an item whose premise is false #3011 premise probes
2. feat(roadmap): the stage context holds the callers and tests of each cited symbol #3012 callers and tests in the context
3. fix(roadmap): the stage job refuses a spec delta that the changed paths do not need #3013 spec delta rule
4. fix(roadmap): the stage job requires devops ci test verify commands and mirrored test paths #3015 verify form and test paths
5. fix(roadmap): the stage job refuses a plan whose group reads a path that a later group creates #3017 group order
6. feat(roadmap): the stage job maps each acceptance criterion of the issue to a task group #3019 acceptance criteria map
7. test(roadmap): a golden set of plan cases and a scoring command for the stage job #3028 golden set and eval stage
8. test(roadmap): a replayable fixture set and scoring command for the implement job #3030 implement fixtures and eval implement
9. feat(roadmap): a trace of every run, kept and shown in the PR comment and in explain #3032 run trace
10. feat(roadmap): the stage job plans with a read-only investigation toolbox #3037 investigation toolbox
11. feat(roadmap): an independent verifier refutes a plan before the draft pull request opens #3039 independent verifier
12. feat(roadmap): the implement agent can use the project's development tools #3041 project tools for the agent
13. feat(roadmap): behavior groups run fail-to-pass and record the result #3042 fail-to-pass
14. feat(roadmap): the implement job runs the scoped CI before it pushes and repairs failures #3043 scoped CI before the push
15. feat(roadmap): Run devops review on the diff before marking PR ready #3049 devops review before ready
16. feat(roadmap): new tests of an implementation are checked for strength #3051 test strength
17. feat(roadmap): every step repairs before it gives up, in bounded rounds #3053 repair rounds
18. feat(roadmap): a stopped step keeps its work and the next attempt resumes from it #3054 keep work and resume
19. feat(roadmap): the implement agent can amend the plan when the plan is wrong #3057 plan amendment
20. feat(roadmap): a model ladder for retries, with the tier chosen per attempt #3060 model ladder
21. feat(roadmap): durable runs: a step survives an eviction, an OOM or a restart #3062 durable runs
22. feat(roadmap): long-running tasks: progress-based deadlines instead of a fixed two hours #3063 progress-based deadlines
23. feat(roadmap): several implement Jobs can run at once, on disjoint files #3064 parallel Jobs on disjoint files
24. feat(roadmap): a failure taxonomy, and one action for each class #3065 failure taxonomy
25. feat(roadmap): cost and token budgets per item and per day #3066 budgets
26. feat(roadmap): metrics for the pipeline outcomes #3069 dashboards and alerts
27. feat(runner): the runner image carries every development tool, and the shell policy is one table #3072 runner image tools and shell policy table
28. test(roadmap): fault-injection tests prove the pipeline recovers #3075 fault injection
29. feat(roadmap): a run console: follow a run and read its history #3076 run console
30. feat(roadmap): an agent that needs a decision asks on the issue and waits #3077 needs-input state

Suspected block

None.

Dependencies

#3011, #3012, #3013, #3015, #3017, #3019, #3028, #3030, #3032, #3037, #3039, #3041, #3042, #3043, #3049, #3051, #3053, #3054, #3057, #3060, #3062, #3063, #3064, #3065, #3066, #3069, #3072, #3075, #3076, #3077