Repository navigation
test(roadmap): a replayable fixture set and scoring command for the implement job #3030
Description
What happened
The implement job has no measure either. Its outcomes so far: #2735 and #2739 delivered; #2656 and #2747 needed a person; #2175 stopped at group 2 on a scratch file; others stopped at group 1 on a wrong plan. A prompt or tool change cannot be compared with an older one.
What should happen
- Fixtures of small, real items: an issue, a plan, a repository SHA, and the properties that a good result has (tests pass, no deleted test, the diff touches only the group files,
devops ciscope passes). devops roadmap eval implementruns a step in a scratch clone with a chosen model and reports pass or fail by property, tokens and time.
Acceptance criteria
- At least 10 fixtures from merged deliveries and stopped runs, including refactor(cli): the five functions and properties that nothing calls and that #1910 does not list leave src #2175 and fix(security): the subprocess environment guard accepts LESS, which makes less run a program #2735.
- The scoring runs in a scratch clone and touches no GitHub state.
- The report lists tokens, wall time, tool calls and the property that failed.
- A change to the implement prompt or tool set states its score before and after in its PR.
Related: tracker for this set (filed with it), #2908 (issue workflow), #2867 (draft PR evaluator), #3005 (task kind), #3010 to #3015 (plan gates), #2919 (stuck-item metrics), #2953 (Ollama parser).
Activity
dan-petty commentedon Nov 1, 2026
Intake placed this item to the backlog: intake puts every new item in the backlog. A person plans the milestones.
testprefix of the title gives the type.(roadmap)scope of the title gives the scope.It enables regression testing of the implement job but does not block delivery of new features.It lets the team compare prompt and tool changes with a measurable score, reducing risk on the implement pipeline.It needs at least 10 real fixtures, a scratch-clone runner, and a multi-property report, which is bounded but non-trivial.Both add fixtures and a scoring command under the roadmap eval pattern, but the candidate targets the implement job and #3028 targets the stage job.