Repository navigation
test(roadmap): a golden set of plan cases and a scoring command for the stage job #3028
Description
What happened
The stage job has no regression set. On 2026-11-01 a hand run of 37 reviews found 34 flawed plans and 1 to reject. Nothing recorded the cases, so a change of the prompt or the checks ships with no measure.
What should happen
- A committed set of cases under
tests/fixtures/roadmap_eval/stage/: the issue text, the tree at the staging commit (a pinned SHA), the plan that was staged, and the defects a reviewer found (wrong path, false premise, missing criterion, wrong spec delta, wrong verify form, wrong order). devops roadmap eval stageruns the stage code on each case with a chosen model and prints per-defect pass rates and a total score. It makes the model calls through the gateway, so it is an on-demand command and not part ofdevops ci.- A replay mode with recorded replies runs offline in CI and checks the deterministic gates only.
Acceptance criteria
- The set holds at least the 37 plans of 2026-11-01, each with its defect list.
-
devops roadmap eval stage --replayruns offline in the test suite and fails when a gate stops refusing a known-bad plan. -
devops roadmap eval stage --model <name>prints a score table and stores it under.data/. - A change to the stage prompt or to
check_proposalstates its score before and after in its PR.
Related: tracker for this set (filed with it), #2908 (issue workflow), #2867 (draft PR evaluator), #3005 (task kind), #3010 to #3015 (plan gates), #2919 (stuck-item metrics), #2953 (Ollama parser).
Activity
dan-petty commentedon Nov 1, 2026
Intake placed this item to the backlog: intake puts every new item in the backlog. A person plans the milestones.
testprefix of the title gives the type.(roadmap)scope of the title gives the scope.It provides regression protection for the stage job but does not block a user-facing delivery.A scored regression set lets the project measure prompt and gate changes, which is important but not directly user-facing.It needs fixture files, a gateway-backed command, a replay mode, and CI wiring.Both concern the quality of the 37 stage plans, but #3027 is the umbrella tracker for the whole pipeline while the candidate asks for a specific test fixture set and an eval scoring command.