dan-petty · Nov 7, 2026 · Nov 7, 2026 · Nov 7, 2026
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-11-07
@@ -0,0 +1,19 @@
+# Design
+
+## Decisions
+
+- **Refuse each `criteria` entry whose `_one_line` text is not in `acceptance_criteria(issue_body)`.**: The prompt tells the model to copy person-written items, and `acceptance_criteria` cuts the refine block off, so the person-written list is the only source. Normalizing with `_one_line` keeps a copy that differs only in spacing valid.
+- **Add the rule to `criterion_problems` and keep `acceptance_criteria` unchanged.**: `criterion_problems` is the one check that the stage job, the implement job and the plan check call on a `criteria` list, and `acceptance_criteria` already returns the person-written criteria in order.
+- **Put the problem text in `MESSAGES.roadmap` as `stage_problem_criterion_unknown`.**: Every problem of the module comes from that table, and the criteria tests assert its exact text.
+- **Give the replay's `paths` gate the issue body of a plan that has a `criteria` list, and no body for a plan that has none.**: `_path_gate` runs `check_proposal`, which holds the criteria rule, and the job gives that rule the body. With no body the new rule would refuse an entry that a person wrote; with a body for a recorded plan that has no list it would refuse a plan that was never asked for one.
+- **Run the gate change in group 1 and the refusal in group 2.**: The replay test at `tests/roadmap/stage/test_roadmap_stage_eval_gates.py:311` calls the `paths` gate on a mapped plan with a body, so the body must reach `check_proposal` before the new rule refuses entries. Each step then keeps one module and one test module.
+- **Add one sentence to the `criteria` bullet of the stage prompt and assert it in the prompt rules test.**: The prompt is the model's only copy of the rule, and a sentence that no test holds can drift from the check.
+
+## Rejected alternatives
+
+- **Refuse only an entry whose text appears in the refine block.**: `_without_refine_block` cuts the block off before `acceptance_criteria` reads the body, so a check of the block would read text that the function already discards, and an invented entry would still pass.
+- **Accept a substring of a criterion as a match.**: A partial text can match a different criterion and weaken the check. The prompt requires the text copied character for character, and the missing-criterion check compares whole texts.
+- **Leave the rule to the prompt and accept an entry that no person wrote.**: The plan for #3320 in #3333 copied five refine-block criteria and every gate accepted it, although the prompt already stated the empty-list rule.
+- **Write the problem text inline in `criterion_problems`.**: `src/devops_cli/roadmap/stage/stage_criteria.py` reads every problem text from `MESSAGES.roadmap`, and `tests/roadmap/stage/test_roadmap_stage_criteria.py` asserts the exact text of a problem.
+- **Let the replay's `paths` gate judge the criteria rule with no issue body.**: `src/devops_cli/roadmap/stage/stage_eval_gates.py:def _path_gate` passes no `issue_body`, so the new rule would refuse the entry of `tests/fixtures/roadmap_eval/stage/3411-a-what-changes-item-that-names/plan.json`, which copies the criterion of `tests/fixtures/roadmap_eval/stage/3411-a-what-changes-item-that-names/issue.txt:13: The stage check refuses a What Changes item that names a repository path`.
+- **Merge the gate change and the refusal into one group.**: One group would hold three source files and two test modules, and a step reads them with the issue, the change and the prompt, which is more than the small step that `budget` sizes. The order of group 1 and group 2 gives the same result with two small steps.
@@ -0,0 +1,55 @@
+# Proposal
+
+## Why
+
+The stage job accepts a `criteria` entry that no person wrote, so the implement job verifies steps against invented criteria and hides that the issue gave none (#3391).
+
+## What Changes
+
+- `criterion_problems` in `src/devops_cli/roadmap/stage/stage_criteria.py` refuses each `criteria` entry whose text is not an acceptance criterion that a person wrote in the issue body.
+- The message `stage_problem_criterion_unknown` in `src/devops_cli/lang/en/messages.py` quotes the entry and says that no person wrote it.
+- `src/devops_cli/roadmap/stage/stage_eval_gates.py:def _path_gate` gives `check_proposal` the issue body of a plan that has a `criteria` list, and no body for a plan that has none.
+- `tests/roadmap/stage/test_roadmap_stage_criteria.py` covers a body with no person-written heading and a body whose only list sits in the refine block.
+- `tests/roadmap/stage/test_roadmap_stage_eval_gates.py` covers the `paths` gate on a plan with a `criteria` list and on a recorded plan with none.
+- `src/devops_cli/ai/tasks/roadmap_stage_change.md` states that the check refuses an entry that no person wrote, and `tests/roadmap/stage/test_roadmap_stage_prompt_rules.py` asserts that sentence.
+- The spec delta of `roadmap-stage` modifies the requirement that refuses an unmapped criterion and the requirement that the replay judges a plan with a `criteria` list.
+- The change fits one pull request.
+
+## Capabilities
+
+### New Capabilities
+
+None.
+
+### Modified Capabilities
+
+- `roadmap-stage`: An unmapped criterion is refused, The replay judges the criteria gate on plans with a criteria list
+
+## Impact
+
+- Code: `src/devops_cli/roadmap/stage/stage_eval_gates.py`, `src/devops_cli/roadmap/stage/stage_criteria.py`, `src/devops_cli/lang/en/messages.py`
+- Documents: `src/devops_cli/ai/tasks/roadmap_stage_change.md`
+- Tests: `tests/roadmap/stage/test_roadmap_stage_eval_gates.py`, `tests/roadmap/stage/test_roadmap_stage_criteria.py`, `tests/roadmap/stage/test_roadmap_stage_prompt_rules.py`
+
+## Feasibility
+
+- The premise holds in the moved module. The probe says "The path `src/devops_cli/roadmap/stage_criteria.py` is no file at the root of the staging commit. These files of the commit are candidates: `src/devops_cli/roadmap/stage/stage_criteria.py`. Commit bff2f14f 2026-11-06 refactor(roadmap): group the job modules in subpackages (#3897) deleted or renamed it." `read_file` of `src/devops_cli/roadmap/stage/stage_criteria.py` shows `src/devops_cli/roadmap/stage/stage_criteria.py:def criterion_problems` building `mapped` from the entry texts and refusing only a missing criterion and a group over the count, so no line compares an entry with the person-written list.
+- The test module moved with the code. The probe says "The path `tests/roadmap/test_roadmap_stage_criteria.py` is no file at the root of the staging commit. These files of the commit are candidates: `tests/roadmap/stage/test_roadmap_stage_criteria.py`." The plan writes the tests in `tests/roadmap/stage/test_roadmap_stage_criteria.py` and plans no code for the old path.
+- The prompt exists. The probe says "The path `src/devops_cli/ai/tasks/roadmap_stage_change.md` is a file of the staging commit." `search_code` of `An issue with no such list gets an empty list` returned one hit, and the file holds `src/devops_cli/ai/tasks/roadmap_stage_change.md:46: An issue with no such list gets an empty list.` beside `The check refuses a reply that leaves one item out.` The empty-list rule stays and the refusal sentence joins it.
+- The replay holds no body for the check. `read_file` of `src/devops_cli/roadmap/stage/stage_eval_gates.py` shows `src/devops_cli/roadmap/stage/stage_eval_gates.py:def _path_gate` calling `check_proposal` with no `issue_body`, while `PlanUnderTest.of_case` sets `issue_body=case.issue_body` and `mapped="criteria" in case.plan`, so a mapped recorded plan holds its body and an unmapped one is not judged.
+- `search_code` of `check_proposal(` shows the callers that hold a real body, `src/devops_cli/roadmap/stage/stage.py:1126: problems = check_proposal(`, `src/devops_cli/roadmap/implement/implement.py:404: problems = check_proposal(` and `src/devops_cli/roadmap/plan_check.py:380: for message in check_proposal(`, and the replay call `src/devops_cli/roadmap/stage/stage_eval_gates.py:161: return check_proposal(` with no body. The rule therefore holds in the job and the plan check, and the replay is the one caller that group 1 repairs.
+- The first review said group 1 needs group 2. The groups are now in the other order: group 1 gives `_path_gate` the issue body, group 2 adds the refusal. `tests/roadmap/stage/test_roadmap_stage_eval_gates.py:311: assert not replay_case(_with_criteria(good, both)).refused` passes after group 1 alone, because the old rule reads a body whose two criteria the entries copy, and it still passes after group 2 for the same reason.
+- The second review is right about case 3174, and the earlier line of this plan was wrong. Its one defect is pinned to no gate at `tests/fixtures/roadmap_eval/stage/3174-a-plan-with-a-placeholder-group/case.json:29: "gate": "none"`, and `_status` returns `UNJUDGED` for that gate, so the pinned counts hold through the `UNJUDGED` path and not through the `content` gate. The plan of 3174 holds `tests/fixtures/roadmap_eval/stage/3174-a-plan-with-a-placeholder-group/plan.json:64: "criteria": []` and its issue body holds three criteria, so the `criteria` gate already refuses it and the `paths` gate names the same three criteria after group 1. Its verdict is `reject`, so the replay fails nothing and no pinned count in `tests/roadmap/stage/test_roadmap_stage_eval_gates.py` changes.
+- Case 3411 keeps its verdict. Its entry copies a person-written criterion: `tests/fixtures/roadmap_eval/stage/3411-a-what-changes-item-that-names/issue.txt:13: The stage check refuses a What Changes item that names a repository path` and `tests/fixtures/roadmap_eval/stage/3411-a-what-changes-item-that-names/plan.json:70: "criteria": [`. So the `paths` gate finds no unknown-entry problem for that entry and still names the missing What Changes path.
+- `docs/ROADMAP_WORKFLOW.md:314: A pull request that changes the stage prompt` holds the rule that such a plan states the replay score before and after, so the task of group 3 holds `uv run devops roadmap eval stage --replay` with the words score, before and after.
+- No call of the plan needs a checkout of the repository, a client of a service or a network: `criterion_problems` and `acceptance_criteria` take strings, and `_path_gate` takes a `PlanUnderTest` whose body and tree the replay reads from the recorded case files.
+- The first checks against the real system: `uv run devops ci test tests/roadmap/stage/test_roadmap_stage_eval_gates.py`, then `uv run devops ci test tests/roadmap/stage/test_roadmap_stage_criteria.py`, then `uv run devops ci test tests/roadmap/stage/test_roadmap_stage_prompt_rules.py`, then `uv run devops roadmap eval stage --replay` with the score before and after the change.
+- Pull request #3989 changed `src/devops_cli/ai/tasks/roadmap_stage_change.md` too and merged, so this plan adds the refusal sentence beside the merged changes.
+
+## Existing Tool Considered
+
+No open, maintained tool covers the need, because the rule compares a reply with the text of one issue of this repository and belongs in the existing `criterion_problems`. The change builds about 4k tokens of code to keep and fix, in about 1 hour, and deletes no internal code: `acceptance_criteria`, `criterion_problems` and `_path_gate` stay.
+
+## Dependencies
+
+- None. Pull request #3989 merged into main, so no pull request blocks this change.
@@ -0,0 +1,39 @@
+## MODIFIED Requirements
+
+### Requirement: An unmapped criterion is refused
+`check_proposal` SHALL refuse a reply that leaves an acceptance criterion of the issue with no entry in `criteria`, an entry that names a group the plan does not have, and an entry whose text is not an acceptance criterion that a person wrote in the issue body. The problem SHALL quote the exact text of the entry. The check SHALL read the body of the issue with the block of the refine job cut off, so a list inside that block is no criterion. The stage prompt `src/devops_cli/ai/tasks/roadmap_stage_change.md` SHALL state that the check refuses an entry that no person wrote.
+
+#### Scenario: One of three criteria is unmapped
+- **WHEN** The issue has three acceptance criteria and the reply's `criteria` list maps only two
+- **THEN** `check_proposal` returns a problem that quotes the text of the unmapped criterion
+
+#### Scenario: The retry copies the text
+- **WHEN** The first reply leaves a criterion out and the second reply maps it
+- **THEN** the second request to the model holds the problem with the exact text, and the item is staged
+
+#### Scenario: An entry names a group that does not exist
+- **WHEN** An entry of `criteria` names group 3 and the plan has 2 groups
+- **THEN** `check_proposal` returns a problem that names the criterion and the count of groups
+
+#### Scenario: An entry copies a criterion of the refine block
+- **WHEN** The issue body has no `Acceptance criteria` heading outside the refine block and the reply holds one criterion of the refine block
+- **THEN** `check_proposal` returns one problem that quotes the copied criterion and says that no person wrote it
+
+#### Scenario: An issue with no person-written list takes an empty list
+- **WHEN** The issue body has no `Acceptance criteria` heading outside the refine block and the reply's `criteria` list is empty
+- **THEN** `check_proposal` returns no problem for the criteria mapping
+
+### Requirement: The replay judges the criteria gate on plans with a criteria list
+The golden-set replay SHALL run the `criteria` gate on a plan that has a `criteria` list, and SHALL not judge a recorded plan that has none. The `paths` gate SHALL give `check_proposal` the issue body of a plan that has a `criteria` list, so the criteria rule reads the same body as the stage job, and SHALL give no body for a plan that has none.
+
+#### Scenario: A plan leaves a criterion out
+- **WHEN** The replay runs a plan with a `criteria` list that leaves one criterion of its issue out
+- **THEN** the `criteria` gate refuses the plan
+
+#### Scenario: A recorded plan has no list
+- **WHEN** The replay runs a recorded plan with no `criteria` key
+- **THEN** the `criteria` gate finds no problem
+
+#### Scenario: The paths gate reads the body of a plan with a criteria list
+- **WHEN** The replay runs a plan whose `criteria` entry copies a criterion that a person wrote in its issue
+- **THEN** the `paths` gate returns no problem for that entry
@@ -0,0 +1,32 @@
+# Tasks
+
+## 1. Give the replay's paths gate the issue body
+
+Reads: `src/devops_cli/roadmap/stage/stage_eval_gates.py`, `src/devops_cli/roadmap/stage/stage_criteria.py`, `tests/roadmap/stage/test_roadmap_stage_eval_gates.py`. Changes: `src/devops_cli/roadmap/stage/stage_eval_gates.py`, `tests/roadmap/stage/test_roadmap_stage_eval_gates.py`.
+
+- [ ] 1.1 In `tests/roadmap/stage/test_roadmap_stage_eval_gates.py`, build the good case with `_with_criteria(good, [])`: the `paths` gate names the criteria `First.` and `Second.` of `ACCEPTANCE_BODY`. With both entries mapped, the `paths` gate names no criterion, and a case with no `criteria` key and the same body finds no criterion problem at the `paths` gate
+  - Verification: `$ uv run devops ci test tests/roadmap/stage/test_roadmap_stage_eval_gates.py` fails on the base
+- [ ] 1.2 In `src/devops_cli/roadmap/stage/stage_eval_gates.py`, pass `issue_body=plan.issue_body if plan.mapped else ""` to `check_proposal` in `_path_gate`, and state that rule in its docstring
+  - Verification: `$ uv run devops ci test tests/roadmap/stage/test_roadmap_stage_eval_gates.py` passes
+
+## 2. Refuse a criteria entry that no person wrote
+
+Reads: `src/devops_cli/roadmap/stage/stage_criteria.py`, `src/devops_cli/lang/en/messages.py`, `tests/roadmap/stage/test_roadmap_stage_criteria.py`. Changes: `src/devops_cli/roadmap/stage/stage_criteria.py`, `src/devops_cli/lang/en/messages.py`, `tests/roadmap/stage/test_roadmap_stage_criteria.py`.
+
+- [ ] 2.1 In `tests/roadmap/stage/test_roadmap_stage_criteria.py`, write a test with a body whose only `Acceptance criteria` heading sits in the refine block and a reply that holds one criterion of that block: `criterion_problems` returns one problem that quotes the copied text. Add a second test with the same body and an empty `criteria` list: `criterion_problems` returns no problem
+  - Verification: `$ uv run devops ci test tests/roadmap/stage/test_roadmap_stage_criteria.py` fails on the base
+- [ ] 2.2 In `src/devops_cli/roadmap/stage/stage_criteria.py`, make `criterion_problems` add `MESSAGES.roadmap.stage_problem_criterion_unknown` for each entry whose `_one_line` text is not in `acceptance_criteria(issue_body)`, after the problems for a missing criterion. Add `stage_problem_criterion_unknown` to `src/devops_cli/lang/en/messages.py`: it quotes the entry, says that no person wrote it in the issue body, and says to copy the text of a top-level item under the heading `Acceptance criteria` or to give an empty list
+  - Verification: `$ uv run devops ci test tests/roadmap/stage/test_roadmap_stage_criteria.py` passes
+
+## 3. State the refusal in the stage prompt
+
+Reads: `src/devops_cli/ai/tasks/roadmap_stage_change.md`, `tests/roadmap/stage/test_roadmap_stage_prompt_rules.py`. Changes: `src/devops_cli/ai/tasks/roadmap_stage_change.md`, `tests/roadmap/stage/test_roadmap_stage_prompt_rules.py`.
+
+- [ ] 3.1 In `tests/roadmap/stage/test_roadmap_stage_prompt_rules.py`, assert that the stage prompt states that the check refuses a criteria entry that no person wrote
+  - Verification: `$ uv run devops ci test tests/roadmap/stage/test_roadmap_stage_prompt_rules.py` fails on the base
+- [ ] 3.2 In `src/devops_cli/ai/tasks/roadmap_stage_change.md`, add to the `criteria` bullet that the check refuses an entry whose text is not one of the person-written criteria, beside the rule that an issue with no such list gets an empty list. Run `uv run devops roadmap eval stage --replay` and state the score before and after the change
+  - Verification: `$ uv run devops ci test tests/roadmap/stage/test_roadmap_stage_prompt_rules.py` passes
+
+## Workflow follow-up
+
+- Pending a person: review the diff in tuicr (tuicr -r <base>..HEAD).