Skip to content

feat(roadmap): behavior groups run fail-to-pass and record the result #3042

New issue
New issue
Closed
#3602
Closed
feat(roadmap): behavior groups run fail-to-pass and record the result#3042
#3602
Labels
effort/mediumEffort Mediumpriority/p2-mediumStandard priority feature work and architectural improvementsscope/roadmapThe roadmap jobs: intake, refine, stage, implement and the Service that runs themsource/agentFiled by an agenttype/featureNew user-facing functionality or capability additionvalue/mediumValue Medium

Description

@dan-petty
dan-petty
opened on Nov 1, 2026

What happened

A group that adds behavior should start with a failing test (the rule of 2026-11-01), but the implement step does not check that the test fails on the base for the right reason, or that it passes after. A test that never failed proves nothing.

What should happen

Acceptance criteria

  • A behavior group whose test passes on the base is refused with a clear message.
  • The PR comment shows the fail-to-pass result of each behavior group.
  • A group of kind other runs its verify command and records the output instead.
  • A test replays the step with a test that passes on the base.

Related: tracker for this set (filed with it), #2908 (issue workflow), #2867 (draft PR evaluator), #3005 (task kind), #3010 to #3015 (plan gates), #2919 (stuck-item metrics), #2953 (Ollama parser).

Proposed design

Problem statement

The implement step does not verify that a behavior group's new tests fail on the base commit before the change, nor that they pass after. A test that never failed proves nothing. The step must run the tests on the base, record the failure, run them after the change, record the pass, and put both results in the trace and the PR comment. A test that passes on the base is a step failure with a repair round.

Acceptance criteria

Requirement: Behavior group fail-to-pass check

A behavior group's new tests MUST fail on the base commit before the change is applied.

Scenario: Test passes on base

  • WHEN A behavior group's test is run on the base commit and exits with a success code
  • THEN The step fails with a clear message and enters a repair round

Requirement: PR comment shows fail-to-pass result

The PR comment SHALL show the fail-to-pass result of each behavior group.

Scenario: Successful fail-to-pass

  • WHEN A behavior group's test fails on the base and passes after the change
  • THEN The PR comment records the failure on the base and the pass after the change

Requirement: Other kind group runs verify command

A group of kind other SHALL run its verify command and record the output in the trace.

Scenario: Other kind group processed

  • WHEN A group of kind other is processed by the implement step
  • THEN The step runs the group's verify command and records its output in the trace

Requirement: Test replays step with passing base test

A test SHALL replay the implement step with a test that passes on the base commit.

Scenario: Replay with passing base test

  • WHEN The test harness replays the implement step with a test that passes on the base commit
  • THEN The step is refused with a clear message

Existing Tool Considered

No check has run. No open, maintained tool or library covers the need of running a group's tests on a base commit, verifying they fail, running them after the change, verifying they pass, and recording both results in the step trace and the PR comment. The project builds this as part of its implement step. The written cost is the tokens to build, keep and fix this verification logic in the implement step until it matches the behavior described in the issue.

Key questions

  • What is the repair round when a test passes on the base?: The step fails with a clear message, and the agent gets one more turn to write a test that fails first. (proposed)
    • Chosen: Step failure with one repair round. The body specifies this as D1. The agent must write a test that fails first.
    • Rejected: Step failure without repair. The body explicitly requires a repair round for this case.

Open questions

  • How does the step determine the test command for a behavior group? (missing verified source)
  • How are the fail-to-pass results recorded? (missing verified source)
  • What does the step do for a group of kind other? (missing verified source)

Fit

Fits in one pull request.

Suspected block

None.

Dependencies

#3005

Alternatives check of 2026-11-04: partly.
Tool: bhaskargurram-ai/proofcheck (https://github.com/bhaskargurram-ai/proofcheck), MIT, last activity 2026-10-19, maintained.

Activity

  1. dan-petty
    added
    source/agentFiled by an agent
    type/featureNew user-facing functionality or capability addition
    scope/roadmapThe roadmap jobs: intake, refine, stage, implement and the Service that runs them
    status/newStatus New: the item is not ready yet
    value/mediumValue Medium
    effort/mediumEffort Medium
    on Nov 1, 2026

  2. dan-petty commented on Nov 1, 2026

    @dan-petty
    dan-petty
    on Nov 1, 2026
    OwnerAuthor

    Intake placed this item to the backlog: intake puts every new item in the backlog. A person plans the milestones.

    • Type: type/feature. The feat prefix of the title gives the type.
    • Scope: scope/roadmap. The (roadmap) scope of the title gives the scope.
    • Priority: P2-Medium. It adds a required verification step that prevents invalid tests from passing silently in the roadmap workflow.
    • Value: Medium. It closes a gap where a test that never failed was accepted, weakening the evidence standard for behavior changes.
    • Effort: Medium. Requires base-commit test execution, result recording in trace and PR comment, a refusal path with repair round, and a replay test.
    • Related: fix(roadmap): the stage job plans a failing test for groups that change no code with behavior #3005, which asks for a different change. Both target behavior groups defined by #3005, but #3005 plans a failing test task and this candidate verifies at implement time that the test actually fails on base and passes after.
  3. dan-petty
    added
    priority/p2-mediumStandard priority feature work and architectural improvements
    on Nov 1, 2026
  4. dan-petty
    added a parent issue on Nov 1, 2026

  5. devops-cli-service commented on Nov 1, 2026

    @devops-cli-service
    devops-cli-serviceBot
    on Nov 1, 2026
    Contributor

    Refine cannot set this item to Ready at aad0115 until a person answers the questions below. Reply in a comment or edit the body, and the next refine round looks at the item again.

    • How does the step determine the test command for a behavior group? (missing verified source)
    • How are the fail-to-pass results recorded? (missing verified source)
    • What does the step do for a group of kind other? (missing verified source)

  6. devops-cli-service commented on Nov 4, 2026

    @devops-cli-service
    devops-cli-serviceBot
    on Nov 4, 2026
    Contributor

    Alternatives check: partly

    Tool: bhaskargurram-ai/proofcheck (https://github.com/bhaskargurram-ai/proofcheck)

    • License: MIT
    • Last activity: 2026-10-19
    • Maintained: yes

    Covers:

    • Runs a pull request's changed test files on the base and on the changed commit and checks that the test fails before and passes after (fail-to-pass evidence).
    • Replays reproducer snippets from the linked issue on both commits to show the change is what makes them pass.
    • Reports the fail-to-pass evidence on the pull request (GitHub Action and a Python package/CLI), including which parts of the diff the evidence depends on.

    Lacks:

    • No step-level integration: it does not fail the roadmap implement step and enter a repair round so the agent must rewrite a test that passes on the base.
    • No per-group model: it has no notion of the roadmap's behavior groups or of a group's base commit, and no run/record of a verify command for a group of kind other.
    • No trace record of the base failure and post-change pass in the implement step's own trace, and no replay test of the implement step with a passing-on-base test.

    Searches run:

    • tool verifies test fails on base commit then passes after change fail-to-pass
    • enforce failing test first TDD verification tool CI
    • "fail-to-pass" test verification tool github python package
    • TDD Guard nizos tool enforces failing tests before implementation
    • proofcheck PR evidence github action fail-to-pass repository
    • "proofcheck-pr-evidence" github repository action
    • pytest plugin assert test fails before fix verify test would have failed on parent commit
    • bhaskargurram-ai proofcheck README how it works base commit fail pass test comment
    • "proofcheck" GitHub Action "fail-to-pass" test command inputs base ref worktree
    • proofcheck-ai pypi fail-to-pass reproducer replay command line

    Sources:

    Checked on 2026-11-04. A person decides what this means for the item. The check changes no label and closes no issue.

  7. dan-petty
    added
    owner/agentOwner Agent: an agentic coder works on the item
    status/in-progressActively being designed or implemented by developer or agent
    and removed
    status/newStatus New: the item is not ready yet
    on Nov 5, 2026
  8. dan-petty
    closed this as completedin 7b8a4a1on Nov 5, 2026
  9. github-actions
    removed
    status/in-progressActively being designed or implemented by developer or agent
    owner/agentOwner Agent: an agentic coder works on the item
    on Nov 5, 2026

  10. devops-cli-service commented on Nov 5, 2026

    @devops-cli-service
    devops-cli-serviceBot
    on Nov 5, 2026
    Contributor

    Released in v0.61.0: https://github.com/dan-petty/devops-cli/releases/tag/v0.61.0

  11. devops-cli-service commented on Nov 5, 2026

    @devops-cli-service
    devops-cli-serviceBot
    on Nov 5, 2026
    Contributor

    Live since 2026-11-05 12:32 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    effort/mediumEffort Mediumpriority/p2-mediumStandard priority feature work and architectural improvementsscope/roadmapThe roadmap jobs: intake, refine, stage, implement and the Service that runs themsource/agentFiled by an agenttype/featureNew user-facing functionality or capability additionvalue/mediumValue Medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions