Skip to content

feat(roadmap): stage requires workflow contract tests for workflow changes #4083

Open

Description

@dan-petty

Problem or Motivation

The stage planning job permits a task group to change a workflow file under .github/workflows/ without requiring a workflow contract test in tests/invariants/test_workflow_contracts.py.

In issue #2163 (pull request #4031), the stage planner placed the .github/workflows/ci.yml edit in Group 2 with the verification command uv run devops ci --check --only actionlint,zizmor, and deferred the contract test to Group 3.
This design has two defects:

  1. It breaks the test-first rule of AGENTS.md (Rule 4: "Test new behavior first"): Group 2 modified behavior without an executable test proving the change.
  2. The verification command uv run devops ci --check --only actionlint,zizmor is vacuous for validating the added step because it passes on the base before the change and only verifies syntax, not step configuration or existence.

Proposed Solution & Architectural Design

  1. Extend path_problems in src/devops_cli/roadmap/stage/stage_paths.py:
    • When a task group lists a workflow file (.github/workflows/*.yml) in changes, require that the same group (or change) lists tests/invariants/test_workflow_contracts.py in changes.
  2. Update the stage planner prompt (src/devops_cli/ai/tasks/roadmap_stage_change.md) to instruct the model that workflow changes require an invariant contract test in the same group.
  3. This prevents vacuous verification commands on workflow modifications and ensures test-driven implementation for CI automation.

Target Subsystem

AI & Agentic Review Pipeline

Existing Tool Considered

path_problems in src/devops_cli/roadmap/stage/stage_paths.py. The change extends existing path validation rules: approximately 8,000 tokens to build and test.

Standards Compliance

  • I confirm this proposal aligns with AGENTS.md engineering principles (cyclomatic complexity <= 10, nesting <= 5, zero bare exceptions, tests that cover the changed lines).
  • I am willing to submit a pull request implementing this feature with accompanying tests.

Source: #4031

Activity

  1. added
    type/featureNew user-facing functionality or capability addition
    scope/roadmapThe roadmap jobs: intake, refine, stage, implement and the Service that runs them
    status/newStatus New: the item is not ready yet
    effort/lowEffort Low: little work
    on Nov 7, 2026
  2. dan-petty commented on Nov 7, 2026

    @dan-petty
    OwnerAuthor

    Intake placed this item to the backlog: intake puts every new item in the backlog. A person plans the milestones.

    • Type: type/feature. The feat prefix of the title gives the type.
    • Scope: scope/roadmap. The (roadmap) scope of the title gives the scope.
    • Priority: P2-Medium. It closes a gap in stage validation but does not block a release or fix a user-facing defect.
    • Value: Medium. It prevents one class of vacuous workflow verifications in the roadmap pipeline.
    • Effort: Low. It extends an existing path_problems rule and a prompt file; the body estimates about 8,000 tokens of work.
    • Related: fix(roadmap): stage includes perimeter tests for network policies #3972, which asks for a different change. Both extend path_problems to require a specific test file in the changes list when a specific file type is changed; #3972 targets network policy files, the candidate targets workflow files.
  3. added
    priority/p2-mediumStandard priority feature work and architectural improvements
    on Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    effort/lowEffort Low: little workpriority/p2-mediumStandard priority feature work and architectural improvementsscope/roadmapThe roadmap jobs: intake, refine, stage, implement and the Service that runs themsource/agentFiled by an agentstatus/newStatus New: the item is not ready yettype/featureNew user-facing functionality or capability additionvalue/mediumValue Medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions