[BUG] MUT-023: mutiny test exits 0 on PASS + SKIPPED

[CodewithJha]

Problem

Invalid JSON regression files are SKIPPED. mutiny test exits 1 only if skipped and not passed. PASS + SKIPPED → exit 0. Corrupt files can hide in CI if any other case passes. The SKIPPED path itself is weakly tested (MUT-027).

#39 (merged) only formatted the summary line (N passed, N failed, N skipped). It did not change this exit-code rule. test_cmd.py still has if skipped and not passed: return 1 then return 0.

Why it matters

CI green with a broken regression artifact in the suite.

Evidence

• Audit ID: MUT-023 (MEDIUM, confirmed).
• packages/mutiny_cli/src/mutiny_cli/test_cmd.py run_tests L200–204; discover_regressions SKIPPED on load error.

Reproduction

Code path as written. Audit catalogued this as good first issue material. Not independently re-run in this filing pass.

Expected behavior

Load errors are FAIL, or exit 1 if any SKIPPED unless --allow-skip.

Suggested implementation direction

Treat corrupt JSON as FAIL (simplest) or add --allow-skip and default to non-zero when skipped > 0. Add a unit test with a temp invalid JSON file plus one valid PASS.

Acceptance criteria

• A suite with one PASS fixture and one unreadable JSON does not exit 0 (unless --allow-skip is explicitly used).
• All-PASS still exits 0; all-FAIL still exits 1; skip-only still non-zero.
• Unit test covers the SKIPPED/corrupt path.
• Summary line from cli: clearer PASS/FAIL/SKIPPED summary line for mutiny test #39 remains readable.

Scope

test_cmd.py + tests. Out of scope: broader CLI test coverage (MUT-027).

Related audit findings

MUT-023. Related: MUT-027, #39 (summary format — done).

Contributor notes

Generated from the Mutiny deep audit.

good first issue. help wanted.

[CodewithJha]

Landed fix is now #88 (f552b23). #87 was correct and was reverted in #90 so this equally valid --allow-skip approach could be the one on main.