fix(cli): exit non-zero when tests have skipped cases unless --allow-skip (#71)

[littfed]

Summary

Fixes #71 (MUT-023).

Changes

• Updated run_tests in packages/mutiny_cli/src/mutiny_cli/test_cmd.py to treat skipped test cases as non-zero exit unless --allow-skip is explicitly enabled.
• Added --allow-skip argument to mutiny test subcommand in packages/mutiny_cli/src/mutiny_cli/main.py.
• All-PASS still exits 0, all-FAIL exits 1, and skip-only remains non-zero.
• Added comprehensive unit tests in tests/unit/test_mutiny_cli_test.py covering PASS + corrupt JSON fixture with and without --allow-skip, as well as skip-only suites and summary line formatting.

[vercel]

@littfed is attempting to deploy a commit to the priyanshu's projects Team on Vercel.

A member of the Team first needs to authorize it.

[CodewithJha]

Thanks @littfed — I reviewed this against #71 / MUT-023.

The core diagnosis is right: PASS + SKIPPED currently exits 0, and your tests (corrupt JSON mixed with a passing fixture, skip-only, and CLI --allow-skip wiring) would lock the intended CI behavior. GitHub Actions were green on 3.11/3.12.

We're landing the same default in #87: any skip → exit 1, without a new public flag. #71 allowed either treating load errors as failure or adding --allow-skip; #87 also documents the exit rule in CLI.md and covers extra corrupt encodings. Adding --allow-skip without a CLI.md / flag-table update would leave the new switch undocumented.

I'll close this as a duplicate of #87 so you don't keep rebasing against a conflicting test_cmd.py. The --allow-skip escape hatch is a reasonable follow-up if you want it later, with docs — not required for #71.

[CodewithJha]

Closing as a duplicate of #87, which is now on main and closes #71.

Thank you @littfed — the diagnosis and the mixed PASS + corrupt-JSON tests were correct. We merged the no-new-flag path because it was documented and already covered the extra corrupt encodings; your --allow-skip wiring is a solid optional escape hatch if you want to propose it as a small follow-up with CLI.md.

If you'd like to continue working on Mutiny, these would be good related areas to look at:

• #57 — mutiny run still tracebacks on adapter/config errors (same CLI fail-closed mindset)
• #75 — CLI / reliability tests still miss several product paths
• #58 — env parsing (OPENAI_API_KEY / MUTINY_LLM_TIMEOUT) should fail closed

No pressure — only if you want another one.

[CodewithJha]

Thanks @littfed — this hits #71 / MUT-023 cleanly: mutiny test exits 1 on any unexpected skip (including PASS + corrupt JSON), skip-only stays non-zero, and --allow-skip is the explicit hatch for mixed suites. CLI wiring and unit tests cover both paths. We reverted #87 in #90 so this equally valid approach can be the landed fix. GitHub Actions from the original run were green on 3.11/3.12; re-approving the fork workflow after reopen. Ignoring Vercel auth.

[CodewithJha]

Reopened after reverting #87 (579104a). Your branch applies cleanly on current main. Reviewing and landing this as the canonical #71 / MUT-023 fix — same fail-closed default, plus the --allow-skip hatch.

[CodewithJha]

Merged — thank you @littfed. Exactly the kind of CLI fail-closed fix Mutiny needs: mixed PASS + SKIPPED can no longer leave CI green, skip-only stays non-zero, and --allow-skip is the explicit hatch when a local suite has known unreadable fixtures.

If you'd like to continue working on Mutiny, these would be good related areas to look at:

• #57 — mutiny run still tracebacks on adapter/config errors (same CLI fail-closed mindset)
• #75 — CLI / reliability tests still miss several product paths
• #58 — env parsing (OPENAI_API_KEY / MUTINY_LLM_TIMEOUT) should fail closed

No pressure — only if you want another one.