fix: fail regression runs when any selected case is skipped

[fatihcvs]

Summary

Fixes #71. Return exit code 1 whenever a selected regression is skipped, including mixed PASS + SKIPPED runs. Corrupt artifacts and replay errors keep their SKIPPED report status, but can no longer leave CI green just because another case passed. No new bypass flag is introduced.

Add real temporary-file regressions for malformed JSON, invalid schema and invalid UTF-8, both alone and alongside a passing fixture. Check the CLI exit code, human summary and persisted error details, plus mixed-result JSON output. Document the exit behavior in the CLI guide.

Type of change

• Bug fix
• Tests
• CLI / DX

Checklist

• Read CONTRIBUTING.md
• Linked related issue
• Core stays free of framework SDK imports
• Behavioral tests added and full unit suite run
• Documentation matches implemented behavior
• One concern per PR
• Authorized offline testing only

Test plan

• Before the fix, the focused module had 4 failures and 13 passes; the mixed corrupt/valid cases incorrectly returned 0.
• pytest tests/unit tests/integration tests/reliability -q --tb=short: 523 passed, one existing Starlette deprecation warning, on Windows/Python 3.12.14 with MUTINY_SAMPLE_OFFLINE=1 and PYTHONUTF8=1.
• Existing all-PASS, all-FAIL and no-regression behavior remains covered; new skip-only cases return 1.
• git diff --check passed. The existing tracked trace generated by the unit suite was restored after testing (test: isolate persisted violation traces from committed examples #85 handles it separately).

Independent branch from main, without the other open Mutiny contributions.

[vercel]

@fatihcvs is attempting to deploy a commit to the priyanshu's projects Team on Vercel.

A member of the Team first needs to authorize it.

[CodewithJha]

Thanks @fatihcvs — this hits #71 / MUT-023 cleanly: mutiny test now exits 1 whenever any selected case is skipped, including PASS + corrupt JSON. Status stays SKIPPED (still visible in the #39 summary line) but CI cannot go green on a broken artifact. CLI.md documents the exit rule; tests cover malformed JSON, invalid schema, and invalid UTF-8, mixed with a passing fixture. GitHub Actions green on 3.11/3.12; ignoring Vercel auth.

[CodewithJha]

Merged — thank you @fatihcvs. Exactly the kind of CLI correctness Mutiny needs: a mixed PASS + SKIPPED suite can no longer leave CI green, the #39 summary line still reports skips honestly, and corrupt JSON / invalid UTF-8 are covered.

If you'd like to continue working on Mutiny, these would be good related areas to look at:

• #57 — mutiny run still tracebacks on adapter/config errors
• #75 — broader CLI test coverage beyond the skip/exit path
• #58 — env parsing (OPENAI_API_KEY / MUTINY_LLM_TIMEOUT) fail-closed behavior

No pressure — only if you want another one.

[CodewithJha]

Quick follow-up @fatihcvs — we are reverting this squash merge (836e9fe) in #90 so we can land the equally valid #88 from @littfed.

Your implementation was correct: mixed PASS + SKIPPED now fails CI, the #39 summary stays honest, and the corrupt JSON / UTF-8 coverage was real. We are not changing that product default. #88 adds an opt-in --allow-skip hatch on the same fail-closed rule, and swapping lets a second first-time contributor land.

Your other merges (#83, #85, #86, #89) stay on main. If you want another CLI/Core bite later, #57 / #75 / #65 are still open.

No action needed on your side.