Skip to content

About

Targeted mutation testing for any language: break the lines that matter (auth checks, boundaries, retry limits) and see if your tests catch it. Python CLI, zero dependencies, CI-ready.

Topics

Resources

Security policy

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

Repository files navigation

targeted-mutation-testing

CI

Do your tests actually catch bugs? mutprobe is a small, dependency-free mutation testing tool written in Python that checks test suite effectiveness with mutants you choose. You list exact code edits ("break this comparison", "drop this guard"), and mutprobe applies them one at a time in a scratch copy, runs your test command, and tells you which bugs your tests caught and which slipped through. It is language-agnostic: anything with a test command works (pytest, unittest, Jest, Vitest, go test, cargo test, a shell script). It fits in CI as a test quality gate, because it exits non-zero when a mutant you expected to be caught survives.

$ mutprobe examples/basic/mutants.json --quiet
mutant                  status    expected  ok   failing tests
lower-bound-flipped     KILLED    killed    yes  test_clamp_above (test_calc.ClampTests), test_clamp_below (test_calc.ClampTests), test_clamp_inside (test_calc.ClampTests)
even-inverted           KILLED    killed    yes  test_even (test_calc.EvenTests), test_odd (test_calc.EvenTests)
range-check-too-strict  SURVIVED  killed    NO

killed 2/3  baseline passed: True  root unchanged: True

The surviving mutant shows a real gap: no test calls clamp with low == high. That is why this example exits with code 1 on purpose: an unexpected survivor is exactly what mutprobe exists to report.

How it differs from mutmut, Stryker, PIT

Tools like mutmut (Python), Stryker (JavaScript/TypeScript, C#, Scala) and PIT (Java) generate thousands of mutants automatically from a language's syntax tree. That is great for a broad score, but slow, tied to one language, and noisy with equivalent mutants. mutprobe does the opposite. It runs a short list of hand-picked, targeted mutants aimed at the lines that matter: a permission check, a boundary, a retry limit, a "never send twice" guard. It needs no parser, so it works with any language and any test runner. It can require that a specific test fails (expect_failures), so a mutant only counts as caught when the test written for it catches it. Use it when you want to prove that particular safety checks are tested, not to compute a coverage-style percentage.

Install

Python 3.9+ and no third-party dependencies.

pip install .            # from a clone; provides the `mutprobe` command
python -m mutprobe ...   # or run without installing

Quick start

  1. Make sure your tests pass on the unmodified code.
  2. Write a spec file, mutants.json:
{
  "root": ".",
  "test": ["python3", "-m", "pytest", "-q"],
  "failure_pattern": "pytest",
  "timeout": 300,
  "mutants": [
    {
      "name": "allow-negative-amount",
      "file": "src/payments.py",
      "find": "if amount <= 0:",
      "replace": "if amount < 0:",
      "expect_failures": ["test_rejects_zero_amount"]
    },
    {
      "name": "skip-duplicate-check",
      "file": "src/payments.py",
      "find": "    if already_sent(order_id):\n        return\n",
      "replace": ""
    }
  ]
}
  1. Run mutprobe mutants.json (add --json report.json for a machine-readable report).

Spec reference

key required meaning
root no (default .) project directory, relative to the spec file
test yes test command as an argument list (no shell string)
build no command run before the tests on every mutant (for example ["npx", "tsc", "-b"]); a failing build counts as BROKE
timeout no (default 600) seconds per command
failure_pattern no, but strongly recommended preset (unittest, pytest, vitest, jest, go) or a regex with one capture group that extracts a failing test's name from the output. Without it, any non-zero exit counts as KILLED, including syntax errors and import errors, so broken mutants inflate the result; mutprobe prints a warning when it is missing
ignore no glob patterns left out of the scratch copy and of the root hash (default .git, __pycache__, .pytest_cache, .mypy_cache)
mutants[].name yes unique label
mutants[].file yes path relative to root
mutants[].find / replace yes exact text to replace; find must occur exactly count times (default 1)
mutants[].expect no killed (default) or survived, to document a known, accepted survivor
mutants[].expect_failures no substrings of test names; the mutant only counts as KILLED if one of these tests failed (needs failure_pattern)

TOML specs (.toml) work on Python 3.11+.

Results and exit codes

status meaning
KILLED the tests failed (and, with a failure_pattern, at least one named test failed)
SURVIVED the tests passed with the bug in place
BROKE non-zero exit with no named test failure (syntax error, import error, failed build), or the wrong tests failed for expect_failures. Needs failure_pattern (except a failed build)
TIMEOUT a command exceeded timeout
NOT_APPLIED the file is missing, is not UTF-8, resolves outside the scratch copy through a symlink, or find did not occur exactly count times

Exit code 0: baseline passed, every mutant matched its expectation, and the project root is byte-for-byte unchanged. 1: at least one mismatch. 2: invalid spec or arguments. 3: the baseline failed or the root changed.

Safety properties

  • Mutations are applied only in a scratch copy (a temporary copy of root, or an existing copy passed with --workdir). The original tree is hashed before and after the run, and a change fails the run.
  • Each mutated file is restored after its run and checked by hash.
  • Symlinks are copied as links, not followed. A mutant whose file resolves outside the scratch copy (through a symlinked file or directory) is NOT_APPLIED and never written; the same check applies inside a --workdir copy. The root hash records link targets as text and does not follow links, which is why this check happens before every write.
  • A command that times out, or is running when mutprobe is interrupted (Ctrl-C) or fails, is killed together with its whole process group (POSIX), so stray child processes cannot keep writing into the copy afterwards.
  • Every write gets a distinct modification time, and PYTHONDONTWRITEBYTECODE=1 is set for child processes. This stops mtime-based caches (Python bytecode, incremental builds) from running a previous mutant's code. The bug was real: two same-length edits written in the same second reused stale .pyc files.

Using it in CI

- name: Targeted mutation tests
  run: python -m mutprobe tests/mutants.json --json mutation-report.json

Keep the mutant list next to the code it protects, and add a mutant whenever you fix a bug that a test should have caught.

Limitations

  • It runs only the mutants you write. It does not generate mutants, compute a mutation score for the whole codebase, or find untested code by itself.
  • Edits are exact text replacements in UTF-8 files. Reformatting the target line makes the mutant NOT_APPLIED, which is reported rather than silently skipped.
  • Mutants run sequentially, and each one runs your full test command. Point test at the relevant test files to keep runs short.
  • The temporary copy includes everything not ignored. For very large trees (for example node_modules), prepare a copy once and pass it with --workdir. A temporary copy is deleted at the end, but a --workdir copy is yours: if mutprobe is killed mid-run (for example with SIGKILL), the file being mutated stays mutated there, so recreate or reset that copy before trusting it again.
  • The spec runs commands you give it. Treat spec files from others like any other script.
  • Failure-name presets parse common runner output formats. If your runner prints differently, supply your own failure_pattern.

Testing this project

python -m unittest discover -s tests
python -m mutprobe tests/self-mutants.json    # mutprobe checking its own test suite

Run these from the repository root. If you use python -I (isolated mode, which drops the current directory from the import path), install first with pip install -e ..

License

MIT. See LICENSE. Security reports: see SECURITY.md.

About

Targeted mutation testing for any language: break the lines that matter (auth checks, boundaries, retry limits) and see if your tests catch it. Python CLI, zero dependencies, CI-ready.

Topics

Resources

Security policy

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages