Do your tests actually catch bugs? mutprobe is a small, dependency-free mutation testing tool written in Python that checks test suite effectiveness with mutants you choose. You list exact code edits ("break this comparison", "drop this guard"), and mutprobe applies them one at a time in a scratch copy, runs your test command, and tells you which bugs your tests caught and which slipped through. It is language-agnostic: anything with a test command works (pytest, unittest, Jest, Vitest, go test, cargo test, a shell script). It fits in CI as a test quality gate, because it exits non-zero when a mutant you expected to be caught survives.
$ mutprobe examples/basic/mutants.json --quiet
mutant status expected ok failing tests
lower-bound-flipped KILLED killed yes test_clamp_above (test_calc.ClampTests), test_clamp_below (test_calc.ClampTests), test_clamp_inside (test_calc.ClampTests)
even-inverted KILLED killed yes test_even (test_calc.EvenTests), test_odd (test_calc.EvenTests)
range-check-too-strict SURVIVED killed NO
killed 2/3 baseline passed: True root unchanged: True
The surviving mutant shows a real gap: no test calls clamp with low == high. That is why this example exits with code 1 on purpose: an unexpected survivor is exactly what mutprobe exists to report.
Tools like mutmut (Python), Stryker (JavaScript/TypeScript, C#, Scala) and PIT (Java) generate thousands of mutants automatically from a language's syntax tree. That is great for a broad score, but slow, tied to one language, and noisy with equivalent mutants. mutprobe does the opposite. It runs a short list of hand-picked, targeted mutants aimed at the lines that matter: a permission check, a boundary, a retry limit, a "never send twice" guard. It needs no parser, so it works with any language and any test runner. It can require that a specific test fails (expect_failures), so a mutant only counts as caught when the test written for it catches it. Use it when you want to prove that particular safety checks are tested, not to compute a coverage-style percentage.
Python 3.9+ and no third-party dependencies.
pip install . # from a clone; provides the `mutprobe` command
python -m mutprobe ... # or run without installing
- Make sure your tests pass on the unmodified code.
- Write a spec file,
mutants.json:
{
"root": ".",
"test": ["python3", "-m", "pytest", "-q"],
"failure_pattern": "pytest",
"timeout": 300,
"mutants": [
{
"name": "allow-negative-amount",
"file": "src/payments.py",
"find": "if amount <= 0:",
"replace": "if amount < 0:",
"expect_failures": ["test_rejects_zero_amount"]
},
{
"name": "skip-duplicate-check",
"file": "src/payments.py",
"find": " if already_sent(order_id):\n return\n",
"replace": ""
}
]
}- Run
mutprobe mutants.json(add--json report.jsonfor a machine-readable report).
| key | required | meaning |
|---|---|---|
root |
no (default .) |
project directory, relative to the spec file |
test |
yes | test command as an argument list (no shell string) |
build |
no | command run before the tests on every mutant (for example ["npx", "tsc", "-b"]); a failing build counts as BROKE |
timeout |
no (default 600) | seconds per command |
failure_pattern |
no, but strongly recommended | preset (unittest, pytest, vitest, jest, go) or a regex with one capture group that extracts a failing test's name from the output. Without it, any non-zero exit counts as KILLED, including syntax errors and import errors, so broken mutants inflate the result; mutprobe prints a warning when it is missing |
ignore |
no | glob patterns left out of the scratch copy and of the root hash (default .git, __pycache__, .pytest_cache, .mypy_cache) |
mutants[].name |
yes | unique label |
mutants[].file |
yes | path relative to root |
mutants[].find / replace |
yes | exact text to replace; find must occur exactly count times (default 1) |
mutants[].expect |
no | killed (default) or survived, to document a known, accepted survivor |
mutants[].expect_failures |
no | substrings of test names; the mutant only counts as KILLED if one of these tests failed (needs failure_pattern) |
TOML specs (.toml) work on Python 3.11+.
| status | meaning |
|---|---|
| KILLED | the tests failed (and, with a failure_pattern, at least one named test failed) |
| SURVIVED | the tests passed with the bug in place |
| BROKE | non-zero exit with no named test failure (syntax error, import error, failed build), or the wrong tests failed for expect_failures. Needs failure_pattern (except a failed build) |
| TIMEOUT | a command exceeded timeout |
| NOT_APPLIED | the file is missing, is not UTF-8, resolves outside the scratch copy through a symlink, or find did not occur exactly count times |
Exit code 0: baseline passed, every mutant matched its expectation, and the project root is byte-for-byte unchanged. 1: at least one mismatch. 2: invalid spec or arguments. 3: the baseline failed or the root changed.
- Mutations are applied only in a scratch copy (a temporary copy of
root, or an existing copy passed with--workdir). The original tree is hashed before and after the run, and a change fails the run. - Each mutated file is restored after its run and checked by hash.
- Symlinks are copied as links, not followed. A mutant whose file resolves outside the scratch copy (through a symlinked file or directory) is NOT_APPLIED and never written; the same check applies inside a
--workdircopy. The root hash records link targets as text and does not follow links, which is why this check happens before every write. - A command that times out, or is running when mutprobe is interrupted (Ctrl-C) or fails, is killed together with its whole process group (POSIX), so stray child processes cannot keep writing into the copy afterwards.
- Every write gets a distinct modification time, and
PYTHONDONTWRITEBYTECODE=1is set for child processes. This stops mtime-based caches (Python bytecode, incremental builds) from running a previous mutant's code. The bug was real: two same-length edits written in the same second reused stale.pycfiles.
- name: Targeted mutation tests
run: python -m mutprobe tests/mutants.json --json mutation-report.jsonKeep the mutant list next to the code it protects, and add a mutant whenever you fix a bug that a test should have caught.
- It runs only the mutants you write. It does not generate mutants, compute a mutation score for the whole codebase, or find untested code by itself.
- Edits are exact text replacements in UTF-8 files. Reformatting the target line makes the mutant NOT_APPLIED, which is reported rather than silently skipped.
- Mutants run sequentially, and each one runs your full test command. Point
testat the relevant test files to keep runs short. - The temporary copy includes everything not ignored. For very large trees (for example
node_modules), prepare a copy once and pass it with--workdir. A temporary copy is deleted at the end, but a--workdircopy is yours: if mutprobe is killed mid-run (for example with SIGKILL), the file being mutated stays mutated there, so recreate or reset that copy before trusting it again. - The spec runs commands you give it. Treat spec files from others like any other script.
- Failure-name presets parse common runner output formats. If your runner prints differently, supply your own
failure_pattern.
python -m unittest discover -s tests
python -m mutprobe tests/self-mutants.json # mutprobe checking its own test suite
Run these from the repository root. If you use python -I (isolated mode, which drops the current directory from the import path), install first with pip install -e ..
MIT. See LICENSE. Security reports: see SECURITY.md.