Mutation testing

A passing test suite is only reassuring if the tests can actually fail. Mutation testing asks that question directly: inject a fault, and see whether a test dies.

It is opt-in. You run it when you want to know, and no gate reads its result. A unit reaches Done on its acceptance criteria and its review; nothing asks it for a mutation run.

How it works

mutation.py run --story US0051 --test "pytest -q"       # the story's declared surface
mutation.py run --since HEAD~1 --test "npm test"        # what changed since a ref
mutation.py run --files src/loader.py --test "pytest -q"

It mutates the surface - flips a comparison, drops a guard, stubs a return - and re-runs the tests once per mutation. Each mutant is reported killed (a test caught it) or survived (nothing did). A survivor is a finding worth reading: the test stayed green over broken code, so that path is not really under test.

Three more verbs sit beside it: mutation.py yield --run <id> reads back what one run cost and what was filed from it, mutation.py prefilter lists test files with no recognisable assertion, and mutation.py window marks source as being rewritten in place, so a concurrent commit is refused rather than staging a half-mutated tree.

Honest by construction

• A red baseline refuses the run. No mutant is applied over tests that already fail, because every one would read as an error and prove nothing.
• An un-mutatable surface reads un-checked, never as a silent pass, and mutants beyond the cost ceiling are counted as truncated, not clean.
• A mutant that cannot parse is unviable, evidence of nothing - any suite fails on it, however vacuous.
• The original bytes always come back. Each file is saved before its mutant lands, so even a killed process leaves nothing stranded: the next run restores it first.
• The bytecode cache is purged, so a same-length mutant cannot reuse a cached .pyc and report SURVIVED without ever running.

Tip

The discipline in one line: mutate the call site, purge the cache, and assert the patch changed the file. A mutation harness that does not do all three can report a kill it never earned.

Why it is not a gate

v5 kept a per-target record of hand-applied mutants and could hold a repair until its mutants were registered. v6 retires both: mutation.py register, retract, retractions and audit are refused by name, and gate.py --only mutation is refused as an unknown lane. The record grew into paperwork about tests rather than a check on them.

What stays is the question, asked when it is worth asking: before trusting a guard that decides something expensive, or when a review suspects a test cannot fail. The run writes the latest report to sdlc-studio/.local/mutation-report.json and appends one row per run to a series you can read back.

Where it fits

Mutation testing is a backstop you reach for under executable acceptance criteria. The criteria prove the behaviour is present; a mutation run proves the test would notice if it broke. A guard worth trusting is a guard whose mutant was killed.

See also

• Executable acceptance criteria - the tests a mutation run probes.
• Specs, code and environments - naming the mutant before writing the test.
• Reconcile & drift - keeping the evidence honest.