Mutation Testing: Verifying Test Effectiveness

Mutation Testing: Verifying Test Effectiveness

Mutation testing verifies the effectiveness of a test suite by introducing small, deliberate bugs ("mutations") into the source code and checking whether the test suite catches them. A test suite that fails to catch mutations is providing false confidence — the code is covered but the tests aren't actually verifying correct behaviour.

How It Works

A mutation testing tool automatically creates many slightly-modified versions of the code ("mutants") — changing a + to -, flipping a true to false, removing a return statement. It then runs the test suite against each mutant. A mutant that is caught (causes a test to fail) is "killed." A mutant that is not caught (all tests still pass despite the bug) is a "survivor." The mutation score is the percentage of mutants killed.

What Surviving Mutants Reveal

• Tests that assert the wrong thing — verifying a side effect rather than the actual return value
• Untested branches — conditions that execute but whose correctness isn't verified
• Weak assertions — tests that pass regardless of value (e.g., assert(result !== null) instead of assert(result === 42))

Tools

• Stryker: Mutation testing for JavaScript/TypeScript, C#, Scala
• PIT: Java mutation testing — widely used in Java projects
• mutmut: Python mutation testing

Practical Use

Mutation testing is computationally expensive — running full mutation testing on every CI build is usually impractical. Use it periodically, on critical modules, or incrementally (only mutate changed code). Target >80% mutation score for business-critical logic.