Your AI tests pass. Do they check anything?

An assistant can give you a test suite with high coverage in an afternoon. Coverage only tells you which lines ran during the tests. It says nothing about whether any test would notice if those lines were wrong.

Mutation testing does. It breaks your code on purpose, many times, and tells you which breakages your tests missed.

How it works

A mutation tool makes one small change to your code, called a mutant: it flips a < to <=, swaps true for false, deletes a function call or returns an empty value. Then it runs your tests.

If a test fails, the mutant is killed. Good: your tests noticed. If every test still passes, the mutant survived, and you have found a change to your code that no test cares about. The tool repeats this hundreds of times and reports the survivors.

That is the habit we recommend for every new test, "break the code and watch the test fail", done automatically and at scale.

Why it matters more with AI

When tests are generated from the code, they describe what the code does. They run every line, so coverage looks excellent, but their assertions are often weak: checking that a result exists rather than what it is, or asserting on a mock. Mutation testing is very good at exposing exactly that. A suite with near-total coverage and many surviving mutants is a suite that runs your code without checking it.

Try it on one module

Mutation testing is slow, because it runs your tests once per mutant. Don't start with the whole codebase. Pick one module with AI-written tests, ideally one that matters.

• JavaScript and TypeScript: Stryker. Run npm init stryker once to set it up, then npx stryker run.
• Python: mutmut. Run mutmut run, then look at the survivors with mutmut results.
• Java: PIT, usually through its Maven or Gradle plugin.

Check the current setup steps in each tool's documentation, since configuration varies by project.

Reading the results

Open each surviving mutant and ask one question: if someone made this change for real, would it be a bug?

• Yes: you have a missing or weak assertion. Add a test, or tighten one, that fails on this mutant.
• No: some mutants change nothing that matters, such as a log message or an equivalent condition. Mark them and move on.

Don't chase a perfect score. The point is the list of survivors, not the number. Ten minutes on the survivors in one module will tell you more about an AI-written suite than any coverage report.

Make it routine

Once a module is in good shape, run mutation testing on it occasionally, or only on changed files where your tool supports it. Use it when a large batch of generated tests lands. Pair it with the manual habit for every new test, and mutation testing becomes the backstop rather than the only check.

The free Testing with AI guide covers the rest of the workflow: asking for test cases from the requirement, proving each test can fail, and checking for flaky tests.