Flaky tests need a method, not a retry button.

A test fails in CI, passes when you rerun it, and everyone moves on. Do that enough times and the team stops trusting red builds at all, which is worse than having no tests.

The usual fix is automatic retries. They make the build green and hide the problem. A flaky test is a bug that only shows up sometimes. It might be in the test or in the code, and it deserves the same method as any other bug.

Make it fail on demand

You cannot fix what you cannot trigger. Turn off retries and run the one test many times in a row:

• Playwright: npx playwright test path/to/spec --repeat-each=50 --retries=0
• pytest: install pytest-repeat and add --count=50
• Anything else: a shell loop that stops on the first failure

If it fails once in fifty runs, you have a reproduction. If it never fails alone, that is a clue too.

Isolate it

Run the test on its own, then with the rest of its file, then with the full suite. If it only fails alongside other tests, the cause is shared state: a database row, a file, a global variable or a cached value that another test leaves behind. Try changing the test order or running tests in parallel. Many test runners can shuffle order with a plugin or a flag.

If it fails alone, the cause is inside the test or the code it exercises.

Compare a pass with a fail

Capture everything you can from one passing run and one failing run: logs, timings, screenshots, traces, the data involved. Then compare them line by line. The first place they differ is usually close to the cause.

The usual suspects

Most flakes come from a short list:

• Fixed sleeps. wait(2000) passes on a fast laptop and fails on a busy CI machine. Wait for a specific condition instead.
• Missing awaits. In JavaScript, an assertion that runs before the work finishes.
• Order and shared state. Tests that depend on what an earlier test created or left behind.
• Time and dates. Midnight, month ends, time zones and daylight saving changes.
• Randomness. Random data without a fixed seed, or results whose order is not guaranteed.
• The network. Real calls to services that are sometimes slow or down.
• Parallel runs. Two workers sharing one account, port or record.

AI-written tests are especially prone to the first three, because a test that passed once on the assistant's run looks finished. Check generated tests for sleeps, missing awaits and dependence on earlier tests before you merge them.

Fix the cause, then prove it

Change one thing, the thing your evidence pointed to. Then repeat the fifty-run loop. If it passes every time, you have a fix. If not, you have a new clue.

If you cannot fix it today, quarantine the test: move it out of the blocking suite, give it an owner and a date, and keep it visible. A quarantined test with an owner gets fixed. A test retried forever never does.

The free Debugging Is the Skill book covers this loop in depth, including bugs that only appear sometimes. The Testing with AI guide shows how to catch flaky patterns in generated tests before they merge.