Testing with AI

AI can draft a test suite in minutes. The danger is a suite that passes whether the code works or not. This guide shows QA and test engineers where AI helps, where it fools you, and the one habit that catches most of it: prove every new test can fail.

Where AI earns its place

• Listing cases from a requirement. Ask for cases, not code, and pick the ones that matter. It is good at the edges people skip: empty, zero, one, the limits, bad input, slow or missing responses.
• Boilerplate. Page objects, fixtures, request builders, parametrised tables of inputs.
• Turning manual cases into automation, one case at a time, with the manual steps pasted as the specification.
• Fake test data with the same shape as the real thing, so nothing real leaves your machine.
• Reading a failure. Paste the error and the test, ask for three possible causes, and rank them yourself.

How AI-written tests fool you

1. Tests written from the code. If you give it the code and ask for tests, it describes what the code does, bugs included. A test that expects the wrong answer passes, and protects the bug.
2. Assertions that can't fail. It checks that a value exists rather than what it is, compares something with itself, or asserts on a mock it set up two lines earlier.
3. Assertions that never run. In JavaScript, a missing await lets the test finish before the check happens. A loop over an empty list checks nothing.
4. Brittle selectors. Generated class names, nth-child, long CSS paths and text that changes with every copy edit.
5. Fixed sleeps. wait(2000) passes on a fast laptop and fails in CI, or hides a real race.
6. Invented APIs and options. Matchers, config keys and command-line flags that sound right and do not exist in your version.
7. Mocking the thing under test. The test passes because the mock answers, not because the code works.
8. Happy path only. Ten tests of the main case and none of the errors the requirement mentions.

The workflow

1. Start from the requirement, not the code. Paste the ticket, the acceptance criteria or the manual test case. Leave the implementation out of the first prompt.
2. Ask for cases first. A plain list of what to test and why. Cut, merge and add to it before any code exists. This list is your specification.
3. Generate the tests, one small group at a time, against your list.
4. Read every assertion. For each one, ask what wrong behaviour would make it fail. If you cannot name one, rewrite it.
5. Prove each test can fail. See the next section. This is the step that catches most bad tests.
6. Check for flakiness. Run each new test many times with retries turned off before you trust it.
7. Review it like code. Use the AI code checklist on the test change, and pay special attention to edited or deleted assertions in existing tests.

Prove every test can fail

A test you have never seen fail is a test you are trusting, not checking. Before you merge a new test, break the behaviour it covers and watch it go red:

• Change one thing in the code under test. Flip a condition, return the wrong value, remove a validation. Run the test. It should fail with a message that points at the problem. Then undo the change.
• For a bug fix, run the test against the old code. Check out the commit before the fix, or revert it locally. The test must fail there and pass with the fix.
• For UI and API tests, point the test at wrong data. Change the expected text, status code or field and confirm the failure message is clear.
• Automate it with mutation testing once a suite matters. These tools make many small changes to your code and report which ones no test noticed: Stryker for JavaScript and TypeScript, mutmut for Python, PIT for Java. A surviving change is a missing or weak assertion.

If the test still passes when the behaviour is broken, it is not a test. Fix it or delete it.

Prompts that work

Ask for cases before code:

Here is a requirement. List the test cases you would write, as one line each:
the input or situation, the expected result, and why it matters.
Include edge cases and error cases the requirement implies. Do not write code yet.

Get a second opinion on a test you already have:

Here is a test and the requirement it checks. Describe three ways the code
could be wrong while this test still passes. For each, suggest the smallest
change to the test that would catch it.

Make fake data instead of pasting real records:

Here is the shape of a customer record, with field names and types only.
Generate 20 realistic but fake records, including edge cases: empty optional
fields, very long names, non-ASCII characters, and boundary dates.

Framework notes

To check a new test for flakiness, run it repeatedly with retries off: in Playwright, npx playwright test path/to/spec --repeat-each=20 --retries=0; in pytest, install pytest-repeat and add --count=20. Retries hide flaky tests; switch them off when you are judging a new one.

What never goes into an AI tool

• Real customer or user data: production exports, support tickets, logs with emails, names, addresses or account numbers.
• Secrets: API keys, tokens, passwords, connection strings, .env files, and recorded requests or HAR files that contain them.
• Anything your team's policy restricts. If there is no policy yet, the team AI policy template is a starting point.

Use fake data with the same shape instead: generate it with the prompt above, or with a library such as Faker for Python or @faker-js/faker for JavaScript.

Before you merge

• Every new test was seen failing for the right reason.
• Each assertion checks a specific value, not just that something exists.
• Selectors use roles, labels or test IDs, and there are no fixed sleeps.
• Each test ran many times with retries off and passed every time.
• No existing assertion was weakened, edited or deleted without a reason in the PR.
• Every library call, matcher and flag exists in the version you run.

By Ghost in the Repo, ghostintherepo.com. Free to print and share in your team.