A practical guide for engineering managers. It covers how to tell whether AI coding tools are helping your team using numbers you already have, what to stop counting, and a 90-day plan for rolling the tools out to one team with a clear decision at the end. You need no analytics platform: your Git host, your deploy log and a ten-minute survey are enough.
What not to measure
The numbers vendors show first are the easiest to collect and the least useful:
- Lines of code, or lines "written by AI". More code is not more value, and generated code inflates it.
- Suggestion acceptance rate. It shows people pressed Tab. It says nothing about whether the code survived review or production.
- Pull requests opened per person. AI makes it cheap to open many small or half-finished changes.
- Hours of AI usage per person. Measure usage and people will use the tool more, whether it helps or not.
None of these is wrong to look at for curiosity. All of them go up the week you roll out a tool, and none tells you whether your team ships better software. Never use them in individual performance reviews.
Five numbers that matter
Measure what reaches users and stays there. Collect these for the whole team, monthly.
| Number | What it tells you | Where to get it |
|---|---|---|
| Time to first review | Whether more changes are piling up waiting for a reviewer | Your Git host: the first review on each merged pull request |
| Time from open to merge | Whether changes actually flow faster | Your Git host |
| Rework | How much "almost right" code gets through: changes reverted or fixed again within 14 days | Reverts in Git, plus a monthly sample of merged pull requests |
| Change fail rate | Deploys that needed immediate intervention, such as a rollback or hotfix | Your deploy log and incident tracker |
| Team pulse | Whether people think it helps and trust what they merge | A two-question survey, monthly |
The pulse is two questions on a 1–5 scale: "AI tools save me time" and "I trust the code we merge". The second answer often moves before any of the other numbers.
Getting the numbers without a platform
Merged pull requests with their open and merge times, using the GitHub CLI:
gh pr list --state merged --limit 200 --json number,createdAt,mergedAtFor the first review on a pull request, gh pr view <number> --json reviews lists each review with its time. GitLab and Bitbucket show the same data in their merge request APIs.
Reverts in the last 30 days:
git log --oneline --since="30 days ago" --grep="^Revert"For rework that is not a revert, take a sample of 20 merged pull requests each month. For each one, check whether the same code was fixed again within two weeks. It takes about half an hour, and it is the most honest number on this page.
How DORA metrics change when AI writes code
DORA's five software delivery metrics are a good frame, but AI changes how far you can trust each one:
- Deployment frequency and change lead time rise easily when AI makes changes smaller and more numerous. On their own, they can look like progress when nothing better is reaching users.
- Change fail rate and deployment rework rate, DORA's measure of unplanned deploys caused by production incidents, matter more. They are where code that passed review but was wrong shows up.
- Failed deployment recovery time tells you whether the team still understands the code well enough to fix it quickly.
Watch throughput and instability together. Faster delivery with a rising fail rate is not a gain. DORA's research makes the same point at scale: AI amplifies the strengths and weaknesses a team already has. See dora.dev for the definitions and reports.
Take a baseline first
Every number above is meaningless without a "before". Collect four to six weeks of data before the rollout, or before any policy change, and compare the same team afterwards. Do not compare yourself with another company's published figures: their codebase, team and definitions are different.
Expect a dip at first. Learning a new tool takes time, and the numbers that matter most, rework and fail rate, show up weeks after the code merges.
A 90-day rollout plan for one team
This plan suits one team of 5 to 30 engineers. Larger organisations can run it team by team.
| Weeks | Phase | What happens |
|---|---|---|
| 0–2 | Baseline and rules | Collect the five numbers. Agree the team's first rules using the AI acceptable use policy template. Pick the tool and set up company accounts |
| 3–6 | Volunteer pilot | Two to four volunteers use the tool on real work. Every AI-assisted pull request says so and gets reviewed with the AI code review checklist. Meet weekly for 15 minutes on what worked and what broke |
| 7–8 | Team session | Run the one-hour team session with everyone, led by the pilot group. Update the policy with what the pilot learned |
| 9–12 | Whole team | Everyone may use the tool under the agreed rules. Keep collecting the numbers and the pulse |
| Day 90 | Decision | Compare with the baseline and decide: keep, adjust or stop |
The day-90 decision
Decide with the numbers in front of you, against criteria you wrote down in week 0:
- Keep if time to merge is the same or better, rework and fail rate have not risen, and the pulse says people save time and trust what they merge.
- Adjust if throughput improved but rework, fail rate or review time got worse. That usually means smaller pull requests, stricter review for high-risk changes, or more practice checking AI output. Run another 30 days.
- Stop or narrow if the numbers got worse and the team does not trust the output. Limit the tool to low-risk work, such as tests, scripts and documentation, and try again later.
Whatever you decide, share the numbers and the reasoning with the team. People adopt rules they helped test.
Common mistakes
- Rolling out to everyone on day one. You lose the comparison and the chance to learn from a small group.
- Measuring individuals. Measure the team's delivery and let people choose how they work.
- Fixing the tool instead of the bottleneck. If review time climbs, the fix is smaller pull requests and review rules by risk, not a different assistant.
- Declaring success in week three. The costs of almost-right code arrive later than the speed-up.
By Ghost in the Repo, ghostintherepo.com. Free to use, adapt and share in your team.