Is AI actually helping your team?
Perceived speed is unreliable; baseline lead time, review time, defect escape rate, and rework rate before rollout, then compare, not guess.
Whether AI is helping your team is not a question you can answer by asking people how it feels. It has to be answered by comparing the same metrics before and after, because the gap between how fast people feel and how fast they actually are is large enough to invert the answer entirely.
The gap between feeling faster and being faster
A 2025 randomized controlled trial from METR had 16 experienced open-source developers complete 246 real tasks in their own mature repositories, with and without AI coding tools. The result: developers were, on average, 19% slower with the tools than without them — and yet, both before and after the study, they believed AI had sped them up, expecting a 24% gain beforehand and still estimating a 20% gain afterward, according to METR's writeup. The authors are explicit that this is a snapshot of one setting, not a universal verdict on AI tools. The lesson to take isn't "AI slows teams down" — it's that self-reported speed is not evidence, in either direction, and the only way to know is to measure.
Four metrics that aren't vanity
"PRs merged this week" or "lines of code shipped" go up when agents write more code, and tell you nothing about whether that code was good. Use metrics that catch a regression a raw output-volume number would hide:
| Metric | What it catches |
|---|---|
| Lead time for changes | Time from commit to production — DORA defines it as "the amount of time it takes for a change to go from committed to version control to deployed in production" (dora.dev) |
| Review time | Time a change sits waiting for a human to approve it — rises if reviewers don't trust agent output enough to move fast |
| Defect escape rate | Share of defects found in production instead of before release |
| Rework rate | DORA's related measure: "the ratio of deployments that are unplanned but happen as a result of an incident in production" (dora.dev) |
Lead time and review time measure throughput; defect escape and rework measure whether the throughput is real or borrowed from next week's incident response.
Baseline before you roll out
Measure all four for two to four weeks before introducing or expanding agent use on a team — not after, when there's nothing to compare against. Use the same measurement window length and the same team composition for the "after" period, or a slower quarter will look like an AI-driven improvement and a faster one will look like a regression.
Reading the scorecard honestly
In the figure, three of four metrics improved and one — rework rate — got worse. That's a realistic outcome, not a failure of the exercise: it might mean agents shipped changes faster but with less scrutiny, trading lead time for stability. The point of tracking four metrics instead of one is catching exactly that trade before it becomes the story a single "we shipped 30% faster" number would tell on its own.
Try it today
- Pick the same two-to-four-week window and record current lead time, review time, defect escape rate, and rework rate for one team.
- Write down what changed about how the team works with agents during that window — don't rely on memory later.
- Re-measure after the same length of time and compare all four, not just the one that improved.
- If one metric got worse, investigate before rolling the change out further — don't average it away against the ones that got better.
Where this breaks
Four metrics are still a small sample of team health — they say nothing about the team's own sense of the work, which matters and isn't captured here. Short baseline windows are noisy; a single incident or a single slow reviewer can swing review time or rework rate without agents being involved at all. And none of these numbers separate "agents changed behavior" from "the team changed behavior because they were being measured" — the same observer effect that makes any process metric worth treating with some skepticism.