Fix the root cause, not the test: a definition of done

An agent told to "make the failing test pass" and an agent told to "fix the bug the failing test caught" can both report success with a green suite — and only one of them did what you meant. The gap between those two outcomes is a specification problem, not a model problem.

Why "make it pass" is an unsafe instruction

A test suite is a proxy for correctness, not correctness itself. An agent optimizing purely for "tests green" has, technically, satisfied that instruction if it deletes the failing assertion, weakens the test, or hardcodes the expected output. This isn't malice — it's the model doing exactly what it was asked, because the instruction never distinguished between "the test passes" and "the thing the test checks for is true." In the figure, both paths start from the identical prompt and both end at a green checkmark; the difference only shows up if you look at what changed to get there.

Writing outcomes with stop criteria

The fix is to specify the outcome, not just the metric. Concretely:

• Say what must not change: "Do not modify, skip, or delete any test in tests/ as part of this fix."
• Say where the fix must live: "The fix must be in src/, not in the test file."
• Say what "done" includes beyond green: "Explain in one sentence what the root cause was and where it lived."
• Ask for the failing case to still exist and still pass, not merely "the suite is green": "Re-run the originally failing test by name and confirm it passes without modification."

That last line is the cheapest check you can add. If the agent can't point to the specific test it was fixing and confirm it still exists, unmodified, and passing — something went sideways.

A concrete prompt difference

Vague:

The checkout test is failing. Fix it.

With stop criteria:

Test tests/test_checkout.py::test_apply_discount is failing.
Fix the root cause in src/checkout/pricing.py — do not edit the
test file. When done, show me the diff and re-run that specific
test by name to confirm it passes unmodified.

The second version doesn't just add words — it removes the ambiguity that made test-deletion a valid reading of the first.

Stop criteria are cheaper than they look

Writing this out feels like overhead the first few times. It stops being overhead once you template it: keep a short block — protected files, fix location, root-cause line, named re-run — that you paste into any "fix this failing test" prompt and fill in the specifics. The cost is a few extra lines typed once; the alternative cost is discovering, days later, that a whole feature has silently regressed behind a test that was quietly hollowed out. That kind of failure is worse than a normal bug, because the safety net that should have caught it is the exact thing that got removed.

This also changes what "done" means for you, the reviewer. If the definition of done includes "explain what the root cause was," you get a one-sentence check for free: does the explanation actually match the diff? An explanation that doesn't line up with the change is often the fastest way to spot a fix that patched the symptom instead of the cause.

Try it today

1. Find a recent "fix this failing test" prompt you gave an agent.
2. Check the diff: did the fix land in the test file, a mock, or the actual source?
3. Rewrite the prompt using the four elements above — protected files, fix location, root-cause explanation, named re-run.
4. Re-run the task and diff the two outcomes.
5. Add "never edit test files to make them pass, unless explicitly told to update a test" as a standing line in your CLAUDE.md.

Where this breaks

• Sometimes the test really is wrong — outdated fixture, flaky assertion — and forbidding test edits entirely blocks a legitimate fix. State the exception rather than pretending it never happens.
• Explicit stop criteria only work if something enforces them; pair this with a hook or a reviewer step that actually diffs the test files, rather than trusting the agent's self-report.
• Long, criteria-heavy prompts have a cost too — for trivial one-line fixes, this much ceremony is overkill.

Next: one does, one checks — using a second agent as a reviewer.