How to read an agentic benchmark
Chat benchmarks score one answer; agentic benchmarks score a whole trajectory of tool calls and recoveries — read them differently.
A chat benchmark asks a model one question and grades one answer. An agentic benchmark drops a model into an environment — a repo, a terminal, a browser — and grades whether it reaches a verified end state after any number of tool calls, errors, and retries. Reading the two the same way is the most common mistake when picking a model for agent work.
Why the two scores diverge
A model can be excellent at single-turn reasoning and mediocre at agentic work, because agentic tasks add failure modes that never show up in a one-shot answer: losing track of state across 40 tool calls, misreading a tool's error message, or quitting one step before the fix would have worked. SWE-bench Verified — OpenAI's human-validated subset of 500 real GitHub issues, released in August 2024 — exists precisely because the original SWE-bench had test setups an agent could pass without truly fixing the issue. Verified tightened the harness so the score reflects the fix, not the scaffolding.
What "long-horizon" actually measures
The more revealing number for agent work is not accuracy on a fixed task set but how long a task the model can carry to completion unsupervised. METR's time-horizon metric asks: for tasks of a given human-completion-time, what's the length at which a model succeeds only half the time? Their research found this "50%-task-completion time horizon" has been doubling roughly every seven months since 2019, with an even faster doubling under three months on real software tasks based on SWE-bench Verified (METR, March 2025). That trend line, not a single leaderboard percentage, tells you whether a model is ready for a task that takes your team an hour versus a day.
Reading a benchmark table without getting fooled
Before trusting a number, check three things:
| Check | What it tells you |
|---|---|
| Harness | Was the model given the same tools, retries, and timeouts you'll give it? |
| Verifier | Does "pass" mean tests ran green, or that a human re-checked the diff? |
| Task realism | Is this a curated issue set, or a live environment like a terminal (Terminal-Bench 2.0 uses 89 hand-verified tasks across scientific computing, security, and sysadmin work)? |
A model at the top of a chat leaderboard can sit mid-table on an agentic one, because the chat score never had to survive a bad tool response.
Try it today
- Pick the agentic benchmark closest to your actual work (coding: SWE-bench Verified; terminal ops: Terminal-Bench; tool-using customer workflows: tau-bench).
- Read the benchmark's own methodology page, not a summary — find out what counts as "pass" and how many retries the harness allows.
- Run one real task from your backlog through your two candidate models with the same tools and note where each one gets stuck, not just whether it finishes.
- Weight the trajectory over the score: did the model recover from its own error, or did it need a human nudge?
- Re-check the benchmark's leaderboard date — agentic scores move faster than chat scores as harnesses improve.
Where this breaks
Benchmark scores are a ceiling, not a guarantee: a model that tops SWE-bench Verified can still fail on your codebase's specific conventions, because the benchmark's repos are not yours. Public benchmarks also invite overfitting once a lab knows the test set, so a jump on one leaderboard is weaker evidence than steady improvement across several unrelated ones. And no benchmark yet scores the thing you actually care about most: whether the fix was the right fix, not just a passing one.
Next: prompts are disposable, skills compound.