How to read a coding-agent leaderboard without being misled
The same agent can appear at $0.85 per task in one table and $13.30 in another. Both can be correct. The difference is the task definition, and it decides which table you should use.
Published · 10 min read · by AgentBurn Editorial · dataset snapshot 2026-10-27
1. Three tables, three task definitions
Our snapshot carries three public benchmark boards, and they get quoted as if they were three views of one thing. They are not. Each fixes a different task set, a different judge and a different score scale, and only one of the three publishes a token count for every row.
| Board | Metric | What one task is | Rows in snapshot | Known caveat |
|---|---|---|---|---|
| SWE-bench Verified | resolved % | a real GitHub issue in a popular Python repository, patched and checked by the repository's tests | 8 | official table is client-rendered; row values cross-checked against cited sources |
| Terminal-Bench 3.0 | reward % | a verifiable terminal task, scored by its reward function | 7 | tokens and cost published for every row |
| AA Coding Agent Index v1.5 | index score 0–100 | a composite coding run on Artificial Analysis's own task set | 8 | UNVERIFIED/STALE: the page now serves v2.0 |
Two consequences follow. A score of 56.6 on Terminal-Bench 3.0 is not "worse" than 72.8 on SWE-bench Verified; the numbers answer different questions. And a cost column without a token column tells you what a task cost, not why it cost that — which is where most cross-table comparisons go wrong.
2. The same agent at $0.85 and $13.30
Claude Code appears twice in our files. The SWE-bench Verified row: 72.8% resolved at $0.85 per task, from TrustTheBench. The project card, sourced to Artificial Analysis: 24.1M tokens and $13.30 per task. Same product name, same snapshot, 15.6x difference in cost per task. Both rows are defensible.
The difference is the task set. SWE-bench Verified asks for a repository patch that passes the repository's tests on a curated issue set; the AA index runs a composite of coding tasks with a different harness configuration and a much longer interaction budget. The cost numerator is defined per task in both cases, but "task" is not the same object. There is a provenance wrinkle too: the SWE-bench costs come from a third-party leaderboard that reproduces the official scores, and that leaderboard labels the underlying models differently from the agent labels we display. The file records that caveat instead of editing the labels.
This is the cleanest demonstration of why cross-table comparison fails. The product name is stable; the thing being measured is not.
3. Resolved, reward and index are different scales
The SWE-bench rows also dismantle the idea that a leaderboard ranks agents by cost. Cost per task across the eight rows runs from $0.42 (Cline) to $1.40 (Devin), a 3.3x range, while resolved percentage runs from 46.2 to 72.8, a 26.6-point range. The cheapest row is not the best, and the most expensive is not the worst.
Divide cost per task by resolved share and the ordering changes again. The last column is our arithmetic, not a published figure, and it assumes the published cost is per attempt:
| Agent | Resolved % | Cost per task | Cost per resolved task (derived) |
|---|---|---|---|
| Claude Code | 72.8 | $0.85 | $1.17 |
| OpenHands + critic | 72.0 | not published | — |
| Devin | 70.4 | $1.40 | $1.99 |
| Cursor Agent | 69.2 | $0.68 | $0.98 |
| Aider | 67.5 | $0.48 | $0.71 |
| Cline | 66.8 | $0.42 | $0.63 |
| GitHub Copilot coding agent | 65.0 | $0.75 | $1.15 |
| AutoCodeRover | 46.2 | $0.70 | $1.52 |
Devin moves from a mid-table $1.40 to the most expensive per resolved task at $1.99. Cline, cheapest per task, stays cheapest per resolved task. Cursor Agent beats Claude Code on cost per resolved task ($0.98 against $1.17) while scoring 3.6 points lower. AutoCodeRover has the worst score and still costs $1.52 per resolved task — more than Cursor Agent, which is the table's clearest warning about cheap-looking per-task prices.
Terminal-Bench 3.0 shows the same non-monotonicity with a different metric: the top reward row costs $0.99 per task, while the $5.54 row scores 36.6. Score and cost are two columns, never one ranking.
4. Five checks before you cite a row
- Name the task set and the judge. "Resolved", "reward" and "index" are different words because they are different measures. A row without a task-set definition is a number without a unit.
- Name the configuration, not just the product. Cursor Composer 2.5 scores 62 both in its standard configuration at $0.07 per task and in its fast configuration at $0.44 — same score, 6.3x the cost. The variant label is part of the number.
- Check what cost per task includes. Retries, caching and harness overhead are usually not itemised. The Codex CLI card notes that its $1.50 is the published figure for the GPT-6.1 Sol configuration, and that an earlier version quoted a lower number from an earlier configuration.
- Check the snapshot date and the stale flag. The AA v1.5 rows are marked UNVERIFIED and STALE, and the publisher's current page scores the same variants differently. A stale row is evidence of a mechanism, not a current score.
- Convert to cost per solved task, and compare only inside one table. Then take the comparison out of the table and into your own telemetry, because none of these rows knows your retry policy.
5. Three traps in this snapshot
Trap one: cross-table ranking. Putting SWE-bench resolved percentages next to Terminal-Bench rewards and calling one agent "better" mixes three judges. The dataset keeps the boards separate for exactly that reason.
Trap two: stale rows reused. The v1.5 AA rows are the most quotable — five configurations of one product, tokens and cost side by side — and the most dangerous to reuse. The publisher's v2.0 page scores the same variants differently; the dataset's note lists the current values rather than silently refreshing the old ones.
Trap three: variant labels. "Composer 2.5 standard" and "Composer 2.5 fast" are both 62 and differ by 6.3x in cost. "Sonnet 5.5 max" and "Sonnet 5.5 high" are both Claude Code and differ by 10.3x in tokens in the v1.5 rows — provisional figures, but enough to show that the configuration column is not boilerplate.
What the boards do support: the order of magnitude of tokens per task on a comparable set, the direction of cost differences within one board, and the identification of which harnesses are token-heavy. Use them for that, then instrument your own stack. The SDK records what the tables cannot, and the calculator turns those numbers into a bill. The monthly bill formula is the next step.
Sources
- AgentBurn dataset snapshot 2026-10-27 — benchmarks.json rows for SWE-bench Verified, Terminal-Bench 3.0 and the AA Coding Agent Index v1.5, each with its source URL; projects.json for the Claude Code and Codex CLI cards.
- SWE-bench Verified leaderboard — swebench.com.
- TrustTheBench, Autonomous Coding Agents Leaderboard 2026 — trustthebench.com/agents (source of the per-task costs in the SWE-bench rows).
- Terminal-Bench 3.0 — bench.turing.com.
- Artificial Analysis — Coding Agent Index and Composer 2.5 article; v1.5 rows are flagged UNVERIFIED/STALE against the current v2.0 build.
- AgentBurn methodology — provenance labels, the derived-metric policy and the stale-row rule.