Cost per solved task: the agent metric vendors leave out
"Cost per task" is the number in every agent pitch deck. It is also the number that flatters failure: an agent that misses a third of its tasks still gets paid for the attempts. Divide by the success rate and the ranking changes — sometimes the agent with the better score is no longer the better buy.
Published · 8 min read · by AgentBurn Editorial · dataset snapshot 2026-10-27
1. The denominator that is missing
Cost per task answers "what does one attempt cost". Cost per solved task answers "what does one success cost". The second is the number a buyer needs, because failed attempts are not free — the tokens were consumed, the sandbox ran, the engineer reviewed the diff. The only question is whether the invoice is attached to a delivery.
AgentBurn defines the metric bluntly on the methodology page: cost per task divided by success rate — failed runs are paid for, so they are in the bill. One line, no weighting, no composite score.
2. The arithmetic, on published rows
The dataset's SWE-bench Verified table pairs a resolved percentage with a per-task cost for seven agents. Both numbers come from the cited public leaderboards; the cost-per-solved column is this post's arithmetic, nothing more:
| Agent | Resolved % | Cost per task | Cost per solved task |
|---|---|---|---|
| Cline (VS Code, BYOK) | 66.8 | $0.42 | $0.63 |
| Aider (open source CLI) | 67.5 | $0.48 | $0.71 |
| Cursor Agent (Composer 2.5) | 69.2 | $0.68 | $0.98 |
| GitHub Copilot coding agent | 65.0 | $0.75 | $1.15 |
| Claude Code (Sonnet 5.5) | 72.8 | $0.85 | $1.17 |
| Devin (cloud agent) | 70.4 | $1.40 | $1.99 |
| AutoCodeRover (AST agent) | 46.2 | $0.70 | $1.52 |
Formula used: cost_per_solved = cost_per_task / (resolved_pct / 100). No retry overhead is added because a benchmark run is already a completed attempt; production retries are handled in the next section.
Three results worth pausing on:
- The score leader is not the cost leader. Claude Code resolves the most tasks (72.8%) at $0.85 per attempt, but Cline delivers a solved task for $0.63 — 46% cheaper per success, at 6 points lower resolution.
- A lower score can cost more. GitHub Copilot resolves 65.0% at $0.75 per attempt: $1.15 per solved task, more than Cursor's $0.98 at a higher 69.2%. The cheaper-looking attempt price is the more expensive success.
- OpenHands + critic resolves 72.0% but publishes no cost — and that is the correct treatment. A missing denominator stays missing rather than being estimated into a favourable position.
3. Production adds a retry term
Benchmark tasks are single-shot; production agents retry. The dataset's formula carries that explicitly:
attempt cost = (tokens_per_task / 1e6) * blended $/1M
solved cost = attempt cost * (1 + retry_overhead) / success_rateTake Claude Code at 72.8% and $0.85 per attempt. At zero retry overhead the solved cost is $1.17. At 20% retry overhead it becomes $1.40; at 50% it becomes $1.75. Retry overhead is not a rounding error — it is the second-largest lever after the success rate itself, and it is exactly the line vendors omit when they quote "cost per task".
4. How the metric gets gamed
- Cost per attempt, presented as cost per outcome. The most common substitution. Ask what happens to the price when the agent fails.
- Subset selection. A resolved percentage is only comparable on the same task distribution. "72% on our benchmark" and "72% on SWE-bench Verified" are different claims.
- Success redefined. Partial credit, "acceptable" diffs, human-completed tasks and near-miss counting all change the denominator. The definition matters more than the decimal.
- Retries excluded from cost. Some price sheets count one attempt; the harness may run five. The gap lands on your invoice.
- Infrastructure outside the number. Sandboxes, retrieval, vector stores and review time are real costs that "tokens per solved task" does not include.
5. Five questions for a procurement call
- What exactly counts as solved, and who adjudicates it?
- Is the quoted cost per attempt or per solved task?
- How many attempts does the agent typically make before it stops?
- On what task distribution was the success rate measured?
- Which cost basis — list prices, negotiated rates, or credits converted at what ratio?
The calculator answers the arithmetic side of these questions for one task across all 22 models in the snapshot, including retry overhead, cache hit rate, success rate and a human baseline. The leaderboards show the same dataset ranked by cost per solved task where success rates are published.
6. What this arithmetic does not cover
SWE-bench Verified measures issue resolution on a fixed repository set, not your repository. The rows come from public leaderboards and the dataset keeps their provenance labels; a benchmark number is an upper bound on comparability, not a forecast of your bill. Human review time, CI cost and the cost of a wrong merge are outside the unit entirely.
Also note that "cost per task" across different tables is not comparable. Claude Code appears at $0.85 per SWE-bench task in one table and at $13.30 per full coding task in the project library — same product, different task definitions, both correctly labelled. Compare within a task family, never across one.
Sources
- AgentBurn SWE-bench Verified table — benchmarks.json, snapshot 2026-10-27; row values cross-checked against the TrustTheBench agents leaderboard and the Wazir SWE-bench notes as documented in the dataset.
- AgentBurn methodology — definitions, provenance and formulas.
- Claude Code workflow split — project page, dataset snapshot 2026-10-27.