Every number has a provenance tier.
AgentBurn exists because agent costs are quoted in incompatible units: tokens per million, credits per action, seats per month, dollars per solved ticket. This page documents exactly how each figure is defined, where it comes from, and what we do when a vendor publishes nothing.
1. What we measure
Every project card reports the same six numbers so a coding agent and a customer-service agent can be compared. Definitions are deliberately blunt — no weighted composite scores.
| Metric | Definition | Unit |
|---|---|---|
| Tokens per task | Input + output + cached tokens consumed by one completed unit of work (a PR, a case, a paper, a report) | tokens |
| Cost per task | Model spend for one attempt, at list prices, using the project's model mix | USD |
| Cost per solved task | Cost per task divided by success rate — failed runs are paid for, so they are in the bill | USD |
| Runs per day | How often the workflow completes per day across all its users | tasks/day |
| Tool calls per task | Discrete tool invocations in the agent loop, including retries | calls |
| Retry overhead | Share of spend caused by tool errors, loops and failed validation | % |
| Framework overhead | Extra tokens an orchestration layer adds over a raw API loop on matched tasks | % |
2. Two data modes
Seed dataset (current). A maintained dataset of 30 production agents, 22 models and three public benchmarks. It is assembled from published benchmarks, papers, vendor documentation and community reports, and it is what all boards currently render.
Live telemetry (beta). When developers instrument their agents with the AgentBurn SDK or an OpenTelemetry bridge, token counts, durations and outcomes are posted to the ingest API and aggregated per project and per day. Live aggregates never mix with seed rows: the leaderboard shows the live panel separately and only when events actually exist.
3. Provenance tiers
Every project carries a source object with a URL and a tier. The badge on the card is a summary of this table — click through to the actual source before you quote a number.
| Badge | Meaning | Evidence required |
|---|---|---|
| Verified | At least one core metric is stated in a public benchmark, paper or vendor document | A URL that contains the score, price or token count being displayed |
| Community | A credible public report — issue thread, teardown, practitioner write-up — states the figure | Link to the report plus a note on what exactly it covers |
| Estimated | We modelled the figure from adjacent public data (credit rates, seat prices, comparable tasks) | A metrics note explaining the derivation, plus a real product source |
Estimated does not mean invented: the derivation is shown on the project page. If a vendor publishes seat pricing only, we convert credits or per-action price points and state the conversion. We never publish an estimated figure as verified.
4. Sources
- Public benchmarks — SWE-bench Verified, Terminal-Bench 3.0 and the Artificial Analysis Coding Agent Index, for agents whose per-task cost or token count is published alongside the score.
- Papers and lab write-ups — research agents (AI Scientist, Coscientist, FutureHouse Robin, GPT Researcher) where the paper or documentation states cost per run.
- Vendor documentation and rate cards — consumption-priced enterprise agents (Agentforce Flex Credits, Copilot Studio credits) where the price per action is public but tokens are netted out.
- Community reports — practitioner threads and public issue reports where someone published a real bill, such as an n8n run at $0.42.
- Model list prices — the official pricing pages of Anthropic, OpenAI, Google, DeepSeek, Alibaba, Moonshot, Z.ai, Mistral, xAI and Meta, captured in a dated snapshot.
5. Model pricing
Prices are USD per one million tokens, split into input, output and cached input where the provider offers prompt caching. The calculator blends them for a workload:
blended $/1M = (input_share * input_price)
+ (output_share * output_price)
- (cache_hit_share * input_share * (input_price - cached_price))
attempt cost = (tokens_per_task / 1e6) * blended $/1M
solved cost = attempt cost * (1 + retry_overhead) / success_rateCaveats we apply: list price only (no negotiated discounts), no batch-API discount, peak/off-peak pricing for providers that use it, and cached-token discounts only when the provider publishes a cache rate. Model prices change often — the snapshot date is shown on the calculator and in the dataset's metadata.
6. Usage volumes
The leaderboard windows are derived, never scraped as totals:
24h tokens = runs_per_day * tokens_per_task
7d tokens = 24h tokens * 7
30d tokens = 24h tokens * 30
share = project 24h tokens / sum(all projects 24h tokens)Runs per day is the softest number on the site. Where a vendor publishes usage we use it; otherwise the volume is a community-reported or modelled estimate and the project's usage basis says so. Treat window totals as an order-of-magnitude index of relative agent activity, not as an audited market size.
7. Framework overhead
The overhead board compares orchestration layers on matched tasks, with a raw API loop as the 0% baseline. Reported percentage is the extra token spend attributable to the framework: system prompts, tool schemas, serialised state and duplicated message history. A multi-agent framework that resends a growing transcript every turn will show a large number — that is the point of the board.
8. Corrections
If a source is stale, misattributed, or a number does not reproduce, tell us and we will fix it inside the dataset with a changelog entry. Corrections are shipped with the next dataset revision; the card's source link is the place to start.
9. Changelog
10. Using the dataset
The dataset is openly readable at /data/projects.json (projects), /data/models.json (pricing) and /data/benchmarks.json (benchmarks). Cite it as AgentBurn, getagentburn.com, dataset snapshot <date>, link the project page, and keep the provenance tier attached to any figure you reuse. For commercial redistribution or a bulk feed, contact us first.