AgentBurn getagentburn.com
Submit Agent
Home/Blog/What 20 of 30 agent projects don't publish — and why that matters
Measurement

What 20 of 30 agent projects don't publish — and why that matters

A dataset that fills every cell looks complete and is useless. The 20 empty rows are the most informative part of the AgentBurn project library.

Published · 9 min read · by AgentBurn Editorial · dataset snapshot 2026-10-27

1. The empty cells are the finding

A dataset that fills every cell looks complete and is useless. The AgentBurn project library (snapshot 2026-10-27) has 30 projects, four metric fields each, 120 provenance labels in total. Exactly 15 labels say a cited public source states the figure. One is derived. Twenty-four are model estimates of run volume. And 80 — two thirds of the dataset — say not published, with the field left empty.

That is not an apology. Twenty of the 30 have no public per-task metrics at all: neither a token count nor a cost per task. If you assumed the AI agent market publishes enough for a clean comparison, this is the counter-evidence, and it holds across domains — the empty rows include coding agents, research agents, CRM and support agents, and no-code workflow tools.

2. Four labels, four different kinds of absence

The library has no single known/unknown switch. Each field carries a basis, and the four bases mean different things — the part that keeps the database honest, and the part readers skip.

FieldStated by sourceDerivedNot publishedModel estimateRows with a number
tokensPerTask3027—3
costPerTaskUsd9120—10
successRatePct3027—3
runsPerDay0062424
All 120 labels1518024—
  • Stated — the cited benchmark, paper or vendor page publishes that exact figure. Example: Claude Code at 24.1M tokens and $13.30 per task from Artificial Analysis; Aider at $0.48 and 67.5% success from TrustTheBench.
  • Derived — computed from public inputs with the formula written down. The library has exactly one: ai-scientist-v2 at $10.50 per paper, the midpoint of a $6–15 range in public evaluations. Its note concedes the weakness — “secondary reporting; the paper itself does not state cost per paper.” A derived number can be the least reliable value here and still be more honest than an estimate presented as fact, because you can see the formula and disagree.
  • Not published — no public basis exists, so the field is empty. That is a legal state in our schema, not a value to impute. It appears 80 times.
  • Model — daily run volume only. No public source publishes how often these workflows run, so the figure is a labelled estimate, never used to rank per-task efficiency. It appears 24 times.

3. The asymmetry: cost without tokens

The headline gap looks simple — 3 of 30 publish tokens, 10 of 30 publish cost — but the interesting part is the seven-row difference: projects where a source published a cost per task and said nothing about the tokens behind it. TrustTheBench is the clearest case: it scores coding agents on SWE-bench Verified, publishes a cost per task and a success rate, and the provenance note on Copilot's token field says, in effect, “does not publish token counts.”

Cost per task without tokens is a closed box. You can rank those rows against each other — Aider $0.48, Cline $0.42, GitHub Copilot $0.75 — and you cannot do anything else with them. You cannot recompute at your own model price, cache hit rate or retry rate: you do not know the input/output/cached split behind the number. You rent the figure; you cannot inspect it. Tokens per task is the only field that transfers, which is why three rows out of 30 is the number we most want to change.

What the empty rows protect. Every aggregate on this site — 24-hour token totals, leaderboard shares, the estimated-volume board — is computed client-side from published rows only. If the 20 empty projects carried guessed numbers, those aggregates would silently include fiction and no reader could tell which share was invented. Empty fields make the aggregate smaller and the denominator visible. The methodology documents the rule; the JSON enforces it field by field.

4. Fill or leave empty

For any empty cell there are three ways out, and only one survives contact with an aggregate. Estimating from a similar project is fast and silent: a plausible token count for an enterprise support agent makes the table look complete, and nobody knows it was inferred from another vendor's workload. Publishing a range is better and still broken: ranges do not sum — you cannot add a $6–15 band to a $0.42 point estimate and get a monthly bill.

Leaving the cell empty and labelling it costs the table its completeness and buys three things: the aggregate covers only what it covers, coverage becomes a number you can count (section 2 is that count), and a later source can fill the cell without anyone walking back a fabricated figure. We chose empty. The single derived row shows the template for adding one honestly: a narrow public range, a stated formula, and a note naming the weakness of the input.

One more detail. Six projects carry no run-volume estimate at all — their runsPerDay is null and the provenance note reads “No public run volume” — while the usage block on every row still labels its basis as model. That label is stale on those six rows and will be corrected in the next revision. We would rather describe the inconsistency than let a reader discover it.

5. Read the provenance before the number

Every field ships with a basis, a source URL where one exists, and a note explaining how the figure was obtained:

  1. The basis. If it is not_public, there is no number to cite — the field is empty and the honest citation is “not published.” If it is derived, read the formula before you repeat the value.
  2. The source URL. For a benchmark page, check whether it states a task definition and a sample size; for a vendor page, whether the number is list price or effective contract price.
  3. The note. This is where caveats live: which model variant was measured, whether a range was collapsed to a midpoint, whether a figure is a single anecdotal run. The n8n row, for instance, is one reported execution of a five-word question — real, stated, and not a population estimate.

Then read the count. Nine projects carry a metric stated by a public source; 20 carry no per-task metric. A leaderboard that ranks the nine and leaves the 20 unranked tells you the truth about disclosure here. The project library marks the empty rows rather than hiding them, and the machine-readable dataset summary says the same thing in the same words to crawlers, so an assistant summarizing our coverage cannot accidentally claim more than we have.

6. Three things to do on Monday

1. Check provenance before you use a number. Open the row in projects.json, find the field, read its basis. If you build on this data, key on the label and refuse to aggregate across not_public. The methodology has the definitions and the citation format.

2. Ask your vendor for tokens per task. Specifically: input, output and cached tokens per completed task; the definition of a completed task; the success rate over a stated sample; and whether the cost is list price or your effective price. A vendor that publishes only a seat price or a credit ratio may be hiding nothing — but you cannot compute cost per solved task from it, and the procurement note should say so. The pricing-units piece walks through the conversion you will be missing.

3. Measure your own. The fastest route out of a data gap is telemetry you control: the SDK posts input, output and cached tokens, tool calls, retries and outcomes to the ingest API — how a measured row reaches the boards without a vendor's permission. Until then, the calculator will price a task at list prices with your retry and success assumptions made explicit — a scenario, not a measurement, and labelled as such.

The short version: “not published” is a legitimate value. A database that says it 80 times out of 120 is more useful than one that guesses — because you can see which third of the market you can compare.

Sources