When benchmarks stop helping: instrument your own agent

1. What a benchmark can and cannot answer

A published benchmark row answers one question: on a fixed task set, with a fixed configuration, this product scored X and cost Y per task. That is a description of a product. It says nothing about your task mix, your retry policy, your cache hit rate or your volume.

Our dataset makes the size of the mismatch visible. Claude Code appears at $0.85 per task on SWE-bench Verified and at $13.30 per task on the Artificial Analysis index — same product, same snapshot, 15.6x apart, and both rows are correct. If two published rows for the same agent differ by 15x, no published row can settle what your agent costs.

Three inputs of the monthly bill formula are absent from every benchmark row. Runs per day is a model estimate wherever our dataset carries one, and absent where it is not. Retry overhead is published for no project. Task success rate is absent for most projects, and the values that do exist are benchmark scores rather than production success rates. Telemetry is the only instrument that produces those three for your stack.

2. The minimum viable measurement set

Six numeric fields per task are enough to price an agent:

Three identity fields make the numbers joinable: model id, a task label of at most 80 characters, and duration in milliseconds. Nothing else is needed. Prompts and completions are not cost inputs; they are the part you should not collect.

3. Two ways in, neither needs a build step

The telemetry SDK is the short path. The JavaScript module is served from the site, has no dependencies and no build step; the Python client is a single standard-library file. One call per task carries the fields above:

import { createClient } from '/sdk/js/agentburn.js';
const client = createClient({ project: 'my-agent' });

client.track({
  model: 'claude-sonnet-5-5',
  inputTokens: 1240,
  outputTokens: 380,
  cachedInputTokens: 256,
  toolCalls: 6,
  retries: 0,
  durationMs: 1820,
  success: true,
  task: 'resolve-issue-412'
});
await client.flush();

The other path exists for stacks that already emit spans. Run an OpenTelemetry Collector with an OTLP receiver, write spans to a JSON-lines file, and let a relay you control map gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and otel.status_code to the ingest fields, then post to /api/telemetry with source: "otel". LangGraph, CrewAI, the OpenAI Agents SDK and LlamaIndex already emit those spans; the mapping example is in the OpenTelemetry bridge section of the docs.

The endpoint takes 1 to 100 events per request, a 32 KB body, timestamps no older than seven days and 120 requests per IP per hour. A successful ingest returns 202 with a summary computed from the batch, so it is safe to log and move on.

4. What the measurement costs to run

The SDK is built so measurement rarely becomes its own incident. Events queue in memory and flush every five seconds or at the batch size; memory is bounded at 500 queued events; failures back off from 500 ms to 60 s; a rejected batch is dropped rather than retried forever. Sampling happens before queueing, so unsampled events cost nothing. Nothing throws into your agent code.

On the server side, raw events expire after 30 days and daily counters after 90. The aggregates are counters only — events, tokens in and out, cached tokens, tool calls, retries, successes, failures — with no content. The API is four Cloudflare Pages Functions and one KV namespace, and the docs describe how to self-host if the data cannot leave your account.

The real cost is engineering attention, not bytes. The hard part is deciding where a task begins and ends in your code and recording the outcome there; the SDK provides trackTask for that. One week of data on one task type is usually enough to replace two guesses in the monthly bill.

5. Measure the numbers, not the content

Cost accounting needs counters, so the privacy boundary can be drawn sharply. Collected: project slug, model id, SDK version, input, output and cached token counts, tool-call and retry counts, duration, a success boolean, a task label of at most 80 characters, an error string of at most 160 characters, a random client id and the timestamp.

Two operational rules keep that honest. Write task labels as identifiers, not payloads — "resolve-issue-412", never an email address — because an 80-character string can still carry a customer name. And treat the project slug as the aggregation boundary: stats are per project, never per user. If either rule is too tight for your environment, self-host and point the SDK at your own origin; opting out is simply not calling the client.

The trade-off is narrow. You give up per-request debugging detail, and in return you can compute tokens per task, retry overhead, cache hit rate and cost per solved task. Content-level tracing belongs in your existing observability stack. Cost telemetry needs counters, and counters are the part that can be shared.

6. A four-week rollout

1. Week 1 — tokens. Log input, output and cached tokens for one task type on one model. Reconcile the daily total against your provider invoice before trusting anything downstream.
2. Week 2 — overhead. Add tool calls, retries and duration. The ingest flags a day above three times the trailing seven-day median, and retry overhead above 25% of requests, so an expensive retry loop shows up before the invoice does.
3. Week 3 — outcomes. Add the success flag and a generic task label. Compute cost per solved task and compare it with the published row for the same agent; the gap is your stack, and it is usually the cache rate or the retry policy.
4. Week 4 — replace the guesses. Put the measured volume, tokens per task, cache share, retry and success into the monthly formula and retire the assumptions.

When benchmarks are enough: before you have production traffic, when you are choosing between products, and when you need an outside reference point. When they stop helping: the moment you need a number for your own bill. At that point the published row has done its job, and the only instrument that answers the question is the one running inside your agent.

Sources

• AgentBurn telemetry SDK documentation — quickstarts, ingest API, limits, retention, OpenTelemetry bridge and privacy policy.
• AgentBurn methodology — provenance labels, derived metrics and the runs-per-day estimate caveat.
• AgentBurn dataset snapshot 2026-10-27 — projects.json (Claude Code card and run-volume basis) and benchmarks.json (SWE-bench Verified, Terminal-Bench 3.0, Artificial Analysis rows and framework overhead).
• OpenTelemetry semantic conventions for generative AI — opentelemetry.io.
• Artificial Analysis — Coding Agent Index; v1.5 rows flagged UNVERIFIED/STALE against v2.0.
• SWE-bench Verified leaderboard.
• AgentBurn cost and ROI calculator and leaderboards.