Agent framework token overhead: AutoGen vs LangGraph vs CrewAI
Frameworks are sold on capability — graphs, crews, handoffs, memory. They are not sold on the context they resend on every turn. On one matched task set, the orchestration layer alone moved token spend from 379 to 4,382 tokens per run. That is an 11.6x tax, and it is measurable before you commit.
Published · 10 min read · by AgentBurn Editorial · source: framework overhead board, dataset snapshot 2026-10-27
1. What "framework overhead" measures
AgentBurn defines it on the methodology page as the extra tokens an orchestration layer adds over a raw API loop on matched tasks. The reported percentage is the share of token spend attributable to system prompts, tool schemas, serialised state and duplicated message history. A raw loop — one model call, one tool call, one answer — is the 0% baseline.
This is not a benchmark of intelligence. It is an accounting question: when two stacks solve the same task set, how much of the invoice is the framework talking to itself?
2. The measured spread: 0% to 1,056%
The dataset's framework overhead board cites a 450-run study on GPT-4o, matched tool-use tasks at temperature 0. The numbers are tokens per run, with AutoGen as the leanest baseline:
| Framework | Tokens per run | Overhead vs AutoGen | What drives it |
|---|---|---|---|
| AutoGen | 379 | 0% (baseline) | Leanest loop in the study |
| LangGraph | 422 | +11% | Graph state and node bookkeeping |
| OpenAI Agents SDK | 472 | +25% | Role and tool scaffolding |
| OpenAI Agents SDK + MCP tools | 472 | +25% | MCP adds no measurable tokens once the server connection is reused |
| CrewAI | 4,382 | +1,056% (11.6x) | Role, goal and backstory prompts dominate |
Two findings deserve to be separated from the headline:
- MCP is not the problem. Adding MCP tools to the OpenAI Agents SDK left the token count unchanged once the server connection was reused. Tool routing itself is cheap; the expensive part is what the framework puts in the prompt around the tools.
- Multi-agent frameworks are a different cost class. CrewAI's 4,382 tokens per run is not 11% worse than AutoGen — it is 11.6 times the total. Role/goal/backstory prompts are sent per agent, and a crew repeats the transcript as it hands off.
3. Why multi-agent overhead compounds
A single-agent framework overhead is roughly constant per run: fixed prompt plus tool schemas. A multi-agent framework grows with every handoff. Each agent gets its own instructions; each turn tends to resend the conversation so far; state serialisation adds a copy of what the previous agent already knew. If a crew has four agents and three handoffs, the same history can appear in the bill four times.
That is the mechanism behind the 11.6x figure, and it predicts where the tax gets worse: more agents, longer transcripts, more context retention between steps. The same mechanism also pushes you into context compaction, which is itself a token cost.
4. When 93 tokens per run matters
Per run, the difference between AutoGen and the OpenAI Agents SDK is 93 tokens — irrelevant for a demo. At fleet scale it is a line item. The arithmetic below is illustrative of the method, not a measurement of anyone's fleet: it assumes 30,000 runs per day and lists tokens at $2 per million, with no caching.
| Stack vs AutoGen | Extra tokens / run | Extra tokens / month (30k runs/day) | Monthly cost at $2/1M |
|---|---|---|---|
| LangGraph (+11%) | 43 | 38.7M | $77 |
| OpenAI Agents SDK (+25%) | 93 | 83.7M | $167 |
| CrewAI (+1,056%) | 4,003 | 3.6B | $7,206 |
Two caveats keep this honest. First, prompt caching changes the price: cached input at $0.20 per million reduces the dollar column by up to an order of magnitude on a warm context. Second, a framework with more overhead can still be the cheaper stack overall if it raises the success rate — overhead is one term in cost per solved task, not the verdict.
5. How to measure your own stack
- Freeze a task set. Ten to twenty tasks with a deterministic check. Without a fixed set, every number is noise.
- Run a raw-loop baseline. One model, one tool loop, no framework. This is your 0%.
- Log tokens per run, split by input, output and cached. A single total hides the cache hit rate, which is where most of the difference lives.
- Count retries and compaction separately. Frameworks often surface their overhead as extra retries rather than as a bigger first prompt.
- Divide by success rate. A cheaper run that fails more often may cost more per solved task.
The telemetry SDK and OpenTelemetry bridge exist for step 3: post input, output, cached tokens, tool calls, retries and outcomes per run, then compare the aggregate against the same task set on the raw loop.
6. What overhead does not capture
Latency, reliability, maintainability and developer velocity are outside the unit. A framework that adds 25% tokens but removes two days of plumbing is not necessarily the wrong choice. The point is to make the trade explicit: at fleet scale, "we chose a framework" is also a recurring line item, and it deserves a number rather than an assumption.
One more boundary: this board measures a single 450-run study on one model family. It is evidence of a mechanism, not a universal ranking. Your stack, your tasks, your cache policy — measure it.
Sources
- AgentBurn framework overhead board — benchmarks.json (frameworkOverhead), snapshot 2026-10-27.
- agentic-framework-bench, 450-run tool-use study on GPT-4o, temperature 0 — RESULTS.md.
- AgentBurn methodology — framework overhead definition and provenance rules.