Prompt caching and the agent bill: what a 90% hit rate is worth

1. The cached-input price gap

Most providers in the pricing snapshot sell the same token at two prices. A token that has to be read fresh costs the full input rate. A token the provider already has in its cache, at a prefix it has seen before, costs a fraction of that. The gap is not a rounding difference.

Three details matter more than the headline percentages. The discount applies to input tokens only — output is billed at the full output rate, so the most a cache can save is the input share of the work. It is not universal: Kimi K3's row in the snapshot carries no cached-input rate, so the calculator disables its cache control for that model rather than assuming a discount. And the discount varies by more than 20 points between providers, which means a cache hit rate is only meaningful next to the model it was achieved on.

2. What $13.30 per task implies

The Claude Code row in the open dataset is 24.1M tokens and $13.30 per task (Artificial Analysis Coding Agent Index v2.0, Sonnet 5.5 at max effort). Divide the cost by the tokens and you get an implied blended rate of $0.55 per million tokens.

That number is strange until you price the same token pool without a cache. All 24.1M tokens at Sonnet 5.5's fresh input price of $2/1M is $48.20. The published cost is a little over a quarter of that. The comparison is a sanity check rather than a model — it charges every token at the input rate and ignores output, so it is not the correct uncached figure either — but it fixes the order of magnitude: cache working versus not working is a 3.6x swing in the bill before any other variable moves.

The dataset's definition explains why. Tokens per task counts input, output and cached tokens, so a long session that re-reads a 1M-token context on every turn produces an enormous token count at a very small average price. The Claude Code profile describes exactly that: the context is kept warm through caching, and most of the spend is re-reading context rather than generating output. So a cache hit rate is not a discount bolted onto an agent's cost: on long-context agents it determines the token pool's average price.

3. Sensitivity: 0%, 50% and 90% hit rates

To see the slope, hold everything else fixed. Assume an 80/20 input/output token split, Sonnet 5.5 list prices ($2 input, $10 output, $0.20 cached input) and a 24.1M-token task. The split is our assumption, not the dataset's, stated so you can substitute your own.

blended $/1M = (input_share * input_price) + (output_share * output_price)
             - (cache_hit_share * input_share * (input_price - cached_price))
task cost     = (tokens_per_task / 1e6) * blended $/1M

The ceiling is the point. Under an 80/20 split, caching can never remove more than 40% of the bill, because only the input half is discounted. Going from 50% to 90% hit rate is worth 16 points of cost; from 90% to 100%, only 4. Returns flatten well before the hit rate does, which is why "get to 90%" and "get to 99%" are different projects.

4. Four ways the cache goes cold

Every mechanism below has the same effect: the cached prefix stops matching, and the next turn is billed at the full input rate for context the agent had already read.

• Prefix edits. A cache is a prefix match, so a change early in the prompt invalidates everything after it. Injecting a timestamp, ticket ID or user name at the top of the system prompt is the classic own-goal: a one-line change that re-prices the entire context. Reordering the tool list does the same thing.
• Tool schema changes. Tool definitions normally sit in the cached prefix. Adding or removing tools per task, connecting an MCP server mid-session, or editing a tool description rewrites the prefix.
• Context compaction. When the window fills, the harness summarises the transcript. That replaces the prefix: the next turn pays fresh input on the new one and has lost the warm cache. The sessions that benefit most from caching are the long ones, and those are also the ones that compact.
• Model switches. A cache belongs to the model that wrote it. Routing a step to a cheaper model and back starts each model cold, so a cost-saving router can cancel part of its own saving in input pricing.

There is also a failure the snapshot cannot price. models.json records a cached-input read rate only; it records no cache-write premium and no cache lifetime. Providers may charge more to write an entry than to read it, so a low hit rate can be worse than no caching at all. That break-even is provider-specific and outside the dated snapshot — it is a line to check on your own bill, not one we can compute for you.

5. Which agents are cache-friendly

Caching rewards repetition and punishes variability, which gives a short and fairly reliable checklist.

Good fit: a long, stable system prompt and instruction set; a static tool schema; repository or document context that is re-read every turn; multi-turn sessions where each turn adds a small delta to a large prefix; deterministic ordering, so the same prefix hash recurs.

Poor fit: one-shot calls that never read the same prefix twice; short prompts where the cacheable prefix is a small part of the request; per-request tool sets or MCP servers; workflows that inject volatile data at the top of the prompt; sessions long enough to compact repeatedly.

The measurement that settles it is cache-read tokens per call, split from fresh input. A long-prompt agent with a low cache-read share usually has an ordering problem: move every volatile field after the stable block. The telemetry SDK and OpenTelemetry bridge captures that split; without it cache behaviour is invisible, because the invoice shows a total and not a hit rate.

6. What this snapshot cannot tell you

Provenance limits are part of the analysis. Prices are official list prices from the 2026-10-27 snapshot: no negotiated rates, no batch discount, DeepSeek at peak-hour rates. No vendor publishes a cache hit rate, so the sensitivity table is a model with a stated assumption. The write premium and cache lifetime are not recorded at all, and $13.30 is a benchmark-harness figure, not a median repository.

The honest reading is narrower than the numbers suggest: on long-context agents the cache hit rate is the difference between paying the cached price for most of your tokens and the input price for all of them, and it is a property of your prompt structure that you control. Check the model-level rates in the pricing snapshot, then price your own split in the calculator and carry the result into cost per solved task.

Sources

• Cached-input and input list prices — models.json, dataset snapshot 2026-10-27 (22 models; official list prices; DeepSeek at peak-hour rates; Kimi K3 has no published cache rate). Provider rate cards, e.g. claude.com/pricing.
• Claude Code tokens and cost per task, plus the warm-context description — projects.json (claude-code), snapshot 2026-10-27; underlying figures from the Artificial Analysis Coding Agent Index v2.0 comparison.
• Blended-rate formula, token definition and pricing caveats — AgentBurn methodology.
• Model pricing pages cited per row (Anthropic, OpenAI, Google, DeepSeek, Alibaba, Moonshot, Z.ai, Mistral, xAI, Meta) — recorded in models.json provenance notes.