I've had a problem for a while with the cost-per-task charts that show up in every benchmark right now. So I asked to have my hunch tested against our own data:
(The prompt is verbatim, in Danish, typos included — that's how it was actually typed. Roughly: "I've always hated these cost-per-task graphs people use in benchmarks. Can we prove from our own transcripts that we use fewer and fewer tokens to solve tasks over time?")
The answer was no, and not for the reason I expected. The price per unit rose on all three measures — because the unit itself grew faster than the price fell. The "no" is a symptom, not a verdict: in a real project the task never stands still long enough to put a price on it.
What was actually measured
Every session file from February to October 2026: 14,243 JSONL files, 366,966 billed API calls, each priced at its own model's list price.
There's one trap in that arithmetic worth knowing if you try it yourself. Claude Code writes one line per content block in a response, and every line repeats the whole response's usage. Add them up naively and you overcount by a factor of 2.5. requestId is assigned once per request — once per bill — so that's what you deduplicate on.
The hard part isn't the numerator. It's the denominator. "Cost per task" needs a task, and we don't have two alike. So I used three denominators drawn from three different systems, which therefore can't co-vary with token spend by accident:
- Human prompts — how many times a human actually typed something, from the prompt log
- Commits — feat and fix commits across the root repo and every submodule, from git
- Hours at the wheel — hours with at least one agent running, from the agent-hours ledger
If the cost increase were a measurement artifact, the three would diverge.
From March to August a commit got 9.3× more expensive, an hour at the wheel 5× more expensive, and a prompt 7.5× more expensive.
March is the baseline. September and October appear in the charts, but I don't count them, and the reason differs per measure: the prompt filter discards 62% of October's prompts against 9% of March's, making the denominator artificially small, and some of October's commits weren't written yet when I measured. Hours at the wheel is the most robust of the three — that number is simply noisy in a month with 124 hours instead of 356.
Every instruction buys far more machine work
A prompt in March triggered about a thousand output tokens. A prompt in August triggered fourteen thousand. A factor of 12 from March to August.
We didn't make the tasks cheaper. We made the instructions bigger.
Part of that is a dial we turned ourselves. From July almost everything ran at high effort, and from August I can see how much of the output is the model's reasoning rather than delivered text: 32% in August, 29% in October. The field doesn't exist in the data before August, so March is unknown — but the direction isn't in doubt.
We climbed the model ladder, not down it
If we'd got cheaper per task, it should show up as a cheaper basket of models.
The opposite happened. March ran a fifth of its output on haiku; October runs on opus and fable, and fable-5-1 is the most expensive model on the price list — $50 per million output tokens against haiku-4-5's $5.
The blended unit price only doubled, though, rather than going up tenfold. That's the one thing in the dataset pointing the other way: prices within the families fell along the way. Opus cost $5 per million input tokens and $25 per million output tokens in the spring; $4 and $20 in the autumn. So we did get a share of the cut — it just disappeared into the fact that we moved upmarket at the same time.
One last thing that is large but proves nothing about us: caching. Between 95% and 98% of all input tokens arrive as cache reads at a fraction of the input price. The input side cost 255,000 kroner; without caching it would have cost 1.69 million. But the factor is flat — the same from March to October. That's just how agentic loops work on their own — it's the product's architecture, not something we achieved.
Why the question can't be asked
"Cost per task" requires the task to be held constant. We never did, and that's checkable. I clustered all 38,306 prompts on their normalized opening sentence to find tasks we'd repeated over time. Nine clusters came out spanning three months or more. All nine are slash commands and boilerplate — /model, /compact, context continuations. Not one real task was repeated.
We do have ten runs of the same design task on disk, and those genuinely are comparable — but they all sit inside a two-week window in May. They measure prompt changes, not time.
It hits our own numbers too. The same instruction today buys an autonomous run twelve times longer than it did in March, so a curve drawn across those eight months sets two different things side by side and calls them the same.
In a real project you don't sit and run the same tasks over and over. So eight months in a real project looks like this: everything gets more expensive per unit, because the unit keeps growing.
The unscientific part
Then there's the thing I can't measure but can definitely feel: over the months it just feels like it runs better and better, more autonomously, with fewer mistakes.
The first half of that has a number under it. "More autonomously" is precisely what the output-per-prompt chart shows — each instruction buys a longer stretch where I don't have to step in. Feeling and measurement point the same way.
The second half I can't back up. I tried: if there were fewer mistakes, the share of fix commits should fall.
It falls from 45% in March to 30% in July — then jumps back to 54% in September. There's no direction to hang a claim on. And a fix commit isn't the same as an agent mistake anyway; it might just as easily be fixing something a human wrote in 2025.
So "fewer mistakes" stands as what it is: my experience, not a result. I'm including it because it's honest, and leaving it unadorned rather than dressing it in a number that won't hold.
The benchmark is right — it just measures something else
Let me be fair to the charts I started out complaining about. A benchmark that runs the same task over and over genuinely can answer whether that task got cheaper. It's a reasonable question, it's a properly measured number, and the answer is probably yes. Nothing wrong with it.
That number just doesn't transfer to your own budget. Because in a real project three things happen at once, and all of them pull the other way:
- The tasks get bigger. The same instruction buys twelve times more machine work than it did in March.
- They run longer. A stretch I used to have to interrupt now runs to the end on its own. That's the same measurement as above from another angle — I can't separate the two in the data.
- More of them run at once. For every clock-hour with at least one agent running, 1.3 agent-hours ran in March. By August it was 2.4.
All three are direct consequences of the models getting better — and all three cost more tokens, not fewer. The gain never reaches the bank. It goes straight back to work.
So a falling price per task does not mean a falling AI bill. For us it's closer to the opposite: 506 kroner in February, 75,222 in July, 294,167 in total from February to October. The better the models get, the more we hand them.
What the numbers don't cover
For the record, because it changes how hard you can lean on them:
- The prompt denominator is fragile. Machine-generated prompts are filtered out, and the filter removes 9% in March against 62% in October. Errors in that filter bias the trend.
- October commits are undercounted. Some of October's work was uncommitted when I measured.
- Work moved to other tools is invisible here. This is Claude Code sessions, not all work.
- The reasoning share before August can't be recovered. The field wasn't written until August.
The price per task can absolutely fall in a benchmark. The bill does not.