Higher Benchmark Scores Don't Mean Lower Cost
Alibaba released Qwen 3.8-Max this week and marketed the preview as second only to Claude Opus 5. Their launch table was more equivocal, honestly: the model leads on one of 12 coding-agent rows. But an independent benchmark harness came close to the opposite conclusion. A VulcanBench run, apparently using the Preview version, put Qwen 3.8-Max's best-effort setting mid-pack and its default setting dead last.
Both results are real. Both are defensible. The gap between them isn't about model quality—it's about token and time budgets. And those figures are almost never the headline numbers.
That's the problem with raw benchmark scores. They don't predict your bill. What matters is cost allocation for shared GPU clusters, because when you're running multiple models across a team, the cheapest model per token is rarely the cheapest model per outcome.
How Much Does an LLM Cost?
The comparison everyone published in Qwen 3.8-Max's first week was a price comparison. That was the only data available, so it made sense. It's not a cheap model. DeepSeek-V4-Flash-0731, which entered public API beta on July 31, lists at 14 cents per million input tokens and 28 cents output. Qwen 3.8-Max lists at $2 and $6. Kimi K3 sits at $3 and $15.
Those prices tell you less than they used to. The reason is specific to reasoning models like Qwen: getting to a result costs thinking tokens. A model that spends most of its token allowance on reasoning can reach a token cap before it writes the answer, giving you an empty result indistinguishable from a total failure at the cost of a full run.
Artificial Analysis has the cleanest published measurement of how this can affect real agent spend. Running its Intelligence Index on DeepSeek-V4-Flash at maximum effort took 210 million output tokens against a class median of 100 million. Absolute cost stayed low anyway, because the tokens were so cheap. But verbosity costs time, not just money. Depending on your use case, that can sink you.
So how much does an LLM cost? The honest answer is: it depends on what counts as a success, what budget you set, and whether your model runs out of tokens before it runs out of ideas.
The Benchmark Score Illusion
The discrepancy between Alibaba's numbers and independent testing comes down to time budgets. Alibaba's footnotes give its coding numbers a five-hour timeout, and up to 12 hours per run on PaperBench. The independent harness, VulcanBench, allowed between 45 and 60 minutes of wall clock time.
A time budget between five and 16 times larger on Alibaba's side explains the huge difference in results. That's not a flaw in either benchmark. It's a feature of how these models work. Reasoning models need time to think, and the ones that think harder tend to produce better answers—if they have enough budget to finish.
Long-Horizon-Terminal-Bench, published in July, ran 17 frontier models across 46 tasks through a shared harness with one 90-minute attempt each. Timeouts accounted for 79% of unresolved runs, against 19% for agents that stopped on their own and 3% for harness errors. The authors are careful about what that does and does not mean: the timed-out runs were not close to finishing, with mean reward between 0.10 and 0.35, so you cannot assume more time would have resulted in success.
But the lesson is clear: benchmarks are implicitly measuring time efficiency, whether or not they shout about that.
Why Your Failure Rate Is a Configuration Setting
The clearest published example of the mechanism comes from VulcanBench, the same open-source harness behind the Qwen chart. In a report dated July 26, Claude Opus 5's lowest-effort setting was its best, solving 20 of 23 tasks against 18 at high effort. The extra reasoning wasn't useless: high effort returned the fewest wrong answers of any setting, one against three. It ran out of clock instead, and a timeout scores zero. Two of its three regressions were cutoffs on tasks that low effort solves, and given unlimited time on both it only ties its cheapest setting, at 3.1 times the cost.
That has a direct consequence for anyone building a routing ladder. The standard design escalates to more reasoning when a cheap attempt fails, on the assumption that the next rung is better and merely costs more. For a meaningful share of model and task combinations that assumption is wrong, and you pay the higher rung's price to escalate into a timeout or hitting a cap.
A run that produces a wrong answer and a run that runs out of budget are different events with different fixes. Almost no harness distinguishes them, and almost no leaderboard reports the split. I hit this building an agent benchmark of my own: the harness logged a failure and nothing about why, and I had to add the distinction myself. When you do separate them, budget exhaustion turns out to dominate.
Who's Already Measuring This
Several groups have landed on cost per successful task independently in the last few months, which is the strongest signal it's becoming standard.
VulcanBench reports dollars per solved task as a headline column and has since its earliest reports. Long-Horizon-Terminal-Bench publishes per-task cost next to accuracy, and its most instructive row is GPT-5.4 at roughly $26 per task with a much lower pass rate than Grok 4.5 at about $11. TestEvo-Bench runs agents under a cost cap, and Claude Code's test-generation score falls from 71% to 44% at the tighter cap.
Vendors are already on board with the idea of measuring per successful task. HubSpot moved its Breeze Customer Agent in April to 50 cents per resolved conversation, down from $1 per handled conversation. Zendesk bills per automated resolution. Fin charges 99 cents per outcome and bills only on end-to-end resolution.
For teams managing cost allocation for shared GPU clusters, this shift has real implications. When you're splitting compute resources across multiple AI workloads, knowing that a model costs $26 per successful task rather than $11 per attempted task changes your capacity planning entirely. It changes which models you route to which clusters, which effort levels you enable, and whether you're paying for a model's thinking or just its failures.
What to Change This Week
Emit a failure reason on every agent run as a required field, with budget exhaustion, verifier failure and harness error as distinct values rather than one failure flag. Until you can separate a timeout from a wrong answer, your pass rate is measuring two things at once and you cannot tell which one to fix.
Compute cost per successful task per effort level, not just per model. Total spend including failed attempts, divided by tasks that passed your acceptance check. The ranking will not match the rate card, and the cheapest setting may well win.
Cap on tokens rather than wall clock unless latency is genuinely in your service level objective. A wall-clock cap scores your provider's serving speed as model quality.
Check the default effort setting on everything you have deployed. Qwen 3.8-Max runs at its highest reasoning setting when the effort field is unset, and its highest setting was its worst performer in independent testing. A team that never touches that parameter is running the configuration that costs the most per solved task.
The bottom line: if you're measuring benchmark scores to choose models, you're measuring the wrong thing. Cost per successful task is the metric that actually matters for routing decisions, capacity planning, and cost allocation for shared GPU clusters. Everything else is just noise.
Source: Qwen 3.8-Max and Claude Opus 5 show why raw benchmark scores don't predict the bill | Laurie Voss, VentureBeat, August 6, 2026
