If you look at the sticker price of AI models, you’re missing the point. For the last year, engineering teams have been obsessively tracking the price tag on LLM tokens, treating it as the primary indicator for operational success. It’s an understandable mistake; we’ve been trained by years of cloud infrastructure pricing to look for the cheapest component costs. But AI isn't cloud storage.
When you buy a token, you aren’t buying a product—you’re buying an attempt at a result. If that attempt fails, you pay for the tokens anyway and then pay again for the retry. Sometimes, you pay for even more tokens the second time because your harness decided to stuff a larger context into the prompt to "fix" the failure.
It’s becoming increasingly clear that the industry's obsession with per-token pricing is actively misleading. The cheapest model per-token is often the most expensive model per-task, and we have enough data now to stop playing this game.
The Mirage of Per-Token Efficiency
Databricks recently decided to stop looking at standard industry benchmarks like SWE-Bench for their internal decisions. They labeled them as "broken"—and they were right. Instead, they built their own task-based benchmark, putting models through the actual engineering work performed by their own teams.
The findings should be a wake-up call for every CTO in the industry. They compared models like Anthropic's Opus 4.8 and Sonnet 5. On a surface level, Sonnet 5 is significantly cheaper per-token. If you are calculating your AI budget in a spreadsheet based purely on that listed price, you’d assume Sonnet 5 would save you a pile of money.
You’d be wrong.
Opus 4.8, despite its higher token cost, turned out to be cheaper for real-world tasks. Why? Because it completed tasks more often and used its token allocation more intelligently. Sonnet 5 suffered from a lower success rate and, in its attempts to recover, ended up burning through a massive amount of extra tokens. You’re paying for the tokens consumed during the failure, the recovery, and the inefficient context management. Suddenly, that "cheap" model is a massive cost sink.
Your Harness Is the Real Bottleneck
It isn't just the foundational model that drags down your ROI; it’s the layer of software managing that model. We call it the "harness"—the coding agent that sits between you and the LLM, managing the context, calling the tools, and aggregating the results.
This is where the real cost differences lie, and they are shocking. The same model, managed by two different harnesses, can show a 2x cost difference in total performance.
This happens because of context bloat. The harness dictates what information is passed into the LLM with every request. If your harness is poorly engineered—if it’s blindly shoving unnecessary logs, redundant file histories, or oversized context windows into every API call—you are going to pay for it, literally.
Databricks found that harnesses with minimal system prompts performed with drastically smaller total token counts per task compared to their feature-bloated competitors. When you add up the millions of inferences your team runs, that 2x efficiency difference is the difference between an AI team that brings value and an AI team that gets its budget slashed by the finance department in Q4.
Stop Benchmarking for Marketing Stats
There’s a dangerous trend of engineering teams optimizing their systems for benchmarks that don't match their reality. SWE-Bench metrics, for example, are great for public relations but often fail to account for the actual, messy, long-horizon tasks your developers face in production.
If you want to know what your AI costs actually are, stop looking at the vendor’s performance sheet. You need to build a harness that tracks the "cost per completed task" over a sample of your actual existing codebase.
Measure the completion rate. Measure the total tokens spent per completed task, including all retries.
This takes effort. It requires building your own evaluation pipeline, your own logging, and your own observability. But if you aren't doing this, you are flying blind, assuming the lower token price is helping you when it might be quietly cannibalizing your operational budget.
Rethink Your AI Infrastructure
The move to agentic AI requires a shift in how we think about software costs. Tokens are just a raw resource, like electricity. Electricity is cheaper in some regions, but if your machine is fundamentally broken, you’re just wasting cheap electricity on an inefficient task.
Stop looking for the cheapest token. Start looking for the model-harness combination that achieves the highest successful task-completion rate with the smallest amount of context padding.
That’s not flashy, and it’s not something you’ll get from a vendor’s marketing department. It’s hard, boring, iterative engineering. That’s exactly why it works.
