The Gold Rush Isn't Over. It Just Changed Venue.
Early in the AI wave, enterprise strategy was embarrassingly simple. Pick the shiniest model. Buy the most GPUs you can afford. Ship something that demos well at the quarterly review. "They were magic boxes and would do magic things," recalls Brad Shimmin, vice president and practice lead for data intelligence at The Futurum Group. "Companies wanted to get the best model, then spend the most they could on infrastructure to run it."
That sentence reads like a period piece now. It's also a warning. The spend didn't slow down when the magic faded — it just migrated from training to inference, from procurement to operations. And nobody budgeted for that.
If you're a CFO or cloud architect trying to forecast what scalable AI compute actually costs an enterprise in year two of deployment, the honest answer is: more than year one, and more than your vendor's pricing page implies.
Inference Overtakes Training, and Nobody Adjusted the Spreadsheet
Here's a number that should land hard: AI inference costs are projected to increase more than fivefold through 2028, according to Gartner. That's not a rounding error or a pilot project gone sideways. That's the core operational bill for a company that's actually using these models in production.
The shift is structural. Spending on AI-optimized infrastructure as a service — the compute that supports large language model training and operation — nearly doubled to $42 billion by year-end 2025, per Gartner. More telling: global spending on inference ($23.3 billion) surpassed spending on training models ($19 billion). Inference is where the money lives now.
Why? Because agentic AI — multi-step reasoning chains, tool-calling loops, autonomous workflows — multiplies the compute footprint per task. A single chatbot query costs pennies. A five-step agentic pipeline that retrieves data, reasons over it, validates against a knowledge base, generates output, and formats it for a specific system? That's a completely different bill. Product leaders won't be able to rely on more efficient tokens to offset the growing costs of AI infrastructure, according to Gartner's analysis.
The model you chose eighteen months ago isn't what's driving your infrastructure spend anymore. The way your teams use it is.
The Token Price Paradox
This is the part that confuses people. Per-token prices have fallen dramatically. GPT-4's successor models deliver better output at lower unit cost. On paper, the trend line looks like it should flatten total spend.
It doesn't. Gartner's data showed that the rate of innovation in AI capabilities is currently outpacing the cost savings enterprises experience from their AI providers. You get a cheaper token, and within six months there's a new capability, longer context windows, multimodal inputs, reasoning traces, that makes you want to consume ten times more tokens per workflow.
It's Jevons paradox wearing a neural network. Efficiency gains fuel demand growth, which destroys the savings you expected.
How much does an LLM cost your enterprise in practice? Not per million tokens, in aggregate. If you're running agentic workflows across a customer support platform, a code assistant, and a document pipeline simultaneously, the compute bill scales with the complexity of those workflows, not with the per-token price of whichever model you've pinned in your config file.
What Is Cloud Cost When AI Joins the Party?
Cloud cost used to be tractable. You provisioned instances, you tagged resources, you built a dashboard, you had a meeting about reserved instances. AI changed that arithmetic entirely.
Generative AI now accounts for more than half of public cloud services used by enterprises, according to Flexera's 2026 State of the Cloud report. That means the majority of a company's cloud bill is now tied to services where the pricing model is consumption-based, where output quality varies with prompt engineering, and where a developer can accidentally double your monthly spend by switching from a small model to a large one during a debugging session.
73% of organizations operate hybrid cloud architectures, up three percentage points year over year. AI adoption compounds that complexity. You've got training clusters in one provider, inference endpoints in another, vector databases on a third, and a GPU-heavy data pipeline running on bare metal that you own. The "cloud cost" question is no longer "what are we paying AWS?" It's "what are we paying everywhere, for AI, in ways we can't yet attribute to a business outcome?"
FinOps Grew Up Because AI Forced It To
This is where the story gets productive rather than just alarming. FinOps teams, historically the people who nagged developers about stopping idle VMs, have been pulled into a genuinely strategic role because of AI, as engineering and finance learn to manage cloud cost control together.
Nearly half of FinOps teams now align spending with business outcomes, compared with 40% the year before. Four in five report to the CIO or CTO, up from 61% in 2023. At Capital One, Jerzy Grzywinski, senior director of cloud governance and FinOps, described his team sitting "at the center of the conversation" for strategic AI investments with the C-suite.
The questions they ask are concrete and useful. "Do we want to host GPUs to train our own models? Do we want to leverage a SaaS offering for developer productivity or some capability?" Grzywinski told CIO Dive. "And we really model out what we think the cost of those will be. How does that weigh against other spend that we have in our portfolio?"
That's not cost-cutting. That's portfolio management for compute. The shift matters because it reframes the entire conversation from "how do we spend less" to "which investments actually produce value."
The Model Is Rented. The Data Is Owned.
There's a line from Shawn Rosemarin, vice president of R&D at Everpure, that should be printed on the wall of every AI strategy meeting: "The model is what you rent. The data is what you own."
This isn't philosophy. It's infrastructure advice. Models change constantly, providers deprecate versions, reprice tiers, restructure APIs. The one thing you can rely on is your data. And the "harness" around that data, the retrieval systems, the context windows, the validation layers, the AI edge infrastructure that keeps latency tolerable, that's where the durable competitive advantage lives.
"It's the firewall between probabilistic AI models and the deterministic business facts they need to get right," Shimmin says of the data-and-infrastructure layer surrounding the model.
Companies that bet everything on model selection and threw capital at raw compute are finding themselves in an awkward position. They optimized for the variable that changes fastest. They underinvested in the one that compounds.
Building Scalable AI Compute That Doesn't Bankrupt You
So what does the responsible play look like when inference costs are rising fivefold and your FinOps team can't model the spend because nobody knows what workflows will look like in Q3?
Start with attribution. You can't manage what you can't see. The teams that are pulling ahead on AI efficiency are the ones who built per-workflow cost tracking before they had the budget crisis, not after.
Then build the data layer properly. Retrieval-augmented architectures, fine-tuned small models for narrow tasks, caching strategies for repeated queries, these aren't exotic optimizations anymore. They're table stakes for anyone running scalable AI compute in production.
And the compute itself? The global infrastructure scramble is real. Reflection AI's billion-dollar compute pact with Nebius shows that frontier labs are locking in capacity years ahead. Enterprise buyers are competing with AI-native startups for the same GPUs. If you're waiting for a price inflection point, understand that the supply side is being shaped by buyers who think ten years ahead and price accordingly.
The model gold rush gave way to a data gold rush. The data gold rush is giving way to an infrastructure reality check. Each phase was shorter than the last. Budget for that velocity.
