Introduction
The training-inference compute tradeoff shapes how AI labs balance resources between building models and running them. Ege Erdil's analysis at Epoch AI reveals that, under typical conditions, the setup minimizing total compute cost will generally spend comparable amounts on training and inference. Yet the task before us surfaces a more pointed question: if every known tradeoff has been adjusted to maximally favor inference, and inference still dominates compute costs, is there any remaining room to save? The answer, rooted in the empirical evidence summarized by Villalobos and Atkinson (2023), is that we may have simply run out of levers.
The Tradeoff Mechanic
In general, it is possible to improve model performance through two distinct paths: spending more compute on training, or spending more compute during inference. The tradeoff table from Villalobos and Atkinson catalogs the principal techniques. Overtraining on a bigger dataset, chain-of-thought prompting, web browsing, and repeated sampling each offer a span of roughly one order of magnitude (OOM) of compute savings in one domain at the expense of increasing the other domain by somewhat more than one OOM. Monte Carlo Tree Search (MCTS) for game-playing models delivers a comparable 1.6 OOM span in both directions. Pruning and repeated cheap verification each hover near 1 OOM; repeated expensive verification can save 1 OOM of inference compute at the cost of 1.45 OOMs in training. Chain-of-thought prompting's span remains uncertain.
The table's central thesis, as Villalobos and Atkinson articulate it, is that each technique makes it possible to save around one OOM of compute in either training or inference, in exchange for increasing the other factor by somewhat more than one OOM. For exposition, the "somewhat more than" is ignored and a clean rule of thumb is adopted: approximately, we can spend one OOM more on training compute and one OOM less on inference, or vice versa, without changing model performance. This tradeoff eventually breaks down, but it holds on a wide range of tasks near current levels of training and inference compute. Moreover, the techniques can often be stacked, extending the range over which the tradeoff remains viable.
Why Investment Parity Emerges
The parity result follows from a simple algebraic insight. Suppose a lab spends 1 zettaFLOP on training and 100 zettaFLOPs on inference. By increasing training compute by a factor of 10 and reducing inference compute by a factor of 10, the lab preserves performance while driving total compute from 101 zettaFLOPs down to approximately 20 zettaFLOPs — a fivefold reduction. The general formula is equally clear: if we can trade β orders of magnitude of training compute for α orders of magnitude of inference compute, the cost-minimizing allocation spends a fraction α/(α+β) on training and β/(α+β) on inference. When α and β are both near 1, as the empirical evidence suggests, the optimal split lands near 50-50.
A concrete numerical illustration drives the point home. With a total compute budget of 300 zettaFLOPs, a naive 150-150 split is suboptimal. Shifting training up by 0.1 OOM and inference down by 0.2 OOM reduces total use to about 283.4 zettaFLOPs. The efficient allocation places 200 zettaFLOPs on training and 100 on inference; starting from that point, no further tradeoff can reduce total compute without degrading performance. The reason is that the marginal benefit of rebalancing diminishes as the allocation approaches the efficient ratio.
Empirically, the tradeoff table does not support values of α distant from 1. Minor deviations of α from 1 have small effects on the optimal split; only extremely large departures — α below 0.1 or above 10 — would make training and inference compute orders of magnitude apart. Since the known techniques cluster around α ≈ 1, we should expect training and inference compute to remain in the same ballpark.
When Tradeoffs Run Out
The conclusion that training and inference spending should be roughly comparable is robust to plausible uncertainty about the tradeoff size. But the analysis explicitly acknowledges a boundary condition. If every tradeoff has been adjusted to maximally favor inference and inference still dominates the compute cost, there may simply be no more we are able to do to save on inference costs. Perhaps we have exhausted the known techniques, or perhaps unknown methods will eventually emerge to extend the tradeoff's range. The possibility remains open, but it is uncertain.
A few less significant factors also merit mention. Inference may face tighter memory bandwidth constraints than training, and future compute could be cheaper due to Moore's law and time discounting. The model assumes AI developers have perfect foresight about future model demand — an assumption that fails in practice. These effects are real, but they do not overturn the basic qualitative result: training and inference compute should stay within an order of magnitude of each other.
Notes on Remaining Uncertainty
When training a model, both forward and backward passes are required, and backward passes are generally twice as expensive as forward passes. Inference, by contrast, requires only forward passes. The analysis also ignores heterogeneity in the models used to serve requests and the variance in context lengths across users. Importantly, the conclusion does not require the 1:α tradeoff to hold across many orders of magnitude; it suffices that the tradeoff holds locally in the neighborhood of actual lab choices. Along a performance Pareto frontier, α equals the negative of the local elasticity of inference compute with respect to training compute, making the conclusion somewhat more robust than it might first appear.
Related
For related coverage on compute tradeoffs, see How AI Models Trade Training Compute for Inference Power. For practical inference cost reduction approaches, see Execution-Grounded LLM Routing: How ACRouter Slashes Inference Costs by 2.6x.
Source: Erdil, E. (2024), "Optimally allocating compute between inference and training", Published online at epoch.ai. Retrieved from https://epoch.ai/publications/optimally-allocating-compute-between-inference-and-training
