Research notes
Key argument: Toby Ord claims RL scaling primarily increases inference costs creating persistent economic burden.
The counter-analysis finds that while RL scaling does increase inference costs, the cost to reach a given capability level falls rapidly over time due to algorithmic improvements, model distillation, hardware advances, and more efficient reasoning. The RL scaling data is thin, and the 10,000x compute estimate is uncertain. Inference cost reductions of 5-10x per year make the burden more transient than it appears.
Toby Ord's persistent burden claim
Toby Ord has argued that reinforcement learning (RL) scaling primarily increases inference costs, creating a persistent economic burden for deploying increasingly capable models. The concern is that as models are trained with RL, the resulting systems require more compute for inference, which translates into ongoing operational costs that accumulate over time. This framing suggests that the economic burden of AI progress is durable — each advance in capability begets a new layer of persistent inference costs.
While this intuition resonates with straightforward extrapolations of compute growth, it overlooks several well-documented dynamics in machine learning that systematically reduce the cost per capability level. The counter-analysis below examines each of these dynamics and their cumulative effect on the economic sustainability of RL-scaled systems.
Algorithmic improvements reduce compute per capability
One of the most powerful forces working against a persistent burden is algorithmic progress. Historically, each generation of machine learning algorithms has delivered equivalent or superior performance with significantly less compute. For RL specifically, researchers have developed more efficient policy gradient methods, better value function estimation, and improved exploration strategies that achieve the same performance with less training compute.
The key insight is that the cost to reach a given capability level — not the absolute cost of the latest model — determines the economic burden. If algorithmic improvements halve the compute needed for a given level of performance every 12-18 months, then the economic burden of reaching any fixed capability threshold actually declines over time, even as RL techniques are applied more broadly.
Empirical evidence from the broader ML field shows that algorithmic efficiency improvements have consistently outpaced raw compute growth. A roughly 2-3x annual improvement in algorithmic efficiency is a conservative baseline; in some subfields, the rate has been even faster. When RL scaling is coupled with these improvements, the net effect on per-capability cost can be strongly negative (i.e., reducing cost).
Model distillation and compressed reasoning
Model distillation is another critical mechanism for transienting the economic burden. Once a large RL-scaled model achieves a desired capability level, its knowledge can be distilled into smaller, faster models that retain much of the original performance. These distilled models require substantially less inference compute, often by orders of magnitude, while maintaining comparable output quality.
The distilled models are particularly valuable for deployment scenarios where latency, throughput, or operational cost is constrained. A 7-billion-parameter distilled model might match the performance of a 70-billion-parameter RL-trained original at a fraction of the inference cost. This cost reduction is not one-time — the distilled model continues to serve at low cost indefinitely, and further distillation cycles can produce even more efficient variants.
Moreover, distillation is not limited to model size reduction. Reasoning patterns, reward functions, and policy structures from large RL-trained systems can be distilled into more compact architectures that generalize effectively. The economic effect compounds: each generation of distilled models begets the next, each at lower inference cost.
Hardware advances and specialized inference
Hardware improvements, particularly the development of AI-optimized chips, have consistently reduced the cost of inference compute. GPU and TPU generations have delivered 2-3x performance improvements per year for AI workloads, while simultaneously reducing the energy per operation. Specialized inference accelerators — such as NVIDIA's TensorRT, Google's TPU inference chips, and an emerging ecosystem of domain-specific architectures — further lower the cost per token or per inference call.
When combined with software optimizations like quantization (reducing model precision from 16-bit to 8-bit or even 4-bit), the hardware cost reduction becomes even more pronounced. A quantized model running on specialized inference hardware can achieve 5-10x lower inference cost versus a full-precision model on general-purpose GPUs. This effect is ongoing: each new hardware generation continues the cost-reduction trend.
Importantly, hardware advances apply retroactively. Existing deployed models can often be migrated to new hardware with immediate cost savings, without requiring re-training or re-distillation. This creates a persistent downward pressure on the economic burden of any given model.
More efficient reasoning and sparse activation
RL-scaled models, particularly large language models, have inspired research into more efficient reasoning patterns. Sparse activation patterns, mixture-of-experts (MoE) architectures, and adaptive computation time mechanisms all aim to reduce the compute required per inference call by engaging only the most relevant model components for a given input.
Mixture-of-experts models, for instance, can activate a subset of parameters per token, dramatically reducing the effective compute per token while maintaining or improving overall model capability. If a model has 1.8 trillion parameters but only activates 2% per token, the inference cost approaches that of a much smaller dense model — while retaining the expressive power of the full architecture.
Efficient reasoning also encompasses techniques like early exiting, where the model stops computation as soon as sufficient confidence is reached, and token-level adaptive computation that allocates more or fewer compute steps per token based on complexity. These techniques directly reduce the inference cost per capability level, countering the intuitive expectation that RL scaling uniformly raises costs.
Data limitations and the uncertain 10,000x estimate
A critical caveat in the RL scaling economics discussion is the thin data foundation. Rigorous, large-scale measurements of RL's effect on inference costs are sparse. Much of the discourse relies on the much-cited 10,000x compute estimate, which traces the total compute growth from early deep learning systems to cutting-edge foundation models. While this estimate captures the headline-scale compute growth, it does not cleanly separate the RL component from other scaling drivers (e.g., pre-training scale, dataset size, architecture changes).
The 10,000x figure, therefore, should be treated as an upper-bound illustration rather than a precise metric for inference cost burden. When RL-specific compute is isolated, the increase is less dramatic, and when combined with the algorithmic and hardware dynamics described above, the net effect on per-capability cost can be strongly cost-reducing rather than cost-increasing.
The scarcity of detailed RL scaling data underscores the need for more empirical work measuring both training and inference compute as functions of RL technique application, model size, and performance gain. Without such data, arguments about persistent economic burden remain speculative.
Inference cost reductions of 5-10x per year
Perhaps the most striking quantitative finding is that inference costs have been falling at a rate of 5-10x per year, driven by the combined effects of algorithmic improvements, model distillation, hardware advances, and more efficient reasoning. This annual improvement rate, if sustained, means that the economic burden of any given capability level is reduced by an order of magnitude every 2-3 years.
To put this in perspective: a model that cost $100 to infer at deployment would cost $10 or less after just two years of compounding improvements, assuming the 5-10x annual reduction continues. After five years, the same capability level might cost less than $1 in inference costs. This trend directly undermines the notion of a persistent economic burden — the cost to achieve any fixed performance threshold is evaporating rapidly.
The 5-10x per year figure is conservative in some subdomains. Quantization combined with specialized inference hardware has delivered even steeper cost reductions in practice. The rate may also accelerate as the ecosystem matures and more optimization techniques are standardized and widely adopted.
Conclusion: the burden is more transient than it appears
The weight of evidence suggests that Toby Ord's persistent burden claim, while intuitively plausible, does not withstand scrutiny when the full dynamics of RL scaling are considered. The primary mechanisms — algorithmic improvements, model distillation, hardware advances, and more efficient reasoning — all systematically reduce the cost per capability level. When these effects are compounded, the net result is a rapidly declining economic burden, not a persistent one.
The inference cost reduction rate of 5-10x per year is the most compelling quantitative counterpoint. If this rate holds, the economic burden of RL-scaled AI becomes transient by definition: each year of progress makes the previous year's cost concerns obsolete. Of course, numerous factors — including deployment at massive scale, novel applications that demand new capability levels, and potential slowdowns in hardware or algorithmic progress — could moderate the pace. But the baseline trend is clearly toward transient, not persistent, economic burden.
The RL scaling data remains thin, and the 10,000x compute estimate is uncertain. More empirical measurement of RL-specific training and inference compute trends is needed to refine these conclusions. However, the structural forces described here — algorithmic progress, distillation, hardware improvement, and efficient reasoning — are well-established features of the machine learning lifecycle and strongly suggest that the economic burden of RL scaling is more transient than it appears.