Introduction
Training compute has exploded over the past decade, with most agree it’s roughly quadrupling each year. This rapid scaling fuels the capabilities we see in today’s most powerful AI systems. The trend isn’t a fluke; it’s a consistent pattern across model families, companies, and time horizons. In this article we unpack the numbers behind that growth, showing how a 4‑5× yearly increase shapes everything from frontier research to everyday chatbots. All figures stem from a single 2024 Epoch AI analysis of training compute (https://epoch.ai/publications/training-compute-of-frontier-ai-models-grows-by-4-5x-per-year).
The 4‑5×/Year Growth Trend
From 2010 through May 2024, the amount of compute used to train notable AI models rose about 4.1× per year (90 % CI: 3.7× to 4.6×). That estimate comes from a log‑linear fit across hundreds of model releases, and it holds even when we update the dataset threefold since the 2022 baseline. The confidence interval is tight enough to feel solid, yet wide enough to acknowledge measurement uncertainty. If you look at the frontier subset—models that sit in the running top‑10 by compute—the growth rate climbs to 5.3× per year (90 % CI: 4.9× to 5.7×) over the same span. Recent years show a modest deceleration, settling around 4× per year after 2018, but the overall trajectory still averages 4‑5× annually. In short, the community has settled on 4‑5× per year as the baseline expectation for future compute growth, unless new constraints emerge.
Frontier Models
Frontier models, defined as those in the current top‑10 by compute, have driven much of the narrative around AI progress. Their compute grew at 5.3× per year from 2010 to May 2024, with a 90 % CI of 4.9× to 5.7×. After roughly 2018 the trend eases to about 4× per year, reflecting a slowdown that coincides with the maturation of large‑scale models and the emergence of more efficient training techniques. The slowdown isn’t uniform; it’s sensitive to whether we include outlier runs like AlphaGo Master or Zero, which can artificially inflate early growth numbers. Even with those outliers excluded, the recent frontier growth hovers near 4× per year, suggesting that while the pace may be moderating, the underlying capacity to train ever larger models remains strong.
Notable Models
When we broaden the lens to all notable models—those that meet citation, benchmark, or product‑deployment thresholds—the growth rate settles at 4.1× per year (90 % CI: 3.7× to 4.6×). This figure is remarkably stable across different definitions of “notable,” whether we count papers with over a thousand citations, state‑of‑the‑art benchmarks, or commercially deployed systems. The consistency of this rate underscores how compute has become the primary engine of progress, even as model architectures evolve. The 4‑5× yearly multiplier also aligns with the retrodiction of today’s largest models: GPT‑4 and Gemini Ultra appear to have been trained on on the order of 2e25 to 5e25 FLOP, which matches the expected size if we start from GPT‑3’s 3e23 FLOP and apply a 4‑5× per year multiplier over the decade.
Language Models
Language models show a notably faster ascent, especially in the early years after the Transformer’s 2017 debut. Notable language models grew at an astonishing 9.5× per year (90 % CI: 7.4× to 12.2×) between June 2017 and May 2024, reflecting their rapid climb from modest beginnings to frontier status. Once these models reached the broader AI frontier around mid‑2020, their growth decelerated to roughly 5× per year (80 % CI: 3.1× to 7.3×) through 2024. This kink in the curve mirrors how the field caught up to the compute frontier, then stabilized. The faster early growth likely stems from low baseline compute that quickly amplified as the Transformer architecture proved its worth, while the later slowdown reflects diminishing returns as models approach the frontier’s ceiling.
Company‑specific Scaling
The three leading AI labs—OpenAI, Google DeepMind, and Meta AI—each exhibit a 5× per year scaling pattern for their flagship models, though confidence intervals differ. OpenAI’s top models have risen roughly 5.3× per year since August 2017, with a wide CI that reflects the small sample size of high‑profile releases. DeepMind’s Gemini Ultra follows a similar 4.9× per year trajectory (90 % CI: 3.8× to 6.1×) since July 2012, indicating that their recent surge aligns with historical scaling rather than a breakthrough acceleration. Meta AI shows the steepest curve, with a 7.1× per year increase (80 % CI: 4.8× to 10.1×) for models dating back to July 2015. These company‑level trends reinforce the article’s central claim: compute growth is a shared, cross‑organizational phenomenon, not confined to any single vendor.
Predicting the Size of Modern Models
If we extrapolate the 4‑5× yearly growth from GPT‑3’s 3e23 FLOP (released May 2020) forward, we predict roughly 1e25 FLOP for GPT‑4 (March 2023) and 4e25 FLOP for Gemini Ultra (December 2023). Those predictions sit comfortably within the reported training compute of the actual models, confirming that the 4‑5× per year rate is an accurate baseline for forecasting today’s largest systems. The proximity of these estimates to measured FLOP counts suggests the community has a reliable gauge for future compute needs, provided no new hardware bottlenecks or fundamental algorithmic shifts arise.
Conclusion
Across model categories, companies, and time horizons, the data converge on a 4‑5× per year growth in training compute from 2010 to 2024. Frontier models peak at 5.3× annually before easing to about 4×, while notable models hold steady near 4.1×. Language models surged faster early on, then settled into the same 4‑5× range after 2020. Company‑specific analyses echo the overall pattern, with OpenAI, DeepMind, and Meta each scaling their flagship systems at around five‑fold yearly increments. This consistent upward trajectory underpins the rapid capabilities we observe in modern AI, and it suggests that, barring new constraints, compute will continue to expand at the same pace, fueling ever more powerful models.