Inference economics used to be a black box. Everyone felt it: models get bigger, bills get bigger, and yet the tokens still trickle out at roughly the same pace. Ege Erdil’s 2025 work from Epoch AI finally opens the hood. The trade-off between speed and cost isn’t linear. It is shaped by square roots and cube roots, by memory bandwidth you cannot cheat, and by network latency that almost no one talks about when tallying operational overhead and the broader artificial intelligence environmental cost.
The paper models a Transformer forward pass as four distinct pieces that don’t just add up neatly. There’s arithmetic time for the GPU cores to do the multiply-adds, memory read-write time to shuttle weights through high-bandwidth memory (HBM), network send-receive time for collective operations, and a flat latency cost for kernel launches and collectives. NCCL’s LL128 protocol on a DGX H100, for example, carries a base latency of about 30 microseconds even if you’re reducing a one-byte tensor. You can overlap some of those pieces, but you can’t erase them.
Latency follows square root of size
The most quoted finding is also the most surprising. Token generation speed scales roughly with the inverse square root of parameter count and with the cube root of memory bandwidth. Flip that around and latency scales with the square root of model size and the cube root of memory bandwidth.
That means doubling parameters adds about 41% to per-token latency. It’s not two times slower, but it is noticeably heavier. Because scaling follows a square-root relationship, going from a 7B model to a 70B model implies roughly a square-root-of-ten increase in latency—about a 3.16× jump. Similarly, stepping from 70B to 700B scales by another square-root-of-ten factor, adding another 3.16× increase in time per token. The square-root curve is deliberately humbling: you pay heavily for the initial parameter jumps, then the marginal latency increases flatten out mathematically, though the absolute compute cost climbs steeply.
Memory bandwidth helps, but only modestly. Doubling bandwidth cuts latency by roughly 21% because of the cube-root scaling law. In practice, you can throw a faster HBM2e or HBM3 memory stack at the problem, but the throughput gains feel small relative to hardware upgrade costs. The toy model underlying these laws assumes infinite network bandwidth but nonzero network latency, mirroring real-world distributed serving clusters today.
When organizations ask how much does an llm cost, they usually look only at API token pricing tables or raw GPU rental rates. But the true ledger includes the physical energy footprint, cooling infrastructure, and the cascading artificial intelligence environmental cost tied to these memory-bound operations.
The Hidden Artificial Intelligence Environmental Cost of Latency and Scale
To understand how enterprise budgets and compute footprints align, it helps to look at foundational definitions. When cloud providers like Google Cloud describe What is Artificial Intelligence (AI)? | Google Cloud, they emphasize elastic compute clusters designed to handle massive parallel matrix multiplications. Yet in production inference, raw FLOP throughput is rarely the primary bottleneck.
Network latency sets a stubborn floor. Bandwidth limits only dominate when you’re executing GPU collectives over tensors larger than about 10 megabytes. With a model dimension of 10,000 and 16-bit activations, reaching that threshold requires per-GPU batch sizes of 500 or larger. In practice, nobody runs fast, interactive user-facing inference at batch 500 because you are optimizing for user latency, not raw batch throughput.
In fast inference regimes, communication time is dominated by latency rather than bandwidth. That fundamentally alters the economics. Without latency constraints, you could theoretically quadruple inference speed for each doubling of cost per token by sharding the forward pass across more GPUs. The reality that latency is binding explains why frontier LLMs aren’t served magnitudes faster right now. Each extra GPU adds another network hop, another collective, and another 30-microsecond tax. At scale, you are paying heavily for synchronization and coordination, not just raw compute.
Pipeline parallelism is a memory workaround
Data parallelism is the default architecture because it is straightforward to manage. Pipeline parallelism only earns its keep when you need to fit a model that exceeds a single GPU’s high-bandwidth memory. If you split a model into multiple stages, you must slice your inference batch into matching pieces to keep every stage busy and avoid idle bubbles. Most of the time, this makes pipeline parallelism a less efficient version of data parallelism because you introduce inter-stage communication overhead for free.
That inefficiency stems directly from a hardware memory constraint. Pipeline parallelism is only useful when memory limits bind—meaning when decoding speed is restricted by model weight reads. More precisely, if a model is served faster than the time it would take to read the entire content of a GPU’s HBM, pipeline parallelism becomes counterproductive.
For an NVIDIA H100, the math is straightforward: 80 GB of HBM divided by 3.3 TB/s of memory bandwidth yields roughly 24 milliseconds per token, or about 42 tokens per second. Serve faster than that threshold and you start thrashing the pipeline. Serve slower, and you may need the memory staging. It is a strict operational threshold rather than a design preference.
Speculative decoding amortizes the real bottlenecks
Speculative decoding is one of the few practical workarounds in modern inference engineering. It can nearly double inference speed for large models without raising token cost or degrading output quality. The trick isn’t algorithmic magic; it is pure amortization.
Arithmetic and network bandwidth costs exist, but in fast decoding loops they are rarely the binding constraints. What binds is memory bandwidth and network latency. If a smaller draft model generates several candidate tokens in a single forward pass and the larger target model verifies them together in parallel, you spread that fixed 30-microsecond latency and memory read overhead across multiple outputs. The per-token penalty of the memory wall drops significantly.
Empirically, this effect can double token generation speed at fixed cost. It doesn’t help with arithmetic-heavy prompt ingestion phases, but prompt ingestion isn't what frustrates users waiting for token streams. Speculative decoding targets the exact bottleneck holding back interactive experiences.
Making practical serving decisions
The scaling laws mapped by Erdil provide a sober reality check for engineering teams. Want a 2x speedup by shrinking model parameters? You will need to cut parameter count by roughly 4x. Want 2x speedup purely from hardware upgrades? You need an 8x increase in memory bandwidth.
Latency remains the stubborn governor of interactive AI. Pipeline parallelism is primarily a memory crutch, and speculative decoding is the most reliable tool for shifting the speed frontier without inflating token budgets.
Ultimately, inference economics is no longer just about FLOPs per dollar. It is about how many milliseconds systems spend waiting for memory buses to clear and GPUs to synchronize. Recognizing these physical limits helps engineering leaders build more realistic infrastructure plans and understand the true cost structure of modern AI workloads.
Source: Ege Erdil, Inference economics of language models, Epoch AI, 17 Jun 2025. https://epoch.ai/blog/inference-economics-of-language-models
