Reiner Pope – The math behind how LLMs are trained and served

In this comprehensive technical deep dive on the Dwarkesh Podcast, Reiner Pope, CEO of chip startup Maddox and former TPU architect at Google, dissects the intricate interplay between hardware constraints, model architecture, and economic efficiency in large language model (LLM) training and serving. He begins by analyzing the critical balance between compute and memory bandwidth, demonstrating that optimal inference efficiency occurs at batch sizes around 2,000 tokens—where compute and memory times are matched—driven by model sparsity and hardware ratios. Pope highlights how physical limitations in data center racks, particularly the all-to-all communication demands of mixture-of-experts (MoE) layers, create hard ceilings on model size and context length unless leveraging scale-up domains like NVIDIA’s Blackwell racks with 72 GPUs. He further reveals that the 50% price jump at 200K tokens in models like Gemini 3.1 likely marks the inflection point where memory bandwidth (KV cache) becomes the dominant cost, not compute. The discussion then shifts to inference economics, showing that decode is memory-bandwidth-limited while prefill is compute-limited, explaining why output tokens are priced up to 5x higher than input tokens. Cache hits are 10x cheaper than misses, indicating strategic use of memory tiers—HBM for short-term retention and flash for longer durations—based on retention time and cost. The episode culminates in a fascinating exploration of the cross-pollination between cryptography and AI, where Pope explains how the Feistel network, originally designed for secure encryption, inspired reversible neural networks (RevNets), enabling memory-efficient training by rematerializing activations instead of storing them—a clever trade-off of compute for memory savings.