Fixing GPU Starvation in Large-Scale Distributed Training

This episode of MLOps.community dives deep into the critical challenge of GPU starvation in large-scale distributed training, using real-world war stories from Kashish, a Staff Engineer at Uber leading the marketplace matching team. The core issue isn't model complexity or hardware limitations, but data I/O bottlenecks that starve GPUs of the data they need to operate efficiently. Kashish recounts a pivotal case where training utilization was stuck at 15–20% on A100 chips, despite a seemingly sound model. After eliminating variables by loading data entirely into RAM (boosting utilization to 85%), the team discovered the real culprit: the PyArrow-to-NumPy transformation overhead during data pipeline processing. By caching the transformed NumPy data instead of reprocessing it, they achieved 85% utilization and reduced training time from a day to just a few hours. The episode also explores broader themes like the trade-off between efficiency and reproducibility in parallel data loading, the importance of warm-starting models for inference, and the growing role of AI agents in debugging and workflow automation. The narrative unfolds like a detective story, emphasizing that the most impactful optimizations often lie in data engineering, not model architecture.