Why Video Agent models are next — Ethan He, xAI Grok Imagine
Ethan He, former lead of video and world models at XAI, reveals a radical shift in AI development: the future of generative media isn't in better video models, but in smarter language models. In a candid conversation with Latent Space, He argues that the most significant advances in video generation—like longer videos, better consistency, and interactive features—come not from improvements in diffusion models or video-specific architecture, but from the reasoning, planning, and prompt engineering capabilities of large language models. He describes how his team at XAI built Grok Imagine from scratch in just three months, relying on a tight-knit team, strong infrastructure, and a focus on iteration speed. But the real breakthrough, he says, was realizing that video models are 'dumb'—they follow prompts literally. The intelligence comes from a separate, larger language model that rewrites user prompts into rich, detailed descriptions. This insight leads to a bold prediction: video agents—AI systems that use language models to plan, iterate, and combine tools like FFMPEG and Photoshop—will dominate the next wave of creative production. He believes this shift will happen within the next year, with video agents producing ad-ready, production-quality content.
Video model improvements come primarily from language model intelligence, not video-specific architecture.
The most powerful AI systems will be video agents that use language models to plan, iterate, and combine tools like FFMPEG and Photoshop.
Video agents can generate production-quality content in minutes, not hours, by using iterative refinement and tool calling.
Context management is the core challenge in long-form video generation, and self-managing context is the next frontier.
The future of AI is not end-to-end video models, but language models that orchestrate a suite of tools to achieve complex creative goals.
…and 3 more takeaways available in PodZeus
Welcome to the Studio: Ethan He Joins Latent Space
The episode opens with the hosts welcoming Ethan He, a former lead of video and world models at XAI, to the Latent Space podcast. They acknowledge his background in computer vision, his work on the Cosmos world model at NVIDIA, and his previous presentations at the podcast's paper club. The hosts express appreciation for the community-driven nature of the podcast and its focus on deep technical discussions.
From Cosmos to XAI: Building Video Models from Scratch
Ethan recounts his journey from working on the Cosmos world model at NVIDIA to joining XAI in May 2025. He describes how, with no existing infrastructure, data, or model, he and a small team built Grok Imagine 0.9 in just three months. The key factors were strong talent, a shared goal, and XAI's robust data and model infrastructure, which enabled rapid iteration.
The Real Bottleneck: Iteration Speed and Infrastructure
Ethan emphasizes that the most critical factor in model development is iteration speed—the number of experiments you can run per day. He explains that strong infrastructure allows for rapid training cycles, which in turn enables faster bug detection and model improvement. He notes that the biggest gains often come from fixing small bugs in data or training pipelines, not from new algorithms.
The Hidden Power of Language Models in Video Generation
“The visual intelligents are actually mostly coming from language. These video models, especially from now, since the diffusion model technology is more mature, like every time you see there are some improvement on these models, I would say mostly this again comes from language model.”
The Rise of Video Agents: From Generation to Orchestration
“These agents should be able to understand this kind of long horizon task to be able to easily create a long form video.”
“The visual intelligents are actually mostly coming from language. These video models, especially from now, since the diffusion model technology is more mature, like every time you see there are some improvement on these models, I would say mostly this again comes from language model.”
“I feel right now the bottleneck for video models is actually the language part, the agent, which is why I want to go to work more on all lamps.”
“So these agents should be able to understand this kind of long horizon task to be able to easily create a long form video.”
Host
Guest
Ethan He
person
XAI
organization
Grok Imagine
product
Cosmos
product
NVIDIA
organization
Elon Musk
person
Latent Space
media
Flipbook
product
Neural OS
product
Stable Diffusion
product
SpaceX IPO Breakdown — Everything You Need To Know Before Buying This Stock
24m • 6/5/2026
Ian Bremmer on the Risks America Poses to the World
1h 31m • 6/2/2026
How Elon Musk Engineered the World’s Biggest I.P.O.
30m • 6/2/2026
AI goes IPO
25m • 6/2/2026
AI's job apocalypse & the transhumanist endgame
34m • 6/2/2026
Start discovering podcast insights today
Start with a 7-day trial and explore a growing catalog of popular podcasts. No credit card required.
No credit card required • 7-day trial • Cancel anytime