[BidClub_]
Latent Space · · 103 min

Inside xAI: Building Grok Imagine in 3 Months, Videogen vs World Models, and Video Agents— Ethan He

Ethan HeswyxVibhu

YouTube ↗
TL;DR

  • Ethan Caballero’s core claim is that future gains in video models will come primarily from language intelligence, not diffusion breakthroughs. Diffusion models remain literal generators: a prompt such as “a cat” may yield a stationary cat on a blank background, while a larger language model supplies motion, setting, composition, and intent. His conclusion after moving from Cosmos to Grok Imagine: “The bottleneck for video models is actually the language part, the agent.”

  • Video agents could push generative media across the commercial usability threshold by the end of the year. Instead of producing one clip, an agent can reason over a brief, generate variations, edit assets, write code, and invoke tools such as FFmpeg or deterministic text overlays. The guest expects “production grade quality” suitable for advertising; because agents run many expensive intermediate passes, that would unlock materially larger enterprise budgets for model and inference providers.

  • The capital barrier in video is comparable to medium-scale language-model training—and includes a neglected storage and I/O bill. A billion 5 MB videos imply 5 PB before storing similarly sized VAE features; Ethan put a serious corpus at tens of petabytes, while the discussion estimated that network costs alone could reach “a few million per month.” Dense models can reach roughly 19B parameters, explored MoEs about 20B active/100B total, and Cosmos disclosed tens of trillions of visual tokens.

  • xAI’s three-month path from “no infra, no data, and no model” to Grok Imagine 0.9 was an iteration-speed story more than an algorithmic breakthrough. Ethan credited a small, tightly aligned team, mature internal infrastructure, compute, and rapid end-to-end cycles from new data or algorithms through evaluation. The uncomfortable lesson was that “a lot of the improvements does not come from new algorithms”; small data and training-pipeline bugs often produced the biggest quality gains.

  • The guest’s definition of a world model is “real-time interactive long horizon videos,” making context management the central technical constraint. He contrasted demanding game interaction at 300 FPS—about 3 ms per frame—with a digital human that may tolerate about 200 ms; either way, minutes or hours of video overwhelm naïve attention. Five seconds can already represent 50K–60K tokens, so the path forward is selective retrieval—remembering characters, objects, and relevant scenes—not carrying every prior frame.

  • The generative-UI discussion made a larger interface bet: language models handle intent and tools behind the scenes while diffusion renders personalized pixels. The proposed shorthand was effectively “user intention to the pixels directly,” with interfaces that can be invented per user rather than authored once in code. At an illustrative $1 per H100-hour, eight hours daily would cost $240 monthly, but falling inference costs could eventually support what the hosts called a “diffusion front end, deterministic back end.”

  • The guest left xAI because the video thesis redirected him toward context-aware, potentially self-modifying language models. He expects models to know when they are approaching their context limit, decide what to retrieve or discard, and absorb today’s harness-level pruning, compaction, and time-awareness into learned behavior. His more speculative endpoint is a “model [that] can program itself online in test time,” with strong video understanding potentially making a robot simply another tool such an AI can operate.

Digest · the substance, structured for research

1. Grok Imagine compressed a year of learning into three months

  • Ethan Caballero built Cosmos at NVIDIA for roughly a year and concluded that video foundation models exhibited a scaling law similar to language models. That pushed him toward xAI—and, surprisingly, toward a company where he expected access to still more compute than at “the GPU-rich kingdom themselves.”

  • He joined in May 2025 as xAI was beginning video and multimodal work. “There were no infra, no data, and no model,” only a few engineers; three months later, the team released Grok Imagine 0.9.

  • The operating model was one daily sync followed by building. Ethan ranked talent first, then low communication overhead and xAI’s existing data, model, and support infrastructure: the research advantage was how many complete data-to-training-to-evaluation iterations the team could run.

  • His anti-glamorous lesson was that small pipeline defects often delivered larger gains than new algorithms. Better coding models now automate more implementation, but that returns compute to the bottleneck: ideas that once took weeks to prepare can reach training within hours, creating pressure to test everything while treating the work as “a marathon.”

2. Video generation begins with synthetic language, not raw footage

  • Internet videos do not naturally arrive with useful conditioning text. The guest’s example was mountain footage titled “I’m so happy today”: titles, descriptions, and comments may have almost no relationship to the visible content, so the text-video pair must be synthesized.

  • Before capable VLMs existed, humans supplied the bootstrap labels. The Cosmos labeling protocol asked annotators to describe every object, character, interaction, and dialogue so thoroughly that “a blind person” hearing the description could reconstruct the video mentally.

  • Generative training may still include a small fraction of unlabeled footage, with the model instructed to generate without text. The guest said this can improve generalization, but dense synthetic captioning remains the essential bridge between human intent and visual tokens.

  • An image model comes first because images are cheaper and provide denser language supervision. Training on a billion image-text pairs teaches far more linguistic coverage than perhaps 10 million videos; the image model can then bootstrap video generation rather than forcing an expensive video corpus to learn language from scratch.

3. Compression buys scale by taxing responsiveness

  • A 1,000-by-1,000 image contains roughly 1 million pixels, making direct Transformer training impractical. A tokenizer or VAE maps patches into a smaller continuous latent space, generation happens there, and a decoder projects the result back into pixels.

  • Alessio Fanelli emphasized that not all compact representations are equally learnable. MP4-derived tokens exploit redundancy but create a latent space that models find difficult to understand; trained VAEs provide a more usable representation, using convolutional networks, Transformers, or both.

  • Once visual and language tokens exist, diffusion-Transformer training is comparatively standard: add random noise, train the model to recover clean visual tokens, then iteratively denoise from 100% noise during inference.

  • Temporal compression creates the world-model trade-off. An illustrative 8×8×4 VAE compresses four frames into one temporal token and cuts context fourfold versus 8×8×1, but it also introduces intrinsic latency; frame-by-frame output is larger and costlier, yet can react immediately to user input.

4. Training costs rival language models while serving can collapse to a few steps

  • The guest declined to disclose xAI-specific training details, but the general arithmetic was blunt: 1 billion videos at 5 MB each require 5 PB. Stored continuous VAE features can be comparably large, pushing a full working set into tens of petabytes.

  • The hosts’ live AWS estimate put 5 PB of standard storage near $100,000 monthly and 5 PB of egress around $230,000. The guest expected network costs alone to reach “a few million per month,” with video training more I/O-bound than normal training.

  • The discussion cited roughly 19B parameters for RTX, with explored MoEs around 20B active and 100B total. Cosmos disclosed tens of trillions of visual tokens, making training comparable to a medium language model—and potentially less efficient because the infrastructure is different from that for language models.

  • Serving has a different escape hatch. Step distillation can teach a 10-step model from a 100-step flow-matching teacher; Cosmos used four- and eight-step variants, with some image-to-image tasks reaching one step. The guest linked this to consistency methods and GAN-style distribution matching: imitating a fixed teacher is simpler than reconstructing the internet’s full distribution.

5. Audio forces video models to understand time

  • Saining Xie described Grok Imagine 0.9 as, to his knowledge, the first audio-video generation model deployed at large scale. The hard problem was not merely adding a waveform but aligning text, imagery, speech, music, and events within the same timeline.

  • Speech has a language-like discrete component, though the hosts cautioned that speech researchers would resist reducing it to text tokens. Music is much more continuous, while existing LLMs struggle to describe beat, tone, and musical detail with enough precision that “someone deaf can reconstruct how the music sounds.”

  • Video descriptions can be loosely aligned with an entire clip; audio must correspond to what happens at each moment. The discussion therefore framed audio-video modeling as intrinsically time-aware—a property ordinary LLMs lack, even if the hosts argued that their human-like duration estimates partly reflect their training corpus rather than total temporal ignorance.

6. Generative UI makes pixels the personalized front end

  • Flipbook illustrated the thesis as an imagined web browser: every page and control is generated, nothing actually exists behind it, and each click produces the next visual explanation. The discussion presented this as where video world models could ultimately land as inference becomes cheap.

  • Jim Fan applied the same idea to Doom and Firefox in Neural OS. The hosts found an imagined copy of an existing OS less compelling, but his counterpoint was generalization: a model trained on ordinary screen recordings might eventually invent interfaces absent from its dataset, just as video models combine learned concepts into supernatural scenes.

  • The proposed architecture was “user intention to the pixels directly.” A powerful LLM or coding model could manage logic behind the scenes while diffusion renders a personal front end—perhaps email presented like TikTok—creating what the hosts summarized as a “diffusion front end, deterministic back end.”

  • At $1 per H100-hour, eight hours daily for 30 days would cost $240, which the discussion conceded consumers would not pay. The hosts disagreed about the scale of future gains: one cited 100–1,000× improvement in equivalent LLM capability per 12–18 months, while the guest noted that this combines model performance with compute and systems improvements. Accessibility remained an explicit unresolved objection.

7. A world model must be interactive, real-time, and long-horizon

  • Saining Xie declined to adjudicate every definition and offered his own: a world model is “real-time interactive long horizon videos.” The three requirements are interaction, real-time response, and long-horizon generation.

  • Interaction should include keyboard, mouse, and possibly voice, with the generated world responding reasonably. A demanding game case could require 300 FPS—about 3 ms per frame—whereas conversational interaction with a digital human could tolerate roughly 200 ms.

  • Even 200 ms is difficult when temporal compression batches frames before output. Eliminating that compression improves responsiveness but explodes sequence length, directly connecting the real-time requirement to long-context efficiency.

  • Long horizon means minutes or hours, not repeatedly generating isolated clips. The endpoint was a video version of Flipbook or Neural OS that remembers the world while producing and revising the interface continuously, though the guest stressed that “getting there is a very long way.”

8. Video extension is the first practical bridge to long-form worlds

  • Standard generators accept a prompt or initial image, produce one short video, and stop. Creators sometimes recycle the last frame as the next clip’s first frame, but quality degrades with every extension because the model receives almost none of the preceding context.

  • The guest said he remembered Veo 3 retaining about one second of prior video, which was better but still insufficient. Across a minute, image quality could decay, earlier events disappear, and speakers’ voices change because the conditioning window no longer contains their history.

  • Grok Imagine’s extension work instead retained historical context about prior generated video—who was speaking, which objects had appeared, and what had happened. The brute-force version is untenable: in Cosmos, five seconds represented roughly 50K–60K tokens and 50 seconds about 500K, making longer sequences reach millions quickly.

  • The hosts pressed whether capability should simply be solved inefficiently and optimized later, as with attention and distillation. The guest accepted the point but saw extension as a useful intermediate product: creators already loved it, and its adoption supplied a concrete step toward long-horizon world models.

9. Selective memory beats carrying every frame forever

  • Grok Imagine’s reference-to-video feature let users condition generation on as many as seven images representing characters, objects, or scenes. It preserved identity without requiring the complete source video to remain inside the active token window.

  • The guest treated seven references as an intermediate mechanism, not the final answer. An autoregressive movie generator should know when a character reappears, retrieve the earlier depiction, and ignore irrelevant intervening footage rather than keeping every frame equally available.

  • FramePack offered a hand-designed approximation: retain the most recent second at full resolution, then progressively shrink older history so total sequence length remains fixed. The guest expects learned selection to replace this heuristic by deciding which past segments matter for the next frame.

  • The host’s pushback—worth keeping—was that video experiments perform this selection inside the model, while language agents currently prune tool results and files in external harnesses. The discussion saw convergence: heuristic context removal, addition, and compaction may eventually be learned by the models themselves.

10. Provenance remains an adversarial contest

  • The guest said Grok Imagine applied watermarks where national rules required them, and that takedowns of videos were happening extremely fast. He did not present either measure as a permanent solution.

  • SynthID’s adoption by more platforms may broaden its use, but its public technical design also makes removal attacks conceivable. The hosts noted that users had reverse-engineered the applied pattern and could manipulate an image to defeat detection; the guest expected even improved watermarking to remain vulnerable.

  • Visual tells such as six fingers are fading. The guest now looks for whether a scene has “logical sense,” while the hosts watch audio synchronization, overly studio-clean sound, perfect lighting, and unrealistically clear skin—the “lack of imperfections” rather than obvious rendering defects.

11. Language models now supply most of visual intelligence

  • The guest’s “pretty big claim” was that visual intelligence is “mostly coming from language.” Diffusion technology has matured; the generator remains literal, while reasoning, interpretation, tool use, and detailed scene construction increasingly happen before pixels are produced.

  • Cosmos paired a 7B video model with a larger prompt rewriter, using Llama or Mixtral. The rewriter expanded short user requests into the exhaustive descriptions used during training; without it, “a cat” could mean a motionless animal on a white background because no setting or action was specified.

  • The guest’s favorite proof required no joint training: rewriting “a happy sheep” transformed a CGI-looking result into something markedly better. He argued that a three-minute image response may spend much of that time thinking, searching current news, organizing layout, and only then generating pixels.

  • Architectures vary: separate language and diffusion models, a language model with a diffusion head, or purely autoregressive discrete image tokens. The hosts pressed whether a generated image can be re-encoded for iterative visual reasoning and stressed that prompt rewriting is not the same as an autoregressive model with a diffusion head; the guest agreed and declined to characterize Grok’s proprietary design.

12. Video agents turn generation into a production workflow

  • The guest’s video-agent concept uses diffusion as one tool among many. A language model can plan a long sequence, generate and refine clips, use Photoshop-like editing, invoke FFmpeg, and assemble the output—closer to how artists actually work than expecting one model call to emit finished pixels.

  • Grok Imagine Agent Beta was an early attempt: users could request something like a one-minute video, beyond a single generator’s horizon, while the agent created frames and stitched them with code. The live request returned 36 seconds, not a full minute, preserving how early the capability remained.

  • One host initially found the thesis unsatisfying: if a frontier reasoning model plus a harness and Grok-as-a-tool captures most gains, what remains for video-model research? The guest’s rebuttal was that “AI models know AI models better”; joint training can improve model-specific prompting, while deterministic tools handle exact text, timestamps, edits, and compositing that diffusion misses.

  • The guest compared the progression to GitHub Copilot evolving into Codex and Claude Code: assisted creation first, increasingly autonomous production later. His conditional forecast was that by year-end agents could reach “production grade quality” distributable in ads; after that usability threshold, higher per-job inference and enterprise budgets could produce “exponential growth.”

13. The video bottleneck pulled the guest toward self-managing LLMs

  • Robotics remains part of the guest’s endpoint, but perhaps indirectly. A sufficiently capable video-aware model that predicts interactive computer worlds might learn to control physical embodiments naturally, making a robot “one of the tools” available to a general AI rather than requiring an entirely separate intelligence stack.

  • The guest left xAI because corporate objectives and priorities change quickly, limiting research he wanted to pursue on language models. This was not a rejection of video: “Most of the gain come from the language models themselves,” so moving toward LLMs followed his diagnosis of video’s current bottleneck.

  • His near-term prediction is context-aware models that know when they approach, say, 80% of their window and deliberately compact, retrieve, or discard information. Today, OpenClaw-like systems inject local time and prune tool output externally; he expects that heuristic engineering to be absorbed into model behavior.

  • The speculative extension is a model given access to its own agent harness, able to choose chunking strategies or modify how future instances operate—a “model [that] can program itself online in test time.” He framed the career risk through earlier pivots: top-PhD rejections led to FAIR, then self-supervision, Cosmos, Megatron MoEs reaching 40% MFU, and xAI; the shared large-model principles made switching fields easier than labels suggest.

Ethan Caballero

I have a pretty big claim: visual intelligence is actually mostly coming from language. These video models, especially now that diffusion-model technology is more mature, are improving mostly because of the language model, not because of the video model itself—the video diffusion models and so forth.

swyx

Before we get into today's episode, I just have this small message for listeners. Thank you. We would not be able to bring you the AI engineering science and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis, but fortunately enough of you actually subscribe to us to keep all this sustainable without ads and we want to keep it that way. But I just have one favor to ask all of you. The single most powerful completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you and it means absolutely everything to me and my team that works so hard to bring the AI in space to you each and every week. If you do it, I promise you we'll never stop working to make this show even better. Now let's get into it.

swyx

Okay, we’re here in the studio with Ethan, most recently of xAI. Welcome.

Yeah, thank you. Glad to be here.

swyx

We were just here with Vibhu. You were first coming to us, or joining the Latent Space world, because you were working on Cosmos at NVIDIA, and you did a great paper. We loved it. You presented it as well, so thank you for doing that.

Yep. I thought so. I also presented the MoEs—

swyx

Yes.

Ethan Caballero

Yes, at Latent Space.

swyx

How did you actually hear about us? Did we reach out to you? Is that how it worked?

No, actually, I realized there was this online community where people talk about AI and learn from each other through papers every week through the paper club. It’s very nice.

swyx

Yeah, I learned a lot. I think we’ve been doing it for 3 years nonstop. We haven’t stopped even on Christmas and New Year’s. There have been many weeks when I wanted to stop.

No, no, it’s good. I think you had posted that you worked on a paper, and I was like, “Oh, very cool. We have a paper club. Present it.”

Yeah, I might have reached out to you afterward.

swyx

Yeah, because it’s an amateur club, right?

Yeah.

swyx

It’s very unusual, but we sometimes have paper authors come by and actually explain the paper. Today, we just did the Poolside paper, which apparently is very good.

Ethan Caballero

It came out yesterday. Pretty interesting, right? Fully open. They talk about everything in the system. It’s a good one. We’ll recommend people read it.

swyx

Bring us up to speed on your transition to xAI, because I actually don’t even know when you joined. Tell us the story about that transition.

Before xAI, I was working on the Cosmos world model at NVIDIA. Cosmos is a giant video foundation model that aims to summarize the world and serves as a foundation for all roboticists to build on top of. Once I built Cosmos 1, I realized that video also has a scaling law similar to language models. We needed to scale up the video models further. That’s why I realized I needed to move somewhere with much more compute resources.

swyx

Than NVIDIA?

The GPU-rich kingdom themselves.

Yeah.

swyx

Timeline-wise, when was Cosmos? It was pretty early, right? Was it an open-world model and an open paper?

Ethan Caballero

It was toward the end of 2024.

swyx

Toward the end of 2024.

Yeah. Then, in May 2025, I moved to xAI. At that time, I joined when xAI was about to build video models and multimodal models. There was no infrastructure, no data, and no model. It was just a few engineers. We built it in 3 months and released the first model, Grok Imagine 0.9.

Since then, I’ve kept working on video models and moved more from pretraining to post-training of the video models. For example, reference-to-video, kind of like the Cameo feature, and video extensions. Before I left, I worked on a world model, leading a small team focused on real-time, long-horizon video generation.

swyx

Can you give us a rough roadmap? You’re on a brand-new team. Grok was previously only text, so they partnered with BFL for their image generation and stuff. What are the building blocks? You have compute, and data that you can procure somewhere. What’s the sequence of things that people should think about when they’re setting up a new team?

I’ll tell you, it goes even deeper—not just data that you can procure. You guys had to go through getting the data, too, right? You shipped it pretty fast, but, yeah—

Yeah, 3 months is actually very surprisingly fast. One thing I feel is that my experience at NVIDIA helped, because the first time we were building Cosmos together, we built it for about a year. This was the second time I did it, so I roughly had an idea of what to do.

I think the most important thing is the talent. Everyone was very strong and clever, and very closely aligned toward a common goal. That sped things up a lot. You reduce the communication bandwidth among people, and everyone can work toward the same goal.

Every day, there weren’t that many meetings on the calendar—maybe a sync a day—and after that, it was just all building. It was pretty fun at that time. Another thing is that xAI has very strong foundations in data infrastructure and model infrastructure, as well as supporting infrastructure that can help model development a lot.

Actually, the most important thing is how many iterations you can do per day. The more iterations you can do, the faster you can train the model. If you have very strong infrastructure and a lot of compute, you can train these models in a very short period of time. That gives you a much larger buffer for errors, and it also gives you the opportunity to spot more bugs.

swyx

Yeah. What is an iteration? Is it a few hundred steps, or what are we talking about?

Let’s say it’s the whole process of training the model: acquiring new data, maybe designing new algorithms, and training a new model, perhaps at a smaller scale.

swyx

Yeah, so a cycle time for any hyperparameter that you’re testing?

Yeah, the cycle time, and then evaluating the model: Is this model better than my previous iteration?

swyx

So it’s like someone had already set this up so that you could iterate very quickly.

Yeah, I think the foundation there is extremely good for developing and researching models. Often, I find that it’s kind of boring, but a lot of the improvements don’t come from new algorithms. They come from finding small bugs here and there in the data pipeline or in the model-training pipeline. Those give the biggest boost to model quality.

swyx

It’s interesting, right? You say it’s a small team with less communication bandwidth, but a lot of the quality comes from finding little bugs. It seems counterintuitive, right? You might think that having a lot of people would help you iron out more of those, but it’s interesting to see the other side.

Yeah, yeah.

swyx

I also wonder: Do you try using LLMs to look for bugs or anything?

I remember that, at that time—in May 2025—the coding model wasn’t quite there yet. I remember that by December 2025 it was extremely good. I’ve been using it since then, and it’s helpful.

Sometimes it produces code that’s difficult to maintain. The first time, it built something extremely fast, but it gave me spaghetti code with thousands of lines that I couldn’t maintain. The LLM itself couldn’t figure out what was wrong or how to improve on top of it. Now I find it much, much better.

I want to bring up another point here: Now that coding models are much more efficient and can help us implement things much faster, compute might become a bottleneck again. Previously, if you wanted to train a new model—for example, generate new synthetic data or write a new algorithm—it might take a few weeks. During that period of time, you might not have experiments to run.

Now you can build that thing within a few hours, and then you can immediately train a model. You have to have enough compute to try all of the ideas. Compute might be the bottleneck for iteration speed again.

swyx

Yeah. Honestly, I think that’s kind of a stressful job, because you’re thinking, “Well, I should be trying everything. If I’m not, then I’m not doing my job well.”

There’s also the stress of eating thousands of GPUs per hour, which is very expensive. Compute can go to other researchers. Daddy Elon.

swyx

Yeah, Daddy Elon.

But, you know, there’s still a finite amount of compute. You want to use it well, and you want more of it.

swyx

That was quite stressful indeed.

Yeah, I think one thing is that these coding models can now automate a lot of these jobs, which is much better. The second thing is that it’s a marathon, so you’ve got to maintain good health and a regular schedule.

swyx

It’s hard to hear that when you ship from zero to nothing in 3 months.

swyx

Yeah, and I think, obviously, the culture is very famously one where people work very hard. One thing I did want to dive into, in our notes that you sent ahead of time, is that you had specific comments about the cost of video generation training. Presumably, this is on Cluster 1, right? The 200-megawatt cluster? Whatever you want to share with us.

There are 3 things we're talking about. There's also video generation, and there's also the image-generation model that you put out. Do you want to complete the—Okay, so from 0 to 1, you have a few months. What are the stages of creating—

Oh, yeah, maybe I got distracted. Sorry. From there, there's video generation and audio generation. I'd love to get into those next, but what are those first few months like? So, a small team, a lot of bugs and iterations, but what does it look like? Do you take something off the shelf? Do you just get data and compute? What's the first few months like? How do you get to a state-of-the-art image-generation model? How do you just start?

Yeah. I cannot comment specifically on how I actually did it, but it's quite a standard process. I can draw some examples from Cosmos. Mainly, when you're building a video model, you actually need to build an image model first.

In building these 2 models, the data you need is 100% synthetic pairs of language and image, or language to video. Because on the internet, videos don't naturally associate with text. You can say, on YouTube, you have the title, the description, and the comments of a video, but they're not really relevant to the video itself.

Let's say maybe the video is a natural scene of mountains or something, and the title is, “I'm so happy today.” They have no correlation at all. The first step is that you have to generate synthetic pairs of language with the videos. You get videos from the internet and use a VLM to caption the videos.

So, that part raises the question: How do you get the VLM to begin with? If there's no—

Shawn Wang

The model, right? Like—

Alessio Fanelli

Say there's no VLM. How do you generate the text to begin with? It's impossible.

Shawn Wang

I see.

Alessio Fanelli

In the beginning, you ask humans to describe the video as detailed as possible. For example, you ask them to describe everything: all objects, all characters, and all interactions and dialogue in the videos. That seems to be the protocol for Cosmos labeling. The objective given to the labelers was that you have to describe the video as detailed as possible, such that a blind person hearing the blob of text can reconstruct what the video is like in their head.

Shawn Wang

Video or image? You're talking about an image.

Alessio Fanelli

Video or image, either one of them.

Shawn Wang

Okay. This was pretty common when we went from CLIP and DALL-E, right? It was all training on really detailed captioning of images. The same is applied to video, but instead of using a multimodal model to pass in video or images and write rich descriptions, you can also—

I think there's this traditional perspective of supervised, or very highly human-curated, data. I feel like there's an unlock with unsupervised, right? Where you have enough to bootstrap that you can just throw a common corpus on it or whatever—unsupervised vision-and-language pairing, where you just have interspersed image and text and it just learns. To me, that is the VLM breakthrough that is different from CLIP, different from the pre-LLM era.

Alessio Fanelli

Yeah, yeah. It's interesting to see that you kind of need both data.

Shawn Wang

Yes.

Alessio Fanelli

For example—

Shawn Wang

You needed to bootstrap it up.

Alessio Fanelli

Yeah. For generative model training, there's also usually a small percentage of unlabeled data. The model is instructed to generate a video without any text instruction. That can also help the model generalize.

After this stage of generating synthetic pairs, one important common step is to train a compressor or tokenizer of the images or videos. Because technically and theoretically, you can train image or video models on pure pixels, but the problem is that there are a lot of tokens. One image that's 1,000 by 1,000 is 1 million pixels, so it's impossible to train a Transformer on that. You need to train a tokenizer that can go from image to latent space and latent space back to image.

Shawn Wang

That's what we named the podcast.

Alessio Fanelli

Exactly.

Shawn Wang

But basically, you're talking about vocabulary size. And so, like, 1 million is impossible?

Alessio Fanelli

In generative models, the vocab is continuous. It's a continuous space. You can think about it like you map an image to a vector. It's a fixed-length vector of, like, 16 or 48, something like that. And then you map that vector back to the image space.

The mapping is patch-based. You say you have a 16-by-16 patch, and you map that patch of pixels into this latent space.

Shawn Wang

We've covered the Vision Transformer.

Alessio Fanelli

Yeah, VAEs.

Shawn Wang

You basically compress your input, you do your generation, your reasoning, all that generation in a smaller dimension, and then you project back out.

Alessio Fanelli

Yeah.

Shawn Wang

VAEs are a form of compression, but I think that, for me, the patching thing is from ViT, right?

Alessio Fanelli

Yeah, you can make—

Shawn Wang

Literally, the paper is titled something like “An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale.”

And then I think people make a lot of comparisons between this kind of patching and convolutions, which is kind of reconstructing the old paradigm with the new.

Alessio Fanelli

Yes. Yes.

Shawn Wang

Yeah.

Alessio Fanelli

Actually, in VAEs, there are both convolutional networks and Transformers. You can actually do both.

After this VAE, what you get is latent-space tokens and language tokens. So now the training of the diffusion Transformer—you've already trained these models using diffusion Transformers—is actually quite standard. It's very similar to how you train language Transformer models. There's not that much difference. It's just the tokens: visual tokens in, visual tokens out.

The only difference is that there's a denoising process. You train the model to unmask some of the noise. You add random noise to the visual tokens, and then you train the model to remove that noise and generate the clean tokens. In inference, the model can iteratively remove noise from 100% noise.

Shawn Wang

Yeah. And then there's also, to speed things along on the tech tree of diffusion, CFG, and then also, I guess, latent diffusion. I think somewhere along the line, obviously, Stability AI and all these other guys pioneered a lot of this architecture. But I don't know if you want to get into that, or just do the video side—up to you.

Alessio Fanelli

After you train such an image model, the reason it's a foundation for video models is that image models are cheaper to train and they have much denser connections between language and images. For example, you train on 1 billion images, and there's a mapping from the text to the image. The cost to train the same—1 billion text-to-video pairs—is much more expensive because videos naturally have more tokens than images.

Because diffusion models' understanding of language purely comes from this mapping, if you don't have enough mapping—say, if you only train on 10 million videos or something—you might not see enough language tokens in your training. So your model does not understand human intention enough. That's why you first train these image diffusion models and then bootstrap the video model from there.

Shawn Wang

One thing I did want to ask, because I think you're the first video-model person I've ever talked to, I think. We've talked to Luma and all those folks. There are all these tricks in video compression where, basically, frame by frame there's not that much difference, so you don't have to regenerate or resave the whole frame, right? I think MP4 compression or something else like that.

Is it tempting to use that, or, as far as I can tell, does everyone just treat it as, no, we will just generate every frame? Is that roughly the state of the art?

Alessio Fanelli

There are a few different approaches. First, you want to just directly use MP4 compression and use that as the tokens for the Transformer to train, right? People have tried that, but the main challenge is that the latent space for the MP4 tokens was not very comprehensible for the models. It's extremely hard to train on that.

So that's why they created VAEs, which create a more continuous latent space. The models can understand that latent space and learn from it much more easily. Even within VAEs, there are different difficulties of the latent space.

You can imagine something—the simplest, most naive VAE is that you have an image and you just shuffle all of the pixels into a vector. You don't need to train any VAE, right? But that latent space is extremely hard for models to train on.

So that's why there's some debate about how you compress the tokens. You mentioned that you can compress frame by frame, so you can compress the temporal dimension.

Shawn Wang

Yes.

Alessio Fanelli

The difference is, if you compress the temporal dimension, you get a much higher compression rate because there exists temporal redundancy between frames. This frame and the last frame are likely mostly similar, so there's only some small difference. For example, I think in Wan 2.1 VAE, they have an 8 × 8 × 4 compression rate. The 4 temporal tokens are compressed into 1 token. That can save a lot of context length.

If you do it frame by frame, you have to do maybe 8 × 8 × 1, and your context length will be 4 times larger. That being said, the benefit of frame-by-frame compression—we may come back to this later—is real-time-ness and interactivity, because if you stream the output of the model frame by frame, the model can respond to any user request immediately. So if you have a temporal 4, 4-times compression, then—

Shawn Wang

It might be laggy.

Alessio Fanelli

There's a lag there inherently.

Shawn Wang

So you're very keen on this. Let's just go ahead and bring it up, because we have the visual prepared anyway. There are some frontier applications of real-time video generation. Flipbook is one of the examples that went viral recently, right? What is Flipbook?

Alessio Fanelli

Flipbook is kind of like a web browser. You can see it has a web browser UI on top. The difference is that all of the UIs are generated by a generative image model in real time. Anything here is fake, but you can explore inside this imaginary world.

Say, here we have “Engineering the Great Pyramid.” The model generates it for us to understand how it works, and if we want to navigate around to understand it further, we can click on some of the descriptions here, and the model will generate a new page, a new subpage, describing the details we want to know about.

Shawn Wang

So it's basically a kind of—we're playing a video, but it's pausing for our next interaction, and then it just plays the next thing based on our interaction.

Alessio Fanelli

Which is kind of cool.

Shawn Wang

Yes. And you kind of decide your story. So this was, “How do you make a pyramid?” Leveraging techniques seem interesting, right?

Alessio Fanelli

The demo tweet had more animation between frames.

Shawn Wang

I—it’s just skipping.

Alessio Fanelli

It was skipping a lot of frames. They also have a video mode, but I guess a lot of people are using it, so it's not really—

Shawn Wang

There's a live video stream. We can try.

Alessio Fanelli

Yeah.

Shawn Wang

So this is an example of the kind of future that you see at the extreme. We obviously aren't in it today, but in a world where inference is completely free—

Alessio Fanelli

Yeah.

Shawn Wang

This is better than generating code and text.

Alessio Fanelli

Yeah, so this is the final stage of video generation, where it'll be a world model, I think. Imagine the internet doesn't exist, and then you'd type in Google.com. What should a model show you? If a model can imagine something, this is what the model imagines. This webpage completely does not exist.

Jim Fan

So I think as inference costs come down, we're going to have generative UI for everything. If you think about how coding models work, they write code for a webpage, and they render the code. It might be converted into binary, and the binary renders the pixels on the screen. In machine learning, every time we have some breakthrough, obviously it's more intuitive. So why don't we have user instruction to the pixels directly? Generative UI will be user intention to the pixels directly.

And say, even if I want email, let's say everyone has the same interface, but I want it slightly different. I want the email to show to me like TikTok, so I can swipe left and right for the emails. Or maybe you want something else; we can have completely different things. Or, like, I'm looking at Instagram Stories. I don't like the UI; I always have to click through, and our channel is a UI wizard. So it's going to be a revolutionary replacement of the interface.

In the future, we might have much more powerful LLMs and coding models running behind the scenes. In the front end, the diffusion model will actually be the front end to show stuff to you. That's how I imagine it.

Shawn Wang

Yeah, diffusion front end, deterministic back end.

Jim Fan

Yes.

Shawn Wang

I find that very expensive, but you know—

Jim Fan

I find it interesting that you called LLMs writing code on the back end deterministic, but okay. [laughter]

Shawn Wang

Yeah, you write it once and then you execute it.

Jim Fan

If you think about the cost, let's say each H100 costs $1 per hour. If you use this for 8 hours a day and 30 days, every month you're paying $240. You would actually not, and no one will pay for that. That's even more expensive than Claude Max.

Shawn Wang

Compute cost comes down, compute gets faster, models get smarter, models get smaller.

Jim Fan

Yeah, I don't know why you say 2 times, because I think it's like 100 times.

Shawn Wang

In language models, it is roughly 100 to 1,000 times every 12 to 18 months for the same given level of LLMs, you know—

Jim Fan

That's an additive of everything, right? That's model performance alongside compute. So it's different from just compute costs coming down, but, you know, a very interesting future.

Shawn Wang

Yeah. The web designers will have to shout out that accessibility is an issue, right? How do you deal with screen readers or whatever? But yes, this is higher-bandwidth storytelling than anything you can possibly generate with code. So I think that's the rough idea.

Jim Fan

I'd like to add a little bit to that. Humans naturally have the maximum bandwidth when we're looking at things—look at videos. And we also have maximum output bandwidth when we're talking. So in the future, it might be something like we talk to AI models, and the AI model responds back with a generative UI. That would be the maximum input and output bandwidth to interact with AI models before Neuralink happens.

Shawn Wang

And I mean, it's also very custom, right? Some people are very visual, some people are not as visual, right? They prefer the text, but that's the thing about generative UI, right? It can also be text.

Jim Fan

Yes.

Shawn Wang

There's another project that we wanted to highlight, which is Neural OS. Kind of a similar idea, but here you're literally operating a simulated operating system with a video model.

Jim Fan

Yes.

Shawn Wang

And you can play Doom, you can do Firefox. I find this mildly less impressive, obviously, because it's an OS that I can run. [laughter] But here, everything is imagined.

Jim Fan

I was used to the Command-W shortcut to close the Firefox tab. [laughter] It didn't crash.

Shawn Wang

It's too immersive.

Jim Fan

It's too immersive for me. I wanted to close the tab. But yes, I can play generative Doom.

Shawn Wang

It is shockingly fast.

Jim Fan

Yeah. Because I remember there was a demo about maybe 1 or 2 years ago where someone tried to do a first-person shooter with an image model. There was no consistency; it was very slow. But here, realistically, this is Doom.

Shawn Wang

I mean, I think there's 2 sides to that, right? There's, okay, what is running a game? The heavy part of it is actually the game engine, all the lighting, all that stuff, the graphics. This is just kind of video, right? We've solved consistency. This is still—it looks like a few-years-old image generation. There's some temporal consistency, but it's kind of just images stitched together as a video frame by frame. But it's a good visual representation to picture the future you want to see, right? That's what I see in these, more so.

Jim Fan

This reminds me of how the video models get better and better. Neural OS is kind of—if you just look at it, it feels like it's just a crappy version of the Windows we could have, right? But the difference is that this model is overfit on the existing operating systems. It can't generate anything different from that.

But it's actually also similar to video models. When we're training these video and image models, we train them on the internet. There's no imaginary supernatural stuff on the internet, but once we train this model, you can prompt the model to generate something supernatural that has never existed in the dataset. So, if you train your Neural OS for a neural computer using the standard screen recordings from the entire internet, the model can imagine a completely new interface to interact with the computer.

Shawn Wang

Yeah. This is one of the things that is magical to me. Usually, generalizing out of distribution is bad. But somehow we have learned some kind of internal world model. You say, you know, “This, but it looks like rainbows and butterflies,” and it'll do it, and it'll kind of make sense.

Jim Fan

Yeah. [laughter]

Shawn Wang

So, yeah, that's kind of cool. I don't know if there's any more comment on that. I did want to touch a little bit more on the model architecture stuff, which I think you were getting at. It's really fascinating. We don't get a chance to talk about this enough.

One of the papers that we covered—we've covered every annual Segment Anything release.

Alessio Fanelli

I don't know if you follow many computer vision guys, but they did memory attention, which is kind of interesting. I always think that anything where you can keep some consistency across the temporal dimension is fascinating. I don't know if the CV side is bleeding into the video generation side, but I think it's underexplored, right? We talk about it for labeling, but you can actually borrow the architecture itself.

There are also completely different approaches, right? You brought up the term world model, so we went from video model to world model. There is diffusion, but there are also other approaches that people are exploring, so maybe we can get into those afterward as well.

Shawn Wang

Yeah. You have a whole definition of world models and stuff. I feel like we threw a lot at you. What do you want to comment on?

Jim Fan

I think one thing we should actually comment on is that we were talking about the steps to train an image-generation model into a video model. One thing we don't see as much is the delta in training data, right? A video model might not generalize as much, but what is the cost of training a large video model?

We know for LLMs, roughly—even the post that came out today, right? It's a Gemma-level model trained on roughly 40 trillion tokens on this many H200s over this much time. You can see the exact cost of that: how many GPU hours over how much H200 cost. How do we do the back-end math for video models and image models? How do you break that down?

Ethan

Okay, sure. So, I'm back to see I'm a little calculation. So, surprisingly, video models are very comparable in cost to language models. Obviously, the largest scale is language models, maybe a medium scale to language models. I'd say just storing the videos alone costs a lot. You can maybe look up on AWS or something. If you have a billion videos, and let's say each video is 5 MB, then you will need 5 PB to just store those videos. Also remember we talked about using a VAE to compress the videos, and you also need to store those continuous features in storage. That's also comparable in size to the videos themselves. So just storing these videos and the features is tens of petabytes alone.

swyx

Oh yeah, I just looked up the calculation. 5 petabytes on S3 Standard is $100K per month.

Jim Fan

Okay.

Shawn Wang

You said that's affordable?

Jim Fan

At tens of petabytes, it's $200K. Even more expensive is the ingress and egress.

Shawn Wang

Oh yeah, I get it.

Jim Fan

It's through the internet. You have to download those videos. I believe it's more expensive on AWS than just storing those videos. Each training run, you probably need to pull them once, and if you train multiple times, it's even more than that. So the network costs alone would be a few million per month, just for storing everything.

Shawn Wang

Okay, my side tangent: GPU rental, or compute rental, is very efficient. On one side, you can be xAI and build your own data center. Shouldn't we just build our own storage and compute as well, especially with egress and stuff?

Jim Fan

That's a good idea, but it also comes with some of its own challenges. People who build GPU data centers might not expect this much storage. And people who build storage tend to build it somewhere separate. It's not just GPUs.

Shawn Wang

I just looked it up: it only charges for egress, not ingress. Tier 5 for 5 petabytes is $230K.

Jim Fan

Yeah, even more expensive than that.

Shawn Wang

But storage is per month, right? You check in and you cannot check out.

Jim Fan

That's good. Cool.

Shawn Wang

It's okay, so that's fine.

Jim Fan

Yeah, you know, my data is larger than you think.

Shawn Wang

Yeah, my back-of-the-napkin math of GPU hours times GPU cost is also very much missing some storage.

Jim Fan

You're also more I/O-bound than normal training.

Shawn Wang

Yes.

Jim Fan

Because data loading and caching everything becomes super important. In Cosmos, we did a lot of optimizations to make it not I/O-bound, though.

Speaking of training, if you look at open-source models and how big these video models are, I think RTX has 19B parameters. That's a dense model. People are also exploring MoEs, so it might be 20B active and 100B total. That's similar in size to medium-sized language models.

If you look at the number of tokens, we disclosed that in Cosmos: it's also tens of trillions of visual tokens. Putting these together, the cost of training these video models is actually comparable to LMs. Not to mention that the infrastructure is slightly different from LMs, so it might be less efficient to train these models.

Alessio Fanelli

Do you get the benefits of traditional diffusion speedups? For images, there are LCM LoRAs for fine-tuning, there's flow matching, and there's a lot of other work that's been done. Is there overlap that applies to diffusion on the inference side and stuff?

Saining Xie

The inference side is a completely different story. For the training side, it might be a little bit hard to reduce the cost. For inference, the biggest gain comes from distillation of these models. It's called step distillation, which is slightly different from knowledge distillation in LLMs.

Typically, for flow-matching models, you need around 100 steps or something. Diffusion models might need even more—around 1,000 steps—to generate a good image or video. Step distillation tries to learn to generate in fewer steps from the model itself. You use the full model to generate in 100 steps, and then you take a model that only generates in 10 steps and let that model learn from the perfect one.

Alessio Fanelli

Why does this work?

Saining Xie

It's distillation, kind of like strong-to-weak supervision. From the modeling perspective, the strong model, the teacher model, is trying to model the images and videos of the entire internet, and that distribution is extremely complex. A step-distilled model is just trying to learn from the teacher. The teacher is a model whose size is fixed, so its distribution is much simpler than the whole internet. That's the intuition I have for why distillation can work.

Earlier, when these models were served in production, they only ran in a few steps. In Cosmos, I believe we have 4-step and 8-step models. If you do a simpler task, like image-to-image translation, it can even run in 1 step in Cosmos Transfer.

Alessio Fanelli

I think this is the same intuition that guides a lot of the consistency-model work. I sent you a link about consistency models. I don't know if you covered that. To me, that was one of the most impressive papers I've ever seen from OpenAI: this unifying grand concept of consistency models. I don't know if you have any comments on this.

Saining Xie

There are a few different approaches, like consistency models.

Alessio Fanelli

Oh yeah, here it is: 2 steps versus 20 or 100 steps, whatever. It's already done.

Saining Xie

There are a few different approaches. For example, consistency models. We also shouldn't forget GANs. GANs were the OG step distillation because they trained in just 1 step to begin with.

A lot of approaches—for example, distribution-matching distillation—use GANs as one of the losses for distillation. GAN just tells you, “Hey, generate the image,” and then it has a discriminator to tell you, “Is this image real or not?”

The model just needs to learn one distribution, not the full distribution. In training, the model is asked to reconstruct the ground-truth image from the internet, which is extremely hard. When you're training a GAN, it's a 1-step process: you generate an image and ask whether it looks as real as the image from the internet, which is a much simpler task. Combining a lot of these approaches—consistency models, distribution matching, and GANs—people can get these few-step models.

Shawn Wang

Okay, then there's one thing I wanted to add, which is audio.

Saining Xie

Yes.

Shawn Wang

And video.

Saining Xie

Yeah, Grok Imagine 0.9, I believe, is the first audio-video transformer model deployed at large scale.

Shawn Wang

And that was your first model?

Saining Xie

Yes, that was Grok Imagine's first model. It's an audio-video generation model. I think the hard part is the modality alignment.

Before this transformer model, we had text-to-video alignment. We had corresponding text and video. Typically, most VLMs understand images, and videos are very rare. They mostly don't understand audio. If you look at audio generation on the LLM side, you can talk to them perfectly fine, but if you ask them to sing a song or something, they typically aren't very good.

Also, they don't have music either. The hard part is that audio actually has 2 components: a discrete component and a continuous component. The discrete component is language. When we speak, it's just a text token with some characteristics, I would say.

Shawn Wang

Is it an ASR issue?

Saining Xie

Yeah, it's a text token with some characteristics, I would say.

Shawn Wang

I think the speech guys would disagree. It's like, this one says “and then,” you know?

Saining Xie

Yeah, I'd say speech is largely discrete, but music is completely different. It's very continuous, and you cannot model it like discrete tokens in language models. This is the hard part for models, not to mention that we have to align text, video, and audio together.

Shawn Wang

How?

Saining Xie

One significant challenge is that most VLMs cannot understand audio. You need some way to generate synthetic data for audio. You have to caption the audio, and that involves a lot of synthetic-data and human-data effort. Not surprisingly, most LLMs are very bad at recognizing the beat, tone, and details of music. They can make a general prediction of which song it is, but it's very hard to describe the details of the music.

Like we mentioned with image generation, you have to describe an image as detailed as possible so that someone who is blind can reconstruct it. So here, it's like someone—

Shawn Wang

Deaf.

Saining Xie

Someone who is deaf can reconstruct how the music sounds without actually listening to it. Maybe you can think of it as needing what they call the subtitles.

Shawn Wang

Subtitles, yeah.

Saining Xie

You have to have all the details of the music and the dialogue.

Shawn Wang

Is the challenge there typically stuff like music and audio, or is there a baseline where there's enough data that we can understand narration and conversation, but there are nuances in audio where you hit all the data issues? Or is it just from stage zero—you just do it all, right?

Saining Xie

One important thing is the alignment. The model has to know that the video and audio have a time-based alignment: at which time step do the video and audio tokens correspond to each other? We actually don't have this kind of alignment for most other modalities. If you think about text and image, or text and video, they're loosely aligned. You can have a description of what's going on in the video, but you don't have to have an exact description at, say, 1 second—what happened?

Shawn Wang

Very coarse.

Saining Xie

Yeah, yeah.

Shawn Wang

So, what's the ideal time step? You have to deal with it, and then it's like 4 seconds or something?

Saining Xie

That comes down to how you design the model, for the model to be aware of time—the time modality. So the model is time-aware. That's something pretty unique if you think about LLMs. If you ask an LLM to complete a task, it will say, “This task will probably take 12 hours to complete,” and it comes back in 1 hour and says, “I've already spent 2 days on this, and I've exhausted everything.”

Yeah, LLMs themselves don't have a sense of time.

Shawn Wang

I actually don't think that's just them not having a sense of time. I think it's somewhat grounded, right? Like, you tell someone, “Okay, go work on this feature. Go implement this.” There's a general understanding you would have of how long that would take without LLMs working at LLM speed, right?

So you think back—2 years ago, if I told you to build a new front end for Latent Space, have a search bar, have all this, you'd estimate that it would take a few days, right? So you tell an LLM, “Go build this.” “It'll take me a few days.” But I think it's somewhat grounded, as opposed to them not having the best understanding. I'm not saying that they have a great understanding, but I think that example is—you can see where it comes from, right? You're trained on all the text.

Saining Xie

They're trained to say what a human would say.

Shawn Wang

Yes, because that's what the data kind of represents.

Saining Xie

Not just the cars for sale on the internet. People have an estimate.

Shawn Wang

Yeah, and not even just indirect training examples, right? Just your world understanding of tokens, of how long stuff takes, right? Go read a book. It'll take you a while, right? Even if you do nothing but read a book, it takes a few days. So, yeah, an LLM might say, “I read it. It took me a few hours. It'll take me a few hours to go through this research.” But this is a tangent.

This is a train of thought I haven't really expressed until now: a full world model must also be recursive, meaning that the participant in the world model must also be aware that they have a world model. It's this whole recursive thing down the line. But yes, the world model can be wrong, and then it needs to update it, and so on. We've argued in the newsletter as well that there need to be recursive adversarial world models.

Okay, just to ask, how do you define a world model?

Saining Xie

Let's go there.

Shawn Wang

Just for context, we talked about video generation, and then there's a distinction between world models and video generation. What's your definition? How do you see the 2?

Saining Xie

As a disclaimer, I'm not going to debate what a world model is. There are many definitions, so I'll just talk about my definition. Since I came from the multimodal domain, I'll mainly talk about video.

A world model is real-time, interactive, long-horizon video. There are 3 parts, so let's talk about them 1 by 1.

First, interaction. We look at Facebook and a neural computer. The interaction part means that a world model can allow you to interact with it through a keyboard, mouse, and maybe voice. Through all these modalities, you can interact with the model, and the model should respond reasonably.

The second part is real time. Once you move your mouse, say the world model generates a game, how fast can that game respond? If you're a professional CS:GO player, you might say, “You have to respond in under 10 milliseconds or even less.”

Shawn Wang

No, 60 FPS, let's go.

Saining Xie

300 FPS.

Shawn Wang

No, 500 FPS. Wait—okay, yeah, I didn't do the math right.

Saining Xie

Yeah, 300 FPS—that's 3 milliseconds. So you have to respond in 3 milliseconds. Most video models cannot do that.

Shawn Wang

Yeah.

Saining Xie

But if you have a video model that is, say, a digital human, the response time may be more generous. Typically, for real-time interaction, it might be 200 milliseconds. That's much more generous. But even 200 milliseconds is pretty tricky, because remember we mentioned the temporal compression coming from the VAE. If you don't compress the temporal dimension, your sequence length is going to explode. So if you want real-time performance in your model, you have to deal with this long-context problem.

The third part is long horizon. You're not going to just play these video games for a few seconds—most video models only generate a few seconds. You want to play for minutes or hours. The model has to be able to generate long-form content.

Putting these 3 together, it's real-time, long-horizon, interactive video. I think the final state will be, for example, a video version of PlayBook, where you can interact with a neural computer: you move your mouse, click on the generative interface, and it replies to you through pixels, generally in real time. But getting there is a very long way.

One of the first steps at Google Imagen, where I led a small world model team, was to build video extension.

swyx

Ah, it's the first step of interactivity.

Guest

It's the first step.

swyx

It's like video editing, yeah.

Yeah. The first step is that this unlocks long-horizon videos. Typically, for most video-generation models, you give it a prompt or an image as the initial frame, and it generates a video. That's it—1 time, done. Some creators try to use the last frame as the first frame for the second video. It sometimes works, but if you do it a few times, the quality degrades.

swyx

It doesn't have that context over the full video, so the temporal context is limited.

Yeah, because you only give it the last frame, of course, right?

swyx

Exactly.

Guest

But it's actually a pretty fun hack.

swyx

He has something better.

Yeah, yeah, yeah.

For example, I remember Veo 3 has a 1-second context of the last video. It's slightly better than using the last frame, but it has the same problem—the quality will degrade. If you extend it a few times to 1 minute, the video quality will look much worse than the first video.

Second, another problem is that the model doesn't have long-range knowledge of what's happening before. So, if they generate some dialogue—two people speaking—their voices might change over time, especially if the 1-second conditioning does not cover the previous context. These are the core challenges.

The Grok Imagine video extension has historical context of all of the previous generated videos.

swyx

Mhm.

Guest

It has a context of who's speaking, what objects have appeared, and everything else needed to generate the next video. If we naively do this, you can imagine just putting all of the previous history video tokens into the context. The context length will easily explode. For video models, it could easily be a few million tokens, I would imagine.

swyx

What's the number there?

For example, in Cosmos, I think just 5 seconds of video is around 50,000 or 60,000 tokens. If you do 50 seconds, that's 500,000 tokens. If you go longer than that, it can easily explode.

This long-horizon problem was the first step they were trying to solve with the model. It turns out people love video extension. A lot of creators love using video extension to create longer-form videos. This is the part I like: you have an intermediate step towards the final goal instead of just a straight shot to the final version.

swyx

Yeah, but I can see you have a strong vision of where we want to end up.

Yeah.

swyx

Does it seem like it's an efficiency issue? We're at a few million tokens of context. If you draw the parallel to language models, we started with very short contexts—2,000, 8,000—and then scaled them up to 1 million or 10 million. There's an effective context, but at the end of the day, what's it worth? There's also the training-data side. In video, it might be slightly easier because we have 100 million-token videos. You can just take a movie with the full context.

Is this an efficiency issue from an inference standpoint—it's expensive, but we know how to solve it—or why isn't this the approach? My broader point is about your second point on world models. You say they need to be interactive and live, right? You should be able to play a game and see the interaction live.

One thing I see with research is that a lot of what you actually serve is different from what you build, right? We talked about distillation. You train a big model, distill it, do quantization and speculative decoding. We do all this stuff to serve it efficiently. Shouldn't we just have a solution—a world model that can interact well—do the inference optimization, serve it, and distill it secondarily? Make it real-time after you solve it.

Another parallel is continual learning. What we need is someone to solve it and show that it works inefficiently. Give it a few years, and people will make it efficient. The same thing happened with regular attention. It worked, and over a few years people developed different forms of attention and scaled it to be efficient at long context.

So, there are 2 things there. One is that it seems like it works, and you've scaled it. Can we not just scale it much more efficiently over time? Do we need a separate approach if this stuff works? The same thing applies to interaction. If we can solve it somehow and make it work, we can solve the inference-efficiency problem later.

Guest

Yeah, that's actually a very good point. In videos, there's a lot of redundancy. We solve a lot of the pixel redundancy with a VAE, but there's more redundancy in long-range and long-horizon videos. Say a character appears in the first clip and then disappears. If it doesn't reappear until the end of the video, you probably don't need the context from the middle of the generation. You only need that character where you need it.

That's why I built another feature: Reference to Video.

swyx

Is it here? Is it the same model release or a different one?

It's a different one. You probably need to search in the app for “Reference to Video.”

swyx

Okay.

Guest

Reference to Video allows you to upload up to 7 images as conditioning and generate a video. They can be characters, objects, or even scenes. For example, I want to condition on Sean's selfie and him holding a blade—

swyx

Yeah, we have a dog. We put the dog in the thing.

Yeah, you can put them there, and the video model will generate the video from that and copy the context over.

That can solve a lot of the problems there. It probably doesn't need to have a very long context, but I feel like it's an intermediate solution. The model—

swyx

Yeah.

Guest

Yeah, the model should be able to selectively know where it should draw the references. Say if I want to generate a movie, I generate it autoregressively, 10 seconds at a time or something. Now this character appears, I can look back to where it first appeared and bring that back. Yeah, this one. I put in the references.

swyx

Yeah, that's Optimus Ice-T.

Oddly enough, I used Grok Search to find it, and it pulled up your LinkedIn post. We found it interesting.

But this is a problem. I know this isn't your fault, but xAI doesn't communicate all the work that you do very well. They just have the model release, and then that's it. These details are actually very, very good. As far as I understand, everything you just described is state of the art. No one else has done it.

Thanks.

swyx

And then you just put this blog post and the code there. I'm like, this is not enough, you know? Obviously, these are the high-level numbers that people want to know.

Guest

I wonder if part of that is also that some labs don't share research into what happens.

swyx

No, but this is literally bragging about how good they are. Why would you not say that you're capable of extending with full context? This isn't secret sauce. This is just, “We did the work.” I don't know.

Yeah, I guess different labs have slightly different communication styles.

swyx

Anyway, if anyone from xAI is listening, we're always happy to help you tell your story.

Okay, so you did references, and I think the point you're making here is that it's sort of a kludge, right? You can do 7, but what about 100? Then you need a completely different thing.

I think this is a mechanism to select the context from the history. You might not put the entire history into the context. For example, there's a paper called FramePack, which has a heuristic: for the latest history, like the last 1 second, I put in the entire history, and for the history before that, I compress it and make the video smaller.

It follows this beautiful pattern where the maximum sequence length is fixed. The farther you are from the current frame, the smaller the image. This is just a heuristic. I think it can be more automatic. The model can be aware of which part of the history can be selected.

This part of the research is actively being worked on by a lot of people. It's also kind of interesting. I feel like this part of long context is a little bit ahead of the LLM part. For example, in LLMs, if you call a tool and the tool-call history is extremely long, that's still in context. It keeps growing, even if you switch the topic to something else. The whole context will still be there.

There are some agentic harnesses that help you prune the tool results. When you query a file, they might only show the top 200 lines or something. Those are very heuristic-driven.

swyx

For listeners, we did a write-up on the Claude Code leak, where there are 8 different kinds of pruning, including pruning the tool results and all that. You can read up on that kind of thing.

I think one breakthrough in continual learning might be a way to automatically manage its own context.

swyx

All the heuristics will be replaced by machine learning.

Yes. Interestingly, the same thing is being researched in both LLMs and video models.

swyx

The interesting thing is that, in the paper you showed, it's actually happening at the model level, right? Compared to language models, where we have base attention, but we do our own compression and our own pruning, which is separate from the model layer.

Eventually, it all just boils in, hopefully.

swyx

Yeah. This is a form of attention, but also some sort of reasoning attention. I feel like that's different from normal attention. Does that make sense?

Yeah, it's different in the sense that, with attention—not to mention self-attention—normal attention can attend to all of the tokens.

swyx

Like, you can—yeah.

Yes. You don't have a high-level mechanism to drop the tokens you don't want to attend to. As humans, human attention span is surprisingly small.

swyx

Yes.

Guest

You can only remember an 11-digit phone number.

swyx

But I have feature detection, right? I can detect, “Oh, that’s a sequence of 1, 2, 3, 4 in an 11-digit phone number.”

Yeah. Very good pattern matchers.

swyx

But human attention can work because we can dynamically pull in concepts from different places.

Yeah, for LLMs, it’s recent. It’s a recent line of work.

swyx

Yeah.

Guest

Which is not that crazy, but it’s just recursive.

swyx

I think it’s somewhat inherent in models, too, right? It’s a nice example here. You pull these up, and you can read them fine, but language models are also very good at slot parsing.

Alessio Fanelli

I throw my typos in there; it doesn’t matter.

Guest

Yeah, yeah. You have a transcript, you have whatever—just throw it in, and it’s very good at parsing through noise. That may be brute force. It can look over it and reason over it, but there are parallels to both.

swyx

I think it’s just really fascinating how you relate the world-model stuff to video generation, which I don’t think a lot of people hear directly from people like you. I think that’s really helpful. Any other work? Did we cover video, audio, world models—any other stuff in that Omni team, I guess?

Alessio Fanelli

Is there any other work at xAI you want to talk about?

swyx

It seems like everything we see publicly announced is, “Oh, cool, cool stuff,” and then there’s so much more to everything.

Guest

There’s a lot of that.

swyx

Any underrated stuff, you know, just at the time there?

Yeah, I feel the culture is quite interesting and a bit underrated. The culture is three sentences: “Move fast, build”; “no goal is too ambitious”; and “first principles.” You already set a goal that was very ambitious. It wasn’t possible to achieve when I was first thinking about it. For example, I could build something in 3 months.

swyx

Was that like, okay, we’re starting a team, we want image, we want video, do it by this deadline? Or how do you work backward? Was it just, okay, we have a rough date, and by this date we want something out? Or is this like—

That’s a very good point. It’s first-principles thinking. Some people might say first-principles thinking applies more to the physical world than to models. I would say, for example, if you think about some limitation, like acquiring data, how fast can we acquire the videos? If you think about training the models, what’s the iteration speed for training a model end-to-end? How would adding more GPUs accelerate that timeline?

And maybe if you need human data, what’s the turnaround time for human data to arrive? If you put all of those together, that is first-principles thinking: what is the timeline, and what is the minimum number of days that is possible to achieve something?

swyx

I think that’s a lot of Elon’s type of thinking, right? He’s famous for saying that the only laws you can’t break are the laws of physics, something like that. You worked a lot with Elon.

Yeah. I guess one benefit of working at xAI is that you get a chance to interact more with Elon. I was very fortunate to get a few retweets from him, and that was quite fun. He also worked very closely with people. People imagine online that he’s very hands-on.

swyx

There are 2 things. 1, I was actually looking up Elon retweeting you. I’ll pull it up. He talks about you tweeting that you have a really good Voice Mode. I don’t know—

No, no, no—him, him, him.

swyx

Oh, I also did it, but anyway. I actually DM’d you feedback on Voice Mode because I was like, “Wow, really good.” And then I’m like, “Ugh, this sucks.” But I don’t know—anything you want to talk about building your Voice Mode? Was it a team you worked on as well?

Well, that’s actually not part of the team I worked on.

swyx

Yeah, okay. He probably worked on more of the video.

No, but Grok Voice is actually very good. This is one of those things where, first of all, you can speak at 2×, which is fun. I listen at 2×, so I like to speak at 2×. But also, I think the interruption was better than Gemini. I don’t know how it compares to ChatGPT real-time now, but as far as driving was concerned, having Grok in my Tesla and then driving, I think it was a really good experience.

swyx

Yeah. He likes Voice Mode, but also just the crazy reach by Elon. 15 million views for just saying, “Yes, true.”

Oh my God.

swyx

I guess the other thing is the safety aspect of the video model. Anything interesting to talk about there?

Ah, spicy.

swyx

Spicy question.

A lot of countries don’t allow generative AI videos without watermarks. So, in all of those countries, Grok Imagine had watermarks. A lot of the takedowns of the videos were also happening extremely fast.

swyx

I mean, it’s part of running a social platform, but it also transfers nicely to the GenAI side. Do you have a perspective on SynthID versus other kinds of watermarking?

Yeah, it’s going to be harder and harder to detect these things. SynthID was previously only Google, and now a lot of different platforms are also adopting it. The limitation is that the technology is in a paper that’s out there, and people can reverse-engineer how to get rid of it.

swyx

Yes.

Guest

I think even as it advances, it’s still possible to reverse-engineer it.

swyx

Yeah. If you’re interested, you can go on Reddit, and people have taken out the exact mask—or whatever you call it, the pattern—that Google applies. Then you can apply it to any Google-generated photo, and you can reverse out the SynthID.

Yeah.

Guest

It’s also harder and harder to judge by eye. I remember a couple of years ago, there were 6 fingers or something. It was very obvious.

swyx

My current tell is actually the audio. I feel like the audio is really lacking. My way to tell if something is AI-generated, outside of “Okay, I think I’ve seen enough; I have a decent eye,” is the audio match-up. Especially with Sora, it’s not great. It’s all a similar style.

But those are minor imperfections. I think the point is that my closest reference to this is also Ian Goodfellow, because I think he did the adversarial game thing where it’s like, “Okay, here’s a picture of a zebra.” Then you change 1 pixel, and it becomes a panda, right? That is a classic computer-vision issue.

Yeah. If you think about how these models were trained, as I mentioned before, the judge is in the training process. The objective, again, is that a model generates an image, and there’s a judge to tell whether the image is real or not. The model is trained to make the image more real.

So, as a model becomes more and more advanced, it’s going to be harder and harder. For me personally now, I have to judge whether these videos make logical sense.

swyx

Have a world model.

Yeah, yeah, yeah.

swyx

No, I also like it—the audio is too nice and too studio-quality. The lighting is too good. The skin is too clear. Basically, it’s the lack of imperfections.

Yeah.

swyx

Do we have a good way to do reasoning in diffusion? Is that what separates video generators from world models? We really know how to apply it to autoregressive language models. Is there a parallel for diffusion video-generation world models?

Alessio Fanelli

He has a thing on video agents.

Guest

Yeah, that’s a good question. I actually have a pretty big claim: visual intelligence is mostly coming from language. These video models, especially now, since diffusion-model technology is more mature, improve mostly because of the language model, not because of the video model itself—the video-diffusion model itself.

In Cosmos, typically these models have 2 parts. There’s a prompt rewriter, or prompt-upsampler, part. I think in Cosmos we use Llama, or we use Mixtral. The Cosmos video model itself is only 7B, and the language model—the prompt rewriter—is bigger than that.

So, the prompt rewriter’s task is to take a user instruction and convert it into an extremely detailed description of the video. Because video-diffusion models are visual models, I would describe them as kind of dumb: they take the input instruction literally.

In the training process, remember that we have to describe the video as detailed as possible when we’re creating the synthetic text pair. These models take those kinds of instructions to generate the videos. When you’re giving them user instructions, the user instructions are really simple. You just say, “A cat,” or something.

If you put a cat in the video model, it would take that instruction literally. It would literally show a cat, maybe on a white background because you didn't describe the background. The cat would not be moving because you didn't describe it. It takes the instruction quite literally. It's kind of dumb.

The prompt rewriter is actually a much bigger model, which is a language model that takes the user instruction and expands it. The thinking process you mentioned comes from there. If you look at GPT Image, it generates an image in 3 minutes. Those 3 minutes are not all spent on picture generation; a lot of the time is spent thinking. The prompt rewriter has now evolved to not only do thinking; it can also be an agentic model.

For example, say you wanted to generate an image of today's news. It would likely go fetch today's news online, process and digest it, organize the layout, and generate it.

swyx

If I'm not mistaken, these are no longer diffusion models, right? They're autoregressive, or is there still diffusion involved?

There are different approaches. For example, Gemini Omni—since they said it's Omni, I believe it's a single model. It may be something like a language model with a diffusion head or something. The language model does the thinking and the agentic tool-calling, and then it would use the diffusion head to generate the image in the end.

There were also approaches like Cosmos, where you have a separate language model and separate diffusion models. There are also purely language models where you discretize the images and then generate the image as discrete tokens. So, there are different approaches.

swyx

One of the claims I've seen for why these approaches struggle is that a lot of the benefits of how we currently learn reasoning with language models come from iteratively generating and reasoning. You have your thought, and then you work on that answer, right? So, if you have an Omni model and then a diffusion head, you can't feed that back in to continue reasoning, right?

You can't go text, image, text, image. You can't reason on the output and then go back to diffusion. But I guess in the new Gemini Omni, you would be able to, as long as you have—

I'm not sure if they have that process. I guess it's definitely possible in the Omni paradigm.

So, if you think about a traditional multimodal language model, it would have a ViT encoder that can encode an image. If it has a diffusion head, it can generate the image and then put that back into the ViT encoder, encode that, and then do iterative refinement of the result. Yeah.

swyx

I think you have to jointly train the ViT and the diffusion model to make that somewhat feasible, because otherwise you're kind of mismatching or feeding in slop.

I think it depends on the stage of training. You might be able to freeze it.

swyx

I also wanted to make explicit that we do know that Nano Banana and GPT Image are autoregressive language models with diffusion heads. As far as I can tell from your description of Grok Imagine, it is not. It is end-to-end.

I cannot comment on that.

swyx

Well, the way that you described it, I think there are different approaches, right? You started off saying the prompt rewriter is a big part of the intelligence.

Even on that, I think everyone should try using an early diffusion model. If you've used Stable Diffusion 1 or whatever, and if you've seen the prompts—“ultra high res, 4K, this style”—oh my God. The first time I tried one, I realized you don't talk to them like language models, right? Your prompting is very comma-separated—

swyx

Talking in the labels that were in the dataset, right?

Yeah.

swyx

But basically, I'm just trying to make the point that prompt rewriter and then image is different from an autoregressive language model with a diffusion head, right? They're different things.

Guest

Yes, they're different.

swyx

Just wanted to establish that.

I feel like the common part is the image part. So, it's quite surprising that a lot of the improvement came from the—

swyx

Language side.

The thinking and the tool-calling. I still remember that in Cosmos, I generated a happy sheep, and if there was any rewriting, it looked so CGI. After I rewrote it, it looked so beautiful.

swyx

Without any joint training.

Yeah, actually, without any joint training. This rewriting is already much better.

A very interesting thing I think will happen is that video agents, mostly language models, will use these generative models—whether it's a separate model or a diffusion head or whatever—as tools. The model can iteratively refine the results or even generate longer content through a very long chain of thought. It's actually very similar to how humans create art. We don't generate the pixels directly. We actually draw something, and I think through this process, these models will not only use diffusion as one of the tools; they can also use traditional tools.

They can use image-editing tools from Photoshop. You can use a video editor, FFmpeg, whatever, and take a combination of these tools and generative AI technology as a set of tools. Then they can interactively create a much better video for production-grade quality.

If you look at existing professional creators, they don't stop at generating a video from these models. They'll take this video to their editor and edit here and there.

swyx

So much post-production.

Yeah. Sometimes, actually, the reason the video is good is not really the video model; it's actually the editing.

We are engaged in the same process as well. We would love to use a video-editing model.

swyx

Yeah, actually, there's Grok Imagine Agent Beta.

That was the first attempt in that direction.

swyx

Yeah.

Guest

I think the process would be similar.

swyx

What is this agent mode?

You can ask it to—

swyx

There's no blog post for it.

Maybe generate a 1-minute video, which is not possible if you ask the same prompt to video models, but this model will really call different tools to do that.

When we first released a video-editing model, I saw a lot of people ask—some people tried the video-editing feature with, “Add this video to be 1 minute,” because they didn't understand how video editing works. Video editing is typically just removal, adding, replacing, or style transfer, these kinds of things.

But that's actually a valid task under the assumption of video agents. These agents should be able to understand these kinds of long-horizon tasks to actually create a long-form video.

I think this is really fascinating because it's taking the same direction as AI-assisted coding. First, you have things like tab completion and GitHub Copilot. From there, you gradually evolve to Codex and Claude Code, where you do things fully automatically. In Grok Imagine Agent mode, you can still go in there and do stuff by yourself. Gradually, as the model capability increases, it will be able to do everything fully automatically.

swyx

Yeah. I like that. Okay, so it looks like it's still generating. I also noticed that Grok image generation was always very, very fast. I don't know if this is something you guys benchmark, but this is just a tangent. Compared with what I used before OpenAI's latest image-generation model, and with Gemini's Nano Banana, I would often use Grok just—

It's in the benchmark somewhere. In the Imagine API blog post, they have all the speed numbers.

It's mostly a combination of distillation plus inference. There are a bunch of things: we talk about distillation, and then we talk about thinking. If you don't have any thinking budget, the model can just think for 30 minutes and then come back to you. Also, the inference infrastructure team was very talented, and they were able to accelerate a whole lot of these models.

swyx

Yeah. My comment on the video-agents thing is that I'm trying to figure out what people mean by video agents. When you initially told me about it—your bet on video agents, or your vision for video agents—I was a little disappointed. I thought, “You mean models are tapped out? Now we have to do agents?”

But I think you have to, right? The question now is how much model training is really going to make a difference versus just putting on a better harness. Like you said, the models don't have to be jointly trained. You can just take an off-the-shelf frontier reasoning model, slap it on a harness, give it Grok as a tool, and that's it: that's your video agent.

It doesn't seem super satisfying. Obviously, you can co-train and get some more percentage points of performance, but if your central claim is that the majority of video or generative-media alpha, or whatever, is actually coming from language intelligence and not image diffusion or video diffusion, then that is the future. That is the core of it. I mean, just wait—

If you pop back to the example, it generated frames. Sorry to interrupt, but it's been saying, “Okay, I'm going to start stitching these frames together.” It's using FFmpeg, using code.

swyx

This is what you see Imagine doing as well, right? It’s also just writing code in the background and then stitching and doing an image pass on the final output. It feels dissatisfying for the people who want to just train models.

It’s interesting, right? It’s also somewhat exciting. You brought up earlier that a lot of the gains don’t come as much from the video. I think you can see that in the language model space, too, right?

Anthropic is very, very good at coding. Their multimodal capabilities are not the best, right? They have basic PDF input, but there’s clearly a disconnect in the quality of their image, video, and audio processing. Yet their intelligence is very top-tier. Other labs—Gemini, OpenAI, xAI—you can add modalities, but it’s not like they’re unlocking crazy capabilities, right? So, it’s interesting.

Yeah, it’s interesting to see that video models’ capability increases actually come from language models being more intelligent. I think video agents can unlock more things than you might imagine. There are a few things.

When we’re prompting these models, most people are actually not very good at prompting. Language models have a better sense of how to prompt AI models. AI models know AI models better. If you jointly train these models, maybe the model has a better sense of how to prompt each model. Different models might be different.

Another thing is that it might not be as simple as just generating a few clips and slapping them together using FFmpeg. There might be more image- and video-editing tools appearing in this process. If you want to add some text at a specific timestamp, video models might not get the intention very precisely, but this is possible using deterministic tools. Video agents can use all sorts of tools, so you don’t have to put all of the capabilities into the transition model itself.

swyx

Yeah, I think that’s very true. For what it’s worth, I think you’re right. I think this will be a big category. I think you’re probably predicting that the next 1 year in video is going to be all this.

I mean, they already started.

swyx

[Laughter.] No, it’s so good. I think the last one is just longer. It didn’t give me 1 minute; it gave me 36 seconds. But are we feeling it now? Is there going to be an inflection? Is there any timeline prediction you want to make?

I guess by the end of this year, this is going to be a big hit, so the inflection point will be there. Videos generated by video agents can get to production-grade quality. They can be presented and distributed in ads.

Once that happens, I think enterprises will have much more budget for video models because agents are inherently more expensive than the video models themselves, because they do this whole process. They generate many, many variations. But once these models have passed this usability threshold, I think it’s going to be exponential growth beyond that.

swyx

Yeah, I would fund a company right now based on this thing. So, I think you’re right. One thing I’m surprised by, reflecting on the whole past hour of our conversation, is that you’re into world models and video generation for video generation’s sake. I think a lot of the other world model people we’ve interviewed—General Intuition, Fei-Fei Li, and all those guys, and Moonvalley, which I think I told you about—Moonvalley. I keep saying Moondream. God damn it. Moonvalley. A lot of them actually say that robotics is the end game—embodied robotics. You want real-time, you want interactive. It is to interact with the physical world. You’re not that concerned about it.

I think robotics will be a big part of it, for sure. I guess the process might happen naturally. My prediction on robotics is that physical AI might be solved, and we don’t actually need to—

swyx

Be in the real world.

Need to be in the real world. So, it might get solved by a video model with very strong video capabilities. Remember, we talked about real-time, interactive, long-horizon video. These models are currently just training on screen recordings and computer screens. Once these models can use computers and understand the future state of a computer extremely well, robots might be one of the tools a very powerful AI can use. Powerful AI might just be able to control a physical embodiment naturally.

swyx

I see that for sure. Cool. I know we’re coming up on time. You had left one more spicy topic, which is why you left xAI.

For me, there’s a lot of research you want to do that you cannot do at a company. Also, the priorities and objectives for a company typically can change very fast. It’s also the same for xAI. Now it’s the time for some research I want to do, especially more on the language model side, and I cannot do it at xAI.

swyx

Oh, okay, yeah. So, you’re basically leaving. You had this whole transition from computer vision to world models, video generation, and now you’re focusing on LLMs. But it seems like, in saying you’re focusing on LLMs, you really described how it all ties together in the past hour, right?

Yeah, but I don’t know—what do you mean by focusing on LLMs?

I realized that the video models—even in the beginning, the game I came from was improving diffusion technology—but this is the point where most of the gains actually come from the language models themselves.

swyx

It’s a huge black pill for anyone who’s spent their career in generative media.

I mean, that’s an extreme view, right? You still definitely need a bit of both, right?

swyx

Yeah. There just seems to be more pressing, impactful work to do now on the language model side. Do you have any similar predictions? You predict video agents, and you’ll be right. On the language side, what are you looking for in the next 1 year?

I think one thing that’s pretty interesting, and I think might happen soon, is that language models will become context-aware and manage their own context.

swyx

Yeah.

Guest

From the video model side, we’ve been suffering from the long-horizon issue. We want to generate longer and longer videos, and we’ve been trying to solve the context-length issue through various ways. One way is brute-forcing longer context lengths. Another is to manage the context better.

I think the same thing in language models is also going to happen soon. For example, language models aren’t aware of how long their own context length is. Once they hit 80% or something, automatic context compaction is triggered. The model isn’t aware of that while it’s working. Maybe it would be good for models to know, “I’m approaching 80%,” or something.

Something else that’s pretty interesting is that, for example, in OpenClaw, every time you type in something, the current local time is automatically attached to your message. So, the model actually knows what time it is. This makes the model time-aware.

Also, in tool calling, a lot of the intermediate tool-call results are automatically pruned. There’s context removal, context addition, and context compaction. All of these come from the harness systems themselves. From our experience, heuristic engineering also helps the models absorb this into the models themselves. I guess that’s something very interesting to explore.

swyx

To infinite context.

Maybe.

swyx

No, [laughter] but it’s interesting, right?

Guest

It is in the space of memory and continual learning.

swyx

I don’t know. It’s also in the space of agent-harness use, right? You’re saying you don’t want to do it in the harness, right? But models are also being trained using harnesses, right? So, some of it, you could say, is implicitly leaking in, right?

Part of that post-training of language models is using them in coding harnesses, in which case, when our subagents run, when is context compaction going to happen? It’s not explicit—you have this much token window, which I don’t know if you want it to be. Is that going to change? But it’s somewhat leaking in there.

I’m imagining what would happen if the model had access to the whole source code of the agent harness itself and were able to modify it however it wanted. If the agent harness is short enough, you can just put it in the context window in the system prompt, and then the model can say, “When I want to spawn a future version of myself, I can modify the agent harness.”

For example, when I’m reading a long document, I can choose to read the whole thing in chunks and then come back and smash the summaries together, or I can just read the first 200 lines and discard the rest. All kinds of choices, if they can be made by the models themselves, might be very interesting to see. A model can program itself online at test time.

swyx

Yeah. Self-modifying harnesses are also part of OpenClaw and Pi, but I think there’s a lot more work to do there.

swyx

Very cool. I think part of me is curious: you're part of a big lab, right? There's this career path of a researcher at a big lab: you train models, get more compute, train better models, and keep going. And, somewhat, I feel like you're opting out of that. If I were you, I'd be like, “Oh, I think this is a bit of a career risk.” You know what I mean?

Mm-hm. Mm-hm.

swyx

I don't have any comment apart from saying that you're very strongly convicted. I think that a lot of people in your shoes would not be doing what you did.

Guest

Yeah. Speaking of my career, as I look back, there were actually a lot of huge transitions. 10 years ago, I was doing research with the ResNet authors, Xiangyu Zhang and Jian Sun. At that time, the research was completely different. It was mostly computer vision, like image recognition, object detection, and object tracking. I was also doing neural network compression at that time. It was quite different from knowledge distillation these days.

At that time, I wanted to be a professor, and when I applied for a PhD, I already had a few first-author papers at top conferences. So I confidently applied to the top schools. It turns out I got rejected by all of the top PhD programs. So I had to go to industry. At that time, I was at Facebook AI Research (FAIR), led by Yann LeCun.

swyx

I want to talk about V-JEPA, but—

Yeah, I know. We can leave it for another time. I switched to self-supervised learning at that time. It was quite different from what I was doing in computer vision.

swyx

Yeah.

Guest

After that, it was NVIDIA Cosmos. So I realized scaling up was extremely important. At NVIDIA, I was mainly focused on scaling. One thing was Cosmos, scaling the video foundation models to a few billion parameters. Another thing was that I was working on MoEs. Megatron MoE was the first open-source framework able to train these MoEs at very large scales, from 100 billion parameters to even trillions of parameters, efficiently at 40% MFU.

Switching to xAI meant trying to work at an even larger compute scale. Looking at this trajectory, I actually worked on a lot of different things. I feel that within ML, it's actually easier to switch than you think. A lot of people might have mentioned, “Oh, I work on computer vision. I always have to work on computer vision, and I cannot switch to language.” But from my experience, at least at NVIDIA, I worked on both language model MoEs and video models. It's actually not the case. A lot of the core principles for training large models are largely the same.

For me, I feel that right now the bottleneck for video models is actually the language part—the agent—which is why I want to work more on LLMs. One thing is, it's a bit of a challenge. I don't think it's a huge jump, though.

swyx

Yeah, I mean, kudos to you. I think you have a lot of strong vision there. I think that was mostly everything that we wanted to cover. You've been very generous with your time, and it's really nice that you're able to share all these things now. We don't have to go through xAI to clear everything.

There's a lot of good stuff about xAI compared to what you just see in the releases, right? You don't realize how many more levels there are to it.

swyx

xAI, please do more podcasts. Anyway, thank you for sharing. It's been very kind. And I want to hear more from you. I think you're going to embark on your next phase. You haven't announced what you're doing next, but clearly you have more vision and more ambition on this path, and I think you're basically gradient-descending to whatever your final form is.

Thank you. Yeah, I'll share more about my next chapter soon. Thank you for having me.

swyx

Thanks for coming.

Inside xAI: Building Grok Imagine in 3 Months, Videogen vs World Models, and Video Agents— Ethan He | BidClub