Type a Sentence, Get a Playable 3D World in 3 Seconds - Shlomi Fuchter & Jack Parker-Holder

By the way, look at this dog. This is amazing. This is insane. What was the prompt to create that?

Today is a world exclusive of what is, in my opinion, the most mind-blowing technology I've ever seen and the most poggers I've ever been. You're not going to believe what Google DeepMind showed me in an exclusive demo in London last week. This technology might be the next trillion-dollar business and might be the killer use case for virtual reality.

Google DeepMind has been slaying so hard recently that even Gemini Deep Think can't count the number of wins in the context window. Let me explain.

Today we're going to talk about a new class of AI models called generative interactive environments. They're not quite like traditional game engines or simulators, or even generative video models like Veo, but they do have characteristics of all three. They're basically a world model and video generator that is interactive. You can hook up a game controller, or any kind of controller for that matter.

DeepMind says that a world model is a system that can simulate the dynamics of an environment.

DeepMind Genie 3 [World Exclusive] (Jack Parker Holder, Shlomi Fruchter)

The consistency is emergent. There is nothing explicit. The model doesn't create any explicit 3D representation.

How do you square the circle between a stochastic neural network and the fact that it has consistency? So I look over here, I look back, I look there again, and the thing is back. Isn't it a bit weird that a subsymbolic stochastic model can give us apparently consistent, solid maps of the world?

Do you remember the Quake engine in 1996? It required explicit programming of the physics, rules, and interactions. But this new generation of AI systems learns real-world dynamics directly from video data. You can control an agent in the world in real time.

The move towards generative world models was born from the limitations of hand-coded simulators. Even their most advanced platform, XLand, which was designed for general agent training, was the frontier for embodied-agent training with curriculum learning. But it felt far from the real world. It was almost cartoon-like.

It could model 25 billion tasks, but it was still handcrafted. It was constrained to the rules of that particular domain, and it was janky. Imagine if you could just generate any interactive world you wanted to train your agents on with a simple prompt.

Now cast your minds back to last year, when I interviewed Ashley Edwards at ICML. This was the first version of Genie, which was trained on 30,000 hours of 2D platformer game recordings.

When we're generating the next frames, the objects that are further away are moving more slowly than objects that are closer. This is a sort of effect that you would often see in games, so that you can simulate depth. It's something that we also have when we observe things moving: we see things moving slowly when they're further away. So, yeah, the model learned that.

Just being able to be that good at understanding the physical world wasn't something we were expecting it to be that good at that quickly.

The core innovation of Genie 1 was a spatiotemporal video tokenizer that converted raw footage into processable tokens, a latent action model that discovered meaningful controls without labeled data, and an autoregressive dynamics model that predicted future states. The latent action model, a form of unsupervised action learning, was the core innovation.

Genie discovered 8 discrete actions that remained consistent across different environments, purely by analyzing frame-to-frame changes in game recordings. This means it knew what “jump” meant or what “move left” meant without being explicitly trained on those actions.

This was an OMG moment for me. How was that even possible from training on offline game episodes? Even more surprising was how it seemed to have emergent capabilities like 2.5D parallax.

Just 10 months later, Genie 2 arrived with 3D capabilities and near-real-time performance. The visual fidelity was much higher. Now it can simulate realistic lighting like the Unreal Engine—things like smoke, fire, water, gravity, pretty much anything you might see in a real game.

It even had a reliable memory. You could look away from something, bring it back into view, and it would remember the thing.

This is GigaChad Jack Parker Holder. He's a research scientist at Google DeepMind on the open-endedness team, talking about Genie 2 with Demis, no less.

DeepMind Genie 3 [World Exclusive] (Jack Parker Holder, Shlomi Fruchter)

This is a photograph taken by someone on our team somewhere in California. What we then do is ask Genie to convert this into an interactive world. So we prompt the model with this image, and Genie converts it into a game-like world that you can then interact with. Every further pixel is generated by a generative AI model.

So the AI is making up this scene as it goes along.

DeepMind Genie 3 [World Exclusive] (Jack Parker Holder, Shlomi Fruchter)

Exactly. Yes. Someone from our team is actually playing this. They're pressing the W key to move forwards, and from that point onwards, every subsequent frame is generated by the AI.

Around the same time last year, you'll probably remember this, DeepMind's Israel team, led by Shlomi Fruchter, showed diffusion models simulating the Doom engine. The system was called GameNGen.

It's almost a meme at this point how Doom runs on calculators and toasters. But here is a neural network confabulating a Doom game frame by frame in real time. Look at how it just knows what the health is. You can shoot characters. You can open doors and navigate around maps.

Occasionally it was slightly glitchy, but this is just unreal. You can simulate Doom at 25 frames per second on a single TPU. The only limitation, of course, was that it could only do Doom and nothing else.

So, last week we waltzed our way into London, and Jack and Shlomi gave us a demo of Genie 3. Honestly, I couldn't believe what I was seeing. The resolution is now 720p, which is firmly in the good-enough territory to suspend disbelief. It's real time.

It can simulate real-world, photorealistic experiences that can continue for several minutes before running out of context. Shlomi had his hands all over Veo 3, by the way, and they seem to have combined elements of the Genie architecture with Veo, producing something I can only describe as Veo on steroids.

Unlike Genie 1 and 2, the input is now a text prompt, not an image. They argued that this is a good thing from a flexibility perspective, but it does mean that you can no longer take a photo of a real place and generate from there.

One of the main features of Genie 3 is that it has a diversity of environments, a long horizon, and promptable world events. On the world events, let's take this ski-slope example. We might type in, “Another skier appears wearing a Genie 3 T-shirt,” or, “A deer runs down the slope,” and there you are. Things just happen in the world.

They say that this might be very helpful for modeling things like self-driving cars, where you can simulate rare events. But I was left thinking that this is just turtles all the way down. How can we write a process to prompt the potentially infinite number of rare things that could happen in a scene?

There was an example they showed of flying around a lake, and it was amazing. But I was thinking, where are the birds, mate? Can you type the birds into the prompt?

The team believes that we haven't yet had the Move 37 moment for embodied agents—the moment when an agent discovers a novel real-world strategy. They see Genie 3 as the key to enabling that.

But the real world constantly surprises us because the real world is creative. Creativity simply means that the tree of things that can happen keeps growing new branches, and leaves just keep appearing. Perhaps in the future we might have an outer loop that makes the system more open-ended.

But right now, in my opinion, Genie 3, like all AI, gives you exactly what you ask for in the prompts and isn't creative on its own. Currently, the system only supports a single-agent experience, but imagine how cool it would be if you could extend that to a multi-agent system. Apparently, they are working on that.

Personally, I'm most excited about a new modality of interactive entertainment. Just imagine YouTube version 2.

DeepMind sees the main use case of being able to train robotic simulations as the real game-changer. This seems plausible too. The miracle of human cognition, in our brains, is that we have evolved to simulate the world without direct physical experience, which is expensive.

This is basically the same idea: why train in the real world if we can just simulate any possible scenario in a computer, just like that Black Mirror episode? Here are a couple of examples they gave of using simulated environments to train an agent to do some specific language tasks.

Now, with Genie 2, they said they were happy if it was consistent even for 20 seconds. But now, when you notice something inaccurate, it's very surprising. The key thing is that it now extends beyond the prediction horizon of the average human, and the glitches are getting harder and harder to spot.

They said that Genie 2 wasn't actually real time. You had to wait a few seconds between taking different actions. It was low resolution and had limited memory. It was superficially really good, but it didn't look particularly photorealistic.

Genie 3 changes all of that. Genie 1 supported around 10 seconds of generation, Genie 2 around 20 seconds, and Genie 3 is able to simulate interactive environments for multiple minutes.

This time around, they were a little more tight-lipped around the architecture. They wanted to focus on capabilities in the interview, and that's fair enough. It's understandable, given that this is potentially a trillion-dollar business and Zuck will be sniffing around like a truffle hound.

My biggest concern with this is that as soon as Zuck gets wind of this, he is going to be getting out his checkbook.

He's going to go straight to Jack and Shlomi, and he's going to be like, “Come on, boys. $100 million. Come to work for me.” Zuck, mate. Seriously, no. Don't do it. These guys are doing God's work over here. You need to just let them do what they're doing. You can make it yourself if you want, Zuck. Leave them alone.

I should say, I did joke at the end of the interview that if you're learning Unreal Engine right now, you might want to pivot to a different career. But the Google guys were quite grounded. They argued that this is a different type of technology. There are pros and cons, which is fair.

I should stress that, as amazing as this technology is, it's still a neural network, and it still has many important limitations. Certainly, though, just imagine how easily you could generate interactive motion graphics with this technology. That's something that Unreal Engine has been leaning hard towards in version 5.6. So, do I need to fire my motion graphics designers? Victoria, will users be able to use this? Not anytime soon.

This is still a research prototype, and given the obvious safety concerns, they're going to open this up progressively through their testing program. One question did come up in the press conference yesterday, though: could it generate an ancient battle? Shlomi said that it's not trained on that kind of data and wouldn't be able to do that yet. So, certainly not a specific historical battle, anyway. It does sound like there are still some limitations.

How can a system like this ever be fully reliable? Well, they did say that with better models, the trend is that they get more and more accurate. The glitches become fewer, and they expect to see further improvements. There's this annoying phrase, “This is the worst the model will ever be,” but even as I said, they can generate some edge cases using a whole bunch of prompt augmentations. It might just be turtles all the way down, though. How do you come up with all of the rare black swan events that might happen?

So, what data was it trained on? They were quite cagey about this as well. It's probably safe to assume that it's been trained on all of YouTube and lots more besides that. How much compute does this thing need? Well, I asked them that, and they were a little bit vague about it. They said that it ran on their TPU network, so I'm inferring from that that it needs a crap ton of compute.

However, I can say that it was demoed in front of me, and it was very responsive. You put a prompt in, it thinks for about 3 seconds, and then you're just in and it works. They also mentioned some cool stuff about how Genie can be used to train agents, as we said, but the agents themselves could be used to better train Genie 3, creating this virtuous cycle of iterative improvement.

If you're in a world walking around and, say, you go to cross the street, you sort of check the cues of the drivers. For example, maybe there's not a crosswalk, and you need to know when to stop. You can see that they're slowing down, so that's when you would go, and the other agents should be simulated in that fashion. Genie 3 and other similar models would be impossible without at least some human feedback in the training loop, the data curation, or the evaluation.

Prolific is a human data platform and they are sponsoring this video today.

Enzo

My name is Enzo. I work at Prolific. I'm the VP of data and AI. I support everything from AI data research and the likes. For those unfamiliar, Prolific is a human data platform for working with everything from academic researchers, but also small and large players in the AI industry. Visit prolific.com.

Yes, this is the demo where they've got the Genie 3 memory test on a blackboard. You see there's an apple and a cup, and then you go out and look out the window. You see there are a few cars, and the purpose of this test is to show that they've got such a long context window—similar to the concept of a large language model—that it still remembers all of the things that it generated, even if it was minutes ago. We've got the blackboard over here; we look up, and there it is. It remembered it: “Genie 3 memory test.”

I've also noticed that this model is even better than Veo 3 at things like text. You would think that they would dumb down the model to make it interactive and to make it this sophisticated, but even as a video-generation model, it seems almost better than Veo 3 for doing a whole bunch of stuff.

All right. So, I'm Shlomi Fruchter. I'm a research director at Google DeepMind. I'm the Veo co-lead. I've been working at Google for about 11 years, recently on diffusion models in the various modalities—image, video—and we'll tell you more about what we're working on right now.

Hey, I'm Jack Parker Holder. I'm a research scientist at Google DeepMind in the open-endedness team, originally working on open-ended learning and open-endedness, and more recently working on world models.

We are here at Google DeepMind in London, and you guys have just demoed to me something which I think I'm more impressed with than anything I've seen probably ever before. I think it's a paradigm-changing moment. Shlomi, can you tell us a little bit about this new version of Genie?

Sure. So, Genie is our most capable world model. By a world model, what we mean is basically a model that is able to predict how an environment would evolve and also how different actions of an agent would affect this environment.

So, with Genie 3, we are able to basically push the capabilities of a world model to a new frontier. That means high resolution, much longer horizon, and better consistency—all that in real time, basically allowing whoever, whether it's an agent or a person that interacts with the system, to walk around it, navigate it, and affect it while the generation happens in real time.

Genie 3 is just ridiculous, right? It's on a completely different level. But maybe we should just contextualize that around Genie 2. So, what was Genie 2?

Yeah, it's a great question. So, Genie 2 was sort of the culmination of 2 years of research in what was quite a new area, which is foundation world models, as we called it at the time.

Essentially, in the past, world models had modeled a single environment. So, the canonical world models paper in 2018 from David Ha and Jürgen Schmidhuber modeled the car-racing environment, which is a MuJoCo environment, and it could just model that 1 environment. We've seen that with the Dreamer series, also from Google DeepMind and Danijar Hafner, with Atari games and other kinds of environments as well, but no one had ever done something that could create new worlds.

With Genie 1, the real novelty there was that we had a model that, for the first time, could be prompted to create completely new worlds that didn't previously exist. But that being said, they were fairly rudimentary. They were low resolution. You could only play with it for a couple of seconds, so agents couldn't really learn the long-horizon behaviors that we wanted them to, and the diversity was still fairly constrained. It also required some form of image prompting.

With Genie 2, we really pushed that to the next level. We trained it on a much larger distribution of 3D environments. We moved to 360p from, I think, 90p before. So, it was closer to what we see now, but it was still sort of scratching the surface because we didn't really know that this approach could scale the way we've seen other methods have.

We wanted to really test this from a research standpoint. But then, I think, for this year, we wanted to really take that to the next level. That's what we think we've done.

Yes. And it's now 720p. It's interactive. So, Genie 2 wasn't interactive; it wasn't fast enough. Steve Jobs said there's something magic about the touchscreen, right? There's something magic about it. And, of course, the magic happens when it's interactive.

Some of the demos you showed me were just insane. They're photorealistic. I mean, it's kind of like a fusion of Veo, I suppose, that can now understand the real world, and you can build essentially a foundation model for the real world which is interactive. That's mind-blowing. Tell me about some of the examples you showed.

Yeah, so I think what you said about Veo, or more generally about video models, is right. There is a way we can think about them as somewhat of a world model, but it doesn't really allow us to actually navigate or interact with it completely interactively. I think that's one of the limitations of video models that we're trying to address with Genie 3.

Basically, in the examples that you've seen, because Genie 3 generates the experience and what we see frame by frame, it lets the user or the agent that is using it basically control where it wants to go at a very low latency. That allows basically exploring the environment and creating new trajectories that are not predefined like video models.

In the examples that you've seen, for example, you can see the character or the agent in this video moving around, maybe going back to the place they've already been to before, and everything remains consistent. I think that's a very remarkable property, or capability, of the model: the ability to preserve the consistency of the environment along very long trajectories.

Yes, and even Genie 2 had some kind of object permanence and consistency, but nowhere near as much as we have now. We'll come back to that in a second.

We can't say too much about the architecture for Genie 3. But in Genie 2, there was an ST transformer, so a spatiotemporal transformer, which was conceptually quite similar to a ViT. There was also a latent action model, which means even from non-interactive data, you could infer some low-cardinality action space, and then those went into a dynamics model.

I think what we can say about the architecture that might be interesting is that, definitely because of the interactive nature of the problem or the setup, the model is autoregressive. What that means is that the model generates frame by frame and has to refer back to everything that happened before. So, if, for example, we're walking around some auditorium or some other environment and we revisit a place that we've already been to, the model has to look back and understand that this information has to be consistent with what's happening in the next frame.

I think the interesting point here is that all the consistency is emergent. There's nothing explicit. The model doesn't create any explicit 3D representation, unlike methods such as NeRF and Gaussian splatting. I think that emergence of capabilities is very interesting and surprising for us.

Yes, and even Genie 2 had emerging capabilities like parallax, and it could model certain forms of lighting and so on, but this just blows my mind. You're involved in that Doom simulation from last year, and even that just blows my mind. We all played Doom in 1993. It was one of John Carmack's finest.

And now you're saying that—I mean, certainly the work that you folks did last year—you've got a neural network model which is subsymbolic. So there's no explicit model of the world. You don't know where the doors are. You don't know where the lights are, where the maps are, and so on. You just kind of take a sample, a traversal through this space, and it just produces the game in pixel space. I mean, that's—

Yeah, I've been playing games, obviously, including Doom and others, and I also worked on game-engine development at some point very early in my teens. I think what I really like about this project is that we're now able to run models that actually generate consistent 3D environments, as in a game engine and the Doom simulation, and they run on GPUs or TPUs, while in the past we were running game engines on the same hardware. So I think it's really something very interesting, and it kind of closed this circle for me.

In particular, in the case of GameNGen, we tried to push the real-time interactive aspect. We basically said, “Okay, would a diffusion model be able to simulate a game environment end to end with nothing explicit—no code, nothing—except for actually generating the pixels and getting the inputs from the user?” We weren't sure if it was going to work, so with this kind of research, we try and it doesn't work, and then all of a sudden something happens and we see that it does work. That's a very rewarding moment.

I think the reception was a bit surprising, because there's something about the real-time interactive capability that really sparks the imagination: I can actually walk into this environment, maybe a generated environment, and actually experience it. Looking back, we were excited about the real-time nature of the simulation, and we really wanted to bring it to higher-quality, more general-purpose simulations.

So, Jack, one of the million-dollar questions is, even with a language model, it's stochastically sampled with this temperature parameter. Same thing here. With Genie 2, the dynamics model is using MaskGIT, and it was run iteratively. How do you square the circle between a stochastic neural network and yet it has consistency? I look over here, I look back, I look there again, and the thing is back. Isn't it a bit weird that a subsymbolic stochastic model can give us apparently consistent, solid maps of the world?

That's a really good question. I think, probably similar to language models, there are some fundamental things about the world that you want to remain consistent. With a language model, even though, as you said, they can be stochastic models, if there are things that are stated as facts in their context, they'll still probably recall them correctly, right? Whereas new things are where they maybe have more degrees of freedom to change things like that.

So I'd imagine in a Genie-generated world, if you were to move around, then maybe new things would have some degree of stochasticity to them, right? But once they've been seen once, then they should be consistent from that point forward, because the model knows when to use this stochasticity. This is an emergent property from the scale that we train at.

Yes. We'll save the emergent discussion. I was just telling the guys about my conversation with David Krakauer the other day, but maybe we won't go there.

The other really interesting thing is, you said David Ha, 2018, with Jürgen Schmidhuber, the World Models thing. And, Shlomi, in the presentation you defined a world model as essentially being able to simulate the dynamics of something, right? If a world model simulates the dynamics of a system, how could you, for example, measure that?

I think it's very hard to measure the quality of world models in general. When it comes especially to visual generation—image models and generative models in general—it's very difficult to measure their quality, because the quality is somewhat subjective, right? I think for LLMs, we're actually in a better place because we can measure their performance. First, of course, there's perplexity, just a next-token prediction problem, but later on we actually care about how they operate for the tasks that we care about, right? So we measure, for example, downstream performance on various tasks.

But when it comes to world models—and today we focus mostly on the visual aspect—it's important to highlight that the world is more than just visuals, right? Again, for Genie, we're focusing more on that because a lot is captured in the visual interaction of the world. So measuring how well a model is doing really depends on the context and also on how we want to use it later. I think that's something we have to keep in mind when we evaluate models.

We have in mind one particular application that we think is really key, and that's to be able to actually train AI agents and let them interact with simulation environments. I'm coming more from a simulation background, but training agents in simulation environments wasn't my original background. Through interacting with other people at DeepMind who have been exploring that for a long time, over the last few years I've come more and more to realize how much potential there is in that.

Because if we really think about it, AI would be limited by the ability to perform physical experiments, right? Imagine that you want to develop a new drug or a new cancer treatment. You cannot really do it in the real world if it takes months for every step along the way, right? The same goes if we want to learn how to assemble something: if I have to train the robot in the real world, it might take a very long time. So that's why simulation of the real world is really key, and that's what we hope we kind of push a bit further with Genie 3.

Yes, very exciting. I spoke to a startup recently, and they sketched out this future where we'll have essentially a model platform where people doing robotics can download policies. I'm in a factory, and I need a policy for doing this particular thing, but of course they imagined that it's so scarce, it's so difficult to get real-world data, that there would be a marketplace. Everyone would train their own policies and sell them to other people on the market.

This is a slightly different vision. You're saying that now we have a world foundation model, and essentially I could say, “Well, in this situation, I need to have a robot policy for doing this particular thing.” So I can just spin off a job, create the policy, and away we go. Is that roughly correct?

I think that is kind of the vision that we have. In robotics in particular, there's a lot of focus on deploying robots in somewhat constrained settings. It might be, for example, in someone's apartment that's very staged—almost as staged as a podcast recording—with all these support staff watching around this robot achieve one goal. From a control perspective, it might be very impressive, but in terms of the stochasticity of the world that it's in, it's very limited, right?

If we look at simulated environments, they might accurately model physics, but they definitely don't model things like weather or other agents or animals. Whereas a model like Genie 3, because it has world knowledge, that world knowledge extends beyond physics to also include the behavior of other agents. And as we showed you in that example at the beginning with the world events that we can also inject, you can prompt it to have another agent cross in front of you, or, as we had, a herd of deer run down the ski slope.

I think these are the kinds of things that are necessary for robots to be deployed at large scale in the real world.

The real world is fundamentally populated by people and other agents. This is something that we can gain from training on this general-purpose world model, and I think we just have no other approach to scalably get this data in a safe way. Safety is a critical element of this: we can simulate things in a realistic way without having to actually deploy agents in the real world.

Yes. And that was a very important detail. So you can prompt an event in, and you gave me an example where there’s a skier going down the slope, and then there’s a guy with a Gemini T-shirt. I guess what I’m thinking about here is that if we did train these robot policies, we would probably need to do some kind of curriculum learning and some kind of diversity.

We would start off with a simple environment, and then we’d add the guy with the Gemini T-shirt; then there’d be a car coming along. Maybe in reality there would be some kind of meta-process creating some gradient of complexity and diversifying environments. I love that Kenneth Stanley paper, the POET paper, doing something like that. But is that a fairly reasonable intuition?

So I think it’s still early to say exactly how world models like Genie 3 will actually be used for AI research. I think we can only say directionally that, in general, we still see in other generative models that there are some capabilities that we discover. We don’t necessarily know that they’re there, and then through interaction and development, we’re actually seeing them emerge.

For example, just a few days ago, we shared that you can write some text on a photo and provide it to Veo, and it reads the text and follows the spatial instructions as well, right? I think that’s something that we didn’t necessarily explicitly train the model to do, but it’s capable of doing. I think here as well, the capabilities of Genie 3 that we’re exploring are still emerging, and we’re still discovering new things. I think that’s something we hope to learn, first by having more testers and external testers to whom we’ve already previewed the model and from whom we’ve gotten feedback. We hope that through this kind of engagement with the community, we can better see how these models will be useful, and that’s something I expect to take some time as we try to understand the best application.

I’m a huge fan of open-endedness, for example. Certainly, at the moment, when we prompt models quite generically—when what we put in the prompt is quite generic—we tend to get quite simplistic answers. A lot of people doing computer graphics, when they prompt image models, have so much specificity, and they deliberately take it out onto the tail of the distribution so they get something that’s novel and interesting, and so on.

The real world just always produces a sequence of artifacts which are novel and interesting: you get random NPCs walking onto the screen, and cars go by, and so on. Is my intuition correct that, at the moment, as good as Genie 3 is, you tend to get quite a specific scene, and you don’t have random planes flying over or just random things happening?

Yeah, that’s a really good intuition, right? It definitely is the case that the model is very aligned with the text prompt that it’s given. Therefore, there’s a lot of emphasis placed on the quality of the text prompt to describe the scene. But I actually wouldn’t see that as a limitation; I would see it as a strength, right?

Firstly, it means that there’s still a lot of human skill involved in creating really cool worlds. You see some of the examples we showed you. We have some very talented people who can do amazing things with these models, and there’s actually a lot of value added there. It’s a tool that can really amplify already creative humans in new ways. I’m definitely not the best at doing this, right? I can tell you that it is really impressive when someone is able to do that.

But on the flip side, from the agent perspective as well, when we’re talking about designing environments for agents—and you referenced POET, which was, for me, like, POET and world models were the 2 papers that I just thought were eventually on a collision course. That’s basically why I started my research career.

I think POET was fundamentally limited because the environment encoding was an 8-dimensional vector, but also because there was no real notion of interestingness. In your recent interview with Jeff, he obviously talked about how this problem is largely now solved with foundation models. These foundation models can not only define what’s interesting based on standing on the shoulders of human knowledge, but they can also steer the generation of worlds in things like OMNI-EPIC to do this.

In that case, it’s done through code, but here we have text as a substrate as well. So in theory, these kinds of open-ended algorithms that use language could actually be quite strong places to have these kinds of notions of interestingness, with agents steering tasks through that space as well.

Yeah, I think this is the fundamental thing, because certainly with creative models at the moment, weirdly, counterintuitively, you need more skill to make them do something interesting than you did before. The average creative process now for someone designing a thumbnail on YouTube is that they mix together a whole set of tools: they might use a ControlNet model, they might use an upscaler, and they might then use another image-generation model.

You get this huge compositional tree of operations that happen, and it’s very, very highly skilled, because a lot of the structure for constraining the generation of these models still comes from our own abstract understanding of the world. This is kind of what Kenneth Stanley was saying: we have this understanding of the world which is constrained by things like symmetry and various different rules, and then we hint to the model—we constrain the model in the prompt using those things. Would the models ever be able to do that without humans needing to prompt them?

So I think what’s interesting is that, eventually, what we find interesting and worth maybe watching, investigating, or researching is eventually being defined by people. In the case, for example, of video generation, we see people find ways that maybe we weren’t expecting: they use the tool that we put in front of them to generate new things.

For example, we have people making ASMR videos of cutting fruits made of glass, right? That’s not something you can do in the real world, and the novelty comes from the prompt, basically. I think that’s what you’re alluding to. In the case of world models, we’re still in a similar place, I would say, because you have to provide the description of the world that you want to maybe walk into and experience.

But some elements would kind of emerge from—and would be inferred from—the prompt that you provide, right? You can maybe write a very short prompt, but the world will still have much more richness. So I think there’s a question of where this richness is coming from, and I think there are different levels of the models’ ability to bring this richness into your experience.

Over time, we see that this ability becomes higher and higher, and a little information provided by users can actually generate very rich videos or experiences. So I would say it’s a bit of an evolving answer. Over time, I expect that more inputs to the model—or you can think about it like the person is providing a seed—and from that seed we can maybe generate more elaborate descriptions and, finally, an experience.

I don’t think of it as a 1-step process, but more as a series of creative steps. Each one of them can be done by a person or by an AI model, and together they generate maybe something new.

Yeah. And that’s what we’re seeing play out on Twitter: the creative process is like generate, discriminate, generate, discriminate, and we memetically share all of the prompts that work. That’s why we’ve just created this beautiful phylogeny of creative artifacts that are exploring the space of these models, which is beautiful.

I’m thinking about the future. I know you probably can’t speculate about this, but this could be the next YouTube; it could be a new form of virtual reality. In philosophy, there’s this thing called the experience machine, where you plug yourself into this better-than-life matrix simulation, and no one wants to leave the experience machine because it’s better than real life.

But we could co-create something like that, right? It could be on a phone or a virtual headset, and we could create these worlds and portals between the worlds, and it would just be a never-ending simulation.

Yeah. That’s a great question. Going back a few steps, I think another really inspiring thought experiment in this space, before the generative models really became capable, was something like Picbreeder, right? In that case, it was a very simple idea: it was just evolving some images, basically, and some quite surprisingly creative things emerged from that experiment that I don’t think many people would have expected, right?

So you had these beautiful images emerging—to use the word “emerging” again—from just evolving user preferences over time, right? We definitely see modern analogies of this, like you described, with social media platforms sharing prompts and people generating ideas, and then it emerges in different ways or goes in different ways, like the Veo ones, with people generating stand-up, for example, and then suddenly there’s tons of exciting content in that space.

I think it’s definitely fair to say that what we’ve done with Genie 3 is create another form, another platform or type of model, where this kind of creativity could happen, and it could also lead to some unexpected, exciting things. But I don’t think we can speculate too much at this point about exactly what those will be, other than to say that it should be interesting and humans will likely do cool things with it.

Yes. I was discussing with Kenneth the other day whether—because he’s a big fan of neuroevolution—I think he’s leaning towards evolution, creating an algorithm that represents evolution in and of itself, as being the way to explore interesting novelties. For me, Picbreeder was a kind of supervised human imitation learning, so it was almost like a reflection of the constraints and the cognition that we have.

I lean externalist a little bit, so I think that a lot of semantics is about this embodied physical interaction with the world that, via osmosis, perhaps gets represented in our brains. But do you have a position on that? Do you think that just pure neural networks simulating the world could understand the world in the same way?

So, first, to talk about immersion, or potentially using these kinds of models to actually be immersed in them, I think we’re still very far away. I said before that I think the visual aspects are pretty much primary, right? We’re generating pixels, and with Genie 3 we also added audio, but our embodied existence is so much more than that.

I think sometimes that gets lost, right? Because, eventually, as people, we feel a lot, we walk around, and we have other senses. We have this sense of where I am right now and, of course, the physical interaction, which is also applicable to robots, right? There’s still a large gap between where we are right now and building a real, full simulation of the world that can actually provide all of the information to an embodied agent.

So I think there is definitely a gap there that is interesting, but it does show that we’re still very far in that regard. As Jack said, basically building those kinds of experiences—we do see people try to build experiences together and explore worlds together—and I think that’s a very interesting direction for us.

Yes. Yes. So many things to talk about there. I suppose one important step is this multi-agent simulation thing, right? Quite a few people have spoken about this, certainly David Krakauer. He said that a lot of emergent intelligence is about coarse-graining, when you have these systems that can, through a variety of tricks, accumulate information over time.

Eventually, we developed a nervous system, culture, and language, and that allowed us to accumulate information, sort of transcending the hardware—the DNA evolution speed. So it’s evolution at light speed. Max Bennett spoke about that in A Brief History of Intelligence, how a lot of the evolution of the brain in culture was about the propagation of information without needing to have direct physical experience.

So we can implicitly share simulations with each other. When we start to build these multi-agent simulations, do you think similar things might emerge where, almost irrespective of the lifespan of an individual agent, the system could accumulate information and develop forms of agency and dynamics that simple systems couldn’t?

That’s a really good question. The way I would see it from the standpoint of Genie 3, where it is right now, is that it’s sort of a multi-agent world, but that’s only controllable in a single-agent setting, right? A lot of the multi-agent nature of the world is baked into the simulation around you. They’re almost like additional characters in the world rather than controllable agents.

You can control them if you wanted to with world events, right? You could actually control what the other agents are doing, but otherwise it’s always implicit in the weights. What you see is that there is some sort of natural behavior from them. If you walk through a crowd, people will move out of the way, for instance. If you create a driving world, then when you drive around, the other cars move in a sensible fashion.

To go back to your actual question, I think you’re saying the system can almost bootstrap from itself and learn across the different agents in the system. I think the way I would see it right now is more that the model’s knowledge of human behaviors can distill into the egocentric agent. That’s actually something quite powerful that we haven’t really got with any other simulation tool, right?

Because if the other agents are moving around in a way that we do, then I think it might even be a way for our embodied agents to learn things like theory of mind. For instance, if you’re walking around in a world and, say, go to cross a street, you check the cues of the drivers. Maybe there’s not a crosswalk, and you need to know when to stop. You can see that they’re slowing down, so that’s when you would go, and the other agents should be simulated in that fashion.

Actually, you can learn these kinds of cues that you can’t really learn any other way, other than being deployed in the real world. That obviously has safety risks and probably wouldn’t be an advisable thing to do with an agent that’s learning from its own experience. What we think with this kind of model is that agents can really learn these social cues, things like theory of mind, and how to operate with human-like other agents.

But it’s not the case that the model itself is then learning back from the agent that’s collecting experience. That might be a future step, but it’s not something we’ve really considered in this work yet.

Yeah, it’s fascinating. Shlomi, what do you think about that? Certainly, we use tools. We have search, GPSs, computers, calculators, and all these different things. Do you think about the locus of intelligence being in our brains, or do you think if we built rich multi-agent systems—or maybe even if we look at humans and LLMs now—where do you think the locus of intelligence in that system is?

I think there are different types of intelligence, eventually. As we make progress towards understanding intelligence and building intelligence, we end up building separate models that can accomplish different tasks along different dimensions of intelligence.

As I said before, if you really think about it, generating and simulating a world is not necessarily something that a person can do, right? Some people say, “Okay, we have a world model,” but definitely we don’t have the same world model or an ability like Veo or Genie 3. We cannot really simulate—if you tell me a sequence of events, I won’t output pixels. I can maybe imagine at a lower level of detail what would happen if, for example, you got up or if something happened in the environment, and I can plan accordingly.

So I think it’s not completely parallel. We cannot just say that these models are exactly how we operate. But I think what we do see is that some capabilities that, a few years ago, if you had come to me and said, “We’ll be able to generate videos from text,” I would have said, “Okay, I don’t think that’s going to happen in a few years.” But it did happen.

Other things that people thought were going to happen way before, like maybe self-driving cars, haven’t happened as fast as people thought. Now we have much better progress towards that. So I think, in this case, different types of intelligence made progress in different ways.

What I’m really interested in is seeing how those types of intelligence can work together. For example, if we have a model that can simulate the world at a different level than was possible before, and we have other models—for example, Gemini—that are able to reason about the world in a different, perhaps less visual way, what would happen when we bring them together?

The examples that we’ve demonstrated of the SIMA agent interacting with Genie are 2 separate models trained completely separately, but when they’re put together, they can accomplish maybe a new thing. So I’m really excited about that.

Yeah, that’s amazing. There’s also this notion that it’s 720p. Genie 3 was just creating these immersive— and I use the word “immersive” intentionally because, as a video editor, I know that it’s all a bit of an illusion.

So you are trying to create a creative artifact that is just beyond the predictive horizon of the consumer, and then they suspend disbelief. Right? In a sense, we are cognitively bounded as observers. We see the world macroscopically. We see chairs; we don't see particles. The world can have descriptions at different levels.

In some of the examples we showed, there's the one where you're controlling this drone by a lake, and there are some trees and very beautiful scenery. You do notice in that one that when you focus your view in different areas, it definitely hones in on detail more. I think the model kind of learns that sometimes you don't need all this detail, right? Actually, it should focus its efforts with the focus of the agent.

I think this comes a bit from our emphasis on this model being agent-centric, sort of egocentric often, but also able to do third person—a model that really feels like it's your view of the world, right? That's in contrast to Veo videos, which are much more cinematic in quality. The whole video is very high quality, whereas Genie 3 often feels much more like your own personal view of the world, which I think is quite a different experience. It does have these different levels of detail to it.

How do you think about that? I mean, it's so difficult for us to know how these inscrutable models work, but do you intuit that it's simulating the world at multiple levels of resolution? If you zoom in on something, does it have a different description, or is it limited in some way? How do you think about that?

That's a really interesting question and way of thinking about it. When I first saw video models and the simulation of, for example, fluid dynamics and other aspects of reality, I was like, how is it even possible to do that in so little time or compute compared to actually running the entire simulation? I think that's the first surprising aspect of these models, but it does come with some limitations.

What we basically see is that the models somehow find ways to simulate, as you said, in a way that looks good and reasonably realistic. We see it with video models, but as they get better, those approximations become even better. Maybe that's a good opportunity to think about the difference when we simulate the environment in an interactive way, because that becomes much harder.

If, for example, you want to just spill water—like a video of someone spilling water on some surface—if the model is a video model, it can just think about it, try it, and generate the entire video end to end. The past and future can be modified at the same time, and eventually you get some video that maybe looks real. But with Genie 3, because it's an interactive model, the user or the agent that controls it can decide to intervene. They can look from a different angle, and we have to create the entire simulation frame by frame in a causal way.

That makes the problem much harder for the model. Basically, it cannot change the past, right? Once the past happened, you cannot change it, like in the real world. I think that's where we hope to see better physical simulation, but it also makes it much more challenging.

So, to your question about different levels of reality, for example, would it work if I just zoomed in and looked at the molecules? I think this highlights the amount of computation that actually happens in the real world. If we actually had to simulate it completely, that would probably be impossible. Some models find ways to approximate it to a certain degree that looks reasonable to the observer, which is us, basically.

So another interesting thing is that we do this thing called thinking, and we know that neural networks are roughly computationally limited. They can be trained to do a certain amount of computation in a certain amount of time. That means we can do lots of things, but there might be certain types of things—for example, if you simulated someone solving a Rubik's Cube—you might find that, for whatever reason, it just doesn't have enough computation to do that thing.

Would there be an opportunity to create a variable-computation version where, for doing certain types of things, it could think more about them?

That's a really interesting question. Some folks on the team were also talking about this. For example, if in the future you wanted to be able to write code inside the model, at that point maybe it would require some different approaches. We already have models that can write very good code, quite widely available now. We also have models that can win a gold medal at the IMO, for example.

Maybe eventually you want to be able to do this inside the simulation, because that might be the next level: to really develop embodied agents that can blend these two different tasks, physical tasks and thinking-based tasks. At a certain point, I think we will probably need to cross that gap.

For now, I think we probably focus much more on visual quality and physical simulation rather than math- and code-type problems, which typically have more thinking-style models. But I definitely think it is an interesting question.

I also think it's something where the model definitely has this physical knowledge in it, but I don't know if the model itself could describe it. It probably just has it implicitly in the weights. Another agent could probably learn about the physical world from the model, but the model doesn't necessarily know it and can't tell you about it. It just sort of implicitly has that in the weights somewhere.

There's this interesting duality, in a sense, which goes back to the agent-environment idea. I think it's much more our belief that, right now, this is a nice setup: to have models that can focus on different strengths, simulating the future versus thinking and understanding the present.

Yes. What's your philosophy on that, Shlomi? In a way, you've built something that's even higher resolution than a language model. In principle, all the things a language model could do, as you were just saying, Jack, could kind of emerge from a model like this. Is your philosophy to build a massive model that does everything?

I'm typically thinking about this more from a practical point of view. There is definitely this kind of purist approach: we should have just one model to do everything. But a lot of the challenges with modern machine learning actually come from building these systems. There's a lot of engineering, software design, and hardware design involved in building them, training them, and running inference.

When we actually try to design those systems, there are a lot of constraints, and those constraints basically impose on us some ways in which we have to prioritize what we want the model to do. Especially for Genie 3, when we're bringing real-time capability, real time basically means that we have to generate frames very fast—multiple times per second—for the person or agent that interacts with it to feel like they can move around and feel the responsiveness of the model. That sets some constraints on how much capacity we actually have.

When it comes to your question of whether we can have one model encompass all of the aspects of intelligence we discussed before, I think it boils down to what set of requirements we have. If we don't care about real-time interaction, maybe we can do that. If we don't care about things like how expensive it is to run, maybe we can do that as well.

Ultimately, we're trying to build models that are not just a theoretical exercise. We hope to bring them, like other models, to people to use and to advance actual applications. That's where we have to make those decisions. Ultimately, we pick the types of capabilities we want to emphasize.

Very cool, and 20-second answer, Jack. Is there a sim-to-real gap?

It depends on how you define it. I think that currently, sim-to-real is actually a bit of a conflated term. It's more sim-to-lab in what people currently do. I think sim-to-real can only really be achieved with a photorealistic world-simulation tool like Genie 3.

Yeah. So you think this is actually a big step in the direction of—

I think it's the only way to solve it: to actually get into the real world, where there are people and other agents in general moving around, rather than just a very constrained, lab-like situation that has real-world physics but nothing else that's real.

Amazing, guys. This has been an absolute honor. Thank you so much for coming on. And for folks at home, if you're developing on Unreal Engine, it might be time to, you know... Yeah. Anyway, cheers.