Google DeepMind Lead Researchers on Genie 3 & the Future of World-Building

All of the applications basically stem from the ability to generate a world just from a few words. You look at it and there’s a world generated in front of your eyes, and it’s amazing that it’s happening. I was very excited about how far we could push that.

And it’s at the point where a human who is not an expert will watch it and think it looks real, right? I think that’s pretty incredible.

Jack, Shlomi, Genie 3 has taken over the internet. We’re honored to have you on the podcast today. Has the response surprised you? Reflect a little bit on the reaction.

We weren’t sure how big it was going to be, but I definitely felt that we had something that had been a long time coming: being able to generate environments in real time. I think a lot of work that was done at Google DeepMind and elsewhere pointed in that direction, but we really wanted to make it happen.

Yeah.

Yeah, team, why don’t we reflect internally a little bit about what we found so game-changing about Genie 3 and why we’re so excited to have this conversation, Marco?

Yeah, for sure. I mean, first of all, it’s an amazing model. I think there’s a lot of excitement around the spatial memory and the consistency across all the frames. I think this is the first time I can see that you can have some sort of interactive way of doing this stuff with videos, because it used to be that you would do 1 prompt and have 15 seconds of video, but now you can actually have some sort of interactive element to it, which I think is very exciting.

Can you elaborate a little bit more on your insights on this? For example, how was figuring out what data you should collect? How do you make it very interactive while keeping the flow of the whole video, which I thought was phenomenal?

Sure. Yeah. So I think you highlighted a few capabilities: the length of the generation, the consistency of the world, and maybe diversity as well in the kinds of things you can generate.

I think the main thing is that last year, obviously, we made progress on quite a few different fronts in separate efforts. We had this Genie 2 project that was much more focused on 3D environments that it could generate, and it wasn’t super high quality. It felt like it was coming from Genie 1, but it wasn’t the same quality as things like Veo 2, which—the state-of-the-art video model at the time—came out in December, roughly at the same time. It came out a week later than Genie 2.

Obviously, internally, there was a lot of discussion between the 2 projects about the different directions we were pursuing. We had also worked on GameNGen, which is the Doom paper, as people know it, and I think you guys also wrote a nice piece on it straight after that came out. So I think that also attracted a lot of attention.

We felt that across these different projects, we had quite a lot of interesting things that would naturally combine, and we could basically take the most ambitious version of the combined project and see if it was possible. Fortunately, it was. I think the timeline is probably the bit that surprised many of us, because obviously we set ourselves these goals and tried very hard to achieve them, but you can never be totally sure how it’s actually going to feel when you’ve got to that point.

I think it ended up being something that resonated with people a lot more than maybe we expected, but we were always believers.

Yeah, I’ll just add to this that I think the real-time component is really important. People don’t often experience it firsthand, but we really tried in the release to at least have a few trusted testers interact with it and also get a feel for it by adding these overlays that show what happens and how people can use the keyboard to control it.

I think there is something magical about the real-time aspect. I felt it for the first time when our model—the actual game-engine model—started working fast enough, and we were just like, “Oh my God, I can actually walk around.” It was a bit of an awe moment.

I think there’s something about when it responds immediately that is really magical. I think that sparked the imagination of many people when the Doom simulation came out, and here we really wanted to push it to somewhere we weren’t sure was going to work. So it was definitely at the edge of what’s possible, I think. That’s how we felt. We just said, “Yeah, let’s try and see if we can make it happen.”

I think you guys—I don’t know if this was on purpose or not—perfectly timed it when everyone on X and Reddit and everywhere was making those videos of characters walking through games, but they obviously weren’t interactive. They weren’t real-time. Then you guys came out with this release that was like, “Now this is an actual product,” and it blew folks away.

I’m curious, because you can imagine so many different applications for this, right? More controllable video generation, making it much easier to create games, even personal gaming where someone’s just creating their own world that they walk through, reinforcement learning environments for agents, robotics—are there any particular use cases that you’re most excited about?

I think all of the applications basically stem from the ability to generate a world just from a few words. When I started looking at video models, I think it was pretty early. I think one of the models was Imagine Video, which was the model by Google Research, but there were a lot of models that were very basic compared to what we have today. But the ability to simulate something like—you look at it and there’s a world generated in front of your eyes, and it’s amazing that it’s happening—and I think at this point I was very excited about how far we could push that.

I think Veo was one way to do it, and Genie is definitely another way to make it a bit more interactive. So I think all of the applications basically stem from this core capability. It can be entertainment, of course, as you said; it can be training agents, helping agents to reason about the world, or education.

I don’t think any particular application is more important than others. I think it’s really up to how developers in the future will build on top of that.

Yeah, I would give basically the same answer in the end, with a different journey to get there, right? I personally worked in reinforcement learning for a few years before starting the Genie project in 2022.

The motivation originally was that, in RL at the time, we had this problem where we’d say, “Which environment should we try and solve?” Once you’d already done Go—which people thought was years or decades away, and which was solved in 2016—we’d reached superhuman level, and then StarCraft 3 years later. That was not a particularly long time for something incrementally significant. By around 2021, it was a big question of what we should try and do with RL.

We know that the algorithms can learn superhuman capabilities if they have the right environment, but we don’t know what the environment would be. So we were working on designing our own ones with code, but then it seemed like the more promising path, when you had the first text-to-image models coming out, was: What if we just think long term? What’s the way to really unlock unlimited environments?

That being said, over the course of the project—and originally we started it, I guess, in 2022—it was very focused on that one application, but it seems quite clear now that this could have a big impact in all those other areas you mentioned.

I think it’s like language models in 2021: maybe you probably wouldn’t have guessed an IMO gold medal a few years later would come that fast. As a direct application of that technology, it was probably, “Oh, it can help me with my emails,” or whatever it was.

I think it’s really cool to build these kinds of new classes of foundation models and then see what people can imagine doing with them. And that’s one of the really exciting things about sharing the research preview, right? We got this kind of feedback, so we’re hoping a lot of these things can happen.

One of the things in the research preview post, Jack, that blew me away was this—and it wasn’t even your first GIF, I think, in the blog post. It was either the 2nd or 3rd. You had this visual of somebody painting the wall with a paintbrush, and then the character moves.

Yeah, the spatial memory—

Right, like out to a different part of the wall, paints—

And then moves back, and the original paint is still there.

And I didn’t believe it. There’s no way. Then I read it, and you’re right: it’s described as spatial memory. So the persistence part, for me—I’m not taking away from all the other stuff; the interactivity is amazing—but I think—

Broadly speaking, folks expected that at some point video generation, for example, would become real-time. When I saw the Genie 3 post, I was like, “Okay, they actually went and did it.” But the spatial memory and the persistence was when I sat up in my chair and thought, “How did that happen?”

Could you talk a little bit about when you discovered that as an emergent property, or was that a specific design goal? What’s the backstory on that? Because that feels like a big unlock.

Jack, why don't we start with you?

Yeah, that's a great question. I'll say a few things. The TL;DR is that it was totally planned for, but still incredibly surprising when it worked that well. That specific sample, when I saw it, was hard to believe. I actually wasn't sure that the model had generated it for a second. I had to watch it a few times, really check it, freeze the frames, look back, and check that it was the same.

Going back a few steps, obviously Genie 2 had some memory, right? This got kind of lost because Genie 2 came at a time when there were lots of announcements—very exciting announcements. I mean, Veo 2 only came a few days later. It was a busy time of year, and the main headline act was that we could generate new worlds at all, right? That was the thing we wanted to emphasize.

But it did have a few seconds of memory, and we had a couple of examples. I created a robot near a pyramid, looked away, looked back, and the pyramid was there, but it was kind of blurry. It wasn't perfect. Some other models around the same time, or more recently, didn't have this feature, right? People kind of indexed to that because they didn't notice the early signs of it in the Genie 2 work.

And then for Genie 3, we basically went much more ambitious on the same sort of approach. We made it a headline goal for ourselves: Can we make the memory be what it is? We said we wanted minute-plus memory, real time, and higher resolution, all in the same model. Those are kind of conflicting objectives, right? So we set ourselves this technical challenge. We said, if we target this, then it's just about feasible and it'll be pretty incredible.

And then you still don't know, obviously, whether it's going to pan out. So when you get to the end of the research—7 months later—and see the samples, it still is quite mind-blowing, to be honest. So, yeah, it's kind of planned for but still pretty cool and exciting when you see it, because at the end of the day, research projects aren't sure things, are they?

One thing we didn't want to do was build an explicit representation, right? There are definitely methods that are able to achieve consistency, and they do that through an explicit 3D representation—NeRFs, Gaussian splatting, and other methods that basically say, okay, if we know what the world looks like and use prior assumptions on how the world remains static, then we can build a representation of what you're looking at. That's great for some applications, but we didn't want to go down this path because we felt it was somewhat limiting.

I think we can definitely say that the model doesn't do that. It generates frame by frame, and we think this is really key for the generalization to actually work.

Every time someone interacts with it for the first time and tests it—they look away and then look back—I'm always holding my breath. Then it looks back and it's the same. I'm like, whoa. It's still really cool.

How long is this spatial memory? I don't know if you can talk about it. You mentioned a minute-plus, but is there some sort of measure that you have? Can you keep it for half an hour, or what is the limit on that?

There was no fundamental limitation, but the current design is limited to 1 minute of this type of memory.

Yeah, it's also a real-time trade-off for the model as well. We felt that, because of the breadth and the other capabilities, a minute was sufficient for this version. It's quite a significant leap.

One more question related to the difference between Genie 1 and Genie 3. In LLMs, for example, with DeepSeek-R1, they saw in this paper that the longer they keep it running, they suddenly see these interesting behaviors. The model will start reasoning or give a—like a—“Oh, I'm wrong on this. I should self-correct.”

Do you see anything in this scaling from 2 to 3? Do you see any sort of interesting behavior that you weren't expecting that suddenly just appeared by increasing the amount of data and the amount of compute?

Yeah, I would just say that there is a bit of an overall trend. Definitely, like many generative models, we see that improvements happen with scale, so I think that's not a secret. I don't know if we can—I don't think it's the same type of intelligence that an LLM has. I'm not sure if reasoning is the right term, but we do see some things.

It can infer that if you approach a door, it makes sense for the agent to maybe open it, so you might see that it's starting to do that, for example. There's some better world understanding that happens over time, and things just look better and more realistic. I think these are the trends that we've observed.

Yeah. From Genie 2 to Genie 3, I think the real-world capabilities really increased, right? On the physics side, some of the water simulations and the lighting are really breathtaking. I think we have this example of the storm on the blog, and that one is super cool.

It's at the point where a human who is not an expert will watch it and think it looks real, right? I think that's pretty incredible. Whereas with Genie 2, it kind of understands roughly what these things should do, but it's not real, right? You can look at it and clearly see that it's not completely photorealistic. So I think that's quite a big leap in quality on that side.

Yeah. One of the things that was really cool in all the examples was the water. It's sort of a great way to see: Does it understand what the world is and how objects interact? That example someone posted of the feet going in the puddle was amazing.

But then there was also that example of a cartoon character—it was more of an animated style—who was running across this kind of green patch of land and then ran into this blue, wavy thing that looked like water, and he started swimming, which I thought was really interesting. Were there particular things you had to do around that for the model to be able to understand how characters should interact in different environments and different styles?

What you're basically describing is the real breadth of different kinds of environments, terrains, worlds, and things like that: water, walking on sand versus going downhill in snow, and how the agent's interactions should differ given the terrain that it's in. I think that really is a property of the scale and breadth of training. This is very much an emergent thing, and I don't think there's anything really specific we do for this, right?

Again, you hope the model has learned this because it should have general world knowledge. It doesn't always work perfectly, but in general it's pretty good. For the skiing examples, you go fast when you go downhill, and then when you turn and try to go back uphill, it's very slow, if not at all possible. When you go into water, obviously you hope, as you said, that the agent will start swimming and splashing, and this does typically happen.

When you look down near a puddle, hopefully you're wearing Wellington boots. This kind of stuff does just make sense, and I think it feels pretty magical because it very much aligns with what you were thinking about the world and the models just generated it all. That's also one of the really exciting things, for sure.

Yeah. And on top of that, one trade-off that we typically have is that we want the model to do 2 things. We want the model to create the world in a way that looks consistent. So, as Jack said, if you walk in rain or in snow, then you're probably wearing boots.

But if we provide it with a different description, or if the prompt is saying something else, we want it to still follow the prompt. There is some tension here because some things are very unlikely. You might say, “I want to wear flip-flops and jump in the rain,” or whatever. Then the model still has to try and create something that is very unlikely, and that's where typically video models may find it more challenging.

That's where our models might find it more challenging, but it's still successful to a surprising degree in going into these low-probability areas. I think that's really, in a way, what we want, right? Many people don't want to just look at a video that looks like their own home—maybe this room—but something a bit more exciting. That's where I think this is the magic of the models: They can take you to places that maybe are not so likely to be in reality.

Text following is really amazing in this model. That does feel really magical. I think this is something that Veo does really well as well, right? Pretty much what you ask for. It's really well aligned with text, and we've had that with Genie 3.

You could describe very specific worlds and really arbitrary, silly things, and it pretty much works. We actually had this discussion because people were very disappointed to find out that the video I made of my dog actually was not my dog's photograph.

I just described her in text. And yeah, I don't know if that's a big secret, but it looked exactly like her. The model just kind of knows, right? I think that's pretty amazing.

So I think that's actually a really important capability that we didn't have with Genie 2 as well, right? Because we relied on image prompting. There was some transfer issue: You rely on an image to generate the image, and that often does look really good, but it's not necessarily a good image for starting the world. Whereas, going directly from text, you get the controllability of pretty much anything you want. Plus, it just naturally works because it's in the correct space for the model to do its thing. That's something really powerful.

And why is that, Jack? What do you think led to such a massive instruction-following, or text-adherence, gain? It's a pretty hard thing to do.

Well, our team had never really worked on this. Genie 1 and 2 both worked with image prompting. Obviously, for this next phase, we leveraged a lot of the research done internally on other projects.

Personnel-wise, Shlomi had obviously been co-leading the Veo project, and so we were able to build on a lot of other work and ideas internally. That basically allowed us to turbocharge progress, right? If we'd done this incrementally, building ourselves in isolation, it would have taken, I think, a lot longer than being part of Google DeepMind, where we have these teams that have a lot of knowledge in different areas that we can lean on and build on. I think that's super exciting about being in this company right now: We have so many experts in different areas that we can seek out for advice and help.

And Shlomi, a question for you on that: Having led the Veo 3 work, which is kind of mind-blowing, is there a reason why this is Genie 3 and not, like, Veo 3 real-time?

So I think it's definitely a bit different, right? Genie allows you to navigate the environment and then maybe take actions, right? That's not something Veo, at this point, can do. But there are other aspects that are different, that Genie doesn't have. Genie generally doesn't have audio, for example, right?

While there are potential similarities, we just think it's sufficiently different. Also, at this point, Genie 3 is not available as a product, and we do think about Veo 3 as a product that's kind of mainstream and has become very popular. I don't know what the future holds, but at this point we just felt it's sufficiently different in terms of what capabilities it has and how we think about it. So, Genie 3 is pretty much a research preview, right? It's not something we are releasing at this point.

You know, something we think about a lot is: What are the edges of a modality? We're talking about this all the time. The lines start blurring pretty quickly between real-time image and video, and then real-time video and interactive—whatever—world generation, world models. I don't think we have a good word for what Genie 3 is yet, but you guys called it a world model, which I think is a great term.

In your mind, where does the video-generation modality stop and real-time worlds take over? Do you think in the future these are converging into basically one modality, or, if you had to predict over the next few years, do you guys think these will diverge into completely different disciplines? It seems like they share one parent today, which is video generation, but where is the world going? Do you think these are 2 completely different fields?

From my perspective, they are different. I would say modality is one thing, right? We have text and audio; even within audio, there are different types of submodalities. Speech is not the same as music. We have different products for music generation, and we have other models for speech generation and speech understanding. Even within one modality, you can have different flavors.

Then, of course, you have video and other things. I think modality is one dimension, and another is how fast, or how quickly, we can create new samples. A completely orthogonal direction, or dimension, maybe, is how much control we have, right? I think we picked a specific direction, or a specific vector in the space, for Genie 3. Different products and different models can try to go in different directions. I think the space is pretty big, and there are a lot of trade-offs to be made.

Yeah, I don't know. I think it really depends. Some people believe there is 1 model that will do everything, but I think it's still open-ended what's the best way. We're in a place where engineering is a big part of our research, right? It's not a paper where we want to build something that people can actually use. So I think this really makes abstract ideas get you to some point, but to actually build things, you have to make some concrete decisions. I think it kind of forces you to decide what you want to do.

Yeah, I think this is a really interesting point, mate. Ultimately, it has to be driven by technical decisions and also the goals, right? If you look at the models right now, we obviously made a choice that we want Veo 3 and Genie 3 to be separate projects this year, right? If you look at them both as they are right now, they have very different capabilities that the other model does not have.

Technically, to combine all of that already into 1 model would be, I think, very challenging. I mean, Veo 3 is clearly a higher quality threshold than Genie 3, right? It has very different priorities, right? The natural thing is you could say, “Oh, well, what if we just took these together and combined them?” But that may not be the best next step for either of those 2 models, right? It may not be the case that the thing that the other one has is actually the most compelling thing for a completely different experience.

Given the breadth of interest in both models, there's actually quite a small set of people who are really actively using both, and they tend to be more folks like yourselves who are just more broadly interested in AI, rather than people with really downstream use cases. You mentioned agent training, for 1, which is a very high action frequency and requires more egocentric—or, I guess, more like—worlds where tasks can be achieved, but it doesn't require the high-quality, cinema-style videos you could generate with a Veo model. It's quite different. On the filmmaking element, I'm not so sure that Genie 3 is really there at this point, and that would necessarily be the goal.

I don't know. In filmmaking, Justine can do some pretty incredible things with the filmmaking tools today. You'd be surprised.

Give me access. I will make amazing films with Genie 3.

I guess that gets to one of my questions, though, which is: The work you guys are doing is incredible, and you probably have so much going on just to coordinate training these models and managing these teams. How much do you also have to think about what the downstream use cases of the model are when you're training it?

You could imagine a world in which you're just like, “We don't really know or care what people are going to do with it yet. We're just going to go in the research direction we think we should go and see what happens.” But based on how you guys are talking about it, it sounds like you've also been pretty thoughtful around what different capabilities or features are needed for different potential use cases, at least for different models.

Yeah, I'll say that we have some applications in mind, but that's not what's driving the research. It's more about: How far can we push in this particular direction? Can we make all of that work—really great quality, really fast generation, real-time, very controllable? I think that's kind of what drives us to develop Genie 3, and the applications kind of follow.

And I don't think, to be honest, that I know what the applications would be. I think we're very surprised. I'd like to mention Genie 3: People find new ways in which it can be useful and prompt it to have visual stuff. People just discover it, right? We didn't even think about it initially.

So I expect kind of the same thing, and I think that's why I'm excited for more people to be able to access it in the future. In general, our approach is to make sure that, over time, there is more access to the models we build, and I think that's the only way to discover what the real potential is.

I guess one question somewhere related to that is: How do you think going forward—Genie 4, 5, or any other models—what is top of mind right now? If you wanted, for example, to focus on—I don't know, it seems like gaming could be one of the applications—having multiplayer-type games where you have 2 separate memories or 2 different views, but at some point they merge.

How are you thinking going forward? What’s next? Is it scaling these models on more data and more compute? Is it creating this sort of multiverse where you have multiple players or multiple people looking at the same model and putting in different views? What’s top of mind for you guys?

Top of mind for the next few days might be a vacation. After that, maybe walking my dog in the real world.

You mentioned a bunch of really interesting things, to be honest. I think we’re still collecting a lot of feedback on this current model, and in general, we’re most interested in building the most capable models. We would hope to have an even broader impact in the future and really enable other teams to do cool things with it, both internally and externally.

For me, I started this with a very, very focused vision about AGI. I still think, honestly, that what I’m excited about for AGI is more embodied agents. I really believe this is the fastest path to getting these agents into the real world, and I think we made a big step toward that.

But I’m sometimes even more excited about applications I never thought of that come up when other people see the model. I think it’s this trade-off: obviously, you want to focus on some applications, but then you want to be open-minded about others. The real joy of building models like this is seeing all of these people who can be way more creative with them than I can.

There are all these really cool things that we can do, and I honestly can’t really tell you what the biggest application will be in 1 year. We’ll definitely be trying to build better models.

I’m really excited, but I think that, as impressive as the model may be, we’re still very far from actually simulating the world accurately and being able to put a person in there and let them do whatever they want. When I say “far,” it doesn’t mean it’s far in terms of calendar time, because we live in an accelerated timeline, but it feels like there’s more work to do to get there.

I imagine that once we can actually—whatever the form factor would be—step into this world and maybe tell it what we want to experience, there will be so many applications. Imagine, for example, someone who’s afraid of talking to people on a stage or in a podcast. They could simulate that. Or someone who’s afraid of spiders could maybe actually see themselves getting over that.

That’s just 1 example. My wife actually thought about it, so it’s not my idea. It all hinges on the ability to simulate the world and maybe put ourselves in it, maybe see ourselves from the side, and potentially have agents interacting with things. The realism, and really making it work in a way that’s similar to our world, is key.

I’m actually personally petrified of skiing, and the models are already quite good at that. So when things quieten down, I might spend some time on it, because I promised my wife that our children would grow up knowing how to ski. We’re getting close to the age where I have to live up to my promise, and I’m not sure if I want to do it yet.

So we have to improve the model for you, Jack, so you can actually get that in distribution.

I hope so.

We were just talking before we started about how we might see applications in robotics. Jack, you were talking about embodied AI, and now the limitation in robotics is the data, right? How much data you can collect. Now you can probably generate a lot of different scenes that you weren’t able to create before, purely from recording videos or something like that.

I think that’s another thing that’s pretty exciting. And congrats on the model. It’s phenomenal.

On the robotics application, there was a conversation I was listening to from Demis yesterday where he was talking about your guys’ work on Genie 3. He mentioned that there’s an agent—I think you guys call it SIMA, right?—which can then interact with the Genie agent.

As I was hearing him describe it, it was kind of breaking my mind. You had 1 SIMA agent asking the world, asking the Genie agent, to essentially create a real-time environment for it to interact in. That’s when I realized that the way you built it, it’s composable with other agents.

Can you talk a little bit about why that’s so important for robotics, like Marco was saying, and what the major limitations are today that we’d have to overcome as a space to make the rate of progress in robotics much faster than it is now?

We designed it to be an environment rather than an agent. Genie 3 is very much an environment model. We don’t see it as an agent itself that can think and act in the world. It’s more of a general-purpose simulator, in a sense, that can simulate experiences for agents.

We know that learning from experience is a really important paradigm for agents. That’s how we got AlphaGo: the agent learned by playing Go by itself, trying new things, and then learning from feedback with reinforcement learning. It learned to improve itself and actually discover new things. It discovered new moves, like move 37, that humans didn’t think was a worthwhile move, but AlphaGo learned that it was because it could experience and try things for itself.

In robotics, we have a paradigm right now where there are some data-driven approaches. You can collect data in a quite laborious way, but it looks real, so it looks like the downstream task, and there’s not so much of a mismatch between the 2 domains. Or you can learn in simulation, but robotic simulations—even the best ones, and we have some of the best ones at DeepMind, such as MuJoCo, which we work with—are still quite far away from the real world. So you have the sim-to-real gap.

Even the sim-to-real gap itself is kind of poorly named, because what people consider to be real in robotics is typically still a lab or some very constrained environment where you’ve got a bunch of spotlights on a robot and tons of researchers crowding around watching.

Really real, for me, is the ability to walk my dog when I’m too busy to hold the lead, cross the street, see someone who’s scared of dogs and know to go around them, see someone with a ball and change directions—all these challenging situations in the real world. Of course, you still have gripping and these other tasks, but you need to really discover your own behaviors from your own experience.

Doing that in physical, embodied worlds is super challenging because there are so many reasons why it’s difficult. First, it could be expensive to collect data in those settings. You’d have to keep moving the robot back to where it started every time it doesn’t do something right. It could also be unsafe. There are many reasons why we can’t really do learning from experience in the physical world.

So we do it in simulation, but what we think with Genie 3 is that it’s the best of both. You’re taking a real-world, data-driven approach, but then you’ve got the ability to learn in simulation. It combines the good parts of each of those, which is why I think it could be super powerful—not just, for example, for robots.

I really love the idea of not having to take my dog for the second walk when it rains a lot in London. That would be great.

As you can see, we built a model basically for Jack’s personal applications.

That’s what’s driving the project.

Well, clearly, dog owners are out there.

Yeah.

I’m just saying, clearly, Jack, it’s time to move to California.

Yeah.

Yeah, that’s the solution. Less rain—

Less lag.

I mean, I personally love California, but my wife’s not convinced. Sorry.

We’re convinced here.

Yeah.

Just to touch on a final point on the robotics part, robotics means more than vision. We need to be able to—and I think this is an important point—drive the decisions of the robot by looking around, but it still has to perform actuations, decide where to move, and decide how to respond to the environment.

There are definitely some gaps, but at the core of the problem, being able to reason about the environment is something that world models—general-purpose world models such as Genie 3—can really help with. Maybe with future research, we can bridge those gaps in physical understanding and actually get physical responses from the world, which is a very interesting direction to explore.

One last question from my side, and I don’t know if you can answer this, but is it going to become public? Can developers access it at some point, or is there some sort of plan for that?

As you can see, we’re very excited about having more people access it.

We definitely want to make it happen. There is no concrete timeline at the moment, but I'm sure once we have more to share, we will.

Awesome. One of the things I've been thinking about a lot is that we see, with every modality—maybe first LLMs, and then image, video, and audio—early glimmers of something really exciting in a project or research preview. Then there's a ton of data and compute, and researchers kind of pour themselves into the problem, and you hopefully see this sort of exponential progress until you eventually get to the point where you're out of data or the improvements don't come as easily. I'm wondering, where are we on that curve for world models?

That's a really good question. I actually have a super hand-wavy, somewhat swerving answer, right? And I think it's actually both. I think the current capabilities are already quite compelling. You could make the case that if what you wanted was a minute of photorealistic any-world generation with memory, that could actually be the end goal, right? And 2 or 3 years ago, I probably would have said that was a 5-year goal.

At that point, if you just wanted to improve that, I think you probably end up with this. The jump from Genie 2 to Genie 3 was absolutely massive, and it went from being kind of a cool bit of research that was showing signs of life to something that could already be very compelling. But I think there's a lot more that you can do with this, and Shlomi kind of referenced this himself, right? It's not the case that you're dropping yourself in the world, right? Actually being in the real world, for example, is quite different from that.

When you do take a minute to look away from a computer screen, it's quite a bit richer out there. And that's just for the real world; we also want this ability to generate completely new things, right? So I think we've got a huge gap to close with the new capabilities that we want to add. I think it's maybe a bit different from language models—or actually, maybe it is similar to language models.

With language models, there have been lots of new steps that have actually come on top, right, that maybe we didn't think were possible. We thought things were plateauing, and then a new idea came that made a significant change. And that has happened a couple of times in the past few years. So I think there are a few more of those left, for sure.

My final question for you guys is: Are we living in a simulation?

Oh yeah. I've thought about it a bit. I think if we live in a simulation, my take is that it doesn't run on our current hardware because it's analog, and it's continuous. All of the observations are continuous, and there is nothing like—maybe the quantum level is some limitation. If you wanted to go philosophical, you could say it's some kind of hardware limitation of the simulation we run on. So yeah, take it or leave it.

It's a great answer. Clearly, there's a lot of work for the TPU team to do.

Yeah, maybe quantum computing will actually be running our actual simulation. So yeah.

That's a great place to wrap. Shlomi, Jack, thank you so much for coming on the podcast.

Thank you guys for having us. Right, come on.