Why World Models Could Change Robotics, 3D, and Creativity

On the path to spatial intelligence, creating pixels that are truly spatially contextualized and grounded is the same super-complex step that Atlas took. Large language models are built on predicting the next token, and video models are built on predicting the next frame. Atlas is, in essence, forecasting a new perspective.

This is a real area where AI can bring huge benefits to people and their processes. We’re talking about cutting costs by 50 to 100 times.

There’s a famous shot in the first Matrix movie where Neo appears to be falling.

Exactly. They had hundreds of cameras for that green-screen angle. With Atlas, we can do this with just 3 cameras. There’s no studio filming, no green screen, and no expensive calibration. No one has ever seen results like this.

When you started this, did you know it would work?

I was pretty confident. Every time we scaled up the model and trained it longer, it got significantly better.

Does this mean we’ll get 4D videos? Will I be able to walk around?

1. What Atlas Is & Why It Matters

Yesterday was a big day. You launched a new cutting-edge model that received a great reception that continues to this day. I think it would be good to structure our conversation like this: let’s discuss what exactly was launched, and then go back through the history and trace the path of development.

So, Justin, can you tell us what was launched yesterday and why it’s important?

Yes. Atlas is our new next-generation world model. It has 3 main characteristics: it can generate, reconstruct, and simulate the world.

It has very high-quality generation under camera control. You can input an image along with a camera trajectory, control the model, and have it generate video frames from any desired perspective. It also performs sparse 3D reconstruction very well. You can load 1 or more—up to 100—frames that show views of the real world and use them to reconstruct that world.

This reconstruction can be presented as a new video of a flight through space or as a clear 3D reconstruction of that space. Finally, it can be used for simulation. We’re showcasing these incredible bullet-time videos that have been garnering a lot of attention online, as well as a robotic simulation.

What is a “bullet-time” video?

The bullet-time style of video comes from The Matrix. There was this famous shot in the first Matrix movie where Neo is falling—

Oh, yeah. That’s right.

You remember that famous moment: he’s falling, everything is in slow motion, and the camera is flying around.

This is how they made the shot: they set up a ring of hundreds of cameras. When he fell in the studio, hundreds of cameras filmed him from different angles against a green screen. They then used those hundreds of cameras to create that famous shot in The Matrix.

With Atlas, we can do it with just a few cameras—3 cameras. That means no studio filming, green screens, or expensive calibration. We can literally put 3 cameras—3 iPhones—on tripods and use them to film something happening, such as someone throwing a ball into a basket or dropping a strawberry into a bowl of milk.

From those 3 iPhone videos, we can recompose the shot and imagine that time has stopped while the camera flies past the splashing milk, creating these amazing “frozen-time” shots. We can even do this with only a pair of cameras.

Can you give us the simplest description of what Atlas does? What goes in, and what comes out?

One of the fundamental principles of Atlas—the most important one—is the anticipation of new perspectives. This is a fundamental primitive that we find extremely exciting: a completely new primitive for basic models that no one has done before.

We know that large language models are built on predicting the next token. We’ve seen that video models are built on predicting the next frame. Atlas is truly a prediction of new perspectives.

Given a certain number of views of a scene or its description, those inputs form what we call a spatial context, which implicitly describes the world we want to talk about. You can then point the virtual camera at any point in space and time, and Atlas will figure out what that world should look like from that position in space and time.

Ben, with all these video models claiming to be models of the world and creating new angles, could you explain more specifically how this is different from the many models that have come before?

Yes, I think what Justin was talking about regarding the spatial-context aspect is extremely important here. There are many video models, many of which have gained popularity by introducing a single image or interpolating between the initial and final frames. Now we’re starting to see models that can do a kind of omni-reference with 20, 30, or 50 images.

The main thing about Atlas is that it has a spatially based meaning for every frame you load into it. This is not just an image that the model will interpret however it wants, or that you try to negotiate with using text prompts to get it to do something specific.

In Atlas, each image actually has a corresponding 3D camera pose. That means you can perform the reconstruction task with an extremely high degree of accuracy. If we have 4 views of this room, 1 from each corner, you can feed them into the model and get an exact copy of everything you see in this room. It won’t have to guess what’s in the other corner or what the relationships are between objects. It will simply reproduce exactly what you gave it.

You can also do this in a creative or imaginative sense. If you take 2 photos from different AI generations or real-world locations, you can place them to build targeted camera passes guided by exactly where you placed the desired content and where the camera will look and move.

In my opinion, this is very different from the “slot machine” effect of having to redo the generation over and over again with only a high level of textual control, as in video models.

2. Is This a Scaled-Up Video Model or a New Architecture?

Is this just an obvious, scaled-up version of the traditional video model, or is it a new architecture?

I think this is a fairly new thing for a few reasons. One of them is what we talked about: it performs both generation and reconstruction together within the same model.

As Ben said, this system can take multiple angles of this room and then reconstruct everything in the room exactly as you see it. Historically, reconstruction was a separate subfield of computer vision, with its own specialized tasks and specialized models.

Generation is something that all text-to-video models do very well, including all the big diffusion models we’ve seen in recent years. They’re great for creative tasks, when I want to imagine something that has never existed before.

But with Atlas, we’re bringing these 2 different parts of visual intelligence together for the first time in a single model. It can perform both 3D reconstruction and generation within the same architecture.

To do that, we had to make a few changes. First, we had to make it multimodal from the start. The system works natively with text, images, and video. It also works with camera positions as native inputs to the model, which I don’t think anyone has done before in the pretraining phase.

It uses 3D as the native modality it works with. The system was designed from the beginning to be natively multimodal, and I don’t think anyone else has done this.

Sorry, I’m just not very good at this. By 3D, do you mean depth maps, or something else? What does this mean?

The wording we’ve been using so far is depth maps. You can have a frame where a virtual camera indicates its position in 3D space, and that camera position and its parameters are input to the model.

Along with this camera position, you can add RGB data that shows what this point in space looks like, as well as a depth map that defines the spatial structure of this position in 3D. Text, images, video, and 3D cameras are the modalities that this system processes together in a multimodal way.

I want to add that what Justin just said is extremely important, and what Ben talked about is also underrated. This is the first time we’ve combined pixel generation and pixel reconstruction.

In the world of computer vision, this field has existed for over half a century. Sitting here and having worked in this field for decades, I can’t tell you how many PhD theses have been written on reconstruction or the synthesis of new perspectives. Our field has traditionally had several directions.

You go to a computer vision conference, and there’s a pixel-generation section, a recognition section, and a 3D-reconstruction section. This is an elegant model that combines or unifies the problems of reconstruction and generation based on viewpoint and perspective estimation. It’s incredibly powerful.

3. Spatial Intelligence & Why New View Prediction Matters

Can we take a step back? When you started the company, I remember you said you wanted to do spatial intelligence. Now we have this new model, and to me, as a layman, it seems very general.

You have a prediction for the next frame, and this is a prediction of a new view. You can get 1 view from a set of views, and you get a new view. Can you explain how this is a significant step toward the general problem of spatial intelligence, perhaps starting by describing what spatial intelligence is?

Spatial intelligence should ultimately enable us to generate space, reason within it, and edit and interact within it. It’s debatable whether that means 3D or 4D. Ultimately, this is 4D, with a temporal dimension, but even 3D represents fundamental tasks that spatial intelligence must perform or provide.

And then we talk about being able to render, simulate, and plan actions with this. But to do this, the fundamental problem that must be solved is understanding the geometry, structure, and physics of space. I really believe that Atlas is a significant step forward because, with every frame, you can generate an estimate of important information—namely, the point of view, or camera pose. This is the most important information needed to understand the geometry of space.

This can lead to all those emergent patterns of behavior that we see in the model results we showed in the blog.

So, on the path to spatial intelligence, generating pixels is certainly an early step that we've seen in, as you say, billions of models. But generating pixels that are truly spatially contextualized and grounded is a completely different and important step. This is the same extremely complex step that Atlas took.

We definitely have more to say, but we can just move on, right? It's as if there's a fourth dimension of time that will add dynamics, as well as higher-fidelity simulation and better spatial detail. This is part of the spatial intelligence roadmap.

Great. I would really like to delve into what this is all about, but first, let's talk about how we got here. How long has World Labs been around?

Two and a half years.

And you've actually released models before. So why don't you just jump straight to Atlas?

Great question. This is real magic, right? Justin's team needs a lot of chips.

Yes, you need a lot of GPUs to scale this thing.

Last year, we released our Marble world model. This was the first big, serious model of the world that we presented. It powers our current product, Marble.

Marble is really cool.

Marble can take images, videos, and text prompts and use them to create 3D worlds. But one of the biggest differences between Marble and Atlas is the output modality. Marble focused on Gaussian splats as a form of output representation. Whatever you input, you get a 3D world represented as Gaussian splats.

And Gaussian splats are very useful, aren't they?

They are really cool. They're easy to render, they can be deployed effectively on mobile devices and VR headsets, and they can interact with game engines and simulation engines. Gaussian splats have many advantages.

But this was a kind of bottleneck in the previous Marble model. With Atlas, we reworked the system and realized that we needed to separate these modalities earlier so that they worked more uniformly within the model. In Atlas, the fundamental primitive is not creating a world of Gaussian splats. The fundamental primitive is, as we have already said, the prediction of a new view.

This allows us to generate RGB frames and generate 3D, and we can use them to create beautiful Gaussian-splat worlds when needed. But we don't need to limit our results to Gaussian splats when there's no need to. It took a lot of sweat, blood, and tears to understand all the advantages and disadvantages of these different representations.

This is just one part. The other part is that you have to climb the scaling ladder. You need to gradually move up, conduct smaller experiments, and create smaller models to build confidence in what will work and what can scale.

If you could instantly know what would scale, you would just do it. But when we started the company, the world was completely different. For example, there was no scaling law for spatial intelligence.

When we started the company, the world was completely different and technology was at a completely different level. We had a lot of ambition about where we wanted to go, but it took us a few iterations to get to a model that we thought was really the same thing that could scale.

Ben, as the creator of NeRF, who has worked extensively with 3D and reconstruction, it's not obvious to me that having multiple angles will necessarily result in a 3D object. But you clearly showed how to get this 3D object. So maybe tell me a little bit about this stage.

As you said, I've spent many years of my career, most of it, working on creating 3D objects from images. This is something we talked about a lot in the early stages of the company: What approach would we take to create 3D? Would we synthesize multiple views and then build 3D from them, or would we try to move directly into a 3D direction?

There has been a lot of uncertainty in the field about which of these approaches will ultimately win out or produce the best early results. But I had a lot of confidence just seeing the power of what I would almost call coarse scaling—scaling to a very, very small scale, not like real-world modeling, but the dense-reconstruction scaling that we've seen over the past 3 years.

So basically, we released this dense reconstruction—

Dense?

Yes, dense. I'll just say dense because I know we're going to be talking about sparse, and I want to make sure people understand what dense is and what sparse is.

I think that, even from a technical and commercial perspective, was one of the challenges to creating a product based on 3D-reconstruction technology at a fundamental level. People don't think about it in the usual sense. They think, “I took 3 photos of this object or 6 photos of this room. I look at the photos, and I can understand in my imagination how they fit together. I can somehow fill in the gaps and understand it.”

But there has never been a real alignment between this a priori knowledge based on data and the kind of crude dense reconstruction we've been doing, which is actually much closer to scientific or medical imaging. You actually need to tell me that, for everything I want to see in this reconstruction, I need at least 3 or 4 angles.

If you think about it, even just in this room, there's space under the microphone, under the table, between every crack and crevice, and among the leaves of the plants. To get an image that covers each of these points requires a very exhausting effort to go around the room. I think you've all seen me running around different places filming them.

For someone who is well trained, it can take minutes. But if you give a regular consumer—or even a professional trying this for the first time—a regular cellphone camera or other shooting device, it will probably take them an hour. I saw someone try to scan a multi-room space for the first time and spend about 2 hours walking around it to get enough coverage. It's a very, very exhausting and boring cycle.

When we say dense, we really mean dense. For this room, I want 100 photos—many photos. I need somewhere between 100 and 300 photos of this room to capture it. We are trying to reduce this number to 3.

We're talking about a 50–100× reduction, and at that scale it completely turns the idea of what kind of data can be reconstructed upside down. You can go back to existing images, find something online, and even create scenes based on them. You can take regular videos and find a lot of material from the past that we would never use, and potentially breathe life into it in 3D.

This is something we experimented with a lot at Atlas. We take old clips—I took a bunch of my old recordings that didn't work before, ran them through the system, and saw a normal reconstruction for the first time. Or I took old recordings, deleted 95% of the photos, and got views that I would never have achieved with traditional reconstruction like NeRF or splatting.

One thing that gets overlooked on the demo site is the Stanford demo, where Ben showed that with 3 to 25 images, you can reconstruct the entire Stanford courtyard. But we had to show it from the air, even though every input image is Ben standing on the ground and taking photos from ground level. Everything you see is generated, but according to the laws of reconstruction. This is truly magic.

This is where generation and reconstruction must interact in a fundamental way to solve this problem. In the classic reconstruction that Ben was talking about, the reason so many angles are needed is that you need multiple images to triangulate a point in 3D space and see it from many perspectives. That is a prerequisite in the traditional version.

On the other hand, anything that wasn't captured in those angles—any pixel not visible in the input images—will be a hole in the 3D reconstruction. Fundamentally, if something wasn't visible in the input frames, you need to imagine it to fill in the gaps. That is, in essence, a generative process.

Even in this room, if we give Ben a DSLR and let him take hundreds of pictures of the room, even the world's expert in such detailed photography will still miss certain places. He won't be able to see under all the microphones or under all the tables or between the legs of chairs. You'll always miss something, no matter how many shots you take.

Therefore, generation is needed here as another mechanism in the model, because you can never get everything. It takes some generative ability for a model to think, “Based on what I see, I can first triangulate what's visible and then fill in the gaps with what inevitably failed to be captured.”

Yes, and there's something extremely cool about that, because large language models figured that out a long time ago, right? There were some kind of context wars when, for the first few years, we went from 128 to 256 and then to 512 tokens. Now we have 1 million of them, right? Everyone now understands on a very tangible level the value of increasing context when you use models for programming.

This is a difficult task. Everyone understands this, but no one has yet exploited it in image and video models in the same principled way. No one is trying to download an hour-long video and find a “needle in a haystack,” such as the frame at minute 37. In the case of reconstruction and generation, you have the same thing: reconstruction is just generation with a very long context, where you put in a lot of data.

This creates a continuum where you combine these 2 things. And Atlas creates the opportunity to do things we could never do with Marble. Marble had a fundamental limitation: you couldn’t cram more than, frankly, a few images into it. But with Atlas, I can take 64 shots and do a flyby of an entire house, where everything is based on what was seen, almost seen, or slightly extrapolated from what isn’t there.

I can take the snapshots I took of 2,000 images of a house with many rooms, reduce them to 30 or 40 inputs, and the flyby looks pretty much the same. This was simply completely unthinkable before. All of this became possible thanks to the creation of a spatially scalable context window where you can drop in data.

So should we think of it as sparse—the photos that you physically took—and then Atlas, as a model, creates the rest of the scene, and you use classical reconstruction methods? Is that how you should think about it?

In a sense, yes. That’s the beauty of Atlas. You can take any number of input views, down to 1, and then use Atlas as a sort of rendering engine to create whatever you want. You can move it around like a virtual camera.

You can just say, “Okay, I have a photo here. I want a photo there, there, and there.” You can do a few of these and then tell it to do a close flyby. You can do this sequentially; it’s an autoregressive model. It’s up to you. You choose what you add to the context interactively during generation.

What amazes me is that I have a very simple mental model: I have 4 photos, and I need the model to extrapolate from them and then have everything come together in the reconstruction. It has to be 3D, and I always find these diffusion models visually great but not accurate.

I don’t even know if there’s a question here, but how is it that the room fits? How is it that it is 3D-consistent? Is it just a large amount of data?

Yes, part of it is belief in the scaling hypothesis.

4. Did You Know It Was Going to Work?

By the way, I have to ask: when you started doing this, did you know it would work?

I was pretty confident.

Were you sure?

I think all 3 of us had complete conviction in the law of scaling.

Yes, I think so. I truly believe that the precise choice of architecture and data mix is where the devil is in the details.

I watched Justin and his team go from, “We really don’t know how long this will take,” to, “Oh, maybe there are signs of life,” to, “Well, it will work.” Nobody had done it, but I think the hypotheses were, first, that you had to get all the data, and second, that you had to predict the next viewpoint. We were convinced of these 2 things from the very beginning.

I was very confident that it would work. I wasn’t sure it would work so well and so quickly. I thought there was a chance we would do it, but it may not have been obvious that the first cycle of pretraining a new model with a new architecture and a new paradigm would work. The fact that the first cycle worked is simply incredible. I thought there was a chance we would have to do a few more iterations of the pretraining cycle before we got the desired level of quality.

Are we already at the point of scaling for this architectural approach, or do we need another breakthrough?

No, no. We’re at the very beginning.

Really? Without changing the architecture?

Yes, I think we’re at the very beginning. At the moment, we’re only limited by computing power. Data is very important, as Fei-Fei likes to point out, but everything has a bottleneck, and I think the main limitation to further scaling is precisely the computational resources for training.

During development, we trained a sequence of models. We blogged about this a bit, but we trained a few models that were the first rungs on the scaling ladder. Every time we made the model bigger, trained it longer, and ran it on more chips, it got significantly better.

The size of the model that we showed in the blog is obviously the biggest and best that we’ve trained, but it wasn’t limited by scale or data or anything like that. We had a release deadline, so we limited ourselves to what we managed to teach before that deadline.

Here’s a little insider information. Justin and the team train from smaller to slightly larger models, and they have a development plan. One summer day, early in the summer, it wasn’t even the current Atlas model yet—it was a smaller model. Ben fed the data into novel-view generation.

Remember that famous table, the garden table, from the NeRF paper and many other works? That night, I got a message on Slack. We all saw Ben’s message about our camera flying under the table.

Yes, with a soccer ball.

It was real. That morning, the 3 of us looked each other in the eye and said, “This is it. This is what we’re going to build.” We made the decision in literally 5 seconds. No one had ever seen a result like this before.

5. Use Cases: Creatives, Games & Robotics

Ben, can you be a little more specific about the use cases? World Labs has historically had many creative users who used Marble for consistency, to create 2D images, movies, 3D objects, games, and more. Maybe you can talk about how Atlas expands or complements those existing use cases, and then I’ll talk about robotics.

Yes, of course. It’s funny enough that one of the main ways people have used Marble fits perfectly into this new view-prediction scenario. Many of our Marble users—Marble is our previous product—would upload an image, get a full 3D scene in the form of a Gaussian splat mesh, take a few screenshots from different vantage points, and leave.

We thought, “We can just generate these images if that’s the data we need, right?” There was a lot of loss of quality there. People would say, “This Gaussian splat could look better.” We decided, “Okay, what if we just model these viewpoints generatively with that same level of control?”

This key possibility of view synthesis has been an academic problem for a long time, in the sense of, “You’re going to do a really tight capture.” But generative synthesis of appearance is a relatively new task.

We see so many people who have multistep workflows in their creative process. I don’t think there’s anyone who uses 1 monolithic model, even Seedance or something else, to accomplish their holistic task. People usually have a bunch of storyboards and mood boards with images that they take from their favorite model collections to generate images. They then turn to various video tools to combine them into keyframes, and then they edit and re-edit them.

We saw a niche but very specific use case for Marble: as a way to ensure that your generation is based on a certain 3D-consistent world. I’ve struggled with different models myself, trying to ask them, “Give me different angles of the ruins.” Every time, you could see, “Oh, they’ve shifted somehow.” It’s unstable, and even this one use case suggests value lurking beneath the surface.

After all, for decades, people have become accustomed to a stable 3D state—a virtual simulation of what they do in the real world, with a stage, props, and elements—whether for movies, shows, commercials, or creating game environments. This consistency and resilience are key to how humans think in spatial categories and develop environments over time.

People don’t think ephemerally, like, “Generated, generated, discarded,” with only text queries left. People want to create collections of assets and model the world that way. We’re trying to provide, through this spatial context mechanism and other means, that level of control and precision, along with the ability to accept different input modalities, starting with images.

We want to give people more control over the elements of the scenes that they view, edit, and interact with in the future. I think this opens up further use cases in the areas that we’re already seeing, and it also extends to any areas where people want to create a virtual replica or a previsualization of the real space that they need to build.

That includes architecture and construction. I once talked to a guy who built conference booths. There are so many things in the world that we don’t think about but that need to be made, and each of them goes through a painstaking phase of virtual design. In this process, part of the work in 3D programs is perhaps the most exhausting and time-consuming part today.

For example, taking feedback on a 3D design—whether verbally, from sketches, or from very quick sketches by a creative director, designer, or architect—and translating that into a 3D model is 95% of the job. You hold a meeting, get feedback, and then spend a week making edits.

That’s because our software is already decades old, and it’s never been as intuitive as playing with LEGO, modeling with clay, or drawing with a pencil. This is where AI can really benefit people in their work, whether it’s in the creative field, industrial design, or anything else. This really motivates me to create different versions of our model to meet the needs of such people.

Yes, I understand how it helps creatives, as it did with Marble, and how it extends to design or architecture.

But, Fei-Fei, you acquired a robotics company, so this—we were just talking about this two minutes ago, I know. However, it is less clear to me, especially in the context of Atlas, how this applies to robotics. Could you explain that a little more?

Yes, Atlas is actually a key piece of the puzzle. We acquired the company formerly known as Synapse. What is its key technology? Currently, its core technology is a system that works on the principle of transitioning from reality to simulation, and then from simulation to reality.

What does this mean in robotics? You want to train a robotic arm to, say, lay cables in an industrial environment. To do this, you need a lot of data to first train the robot-control algorithm to perform cable work. Then you need to evaluate how well this algorithm works. Finally, you deploy the robot into the environment where the cables are being laid.

What this company, and now our robotics team, has done before to teach robots is exactly what I was talking about: dense reconstruction. You take snapshots of the situation and then try to recreate that environment. It is extremely painful, long, and exhausting, and it really slows down the speed of robotics simulation—the transition from reality to simulation.

So Atlas is really the next-generation technology for this. This does not just apply to laying cables with robots or anything like that. We need to look beyond that and recognize that the biggest problem in robotics right now is actually data. Someday it will be chips, but now it is data.

It is very difficult to collect real data where robots are working, and not just collect data about, say, laying cables or washing dishes or whatever. There is also a very important step called randomization, where you take the same environment and randomize the conditions. The cable does not bend in just one way; it can bend differently. The box can have different sizes, colors, and covers, or be in different parts of the scene.

Therefore, you need to go through the process of moving from reality to simulation to get enough data, in addition to other data from the internet. This transition from reality to simulation will be greatly improved by Atlas. This is only the first part of meeting the needs of robotics in modern technology, because we do not yet have a sufficiently advanced foundation model that is reliable enough for robotics.

But Atlas is a new model. It is a multimodal model. It accepts different types of input data and generates different types of output. It is quite conceivable that the next step will be for the Atlas model to accept dynamic data. This can really start to bridge the gap between action planning in robotics and Atlas’s raw data.

So, this is a roadmap. What did you want to say?

What I meant to say is that policy learning for robotics is fundamentally different from any other application of AI we have seen before. If you generate a piece of code, an image, or a video, the model fundamentally creates that artifact. For that artifact, there are many examples that can be found online or elsewhere and collected.

You want to generate images—the world is full of images. You want to generate videos—there are plenty of them in the world. You want to create a codebase—there are plenty of codebases to learn from. Robotic policy is something fundamentally different.

It does not create some static object. Instead, it is a strategy that goes out into the world, acts, and tries to achieve a goal. But the world does not always react the way you expect. Something unexpected will happen. A robotic policy is essentially an agent that acts in the real world, interacts with it, and responds to what happens.

It is critical that during training, these policies are exposed to everything that could go wrong during deployment. This is where simulation plays a key role in robotics.

You can look at this from 2 sides. One is a kind of classic simulation: you can take your favorite physics engine and, as a human designer, try to creatively imagine all the scenarios that could arise when performing this task, and then try to write code that models them all.

That is 1 approach, and it is an interesting one, especially with coding agents actually getting quite a boost from it. But there is another approach: trying to create a data-driven simulation. Maybe we can create a trained model that understands how the environment, or the world, will respond to actions, and maybe sometimes it will respond in other unexpected ways.

Could we create these neural simulators, trained on as much data as possible, and then use them as a base for training robot policies? This is indeed a very interesting direction of development for the future.

But it does not end there, does it? Once you have this trained simulator, it is as if it already has, in its “mind,” an understanding of the world and how the world will respond to actions. So why could not the simulator itself become a player?

Yes. It is the same thing, and that is essentially the key thesis we have about world models and their universality. There are certain fundamental things that a model has to understand about creating worlds, modeling them, understanding what they look like in different situations, and understanding how the world will respond to an action. That is closely related to understanding what action needs to be taken to make the world respond in a certain way.

6. The Elephant in the Room: Video Models vs World Models

One piece of feedback I received—congratulations on the launch, by the way—was extremely positive. I think this was probably the most important model launch this year. Everyone was saying positive things, but 1 expert in the field whom I wrote to asked, “What do you think?” He said, “This is wonderful, even fantastic. It is impressive, but more dynamics are needed.”

So it seemed that, at least for robotics, but in general as well, it would be ideal to have a world that moved. Maybe talk a little bit about that, and also about any other future directions you can discuss that you think are worth discussing.

The dynamics will definitely appear. In fact, we already have the initial dynamics, and that is something that, in my opinion, people have not really appreciated. We have not emphasized this on the blog, but the previous Marble world model was fundamentally static. The model simply could not cope with any dynamics at all. That was built into the architecture of the model and into the training; it was all fundamentally static.

We already knew this was a big problem after Marble, and we already fixed it in Atlas. The Atlas architecture fundamentally supports dynamics, and the Atlas training data fundamentally contains dynamics. If you look closely at some of the videos that we posted—

I saw, I saw. You see a little man.

Yes, waves on the water. Some examples are waves on the water, and some generated aerial views show small cars moving. The dynamics are actually already in this model.

But is that—by the way, dynamics seem very problematic to me if you are trying to reconstruct 3D from multiple angles, right? Do those things contradict each other?

In fact, 1 of our theses is that if you are going to do fundamental 3D reconstruction, you do not really need dynamics. You want to be able to simulate exact scene views with precisely recorded time. But that was actually a problem with our previous approach with Marble.

You can try to find completely static data, but that is very difficult to scale and very difficult to obtain in large quantities. We realized that even if I want to get a static result at the output, the best way to achieve that is to actually show the model dynamics. Show the model as many dynamic objects and as many static ones as possible, and let it figure out how to filter out the dynamic ones.

In particular, in the pretraining of Atlas, it had already seen a lot of dynamics. The post-training that we did specifically for this checkpoint in this release was much more focused on statics—more on spatial movement rather than temporal movement.

But I am sure this pretrained checkpoint already contains a lot of hidden dynamics. This is something we will significantly improve in the future.

7. Will We Get 4D Video You Can Walk Around In?

So, Ben, does this mean we are going to get 4D video?

You will be able to walk around.

I can see the smiles on their faces. If you stopped now and only did “more,” “faster,” and “better,” you could build an entire industry. This seems like a very horizontal primitive to me. If you were to do nothing else, are there other things besides “more” and “faster” that excite you about your applications, which are usually more in the realm of 3D content creation?

Yes, I am very passionate about developing this multimodal aspect. I think different control regimes are extremely important here. In my opinion, it is very underrated, especially in the academic community, how critical it is to add control conditioning to these models to get what is inside.

Honestly, this dynamic-versus-static thing is—I am trying to understand what those words mean. I am trying to understand, in simple terms—sorry—editability.

I think editability is key. We are seeing this in models for individual images and, starting this year, in video models, where the ability to intuitively interpret queries like, “I want this person, this object, and this action to happen at the same time,” and combine all of that into 1 cohesive result without having to manually tweak the system, is starting to open up.

It is interpreted the way advanced image models do, right? In terms of editing.

But we haven't seen it spread as powerfully to video and then to world models yet, have we? We've only seen a few toy examples where you can type in a sentence and a dinosaur appears or something like that, using these real-time models.

But I want to bring this to an industrial level, because the main difficulty is adding control without degrading the quality of the model. Otherwise, it will just become entertainment. No one will seriously want to replace their advanced video model with yours if you provide additional settings but the image quality drops.

So I think the game is how to maintain the high bar of output quality that we set in the current model while adding all those interesting features that we're asked for: interaction with the scene, layout control, object identities, or temporal control. I think this is the direction that opens up a sea of interesting opportunities for working on the product and interface. The more complexity and richness you add, the more it allows you to completely rethink how people interact with stable 3D worlds in the computer.

This is the ultimate goal: to obtain all the necessary capabilities to build such a system.

Perfect. Is there anything you would like to add about the new functionality that would be interesting, beyond just “bigger and better”?

I think it's worth going back to the basics of intelligence for me. Intelligence is not just sitting still, observing, or interpreting something when it comes to space and physical space, right? It truly closes the loop between vision, experience, and interaction. So thinking about how to climb this ladder is exactly what Ben was talking about.

I think one interesting concept here is the concept of AI-completeness. Have you heard of this before?

Yes, yes, I've heard of it.

Aha. Well, like everyone. By the way, I hear about AI-completeness regarding LLMs: you have to be the smartest LLM to answer the question that the smartest LLM has to answer, or you have to solve the problem of general intelligence. This is essentially a connection to Turing completeness, right?

The idea is that if a problem is Turing complete, as in classical complexity theory, then I can take any class of any problems in that category and reduce them to this one problem, right? The 3-SAT problem is a classic example, isn't it? You can take any NP-hard problem and reduce it to 3-SAT. Therefore, you can use 3-SAT to solve any problem.

Yes, that's it. Yes, yes.

So the soft definition of AI-completeness is that there's a fundamental primitive that's an AI problem, but if I could solve this AI problem in its full breadth, I would solve any intellectual problem. The classic example for LLMs is predicting the next token. It's AI-complete because I could—there's a classic example, I think from Ilya, where there's a detective novel, and the system has to read the whole novel, and the final sentence is like, “And the murderer was…” Predict the next token. So you could, in principle, formulate any intellectual task in this way.

Yes, yes. But I think we're starting to understand—and Ben talked about this earlier today—that novel-view prediction, this primitive that we have in Atlas, especially generative novel-view prediction, is also AI-complete. Yes?

8. Why New View Prediction Is the Next Token Prediction

And because I could take something like a movie, watch all the footage of that movie, and then the killer comes out, and you predict exactly who comes out. I want to have a world where, for example, Martin writes on the board the proof of the Riemann hypothesis.

So if you look at it from an evolutionary perspective, this prediction of a new vantage point is exactly what evolution had to solve by making animals move. Nature gave animals eyes, but nature did not give trees eyes. Why? Because when you move, you see a new vantage point.

And this, regardless of whether you call it AI-complete or complete intelligence, is why we firmly believe that predicting the next viewpoint is equivalent to predicting the next token. Surprisingly.

Well, congratulations to all of you on the phenomenal launch of the model. We look forward to future model launches, and thank you for coming.

Thank you very much.