After LLMs: Spatial Intelligence and World Models — Fei-Fei Li & Justin Johnson, World Labs

I think the whole history of deep learning is, in some sense, the history of scaling up compute.

When I graduated from grad school, I really thought the rest of my entire career would be devoted to solving that single problem, which is that a lot of AI as a field, as a discipline, is inspired by human intelligence. We thought we were the first people doing it. It turned out that nature was also simultaneously doing it.

So Marble is basically one way of looking at it: it's a system, a generative model of 3D worlds. You can input things like text, an image, or multiple images, and it will generate a 3D world that matches those inputs.

While Marble is simultaneously a world model that's building toward this vision of spatial intelligence, it was also very intentionally designed to be something that people could find useful today. We're starting to see emerging use cases in gaming, VFX, and film, where I think there's a lot of really interesting stuff that Marble can do today as a product, while also setting a foundation for the grand world models that we want to build in the future.

Hey everyone, welcome to the Latent Space podcast. This is Alessio, founder of Colonel Labs, and I'm joined by Swyx, editor of Blade in Space, and we are so excited to be in the studio with Fei-Fei and Justin of World Labs. Welcome.

We're excited, too.

I nearly said Marble. [laughter]

Yeah, thanks for having us. I think there's a lot of interest in world models, and you've done a little bit of publicity around spatial intelligence and all that. I guess one part of the story that's a rare opportunity for you to tell is how you two came together to start building World Labs.

That's very easy because Justin was my former student. Justin came to my lab in—what year?

2012, actually. The quarter that I joined your lab was the same quarter that AlexNet came out.

Were you involved in the whole announcement drama, I guess?

No, no, not at all. But I was watching all the ImageNet excitement around AlexNet that quarter.

He was one of my very best students, and then he went on to have a very successful early career as a professor at the University of Michigan in Ann Arbor and then at Meta. When we started talking, I think it was more than 2 years ago, for sure, both of us had independently been looking at the development of the large models and thinking about what's beyond language models. This idea of building world models and spatial intelligence was really natural for us, so we started talking and decided that we should just put all our eggs in one basket and focus on solving this problem. We started World Labs together.

Yeah, pretty much. After seeing that ImageNet era during my PhD, I had the sense that the next decade of computer vision was going to be about getting AI out of the data center and out into the world. A lot of my interests post-PhD shifted into 3D vision, a little bit more into computer graphics, and more into generative modeling.

I thought I was drifting away from my adviser post-PhD, but when we reunited a couple of years later, it turned out she was thinking about very similar things.

If you think about AlexNet, the core pieces of it were obviously ImageNet, the move to GPUs, and neural networks. How do you think about the AlexNet-equivalent model for world models? In a way, it's an idea that's been out there, right? Yann LeCun is maybe the biggest, most prominent proponent of it.

What have you seen in the last 2 years that made you think, “Now is the time to do this”? What are the fundamental things that you want to build in terms of data and perhaps different types of algorithms or approaches to compute to make world models really come to life?

I think one factor is that there's a lot more data and compute generally available. The whole history of deep learning is, in some sense, the history of scaling up compute. AlexNet required this jump from CPUs to GPUs, but even from AlexNet to today, we're getting about 1000 times more performance per card than we had in the AlexNet days.

Now it's common to train models not just on 1 GPU, but on hundreds or thousands or tens of thousands or even more. The amount of compute that we can marshal today on a single model is about 1,000,000-fold more than we could have had even at the start of my PhD.

Language was one of the really interesting things that started to work quite well over the last couple of years. But as we think about moving toward visual data, spatial data, and world data, you just need to process a lot more. I think that's going to be a good way to soak up this new compute that's coming online more and more.

Does the model of having a public challenge still work, or should it be centralized inside of a lab?

I think open science is still important. AI, obviously, compared to the ImageNet days, has really evolved, right? That was such a niche computer science discipline; now it's just civilizational technology.

I'll give you an example. Recently, my Stanford lab announced an open dataset and benchmark called BEHAVIOR, which is for benchmarking robotic learning in simulated environments. That's a very clear effort to keep up this open-science model, especially in academia.

But I think it's important to recognize that the ecosystem is a mixture. A lot of the very focused work in industry is seeing the daylight in the form of products rather than an open challenge per se.

That's just a matter of funding and the business model: you have to see some return on investment from it. I think it's just a matter of the diversity of the ecosystem. Even during the so-called AlexNet and ImageNet era, there were closed models, proprietary models, and open models.

If you think about iOS versus Android, there are different business models. I wouldn't say it's just a matter of funding per se; it's just how the market is. There are different plays.

Do you feel like you could redo ImageNet today with the commercial pressure that some of these labs have? To me, that's the biggest question: what can you open versus what should you keep inside?

If I put myself in your shoes, you've raised a lot of money and you're building all of this. If you had the best dataset for this, what incentives do you really have to publish it? It feels like the people at the labs are getting more and more pulled in, and people in PhD programs are getting pulled earlier and earlier into these labs.

I'm curious if you think there's an issue right now with how much money has taken over and how much pressure it puts on the academic open-research space, or if you feel like that's not really a concern.

I do have concerns, but less about the pressure and more about the resourcing and the imbalance in the resourcing of academia. This is a little bit of a different conversation from World Labs. Over the past few years, I've been advocating for resourcing a healthy ecosystem.

As a founding co-director of Stanford's Institute for Human-Centered Artificial Intelligence, Stanford HAI, I've been working with policymakers on resourcing public-sector and academic AI work. We worked with the first Trump administration on a bill called the National AI Research Resource, or NAIRR, which scoped out a national AI compute cloud as well as a data repository.

I also think that open-source and open datasets continue to be an important part of the ecosystem. Like I said, right now in my Stanford lab, we're doing the open dataset and open benchmark on robotic learning called BEHAVIOR, and many of my colleagues are still doing that. I think that's part of the ecosystem.

What the industry and some startups are doing—running fast with models and creating products—is also a good thing. For example, when Justin was a PhD student with me, none of the computer vision programs worked that well, right? We could write beautiful papers.

Actually, even before grad school, I wanted to do computer vision. I reached out to a team at Google and wanted to potentially go and try to do computer vision out of undergrad, and they told me, “What are you talking about? You can't do that. Go do a PhD first and come back.”

What was the motivation that got you into it?

I had done some computer vision research during my undergrad with, actually, Fei-Fei's PhD adviser.

There's a lineage here.

There's a lineage here. So, I had done some computer vision even as an undergrad, and I thought it was really cool and wanted to keep doing it. I was sort of faced with this industry-academia choice even coming out of undergrad that I think a lot of people in the research community are facing now.

But to your question, I think the role of academia, especially in AI, has shifted quite a lot in the last decade. It's not a bad thing. It's because the technology has grown and emerged, right? Five or ten years ago, you really could train state-of-the-art models in the lab, even with just a couple of GPUs, but because that technology was so successful and scaled up so much, you can't train state-of-the-art models with a couple GPUs anymore. And that's not a bad thing. It's a good thing. It means the technology actually worked. But that means the expectations around what we should be doing as academics shift a little bit. It shouldn't be about trying to train the biggest model and scaling up the biggest thing.

It should be about trying wacky ideas and new ideas and crazy ideas, most of which won't work. I think there's a lot to be done there. If anything, I'm worried that too many people in academia are hyperfocused on this notion of trying to pretend like we can train the biggest models, or treating it as almost a vocational training program to then graduate and go to a big lab and be able to play with all the GPUs.

I think there's just so much crazy stuff you can do around new algorithms, new architectures, and new systems. There's a lot you can do as one person.

Academia also has a role to play in understanding the theoretical underpinnings of these large models. We still know so little about this, or extending it to the interdisciplinary—as Justin calls them—wacky ideas. There's a lot of basic science ideas and a lot of blue-sky problems.

So I agree. I don't think the problem is open versus closed, or productization versus open-sourcing. I think the problem right now is that academia by itself is severely under-resourced. The researchers and students do not have enough resources to try these ideas.

Yeah. Just to nerd-snipe people, what's a wacky idea that comes to mind when you talk about wacky ideas?

I had this idea that I kept pitching to my students at Michigan. I really like hardware, and I really like new kinds of hardware coming online. In some sense, the emergence of the neural networks that we use today and transformers is really based around matrix multiplication, because matrix multiplication fits really well with GPUs.

But if we think about how GPUs are going to scale and how hardware is likely to scale in the future, I don't think the current system we have—the GPU hardware design—is going to scale infinitely. We start to see that even now: the unit of compute is not the single device anymore. It's this whole cluster of devices.

Node.

Yeah, it's a whole node or a whole cluster. But the way we talk about neural networks is still as if they are a monolithic thing that could be coded on one GPU in PyTorch, when in practice they could be distributed over thousands of devices.

So, just as transformers are based around matrix multiplication, and matrix multiplication is the primitive that works really well on GPUs, as you imagine hardware scaling out, are there other primitives that make more sense for large-scale distributed systems that we could build our neural networks on?

I think it's possible that there could be drastically different architectures that fit with the next generation of hardware, or the hardware that's going to come 10 or 20 years down the line. We could start imagining that today.

It's really hard to make those kinds of bets because there's also the concept of the hardware lottery, where let's just say NVIDIA has won and we should just scale that out infinitely and write software to patch up any gaps we have in the mix, right?

I mean, yes and no. If you look at the numbers, even going from Hopper to Blackwell, the performance per watt is about the same. They mostly make the number of transistors go up, make the chip size go up, and make the power usage go up. But even from Hopper to Blackwell, we're already seeing a scaling limit in terms of the performance per watt that we can get.

I think there is room to do something new. I don't know exactly what it is, and I don't think you can get it done in a 3-month cycle as a startup. But I think that's the kind of idea that, if you sit down and sit with it for a couple of years, maybe you could come up with some breakthroughs. That's the kind of long-range stuff that is a perfect match for academia.

Coming back to a little bit of background in history, we have this sort of research note on the scene-storytelling work that you did, or the newer image-captioning work that you did with Andrej. I just wanted to hear you guys tell that story about how you were embarking on that for your PhD, and, Fei, the reaction that you had.

I think that line of work started between me and Andrej, and then Justin joined, right? Andrej started his PhD, and he and I were looking at what is beyond ImageNet object recognition. At that time, the convolutional neural network had proven some power in ImageNet tasks. ConvNets are a great way to represent images.

In the meantime, in language, the early sequential model called LSTM was also being experimented with. Andrej and I were just talking about this. It had been a long-term dream of mine. I thought it would take 100 years to solve, which is telling the story of images.

When I graduated from grad school, I really thought the rest of my entire career would be devoted to solving that single problem: given a picture or a scene, tell the story in natural language. But things evolve so fast.

When Andrej started, we thought maybe combining the representation of a convolutional neural network with the language sequential model of an LSTM would allow us to learn through training to match captions with images. That's when we started that line of work. I don't remember if it was 2014 or 2015. It was the CVPR 2015 captioning paper.

It was our first paper. Andrej got it to work: given an image, the image is represented with a ConvNet, the language model is an LSTM, and then we combine them, and it's able to generate one sentence. That was one of the first times it had been done.

I think I wrote in my book that we thought we were the first people doing it. It turned out that Google was also doing it simultaneously at that time. A reporter, John Markoff from The New York Times, was breaking the Google story, but he accidentally heard about us and realized that we had independently gotten there together at the same time.

So he wrote the story about both the Google research and Andrej's and my research. After that, I think Justin was already in the lab at that time.

Yeah. I remember the group meeting where Andrej was presenting some of those results and explaining this new thing called LSTMs and RNNs that I had never heard of before. I thought, wow, this is really amazing stuff. I want to work on that.

So then he had the paper at CVPR 2015 on the first image-captioning results. After that, we started working together. First, we did a paper just on language modeling, back in ICLR 2015.

I should have stuck with language modeling. That turned out to be pretty lucrative in retrospect. But we did this language-modeling paper together—me and Andrej—in 2015, and it was really cool. We trained these little RNN language models that could spit out a couple of sentences at a time, and we could poke at them and try to understand what the neurons inside the neural network were doing.

You guys were doing analysis on the different memory...

Yeah. It was really cool. Even at that time, we had these results where you could look inside the LSTM and say, oh, this thing is reading code.

One of the datasets that we trained on for this was the Linux source code, because the whole thing is open source and you could just download it. We trained an RNN on this dataset, and as the network was trying to predict the tokens, we tried to correlate the kinds of predictions it was making with the internal structures in the RNN.

We were able to find some correlations, such as, oh, this unit in this layer of the LSTM fires when there's an open parenthesis and then turns off when there's a closed parenthesis. We tried to do some empirical stuff like that to figure it out.

That was pretty cool. It was sort of like cutting out the CNN from this language-modeling part and just looking at the language models in isolation.

But then we wanted to extend the image-captioning work. I remember that at the time, we had a sense that captioning did not capture different parts of the image. I was talking to Justin and Andrej about whether we could do what we ended up calling dense captioning, which is describing the scene in greater detail, especially the different parts of the scene.

Yeah, and so then we built this system. It was me, Andrej, and Fei on a paper the following year, CVPR 2016, where we built this system that did dense captioning.

You input a single image, and then it would draw boxes around all the interesting stuff in the image and write a short snippet about each of them. It's like, oh, it's a green water bottle on the table. It's a person wearing a black shirt.

This was a really complicated neural network because it was built on a lot of advancements that had been made in object detection around that time, which was a major topic in computer vision for a long time. It was actually one joint neural network that was learning to look at individual images, and it had 3 different representations inside the network.

One was the representation of the whole image, to kind of get the gestalt of what was going on. Then it would propose individual regions that it wanted to focus on and represent each region independently. Once it looked at the region, it needed to spit out text for each region. So that was a pretty complicated neural network architecture.

Right, and does it do it in one pass?

Yeah. Yeah. So it was a single forward pass that did all of that.

Not only was it doing it in one pass; you also optimized inference. You were doing it on a webcam. I remember.

Yeah. So I had built this crazy real-time demo where I had the network running on a server at Stanford and then a web front end that would stream from a webcam and send the image back to the server. The server would run the model and stream the predictions back. So I was just walking around the lab with this laptop that would show people this network in real time.

Identification and labeling as well. Yeah, it was pretty impressive, because most of our graduate students would be satisfied if they could publish the paper, right? They package the research, put it in a paper, but Justin went a step further. He's like, “I want to do this real-time web demo.”

Well, actually, I don't know if I told you this story, but there was a conference that year in Santiago at ICCV. It was ICCV 2015. I had a paper at that conference for something different, but I had my laptop. I was walking around the conference with my laptop, showing everybody this real-time captioning demo, and the model was running on a server in California. So it was actually able to stream all the way from California down to Santiago.

The latency was terrible. It was like 1 FPS, but the fact that it worked at all was pretty amazing.

I was going to briefly quip that maybe vision and language modeling are not that different. You know, DeepSeek-OCR recently tried the crazy thing of, “Let's model text from pixels and train on that,” and it might be the future. I don't know. I don't know if you guys have any takes on whether language is actually necessary at all.

I just wrote a whole manifesto on spatial intelligence. [laughter]

This is my segue into this. Yes.

I think they are different. I do think the architecture of these generative models will share a lot of sharable components, but I think the deeply 3D, 4D spatial world has a level of structure that is fundamentally different from a purely generative signal that is one-dimensional.

Yeah, I think there's something to be said for pixel maximalism, right? There's this notion that language is this different thing, but we see language with our eyes, and our eyes are just basically pixels, right? We've got sort of biological pixels in the back of our eyes that are processing these things. We see text and we think of it as this discrete thing, but that really only exists in our minds. The physical manifestations of text and language in our world are physical objects that are printed on things in the world, and we see them with our eyes.

Well, you can also think of it as sound, but even sound—you can translate it into a signal. You can encode it as a signal.

Right? And then you actually lose something if you translate to these purely tokenized representations that we use in LLMs, right? You lose the font, you lose the line breaks, you lose sort of the 2D arrangement on the page. For a lot of cases, for a lot of things, maybe that doesn't matter. But for some things it does.

I think pixels are this more lossless representation of what's going on in the world and, in some ways, a more general representation that more matches what we humans see as we navigate the world. So there's an efficiency argument to be made. Maybe it's not super efficient to render your text to an image and then feed that to a vision model.

That's exactly what DeepSeek-OCR did, right? It kind of worked. [laughter] I think this ties into the whole world model. One of my favorite papers that I saw this year was about inductive bias for world models. It was a Harvard paper where they fed a lot of orbital patterns into an LLM and then asked the LLM to predict the orbit of a planet around the sun. The model-generated orbits looked good, but then if you asked it to draw the force vectors, it would be all wacky. It wouldn't actually follow them.

So how do you think about what's embedded into the data that you get? We can talk about maybe tokenizing for 3D world models. What are the dimensions of information? There's the visual, but how much of the underlying hidden forces, so to speak, do you need to extract from this data, and what are some of the challenges there?

Yeah, I think there are different ways you could approach that problem. One is you could try to be explicit about it and say, “Oh, I want to measure all the forces and feed those as training data to your model,” right? Then you could sort of run a traditional physics simulation, know all the forces in the scene, and use those as training data to train a model that's now going to hopefully predict them.

Or you could hope that something emerges more latently, right? You kind of train on something end to end and then on a more general problem, and hope that somewhere, something in the internals of the model must learn to model something like physics in order to make the proper predictions. Those are kind of the 2 big paradigms that we have more generally.

But there's no indication that latent modeling will get you to a causal law of space and dynamics, right? That's where today's deep learning and human intelligence actually start to bifurcate, because fundamentally, deep learning is still fitting patterns.

There you sort of get philosophical and say that we're trying to fit patterns too, but maybe we're trying to fit a broader array of patterns over a longer time horizon with a different reward function. But basically, the paper you mentioned is that problem: it learns to fit the specific patterns of orbits, but then it doesn't actually generalize in the way that you'd like. It doesn't have a causal model of gravity.

Right? Because even in Marble, I was trying it, and it generates these beautiful sceneries, and there are arches in them. But does the model actually understand how the arch is relying on the center keystone and the actual physical structure of it?

The other question is: does it matter whether it understands it, as long as it always renders something that would fit the physical model that we imagine? If you use the word “understand” the way you understand it, I'm pretty sure the model doesn't understand it. [laughter] The model is learning from the data, learning from the pattern.

Yeah. Does it matter, especially for its use cases? It's a good question, right? For now, I don't think it matters, because it renders out what you need, assuming it's perfect.

Yeah, I mean, it depends on the use case. If the use case is, “I want to generate sort of a backdrop for virtual film production,” or something like that, all you need is something that looks plausible, and in that case, probably it doesn't matter.

But if you're going to use this to—if you're an architect and you're going to use this to design a building that you're then going to build in the real world—then, yeah, it does matter that you model the forces correctly, because you don't want the thing to break once you actually build it.

But even there, right? Even if your model has the semantics in it, let's say, I still don't think the understanding of the signal or the output on the model's part and the understanding on the human part are the same thing. But this gets philosophical again.

Yeah, I mean, there's this trick with understanding, right? These models are a very different kind of intelligence than human intelligence. Human intelligence is interesting because I think that I understand things because I can introspect my own thought process to some extent. And then I believe that my thought process probably works similarly to other people's, so that when I observe someone else's behavior, I infer that their internal mental state is probably similar to my own internal mental state that I've observed. And therefore, I know that I understand things. So there, I assume that you understand something.

But these models are sort of like this alien form of intelligence where they can do really interesting things. They can exhibit really interesting behavior. But whatever kind of internal cognition or internal self-reflection they have, if it exists at all, is totally different from what we do.

So it doesn't have self-awareness.

Right. But what that means is that when we observe seemingly interesting or intelligent behavior out of these systems, we can't necessarily infer other things about them, because their model of the world and the way they think are so different from ours.

So would you need 2 different models to do the visual one and the architectural generation? Do you think eventually there's not anything fundamental about the approach that you've taken in model building?

Is it more about scaling the model and its capabilities, or is there something about being very visual that prohibits you from actually learning the physics behind it, so to speak, so that you could trust it to generate a CAD design that is actually going to work in the real world?

I think this is a matter of scaling data and bettering the model. I don't think there's anything fundamental that separates these two.

Yeah, I would like it to be one model, but I think the big problem in deep learning, in some sense, is: How do you get emergent capabilities beyond your training data? Are you going to get something that understands the forces when it wasn't trained to predict the forces, but it's going to learn them implicitly, internally?

I think a lot of what we've seen in other large models is that a lot of this emergent behavior does happen at scale. Will that transfer to other modalities and other use cases and other tasks? I hope so, but that'll be a process that we need to play out over time and see.

Is there a temptation to rely on physics engines that already exist out there? The gaming industry has basically saved you a lot of this work. Do we have to reinvent things because of some fundamental mismatch?

I think that's sort of like climbing the ladder of technology. In some sense, the reason that you want to build these things at all is because maybe traditional physics engines don't work in some situations. If a physics engine were perfect, we would have no need to build models, because the problem would have already been solved.

In some sense, the reason why we want to do this is because classical physics engines don't solve problems with the generality that we want. But that doesn't mean we need to throw them away and start everything from scratch. We can use traditional physics engines to generate data that we then train our models on. You're sort of distilling the physics engine into the weights of the neural network that you're training.

I think that's a lot of what you're seeing if you compare the work of other labs. People are speculating that Sora had a little bit of that. Genie 3 had a bit of that. Genie is explicitly like a video game—you have controls to walk around in. I always think it's really funny how the things that we invent for fun eventually make it into serious work.

Yeah. The whole AI revolution started with graphics chips.

Partially—misusing the GPU for generating a lot of triangles and turning it into generating a lot of everything else, basically.

Yeah. We touched on Marble a little bit. I think you guys chose Marble as kind of your coming-out-of-stealth moment, if you can call it that. Maybe we can get a concise explanation from you on what people should take away, because everyone here can try Marble, but I don't think they might be able to connect it to the differences between your vision and other generative worlds they may have seen from other labs.

Marble is a glimpse into our model. We are a spatial intelligence model company. We believe spatial intelligence is the next frontier. In order to make spatially intelligent models, the model has to be very powerful in terms of its ability to understand, reason about, and generate worlds in a very multimodal fashion, as well as allow the level of interactivity that we eventually hope will be as complex as how humans can interact with the world.

That's the grand vision of spatial intelligence, as well as the kind of world models we see. Marble is the first glimpse into that. It's the first part of that journey. It's the first-in-class model in the world that generates 3D worlds at this level of fidelity and is in the hands of the public. It's the starting point.

We actually wrote this tech blog. Justin spent a lot of time writing that tech blog. I don't know if you had time to browse it, but Justin really broke it down into what the inputs are—we can have multimodal inputs for Marble—what kind of editability it has, which allows the user to be interactive with the model, and what kind of outputs we can have.

Yeah. One way of looking at Marble is that it's a generative model of 3D worlds. You can input things like text, an image, or multiple images, and it will generate for you a 3D world that matches those inputs.

It's also interactive in the sense that you can interactively edit scenes. I could generate this scene and then say, “I don't like the water bottle. Make it blue instead. Take out the table. Change these microphones around.” Then you can generate new worlds based on these interactive edits and export them in a variety of formats.

With Marble, we were actually trying to do 2 things simultaneously, and I think we managed to strike the balance pretty well. One is to build a model that goes toward the grand vision of spatial intelligence. Models need to be able to understand lots of different kinds of inputs, model worlds in a lot of situations, and model counterfactuals of how they could change over time. We wanted to start building models that have these capabilities, and Marble today already has hints of all of them.

At the same time, we're a company. We're a business. We were really trying not to have this be a science project, but also to build a product that would be useful to people in the real world today. While Marble is simultaneously a world model that is building toward this vision of spatial intelligence, it was also very intentionally designed to be something that people could find useful today.

We're starting to see emerging use cases for gaming, VFX, and film, where I think there's a lot of really interesting stuff that Marble can do today as a product, while also setting a foundation for the grand world models that we want to build going into the future.

Yeah. I noticed one tool that was very interesting, because you can record your scene inside.

Yes. It's very important. The ability to record means very precise control of camera placement. In order to have precise camera placement, you have to have a sense of 3D space. Otherwise, you don't know how to orient your camera or how to move your camera. That is a natural consequence of this kind of model, and this is just one of the examples.

Yeah. I find that when I play with video generative models, I'm having to learn the language of being a director because I have to move them—pan, you know. You cannot say, “Pan 63 degrees to the north,” right? You just don't have that control, whereas in Marble you have precise control in terms of placing your camera.

I think that's one of the first things people need to understand: You're not generating frame by frame, which is what a lot of the other models are doing.

What are the atomic units? People understand that an LLM generates 1 token. What are the atomic units of your generations? There are the meshes, the splats, the voxels—there are a lot of pieces in a 3D world. What should be the mental model that people have for your generations?

Yeah. I think there's what exists today and what could exist in the future. What exists today is that the model natively outputs splats. Gaussian splats are tiny particles; each one is semitransparent and has a position and orientation in 3D space. The scene is built up from a large number of these Gaussian splats.

Gaussian splats are really cool because you can render them in real time very efficiently. You can render them on your iPhone—render everything in real time. That's how we get that precise camera control, because the splats can be rendered in real time on pretty much any client-side device that we want.

For a lot of the scenes that we're generating today, that kind of atomic unit is the individual splat. But I don't think that's fundamental. I could imagine other approaches in the future that would be interesting. There are other approaches that we've worked on at World Labs, like our recent RTFM model, which does generate frames 1 at a time. There, the atomic unit is generating frames 1 at a time as the user interacts with the system.

You could imagine other architectures in the future where the atomic unit is a token, where that token represents some chunk of the 3D world. I think there are a lot of different architectures that we can experiment with here over time.

I do want to press on this and double-click on it a little bit. My version of what Speaker 0 was going to say was: What is the fundamental data structure of a world model? Exactly like you said, it's either a Gaussian splat or it's the frame, or what have you.

You also, in your previous statements, focused a lot on the physics and the forces, which is something over time. I don't see that in Marble. I presume it's not there yet. Maybe if there were a Marble 2, you would have movement. Is there a modification to Gaussian splats that makes sense, or would it be something completely different?

Yeah, I think a couple of modifications make sense, and there are actually a lot of interesting ways to integrate things here, which is another nice place to be working in this space. There's been a lot of research work on this. When you talk about wacky ideas, there's actually been a lot of really interesting academic work on different ways to imbue physics.

You can also do wacky ideas in industry.

Yeah.

Right. But Gaussian splats are themselves little particles. There have been a lot of approaches where you attach physical properties to those splats and say that each one has a mass, or maybe treat each one as being coupled with some kind of virtual spring to nearby neighbors. Now you can start to do physics simulation on top of splats.

One avenue for adding physics, dynamics, or interaction to these things would be to predict physical properties associated with each of your splat particles and then simulate those downstream, either using classical physics or something learned. The beauty of working in 3D is that things compose, and you can inject logic in different places.

One way is that we're generating a 3D scene, and we're going to predict 3D properties of everything in the scene. Then we use a classical physics engine to simulate the interaction. Or you could do something where, as a result of a user action, the model is now going to regenerate the entire scene in splats or some other representation.

That could potentially be a lot more general because then you're not bound to whatever physical properties you already know how to model. But that's also a lot more computationally demanding because then you need to regenerate the whole scene in response to user actions. I think this was a really interesting area for future work and for adding on to potential Marble 2, as you say.

Yeah. And there's opportunity for dynamics, right?

What's the state of splat density, I guess? Can we render enough to have very high resolution when we zoom in? Are we limited by the amount that you can generate, or the amount that we can render? How are these going to get super high-fidelity, so to speak?

You have some limitations, depending on your target use case. One of the big constraints that we have on our scenes is that we wanted things to render cleanly on mobile, and we wanted things to render cleanly in VR headsets. Those devices have a lot less compute than you're used to having in a lot of other situations.

If you want to get a splat file to render at high resolution, at 30 to 60 FPS, on an iPhone from 4 years ago, then you are a bit limited in the number of splats that you can handle. But if you're allowed to work on a recent iPhone—even this year's iPhone—or a recent MacBook, or even if you have a local GPU, or if you don't need that 60 FPS at 1080p, then you can relax the constraints and get away with more splats. That lets you get higher resolution in your scenes.

One use case I was expecting but didn't hear from you was embodied use cases. Are you just focusing on virtual for now?

If you go to the World Labs homepage, there is a particular page called Marble Labs. There, we showcase different use cases, and we organize them into visual-effects use cases, gaming use cases, and simulation use cases.

We actually show that this is a technology that can help a lot with robotic training. This goes back to what I was talking about earlier when speaking of data starvation. Robotic training really lacks data. High-fidelity, real-world data is absolutely critical, but you're just not going to get a ton of that.

At the other extreme, with purely internet video data, you lack a lot of the controllability that you want to train your embodied agents with. Simulation and synthetic data are actually a very important middle ground for that. I've been working in this space for many years, and one of the biggest pain points is where you get this synthetic, simulated data.

You have to curate assets and compose these complex situations. In robotics, you want a lot of different states. You want the embodied agent to interact in the synthetic environment. Marble is a really promising technology for helping to generate these synthetic, simulated worlds for embodied-agent training.

Obviously, that's on the homepage. It'll be there. I was trying to make the link to what you said: You also have to build a business model. The market for robotics is obviously very huge. Maybe you don't need that, or maybe we need to build up and solve the virtual worlds first before we go to embodied, and that's obviously a stepping stone.

That is to be decided. I do think that—

Because everyone else is going straight there, right?

Not everyone else, but there is excitement, I would say. I think the world is big enough to have different approaches.

Yeah. Approaches.

Yeah.

Yeah. I mean, we always view this as a pretty horizontal technology that should be able to touch a lot of different industries over time. Marble is a little bit more focused on creative industries for now, but I think that the technology that powers it should be applicable to a lot of different things over time. Robotics is one that is maybe going to happen sooner than later.

Also, design, right, is very adjacent to creative.

Oh, yeah. Definitely. I think it's the architecture stuff.

Yes. Okay. Yeah. I was joking online. I posted this video on Slack: “Who wants to use Marble to plan your next kitchen remodel?” It actually works great for this already. Just take 2 images of your kitchen, reconstruct it in Marble, and then use the editing features to see what that space would look like if you changed the countertops, the floors, or the cabinets.

This is something that we didn't necessarily build anything specific for, but because it's a powerful horizontal technology, you get these emergent use cases that just fall out of the model. We have early beta users using an API key that is already building for an interior-design use case.

I just did my garage. I should have known about this.

Next time you remodel, we can be of help.

Well, the kitchen is next, I'm sure.

Yeah.

Yeah. I'm curious about the whole spatial-intelligence space. I think we should dig more into that one. How do you define it, and what are the gaps between traditional intelligence that people might think about with LLMs? When Dario says, “We have a data center full of Einsteins,” that's traditional intelligence. It's not spatial intelligence. What is required to be spatially intelligent?

First of all, I don't understand that sentence—a data center full of Einsteins.

I just don't understand that.

It's an analogy. It's an analogy.

A lot of AI as a field, as a discipline, is inspired by human intelligence, right? We are the most intelligent animal we know in the universe, for now. If you look at human intelligence, it's very multi-intelligent. There is a psychologist—I think his name is Howard Gardner—who, in the 1960s, literally called it “multiple intelligences” to describe human intelligence. There is linguistic intelligence, spatial intelligence, logical intelligence, and emotional intelligence.

For me, when I think about spatial intelligence, I see it as complementary to language intelligence. I personally would not say it's spatial versus traditional, because I don't know what “traditional” means. I do think spatial is complementary to linguistic intelligence.

How do we define spatial intelligence? It's the capability that allows you to reason, understand, move, and interact in space. I use the example of the deduction of DNA structure. Of course, I'm simplifying this story, but a lot of that had to do with the spatial reasoning of the molecules and the chemical bonds in a 3D space to eventually conjecture a double helix, and that ability that humans—Francis Crick and James Watson—had.

It is very, very hard to reduce that process into pure language. That's a pinnacle of a civilizational moment. But every day, I'm here trying to grasp a mug. This whole process of seeing the mug, seeing the context where it is, seeing my own hand, opening my hand so that it geometrically matches the mug, and touching the right affordance point—all of this is deeply, deeply spatial.

It's very hard. I'm trying to use language to narrate it, but on the other hand, that narrated language itself cannot get you to pick up a mug.

Yeah. Bandwidth constraint.

Yes.

I did some math recently. If you just spoke all day, every day, for 24 hours a day, at the average speaking rate of 150 words per minute, it roughly rounds out to about 215,000 tokens per day. Your world that you live in is so much higher bandwidth than that.

Well, I think that is true. But if I think about Sir Isaac Newton, you have things like gravity that, at the time, had not been formalized in language. People inherently spatially understand that things fall, right? But then it's helpful to formalize that in some way.

All these different rules that we use language to really capture are things that empirically and spatially you can also understand, but they're easier to describe in a certain way. So I'm curious about the interplay of spatial and linguistic intelligence. You need to understand that some rules are easier to write in language for the spatial intelligence to understand, but you cannot write, “Put your hand like this and put it down this amount.”

I’m always curious about how you leverage each other together.

If anything, the example of Newton illustrates that Newton only thinks to write down those laws because he’s had a lot of embodied experience in the world watching baseball.

Exactly. And actually, it’s useful to distinguish between the theory building that you’re mentioning and the embodied, daily experience of being embedded in the three-dimensional world.

So, to me, spatial intelligence is sort of encapsulating that embodied experience of being there in 3D space—moving through it, seeing it, actioning it. And as Fei-Fei said, you can narrate those things, but it’s a very lossy channel. The notion of being in the world and doing things in it is a very different modality from trying to describe it.

Because we as humans are animals who have evolved interacting in space all the time, we don’t even think that that’s a hard thing, right? And then we naturally leap to language and theory building as mechanisms to abstract above that sort of native spatial understanding. In some sense, LLMs have just jumped all the way to those highest forms of abstracted reasoning, which is very interesting and very useful.

But spatial intelligence is almost like opening up that black box again and saying, maybe we’ve lost something by going straight to that fully abstracted form of language, reasoning, and communication.

You know, it’s funny—as a vision scientist, right?—I always find that vision is underappreciated because it’s effortless for humans. You open your eyes as a baby, you start to see the world. We’re somehow born with it.

Right? We’re almost born with it.

But you have to put effort into learning language, including learning how to write, how to do grammar, and how to express yourself, and that makes it feel hard. Whereas something that nature spent way more time actually optimizing—perception and spatial intelligence—is underappreciated by humans.

Is there proof that we are born with it? You said “almost born,” so it sounds like we actually do learn after we’re born. When we are born, our visual acuity is less, and our perceptual ability does increase. But most humans are born with the ability to see, and most humans are born with the ability to link perception with motor movements, right?

The motor movement itself takes a while to refine, but animals are incredible, right? I was just in Africa earlier this summer. These little animals are born, and within minutes they have to get going; otherwise, the lions will get them. In nature, it took 540 million years to optimize perception and spatial intelligence. The most generous estimation of language development is probably 500,000 years.

Wow.

Yeah. [Laughter]

That’s longer than I would have guessed, I’d say. Well, I’m being very generous. Yeah.

No, I was going through your book, and I was realizing that one of the interesting links to something that we covered on the podcast is language model benchmarks and how WinoGrande actually put in all these physical impossibilities that require spatial intelligence.

Like, “A is on top of B, therefore A cannot fall through B,” is obvious to us, but to a language model it could happen. I don’t know; maybe it’s part of next-token prediction.

Is.

And that’s sort of what I mean about unwrapping this abstraction, right? If your whole model of the world is just saying sequences of words after each other, it’s really kind of hard to ask, “Why not?”

It’s actually unfair.

Right? But then the reason it’s obvious to us is because we are internally mapping it back to some three-dimensional representation of the world that we’re familiar with. The question is, I guess, how hard is it going to be to distill from—like, I use the word “distill”; I don’t know if you agree with that—to distill from your world models into a language model? Because we do want our models to have spatial intelligence, right?

Do we have to throw the language model out completely in order to do that, or—

No.

No, right? I don’t think so, right?

I think they’re multimodal. I mean, even our model, Marble, today takes language as an input.

Right.

Right, so it’s deeply multimodal, and I think in many use cases these models will work together. Maybe one day we’ll have a universal model.

I mean, even if you do, there’s sort of a pragmatic thing where people use language, and people want to interact with systems using language. Even pragmatically, it’s useful to build systems and products and models that let people talk to them. So I don’t see that going away.

I think there’s a sort of intellectual curiosity in saying, intellectually, how much could you build a model that only uses vision or only uses spatial intelligence? I don’t know that that would be practically useful, but I think it would be an interesting intellectual or academic exercise to see how far you could push that.

I’m curious—if you had a highly precise world model and you didn’t give it any notion of our current understanding of the Standard Model of physics, how much of it would be able to come up with and recreate from scratch? What level of language understanding would it need? Because we have so many notations that we use, that we created, but maybe it would come up with a very different model and still be accurate.

I wonder how much we’re limited. People say humans always need to be like humans because the world is built for humans. In a way, the way we build language constrains some of the outputs that we can get from these other modalities as well. So I’m super excited to follow your work.

Yeah. I mean, there’s another way to answer that question: you don’t even need to be doing AI. You could discover aliens and see what kind of physics they have, right? And they might have a—

Let’s face it, we are, so far, the smartest animal in the universe, right?

So, I mean, that is a really interesting question, right? Is our knowledge of the universe and our understanding of physics constrained in some way by our own cognition, or by the path dependence of our own technological evolution?

One way to do an experiment is to say, if we were to rerun human civilization again, would we come up with the same physics in the same order? I don’t think that’s a very practical experiment to run.

You know, one experiment I wonder if people could run is that we have plenty of astrophysical data now on planetary or celestial-body movements. Just feed the data into a model and see if Newtonian law emerges.

My guess is it probably won’t.

That’s my guess. The abstraction level of Newtonian law is at a different level from what these language LLMs represent.

Yeah. So I wouldn’t be surprised if, given enough celestial movement data, an LLM would actually predict pretty accurate movement trajectories. Let’s say I invent a planet surrounding a star, and given enough data, my model would tell you, on day 1 where it is, day 2 where it is.

I wouldn’t be surprised. But F = ma, or action equals reaction—that’s just a whole different abstraction level. That’s beyond today’s LLMs.

Okay. What model would you need to not have it be a geocentric model? Because if I’m training just on visual data, it makes sense that you think the sun rotates around the Earth, right? But obviously that’s not the case. So how would it learn that?

I’m curious about all these forces that we talk about. Sometimes maybe you don’t need them, because as long as it looks right, it’s right. But as you make the jump to trying to use these models to do more high-level tasks, how much can we rely on them?

I think you need a different learning paradigm, right? There’s a bit of conflation happening here when we ask whether it’s LLMs and language and symbols versus human theory building and human physics. They’re very different, because an LLM—the human objective function is to understand the world and thrive in your life.

The way that you do that is by sometimes observing data, then thinking about it, and then trying to do something in the world. It doesn’t match your expectations, and then you update your understanding of the world online. People do this all the time, constantly.

Whether it’s, “I think my keys are downstairs,” so I go downstairs and look for them and don’t see them, and, “Oh no, they’re actually up in my bedroom,” we’re constantly interacting with the world. We’re constantly having to build theories about what’s happening in the world around us and then falsify or add evidence to those theories.

I think that kind of process, writ large and scaled up, is what gives us F = ma and Newtonian physics. And I think that’s a little orthogonal to the modality of the model that we’re training, whether it’s language or spatially.

The way I put it is almost like this is more efficient learning. You have a hypothesis of the different possible worlds that are granted by your available data, and you then do experiments to eliminate the worlds that are not possible and resolve to the one that’s right.

To me, that's also how I have a theory of mind. I have a few hypotheses about what you're thinking, and I try to create actions to resolve that or check my intuition as to what you're thinking. Obviously, LLMs don't do any of these.

A theory of mind possibly also breaks into emotional intelligence, which today's AI is really not touching at all. And we really, really need it. People are starting to depend on these things probably too much, and that's a whole topic of another debate.

I do have to ask, because a lot of people have sent this to us: How much do we have to get rid of? Is sequence-to-sequence modeling out the window? Is attention out the window? How much are we rethinking everything?

I think you stick with stuff that works, right? I think attention is still there. You don't need to fix things that aren't broken. There are a lot of hard problems in the world to solve, but let's focus on one at a time.

I think it is pretty interesting to think about new architectures, new paradigms, or drastically different ways to learn. But you don't need to throw away everything just because you're working on new modalities.

I think sequence-to-sequence is actually in world models. I think we're going to see algorithms or architectures beyond sequence-to-sequence.

Oh, but here, actually, I think there's a little bit of technological confusion, and transformers already solved that for us, right? Transformers are actually not a model of sequences. A transformer is natively a model of sets. That's very powerful, but a lot of transformers grew out of earlier architectures based around recurrent neural networks, and RNNs definitely do have a built-in architecture: they model one-dimensional sequences.

Transformers are just models of sets, and a lot of those sets could be 1D sequences; they could be other things as well.

So you literally mean set theory?

Yeah, yeah, so whatever.

Yeah, yeah. A transformer is actually not a model of a sequence of tokens. A transformer is actually a model of a set of tokens, right? The only thing that injects order into the standard transformer architecture, and the only thing that differentiates the order of the things, is the positional embedding that you give the tokens, right?

So, if you choose to give it a sort of 1D positional embedding, that's the only mechanism that the model has to know that it's a 1D sequence. But all the operators that happen inside a transformer block are either token-wise—you have an FFN, QKV projections, and per-token normalization, all of which happen independently per token—or interactions between tokens through the attention mechanism. But that's also permutation-equivariant, so if I permute my tokens, then the attention operator gets a permuted output in exactly the same way. So, it's actually natively an architecture of sets of tokens.

Literally a transform.

Yeah. [laughter]

I know we're out of time, but we just want to give you the floor for some call to action: either on people who would enjoy working at World Labs—what kind of people should apply—or what research people should be doing outside of World Labs that would be helpful to you, or anything else on your mind.

I do think it's a very exciting time to be looking beyond just language models and thinking about the boundless possibilities of spatial intelligence. We are hungry for talent, ranging from very deep researchers thinking about problems like the ones Justin just described, such as training large world models. We are hungry for good engineers building systems, from training optimization to inference to product. And we're also hungry for good business and product thinkers, go-to-market talent, and business talent.

So, we're hungry for talent. Especially now that we have exposed the model to the world through Marble, I think we have a great opportunity to work with an even bigger pool of talent to solve both the model problem and deliver the best product to the world.

Yeah, I think I'm also excited for people to try Marble and do a lot of cool stuff with it. I think it has a lot of really cool capabilities and features that fit together really nicely.

In the car coming here, Justin and I were saying that people have not totally discovered the advanced mode of editing. It's only been 24 hours. Turn on the advanced mode. You can, like Justin said, change the color of the bottle, change your floor, and change the trees.

Well, I actually tried to get there, but when it says “Create,” it just makes me create a completely different world.

You need to click on the advanced mode. It's a good UI.

We can improve on our UI. Remember to click.

Yeah, we need to hire people who work on the product. [laughter]

One thing we got that was clear from you guys is that you're also looking for intellectual fearlessness, which is something that I think you guys hold as a principle.

Yeah, I mean, we are literally the first people who are trying this, both on the model side as well as on the product side.

Thank you so much for joining us. This was fun.

Yeah, thanks for having us.

Yeah.