World Models & General Intuition: Khosla's largest bet since LLMs & OpenAI

You know, in a video model, you might predict the next likely sequence or the next most entertaining frame. What world models do is actually understand the full range of possibilities and outcomes from the current state and, based on the action that you take, generate the next state—the next frame. So it’s a much more complex problem than traditional video models. To me, it’s a world that’s accurately generated based on the actions that you take as a result of what’s already been generated.

Hi listeners, as you may know, I recently wrapped up the AIE Code conference in New York. And while I’m traveling, I do like to visit top AI startups in person to bring you interviews that you don’t find on any other podcast that just does a Zoom call. General Intuition, or GI for short, is a spinout of a 10-year-old game-clipping company called Medal, which has 12 million users, but in comparison, Twitch only has 7 million monthly active streamers. Medal collects this data by building the best retroactive clipping software in the world. In other words, you don’t need to be consciously recording. You just have Medal on in the background while you’re playing, and you hit a button to clip the last 30 seconds after something interesting happens. It’s very similar to how Tesla and self-driving do bug reporting. If you’ve ever done a self-driving bug report in a Tesla, the result is that Medal has accumulated 3.8 billion clips of the best moments and actions in games, resulting in one of the most unique and diverse datasets of peak human behavior, actively mining for the interesting moments. They were also very careful in navigating privacy and data-collection concerns by mapping actions to these visual inputs and game outcomes. As you saw on our Fei-Fei Li and Justin Johnson episode with World Labs, and with the recent departure of Yann LeCun from Meta, there’s a lot of interest in world models as the next frontier after LLMs to improve spatial intelligence and work on embodied robotics use cases. DeepMind has been working on this with Genie 1, 2, and 3 and SIMA 1 and 2. And this year, OpenAI seemed to finally agree because they’ve been betting on LLMs a lot, and they made the news by offering $500 million for Medal’s video-game clip data. Our guest today, Pim, turned down that money and instead chose to build an independent world-model lab. Instead, Khosla Ventures led the $134 million seed round, which is Vinod Khosla’s largest single seed bet since OpenAI. We were able to get an exclusive preview of GI’s models, which unfortunately we cannot show you directly, but I can confirm they were incredibly humanlike, and we chose to include the first 11 minutes of the demo discussion. Even though I couldn’t show it to you, it may be hard to follow, but I tried to call out what was noteworthy for you to know, as would likely be your reaction if you were watching along with us. Now, enjoy the world’s first look at my first look at General Intuition.

So, what I’m about to show you is a completely vision-based agent that’s just seeing pixels and predicting actions the exact same way a human would. What I’ll show you here is what this looked like 4 months ago. This is just an agent that’s receiving frames and predicting actions.

You can see it has a decent sense of how to navigate around. It taps the scoreboard, just like gamers always tap the scoreboard. These are pure imitation-learning models.

I see it slicing the knife.

Yeah, exactly. It’s doing everything that humans would do in this case. Here was the first interesting part that we saw: it gets stuck, and then it has memory as well. You can see that it can get unstuck.

How long is the memory?

4 seconds.

4 seconds. Okay.

This was 4 months ago. This was maybe a few weeks after that. You can see it’s still doing the scoreboard thing, but there’s still quite a lot of weirdness.

These are bots, too?

Yeah.

It’s very human, let’s just say that.

Yeah. This was really the early days of research. You can see it does one thing and then goes for another. We’ve been scaling on data and compute, and we’ve also been making the models better. This is where we are now.

What you’re seeing is, like I said, pure imitation learning. This is just a base model. There’s no RL and no fine-tuning. This model sees no game states. It’s purely pixel-based, not state-based. It’s purely predicting the actions from the frames. That’s it.

It’s playing against real humans, just like a human would play, and it’s also running completely in real time. Everything here plays exactly like a human.

Do you give it a goal?

Yep.

It just figures out its goal because it’s trained on a scene?

Yes. I picked a sequence where it doesn’t do well initially. This is just a random sequence.

But it looks like it’s doing well.

Oh, okay. Yeah, watch.

Yeah, that’s pretty good. Maybe too good.

This is my favorite part. You can see it does something that a human would never do. Then it gets unstuck, has to realize which way to go, and then sees something in the distance.

So you’re saying, one, it makes a mistake that a human will never make, but it unsticks itself. And two, what we just saw is it doing superhuman things.

Yeah.

Okay.

There are things that it does, obviously, but because it’s trained on the highlights—the exceptional things—it inherits those.

So it’s not like Move 37, where they RL’d their way into something.

Yeah.

Replicating something superhuman?

The baseline of our data set is peak human performance.

Yes. Okay.

That’s the agent. Now I’m going to show you that we’re able to take those action predictions and label any video on the internet using those actions.

This is just frames in and actions out. Yellow is the ground truth, purple is the model prediction, the bottom left is the compound error over the entire sequence, and this is reset per prediction.

“Reset” means that you reset every now and then?

Yeah. It just means it resets the baseline. A single error in the entire sequence compounds here, but it doesn’t compound here, if that makes sense.

Yeah.

Again, this is just seeing frames. It’s not seeing any of the actions. What we did was train it on less realistic games and transfer it over to a more realistic game. Then—and this is where it gets really exciting—we transferred it over to real-world video, which means that you can use any video on the internet as pretraining.

What is it predicting?

It’s predicting as if you were controlling it using a keyboard and mouse. If you were basically playing the sequence as a human.

Is there some sense of error?

That’s why you transfer to more realistic games first.

Yeah.

Then you transfer to real-world video because you can’t get a sense of ground truth from real-world video yet.

This is the same agent that I just showed you, playing against other AIs.

This one’s playing against bots. The previous one was against players.

With the sniper, it doesn’t really matter that much, as you’ll see.

It’s like—

One thing that’s really interesting is that you notice it behaves differently when it has different items.

That makes intuitive sense.

Yeah.

I think there’s also a question about egocentricity versus third-person perspective. Does it matter?

Third-person perspective will be very helpful if you’re, for instance, trying to control multiple objects in an environment later on. Right now, I think having a fully immersive first-person view is quite helpful.

This is the policy itself.

What do you mean, “the policy”?

The agent.

The agent?

Yeah. It has the same constraints that I just told you about.

The way it hides was just incredible to me, just from knowing how to hide when you see it.

It also hides when you see it.

Exactly. And while it was reloading.

Okay, so those are—

That’s the policy, and this is a completely general recipe. We can scale this to any environment.

Okay. No, let’s keep going on the demos.

I was going to go into the research.

Yeah, yeah.

What I’m about to show you are the world models. There are a few really interesting parts about our world models. The first is that we made the decision to pretrain world models from scratch, but we’ve also been able to fine-tune open-source video models to get a better sense of physics transfer.

One thing you’ll notice here is that our world models have mouse sensitivity, which is something that gamers absolutely want. You can have these very rapid movements, which you couldn’t do in any other world model.

And so this is a holdout set. This clip was never seen before at training time. You can see it has spatial memory. This is about a 20-second-ish generation.

And here's what's fascinating: This is an explosion that occurs. You can see that in the physical world, the camera would shake, and in the game that would never happen. The world model inherits the physical-world camera shake, but the actual game never does that. That, to us, was quite fascinating, right?

Also, the models that I just showed you—the 2 that we used to transfer over from video—those combined will allow us to push way beyond games in terms of training.

This is another interesting one. This is the world model with rapid camera motion. Again, this is stuff that we're literally just taking 1 second from here, in the context and the actions, and replaying it here, right? You never essentially have what we're saying is the skill that you see in the clips—the speed and the movement—that also pays off at training time when you're doing world models.

This is my favorite example. This shows that the world model is capable of performing with partial observability. What you're going to see is, again, you're replaying the actions from here in here, just using 1 second of video context. Everything after that is completely generated.

So what you're going to see is the model is going to encounter, in this case, smoke. Normally, models break down. What you actually see is it comes out of the same place. So it's capable of, even with partial observability, still maintaining its position in the world.

Then here it is also interesting. This is sniping. This gives you a

Reaction time.

Like the fact that it can do depth and sequences in completely different views, right? This is a completely different view than if you were to be outside of that view, right? It's able to maintain consistency while zooming in.

Yeah, exactly.

And so, yeah, you can see—even while this goes out of scope, right? Watch. Then it comes back, and you'll see it's still there.

Yeah.

This is the work that Anthony has been working on.

I'm just wondering how much game footage you have to watch in order to find these things.

We can ask Anthony. I'm sure he's not going to be too excited to play these games afterward.

You're not playing. You're just watching.

Yeah, yeah, yeah.

Great. Okay, so those were the models.

These are interesting. We were also able to distill them into really, really tiny models. This is, for instance, a long sequence on a very, very tiny one. You can see it makes a few more stupid mistakes, like it does things that are not as optimal.

I haven't seen anything yet.

At the beginning, it was running into a wall for a bit. Yeah, exactly.

I mean, I do that too.

Yeah, yeah.

It's looking—I mean, it's doing pretty well.

Yeah. Again, all these models are running completely in real time, so there's no—

Okay, so I was thinking, your main model does real time anyway. What's the goal of distilling? Is it cost or—

Yeah, parameters?

Yeah.

Yeah. This is the interesting one: It peeks around the corner. That's what we mean by the first-person reasoning aspect. Humans actually sort of simulate the optical dynamics of their eyes and how they actually see, right? You've seen all this.

Yep.

Exactly. Even in the real world, with, for instance, YouTube data, you have to first solve for pose estimation. Then, once you have pose estimation, maybe you do something like inverse dynamics, where you basically are able to somehow label some of the actions that you're seeing. And then you still have to account for the optical dynamics of where your eyes are actually looking before the decision, because there's just 3 levels of information loss.

When you're playing video games, you're actually simulating the optical dynamics with your hand, right? And I think that's why games are a better representation of spatial reasoning initially than YouTube videos, for instance.

Okay, we're in the GI offices with the CEO. Welcome.

Thank you.

Thanks for having us in your office.

Yeah, excited to be here.

If I'm in New York and you've done one of the hardest raises of the year, I have to come and visit. Thanks for taking some time on the weekend.

Yeah.

So, you've raised $133 million in seed funding for General Intuition. Most people didn't know about you, I guess because GI is new, but more gamers would have heard of Medal. And before that, you ran Medal, probably the largest gaming video platform. What's your reflection on that journey to now being an AI founder?

Yeah, you started off RuneScape.

I grew up with Tourette's. I spent most of my time as a teenager coding and playing video games. In that sense, it doesn't feel that much different.

I started the largest private server for RuneScape, worked at Doctors Without Borders for 3 years on Ebola, and then worked on satellite-based map generation for disaster response, which was already very AI-adjacent. I built some models back then and then started Medal, which became one of the largest social networks in video games.

I've always been kind of AI-adjacent. I'm a self-taught engineer, so for me, the modeling itself always felt a little foreign. I actually had to take a ton of classes over the summer and early this year to get better at it, because it still felt like I was really, really good on the infrastructure side. I had written our transcoders for Medal myself, so I was very, very familiar with CUDA, the GPU side, and all the video infrastructure that we were using for this stuff. But the modeling side itself was still quite foreign.

Luckily, obviously, I have really, really good co-founders. They essentially put a bunch of coursework together for me to complete so I could get really, really good at understanding the fundamentals better.

I think for me, I had seen inside of the labs that had really, really good leadership with fundamentals at the top, and also the ones that didn't. I think the ones that did were just much better. And so, for me, I wanted to be more like that.

In that sense, it was at first very foreign, and then now I feel pretty comfortable with everything. But, yeah, I think there's a lot to be explored starting in video games, and also reverse engineering. The interesting thing about reverse engineering is that it kind of teaches you to look at problems very differently. It's the ultimate form of deductive reasoning, in a way.

This is just how I operate, and so for me, it's been a really, really interesting journey. I don't claim to have any of the credentials or skills that some of the other guests have had on, but hopefully it will make for a good time.

Yeah. Well, your co-founders definitely bring a lot of that different ability, and you bring a lot of the gaming expertise with you. We'll see what I bring to the table.

Yeah.

Just a little bit of history of Medal. Let's establish Medal for those who don't know. You have more active users—concurrent users—than Twitch, something like that?

Yeah. On the creator side, I think the reason is because Medal is a lot more like Instagram than it is like Twitch. The way you think about Medal is that it's a native video recorder. Unlike something like Twitch, where you actually have to use other software to record and stream to Twitch, it's not streaming software. It's actually video-recording software.

A lot of gamers love to put things like overlays on top of their footage. As a result of that, we have the largest dataset of ground-truth, action-labeled video footage on the internet, by maybe 1 or 2 orders of magnitude.

Yeah. What's an example of an overlay? I usually think of it as a camera overlay.

Also controller overlays, for instance, if you're playing on a console.

Yeah, like Flight Simulator. You get the joystick and all the controls.

So you get the actual actions that people take inside the games, as well as the frames of the games themselves, which is a loop. It's essentially: You perceive, then you act, and there's a state update; then you perceive again, act, and there's a state update. That's precisely what you use in order to train these agents.

Yeah, it's almost perfect training data. You were showing me in the demo—we'll show some B-roll here—how you don't log keys. It's very important for you to log actions. When did you figure this out?

Maybe starting a year and a half ago.

And we realized that figuring out this side of the research for us was important. We very much never wanted to be in a position where we eroded privacy or something like that. We never wanted to actually log a W, A, S, or D, which, for researchers, the fact that we don't do that often sounds strange. Like, why wouldn't you do that? But I think for us, the privacy—

Yeah, I think a lot of the researchers hadn’t quite understood yet that you can actually get away with just recording the actions. The reason is that, at training time, having the actual keys is noise anyway. If there is text on the screen and you would want to, in theory, make that part of the training, reading text from a frame is really easy.

For us, if we can do that, we convert it—basically, you hit the input, and we convert it to the actual action. We had thousands of humans label every single action you can take in every single video game over the past year and a half, which is an enormous amount of action labels.

When you act, we get the actual action itself. At training time, you can, for the general set of that game, convert back into computer inputs if you want to, but you can never do it for any individual person. From a design perspective, that was important for us. We figured all that stuff out.

Then we started pushing features as well. Gamers already love being able to navigate their clips by things that happened, so we have an event-capture system. We also have overlays, where you can render the actions on top of your clip. We developed those in tandem with the feature set itself. When world models became a thing, it was very clear that all the data for this was precisely that sequence, so we were able to be first to market, recruit the best researchers, and start a lab.

Yeah, that’s incredible. One more question on Medal before we move forward: It’s been 10 years. What is—I don’t even know how you roll something like this. I’m just curious about the opportunity to ask you what really worked and how you became so huge, because you’re not the only one. I’m sure it’s performance and everything, but—

A few things really worked. I think the first was that a lot of our competitors were focused on solving the social network and the recorder at the same time. Our bet was really that we could get so many people to record with us that we could bootstrap the network on top of that, and that worked.

While everyone was distracted trying to bootstrap a social network, we were just focused on building a really good capture tool. Then we got tens of millions of people to use that, which allowed us to bootstrap a network on top of the sharing behaviors. We already had the profile behaviors and the sharing behaviors, obviously, but the actual content-consumption and sharing pieces really only came after we hit critical mass.

It was actually in the early days during COVID that the network really accelerated. Fortnite happened, which was really important, and I think the fact that Discord existed made it quite a different time from when other types of networks had launched. Discord was essentially the connective tissue between gamers that had never really existed before. I think that combination of things really made it.

We also built a product that, with most video recorders, you have to remember to start and stop the recorder. You have to go into the application, hit Start, and then start your game. Maybe you’ll play games for 3 hours, then you’ll close the game, and then you have to close your video application.

Then, well, you have to process a multigigabyte file. Then you have to upload it somewhere, and this was a pain for people.

So what we did is run this kind of recorder. When you hit that button, it does a retroactive video recording. All the recording is initially in memory, and when you hit that button, it exports only that sequence to disk and syncs it to your phone. That became super popular.

It also means that you’re not behaving or acting differently because it’s always there, and you can just export whatever happens, which is also very helpful for training, obviously.

You were the first to do that?

Yeah.

The thing you were explaining just before this is similar to how Tesla does its bug reports, right? You’re driving, disengage Autopilot, and they’re like, “Well, tell us what happened.”

Exactly. See, Tesla doesn’t want to train on the 10 hours of you driving through a desert where nothing interesting happens. You have the clip button on the steering wheel. Something interesting happens while FSD is engaged—and I’m not sure if you can use it without FSD as well—and you hit the clip button. It basically uses that precise sequence to mark what happened, which is more helpful for training because it’s more unique as training data.

Yeah. I mean, one thing—we’re going to get to this on the agent side—one thing that does pop up is that a lot of life is boring. A lot of life is boring for me. A lot of playing games is doing the boring stuff that is not captured.

Yeah, somehow using the generalized fight.

Yeah. [laughter] Yeah, it makes you think, right?

It makes you think.

It makes you think. Yeah, it’s also quite interesting. You showed me the models—what happens when you increase the size of the context window and how behaviors are actually largely shaped by the size of the context window. That, to me, was one of the most interesting parts of the research. It made me think about our own behaviors in a way.

Yeah.

Let’s also talk about forming the team. On your website, you’re 12, with 3 co-founders. Let’s talk about how this team came together, because you yourself don’t have that academic network. You manage the people.

Yeah. I started reading all the research papers. By that time, I was already pretty deep into having a decent understanding of language models and transformer-based models. Genie and SIMA were really interesting.

SIMA in particular was interesting because they took 10 games, and they had a graphic where you could see the precise actions inside those games that they mapped. I believe they found something like 100 actions that also exist in the real world.

What they did, specifically for navigation, was a 9-to-1 holdout set. They trained an agent on 9 games and then had it play the 10th game, the holdout game. They also trained a specialized agent just on the 10th game and compared how well they did.

If I recall correctly, the 9-game agent did roughly as well playing the 10th game—specifically on navigation—as the 1-game agent. That was really interesting to me because that’s precisely the type of data that we had.

For us, the thinking was, “What if we did exactly what LLMs did? What if we used this?” LLMs were trained on predicting text tokens—words on the internet. What if we predicted action tokens on essentially the equivalent of the Common Crawl dataset, but for interactivity?

Vision input?

Yeah, action output. Correct. That’s it.

Well, I think I’m going to double back a little bit to a question I had. One of the reasons why I thought you would prefer keyboard and mouse over actions is that the action space is potentially unbounded, right? You can jump, walk left, walk right, but then also look up, look left, crouch—it’s unbounded. So it’s huge, isn’t it?

Yeah, I think that’s a problem.

Yeah, there are benefits to the action space being small to start with. We’re going to start with anything that you can control using a game controller, but long term, we want to predict maybe action embeddings and have models sit inside a general action space, to be able to transfer out to other inputs as well.

Got it. Yeah. Okay, and then let’s keep going on the research side. So, Genie. And then the co-founders?

Yeah. So there was the DIAMOND paper, there was Genie, and then there was SIMA. The DIAMOND paper was really interesting because they had managed to get this world model called DIAMOND running on a consumer GPU. I believe it was a 4090 at 10 FPS, and you could play it.

They did that on roughly 90 hours of data—95 hours, I think it was 87 hours or something like that. It was incredible that they had something playable on so little data. So I cold-emailed the entire group of students and told them, “Hey, I think we have this thing.”

Right when that happened, a lot of the labs also started understanding what we had. Multiple labs tried to bring us in in various ways, and they were part of that. They were seeing that happen, and I think for them that also solidified how real it was.

And then, when we chose to do our own thing, we initially thought that we were going to have to just work on world models. We thought the main benefit of this dataset was world models like Genie. What we didn’t realize at the time was that we had so much data that we could essentially do these world models in parallel, take the equivalent of the LLM approach, mostly on imitation learning, and then use the world models after that to get into our RL stage.

And eventually getting rid of imitation learning, something like that?

I mean, ideally, you get rid of imitation learning—yes, imitation learning—but we essentially realized that we could get so far on just imitation learning.

The way to look at it is that we essentially have the internet, or Common Crawl, if you will, and every single lab is trying to simulate that in order to get similar data to train their agents. For us, the reason why we stayed independent and just did our own thing was that we think we essentially leapfrog every single company that’s forced to either consume world models or build world models.

We’re taking this foundation-model bet for spatiotemporal agents and putting ourselves in a place where we have a lot of customers years before any of the labs even get there. Maybe the most similar comparison is what Anthropic did with code. Anthropic focused really hard on nailing the code use case, and its models are incredible for it. A lot of its customers use them for that.

We just want to become incredible at this spatiotemporal-agent use case. Likely, that starts in game simulation, and then, using world models, we can start expanding out to other areas.

So would you show me a little bit of how it does generalize up?

Yeah, it does, although games are kind of a common area.

Games and simulation.

I would specify game engines in particular. So even if you’re, for instance, simulating human behavior in Omniverse because they’re trying to create better training data for factory floors, you can use it.

Yeah. Maybe Meta has a similar dataset because of the Quest.

I never really asked them. I never really looked into the Meta Quest specifically. So you need a few things. You can’t just have lots of companies that have, maybe, recorders; you also need the public graph. Otherwise, you can’t train on the data. You can’t train on people’s private videos that they have saved somewhere, right? I think you need the social network graph components because these videos need to be on the internet to train on them.

No, to train on them.

I mean, generally, people don’t want to train on things that live on your device, usually, right? You can’t train on anything that lives on your device. You actually need to go and upload it and do your thing, right?

For Meta specifically, I think VR is still pretty small in scale. The number of environments in VR that have consumption at scale is probably in the hundreds, whereas on PC it’s probably in the tens of thousands. So you get a lot less diversity.

The 3D input space of VR is pretty interesting. We see some of this, too, obviously. I do suspect Meta will start using these types of things, but it’s unclear to me whether they can get to a similar scale of data or diversity of environments as we can.

Yeah, there are a lot of challenges there. I want to take this in a few different ways, but I guess let’s fill out the papers. Maybe one more to mention is T-REX. I actually interviewed the authors, but that, too, seems like the particular insight that brought it overseas.

Yeah. Anthony Hu, who led the research on GAIA-2, is also one of the engineers who joined our team. So it’s all the core contributors to DIAMOND, and then Anthony. We just had 3 more researchers join this week. It’s been a good week.

I think a lot of the approaches in GAIA-2 were heavily inspired by DIAMOND. Vatsa, who was one of the authors of DIAMOND, was already at Wayve by the time that I emailed them. He also realized what this was and realized that you could scale world models to a much larger scale, so he decided to make the leap as well.

I think everybody who sees the dataset makes a leap, but it takes a while to wrap your head around it because it’s like, “Oh, it’s video games.” Intuitively, it doesn’t make sense. When you actually understand it and see how we’ve been able to transfer it to physical-world video and things like that, then it makes sense, and everybody tends to jump. I would call it video games and call it RL. So then, yeah.

If I lived in San Francisco, maybe I would. Yeah. [laughter] Just a quick note, because we actually cover all these papers in Latent Space Club.

SIMA 2 did not seem to have as much impact as SIMA 1, and I don’t really know why. They did a lot more work. Genie 3 had a ton of impact, but I also felt like, because you could play with the model yourself, it just seemed like an extension of all those things. I guess, any quick takes on SIMA 2 and Genie 3, which both came out this year?

I’ll talk about SIMA 2. The steerability of SIMA 2 was, to me, the most impressive part, because getting the action sequences and the text conditioning right is quite hard to do. It’s also quite interesting that they can use Gemini as part of the flywheel. They can scale this orchestrator as an independent, almost like a puppet master, if you will. In theory, Gemini could orchestrate many instances of SIMA.

That, to me, is the most interesting part. I tend to agree that our models will initially be used with an orchestrator VLM of sorts that’s managing instances and instructing them. SIMA showing that you can do this was fascinating.

They didn’t just have text conditioning; they were also able to use drawings and markings to indicate where to go. They took an interesting end-to-end approach, and I look forward to seeing a lot more of it.

But you talked to them? You saw it? Is that the one with the collaborative room?

Yeah. We’re very friendly with DeepMind. We like them a lot. I saw the team not too long ago, and we’re big fans of their work.

The headline that came from Alex Heath’s coverage, really, is: “You’re the biggest bet that Vinod Khosla has made since OpenAI.”

How did that conversation start?

Okay, so, from what I know of his style—and maybe I’ll get slapped on the fingers for revealing this or whatever, so forgive me if I’m wrong—he asks you to draw a 2030 picture of your company. I think he just picked N+5 years, but whatever, I don’t know.

I did the same to you. Yeah.

He asks you to walk that back from first principles, all the way from today, and he expects you to do that flawlessly, where he can challenge any assumption or any part of the vision. He asks you questions. He has a very technical background, and he also has a bunch of technical people on his team.

He truly backs people who have these very large visions based on the vision and their ability to defend it. That’s what he did for us, and I think that’s why he made that bet. Through this question, he gets to know a lot about how technical you are and how well you think from first principles.

If that vision isn’t connected to something real, it’s very easy to suss it out by asking good questions. Then he just backs you fully. He really gets in your corner if it’s the right fit. They’ve been incredible partners. They’ve opened so many doors for us.

I had to ask the question. It’s a very notable story. Obviously, a lot of work went into it, and it’s also worth it when you come out the other side.

One of the things I also wanted to—I guess I kind of asked this question out of sequence—but one of the things that excites me about talking to you is that there are a lot of people like you who are founders of businesses that, along the way, have a ton of data, and yours happens to be highly valuable. You pursued that before deciding to do an independent journey, and you also talked to other companies about potential licensing or acquisition. What are your learnings from those periods?

One version of this is very simply: how do you value data?

I don’t think you can value it unless you actually model it yourself and see what the capabilities are. That’s my real takeaway.

You say “model,” but train a model.

Yeah, but that’s obviously not doable for everyone. My general advice would be that as model capabilities increase—and these foundation models are also very, very good at labeling—you’re going to need less ground-truth data. You can do more model-based data generation or synthetic data generation.

If you’re going to do large data deals, I would recommend trying to get a large chunk of equity in the company that you’re doing it with, if you can. A lot of them won’t do this, but I think that would be the best option. Or just do the research and figure out what’s actually possible.

In our case, we were quite lucky in the sense that this is actually foundational data.

Right. That’s not true for every dataset. We just happened to hit a particular gold mine. But you also did the acquisition thing, like 1 or 5 years ago, yeah?

Yeah, that’s the thing: you have to be grounded. A lot of what’s interesting is that you can also look for whether scaling laws already exist for your data type. For video, there were some, but for these input-action-labeled datasets, there really weren’t any.

The other question is: does it go into LLMs? Does it go into world models? What type of model is it going to be used for? That’s an important thing to know.

If you’re having these conversations with labs about data, make sure that you actually understand what it’s going to be used for. That’s a very good way for you to make the decision yourself about whether you want to pursue it.

A lot of them won’t tell you that. In that case, you generally just don’t want to do it, because in our case, we really cared that there weren’t going to be competing products built with game developers. We didn’t want to bite the hand that feeds us, and we’re part of the games industry.

So those questions are normal, and we eventually decided that he just had the data, so we were going to do it ourselves. That’s when the rest happened.

Yeah, and he assembled the team that didn’t take advantage of that. I feel like you’ve aligned a lot of stars in order to make GI work.

Yeah.

What about other data founders at the beginning of the journey?

Yes. Or, I’m a data founder—founders who happen to have data but have a main business, right? There are 2 sides to this. It’s really easy to be super naive about it, and I had a lot of people tell me initially, “Oh, it’s not that valuable. You’re just making this up.” For me, doing the work and actually understanding it myself was a really big part of building the confidence to go start the company.

But a lot of times, it is true that model capabilities increase so quickly that certain data just isn’t needed anymore. I think it’s really important to get people to do the work so that they can make these types of distinctions. My recommendation would be: go build models with your data, see if you can create any capabilities that aren’t clearly already there or on the path to being there, and then figure out where you go.

Yeah. I didn’t want to ask this earlier, but you gave me the opportunity. When you say “do the work”—do coursework and all that—and your co-founders gave you some homework, is this like some books? I mean, Coursera?

No. This was François Fleuret. He has a little book called The Little Book of Deep Learning, and he also has a full course that he’s published on his website. I went through the entire course over the summer. I believe it’s something like 30 or 40 lectures, with take-home projects and things like that.

I would recommend that anybody do this. It goes through the history of deep learning and the topology. It takes you through linear algebra and calculus, eventually ending up with the chain rule. By this time, you’ve done all the more important concepts. It takes you through how to create neural networks using the concepts that you’ve learned.

Wow. This is super first-principles. I’ve had the opportunity to spend some time with him as well. He’s one of the most first-principles people I’ve met in my entire life. I’m convinced. I actually asked him, “Why did you create the course?” He said, “Because I thought all the other courses weren’t right.”

Because he is so first-principles, he can only explain things from first principles. Everything you see in how he explains this is from first principles, including the history of deep learning itself. It’s part of the course.

Yes, he goes through everything, and by the end of it, I now have a pretty good intuitive understanding of how everything works. Obviously, I’m still—I like to describe it as being the guy who just got his driver’s license. I can drive the car, while my co-founders are the F1 drivers who have done this for years. They know where all the gaps are, and I enjoy getting to learn from them.

The cool thing is also that world models are a very new space, so I get to bring ideas to the table that no one thought of—not because I’m great at this, but because it’s such a new space that people just haven’t tried things yet.

So, to get a handle on the definition, what are world models to you?

In a video model, you might predict the next likely sequence or the next most entertaining frame. What world models do is understand the full range of possibilities and outcomes from the current state. Based on the action that you take, they generate the next state—the next frame.

It’s a much more complex problem than traditional video models. To me, it’s a world that is accurately generated based on the actions that you take, as a result of what’s already been generated.

And just to fact-check that: it needs to understand physics. If I’m building a type of material, you need to understand how it interacts with some other type of material.

Yeah, I think the interactions are the most important part. That’s one of the reasons why world models are so fascinating. One of the things that I did when I was studying over the summer was try to build a super-rudimentary PyTorch-based physics engine. I would not recommend writing a physics engine in PyTorch for obvious reasons, but I wanted to do that because it’s differentiable, so you can generate the model—

Yeah, exactly. You can, and then you can train.

And so I wanted to understand from first principles. I got so many people asking me, “Why aren’t you just simulating or generating this data?” I really wanted to understand why.

The most important thing I figured out was that the computational complexity of simulation goes up really rapidly with 3 variables. First, the number of agents in an environment. Second, their degrees of freedom—their individual freedom.

Yeah.

And third, the information that each action reveals. For instance, if you have a text action or a speech action, the environment can change so much based on whether you say “water” or “fire” that the outcomes are going to be completely different in terms of how a human would behave in that type of situation.

It goes up so quickly with those 3 variables that at some point, you just want to make a maximal bet on either video transfer or generating these environments using world models, because that type of complexity is incredibly difficult to simulate. But it’s already very present in a lot of the video pretraining that goes into these world models.

For us, it’s more about making a maximal bet on video transfer and interacting with things that are difficult to simulate. The steerability with text is also really interesting, rather than betting against simulation or something like that. I think there’s still a large market for traditional simulation engines, specifically in areas where video is really hard to get.

Is this exactly what the big labs are also saying when they’re talking about that?

I honestly haven’t talked to the big labs since we started working on this ourselves. I think people are more reserved about what they share with us.

Yeah, of course. With that said, how would you contrast your version of world models with V-JEPA?

I don’t know exactly what Yann LeCun is doing today. My understanding is that it’s based on the V-JEPA-like approach. So I’ll start with Fei-Fei Li. I think what’s really interesting about Fei-Fei Li’s approach is that you’re able to reuse the splats in game engines and in things that let you stay in a verifiable domain, which I think is a really interesting approach.

However, my understanding is that they’re currently not interactive, which, in my opinion, is the whole point of world models. They’re environments, and they’re great environments. From a business perspective, I think they picked an important part of the toolchain, but to me, that’s not really a world model. My guess is that they’ll get there. They’ll start generating—

Yeah, they just released it.

Yeah, exactly. Exactly. Fei is one of the founders of the entire space, so I think it’s going to be really interesting to see what that interactive piece looks like before I really judge their approach.

I think we interviewed her with Justin Johnson, her co-founder. He was more focused on the physics side of things and game rendering, trying to have good views. I do think that, basically, with the splats, if you just add more dimensions and, I guess, the forces acting on them, then you get 4D out of the box. Basically, these are virtual atoms that then have all the physics applied to them.

Yeah, I’m excited to see what that looks like when they actually release it. It’s really hard for me to comment on anything. I really like the frame-based approach because all of our video—or all of our training data—is in this format.

Yes. We actually asked them about this, and they said it’s possible, but they’re choosing the splat.

Yeah, yeah. You can also go from splat to frames, right? I’m sure you can. It wouldn’t be easy—you’d have to actually render out the environment. So, it’s not going to be a simple problem, but in theory, it has to be something that you can do if you really wanted to.

It’s almost like having a more ground-truth, 3D representation of the underlying world, so I think it’s an interesting approach. It might be overkill. You’re also dealing with a much larger number of degrees of freedom in the output space, so who knows how well it scales.

I like the fact that these video models also use autoencoders. They can have the world models predict something much smaller, maybe in terms of—

Resolution or size.

Yeah, exactly. Then you can use diffusion upscaling or methods like that to enrich it.

And so I think that world models allow for a much more controlled space—a space that we know really well. I'm not suggesting their approach is wrong; I'm just saying this is what we really like about it.

Honestly, Yann LeCun's podcast—I don't remember which one it was, but it was a long time ago—where he basically proclaimed LLMs to be a dead end was one of the things that inspired me to do this. I think this is very much the consensus among world-model people: basically, everyone here starts with their LLM and just builds world models. I would say that, for the main perspective, I asked this exact question to Noam Brown from OpenAI, and he was like, “Well, learning the world models,” right? So there are basically different things in our system.

Yeah, I'm not one to proclaim LLMs are dead ends personally. I think they're actually quite useful, particularly as orchestrators. The way I think about it is that, as humans, we had a sort of 3D world, then we invented text as, in a way, a compression method. We invented text in order to communicate with each other in a common way, in a way that actually compresses all this information that we're perceiving in 3D space into a single sequence. I think that allowed science to emerge; it allowed so many parts of literature and so many parts of the world that we cherish. So I think it's a critical part of the whole picture.

I also agree that it's very clear that they build sort of internal, implicit world models inside LLMs, and so I think they'll be very helpful as orchestrators. The problem is when it comes to generalization. I think text has a generalization backbone, when most of the pre-training is text, or largely text sequences, but then I think you want that backbone to be more spatial and plural in nature, and also just have text as one of the components of that.

And I think the actual argument against LLMs is also, for instance, the autoregressive nature of the prediction itself. The fact that it's running the entire output through the transformer in order to predict the next token doesn't map well to the environment in the real world, which is continuous—it's always changing—and LLMs kind of just forget about that. I think a lot of the argument is in that, right? So I think the fact that text doesn't necessarily generalize well to sufficient temporal context, and the autoregressive nature of the prediction and using text for that—those are the 2 main arguments. I think text prediction is just one of the actions that is going to come out of these policies and world models. I think speech and text generation will just be—

One of the actions that can be a part of that.

I think there will just be labs coming at this problem from both sides. Everyone ends up in roughly the same place, and the same place will be whatever people think is cool—whatever the consumer gets.

Whatever is closest to AGI.

Yeah, and so I don't think there's a clear answer. I think it's really interesting to come at it from the world-modeling side, but it's also because we have to, because text is largely commoditized; we can import all the text. I think it's interesting and tempting. It makes sense that you can probably recover—it's sort of like you're taking a step back, starting your branch of the ML research tree, but you might guess that you just end up recovering all the other text stuff emergently.

Yeah, yeah. We can import a lot of that research, right?

That's really cool on the research side. Let's talk about the stuff that GI is producing—more like the biggest research and product outputs. You mentioned the word “customers.” What are your current customers?

Yeah. So we're already working with some of the largest game developers in the world.

We're also working with game engines directly. Really, what we're doing at the moment is replacing essentially the player controller inside a game engine. Anything that you're currently doing—maybe behavior trees or things that you're deterministically coding—we hope to replace with a single API: you stream us frames, and we predict actions. That can be inside an engine, or it can eventually even be inside the real world. Hopefully, those are then also steerable. The models that you saw weren't steerable yet, but I think we want to get to a point where they're fully steerable.

Well, steerable means, like, “I want you to figure anything else out in the frame”?

Yeah, I think it's text conditioning on the generation. So, yeah, the ability to—you're right, we want to get to a point where you can generalize, and that's why it's called General Intuition, where we can mimic the intuition of all these gamers in humanlike behaviors in any situation.

As I mentioned, the lab is also named after this quote from AlphaFold: “Wouldn't it be amazing if we could mimic the intuition of these gamers, who are, by the way, only amateur biologists?” On his path to AlphaFold, he tried to get an AI to play Foldit to generate a lot of data for AlphaFold.

And so for us, really, the north star—what we hope to get to one day—is being able to represent scientific problems in 3D space and then have a spatiotemporal agent capable of perceiving that space and using, hopefully, the text-reasoning capabilities that LLMs have today, in addition to the spatiotemporal capabilities, to be able to work on the other side of that problem. So that, for us, is sort of the north star.

That's why we're trying to be hyperfocused based on core workloads, the same way that Anthropic was hyperfocused on code, and use that to then get into organizations and expand from there.

Just as a side note, since you mentioned Anthropic, any idea what they did to solve this?

No. Of any lab, I probably know Anthropic least, to be honest. I admired them, though.

Yeah. Well, the current working theory is that they had a super-lucky roll of the dice, [laughter], and then it compounds from there. That sounds like a nice story. I'm sure it's not that.

Why do game developers want this?

So, if you're a game developer, how well you're actually retaining players is decently dependent on how good your bots are. If you're logging in at an obscure time—let's say 3:00 a.m. in America—and your player liquidity is low, then you need really good bots to keep those players engaged.

Is this known? Is this a thing?

Yeah, for sure. Like Fortnite and whatever.

A lot of human players, yeah. And so, as a human, do I want to play against bots?

Usually, it's not just bots. It's players mixed in with bots, because you don't want to play just against bots, but it's better to have a full game than to have an empty game.

Yeah. And so I think as long as it's part of the environment, I think it's okay. That means you also have to sort of grade that skill level.

Yeah, yeah, which we can do, because we know exactly how good people are at these games. I think for us, bots are kind of step 1. What I was showing you is that we're building a general agent that can play any game in real time. But really, that extends into all of simulation, right? Like, in GTA V, for instance, people are genuinely role-playing real life.

Right? And so they're actually behaving in quite aligned ways with the goals they set for themselves. So you have all these examples represented in video games, right? You have Truck Simulator, PowerWash Simulator—

PowerWash—

PowerWash Simulator, where the behaviors that you'd want in nature to be able to perceive are all there.

Yeah. It's really like how seriously some gamers take truck simulator. If you haven't seen this, you should watch it. They buy the whole truck-driving setup, and they're doing the job of a truck driver. What I mentioned to you is that we have more people at any given time on Medal playing with steering wheels in truck simulator and these types of games than Waymo has cars on the road. It's a ridiculous stat, but it's true.

I used to think that, for self-driving, you kind of just need to play a lot of GTA V. I'm bad at it.

Our bet is not that we can zero-shot any of these things. It's just that the next self-driving company can maybe collect 1% of the data, because, for instance, clips already self-select for negative events and adversity, right? A lot of our dataset consists of the highlights.

That's really precisely what a lot of these companies spend their last 20% doing.

Right, and I think that's the main argument. If you're another company that's looking at what we're doing, I think the thing that people won't understand is that anything you're currently doing in pre-training—as long as your robot can be controlled using a game controller—we hope that we can move to post-training for you. So our bet is not that we can create the next self-driving car company.

It's just that the next self-driving car company hopefully only needs 1% of the data, or maybe 10% of the data—I don't know—to be able to deliver a really good product.

Yeah. Yeah. The term that comes to mind a lot is active learning. I don't know if you've used or identified with that. It got less cool for a bit, and now we're seeing the uptrend, which obviously you have the best data set for: the sort of high-intensity, or, as you'd say, negative, events. It could be negative in all parts of it.

Yeah, for sure. I think “negative events” is just the most common term that people use. If you're Tesla, you want the crashes.

Right, right, but it's only gaming. Yeah.

So, you know, the model that you saw obviously had really, really incredible moments, and that was because it had a large representation of people at their best—

Yeah.

—and worst. Yeah.

Yeah. Yeah. Amazing. Okay, cool. Anything else on the customer-development side that you want to touch on?

Yeah. We're also already working with robotics companies and manufacturing, but again, the key is that the robot has to have gaming inputs. Our bet is not that we can transfer over to higher-DOF robots rather than the keyboard and mouse. It's really just that we can move the hard work of pre-training, hopefully, to post-training.

Yeah. It's kind of like the foundation model that is a very good basis to start.

Yeah. You're going to give us frames and likely some text.

Or you'll license the model, too, because they've been wondering.

Yeah. Our business model is initially going to be an API, like the Anthropic API. You also saw, for instance, some of the video-labeling models that we've been able to develop. The goal is for any company to be able to take in their video data as well, and we can create, first, obviously, custom versions of the policy for your agent. If that doesn't work, we've already been working with a customer where we distill a model, and they turn that into a product for themselves.

So people can engage with you at the agent-level API. People can engage with you at the model level. Can you also buy data?

No, we don't sell data.

Okay, cool. So that's the business. Is there a world in which—I mean, I think this is on your landing page—if you are, you know, a frontier lab for world models, is there a world in which there's a more application-layer thing that comes out, like a ChatGPT for whatever?

Yeah. You're going to see us launch a few things on Medal itself that are going to blow your mind as a result of this agent. I'll leave it to the imagination for now.

Yeah, on the world-modeling side, one thing people underestimate is that Medal is already one of the largest video-consumption platforms as well. People watch millions and millions of videos a day. World-model-based entertainment and things like that—while it's not a focus for us right now, I think that on the consumer side, we have the ability to move very, very quickly here and get it integrated in a way that I don't think anyone else can.

Yeah, you could theoretically do video generation like Sora. What's the Meta one? Meta Movie Gen—not Reels? You could theoretically generate clips that nobody would play, but, you know, it's a different device.

Yeah. I think for us, the games being so human-centric is a really big part of what makes us special. I actually just don't think that would work. One thing that we are really excited about, though—and I'll give you one sneak peek of what we're thinking about—is: what if you could literally replay any of the clips that you have inside a world model, or your friends could play them? I showed you a model that already took part of your clip as context.

Instant replay. Enter that world.

But it's also how we go from imitation learning to RL, right? Because it's part of our research roadmap to make every single clip on Medal playable. So who's to say that that doesn't apply to just the actual clips that you take?

Yeah. Yeah. Interesting. Can you say more about the RL potential?

We describe Medal as the episodic memory of humanity in simulation. When you take a clip, really the way to think about it is that you get the highlight of what may be 3 hours of playtime. You maybe get 2 to 3 minutes of the things that were the most out-of-distribution, right? It is genuinely your episodic memory of that playtime in simulation—the things that you most want to remember and share.

We want to be able to load them, and this is the work that's being done. The reason why we built world models is that every crash that you run into in Euro Truck Simulator, American Truck Simulator, or a driving game, we want to be able to capture. This is the work that Anthony Hu is doing. Again, these are ground-truth labels, so we know precisely the actions that lead up to the negative events.

They're also title-labeled when people upload them onto the platform. They say, “Okay, it's a crash.” We can select all these events, and if we can put them inside a world model, we can train reward models to reward based on how you perform in clips that actually contain negative events, for example. For us, it's very much about creating this LLM moment on imitation learning, but actually making every single clip on the platform playable at billions-of-clips scale is how we go from imitation learning to RL.

Cool. We covered a lot of it. Is there anything else that you want to do before we grapple with the long-term vision stuff?

Yeah. I think for us, this is a very, very ambitious long-term bet. We need the best researchers in the world who want to work on this stuff. It's really exciting not being extremely data-constrained. We get so many learnings every week that we didn't think were possible, and it makes it a joy working here.

Also, because we have such large data moats, we don't have to be as concerned as the LLM companies about publishing, because no one can replicate the models. For us, we really want to bring back the original culture of open research, which is why we did the partnership with Kyutai in France.

We just announced our partnership with Kyutai, which is an open-science lab in Paris, one of the best research labs in the world. Eric Schmidt, I believe, funded it, in addition to some French people. They are essentially acting as the partner that's currently doing a lot of open research on the data.

We also want to partner with universities because we do believe this is the frontier, but it's so data-constrained that everyone has their hands tied behind their back right now. We want to help fix that. For instance, we want to work with universities to build negative-event prediction models for maybe trucks in India, using all the truck data where these crashes occur.

We have all these things that we know we can do that we just haven't had the time to do. If you're listening to this and you're maybe an academic institution or something, and you want access to some of this data and research in an educational-research fashion, I think we're quite open to doing that because we want to educate people.

Other than that, we just want to work with the best infrastructure and research engineers on the planet as we're going into scaling runs that have thousands, tens of thousands, and eventually hundreds of thousands of GPUs.

So what does GI become in 2030?

In 2030, we want to be the gold standard of intelligence. Any sequence long enough is fundamentally causal, right? So by nailing spatiotemporal reasoning, you go after the root problem of intelligence itself.

What the world looks like is that we want GI models to be responsible for 80% of all the atoms-to-atoms interactions driven by AI models. I sort of group the stages of AI into 3 stages, and I credit Andrej Karpathy for teaching me this: bits to bits, atoms to bits, bits to atoms, and then atoms to atoms.

In the atoms-to-atoms stage, I want GI models to be responsible for 80% of all atoms-to-atoms interactions driven by these models. The reason for that is because we were able to unblock intelligence so quickly in robotics. Intelligence is the bottleneck, so supply chains actually converged on gaming inputs as their primary input methods, and they converged on simpler systems that let us do a lot more, a lot quicker.

We are essentially taking the 80/20 market approach. You have lots of companies that have specialized, maybe humanoid robot operating-system stacks, that are the other 20%. I want to be responsible for 80% of all the atoms-to-atoms interactions driven by these models, be the gold standard for intelligence, and maybe be 100 times larger in simulation, because I think simulation will actually be the larger market initially.

In simulation, you have very few constraints. Also, from a safety perspective, simulation is much easier.

So I think a lot of the takeoff initially is in simulation. A lot of the simulation use cases, like the scientific use cases I mentioned, I'm really, really excited about. So, yeah, 80% of atom–atom interactions will be coming downstream from these types of physics world foundation models, and then 100x more in simulation.

Yeah, yeah. It reminds me a lot of what Mark and Priscilla from the Chan Zuckerberg Initiative are doing with virtual biology, because you can do a lot of simulation.

Yeah. Oh, you can do it a lot faster with AI.

Amazing. Thank you for inviting us to your office, and thank you for sharing a little bit about your training.

Thank you. Yeah.