物理AI如何跨越语言、视频与动作学习——Ming-Yu Liu

This car is about to make a left turn. Can you envision the trajectory the car is taking? And the video generated.

Guess what? That road was never filmed. It was made by NVIDIA’s Cosmos 3. It reads video, simulates roads, and is learning how to handle objects. Ming-Yu Liu leads the research, and this is a paid partnership with Nvidia.

Now, Cosmos 3 takes text, video, audio, and actions as input. Let’s start with the simplest case. You ask the model what happened in the videos: how many cars are in the video, or whether the robot succeeded in completing an action. “Okay, there are 3 cars, and the robot succeeded in picking up the apple and putting it in the basket.” When you have video and text as inputs and generate text, it becomes a vision-language model.

So why don’t we flip this thing on its head? The model stops describing the scene and starts generating it.

Cosmos Dreams is a set of skills to build a closed-loop simulator, not just for self-driving. I can employ 100 drivers, drive 100 cars, go to different intersections, and figure out whether it works better, or I can use this simulator to verify the accuracy. You are not limited by the size of your fleet. Anytime you need to do a lot of testing, if you have the compute, just launch it.

Okay, so what if we took the same idea but applied it to an even harder problem, like robots?

With navigation, you don’t want your physical device to touch anything else, but manipulation tasks require all kinds of interactions. When there’s an interaction, there’s occlusion and potential deformation when objects are manipulated, so it’s more challenging.

We will reach a state where we can use a simulation to handle this complex manipulation well. Once we reach that status, you can test your robot policy with the world simulator.

Okay, so one model, Cosmos 3, handling three jobs—handling objects in robotics—is the future. It’s very exciting.

1. Inside Cosmos 3: reasoning and generator towers

You can see that you are looking at a very diverse, versatile model architecture. We now have action as a first-class citizen, which means the model might actually take an action, and then its next observation could be caused by what it did before.

We start with a language model. We either train it from scratch ourselves or take an open model. It has a basic understanding of language structure. Then we have a vision encoder that can take video or a set of image inputs. We connect this vision encoder to the large language model, and we build a vision-language model. Sometimes, you can also start with a pretrained vision-language model.

Now we take the pretrained weights from this vision-language model and initialize a generator. One key thing is that this vision-language model is autoregressive, while our generator is actually diffusion-based and bidirectional.

This bidirectional generative tower learns to generate video, action, and audio. In this tower, every token attends to every other token. When you generate the video or the action, there’s coherence among the chunks it generates. The signal within each chunk is coherent, and it also leverages the tokens from the vision-language model part—the reasoning tower, which we call the reason tower. It also understands the instruction and what you want to generate.

The training has 2 stages: pretraining, followed by base and mid-training, where we add action.

2. World models: dynamics, policy and one clock

In your estimation, what is a world model?

This is a very challenging question. Earlier, people tried to give a definition to AGI. I think this started several years ago, but even today there’s still no common agreement on what AGI is. People have their own definitions, and I think a world model is going to be the same.

I think a world model is a collection of useful tools. We model something because we are trying to achieve a goal. For example, it could be to predict the future, or it could be to understand why something happened.

There are 3 important models in robotics: forward dynamics, inverse dynamics, and policy. Forward dynamics means that, given the starting point and the action taken, you predict what will happen in the future. Inverse dynamics means that, given the visual transition, you infer what action was taken. Policy means determining what the robot should do to achieve the task.

We’re combining different modalities, like audio, action, and vision, and they might run on different time scales. How do you make them all run on the same time scale?

Different signals have different frequencies. Even for video, we can have different frame rates—high-speed video or low-speed video. Audio also has different frequencies, and action operates at a different frequency.

We have a temporal positional-embedding scheme in which we normalize all the signals to the same axis and scale. When tokens represent different time chunks or different spatial chunks of the signal, the model knows which other tokens are in the same time instance and knows the relative distance between different time instances.

That is a critical part of making this world model work while handling audio, video, and action, all of which come with different frequencies.

You were talking earlier about these different modalities—forward dynamics, inverse dynamics, and policy—in a single model. How exactly do they reinforce each other?

When we train the model, we put them together and give it an information bottleneck. You only have a certain amount of capacity to explain all of them. These are all trying to connect visual observation to action.

In the world-action model, you have a text instruction describing what you want to achieve, and this is what you observe. You start to generate an action and a predicted output. The predicted output is similar to forward dynamics: when you have an action, what could the output be? In inverse dynamics, when you have this visual transition, what could the action be? That is also similar to what happens in the world-action model, where you have the video and the action and the correlation between them.

3. Learning robot skills from human video

I think all 3 are fundamentally trying to capture the correlation between observation and action. It’s just a different slice, a different perspective. Once you put in the information bottleneck and the capacity constraints, the model learns. Our results in the paper show that there’s a synergy: one does help the other.

I think it’s fair to say there’s a bit of an asymmetry in the training data, right? There’s lots of human-generated video data, and there’s far less action data. You get this virtuous transfer between the modalities, but could the model ever know when not to transfer, when transfer could be harmful?

I think the model doesn’t really know. We have a large amount of egocentric video—the first-person view—with 2 hands trying to accomplish a certain task. We have a relatively small amount, actually a much smaller amount, of robot video. Robots also have a head camera and a gripper camera, and we can see them complete certain tasks.

I think it’s more about having a shared vocabulary for different embodiments. A humanoid robot looks like a human, and the way a human hand manipulates an object has a visual pattern. The visual-action correlation in how humans manipulate an object is very similar to how a robot would manipulate an object.

Even though the action space doesn’t correlate precisely, once you correlate 1 set of actions, the pattern is similar. It’s easier to generalize to the other embodiment. Incorporating a couple of different embodiments also helps you generalize to unseen embodiments.

Humanoid robots are going to look like humans. Some may have a bigger hand or a longer hand, but they are going to be similar. A gripper can be different, but its functions are also similar. The key is really the pattern—the correlation between visual observation and action. Translating an action from one embodiment to another embodiment is relatively easier.

4. Ambiguous tasks and system 2 planning

Apparently, one of the failure modes is when tasks are not specified well enough. I think System 2 is the key to making ambiguous tasks more concrete for System 1 to execute.

Yes, it’s fascinating, because I’m thinking that we want to use this technology in safety-critical environments, right? Some tasks are quite ambiguous. Presumably, we could add layers of tests and checks. We could red-team it and make it improve over time.

You give it a task, similar to an LLM agent. The first call of the agent or the model will define a sequence of tasks that need to be completed, and it will execute them one by one. After each execution, it will check whether it is complete, and at some point there may be choices that need to be made.

So, okay, should I do this or the other? Right. I think this will require us to think about this robotic task more at the system level, a layer above the model level. You have some harness to gather the resources together, either your past memory or the tools that you can use. Right? So, I’m seeing this as a more complex way to look at the physical task.

5. Neural simulators for policy verification

One thing we’ve not spoken about enough is using Cosmos as a simulator, essentially to train a policy. I’m just thinking about whether there would be some potential problems with that because, in reinforcement learning, we have the sim-to-real gap. Could you have a situation where the policy might learn to exploit features inside the simulator?

Yeah, it’s possible. The way I look at how this world simulator will progress is that I think it will first be more useful for policy verification. Policy verification is a task where, if you’re a model builder, any model run is going to have a lot of checkpoints, and you might do a version study. For every variant, you’re going to have many checkpoints, so how do you know which one is better?

In the ideal case, you deploy each policy on a real robot in a real environment and measure the completion rates, or success rates. For a self-driving car, for example, every time you update your driving policy, how do you know it works? You need a fleet of drivers to go to different challenging scenarios and check whether the policy is desirable, right? It’s so costly. There are so many checkpoints.

If we can use a world model as a world simulator, as a replacement for the real world and real drivers, you can pass the task directly into the world simulator. At the end of the rollout, you can measure whether the test is completed or not, and then you have a success rate. If the success rate of the world simulator correlates with real-world testing and preserves the ranking, you don’t need them to be precise.

You just need to know if policy A is better than policy B in the world simulator, and most likely policy A is going to be better than policy B in the real world. You just need this ranking to be preserved. With this, you can quickly narrow down the number of checkpoints you need to deploy in the real world. You’re going to largely improve your development velocity and help yourself find a better policy sooner.

6. Cosmos as a starting point for robot policies

Okay. And in this policy verification, you don’t directly train the model using the visual output, the rollout, right? If what happened with large language models is going to happen with world models, we’re going to see rapid improvements in world models, right? Even before it’s mature enough to provide training data for the policy model, the training data used to train the world simulator can be used for policy verification. That’s what I think. Okay, so now we go back to the question you asked. Hacking is going to happen. Even if you use deep learning, if there’s some pattern and something that can be exploited, there are going to be other examples. I think the new simulator is going to be exploited, and other simulators are also going to be exploited. As we reach that stage, I think there will be ways to regularize the model training and measure those that can be used. I don’t know precisely the approach. I think this is also a very exciting area, and I just believe that people will figure out a way.

Yeah. And on this thing about using Cosmos as a teacher, essentially, for policies, you guys have a set of recipes around this, and you should talk about that. One interesting thing is the ability to imagine the notional elephant in the room. Something that is quite unnatural and would be very low-frequency in the training data—you can imagine those. But you could also have this adaptive system where you recognize edge cases and improve the policy over time.

Cosmos as the teacher. In Cosmos, we try to help the ecosystem in 3 ways: better data, better environments, and better starting points. We believe that a Cosmos model, a world model, links video, action, and text together to learn a shared representation and is a great starting point for building a policy model.

Not just in theory, we actually post-trained a Cosmos model on the DROID dataset and achieved state-of-the-art pick-and-place policy results. Our intuition is that if we can predict the physics dynamics well and learn the correlation between pixels and actions, then it’s going to help you build a policy model. As we keep advancing Cosmos models and bring more embodiments into them, they’re going to become better and better starting points, and the amount of effort required to adapt to a new embodiment will be less.

Well, maybe you should comment on the recipes, because this will really help folks at home get up to speed with it.

As we build Cosmos as a platform, we also provide post-training recipes in our Cosmos repo. A user or developer can use the recipes to reproduce the results we have, starting with the Cosmos model. We look forward to adding more recipes there. Maybe we’ll also add skills to help you provide data, and agents to help you post-train Cosmos to do something you like. We’re trying to serve the community this way.

7. Cosmos Dreams and robot safety

Amazing. And can you tell me about Cosmos Dreams?

With Cosmos Dreams, we provide a set of skills to build a closed-loop simulator, not just for self-driving but for all sorts of embodiments. It falls into the conventional, traditional definition of a world model in robotics: action in, future observation out. You’re not limited by the size of your fleet, and anytime you need to do a lot of testing, if you have the compute, you can just launch it. It can help you with development velocity. It’s very useful for self-driving, and I think this is going to also be true for robotics.

A world model is now good enough for navigation tasks in robotics. The other challenge is manipulation tasks. With navigation, you don’t want your physical device to touch anything else, but manipulation tasks require all kinds of interactions. When there’s interaction, there’s occlusion and potential deformation when things are manipulated, so it’s more challenging.

I’m very optimistic that, with the excitement the whole field has around world models, continued advances in deep learning research, and more compute, we will reach a state where we’ll be able to use world simulation to handle complex manipulation well. Once we reach that stage, you can test your robot policy with the world simulator.

Self-driving cars require a lot of policy verification today because cars are moving around in our human space, so we need them to be safe enough. When humanoid robots reach that stage, you’ll be deploying them in many people’s houses, and safety is even more important. You don’t expect to see a car in your house, but once humanoids are everywhere, you’re going to expect to see humanoids surrounding you, and maybe your kids and pets.

You do want them to be safe enough. How do you know? As a developer of a humanoid, how do you make sure every iteration of your new policy, or the whole system, is safe enough? You need policy verification, and you won’t have enough space to set up all kinds of kitchens for all kinds of different tasks. I think using a world simulator is the only way to give you the development velocity we need.

8. Super, Nano and Edge model sizes

Very cool. And you also have this Cosmos 3 edge model, so this runs on a single device.

Yes. Cosmos comes in 3 sizes. We have Super, Nano, and Edge, right? Super gives you the highest fidelity; it’s a frontier model. If you want very high accuracy and you have the compute resources, definitely go with Super. Nano is in between: smaller and easier to post-train with whatever GPU resources you have.

We want to make sure Edge runs really well on edge devices like the Jetson Thor, Orin, or DGX Spark. The reason we build this one is that we want the world model to live where the robot lives. That way, you don’t need a round trip to a data center when you want to take a certain action, and I think that’s important when you come to real deployments. You may not be able to count on the network connection being good enough to do all the tasks, and sometimes it’s safety-critical. You have to complete the task on time.

We developed Edge so it’s small enough but also powerful enough to run on those edge devices. Because the model is smaller, it takes fewer compute resources, so it can be easily fine-tuned. Earlier, we had a recipe on fine-tuning the Cosmos Edge model in 1 day to boost its visual understanding capability.

I think all 3 versions are going to be super important. I also envision that they may need to work together in some form for future robotic development. Very challenging task.

You probably want a very powerful model. It's like calling out for help. For simpler tasks, you have to finish right away. You want to depend on the model that is on the edge device.

9. Open models, the Cosmos repo and feedback

For folks at home, where can they get their hands on these models and find out more about them?

10. Cosmos 3 collection on Hugging Face

Yeah. All our models are open, and all our code, and even some of the training data we created, are open. The models and the data live on Hugging Face, and the code lives in a GitHub repository. We have github.com/nvidia-cosmos/cosmos as our landing repository. On the front page, you can see links to the Hugging Face models, the datasets, and also the training code and the post-training scripts.

We also have skills in the repository to help your agent quickly pick up the offerings in Cosmos. We built Cosmos to help the ecosystem and a wide range of physical AI developers. We welcome feedback. I have my email inbox, and as you know, I'm watching every issue that comes to the Cosmos repository.

We do want to see your feedback. Good or bad, both are very welcome, and we are committed to making this better and better.

Ming-Yu, it's been a pleasure and an honor having you on MLST. Thank you so much for joining us today.

Thank you.