How Physical AI Learns Across Language, Video and Action — Ming-Yu Liu
- NVIDIA’s Cosmos 3 is presented as one architecture spanning video understanding, generation, and action, with “action as a first-class citizen.” Tim Scarfe discloses that the episode is a paid partnership with Nvidia; Ming-Yu Liu describes an autoregressive vision-language “reason tower” whose pretrained weights initialize a bidirectional diffusion “generator tower” for video, action, and audio. Cosmos can also serve as a starting point for policy models. The opening demo road “was never filmed.”
- The proposed simulator can reduce reliance on physical test fleets. Liu’s framing: instead of “employ 100 drivers, drive 100 cars, go to different intersections,” you launch verification runs on compute — “you are not limited by the size of your fleet.” For self-driving and eventually humanoids, he calls world simulation the only way to achieve the needed development velocity for safety verification.
- The first practical use case is policy verification rather than direct policy training, and the simulation need not be exact. The simulator only needs ranking preservation across checkpoints: “you don’t need them to be precise, you just need to know if policy A is better than policy B” and that the ranking transfers to the real world, letting developers “narrow down the number of checkpoints” needing real-world testing.
- Cross-embodiment transfer is the data-scarcity workaround: abundant egocentric human video can help with scarce robot action data. Because humanoid hands manipulate objects with visual patterns “very similar” to robots’, correlating one action space makes others “relatively easier” — but on Tim Scarfe’s question of when transfer could be harmful, Liu concedes “the model doesn’t really know.”
- Liu is candid that simulators will be exploited and does not have the fix. His hedge: “Hacking is going to happen”; regularization and measurement approaches may emerge, but “I don’t know precisely the approach... I just believe that people will figure out a way.”
- Distribution is open, with an edge option. Three sizes — Super (frontier fidelity), Nano (smaller and easier to post-train), and Edge for devices like Jetson Thor, Orin, and DGX Spark — support the goal that “the world model [should] live where the robot lives.” Models and some data are on Hugging Face; code and post-training recipes are at
github.com/nvidia-cosmos/cosmos.
1. One architecture spanning understanding, generation, and action
- The episode opens with a generated driving clip — “that road was never filmed” — before Liu walks through the stack: start with a language model, attach a vision encoder to create an autoregressive vision-language model, then use its pretrained weights to initialize a bidirectional diffusion generator for video, action, and audio. In the generator, “every token attends to every other token,” giving coherence within each chunk while leveraging instructions from the “reason tower.”
- Training has two stages: pretraining, followed by base and mid-training, where action is added. Action becomes “a first-class citizen”: the model may take an action, and its next observation could be caused by that action.
- Modality alignment across mismatched frame rates and frequencies — video, audio, and action — is handled by a temporal positional-embedding scheme that normalizes signals to the same axis and scale. This tells the model which tokens share a time instance and the relative distance between time instances.
2. What a world model is — and why three tasks in one model help each other
- Liu’s non-definition: like AGI, “there’s still no common agreement” on what a world model is. His view is that it is “a collection of useful tools.” Concretely, robotics needs three: forward dynamics (starting point plus action → future), inverse dynamics (visual transition → action), and policy (what the robot should do).
- Why train them together: all three “capture the correlation between observation and action, just a different slice.” Under an information bottleneck — “you only have this amount of capacity to explain all of them” — the paper’s results show synergy: “one does help the other.”
- On Scarfe’s data-asymmetry point — abundant egocentric human video versus much less robot data — Liu argues for a “shared vocabulary for different embodiments.” Human and robot manipulation produce similar visual-action patterns even when their action spaces do not precisely correlate, making generalization to unseen embodiments easier. But asked whether the model knows when transfer is harmful, he says, “the model doesn’t really know.”
3. Scarfe’s sim-to-real pushback and the policy-verification answer
- Scarfe’s challenge is whether a policy trained in Cosmos could exploit simulator features or artifacts. Liu reframes the roadmap: world simulators will first be more useful for policy verification — ranking checkpoints instead of deploying every policy on a real fleet, which is “so costly.”
- The load-bearing insight is that “you don’t need them to be precise, you just need this ranking preserved” between simulator and real-world success rates. That could “largely improve your development velocity” and help developers “find a better policy sooner.”
- On simulator exploitation, Liu is unvarnished: “Hacking is going to happen.” He expects ways to regularize training and measure the problem, but says, “I don’t know precisely the approach... I just believe that people will figure out a way.”
- On ambiguous tasks, Scarfe suggests System 2 could make tasks more concrete for System 1. Liu describes an LLM-agent-like system that defines a sequence of tasks, executes them one by one, checks completion after each step, and handles choices. He frames this as a system-level harness above the model, gathering memory and tools around the physical task.
4. Cosmos as teacher: recipes, Dreams, and the safety case for simulation
- The ecosystem play is “better data, better environments, and better starting points.” Liu says this is “not just in theory”: post-training a Cosmos model on the DROID dataset achieved state-of-the-art pick-and-place policy results. Post-training recipes ship in the Cosmos repository.
- Cosmos Dreams provides skills to build closed-loop simulators for many embodiments, not just self-driving. A world model is now good enough for navigation tasks in robotics, while manipulation is harder because interactions create occlusion and potential deformation. Liu is very optimistic that continued advances in deep learning and more compute will make complex manipulation feasible in world simulation.
- The safety framing is vivid: humanoids may be “deployed in many people’s houses,” surrounding people, “maybe your kids and pets.” Since developers cannot stage “all kinds of kitchens” for every task, Liu says using a world simulator is the only way to achieve the needed development velocity for verifying each policy iteration.
5. Edge deployment and open models, code, and some data
- There are three sizes: Super, the highest-fidelity frontier model for users with ample compute; Nano, which is smaller and easier to post-train; and Edge, designed for devices such as Jetson Thor, Orin, and DGX Spark. Edge is intended to keep the world model where the robot lives, avoiding a data-center round trip when network reliability or safety makes latency important. Liu cites a recipe for fine-tuning Cosmos Edge in 1 day to improve visual understanding.
- Liu envisions the three versions working together: a powerful model can provide help, while simpler tasks are handled immediately by the model on the edge device.
- The models and code are open, along with some training data. Models and datasets are on Hugging Face; the code and post-training scripts are at
github.com/nvidia-cosmos/cosmos. The repository also includes skills to help agents use Cosmos, and Liu says he watches every issue and welcomes both positive and negative feedback.
Full transcript
This car is about to make a left turn. Can you envision the trajectory the car is taking? And the video generated.
Guess what? That road was never filmed. It was made by NVIDIA’s Cosmos 3. It reads video, simulates roads, and is learning how to handle objects. Ming-Yu Liu leads the research, and this is a paid partnership with Nvidia.
Now, Cosmos 3 takes text, video, audio, and actions as input. Let’s start with the simplest case. You ask the model what happened in the videos: how many cars are in the video, or whether the robot succeeded in completing an action. “Okay, there are 3 cars, and the robot succeeded in picking up the apple and putting it in the basket.” When you have video and text as inputs and generate text, it becomes a vision-language model.
So why don’t we flip this thing on its head? The model stops describing the scene and starts generating it.
Cosmos Dreams is a set of skills to build a closed-loop simulator, not just for self-driving. I can employ 100 drivers, drive 100 cars, go to different intersections, and figure out whether it works better, or I can use this simulator to verify the accuracy. You are not limited by the size of your fleet. Anytime you need to do a lot of testing, if you have the compute, just launch it.
Okay, so what if we took the same idea but applied it to an even harder problem, like robots?
With navigation, you don’t want your physical device to touch anything else, but manipulation tasks require all kinds of interactions. When there’s an interaction, there’s occlusion and potential deformation when objects are manipulated, so it’s more challenging.
We will reach a state where we can use a simulation to handle this complex manipulation well. Once we reach that status, you can test your robot policy with the world simulator.
Okay, so one model, Cosmos 3, handling three jobs—handling objects in robotics—is the future. It’s very exciting.
1. Inside Cosmos 3: reasoning and generator towers
You can see that you are looking at a very diverse, versatile model architecture. We now have action as a first-class citizen, which means the model might actually take an action, and then its next observation could be caused by what it did before.
We start with a language model. We either train it from scratch ourselves or take an open model. It has a basic understanding of language structure. Then we have a vision encoder that can take video or a set of image inputs. We connect this vision encoder to the large language model, and we build a vision-language model. Sometimes, you can also start with a pretrained vision-language model.
Now we take the pretrained weights from this vision-language model and initialize a generator. One key thing is that this vision-language model is autoregressive, while our generator is actually diffusion-based and bidirectional.
This bidirectional generative tower learns to generate video, action, and audio. In this tower, every token attends to every other token. When you generate the video or the action, there’s coherence among the chunks it generates. The signal within each chunk is coherent, and it also leverages the tokens from the vision-language model part—the reasoning tower, which we call the reason tower. It also understands the instruction and what you want to generate.
The training has 2 stages: pretraining, followed by base and mid-training, where we add action.
2. World models: dynamics, policy and one clock
In your estimation, what is a world model?
This is a very challenging question. Earlier, people tried to give a definition to AGI. I think this started several years ago, but even today there’s still no common agreement on what AGI is. People have their own definitions, and I think a world model is going to be the same.
I think a world model is a collection of useful tools. We model something because we are trying to achieve a goal. For example, it could be to predict the future, or it could be to understand why something happened.
There are 3 important models in robotics: forward dynamics, inverse dynamics, and policy. Forward dynamics means that, given the starting point and the action taken, you predict what will happen in the future. Inverse dynamics means that, given the visual transition, you infer what action was taken. Policy means determining what the robot should do to achieve the task.
We’re combining different modalities, like audio, action, and vision, and they might run on different time scales. How do you make them all run on the same time scale?
Different signals have different frequencies. Even for video, we can have different frame rates—high-speed video or low-speed video. Audio also has different frequencies, and action operates at a different frequency.
We have a temporal positional-embedding scheme in which we normalize all the signals to the same axis and scale. When tokens represent different time chunks or different spatial chunks of the signal, the model knows which other tokens are in the same time instance and knows the relative distance between different time instances.
That is a critical part of making this world model work while handling audio, video, and action, all of which come with different frequencies.
You were talking earlier about these different modalities—forward dynamics, inverse dynamics, and policy—in a single model. How exactly do they reinforce each other?
When we train the model, we put them together and give it an information bottleneck. You only have a certain amount of capacity to explain all of them. These are all trying to connect visual observation to action.
In the world-action model, you have a text instruction describing what you want to achieve, and this is what you observe. You start to generate an action and a predicted output. The predicted output is similar to forward dynamics: when you have an action, what could the output be? In inverse dynamics, when you have this visual transition, what could the action be? That is also similar to what happens in the world-action model, where you have the video and the action and the correlation between them.
3. Learning robot skills from human video
I think all 3 are fundamentally trying to capture the correlation between observation and action. It’s just a different slice, a different perspective. Once you put in the information bottleneck and the capacity constraints, the model learns. Our results in the paper show that there’s a synergy: one does help the other.
I think it’s fair to say there’s a bit of an asymmetry in the training data, right? There’s lots of human-generated video data, and there’s far less action data. You get this virtuous transfer between the modalities, but could the model ever know when not to transfer, when transfer could be harmful?
I think the model doesn’t really know. We have a large amount of egocentric video—the first-person view—with 2 hands trying to accomplish a certain task. We have a relatively small amount, actually a much smaller amount, of robot video. Robots also have a head camera and a gripper camera, and we can see them complete certain tasks.
I think it’s more about having a shared vocabulary for different embodiments. A humanoid robot looks like a human, and the way a human hand manipulates an object has a visual pattern. The visual-action correlation in how humans manipulate an object is very similar to how a robot would manipulate an object.
Even though the action space doesn’t correlate precisely, once you correlate 1 set of actions, the pattern is similar. It’s easier to generalize to the other embodiment. Incorporating a couple of different embodiments also helps you generalize to unseen embodiments.
Humanoid robots are going to look like humans. Some may have a bigger hand or a longer hand, but they are going to be similar. A gripper can be different, but its functions are also similar. The key is really the pattern—the correlation between visual observation and action. Translating an action from one embodiment to another embodiment is relatively easier.
4. Ambiguous tasks and system 2 planning
Apparently, one of the failure modes is when tasks are not specified well enough. I think System 2 is the key to making ambiguous tasks more concrete for System 1 to execute.
Yes, it’s fascinating, because I’m thinking that we want to use this technology in safety-critical environments, right? Some tasks are quite ambiguous. Presumably, we could add layers of tests and checks. We could red-team it and make it improve over time.
You give it a task, similar to an LLM agent. The first call of the agent or the model will define a sequence of tasks that need to be completed, and it will execute them one by one. After each execution, it will check whether it is complete, and at some point there may be choices that need to be made.
So, okay, should I do this or the other? Right. I think this will require us to think about this robotic task more at the system level, a layer above the model level. You have some harness to gather the resources together, either your past memory or the tools that you can use. Right? So, I’m seeing this as a more complex way to look at the physical task.
5. Neural simulators for policy verification
One thing we’ve not spoken about enough is using Cosmos as a simulator, essentially to train a policy. I’m just thinking about whether there would be some potential problems with that because, in reinforcement learning, we have the sim-to-real gap. Could you have a situation where the policy might learn to exploit features inside the simulator?
Yeah, it’s possible. The way I look at how this world simulator will progress is that I think it will first be more useful for policy verification. Policy verification is a task where, if you’re a model builder, any model run is going to have a lot of checkpoints, and you might do a version study. For every variant, you’re going to have many checkpoints, so how do you know which one is better?
In the ideal case, you deploy each policy on a real robot in a real environment and measure the completion rates, or success rates. For a self-driving car, for example, every time you update your driving policy, how do you know it works? You need a fleet of drivers to go to different challenging scenarios and check whether the policy is desirable, right? It’s so costly. There are so many checkpoints.
If we can use a world model as a world simulator, as a replacement for the real world and real drivers, you can pass the task directly into the world simulator. At the end of the rollout, you can measure whether the test is completed or not, and then you have a success rate. If the success rate of the world simulator correlates with real-world testing and preserves the ranking, you don’t need them to be precise.
You just need to know if policy A is better than policy B in the world simulator, and most likely policy A is going to be better than policy B in the real world. You just need this ranking to be preserved. With this, you can quickly narrow down the number of checkpoints you need to deploy in the real world. You’re going to largely improve your development velocity and help yourself find a better policy sooner.
6. Cosmos as a starting point for robot policies
Okay. And in this policy verification, you don’t directly train the model using the visual output, the rollout, right? If what happened with large language models is going to happen with world models, we’re going to see rapid improvements in world models, right? Even before it’s mature enough to provide training data for the policy model, the training data used to train the world simulator can be used for policy verification. That’s what I think. Okay, so now we go back to the question you asked. Hacking is going to happen. Even if you use deep learning, if there’s some pattern and something that can be exploited, there are going to be other examples. I think the new simulator is going to be exploited, and other simulators are also going to be exploited. As we reach that stage, I think there will be ways to regularize the model training and measure those that can be used. I don’t know precisely the approach. I think this is also a very exciting area, and I just believe that people will figure out a way.
Yeah. And on this thing about using Cosmos as a teacher, essentially, for policies, you guys have a set of recipes around this, and you should talk about that. One interesting thing is the ability to imagine the notional elephant in the room. Something that is quite unnatural and would be very low-frequency in the training data—you can imagine those. But you could also have this adaptive system where you recognize edge cases and improve the policy over time.
Cosmos as the teacher. In Cosmos, we try to help the ecosystem in 3 ways: better data, better environments, and better starting points. We believe that a Cosmos model, a world model, links video, action, and text together to learn a shared representation and is a great starting point for building a policy model.
Not just in theory, we actually post-trained a Cosmos model on the DROID dataset and achieved state-of-the-art pick-and-place policy results. Our intuition is that if we can predict the physics dynamics well and learn the correlation between pixels and actions, then it’s going to help you build a policy model. As we keep advancing Cosmos models and bring more embodiments into them, they’re going to become better and better starting points, and the amount of effort required to adapt to a new embodiment will be less.
Well, maybe you should comment on the recipes, because this will really help folks at home get up to speed with it.
As we build Cosmos as a platform, we also provide post-training recipes in our Cosmos repo. A user or developer can use the recipes to reproduce the results we have, starting with the Cosmos model. We look forward to adding more recipes there. Maybe we’ll also add skills to help you provide data, and agents to help you post-train Cosmos to do something you like. We’re trying to serve the community this way.
7. Cosmos Dreams and robot safety
Amazing. And can you tell me about Cosmos Dreams?
With Cosmos Dreams, we provide a set of skills to build a closed-loop simulator, not just for self-driving but for all sorts of embodiments. It falls into the conventional, traditional definition of a world model in robotics: action in, future observation out. You’re not limited by the size of your fleet, and anytime you need to do a lot of testing, if you have the compute, you can just launch it. It can help you with development velocity. It’s very useful for self-driving, and I think this is going to also be true for robotics.
A world model is now good enough for navigation tasks in robotics. The other challenge is manipulation tasks. With navigation, you don’t want your physical device to touch anything else, but manipulation tasks require all kinds of interactions. When there’s interaction, there’s occlusion and potential deformation when things are manipulated, so it’s more challenging.
I’m very optimistic that, with the excitement the whole field has around world models, continued advances in deep learning research, and more compute, we will reach a state where we’ll be able to use world simulation to handle complex manipulation well. Once we reach that stage, you can test your robot policy with the world simulator.
Self-driving cars require a lot of policy verification today because cars are moving around in our human space, so we need them to be safe enough. When humanoid robots reach that stage, you’ll be deploying them in many people’s houses, and safety is even more important. You don’t expect to see a car in your house, but once humanoids are everywhere, you’re going to expect to see humanoids surrounding you, and maybe your kids and pets.
You do want them to be safe enough. How do you know? As a developer of a humanoid, how do you make sure every iteration of your new policy, or the whole system, is safe enough? You need policy verification, and you won’t have enough space to set up all kinds of kitchens for all kinds of different tasks. I think using a world simulator is the only way to give you the development velocity we need.
8. Super, Nano and Edge model sizes
Very cool. And you also have this Cosmos 3 edge model, so this runs on a single device.
Yes. Cosmos comes in 3 sizes. We have Super, Nano, and Edge, right? Super gives you the highest fidelity; it’s a frontier model. If you want very high accuracy and you have the compute resources, definitely go with Super. Nano is in between: smaller and easier to post-train with whatever GPU resources you have.
We want to make sure Edge runs really well on edge devices like the Jetson Thor, Orin, or DGX Spark. The reason we build this one is that we want the world model to live where the robot lives. That way, you don’t need a round trip to a data center when you want to take a certain action, and I think that’s important when you come to real deployments. You may not be able to count on the network connection being good enough to do all the tasks, and sometimes it’s safety-critical. You have to complete the task on time.
We developed Edge so it’s small enough but also powerful enough to run on those edge devices. Because the model is smaller, it takes fewer compute resources, so it can be easily fine-tuned. Earlier, we had a recipe on fine-tuning the Cosmos Edge model in 1 day to boost its visual understanding capability.
I think all 3 versions are going to be super important. I also envision that they may need to work together in some form for future robotic development. Very challenging task.
You probably want a very powerful model. It's like calling out for help. For simpler tasks, you have to finish right away. You want to depend on the model that is on the edge device.
9. Open models, the Cosmos repo and feedback
For folks at home, where can they get their hands on these models and find out more about them?
10. Cosmos 3 collection on Hugging Face
Yeah. All our models are open, and all our code, and even some of the training data we created, are open. The models and the data live on Hugging Face, and the code lives in a GitHub repository. We have github.com/nvidia-cosmos/cosmos as our landing repository. On the front page, you can see links to the Hugging Face models, the datasets, and also the training code and the post-training scripts.
We also have skills in the repository to help your agent quickly pick up the offerings in Cosmos. We built Cosmos to help the ecosystem and a wide range of physical AI developers. We welcome feedback. I have my email inbox, and as you know, I'm watching every issue that comes to the Cosmos repository.
We do want to see your feedback. Good or bad, both are very welcome, and we are committed to making this better and better.
Ming-Yu, it's been a pleasure and an honor having you on MLST. Thank you so much for joining us today.
Thank you.