机器人领域的苦涩教训：为何规模化取胜 — Jason Ma，Dyna Robotics

If we want to learn a new task, we basically do a little bit of robot data collection.

What does “a little bit” mean?

A couple of hours.

A couple of hours?

Yeah.

Whereas for some tasks, it’s just tens of minutes, right?

Yeah.

And the chopping one was 45 minutes.

Yeah.

The open-the-fridge one was an interview task. That’s like an onboarding task for one of your engineers.

Our robots today fold thousands of napkins at many different Din Tai Fung locations across the United States. That’s actually our biggest scaled customer.

I want to make a distinction here. Today, a lot of robotics companies talk about deployments and pilots, but what’s really hard isn’t putting your robot in one location or doing a pilot. What’s really hard is getting customers to believe or validate that your robot creates enough value that they want to deploy it across many of their sites.

That’s what’s happened for us with Din Tai Fung, which, to the best of my knowledge, hasn’t really happened for other companies doing similar general-purpose robots. At Dyna Robotics, our robots fold thousands of napkins per day.

Before we get into today’s episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn’t choose to also click in and tune into our content. We’ve been approached by sponsors on an almost daily basis. But fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way. But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It’s the only thing I’ll ever ask of you. And it means absolutely everything to me and my team that works so hard to bring Latent Space to you each and every week. If you do it, I promise you we’ll never stop working to make the show even better. Now, let’s get into it.

1. Introduction and Dyna-2's World-Action Model

All right, we’re here on site with Jason Ma, co-founder and chief scientist of Dyna Robotics. Thanks for having us.

Of course. Thank you for coming.

Yeah, it’s been a blast. We got to see drinks being pulled, a lot of demos, and robots with knives for the first time. Under the hood of all this, you have a world action model. What is that?

Last month, we released our world action model called Dyna-2. It’s the first world action model trained on more than 1 million hours of data. To answer your question, a world action model is a model that can predict the future video, the future state of the world. Hence the name “world model.” But in addition, you can use that future generation to produce actions that robots can execute in the physical world to control the robot.

Why do we do that? It’s a very powerful architecture because, if you think about how humans act in the physical world, our brains implicitly think about how the world will evolve based on our actions. Now we have a model architecture that can do that to guide robot actions. What we find is that, by doing so, the model is much more dexterous, much more robust, and much more capable of generalizing to environments and scenarios it has never seen before.

Yeah. I want to frame this question a bit. I think a lot of people hear “world model” and think you’re outputting video, a world, or a 3D state. You’re also doing action. I think most people don’t understand what models are under the hood of robotics. What are the inputs and outputs? What’s the training data? What’s the architecture? Give us all the details.

For inputs, the most important one is video. You can think of videos the robot captures with its cameras: images of the past several seconds. But what’s really interesting is that a robot also has a lot of actuators and joint encoders on itself, so it knows the angles of its joints. Those can also be inputs to the world model. That’s very different from a pure video model that only looks at videos.

Additionally, you want to be able to talk to the robot and instruct it to do different tasks, so you can also feed it language instructions. Those are what I consider the basic set of inputs that go into any robotics model, including world action models.

At the basic level, the output needs to be actions the robot can execute on the hardware. But in the case of a world action model, what’s really interesting is that in addition to the actions, you can also output future video. You can almost output the future video, or some representation of it, first, and then use that to guide robot actions.

2. Diffusion Transformers, Flow Matching, and Generalist Robots

The way I think about it is that this is kind of analogous to chain of thought. The models are reasoning in visual and physics space, then outputting actions. That’s the key difference between world action model outputs and other kinds of robot model outputs.

You put out a good blog post about your model. I think a lot of people understand language models or transformers as next-token predictors. In this case, you shared quite a bit: there’s diffusion, flow matching, and you’re predicting the next world state plus action. One thing you say is 1 million hours. How do you determine the architecture? What are the choices? What is the architecture?

I feel like with language models, there’s a lot of taking something off the shelf, using trillions of tokens, doing prediction, SFT, and RL, and you have a model. What about this? How do you go from nothing to your model? Is there a vanilla architecture off the shelf, or do you determine all of this? What’s the current state?

The way to think about building models for robotics is that there are essentially a couple of things to look at. One is data, and the second is how we make a model learn as much as possible from that data. Let’s first talk about the model. The blog post we released is about our Dyna-2 world action model. Under the hood, it’s a diffusion transformer.

A transformer is essentially like a language model: a sequence-to-sequence, decoder-only model. But what’s really interesting is that in visual domains—if you think about generating images or videos, and in this case robots looking at images to generate actions—people have found the diffusion architecture works really well.

A diffusion model takes random noise and, through iterative computation, can turn it into something high-dimensional, well-defined, and full of fine-grained features. If you think about generating images, you want sharp edges, sharp corners, and color contrast. For robotics, even though you’re not generating an image, you’re generating actions that need to be very precise and smooth. So there are similar considerations to generating images.

That’s why, from first principles, the whole field—not just us—has been using this kind of diffusion transformer architecture to train robotics foundation models. You need the transformer aspect because it can ingest a lot of data and learn from it. Diffusion helps refine the outcome to get the smooth, precise, fine-grained motion you need to execute on the robot. That’s the architecture behind a lot of our recent breakthroughs.

We’ve covered flow matching before on the podcast. It’s mostly an efficiency speedup. If you normally take 40 to 100 diffusion steps, you can reduce it to 4.

Yeah, that’s right. With a diffusion model, one thing that’s really hard in practice is tuning the scheduler: how much you’re transforming the initial noise input into the final output. Flow matching is a special case of diffusion that makes the transformation much simpler. In practice, you don’t have to tune as much, and it works just as well.

That’s why, I think, two to three years ago, most generative modeling for images and video shifted from diffusion to flow matching. The concept is the same, and flow matching is just a simpler-to-tune variant of diffusion.

Yeah, it makes me think that flow matching obviously has trade-offs in quality.

Yeah, but it’s very suited for action diffusion and robotics because you get to reevaluate in real time.

Whereas with images, you’re basically doing it in one shot.

That’s right.

Which is pretty fun. We got to mess with the laundry-folding robot and throw a random towel in. The models adapt, right?

Yeah.

I think one key thing about your models is that they’re made to be generalist: one foundation model, and you adapt it a lot. How does that work? In 45 minutes of mapped data, you can generalize to pulling drinks out of a fridge. What’s the mindset behind using one general model, and how do you adapt it to different tasks?

The way we think about it is that, for robotics, we need to build a very general, strong base model. If you think about how a human interacts and behaves in the physical world, we have very strong base knowledge and can readily do a lot of tasks. But if you want to become an expert at any given task, some amount of training needs to happen. I think the same principle has to apply to training a general-purpose robot like the one we’re building here.

3. The Robotics Data Bottleneck

Our approach is to first train a general model on as much data as we can find. Our Dyna-2 model was trained on over a million hours of human data as a starting point. With that strong foundation, we found that we can quickly get the robot—or the model—to do many different tasks with very little fine-tuning. We just tell the model what task we want it to do and give it a few demonstrations, and it can readily do a bunch of the tasks you’ve seen in the office.

We actually have quite a few people who deal in data, who sell data.

Yeah.

What is good data? What is bad data? What do you want to buy?

Yeah.

I think these are all open questions. Obviously, one of the most important things is egocentric data of humans doing tasks. What else would you modify about that basic statement?

Yeah, yeah. I think data is the most important thing in robotics today because, unlike language models, there’s no internet, right? Every company building a model and building hardware has to think about data. There are roughly a couple of different ways to operate here. One is that you can collect a lot of data yourself, which is what we’re doing as well.

You have employees here doing that.

I mean, there are robots everywhere in the office, right? But if you think about how much data you need for a robotic foundation model, I think the amount any individual company can collect by itself is nowhere near sufficient. It’s kind of like trying to build an internet by just having Reddit or just having Facebook. They’re great websites with a lot of users, but it’s not enough.

So, when we think about data, there has to be some kind of data acquisition happening, right? You buy data, or you look at data that’s already out there. There are many different tiers. You mentioned data quality. I think data quality in robotics is a non-stationary problem, in the sense that what counts as high-quality data for your model changes over time.

When your model doesn’t know anything—when it’s at zero—any data is useful. You can use a lot of video data. In the case of egocentric data, I think it’s actually a very good way of pretraining robots because human data is so diverse. It covers every physical activity you can imagine, and that gives the model a strong prior on how the world can evolve around an embodied actor—a human.

But the issue with human data is that, until your robot looks exactly like a human, there’s going to be some gap between human actions and robot actions. Just by training on human data, it’s not very likely that you can execute robot actions directly today. You still need some amount of adaptation using the robot’s own data, but collecting data on a robot is going to be much more expensive and much harder.

The recipe we’re thinking about is: How do we take the largest source of data we have today, like egocentric data, to give the model a strong prior over everything physical manipulation can cover, and then combine that with a small amount of robotics data to help the robot master whatever task we’re interested in? Data quality and getting data in robotics is a fascinating, multifaceted problem, but there isn’t a single winning strategy. You can’t just say, “Here’s one kind of data I’m only going to collect; that will give me a strong foundation model.”

You really need to think about what each kind of data covers: human data covers this part of robot cognition, and robot data covers the low-level reflexes and dexterity the robot also needs. Put them into one model, and I think that gives you a really adaptable, really strong model in the end.

I’ll prompt you, for the people listening who want to work with you: what data vendors do you want? This is your call for data.

Yeah, that’s a great question. In our Dyna-2 blog, at the end, we have an email address: data-partnerships@dyna.co. If you’re a data vendor with a lot of high-quality data—whether it’s egocentric data, robot data, video data, or anything you think could help robots—send it to that email. Our team will take a look and get back to you.

4. Synthetic Data and Self-Improving World Models

But at your stage, you’re probably not buying any more egocentric data off the shelf, right? You probably want to say, “Here’s what I need.”

Is there a niche that’s lacking?

You need a Scale AI; you need a Core for doing this. I see.

The folks who don’t work in robotics are probably thinking, “Okay, some niche factory, like you showed putting tubes in—there’s only so much data you’ll find about that. Would a lot of that help you?”

Yeah, no. If you really want to nail a task, having data from that task will certainly help a lot. For us, we still think a million hours of data is not enough; it’s just the beginning. We showed the first model at a million-hour scale, but we need more than that. We’re still looking at a lot of raw, diverse data, but we’re also thinking about how to get more custom data, either collected in-house or by our partners, for really specific niche tasks. I think you still need that to reach the level of excellence any model needs to be useful in the physical world.

It’s kind of like humans. If you or I go work at the Rosewood Hotel tomorrow, we’re not going to be able to make the bed the way they want. We need a lot of training. That kind of special data, like what Mercor is doing today for language models, will be very valuable for robotics as well.

Yeah. So hotels: strap a GoPro on your employees’ heads, and—

And there’s probably more data from them in a week.

One thing we discussed is closing the sim-to-real gap. What’s very interesting is that now you can basically fabricate environments on the fly.

Yes.

Have you tried that with the frontier models? What’s your experience? Are they good enough? Are you using any world-model or robotics companies to generate these environments and train on them?

Generating synthetic environments, synthetic data, and simulation data for robotics is going to be one of the next frontiers. Today people have looked at egocentric data and robot data, and those are scaling, but I think synthetic data will be a new dimension of scaling.

Going back to your question, we’ve been exploring using frontier models to generate synthetic environments for the data and environments we care about. Surprisingly, I think the technology is now at a point where it’s worth going really deep into it. For some context, in my PhD I primarily focused on simulation-based training for robotics. Back then, simulation wasn’t ready, but we had this paper called Eureka, the first paper to take a large language model—in this case, GPT-4—and have it generate reward functions for reinforcement learning and do things in simulation. At the end, we had this really nice demo of the robot spinning pencils. That was really cool at the time.

But that kind of thing was still very limited because the model didn’t have good vision capabilities or an innate 3D understanding. It was basically just generating code and using very limited feedback from text to try to come up with new simulation environments. Today, the vision capabilities of these frontier models are crazy. They can do real-to-sim CAD modeling, 3D reconstruction, and all of that. I think we’re at a tipping point where this technology could really accelerate robotics.

For us, we think about this as a way to accelerate our internal progress, because we’re building models and hardware. If a frontier model can give us more data that way—generate really tailored data for us—that’s extremely useful. Another thing we’ve been thinking about is using our own world model instead of these models to generate 3D environments. The world action model I was telling you about doesn’t just generate actions; it can also generate videos, so it can generate its own training data. That becomes a self-fulfilling flywheel, which I think will also be powerful.

5. Real-Time Inference and Model Distillation

Overall, I’m really excited about this direction. Even though we’ve scaled to a million hours of data and are deploying a bunch of robots, and the robots are already starting to do very cool things, we’re just at the beginning—the tip of the iceberg for what’s to come.

For those on the research side, we recently had Runway on the podcast. One of their things is robotic world models. Shifting a little toward that, one of the questions I have about you making your own models is: what constraints do you operate in? If you draw a parallel to language models, GPT-OSS 20B is a good fit for your average MacBook. Then there’s a larger one that’s fit for a single GPU.

Do you work within the constraints of a robot model on-device, in the cloud, or in a hybrid setup? And how does that affect your scaling law of data?

Yeah.

A million hours is great, but do we just scale up parameters and data?

No, that’s a great question. What’s really different about robots versus language models and other applications is that, for coding agents and language models, latency is okay.

When I ask ChatGPT today, it takes a couple of seconds before it spits out anything. But for robots, the inference latency has to be very short. The robot needs to act; it can’t just hold a knife and do nothing for 3 seconds. That’s going to freak people out.

Yeah, we saw someone put their real hand in.

Our model’s pretty smart, so that’s okay. But in general, if you think about a very dynamic task—for example, holding a tennis racket and juggling a ball—it’s very dynamic and reactive. Physics and gravity don’t wait for you. For that reason, robotics models need to be able to run inference very fast, and the fastest way is to serve them on-prem or on-device. If you call a model—

Over the network—

There’s no guarantee. If you’re driving a car, there’s no guarantee, right? For that reason, the model that eventually gets deployed has to be a reasonable size that can be served locally. But that doesn’t mean the initial model you train has to be small. In language models, people do distillation: you train the biggest model, then distill it. The best way to make a small model very smart is by distilling it. I think robotics is getting there, and that’s what people are going to converge on.

How about inference? Do you have your own inference stack, or is a lot of it based on architecture you’ve pulled from diffusion and other things? Is there anything you want to share?

6. Full-Stack Robotics: Hardware and Supply Chains

We build our own inference stack for whatever model we deploy. As I was saying, latency is so important, so the whole stack is extremely important for robotics. You want to optimize the inference speed for the kind of model you’re running, and there’s a lot of work in that. That’s what we do.

You’re really full stack, from the actuators all the way to ID—which I didn’t realize was an acronym—to the models and data as well. Are these your own actuators? How much of this is your own? Of course, the design is your own; it’s not a humanoid. How far down the robotics stack do you go?

The way we think about what we need to own versus what we can buy is: if the inputs and outputs of a component are very clear, and there’s already an extremely good, mature solution, there’s no point in building it in-house. For example, you can see the LiDARs on this robot. Those black circles help the robot detect obstacles and navigate its surroundings. LiDAR is a very mature technology that people have been using for decades in self-driving cars, so we don’t need to build it from scratch. The same goes for some cameras.

But for anything we think is crucial to the system’s overall performance, our vision is to own and build it ourselves. You mentioned actuators; that’s something we’re working on and are going to own. The quality and design of the actuator really matter for how well you can model the robot. To model your robot well in simulation, or do any kind of modeling, you need to select an actuator that’s good and easy to model. That’s what we care about, so we’re going to own that.

The same goes for the robot’s proportions. We measured this robot’s proportions against real people. Why? Because we’re pretraining on human data.

Oh, yeah.

It’s better for transfer. A lot of decisions go into the hardware, and they’re very much tied to intelligence, because the hardware carries the intelligence. Intelligence is extremely important, and we want to make sure the hardware serves that.

I’ll just highlight something, in case it’s relevant to you. One of the most interesting robotics discussions I’ve had was with someone who was really stressed out about actuators, and specifically magnets, because apparently we don’t make magnets in the U.S.

No, we don’t.

And there are only 2 countries that make magnets.

Rare-earth materials.

Right. I don’t know what the chemistry is or what the material is, but if anyone’s interested, that’s apparently a bottleneck we have.

Yes.

So what do we do?

It’s a good question. There are a lot of exciting discussions about bringing manufacturing back to the U.S., but for robots, that’s much harder than it seems. Even now, there isn’t a company in the U.S. truly building everything from scratch in the U.S. Even when companies say they assemble a robot or build it in the U.S., they’re still sourcing raw materials from elsewhere in the world.

As robot builders, we need to be aware of that. As a full-stack robotics company, we realize just how long the supply chain is and how hard the whole problem is. These days, a lot of the discussion centers around robot foundation models and data, but hardware—and how hard it is to build good hardware—is very underemphasized.

That’s going to be extremely important. This is a platform, right?

It’s actually a platform.

Yeah. Some companies choose to be, let’s call it, platformless. They only do the model and say, “We deploy on any platform. We deploy on Optimus or off-the-shelf robots.” It’s a choice. Obviously, you own everything, so you can do that. Do you see yourself building more in the U.S., or building your own global supply chain? What direction is most likely?

We’re currently exploring many different options. With a global supply chain, there are geopolitical issues—

Shipping.

Shipping delays and all that stuff. In the U.S., going from zero to one will take some time. We’re trying to think of a solution that lets us build more in the U.S. while leveraging the global supply chain.

7. Why Dyna Built a Wheeled Semi-Humanoid

Going back to your previous question about some companies choosing to just build the model: we build everything. I think it’s very important to get this right. The hard thing in robotics is building the whole system and the things necessary to build that system. Previously, I was at DeepMind and NVIDIA, primarily just building models, and I saw the upper bound of a system when you don’t control the hardware and don’t get to optimize it. If you think about value creation for robots, they’re here to deliver a service and solve physical tasks. To solve physical tasks, you want to be able to optimize everything, and that’s important.

I mean, we have the robot. You’ve got to tell us about your own custom platform. What do you call this?

What do we call it?

Yeah.

We think of it generally as a semi-humanoid. The upper body is like a humanoid: it has shoulders and arms, and it also has 4 degrees of freedom in the waist. So it can do all the upper-body motions a human can do.

It’s a semi-humanoid because the lower body has wheels. We chose wheels because there’s a trade-off between practicality and performance. With legs, energy consumption is a lot higher, the robot is a lot less stable, and safety is extremely hard to guarantee. For legged robots, there’s actually no guarantee that the robot will stay upright and balanced the entire time. I’m sure you’ve seen a lot of those robot dancing videos. They fall over very easily.

You also see demos that you can tell aren’t production-ready, where you can box the robot or push it over. That robot is just good at balancing, right?

I think the more interesting thing you’ve innovated on is— we have one of the prototypes, the Mewtwo, back here. This is just an up-and-down axis, and you’ve chosen to move toward something like this, which can lean. We’re going to show the laundry demo, where you have to lean into machines. That’s another degree of freedom that I don’t even have the words for. It’s not pitch, it’s not yaw.

How did we think about that? It’s a useful exercise. A couple of years ago, when I was exercising, I accidentally pulled a muscle. I’m sure you’ve had that experience too. For the next few days, you just couldn’t move. Or you sleep in the wrong position and your neck is very stiff. We’ve all experienced that, and then you realize there are so many muscles and degrees of freedom that we use in daily life and take for granted. When they’re disabled, it’s very awkward to do tasks that feel natural.

Right? That’s also how we think about designing robots. With a design that’s purely an elevator, it can only go up and down. We realize that we use our waist and back for a lot of everyday physical activity. Even if I want to get this bottle, I’m not just moving my arms. It would be very strange if I had to intentionally keep my upper body fixed; I’d naturally move like this.

There are a lot of subconscious ways that humans move their bodies that would be useful for a robot to have as degrees of freedom. And if we want to learn from human data, humans are moving their torsos all the time. We want to be able to ingest that into the model to control the robot. But if the form factor doesn’t allow for that, the transfer is less effective, and the robot is less capable and more awkward.

What’s the data-labeling side of this? Is this the post-training fine-tuning stage, after someone uses the robot—operating it remotely and mapping the motion—or does some of it generalize?

When you say labeling, do you mean—

Taking the model you have that predicts state and mapping it to your specific hardware.

Oh, yeah. It generalizes very well. For all the demos I’ve shown you on the new robot, it got ready not so long ago, so we quite literally don’t have a lot of data compared with the stationary arms we showed you, which had much more data. The model hadn’t seen this new robot during pretraining, but it was able to learn the tasks you saw very quickly.

8. Teaching Robots New Tasks and Evaluating Performance

I credit a lot of that to the fact that, even before the hardware was ready, we had already committed to pretraining on humans because we knew that at some point our robots would be human-like. You can get ahead on the training side while the hardware is getting ready. Now we’re seeing that the transfer is real and actually very helpful.

Talk about the evaluation side. You can have predictions and have it do things, and every now and then we see it miss something. You test very niche things, too, like filling a pill bottle. How long does that process take? How do you think about deploying on a new task, evaluating, and iterating? What does that look like?

If we want to learn a new task, we basically do a little bit of robot data collection, then train the model.

What does “a little bit” mean?

A couple of hours. For some tasks, it’s just tens of minutes.

The chopping one was 45 minutes.

Opening the fridge was one.

And that’s an interview. That’s an onboarding task for one of your engineers.

Yeah. So, a small amount of data is collected, we train a model, then deploy it in the real world and look at its performance.

Fun thing if you want to work here.

Find a task and have the robot do it.

Yeah. Pitch me a new task, and I’ll come to the office, collect some data, try a model, and see what happens.

Tie your shoes.

That’ll be tiring. That’d be cool. But, yeah, we collect a small amount of data, train a model, deploy it in the real world, and look at its performance.

When you say data is collected, do you have a person operating the robot remotely, doing the task for a bit, and then you train?

Yeah. When you’re training the model, there are certain metrics you can look at, just like with any model: does it converge nicely? After that, for robotics today, we run the model in the real world on the same setup the data was collected on and see how it does. Sometimes the model just works great out of the box, and you don’t need to do much more.

9. Autonomous Workflows and Hardware Limitations

But when the model is maybe 70% or 80% there, you see how it behaves and figure out what it’s not good at. Then you can collect a bit more data and iterate. I think of this as the robotics version of reinforcement learning from human feedback: the robot rolls out, you score it as good, bad, or somewhat good, and that data from the robot’s own experience can be used to train the model. You can also collect more data yourself to train a model. Eventually, our goal is for this robot to be a platform anyone can use to train tasks on.

I want to get a bit of a prediction and a timeline for where that goes. Right now, the tests are somewhat simple: take laundry out—it’s a towel, you can’t crush it.

Chop some food.

Yeah. There’s also the side of, “Set up this filming studio,” moving things around, bringing a table, setting up lights.

Manage my data center.

Yeah, plug things in.

And also recharge another robot.

Yeah. But when do we see the factory? What do we need to scale up? Is it the model, or is it everything? It seems like your hardware is set for that, but when do we get to the point where I can go from just cutting one thing to saying, “Okay, here’s a recipe—full send”?

To get to truly general-purpose robots—basically, a robot or a model that does anything—

Not that. To keep it somewhat grounded technically: when do we start pushing that feedback loop? Instead of simple chopping, maybe a 5- to 10-minute sequence, where the evaluation can break at many more points. Instead of filling one pill bottle, it’s 10 different pills in sequence. What’s limiting us? What are we pushing on?

A couple of things. The laundry demo you’ve seen is going to take an hour to finish a whole loop. The longest duration a robot can do a full job is already in the hour range.

Nice.

But to be able to do that for any task, I think the bottleneck comes back to data and hardware. For some tasks, like setting up a space, you might need more than a parallel-jaw gripper. You might need something more dexterous, like a human hand. But if you want to train a model on a hand, you’re bottlenecked by hardware stability. The best dexterous hands today still aren’t durable enough. They overheat pretty quickly, so that becomes a bottleneck.

We’re not just waiting to make a better model; we have a hardware limit.

Yeah. Hardware limitations are a big thing. If the hardware is limited, that slows everything down, because to know you’re making progress on any task, you eventually have to evaluate on the hardware. Without evaluation being very fast, that also limits research progress.

Then there’s the question of how we collect the right data for extremely complex tasks, like setting up this studio. Today, most of the data is vision-based: video streams and some encoding of your hand motion, while robots have joints. Humans have a lot more sensors. We hear things, we feel things—

Smell.

Yeah, we smell things. We subconsciously use all of those to help us manipulate the physical world so well. If you’re not capturing those things at all, then even if we scale to millions of hours, we might be missing critical dimensions of the task. You’ll never be able to solve the task.

10. Real-World Deployments at Best Western and Din Tai Fung

Those are what I consider to be dragging out the timeline. But for tasks or workflows where you can collect high-fidelity data today, I don’t necessarily think there’s a fundamental bottleneck. What we’re demonstrating to the public this week is a physical agent that can do the entire laundry workflow as a human would. For that task, we don’t need more than a parallel-jaw gripper or a sensing modality that isn’t available today. I do think the biggest bottleneck to having something much more general is still hardware and data.

I want to take this chance to let you talk about your customers and partners. You have some announced relationships. We can start with Best Western and then maybe some of the others you’re proud of—any cool stories?

We have a deployment blog post on our website that goes into much more detail. We also have one video for all the customers we’re working with. It shows what we’re doing. We have customers like Best Western in the hotel space. We fold a lot of laundry for them, like towels. Every hotel needs it. Every gym needs it. Every fitness center needs it.

Do you want the Best Western one?

No, we have one video for all the customers we’re working with. It’s a really nice video.

What’s the deployment so far? Where can I go to see it? Which is the best place to go?

If you want to see one, I recommend our other deployment at Din Tai Fung. Have you guys been?

Yes.

Yeah. You know, the best—

Very thin wrapper.

Yeah, very thin. We should have filmed there.

Yeah, we should have just done that there.

At Din Tai Fung, if you go to one of their somewhat upscale restaurants, there are napkins on your table, right? Our robots fold thousands of napkins for many different Din Tai Fung locations in the United States. That’s actually our biggest scaled customer.

I want to make a distinction here. Today, there are a lot of robotics companies talking about deployments and pilots, but what’s really hard isn’t putting your robot in one location or doing a pilot. What’s really hard is having customers validate that your robot creates enough value that they want to deploy it across many of their sites. That’s what’s happened for us with Din Tai Fung, which, to the best of my knowledge, hasn’t really happened for other companies doing similar general-purpose robots.

At Din Tai Fung, our robots fold thousands of napkins per day. The quality standard is extremely high. They want the second fold seam of the napkin to be just a couple of centimeters below the pocket, so you can stick the chopsticks in. It’s very fine-grained and high-precision.

At some level, that’s what robotics are good for, right? Robots are good at that, too. Humans aren’t the most consistent, especially on small details, whereas a robot can do it pretty much perfectly. It’s interesting that that’s the example you bring up for general-purpose robotics.

At some level, I think people would assume you could have a specialized robot that folds napkins perfectly to the exact millimeter, as opposed to something like this.

Yeah, maybe you can. But the hard thing about manipulating deformable objects—napkins, towels, everything—is that there are infinitely many possible configurations. That’s really hard for a specialized robot. What we find is that by developing a general-purpose model in a robot and adapting it to these use cases, you can get to production quality faster.

What’s really nice is that now Din Tai Fung has scaled and signed a much bigger contract with us. They don’t only want our robots to fold napkins; now they want our robots to cut food, too. That’s why we had the example of cutting food.

Is that actually the second use case?

That’s one of the use cases they’re thinking about.

Oh, wow. I think that robot needs to cut a lot of vegetables per day.

They need to do the dumplings.

Exactly. Do the fold. That’s the final boss.

I think you can. I think you have the dexterity and precision to do this.

Yeah.

But, like, don’t break the dumplings.

Yeah. That’s the obvious benefit: you can adapt them to more things. I think the other thing you touch on is speed. To make a custom napkin-folding machine with the exact fold, whatever—

You have to make the machine. You can’t just Amazon this. But this can do it pretty fast, right?

Yeah, this can do it very fast. And now it’s actually at human parity.

Yeah.

This robot doesn’t get tired. I really like that you mentioned consistency. One issue with Din Tai Fung is that their LA, New York, and Seattle locations fold napkins differently, but from their perspective, they want consistency. That’s exactly what these models are good at: the model is just a set of weights, so you can put those weights into different robots across the country and update them over the air. You can update all of them at once. Once it works, I think it’s truly incredible.

11. The Bitter Lesson of Robotics

The final boss is the Cheesecake Factory. If you know the story, they make everything from scratch. Everything is very consistent, down to the measured gram. Every sauce is made the same way, so anywhere you go in the world, you’ll have the exact same thing. If you don’t know, you don’t know. But that’s the fun of it.

Anyway, the term that comes to mind is the “bitter lesson” for robotics. Instead of specialized robots, which take a lot of industrial engineering, you build a generalized platform that can do last-mile tuning and all those things.

What’s the bitter lesson in robotics?

I think there are several bitter lessons for robotics. One is that robotics is going to scale just like any other domain that machine learning has solved. What does that mean? It means having as much data as possible. But in robotics, it’s much harder to figure out what the right data is than in other domains. In language, it’s text; in video games, it’s video game data. But for robotics, the sensors you choose determine what kind of data you have.

If you have the right set of sensors and scale that data, I think you’ll get predictable returns on model performance. In that way, robotics isn’t so different. A corollary of the bitter lesson is that there’s a lot of robotics research coming up with very specific priors. For example, people say we need to make the robot see objects as they are and draw some boxes that will give it better perception. But I think the bitter lesson is to keep the algorithm simpler and scale the right kind of data; then generalization and better behavior will emerge.

That’s the main bitter lesson: scaling will work. But in robotics, figuring out how to scale is much harder because there’s no internet. You have to come up with your hardware, choose your sensors, get all of those right, and then scale. I think that—

There’s the argument that the world is made for humans, so it should be a human. As hard as the hardware problem is, 10, 20, 100, 200 years down the line, can we make a really efficient human? But you have to be practical with what we have now, right?

The way I think about it is, yes, eventually the robot should have a human form factor if you want to build a general-purpose robot—a machine that can be adapted to as many use cases as possible. I think the human form factor will be the best one, just because that’s what we’ve been doing for ages.

But I think the best way to build a humanoid robot is to take an iterative approach. Robotics is so hard, and the whole problem—the whole stack—is so long, that it’s really hard to know a priori what the right decisions are. Hardware decisions are much more expensive because hardware iteration is a lot slower. You want to take an incremental, iterative approach, rather than going all the way to the final form factor. The chances are that the thing you build won’t be good enough, and you’ll end up scrapping it anyway.

It’s public knowledge that Optimus has already gone through many iterations and they’ve had to scrap their designs. By iterating with a semi-human form factor, putting robots out with real customers, and getting models really deployed, you get a feedback loop you wouldn’t get if you spent 5 or 10 years trying to build a warehouse humanoid. I think the iterative approach will get us closer to the final robot faster.

For those listening and not watching, this is probably the smoothest robot we’ve seen. Every humanoid you’ve seen is jerky, and even humans don’t walk as smoothly as—

The model can move every single motor at once.

Yeah, more than 2.

That’s actually a very hard problem. The model is outputting actions, and there’s an action dimension. The bigger the dimension, the harder it is to fit the model to the data. Here, the action space is quite large, but we did a very good job of making sure the model fits well and gives you smooth motion.

Can you bound the order of magnitude of the action space?

It’s on the order of about 100 action dimensions.

And these are all continuous, right?

Continuous, right? The model is basically predicting a vector of at least 50 dimensions at a very high frequency, 10 or 20 times a second, to get that continuous motion.

And potentially predicting 1 to 2 seconds out, so that’s another 100 frames or whatever. There’s a bit of a history, similar to context of the previous state, I guess.

Context is very important in robotics, because otherwise, if you’re just making decisions based on the current observation, sometimes it’s really hard to tell what to do next. For example, if you think about opening a microwave, in the middle part it’s unclear whether you’re closing it or trying to open it. Context is really important.

12. Agentic Harnesses for Physical AI

I have a screenshot of the fridge demo, where there’s agentic planning and tool calling and all those things. We can show that for listeners. I think that’s an interesting aside after the model. How do you wrap it around into an agentic harness? What is the deployment stack? You have a lot of cool UIs, but what goes into wrapping model output toward the final thing? Is this custom per use case? Do you have your own SDK?

Eventually, we will have our own SDK. The way we think about it is that we want a general harness layer for robotics foundation models. Today, that’s our own model. For any real, long task in the physical world, there’s a lot of high-level decision-making you have to do. It’s not just the robot reasoning about the physical world in continuous action trajectories. That’s the low-level thing you’re executing in the physical world, but it’s not the intent.

When we make decisions, we think about steps in a recipe. We think, “I finished this; I need to do that,” or, “While I’m waiting for this part to finish, I need to go somewhere else to do something else.” There’s a lot of high-level cognition and decision-making happening. For robots to handle even the laundry task we’re showing you, they need that capability. The robot needs to reason, “The washer is running, so I should go do something else while it finishes. Now the washer is done; I should take the towels from the washer and put them in the dryer immediately,” because otherwise that becomes a bottleneck in the whole workflow.

Right? There’s a lot of reasoning that needs to happen for robotics. In theory, you could try to learn that with a low-level model that outputs actions, but it would be extremely inefficient. That’s why having a hierarchical system—putting an agentic system around our world action model—allows the model to compose different capabilities and skills to solve much longer-horizon tasks.

13. GPT-6, Coding Agents, and AI-Native Robotics

It’s very much like language models. Two or three years ago, the model was only good at autocomplete. Humans had to tap, tap, tap to get the model to keep spitting out code. Today, an agentic system can carry out the end-to-end task. I think that’s important because, in robotics, people are still thinking about autonomy at the task level, like a 1-minute, 30-second task. But the real value is in tasks or workflows that run unsupervised for hours. That’s what we’re building toward, and that’s why I think about robotics not just as a modeling problem but as a systems problem. You need a good agent, a good harness, a good model, good hardware—everything. That’s how we think general-purpose robots ought to be built.

Do you guys see benefits from language-model coding tools getting better? New Opus 5.5, new Astra—do you have language models in your agent harness? How does AGI being developed in language-model space affect you guys?

No—absolutely. It affects us very deeply in many ways. One is that we’re an extremely AI-native company. Coding agents are a fundamental part of our work, and we really welcome them. Whenever there’s a new model, we’re the first to test it. When GPT-6 came out, we had a company-wide hackathon that same week, and the things our engineers built were incredible. Many of the features they built are already in main and in production. We really like using coding agents to accelerate our work.

Just for people who are interested, was there anything surprising in GPT-6 that you found?

Yeah. I think it’s the ability to construct virtual environments and turn them into simulations. We talked about that 20 minutes ago. I think that’s really exciting.

It’s interesting. There were so many demos, like CAD, Zillow-to-3D-ish.

He lost the hackathon to a CAD.

I won. I did the CAD.

The other way around. The other way around. No—oh.

I won a GPT hackathon just for fun. I know nothing about CAD, so I thought I might as well play around with it, because every demo was doing it. I thought it was going to be garbage, but it was kind of grounded and kind of good. It was just hackathon stuff.

Cool. I was going to round out some of the use cases and then go into a little bit of company culture and your personal background.

Yeah.

I have 2 technical questions. One is: are there any goals you want to set for yourself on a 6-month, 1-year, or 2-year timeline? Is there something you think robots should be able to do by then that you’re working toward?

Yeah. In the 6-month-to-1-year frame, our goal is definitely to scale up our robots and our model and truly get them deployed.

I mean a task. Right now, it’s cutting or opening a fridge. Do you set a goal like, “By this timeline, we should be able to do this,” to stay on track?

14. World-Action Models vs. VLAs

Yeah. In terms of deployments, we’re exploring many different industries. There isn’t a particular task we think we have to nail, because our goal is to build a general model, a general platform. We think of each different task as validation of our platform, and as a wedge for figuring out how to build everything better. Today, we’re doing a lot of that in laundry and restaurants. We’ve already deployed tens of robots, and our goal for next year is to deploy hundreds of robots overall.

My second part on the last technical question: how do you see the rest of the industry? For people listening, what other approaches are people taking to world models? Why are they wrong? What’s different but good about their approaches, too? What’s the landscape of robotic world models, and how have you arrived at your current approach?

Yeah, no, that’s a great question. Robotics is the most exciting space today, and there are so many amazing companies that I truly respect. They’ve been doing great frontier work, like us, and it’s very exciting. What we’re seeing today is that there are still many different approaches to robotics. We believe hardware needs to be built in-house; we need to own the whole chain. We believe scaling human data is extremely important. But there are companies betting on scaling only robot data, and others scaling some kind of intermediate data-collection devices. All of them are in play, and I think they’re different approaches. We’ll just see what happens.

I think it’s still a bit early to tell, but we feel pretty good about the way we’re building things. In terms of company culture and mission, we’re not here just to build a model. We’re here to truly get robots deployed, so we deploy as much as possible and work on the hard problems. That helps us identify problems much earlier and figure out the true bottleneck so we can solve it. For people who are working on only one part of the problem, it’s kind of like fog of war: you don’t really see the whole picture, and you might be building something technologically very cool that isn’t what people need or want in the end. There’s a whole graveyard of robotics companies. They all had very shiny, very cool robots, but figuring out what people actually want is really hard to execute, even though it’s easy to say, “Build something people want.”

On the technical side, you have your world action model. Is that standard across the industry, or are people taking other directions?

Yeah, I’d say that’s not standard, either. We’re building a world action model on top of a world model. There are many companies building world models. You mentioned Runway; they’re also adapting video world models. There are also companies building vision-language-action models, which means taking vision-language models and adapting them to robotics. There are many different ways to go about it.

At the end of the day, what we call the models is less important than what we believe the model needs to be able to do at a first-principles level. I want to explain that a bit. For robotics, the most important thing is being able to predict the future. Any physical system is fundamentally about what you should do next, based on the information you have at the present moment. It’s really about predicting what comes next. Predicting robot action is predicting what the robot should do next. But the robot is just a part of the world: the robot influences the world, and the world also influences how the robot should behave in that environment.

So, the most first-principles thing you can do is predict as much of the world as you possibly can. That’s why we believe predicting the future is extremely important. Today, the instantiation is a world action model, and a VLA only predicts robot actions with some more abstract representation of the world. But whenever you predict an abstract representation of the world, you’re encoding your own bias into what’s important to predict. The Bitter Lesson tells you to remove human biases and just take the rawest representation—the camera inputs—and predict all of that. That’s how I think about modeling for robotics today.

Just to press you a little bit on this: you’re basically saying your approach is better than the VLA, because you’re emphasizing prediction. I fully buy that. I like it; you’ve sold me on it. If you were wrong, what would be the most likely alternative? What’s the number two? If someone else is doing something interesting, what is it?

Yeah. There’s a caveat in what I said before, and I think that’s where I could be wrong. I said you need to predict as much of the world as possible, but that doesn’t tell you whether your sensor has everything about the world that you need to predict. I touched on this a little bit. If you train a world model purely based on videos, I think it’ll be a very powerful world model, but it may not be sufficiently rich because you’re missing critical aspects of the world, like sound and touch.

Where we could be wrong is that we have the best vision-based world model, but tomorrow someone else comes out with a vision-plus-tactile world model that’s 100× more data-efficient. Vision is a very high-information, highly redundant input, because every other pixel doesn’t change that much. You can understand the world without perfect vision, but tactile or another modality might be much denser in information. Predicting that might let you use much less vision and get to the same performance with much less compute and much less data.

I think you’ll be able to adapt very easily.

No, exactly. There’s also the question of how much data. The reason I bring this up is that if a vision-tactile world model, or a vision-tactile-audio world model, is what ultimately solves robotics, then I think robotics companies will be able to do that better than video-generation companies trying to pivot to robotics. Understanding sensing is actually very complicated. We’ve been building that since the company started, so we know the bottlenecks and how to combine the modalities. If you’re just trying to adapt a video model to robotic actions and you’re wrong—if the video model isn’t enough—you don’t really have the know-how to get tactile working very well. But we have that.

This is a side question, but I think it’s a quick one. Heat seems very important. Is infrared just another color to you, or is that—

Yeah, it could be another color, another input.

To the model.

It could be used in a cooking scenario, actually. I know companies that use infrared to tell how cooked their food is, because that’s really hard to tell from an RGB camera.

15. From AlphaGo to Founding Dyna Robotics

Right. My very naive pushback on your last point is that maybe we don’t need to understand hardware, cloud, code, or codecs. We can just vibe-code it in. There’s a lot you can do without understanding, but of course—

It’s just not in the distribution.

So I wanted to bring it back to your personal story and background. We both went to Penn.

I just found out today.

Yes, yes. My big regret is that I didn’t join the self-driving challenge.

Then I could have been early at Waymo or something. You were at GRASP, and then you also worked at Google and NVIDIA. What are the different focuses of academia versus Google versus NVIDIA? You’ve seen all these places, right? That obviously led you to say, “Screw all that. I’m going to do my own thing,” right? You could have joined OpenAI Robotics.

So can you take us through it chronologically, and share one thing you learned from each?

Yeah. Chronologically, it might be useful to talk about how I got into AI in the beginning. As a kid, I spent time in Japan and China. I grew up playing Go, so when AlphaGo came out, I was a senior in high school and I was mind-blown. I’d been playing the game for many years, and it’s so complicated. Before AlphaGo came out, people thought Go was one of the games AI would never solve. The search space is larger than the total number of atoms in the world—how could you possibly search for a solution in that? Then reinforcement learning cracked it. I think that’s how I got into AI initially. I was really interested in models or systems—agents that can self-improve.

When I got to Penn as a PhD student, I really wanted to work on reinforcement learning. I was co-advised, and both my advisers were interested in it. One of them was more focused on robotics at the time; he’d done a postdoc at the Berkeley Robotics Lab. That’s how I got into robotics at Penn. We have the GRASP Lab, one of the best places to do robotics, so I worked on reinforcement learning for robotics in my PhD. I did a lot of research on connecting reinforcement learning with human data, and on doing reinforcement learning theory in a rigorous way.

Then I went to NVIDIA. That’s when GPT-4 came out, and at the time we were very interested in solving the pen-spinning task. I don’t know why.

It’s the dexterity limit, right? You can solve Rubik’s cubes, and then—

At the time, I was working with Jim Fan and Yuke Zhu, who you guys probably know. We thought, “Let’s try to do dexterous pen spinning.” The initial idea was to learn it from human videos. We were trying to watch humans do it, but then I realized that we didn’t have a dexterous hand that could spin a pencil at all. How would I even evaluate the algorithm, even if learning from humans worked?

So I thought we needed to try it in simulation first. As soon as I tried to set up a simulated environment for a robot hand with a pen, I saw the issue. If I had to come up with the simulation environment and the reward function for reinforcement learning to solve pen spinning, that would be impossible. I wouldn’t be able to come up with all the terms you need to get reinforcement learning to learn the right behavior. Then I thought, what if a model that can write code could do that? That became the Eureka project. In my opinion, we had the first recursive self-improving robotic agent in simulation, because we used it to solve a new task that no one had demonstrated before. That showed me the potential of using language models and coding models to accelerate robotics.

Then I went to Google DeepMind, because at the time—and even today—they had one of the frontier models in Gemini. I was looking at how to use Gemini for robotics. Eventually, though, I realized that I’d had a lot of fun doing academic research, but there was always a question in the back of my mind: Does any of this matter? When you’re working on these things in a lab, you’re doing your own evaluations—you’re grading your own homework. I wanted something more real, something that could actually be used by millions of people.

I felt 2024 was the right time to take some of this technology into the real world. That’s why I started Dyna with Lynon New York [?]. Fast-forward 2 years, and we’ve built our platform and our model, and our robots are at Din Tai Fung. Go check it out and go eat.

16. Dyna's Research Culture and the Future of Robotics

It’s been a journey. What still gets me really excited is that a lot of the ideas I temporarily put aside when we started Dyna might become relevant again. To build a company, you have to be practical. I don’t think simulation was ready in 2024, but with the new frontier models and what we’re seeing internally, that might change. That’s very exciting.

And then let’s talk about company culture. What principles are you building the company around? You’re responsible for the science and research agenda, and when we meet the engineers, they seem very engaged. They have a lot of ownership and motivation.

It’s interesting to see how many side projects people just work on. Some of the demos we’ve seen were just engineers thinking, “This seems fun; might as well do it.”

To your point about fun evaluations, right here, 2 feet behind the camera—

Can’t talk about that.

No, I mean, the evaluation is just highlighters rubber-banded together, right? Who comes up with, “Ah, yes, we need to test putting rubber bands on highlighters”? It’s just stationery.

Fun stuff. People are very creative here. It’s a beautiful space.

Yeah. I think culture—let’s first talk about how we run a company. We think of ourselves as eventually becoming a research-and-product company. It’s kind of like OpenAI, where you do frontier research, but eventually it makes its way into Codex and ChatGPT. That’s how we think about it, too. We’re not here to do open-ended work on whatever interests us at the time. We’re product-driven: we want to deploy our robots and platform to a new customer. What are the technical problems we have to solve? We have to solve generalization, whole-body control, adapting quickly, getting robots to follow language, and dexterity.

Once those problems are clear, our engineers are very capable and smart. They create tasks that stress-test those capabilities. The grabbing-a-drink task is how we test language following: you specify what you want, and if the model follows intent and instructions, it should be able to do it. At the top level, we identify what we need to solve. Then we have a lot of bottom-up projects to drive execution. You hire the right people, and everyone here is passionate about robots. They’ve been doing this for a long time and love it. We create a culture where that happens naturally, and it’s a really fun place to work.

What’s the split between technical research and software versus the hardware side of the company?

We’ve grown so much recently that I don’t have the exact numbers, but I’d say around 30% to 40% of the company is on the hardware side, because hardware needs a lot more people. We have around 10 people working on models, plus software engineers, robotics engineers, and a lot of very talented people on the business side.

The most fun thing about building a robotics company is that you see everything. Anything under the sun that could possibly be helpful will be helpful. You get so much exposure to people from different backgrounds who are all passionate about robots. It’s been really eye-opening for me as a founder. When I was a researcher in a lab or a PhD student, I talked to researchers every day, and that was great. I still get to do that, but now I also learn from people from so many different walks of life. Some of the people who collect data had no robotics experience; they were working retail or in other jobs. They came here and became very well trained in robotics.

I think that’s pretty amazing to see. I think of myself as still learning in the robotics space, and I haven’t learned as much as I am now. Even though I’m not a student anymore, and I’m here running a company and managing a lot of things, I still feel like the day-to-day learning is just so much.

To close off, where is Dyna going? Two years, five years—what’s the vision?

Our high-level mission is to bring robots and intelligence into the physical world. That means seeing our robots deployed at thousands of customer sites, and eventually millions of robots. That’s what we want to get to.

What’s the story behind the name Dyna Robotics?

There are many stories. I named Dyna. There were a lot of different internal options, but the name came from 2 things. One is that Turing Award winner Richard Sutton, a pioneer in reinforcement learning, wrote a paper called “Dyna: An Integrated Architecture for Learning, Planning, and Reacting” many years ago. It’s basically about an agent that self-improves in a world model. Even when we started the company, we weren’t working on world models yet, but I felt that’s how robotics would get solved. Now our technology direction is converging toward that, so I felt it was a neat inspiration. I’ve also worked in reinforcement learning, so I have a lot of respect for Richard.

The second thing is more personal. I grew up in Japan, and the year I was born there was a cartoon show. I don’t know if you guys have seen Ultraman.

Yeah, of course.

The Ultraman that year was called Dyna.

Oh.

It’s a very subtle thing. Not many people know, but I guess now everyone will.

Yeah.

The team is finding out, too.

Yeah.

I think Richard would appreciate it. I don’t know if you’ve ever talked to him.

No, I’ve seen him at RL conferences.

You should just be like, “Hey, this company—”

Yeah, I’ll invite him. Come to you.

Everyone knows Richard for The Bitter Lesson, but, you know, yeah. Also, The Bitter Lesson.

Yeah. Any parting thoughts or calls to action?

I do feel like, as an industry, robotics is still very demo-oriented. I’ve done a lot of demos in the past in my lab, and even here we’ve had to show some demos. I think we need more people who are truly mission-focused on doing whatever it takes to get robots deployed, get robotic foundation models integrated with hardware, and see them at real customer sites. I think we’re a pioneer in this regard, and I want to see more in this field.

For the audience watching, if this mission inspires you, if solving the hardest problems today is what makes you wake up every day and you want to contribute, we’re always hiring. Looking forward to that.

Bay Area, or anywhere else?

Just the Bay Area today, but as the company grows, never say never to other locations.

That’s what I wanted to say. I want to see more robots in the world.

Cool. Congrats on everything. This is incredible progress for 2 years.

Yeah, amazing.

I really appreciate it. Thank you for visiting us. I hope you know it’s been fun.

I’d say I don’t regret doing this onsite.

We do a lot in our studio.

Yeah.

This was worth it.

Yeah. For robotics, you’ve got to do it onsite. For people listening in, watch the video version—we have fun B-roll.

This is Dyna 2.1, our brand-new hardware with a physical agentic harness around the World Action Model Dyna 2. Dyna 2.1 can automate an entire end-to-end workflow in a real physical industry. Here, we’re demonstrating an end-to-end laundry task in a real laundry room in a hotel. Behind me are the researchers for our project. They’re going to show you something really cool, a quick demo, and explain some of the technical details behind what made this possible. I’ll pass it to George.

I’m just going to show some quick steps in the entire laundry workflow. I’m going to start the robot. This is the task of transferring towels from the washer into the dryer. Let’s take a look.

This is an extremely hard task because the robot has to maneuver its body and stick its arms in to grab all the towels without colliding with the doors. It has to be very dexterous, and the motion is very smooth because a single model is controlling every joint in the body. There isn’t a separation between classical planning and acting that makes the motion less smooth.

Right?

Okay, cool. We can also simulate the robot realizing it dropped a towel. I’m just going to drop my towel here. We’ll see how it can tell that the towel has dropped and then pick it up. After it’s done putting the towel in the dryer, it also realizes that the towel is now dirty, so it puts it in the dirty basket.

I think this is a perfect case to showcase why we need to build a whole-body robot like this. It can access all the corners of the workspace and do everything a human worker can do. Next, it will close the dryer door and turn it on. It can also reason based on tactile information about which drying mode to use.

The most impressive part to me is turning the knob and actually operating a machine like a human would. It’s worth getting a close shot and making sure it’s on the right mode, then pressing a button to start the dryer.

It’s also important to realize that these are digital displays. You need more context because when you capture an image, the LED screen also blinks. That’s why it’s important for the robot to know how to start the dryer and recognize when it has started. It’s pretty cool.

This is just one part of the whole laundry workflow. On Tuesday, we’re going to drop the entire workflow. The whole workflow will take quite some time, as you can imagine.

Folding, stacking, transferring the basket—everything. Dyna 2.1 is about a robot becoming a whole employee in a workplace for the first time, doing everything. There’s no need for a human or any other machine to feed it items or help it with certain steps of the task. It’s just going to take on the whole thing and do it end to end. I think that’s a big breakthrough because historically we’ve thought of robots as machines, but this is the first time we can really see one working alongside humans to improve and increase physical productivity in the world. We’re super excited about it. Go ahead.

It will be our longest-horizon dexterous manipulation demo. We’re quite proud of the work we’ve done, and it’s exciting.

Here’s another demo we built using Dyna 2.1. The physical agent with our hardware is going to grab a drink for you based on whatever drink you select on this screen. Our researchers will give you a quick demo. In this case, they selected a vanilla shake, and the robot has received a command. It’s going to go to the fridge and use the World Action Model Dyna 2, which controls the whole robot, to get the right drink for you and retrieve it.

What’s really interesting is that this showcases the agent’s capabilities. The model isn’t just doing actions; it’s also following language commands faithfully and can navigate back to you to fulfill the order. It got the right drink. It’s going to close the door, and I think it’s switching modes.

Oh, yes.

It’s switching modes because this is the navigation part, which we can also have another high-level reasoning model do.

Anything you guys want to add about what’s happening? Anything cool you want to point out?

This one has a reasoning layer and an action layer, so it’s a 2-model architecture. Underneath, we send language prompts—a few-sentence instruction—to the reasoning model. The reasoning model plans what it needs to do step by step: first go into the fridge and grab something, then come back and hand it off to a human. It oversees the whole execution, while the lower-level action model performs the manipulation and dexterous tasks.

On the action model side, it also needs to be steerable, so you can tell it which of the 6 drinks you want to pick up. The video action model helps it imagine what it looks like when the robot grabs a drink, and that helps it understand your intent and follow what you tell it to do. There are also timing aspects to this.

It needs to recognize when you’ve grabbed the drink and then release at exactly that point. You don’t want false positives. You don’t want to drop a drink on the ground. There are all sorts of little things you have to be careful about to get a demo like this working.

Yeah. The model is quite good at this kind of dexterity and very robust. Like Chad was saying, it’s able to recognize and imagine what it’s going to look like if it grabs the right drink, and then execute.