No Priors Ep. 107 | With Physical Intelligence Co-Founder Chelsea Finn

Chelsea Finn, thank you so much for joining us today.

Yeah, thanks for having me.

You've done a lot of important, storied work in robotics through your work at Google, Stanford, and elsewhere. I would love to hear a little bit about your background and your path in the world of robotics. What drew you to it initially, and what are some of the projects you've worked on?

It's been a long road. At the beginning, I was really excited about the impact that robotics could have in the world, but at the same time, I was also fascinated by the problem of developing perception and intelligence in machines. Robots embody all of that, and there are sometimes some cool mathematical problems that keep your brain active and make you think. I think all of that is really fun about working in the field.

I started working more seriously in robotics more than 10 years ago, at the start of my PhD at Berkeley. I was working on neural network control, trying to train neural networks that mapped from image pixels directly to motor torques on a robot arm. At the time, this was not very popular, but we've come a long way, and it's a lot more accepted in robotics now. It's also something that a lot of people are excited about more generally.

Since that beginning, it was very clear to me that we could train robots to do pretty cool things, but getting the robot to do one of those things in many scenarios with many objects was a major challenge. Ten years ago, we were training robots to screw a cap onto a bottle, use a spatula to lift an object into a bowl, do a tight insertion, or hang a hanger on a clothes rack. That was pretty cool stuff, but actually getting the robot to do those things in many environments with many objects was where a big part of the challenge came in.

I've been thinking about ways to make broader datasets and train on those broader datasets, as well as different approaches to learning, whether it be reinforcement learning, video prediction, or imitation learning. I moved from Berkeley and spent a year at Google Brain between my PhD and joining Stanford. I became a professor at Stanford, started a lab there, and did a lot of work along all of those lines.

Then, recently, I started Physical Intelligence, almost a year ago at this point. I've been on leave from Stanford for that, and it's been really exciting to try to execute on the vision that the co-founders collectively have, and to do it with a lot of resources. I'm also still advising students at Stanford.

That's really cool. I guess you started Physical Intelligence with 4 other co-founders and an incredibly impressive team. Could you tell us a little bit more about what Physical Intelligence is working on and the approach you're taking? I think it's a pretty unique slant on the whole field.

We're trying to build a big neural network model that could ultimately control any robot to do anything in any scenario. A big part of our vision is that, in the past, robotics has focused on going deep on one application—developing a robot to do one thing—and then ultimately getting stuck in that one application. It's really hard to solve one thing and then broaden beyond it.

Instead, we're in it for the long term, trying to address this broader problem of physical intelligence in the real world. We're thinking a lot about generalization and generalist robots. Unlike other robotics companies, we think that being able to leverage all of the possible data is very important.

That means not just leveraging data from one robot, but from any robot platform. It might have 6 joints or 7 joints, 2 arms or 1 arm. We've seen a lot of evidence that you can transfer rich information across these different embodiments. It allows you to use more data, and if you iterate on your robot platform, you don't have to throw all of your data away.

We've faced a lot of pain in the past where we got a new version of the robot, and then our policy didn't work. It's a really painful process to try to get back to where you were on the previous robot iteration. So we're trying to build generalist robots and essentially develop foundation models that will power the next generation of robots in the real world.

That's really cool. There are a lot of parallels to the language-model world, where a mixture of deep learning, the Transformer architecture, and scale has really demonstrated that you can get real generalizability and different forms of transfer between different areas. Could you tell us a little bit more about the architecture you're using, or how you're thinking about the basis for the foundation model you're developing?

At the beginning, when we were just getting off the ground, we were trying to scale data collection. Unlike in language, we don't have Wikipedia or an internet of robot motions. We're really excited about scaling data on real robots in the real world. This kind of real data is what has fueled machine-learning advances in the past, and we actually need to collect that data. That looks like teleoperating robots in the physical world.

We're also exploring other ways of scaling data, but the bread and butter is scaling real robot data. We released something in late October where we showed some of our initial efforts around scaling data and how we could learn very complex tasks, such as folding laundry, cleaning tables, and constructing a cardboard box.

Where we are in that journey now is thinking a lot about language interaction and generalization to different environments. What we showed in October was a robot in 1 environment, and it had data from that environment. We were able to see some amount of generalization: it was able to fold shirts that it had never seen before and fold shorts that it had never seen before. But the degree of generalization was very limited, and you also couldn't interact with it in any way. You couldn't prompt it and tell it what you wanted it to do beyond fairly basic things that it had seen in the training data.

Being able to handle lots of different prompts and lots of different environments is a big focus right now. In terms of the architecture, we're using Transformers and pre-trained vision-language models. That allows you to leverage all of the rich information on the internet.

We had a research result a couple of years ago where we showed that if you leverage vision-language models, you could get a robot to do tasks that require concepts that were never in the robot's training data but were on the internet. One famous example is that you can pass the cat can to Taylor Swift, or pass a picture of Taylor Swift. The robot has never seen Taylor Swift in person, but the internet has lots of images of Taylor Swift, and you can leverage all of that information and the weights of the pre-trained model to transfer it to the robot.

So we're not starting from scratch, and that helps a lot as well. That's a little bit about the approach. I'm happy to dive deeper.

That's really amazing. What do you think is the main basis for getting to generalizability? Is it scaling data further, scaling compute, or some combination of the two? Is it other forms of post-training?

As you think through the common pieces that people look at now, I'm curious what you think needs to be filled in. In the language-model world, people are spending a lot of time on reasoning modules and other things like that. What are the components that you feel are missing right now?

I think the number-one thing—and this is kind of the boring thing—is just getting more diverse robot data. For the release we had in late October last year, we collected data in 3 buildings. The internet, for example, and everything that has fueled language models and vision models, is way more diverse than that, because the internet contains pictures taken by lots of different people and text written by lots of different people.

Trying to collect data in many more diverse places, with many more objects and many more tasks, is very important. It's not just about scaling the quantity of the data; it's about scaling the diversity of the data. That's a big thing we're focusing on right now: bringing our robots into lots of different places and collecting data there.

As a side product of that, we also learn what it takes to get a robot to be operational and functional in lots of different places. That's a really nice byproduct, because if you actually want to get robots to work in the real world, you need to be able to do that.

That's the number-one thing, but we're also exploring other approaches: leveraging videos of people, leveraging data from the web, leveraging pre-trained models, and thinking about reasoning. More basic forms of reasoning can help with things like putting a dirty shirt into a hamper. You need to recognize where the shirt is, where the hamper is, and what you need to do to accomplish the task.

Or, if you want to make a sandwich and the user has a particular request in mind, you should reason through that request. If they're allergic to pickles, you probably shouldn't put pickles on the sandwich. Those are some of the basic things we're thinking about, although the number-one thing is still more diverse robot data.

A lot of the work you've done to date has emphasized releasing open-source models and packages for robotics. Do you think that's the long-term path? Do you think it's open core, or do you think it eventually becomes proprietary models?

How do you think about that in the context of the industry? There are a few different robotics companies taking different approaches: hardware only; hardware plus software focused on a specific hardware footprint; and software companies choosing between closed source and open source. Where in that spectrum does Physical Intelligence lie?

We've actually been quite open. We've open-sourced some of the weights and released details in technical papers. We've also been working with hardware companies and giving them designs for robots.

When I tell people this, they're sometimes shocked. They ask, “What about the IP? What about confidentiality?” We've made this a very intentional choice for a couple of reasons.

First, we think the field is really just at the beginning. These models will be so much better in 1 year and in 3 years, and the robots should be so much better as well. We want to support the development of the research, support the community, and support the robots so that when we hopefully develop the technology for these generalist models, the world will be more ready for it. We'll have more robust robots that are able to leverage those models, as well as people who have the expertise and understand what it requires to use them.

The other thing is that we have a fantastic team of researchers and engineers. Really great researchers and engineers want to work at companies that are open, especially researchers. They want to get credit for their work, share their ideas, and talk about their ideas. We think that having the best researchers and engineers will be necessary for solving this problem.

The last thing I'll mention is that I think the biggest risk with this bet is that it won't work. I'm not really worried about competitors. I'm more worried that no one will solve the problem.

Why do you worry about that?

I think robotics is very hard, and there have been many failures in the past. Unlike recognizing an object in an image, there's very little tolerance for error. You can miss a grasp on an object, or fail to make contact with an object, and the difference between making contact and not making contact can be very small. That has a massive impact on whether a robot can successfully manipulate the object.

That's just 1 example. There are challenges on the data side of collecting data, and anything involving hardware is hard as well.

We have a number of examples now of robots in the physical world, from autopilot on a jet to forms of pick-and-pack and other types of robots in distribution centers. There are also the different robots involved in manufacturing, particularly in automotive.

There have been a handful of more constrained environments where people have been using robots in different ways. Where do you think the impact of these models will first show up? There are certain things where you have very low tolerance for error, and then there are a lot of fields where it's okay, or where you can constrain the problem sufficiently relative to the capabilities of the model that it works fine.

Where do you think Physical Intelligence will have the nearest-term impact? Or, in general, where will the field of robotics and these new approaches first substantiate themselves?

As a company, we're really focused on the long-term problem and not on any 1 particular application, because of the failure modes that can come up when you focus on one application. I don't know where the first applications will be.

I think one thing that's challenging is that, typically in machine learning, a lot of the successful applications—recommender systems, language models, image detection—have humans as the consumers of the model outputs. Humans can check those outputs, and humans are good at the thing being evaluated.

A lot of the very natural applications of robots involve a robot doing something autonomously on its own. It's not like a human consuming the commanded arm position, checking it, validating it, and so forth. I think we need to think about new ways of having some kind of tolerance for mistakes, scenarios where mistakes are fine, or scenarios where humans and robots can work together.

That's 1 big challenge that will come up when we're trying to deploy these systems. Some of the language-interaction work we've been doing is motivated by this challenge. We think it's really important for humans to be able to provide input about how they want the robot to behave, what they want the robot to do, and how they want the robot to help in a particular scenario.

That makes sense. The other form of generalizability, at least in our current world, is human form. Some people are specifically focused on humanoid robots, like Tesla and others, under the assumption that the world is designed for people and therefore the humanoid form is the perfect form factor to coexist with people. Other people have taken very different approaches, saying they need something more specialized for the home, for factories, for manufacturing, or for some other environment.

What's your view on humanoid versus non-humanoid robots?

On one hand, I think humanoids are really cool. I have one in my lab at Stanford. On the other hand, I think they're a little overrated.

One practical way to look at it is that we're generally fairly bottlenecked on data right now. Some people argue that with humanoids, you can collect data more easily because the form factor matches the human form factor, so it might be easier to mimic humans. I've heard people make those arguments, but if you've ever actually tried to teleoperate a humanoid, it's a lot harder to teleoperate than a static manipulator or a mobile manipulator with wheels.

Optimizing for the ability to collect data is very important. If we can get to the point where we have more data than we could ever want, then it comes down to research, compute, and evaluations. That's 1 of the things we're optimizing for. We're using cheap robots and robots for which we can very easily develop teleoperation interfaces, so that you can teleoperate them very quickly and collect diverse data and lots of data.

It's funny: there was that viral fake Kim Kardashian video of her going shopping with a robot following her around and carrying all of her shopping bags. When I saw that, I really wanted a humanoid robot to follow me around everywhere. I thought it would be really funny to do that.

I'm hopeful that someday I can use your software to cause a robot to follow me around and do things. It's an exciting future.

How do you think about the embodied model of development versus non-embodied approaches? A lot of the AI community is focused on language models, vision-language models, and so forth, and there's a ton of hype around reasoning. People may underestimate how much intelligence goes into motor control.

Many years of evolution led to us being able to use our hands the way that we do. There are many animals that can't do it, even though they've had so many years of evolution. I think there's actually an enormous amount of complexity and intelligence involved in doing something as basic as making a bowl of cereal or pouring a glass of water.

In some ways, I think embodied intelligence, or physical intelligence, is very core to intelligence and is perhaps underrated compared with some of the less-embodied models.

One of the papers I really loved over the last couple of years in robotics was your ALOHA paper. I thought it was a very clever approach. What research over the last 2 or 3 years do you think has really caused this flurry of activity?

A number of people are now starting companies in this area because they feel like now is the time to do it. I'm curious which research you think was the basis for that shift and for people thinking this was a good place to work.

For us, there were a few things that felt like turning points—places where the field was moving a lot faster than it had before. One was the SayCan work, where we found that you could plan with language models as the high-level component and then plug that into a low-level model to get a robot to do long-horizon tasks.

Another was the RT-2 work, which showed that you could do the Taylor Swift example I mentioned earlier. You could bring in a lot of the web data and get better generalization on robots.

A third was our RT-X work. We were able to train models across robot embodiments. We took all of the robot data that different research labs had, which was a huge effort to aggregate into a common format, and trained on it.

When we trained on that data, we found that we could take a checkpoint, send it to another lab halfway across the country, and have a graduate student at that lab run the checkpoint on their robot. More often than not, it would do better than the model that they had specifically iterated on in their own lab. That was another big sign that this was starting to work, and that you could get benefits by pooling data across different robots.

As you mentioned, I think the ALOHA work, and later the Mobile ALOHA work, showed that you could teleoperate robots and train models on pretty complicated dexterous-manipulation tasks. We also had a follow-up paper on tying shoelaces. That was a fun project because someone said that they would retire if they saw a robot tie shoelaces.

Did they retire?

They did not retire. We need to force them into retirement, whoever that person is. We need to follow up on that.

I think those are a few examples. We've seen a ton of progress in the field. I also think that after we started Physical Intelligence, it was a sign to others that if the experts were really willing to bet on this, then something might happen.

One thing that you released from Physical Intelligence today is what you call a hierarchical interactive robot, or Hi Robot. Can you tell us a little bit more about that?

This was a really fun project. There are 2 things we're trying to look at. One is that if you need to do a longer-horizon task—a task that might take minutes—then training a single policy to output actions based on images might not work as well as something that is actually thinking through the steps required to accomplish the task.

That was the first component, and that's where the hierarchy comes in. The second component is that, a lot of the time, when we train robot policies, we take our data and annotate it by saying, “This is picking up the sponge,” “This is putting the bowl in the bin,” or “This segment is folding the shirt.” Then you get a policy that can follow basic commands like “fold the shirt” or “pick up the cup.”

At the end of the day, though, we don't just want robots to be able to follow those commands. We want them to interact with us. We want to be able to say, “I'm a vegetarian. Can you make me a sandwich? I'm allergic to pickles, so don't include those.” We might also want to interject in the middle and say, “Hold off on the tomatoes.”

There's a big gap between something that can follow an instruction like “pick up the cup” and something that can handle those kinds of prompts, situated corrections, and so forth. We developed a system that has 1 model taking as input the prompt and reasoning through it. That model outputs the next step the robot should follow, such as “pick up the tomato.”

Then there's a lower-level model that takes “pick up the tomato” as its input and outputs a sequence of motor commands for the next half second or so. That's the gist of it.

It was a lot of fun because we got the robot to make a vegetarian sandwich or a ham-and-cheese sandwich. We also did a grocery-shopping example and a table-cleaning example. I was excited about it, first, because it was cool to see the robot respond to different prompts and do these challenging tasks, and second, because it seems like the right technical approach for solving the problem.

On the technical-capability side, I was wondering about something. If I look at the world of self-driving, there are a few different approaches being taken. One of the more Waymo-centric approaches incorporates a variety of sensors besides vision, such as lidar, to augment the self-driving capabilities of a vehicle.

Where do you think we are in terms of the sensors we use in robots? Is there anything missing, or anything we should add? Are there types of inputs or feedback that we need to incorporate that haven't been incorporated yet?

We've gotten very far just with vision, even with RGB images. We typically have 1 or multiple external cameras looking at the scene, as well as cameras mounted to each of the robot's wrists. We can get very far with that.

I would love it if we could give our robots skin. Unfortunately, a lot of the tactile sensors that are out there are either far less robust than skin, far more expensive, or very low resolution. There are a lot of challenges on the hardware side there. We've found that mounting RGB cameras to the wrists ends up being very helpful and probably gives you a lot of the same information that tactile sensors can provide.

When I think about the set of sensors incorporated into a person, obviously there's the equivalent of tactile sensors, as you mentioned, and then heat sensors. There are actually a variety of things incorporated into people that we usually don't think much about.

How many of those are necessary in the context of robotics versus not? If we extrapolate from humans, animals, or other biological systems, what are some of the things we should think about?

It's a great question. For sandwich-making, you could argue that you'd want the robot to be able to taste the sandwich to know if it's good or not, or smell it at least.

I've made a lot of arguments for smell to Sergey in the past because there are a lot of nice things about it, although we've never actually attempted it. In some ways, the redundancy is nice. For example, audio can help a human: if you hear something unexpected, it can alert you to something.

In many cases, audio might be very redundant with your other sensors, because you might be able to see something fall, for example. That redundancy can lead to robustness. For us, it's not currently a priority to look into these sensors because we think the bottleneck right now is elsewhere—in the data, the architectures, and so forth.

The other thing I'll mention is that our policies currently don't have any memory. They only look at the current image frame; they can't remember even half a second prior. I would much rather add memory to our models before we add other sensors. We can have commercially viable robots for a number of applications without adding other sensors.

What do you think is the time frame on adding memory?

I have no idea. Some parts of robotics make it easier than self-driving, and some parts make it harder.

On one hand, robotics is harder because it's a much higher-dimensional space. Even our static robots have 14 dimensions—7 for each arm—and you need to be more precise in many scenarios than you do when driving. We also don't have as much data right off the bat.

On the other hand, with driving, you kind of need to solve the entire distribution to have anything viable. You have to be able to handle an intersection at any time of day, with any possible pedestrian scenario, other cars, and so forth. In robotics, I think there are lots of commercial use cases where you don't have to handle the whole huge distribution, and you also don't have as much of a safety risk. That makes me optimistic.

I think the results in self-driving have been very encouraging, especially the number of Waymo vehicles I see in San Francisco. It's been very impressive to watch them scale up usage.

What I found striking about the self-driving world is that 2 dozen startups started roughly 10 to 15 years ago, and the industry is largely consolidated, at least in the United States. The China market is a bit different, but in the United States it's consolidated into Waymo and Tesla, which effectively were 2 incumbents: Google and an automaker.

There are maybe 1 or 2 startups that either SPAC'd and went public or are still working in the area, and most of the rest have fallen off. The set of players that existed at that starting moment 10 or 15 years ago was largely the same set of players that ended up winning. There hasn't been a lot of dynamism in the industry, other than consolidation.

Do you think the main robotics players are the companies that exist today? Do you think there's any kind of incumbency bias that's likely to persist?

A year ago, it would have been completely different. We've had so many new players recently. The fact that self-driving ended up that way suggested that it might have been a bit too early 10 years ago, and I think deep learning has come a long way since then.

With robotics, if you had asked me 10 years ago, or even 5 years ago, honestly, I think it would have been too early. The technology wasn't there yet. We might still be too early for all we know. It's a very hard problem, and how hard self-driving has been is a testament to how hard it is to build intelligence in the physical world.

In terms of major players, there are a lot of things I've liked about the startup environment and a lot of things that were very hard to do when I was at Google. Google is an amazing place in many ways, but, as 1 example, taking a robot off campus was almost a nonstarter just for code-security reasons. If you want to collect diverse data, taking robots off campus is valuable. You can move much faster when you're a smaller company and don't have restrictions, red tape, and that sort of thing.

The really big companies have a ton of capital, so they can last longer, but I also think they're going to move more slowly.

If you were to give advice to somebody thinking about starting a robotics company today, what would you suggest they do? Where would you point them in terms of what to focus on?

The main advice I would give someone trying to start a company would be to try to learn as much as possible, quickly. I think that trying to deploy quickly, learn, and iterate quickly is probably the main advice.

Actually get the robots out there, and learn from that. I'm also not sure I'm the best person to give startup advice because I've only been an entrepreneur for 11 months, but that's probably the advice I would give.

You're running an incredibly exciting startup, so I think you have every ability to give people advice in that area.

I've heard a number of different groups say that they're using observational data of people as part of their training set. That could be YouTube videos or things people are recording specifically for that purpose. How do you think about that in the context of training robotic models?

I think that data can have a lot of value, but by itself it won't get you very far. There are some nice analogies. For example, if you watch an Olympic swimmer race, even if you had their strength, their practice at moving their own muscles to accomplish what they're accomplishing is essential for being able to do it.

If you're trying to learn how to hit a tennis ball well, you won't be able to learn it just by watching the pros. These examples might seem a little contrived because they're about experts. The reason I make those analogies is that we humans are already experts at low-level motor control for a variety of things, and our robots are not.

I think robots need experience from their own bodies in order to learn. It's really promising to leverage observational data to expand on the robot's own experience, but it's also essential to have data from the robot itself.

Is that just general data that you're generating around the robot, or would you actually have it mimic certain activities? How do you think about data generation?

You mentioned transfer and generalizability. It's interesting to ask what is generalizable and what isn't, and what types of data are in or out.

When we collect data, it's kind of like puppeteering, as in the ALOHA work. You can record both the actual motor commands and the sensor data, such as the camera images. That becomes the experience for the robot.

I also think autonomous experience will play a huge role. Just as we've seen in language models, after you get an initial language model, if you can use reinforcement learning to have the language model bootstrap from its own experience, that's extremely valuable.

In terms of what's generalizable and what isn't, I think it all comes down to the breadth of the distribution. It's really hard to quantify or measure how broad the robot's own experiences are. There's no way to categorize the breadth of the tasks—how different 1 task is from another, or how different 1 kitchen is from another.

We can at least get a rough idea of that breadth by looking at things like the number of buildings or the number of scenes. Those are the kinds of measures we can use.

We've talked a lot about humanoid robots and other formats. If you think ahead in terms of the form factors likely to exist in N years, as this robotic future comes into play, do you think there's 1 singular form, a handful of forms, or a rich ecosystem like biology?

I don't know exactly, but my bet would be on a world with a really wide range of different robot platforms. Sergey, my co-founder, likes to call it a Cambrian explosion of different robot hardware types and so forth, once we have the technology and intelligence to power all of those different robots.

I think it's similar to how we have all these different devices in our kitchen that can do different things for us, rather than 1 device that cooks the entire meal. We can envision a world with 1 kind of robot arm that does things in the kitchen, with hardware optimized for that use case and perhaps optimized to be cheap for that particular application.

There might be another kind of hardware designed for folding clothes, dishwashing, or other tasks. This is all speculation, of course, but I think that kind of world is possible.

In the book The Diamond Age, there's a view of matter pipes going into homes, where you have 3D printers that make everything for you. In one case, you're downloading schematics and then 3D-printing the thing, while people who are bootlegging some of the designs end up with almost evolutionarily based processes to build hardware and select for certain functionality as a mechanism for optimizing things.

Do you think a future like that is at all likely? Or do you think it's more that you make the foundation model really good, have a couple of form factors, and don't need that much specialization if you have enough generalizability in the underlying intelligence?

I think a world like that is very possible. You can make a cheaper piece of hardware if you're optimizing it for a particular use case, and it might also be a lot faster.

It's obviously very hard to predict.

It's super hard to predict because one of the arguments for having a smaller number of hardware platforms is supply chain. It will be cheaper at scale to manufacture all the subcomponents, so you'll collapse down to fewer things. Unless there's a dramatic cost advantage, those fewer things will be more easily scalable, reproducible, and cheap to make.

If you look at general hardware approaches, it's an interesting question in terms of the trade-off between those 2 tensions.

Although maybe we'll have robots in the supply chain that can manufacture any customizable device you want. It will be robots all the way down.

That's our future. Thanks so much for joining me today. It was a super interesting conversation, and we covered a wide variety of things. I really appreciate your time.

Yeah, this was fun.