No Priors 第141期|与 Sunday Robotics 联合创始人 Tony Zhao 和 Cheng Chi 对谈
Tony Zhao 将机器人行业定位在“GPT 时刻和 ChatGPT 时刻之间”:这个领域似乎已经找到了可规模化的配方,但尚未把规模转化为优秀的消费级产品。 传统的感知—规划—执行系统要求每个任务、每种环境都定制接口;新一代学习方法则试图扩展数据和模型,而不是反复重建面向单一任务的系统。他押注机器人会沿着其他 AI 领域的路径发展:相关数据越多,实用性就越强。
Sunday 的核心资产,是一套真实世界数据引擎:已采集近1000万条长时程轨迹,参与者超过500人。 Cheng Chi 的 UMI 工作以 GoPro 和夹爪替代受实验室限制的遥操作;Diffusion Policy 稳定了多模态模仿学习,ALOHA 让灵巧数据采集更直观,ACT/action chunking 加上 transformer 则帮助双手任务实现规模化。3名学生在约2周内采集了1500段端到端的浓缩咖啡制作视频。
手套式采集的表面简单,掩盖了全栈执行的难度。 Sunday 将数据设备、机器人控制、自动筛选与清洗、校准、训练流水线和机械设计放在一起迭代;手套从 V0 做到 V5,每个版本约迭代20次。正如 Zhao 回忆,创始人最初担心有人会“直接拿走我们的手套”(“just take our glove”),后来才发现“事情比我们想象的难得多”。
当简化能更快、更便宜地造出有用机器人时,Sunday 不追求人形机器人的完整性。 这款友好、卡通化的机器人只有3根手指,因为人们做家务时经常会并用它们,比如抓握把手或打开洗碗机。感知能力可以修正廉价、柔顺且不精确的硬件,让机器人在保持机械安全与柔顺性的同时,达到足够的精度。
创始人目前认为,模仿学习在操作任务上的样本效率更高,而强化学习在移动任务上表现良好。 地面接触相对容易模拟,但要复现会变形的手、透明杯、橙汁、反光和各种物理属性,极其困难且昂贵。对于操作任务而言,“把手放到杯子前面,再以合适力度合拢”可能比模拟整个世界更容易示范。
商业化将从2026年的家庭内测开始,2027年或2028年能否向大众交付,明确取决于可靠性、能力、安全性和价格。 当前原型机成本为6000美元至2万美元;达到几千台的产量后,Sunday 预计材料成本大概率低于1万美元,因为昂贵的小批量外壳加工会转向注塑成型,意味着售价可能在这一水平附近。Zhao 设想10年内家庭机器人数量超过10亿台,而 Cheng 描绘的一种未来是:家庭劳动的边际成本趋近于零。
创始人评估机器人视频的原则是“零假设”(“make zero assumptions. No Priors.”)。 先验证机器人是否自主运行,再把视频中被验证的能力限定为那个具体物体、具体人、具体环境和具体操作序列,而不是据此推断机器人具备泛化能力。Sunday 的证据包括长时程餐桌清理、易碎玻璃处理、在约6套 Airbnb 房源进行零样本测试、制作浓缩咖啡和叠袜子;不过 Cheng 表示,团队开发期间打碎了很多玻璃。
1. 机器人已有规模化配方,但还没有 ChatGPT 级产品
Zhao 的框架是:机器人处在“GPT 时刻和 ChatGPT 时刻之间”(“in between the GPT moment and the ChatGPT moment”)。研究人员越来越认可一些有前景的操作方法,但没有人知道从机器人领域相当于 GPT-2 的阶段跃迁到 GPT-3 会带来什么,因为这个领域直到最近才获得具有实际规模的数据。
传统机器人学依赖人类设计的感知—规划—执行模块。每个任务和环境都要重新设计接口——本质上“每个任务都对应一篇论文”——因此研究机构和公司不断丢弃面向单一任务的工程成果,而不是积累通用能力。
Tony 介绍,Diffusion Policy 解决了示范数据对同一观测包含多个有效响应时模仿学习不稳定的问题。这使多个操作员、包括一些未经训练的操作员能够贡献数据,同时避免训练发散或机器人行为异常。
2. 更好的接口先释放灵巧性,真实世界数据再释放规模
ALOHA 大幅缩短了人体动作与机器人响应之间的延迟,让遥操作更像玩电子游戏。示范动作变得流畅且灵巧后,transformer 开始表现良好;此前机器人学依赖3层 MLP 和 ConvNet 多年,一直难以用好 transformer。
ACT 和 action chunking 预测的是一段轨迹,而不是单个毫秒级动作。Zhao 的直觉来自生物学:人类先感知,再持续移动一段时间而无需再次观察,因此动作分块能带来更一致的运动和更好的整体表现。
Chi 摆脱实验室遥操作限制的关键是 UMI:用3D打印夹爪和 GoPro 同步捕捉视频与手部动作。3名学生把设备带进餐厅,在2周内采集了约1500段制作浓缩咖啡的视频,形成异常庞大的数据集,并训练出一套端到端策略,让机器人能在 Stanford 周边未见过的地点完成出杯。
失败案例同样提供了关键信息:由于采集期一直下雨,这套策略在直射阳光下失效。“要让机器人在阳光环境中工作,它必须见过阳光环境”说明,决定泛化能力的是真实世界数据的覆盖广度,而不只是轨迹数量。
3. 全栈迭代把采集规模转化为执行难题
Sunday 从两位创始人在 Chi 的公寓里把机器人固定在桌上起步,发展到2024年末约8人,再到约30-40人。打造产品而不是演示原型,要求机械工程、控制、软件、AI、供应链和运营共同优化同一套系统。由于通用家用机器人尚不存在,正确的接口和标准仍未确定;创始人表示,这也让外部合作伙伴很难使用,因为他们对“好”的标准一直在变化。
目前近1000万条真实世界轨迹已覆盖导航和长时程任务,而不只是孤立的拿杯子动作。超过500人使用手套,持续暴露各种故障模式;V5 设备从 V0 到 V5,每个版本约经历20次迭代。自动校准和故障检测保护了数据质量,不必让人逐段检查每个视频。
4. 有用的家用机器人,靠简化人形取胜
Sunday 的使命,是把家用机器人放进每个人家里,接管那些对人之所以为人“真正重要”程度不高的家务,把时间还给家庭、爱好和热情。如果机器人变得便宜、安全且有能力,Zhao 设想10年内家庭机器人数量超过10亿台。
Sarah Guo 追问:为什么不直接造一个完整的人形机器人?核心设计答案是,只要不影响实用性,就尽可能简化。3根手指已经覆盖抓握把手或打开洗碗机的大部分收益,无需为通常会协同运动的手指分别增加执行器。
工业机器人必须快速、刚性高且精确,因为它们是在“盲目执行一条轨迹”。感知能力改变了这一约束:廉价、柔顺的执行器在机械上可能不够精确,但 AI 系统可以修正硬件误差,在保持“机械上天然安全且柔顺”的同时,满足家庭场景所需的精度。
商业化仍取决于测试结果。2026年,经过筛选的用户将把真实机器人带回家;能否在2027年或2028年向大众交付,要看实际表现。原型机成本为6000美元至2万美元,但达到几千台的产量后,Sunday 预计材料成本大概率低于1万美元,主要因为 CNC 加工、手工喷漆的外壳会转向注塑成型。
5. 模拟越难,操作任务越偏向模仿学习
创始人最初预计,手套数据会落后于分布完全匹配的遥操作。实际情况相反:这种形态促成了更自然、更灵巧的动作;经过约20轮工程迭代,以及将人类影像转换为机器人风格数据的全流程软硬件工作后,他们已经不再认为两者存在有意义的质量差距。
强化学习适合那些易于模拟的环境,尤其是移动任务,因为相关物理模型主要是刚体动力学和地面接触。Chi 表示,完美的模拟器可以让任何任务都成为可能,因此实际问题在于哪种方法能更快实现目标。操作任务则恰好相反:动作本身可能更容易采集,而准确模拟透明容器、液体颜色、反光、会变形的手、接触和力度,成本极高。
数据规模扩大后,数据质量的重要性反而上升。硬件故障和不受控的真实世界行为要求持续监控和可重复的清洗流程;与此同时,Sunday 在采访前约3个月才开始认真开展研究,因为数据稀缺时期形成的“可爱、花哨的研究想法”未必能扩展成产品。
Chi 认为仍有两项挑战:找到能在大规模数据上稳定运行的训练配方,以及让硬件可靠。学习团队不断把硬件推向极限,因此零部件会损坏;机械团队和学习团队在同一家公司内协作,可以迅速把故障反馈到设计环节。
6. 演示可信度取决于自主性、泛化能力和序列长度
Zhao 的第一道检验,是判断演示由机器人自主完成,还是由人遥操作。如果视频只展示把一个杯子递给一个人,就只能假设这一个具体交互有效;观众会本能地联想到不同的杯子、不同的人、衣物和餐具,但应该“只根据被充分展示的内容做判断”。
序列长度很重要,因为每一次交互都会引入新的失败概率。Sunday 的清理演示涵盖移动操作、倾倒厨余、装载餐具和操作洗碗机,还包括单手处理两只透明酒杯时对力度的敏感控制。Chi 表示,团队实验过程中“打碎了很多很多玻璃”。
为测试泛化能力,团队预订了约6套 Airbnb 房源,并进行零样本任务:将餐具收进收纳筒、把盘子装入洗碗机。机器人没有接受任何针对特定家庭的示范,却能处理反光银器、毫米级精度的抓取,甚至透明桌面;团队将这种表现归因于数据集中超过500人带来的多样性覆盖。
制作浓缩咖啡和叠袜子都在测试细粒度力度控制。传统遥操作让操作员的手实际上处于“麻木”状态,可以施加很大的力而没有感知;使用手套则能自然感受到接触。叠袜子还会形成一个力闭环:僵硬的抓握可以施加事实上不受限的力量,但外观上没有变化,因此手套提供的自然力度反馈对灵巧操作很有价值。
Nobody wants to do their dishes. Nobody wants to do their laundry. People would love to spend more time with their family and loved ones. So what we believe is that if the robot is cheap, safe, and capable, everyone will want our robot. We see a future where we have more than 1 billion of these robots in people's homes within a decade.
Thanks, Memo.
Today we're here with Tony Zhao and Cheng Chi, co-founders of Sunday and makers of Memo, the first general home robot. We'll talk about AI and robotics, data collection, building a full-stack robotics company, and a world beyond toil. Welcome, Cheng. Tony, thanks for being here.
Thanks for having us.
First, I want to ask: where are we here? Classical robotics has not been an area of great optimism over time, or of massive velocity of work, and now people are talking about a foundation model for robotics or a ChatGPT moment. Can you contextualize the state of AI robotics and why we should be excited?
I would say we're kind of in between the GPT moment and the ChatGPT moment. In the context of LLMs, what it means is that it seems like we have a recipe that can be scaled, but we haven't scaled up enough to have a great consumer product out of it. That's what I mean: GPT, which is a technology, and ChatGPT, which is a product.
We're seeing that across academia, there's consensus around what the method is for manipulation, but everybody's talking about scaling up. We know there's a sign of life in the algorithms people are picking, but people don't know what will happen if we have more data—like what happened with GPT-2 and GPT-3. We see a clear trend that there's no reason to believe that robotics doesn't follow the trajectory of other AI fields, and that scaling up is going to improve performance.
Maybe if you took a step back, what was the process for deploying a robot into the world 10 years ago? What was the set of generalizable AI algorithms, and why was it so slow as a field?
Previously, classical robotics had this sense-plan-act modular approach, where a human designed the interface between each of the modules. Those interfaces needed to be designed for each specific task and each specific environment. In academia, that means for every task, there's a paper. You design a task, design an environment, design the interfaces, and then produce engineering work for that specific task.
But once you move on to the next task, you throw away all your code and all your work and start over again. That's also what happened in industry. For each application, people built a very specific software and hardware system around it, but it wasn't really generalizable. It just felt like we were running in loops: we built one system, and then we built the next one, but there was no synergy between them. As a result, progress was somewhat slow.
I feel like that's a good segue into some of the amazing research work that you guys have contributed over the last 5 years to the field. Should we start with Diffusion Policy? What was the impact of that?
Diffusion Policy is a specific algorithm for a paradigm called imitation learning. That's really the most intuitive way of using machine learning for robotics. You collect paired action-and-observation data of what the robot should do, use that to train a model with supervised learning, and then the robot does the same thing.
The problem is that, in the field, it's known to be very finicky. When I talked to researchers after I started in the field, the researchers themselves—the specific researcher—needed to collect the data so that there was exactly one way to do everything. Otherwise, either the model training would diverge or the robot would behave in some weird way.
A diffusion model really allows us to capture multiple modes of behavior for the same observation while preserving training stability. That really unlocked more scalable training and more scalable data collection.
So it doesn't have to be you personally wearing a teleoperation headset in order to make a robot learn?
Yep. We can have multiple people, sometimes even untrained people, collecting data, and the result will still be great.
Where do ALOHA and ACT play into this?
These 2 papers are actually super close to each other. They're 1 or 2 months apart. That's actually how Cheng and I know each other: we were looking at each other's papers, and we met on Twitter, I think, when Cheng was back in Colombia, before ALOHA.
The typical way people collect data is with a teleoperation setup with a VR headset. It turns out to be very unintuitive, and it's hard to collect data that is actually dexterous. ALOHA is a very simple and reproducible setup, so it's very intuitive.
Sorry—in terms of most people who haven't worn a teleoperation setup, is it the lag? How should I compare it to playing a video game or something?
ALOHA makes it feel more like playing a video game. Normally, it feels disconnected: you're moving in free air, and the robot is moving with some delay. But ALOHA reduces that delay by a lot, and that contributes to the smoothness and how fast humans can react.
Once we get that really dexterous data, what it allows us to do is investigate algorithms that are actually solving difficult problems. In this case, it's through introducing transformers in robotics. There was a long period of time when I think robotics was stuck with 3-layer MLPs and ConvNets, and as you made them deeper, they worked worse. But once you have very strong and dexterous datasets, you can just throw a transformer at them, and it works quite well.
Actually, just in terms of the progress of the industry over time, transformers didn't make sense without a certain level of data-collection capability.
And also an entire system around it—for example, action chunking, which is predicting a trajectory as opposed to predicting single samples of actions. All these things combined to make dexterous bimanual tasks more scalable.
Why is chunking important here? If I think about the analogy to LLMs and text-sequence prediction—
I think it just throws your model off if you're trying to force it to react every millisecond. That's not how humans act. We perceive, and we can actually move quite a bit without looking at things again. That turns out to make the motion a lot more consistent and overall performance a lot better.
And you discovered that transformers architecturally did apply to robotics. Cheng, you felt then that data collection was still a problem. So enter UMI.
After ALOHA and Diffusion Policy, I was super excited about imitation learning, but at the time, both of us were still doing teleoperation, and that just felt super limiting. The problem is that in a teleoperation setup, it takes a PhD student a couple of hours to set it up in a lab. That pretty much restricts data collection to a lab.
But in order for the robot to actually work as a product, it needs to work in the wild, in unseen environments. That requires data to also be collected in the wild. At the time, I was thinking: Is there a way we can collect robotic data without actually using a robot? That forced me to think: What's the most essential part of robotics data?
After Diffusion Policy and ACT, the paradigm is simple: you just need paired observation and action data. In our case, the observation is a video clip; the action is the movement of your hand, plus how your fingers move. I realized that you can get all this information from a GoPro. You can track the movement of the GoPro in space, and you can track the motion of the gripper and the fingers through images as well.
That's why I built this UMI gripper, which was 3D-printed at the time. The project had 3 students, and we just took the grippers everywhere. Every time we went to a restaurant, before the waiter came in, we collected some data. Very quickly, we got 1,500 video clips of this espresso-cup-serving task.
That turned out to be one of the biggest datasets in robotics, simply by 3 people. That's where the power really shines. With that amount of data, we were able to train the first end-to-end model that could generalize to unseen environments. We could push the robot around Stanford's campus—Tony was there as well—and anywhere, the robot could serve you a drink.
Yeah, I think that was the moment when I was like, “Hey, maybe we should start a company. This is actually working so well.”
I remember just following Cheng, and at times it didn't work well.
Yes.
I think the only exception I saw was when it was under direct sunlight, right? I think the reason was that, over that whole 2- or 3-week period of data collection—
Those 2 weeks were all rainy, so there was no sunlight data. It fails. That also demonstrates the importance of distribution matching. In order for a robot to work in a sunny environment, it must have seen sunny environments during training.
Yeah. This is really interesting because I remember when I first met you guys, you spent, I don't know, $200,000 across all of your academic research, and yet the scale of data collection, as translated to model capability, is leading. Right? So it's very interesting that we look at where we are—maybe going back to Tony's point of scaling and massive capital deployment—but that entire paradigm actually wasn't relevant before people realized you should train on all of the internet data, and we just don't have that in robotics. So the entire field is just blocked on having any scale of data that's relevant.
Yeah. I think these days there are so many debates about what is even the right way to scale. There are world models, there are simulations, there is teleoperation, and there are all these new ideas. I think this is the area that we really want to innovate in, that we want to differentiate in, that we want to find out something that is both high-quality and scalable.
And then you guys decide to start a company pushing this cart around Stanford. Tell me about that decision, and congratulations on the launch and the direction and team you've built.
Yeah, it's a very interesting journey. I remember in the beginning it was actually the 2 of us, in Cheng's apartment, on his desk. We clamped a robot there and tried to do some tasks, and it soon became, I think, an 8-person team towards the end of 2024, and now we're at around 30 to 40 people.
We're not the best at everything, right? But starting a company allows us to find people who we really love working with and then bring all the expertise together—from mechanical engineering and supply chain to software engineering and controls—to build a system together that is not a demo but a real product.
You've built this amazing team. What are people actually signing up for? What's the mission of Sunday?
Yes, it is to put a home robot in everyone's home. I think there are a lot of AI systems trying to make you more efficient during work. But there is not enough AI that actually helps you with all these mundane things that are not creative, that really have nothing to do with what's making us intrinsically human.
What's ideal for people to spend more time on is actually their hobbies and passions, as opposed to spending more time doing chores.
So if you guys are going from these amazing research breakthroughs to actually shipping a home robot, that's a product. You have to talk about cost and capability and robustness. What's the design philosophy?
As these AI models become more capable and as hardware costs continue to go down, home robots—or all kinds of robots—will be everywhere. So if we start from the most surface level, which is the design of the robot, when we design it, we think about what a robot should look like if it is ubiquitous. You need to see it every single day. What should it look like?
What we end up with is that we really think the robot should have a face. It should have a cute face, and it should be very friendly. So instead of a Terminator doing your dishes, we want the robot to feel like it's out of a cartoon movie.
And then a huge decision is how many arms should the robot have? Should it have 4 arms? Should it have 1 arm? Should it have legs? Should it have 5 fingers, 2 fingers, or 3 fingers? It's a huge space.
Why isn't the obvious answer that it should just be a full human?
I think the core motivation for us is: how can we build a useful robot as soon as possible? So whenever we see something that we can accelerate with simplification, we'll go simplify that.
One example of that is the hand that we designed, which has 3 fingers. We combine the 3 fingers that we have together. The reasoning there is just that most of the time when we use those fingers, we use them together, whether it's grasping a handle or opening the dishwasher.
So it really doesn't make sense to multiply the cost by 3× to separate it into 3 when we can do 1 with most of the benefits. So this is how we think about the whole robot as well. It's with the constraint that we are building a general-purpose robot that can eventually do all your chores, and we'll simplify everything we possibly can so that the robot can be as low-cost and as easy to repair as possible.
Yeah, I just want to add a little bit more to the actuation and mechanical design. Traditionally, most robots are designed for industrial use cases, and the robots are very fast, very stiff, and very precise. The reason is that all the industrial robots are blind, so they're blindly following a trajectory that's programmed by someone.
It's not reacting to perception.
Correct. But because of the breakthroughs we had in AI, now the robot has eyes. So it can actually correct its own mechanical and hardware inaccuracies. That kind of opened up a new, different space of design.
Intuitively, it should be like, I can't tell you exactly what the distance is here on a millimeter scale, but I'm going to get to the cup because I can stop.
Yeah, exactly. So that allows us to use these low-cost actuators that are cheap and compliant, but imprecise. But because of the AI algorithms and systems we build, it allows us to build a robot that's mechanically inherently safe and compliant while simultaneously being able to achieve the sufficient accuracy we need for the home tasks.
Where are we in that timeline? You said we're between GPT and ChatGPT. And so, when do consumers get ChatGPT, and when will you guys ship something?
Yeah. It's actually a really exciting time because we have so many prototypes internally. What we will do next year, 2026, is actually start doing beta programs. We'll have these robots—all kinds of different ones—in people's homes and see how they react to them.
That will be when we learn the most about what people like. Do people want to talk to a robot? Do people want to have the robot maybe teach their kids some new knowledge about the world? This will inform us what the eventual product should look like.
Internally, we just have an extremely high standard for what the minimal consumer product we want to ship is. It needs to be extremely safe. It needs to be extremely capable and low-cost.
Do you feel like you know something now that you didn't when you started the company?
Absolutely. At the beginning, I would describe it as seeing light at the end of the tunnel. There are 2 axes: there's dexterity, and there's generalization. When we add more data, things work better. What is this company about? It's the cross product of these 2: how can we scale and have both dexterity and generalization?
This is something we're able to show in our generalization demo, which is that we can pick up these very precise forks—actual metallic forks—on ceramic plates with very high success rates. Honestly, this is not something that we thought would work so easily just by having so much more data.
Yeah. I actually just want to expand a little bit. The process was long and painful. There were so many issues; scaling up a robotic system is very, very hard. There are mechanical issues, reliability issues, and data-quality issues that come out of it.
In the beginning, I actually thought it was going to be much easier than this. Compared to teleoperation, it's much harder to get this system scaled up, but once it's scaled up, it's very powerful and very repeatable.
So it is both harder than you thought it would be to get to here, and you are further than you thought you would be.
Yes. And I remember in the beginning we were having this funny conversation. We were like, if we build this, someone can just take our glove and build the same thing. What moat do we have? Are we worried about that?
I think in the beginning, actually, we were a little bit worried because we thought they could probably just replicate it. But as we go along the path, it turns out things are so much harder than we thought they were.
Yeah. And when you say scaling up the robotic system, you mean the data-collection-to-training pipeline and the hardware itself?
Yeah. For this to work at all, you need the data-collection system. You need the robotics and control systems to be able to deliver the hand to where we want it to go, and you also need the data-filtering pipeline, data-cleaning pipeline, and training pipeline. All these things need to be iterated together.
We've actually gone through several loops of these. It's kind of hard to imagine, without having a full-stack team in-house, how this can even be done. The glove we're using right now is—we call it V5.
For V0 to V5, each version has around 20 iterations.
Okay. So 100.
Yes. Also, when you make these at scale, right now we have more than 500 people using these gloves in the wild. All the things that could go wrong will go wrong.
They did.
Yes. For example, how things are assembled. If you don't specify exactly how it should be done, people will assemble it in creative ways. And the creativity doesn't help us here because we really want the data-collection device to be extremely precise.
You obviously can’t know everything that’s happening in every company, in academia, and in industry. But from what you know, how would you compare the scale of the training data you have today relative to the industry?
At this point, we have almost 10 million trajectories being collected in the wild. Those trajectories aren’t just, “Oh, pick up a cup.” These are long trajectories involving walking, navigation, and long-horizon tasks.
Elad Gil
Tony, as you mentioned, it’s an open question what the right way to scale data up is. There are strong theories around teleoperation, pure RL, video, and world models. How have you thought about all of these?
From our perspective, it was somewhat surprising. In the beginning, we worried that data from the glove would have higher quantity but lower quality compared to teleoperation, because with teleoperation you’re using exactly the same hardware and software stack between training and testing, so it’s perfectly distribution-matched.
What we realized is that the glove form factor encourages people to make more dexterous and natural movements. Those movements actually result in more intelligent behavior on the modeling side. In terms of data quality, we don’t really see a gap between teleoperation and glove data after we did the 20 engineering iterations.
Yeah. Apparently, there is a mismatch: in the camera frame, there’s a human instead of the robot. There are a lot of things we need to do to convert human data one-to-one into robot data, as if it were robot data, and make sure the model can’t tell the difference. That relies on the full-cycle iteration between hardware and software.
Elad Gil
What about RL? We see a lot of promise for RL in locomotion, and we think that will continue to be true for locomotion.
From what we see, RL as a method is very powerful, but it’s much less sample-efficient compared to imitation learning. We see it working well in environments where it’s easy to simulate. For locomotion, you only need to worry about rigid-body dynamics and rigid-body contact between the robot and the ground. Because you engineered the robot, you know everything.
For manipulation, it’s hard for us to imagine having the same amount of diversity and the same distribution of real objects, in terms of matching both appearance and physical properties. We think that’s going to be challenging compared to glove data collection and teleoperation.
I think it’s really about which method can get us there faster. There might be different methods that will eventually get there. For example, simulation and world models. It’s almost a tautology to say that if I have a perfect world simulator, anything can be done there. As long as you can do it in the real world, you can do it in a simulation. You could cure cancer in a simulator.
But what it turns out for robotics is that some things are harder than others, and it really depends on the problem itself. In the case of locomotion, as I mentioned, all we need to model in a simulator are point contacts with somewhat flat ground, like feet. But the behavior we want out of it is actually very difficult to model.
It’s all these reactive behaviors: when you feel like your leg is hitting something, you should retract and step again. These are very hard to describe or learn from demonstrations directly. In the case of manipulation, I think the difficulty is flipped: it’s a lot easier to capture the behavior itself, and it’s a lot harder to simulate the world.
For example, if you were to grasp a transparent cup with some orange juice in it, it’s ridiculously hard to simulate how your hand deforms around the cup, how all those ripples and the color of the juice affect the rendering, and what the policy ends up seeing. Simulating that is very expensive and difficult.
But all we need to learn is to get your hand in front of the cup and then close it with the appropriate amount of force. That’s actually very easy to learn. That’s why we see so many successes with imitation learning in the case of robotic manipulation: the behavior itself is not as hard as simulating the world, and that’s why we see faster progress there.
Is there anything that you’ve changed your point of view on in data over the last year?
One thing I wouldn’t say changed is that data quality really matters. I always knew data quality mattered, but once you scale up, it really matters. The diversity of behavior that you experience in the wild is very hard to control, and hardware failures are hard to control as well. You need to constantly monitor them and spend a huge amount of engineering effort just to make sure that the data is clean.
Also, building all those automatic processes is important. We have our own way of calibrating the glove before we ship it out, and we have this whole software system to catch when something is broken on the glove so we can detect it automatically. The importance of data quality translates into all these repeatable processes, so we don’t need a human staring at the data to know that something is wrong.
When you described the beta for next year, a lot of it sounded like, “We just want to understand behavior—how people actually want to use it—and we can make some design decisions for the actual product.” What technical challenges do you still see?
To me, there are 2 kinds. Number 1 is really figuring out the training recipe at scale. As a field, we’ve just entered the realm of scaling, and we’ve just gotten the amount of data that we need. I think now is the perfect time to start doing research and actually figure out what exact training recipe we need to get robust behaviors. We’re in a unique position because of the amount of data and the entire pipeline we’ve built around data.
The second point is that hardware is hard. We’re still pushing the performance envelope of hardware. It’s not really clear what is needed for the hardware to be reliable, because whenever the mechanical team builds hardware, the learning team will try harder to push it against the boundary, and then it will break at some point.
What’s interesting about this company is that everybody’s under the same roof. Immediately after something breaks, it goes straight back into mechanical design, and then we have another iteration—for the hand parts, for example—very quickly. Hardware is hard, but it is important. It’s the hard but right thing to do, and we as a field shouldn’t avoid doing the hard things just because they’re hard.
I want to echo Cheng’s point about research. When there is data scarcity, it’s really easy to come up with cute, fancy research ideas that don’t end up scaling very well. This is why, when we built the company, we focused on the infrastructure, a scalable data pipeline, and operations before we started to really dive into research, which we only started doing 3 months ago. We want to avoid doing research that doesn’t scale and focus on things that contribute to the final product.
The second point is that robotics is intrinsically a system. Right now, there isn’t an existing general-purpose home robot out there, and we don’t really know what the interface between different systems is or what is even good. If you’re working with a partner, it’s actually really hard for them to understand your standard of good, because your standard of good is changing all the time.
This is why we’re building everything in-house, taking a more full-stack approach. We build our own data collection device, which is co-designed with the robot. We build our own operations team to figure out how to get the most high-quality data out as efficiently as possible, and of course our own AI training team to make the best use of that data.
These things are really not easy. They make a company a lot harder to build, because you suddenly need so many teams and they need to work together. But we believe it’s the right thing to do.
Okay. I’m going to ask you a few questions that are uncomfortable guesses now. When will people be able to buy robots commercially for the home?
This is something we’re really excited about, because we have so many prototype robots in our office and we really want to get them out there. The next step of our plan is to have a beta program in 2026. For people who sign up and are selected, they will have a real robot in their home, and it will start doing chores for them.
It’s going to be a really interesting learning experience for us, because we’ll see how humans interact with the robots and what kinds of things people really want the robot to do. I think this will happen before we actually ship it to the masses, because we have an incredibly high standard for what we’re willing to ship from a consumer-experience standpoint.
We want the robot to be highly reliable. We want it to be capable. We want it to be cheap. I think it really depends on the results of the beta program, which will decide when is a good time to ship it. Is it 2027? Is it 2028? All of those are possible.
But it's not a decade away.
No, it's definitely not that far away.
How much do you think it could cost?
Right now, the prototype robots we have in-house range from around $6,000 to something like $20,000. What's interesting is that the big difference here isn't that we found a better actuator. We're using the same actuators, which are very low-cost. The real difference is the cladding of the robot. When you're trying to make them at low scale, it's just really expensive. The claddings cost a few thousand dollars to make.
But these are the types of things that, as we scale up, become dirt cheap. Instead of doing CNC and hand-painting them, we'll use injection molding. As we get to a scale of a few thousand units, we can drastically reduce the material cost, likely to under $10,000. What that implies is that when we sell the robots, the price will be somewhere around that.
Okay. Fast-forward 2 or 3 years. If you look 5 years and beyond, and home robots are ubiquitous, what does life look like? How does it change for your average person?
This is a different answer for everyone. For me, I really hate dishes. In my sink, there are always 4 or 5 dishes that are somewhat dirty, and it kind of stinks. After a long day of work, it really doesn't feel good to come home and see a home like that. I think the world we'll live in is—
It's going to be cleaner.
It seems like it will be cleaner. I was just thinking about it as the marginal cost of labor in homes goes to zero.
The last thing I want to make sure we do is talk about demos. There are a lot of robotics launch videos today. It's been years since you saw an Optimus serving drinks at a bar. Why aren't those robots available, and what is actually hard?
I think the way I would put it is: make zero assumptions. No Priors.
Okay.
If you see a robot handing one drink to one person, first ask whether that's autonomous or teleoperated. That's the first thing. We should look at the tweet and see what they say about it. Then ask whether it shows the robot giving another, slightly different-colored cup to the same person. If they didn't show it, it means that the robot may literally only be able to pick up that single cup and give it to that same person.
When we look at demos, we tend to put our human instincts into them. If you can hand a cup to that person, it must be able to hand a different cup to another person. Maybe it can also do my dishes. Maybe it can do my laundry. There are a lot of visual inferences we can make, which is what's great about robotics: there are a lot of possibilities. But when we look at demos, we should only index on the things that are demonstrated strongly, and assume that's likely the full scope of the task.
Another aspect is that, at least as a researcher, I appreciate the number of interactions that happen in a demo. Usually, the more interactions you have, the more opportunities there are for failure. The longer the sequence is, the harder it actually is. That's something we really emphasize here, and it's somewhat uniquely easy for us because the glove-based method of data collection is so intuitive for people. It's really about generalization and reliability.
Can you explain the demos that you guys are showing?
Yeah, of course. We're showing 3 categories of demos. The first one, as you saw, is a messy table. The robot cleans up the whole table, dumps the food into the food-waste bin, loads the dishes into the dishwasher, and then operates the dishwasher.
What makes this demo really hard is that it's a mix of very fine-grained manipulation with these super-long-horizon, full-range tasks. You need to go up, and you also need to go down a lot.
It's mobile manipulation.
Exactly. The reason we can show this is that our data-collection process is so nimble and easy that we can make long-horizon dexterous demos possible. It's also about the forces involved. You might have seen that we're trying to pick up 2 wine glasses with 1 hand.
I struggle with this, but yeah.
It's actually really hard. Because they're transparent objects, we also need to load them very precisely into the dishwasher. A lot of it is about how much force you apply. If you're trying to grasp 2 glasses in 1 hand and you squeeze a little bit harder, you're going to break one of them. When you load them into a dishwasher, if you're pushing in the wrong direction and they hit something, they're going to shatter.
We shattered a ton of glasses when we were experimenting with this. These tasks are really high-stakes. It's not just about recovering from mistakes; it's about not making those mistakes in the first place. That's generally the case in a lot of home tasks: you're just not allowed to make any mistakes.
Then we get into the generalization demos. We book around 6 Airbnbs and get there zero-shot to see if the robot can do part of the task. The 2 tasks we use are, first, going around the table and collecting all the utensils into the caddy. The other is grabbing a plate and loading it into the dishwasher.
What makes these demos very interesting is that we don't need any data when we enter that home. It's pure generalization. This is as close to a real product as you can get, because when someone buys our home robot, we really don't want them to have to collect a huge set of data themselves just to unbox it.
In addition to generalization, those 2 tasks are also really precise. We're using the exact silverware in the home, and you need a few millimeters of precision to grasp it properly. Those forces are also hard to perceive because the objects are reflective, and the lights look weird on them. We have a transparent table in one home. I think the table looks like nothing, and the robot still reacts very well to it.
Again, the reason we can do it is that we have more than 500 people in our data set, and we've seen so many glass tables. The robot is able to handle it.
The last set of tasks we did is about pushing what's possible in terms of dexterity. The 2 tasks we chose were operating an espresso machine and folding socks.
Mhm.
What makes these hard is that they require very fine-grained force, which is hard to get with teleoperation. These days, there isn't a good teleoperation system that lets you feel how much force the robot is feeling. Basically, when you're teleoperating, your hand is numb, and sometimes you're applying a huge amount of force to the robot without knowing it. That can result in very low-quality data, with the robot also doing things in that aggressive way that we really want to avoid.
The sock is a very good example. When you're trying to fold it, your 2 fingers can touch, and that forms what we call a force closure. You have a closed loop for the force, and if your grip is stiff, you can apply an infinite amount of force to it and it doesn't look like anything. But because we're using the glove to collect the data, the human collecting it can naturally feel that. It's very intuitive.
I think we're the first in the industry to do sock folding and use end-to-end learning to operate an espresso machine.
One of the things that you will also need to scale as you scale up the company is the team. What are you hiring for? What are you looking for?
One thing I'm really looking for is full-stack roboticists and people who aspire to become full-stack roboticists. What you learn in this company is that robotics is such a multidisciplinary field. You need to know a little mechanical engineering, a little electrical engineering, a little code, and a little bit of data to actually fully optimize the system.
We have a couple of examples of training full-stack software engineers to become roboticists and training ML engineers to become roboticists. If you want to learn about robotics and learn the whole thing, rather than just be boxed into your small little cubicle, let us know.
You told me that you didn't write code until you got to college or something.
I was super enthusiastic about robotics, but before that I was mostly doing mechanical and electrical design. Then I realized that the bottleneck was actually how the robot would move, and that there was something called programming. The more I got into it, the deeper it got.
Toward the end of college, I realized there was a thing called machine learning, and that you could train models. The field just goes on and on. I think it's very natural for me to gradually expand my skill set because I'm always looking to build a robot.
Well, I hope you discover the next field, because you're no longer doing dishes, too.
It's a very fun place to work. Whatever you can imagine about robotics, consumer products, and machine learning, you can find it here, because we're fundamentally such a full-stack company.
We're not just about the software. We're not just about the hardware, but we're about the whole experience, the whole product, and making sure that product is general and scalable in the future.
Awesome. Congratulations. It's really exciting.