[BidClub_]
The a16z Show · · 42 分钟

Fei-Fei Li 正在解决机器人领域最难的问题|a16z 对话 World Labs

Martin CasadoFei-Fei LiYunzhu Li

YouTube
TL;DR
  • World Labs 正将空间智能路线延伸至机器人领域,通过将最初的 Marble 客户 SpAItial 纳入公司,而非转型为机器人制造商。双方合并后的技术栈,将 World Labs 的生成建模与3D重建能力,叠加 SpAItial 在机器人、仿真和硬件方面的专长。Yunzhu 的核心目标很直接:“我希望机器人能真正工作”(“I want the robot to work.”)。
  • 核心瓶颈是缺乏可规模化的机器人数据和评测体系。与语言模型不同,机器人无法从互联网获取海量数据;物理测试缓慢、昂贵且危险,因为原子必须在空间中移动。SpAItial 的真实世界—仿真—真实世界流程,旨在用对齐的数字环境替代其中相当一部分数据和评测工作。
  • World Labs 认为,一致的世界模型能够提供纯视频路径仍难以保证的能力。一个有用的环境必须在空间、时间、视角和交互上保持一致:如果机器人推了一个物体,而它“就这么神奇地消失了”,这样的预测就无法提供良好的学习信号。Marble 可从文本或图像生成几何一致的世界,并以 Gaussian splats 或 mesh 表示。
  • 仿真与真实世界数据是飞轮中互补的两个阶段,而非互相竞争的两种教条。早期系统可能更依赖物理、几何和随机化;随着客户数据和机器人数据积累,建模会逐步转向学习型动力学。Fei-Fei Li 的关键区分在于,仿真可以对尚未发生、不可能发生或现实数据不足的事件进行“反事实推理”(“counterfactual reasoning”)。
  • 评测可能是平台近期最锋利的切入口,因为迭代速度决定机器人开发速度。SpAItial 希望无需在现实中重做每一轮试验,就能区分90%和92%的 checkpoint,或测量95%与99.9%的可靠性。其卖点是“可规模化、安全且快得多的评测”,且结果与真实世界表现保持一致。
  • 商业部署应先从结构化工厂推进到半结构化仓库、酒店和餐厅,再进入家庭。可靠性来自足够广泛的场景覆盖,而受控环境更容易实现这种覆盖;完全非结构化的家庭仍是“终极挑战”。Martin 认为,这一过程更有利于专用具身形态,而不是过早押注通用人形机器人;SpAItial 则不绑定模型或具身形态。
  • 机器人达到人类水平的能效,并不是5年内必然发生的事。Yunzhu 预计需要“非常长的时间”,因为可靠机器人是一个整合系统,覆盖硬件、软件、机器人“大脑”、动力学,以及指尖摩擦力等细节;Martin 指出,即便语言模型也尚未达到约30瓦的人脑水平。更近的2年目标是可量化的:在少数垂直领域验证价值,并将这些客户打造为“灯塔案例”。
摘要 · 为研究而整理的核心内容

1. 让空间智能真正落地

  • Fei-Fei 将 World Labs 定义为一家成立2年的前沿模型实验室,目标是打造“能够生成、理解、推理并与空间互动的 AI”,空间既可以是物理空间,也可以是虚拟空间。大型世界模型是手段,空间智能才是目标。

  • 在空间中采取行动并不只对应机器人,还包括 VFX、游戏和设计。她更大的设想是一个“多元宇宙”,让开发者能够在不同空间中创建并开展活动;而物理行动仍是 AI 最具影响力的落地方向之一。

  • 这次结合始于产品需求牵引:World Labs 去年冬天发布 Marble 后,大约在11月或12月,SpAItial 注册成为客户,而 Fei-Fei 当时并不知道这是 Yunzhu 创办的公司。Fei-Fei 还表示,World Labs 当时已经收到早期机器人公司的主动咨询,以及来自实际下游应用场景的需求,只是还无法服务这些客户。

  • Marble 可接收文本、1张图或多张图,并生成以 Gaussian splats 或 mesh 等3D几何形式呈现的几何一致世界。SpAItial 提供全栈机器人技术和仿真能力;World Labs 则提供生成建模、计算机视觉和稀疏3D重建能力,让 SpAItial 更为繁重的真实世界采集流程变得更高效。

2. 机器人基础模型必须理解动作及其后果

  • SpAItial 的真实世界—仿真—真实世界(real-to-sim-to-real)流程,会重建外观、几何结构和动力学——也就是施加动作后环境如何变化。理想状态是,数字环境中观察到的行为在物理世界中也大概率发生,从而让仿真数据同时服务于训练和评测。

  • 机器人基础模型可能会是一个覆盖视频帧、文本、图像、深度信息和动作的全模态模型。Yunzhu 强调的区别在于架构:动作作为输入时,模型是预测下一个世界状态的前向模拟器;动作作为输出时,模型是选择如何朝目标移动的策略。

  • Martin 将其与主流视频模型路径作对比。Yunzhu 的答案是,系统必须在“空间、时间、不同视角以及不同类型交互”上保持一致;最终形态可能介于物理建模与学习之间,并随着执行后的策略不断带回新的真实世界数据而持续改进。

3. 仿真提供真实数据无法给出的反事实

  • Martin 保留了核心质疑,引用 Sergey Levine 的观点称,仿真最终会与现实产生偏差,真实世界数据采集仍然关键。Yunzhu 则认为双方立场并不冲突:先依靠更强的物理结构,再吸收真实世界结果,随着飞轮积累数据,逐步转向学习型建模。

  • Fei-Fei 的哲学论据是,人类一直在进行模拟,因为现实不可能提供每一种相关经历。Martin 以世界杯比赛中的战术规划作类比;Fei-Fei 将这种能力称为“反事实推理”(“counterfactual reasoning”)。她引用 Waymo 所称的数十亿小时仿真,并补充说,汽车已经是最简单的机器人之一。

  • 仿真逼真度不等于复现每一片雪花或每一丛灌木。四足和双足机器人可以穿越雪地和植被,无需精确模拟每个元素;模型需要抓住“问题的核心结构”,再引入足够的随机变化,确保能力可靠迁移。

  • 仿真通过系统性覆盖光照、摩擦、几何结构、物体和物理参数,提升可靠性。它也能提高效率:远程操控采集数据可能比人类亲自完成任务还慢,而客户可能要求机器人以高于人类的速度工作;单纯加速策略并不奏效,因为“重力不会改变”(“gravity doesn’t change”)。

4. 更快的评测是眼下的平台切入口

  • 机器人评测关注的是,一个模型检查点执行任务的可靠性究竟是95%还是99.9%,以及需要多少实际耗时才能区分90%和92%。物理试验让迭代速度比语言模型开发慢“多个数量级”,同时还带来危险和成本。

  • 当数字结果与物理结果对齐时,在仿真中表现更好的模型检查点,大概率在现实中也表现更好。训练同样获得可控性:团队可以明确覆盖了哪些状态分布,以及合理预期机器人在哪些场景中有效。

  • 这是让其他公司的机器人在这些世界中学习的基础设施,不是制造机器人的计划。它支持固定式机械臂、双臂机器人、移动操作机器人、夹爪及其他具身形态;生成数据既可用于从头训练模型,也可用于对 vision-language-action 模型和 world-action 模型进行后训练。

5. 商业化从受限环境开始

  • 机器人商业化历来是从高度结构化工厂走向半结构化仓库、餐厅和酒店,再进入非结构化家庭。Yunzhu 认可这一路径,因为可靠性取决于场景覆盖;一项涵盖约1,000项用户希望机器人完成任务的调查显示,约1/3涉及清洁。

  • Martin 认为,人形解剖结构是为在非结构化环境中普遍生存而进化,并非为任何单项任务实现最优表现。专用机体可能更好地解决更窄的商业问题,因此 SpAItial 不绑定具身形态的基础设施,可能比押注单一人形机器人更有用。

  • 当被问及机器人能否在5年内——甚至最终——达到人类的能效水平时,Yunzhu 回答需要“非常长的时间”(“a very long time”)。进展一再超出他的预期,但可靠性取决于整个系统,细节甚至包括指尖摩擦力;Fei-Fei 将应有的态度概括为“克制的乐观”(“measured optimism”)。

  • 整合会循序推进,团队“不会急着把双方搅成一大碗沙拉”(“not rushing to blend” into a “full salad bowl”),但 SpAItial 已在内部使用 Marble,双方也已开始围绕仿真和动作条件模型展开合作。2年后的成功标准,是在几个重要垂直领域验证自动化价值,打造灯塔客户,进而推动业务扩张。目前客户已接近部署阶段,瞄准的是具体实用任务,每项任务对应几十到几百个待自动化的候选场景。

Fei-Fei Li

We’re building the next frontier of AI, which is what we call spatial intelligence.

Yunzhu Li

At SpAItial, we are developing what we call a real-to-sim-to-real pipeline. We can replace all the data and all the evaluation we need in the real environment by using data that we can generate at scale in our digital world.

Fei-Fei Li

Think about human intelligence. We do a lot of simulation in our head. Why? There’s a very important role simulation plays that real-world data doesn’t play, which is counterfactual reasoning.

Yunzhu Li

What we are building is a consistent world, consistent over space, over time, over different viewpoints, and over different types of interactions. My north star is: I want the robot to work.

Fei-Fei Li

The world we live in can be a multiverse, and we can create technology to allow people—builders, developers—to act within different spaces.

Martin Casado

Do you believe we’ll ever be able to build robots that have the power efficiency of a human being? How far away are we from this? Is this 5 years away, or is this never?

All right. Well, it’s great to have you both here. Fei, for the listeners who may not have the background, maybe you can give an overview of what World Labs does.

Fei-Fei Li

World Labs is a 2-year-old startup. I think we should just recognize that it’s a frontier model lab. We’re building the next frontier of AI, which is what we call spatial intelligence.

Spatial intelligence is about creating AI that has the ability to generate, understand, reason with, and interact with spaces, whether they’re physical or virtual. Of course, a means to an end toward spatial intelligence is building large world models. That’s what World Labs is mostly focused on.

Martin Casado

You’ve been saying this since the very beginning: the machine’s ability to perceive and reason about spaces and act on spaces. But I always had the assumption that acting on spaces was some long-distant-future thing. Now you’re acquiring a robotics company, so maybe talk a little bit about the timeliness of this and the intentions.

Fei-Fei Li

First of all, it doesn’t just take robotics to act within spaces or to interact. Look at the creative field, whether it’s VFX, gaming, or design. There are many use cases where you can create and act within virtual spaces. World Labs’ thesis has always been that the world we live in can be a multiverse, and we can create technology to allow people—builders, developers—to act within different spaces.

Having said that, the ability to act within physical space is one of the most exciting and profoundly important capabilities of the future AI world. Robotics is very much that. World Labs has always believed that robotics is an important application, as well as a use case, of spatial intelligence and world modeling.

So, inviting the SpAItial team to join World Labs is part of our long-term vision and mission. We’ve always been committed to that.

Martin Casado

Amazing. Yunzhu, you’re the co-founder of SpAItial. Maybe provide everyone with a quick overview of your background and what SpAItial does.

Yunzhu Li

I’m Yunzhu. I’m currently a co-founder of SpAItial and also an assistant professor at Columbia University.

Martin Casado

Wow.

Yunzhu Li

My research started with my PhD at MIT and then a postdoc with Fei-Fei at Stanford University.

Martin Casado

That’s great.

Fei-Fei Li

The world is small.

Martin Casado

It is.

Yunzhu Li

Throughout my career, my goal has been very simple: trying to help robots better perceive and interact with the physical world. I’m a very practical person. I want my robot to work in real physical environments.

For SpAItial, the unique opportunity we see is that there have been a lot of bottlenecks faced by the development of general-purpose robots, especially around training and evaluations. We’re developing what we call a real-to-sim-to-real pipeline.

We want to map real environments into the digital world that has the best alignment with the real environments. By alignment, we mean that whatever happens in the digital world is also going to happen in the real environments, such that we can replace all the data and all the evaluation we need in the real environment by using data that we can generate at scale in our digital world.

That’s how everything started at SpAItial. We put together a very, very strong team around robotics, robot learning, simulation, and rendering, trying to build this real-to-sim-to-real stack to solve some of the key bottlenecks.

Martin Casado

It’s amazing that you two work together.

Fei-Fei Li

There’s a funny story here, because you would think that, because we worked together—he was my amazing postdoc—we’d been talking about this World Labs integration for a long time. It’s actually not true. They came into World Labs as a customer.

Martin Casado

Really?

Fei-Fei Li

When we released the 1st version of our generative model, called Marble, last winter, around November or December, SpAItial just signed up.

Martin Casado

No kidding—as a customer.

Yunzhu Li

Yes.

Fei-Fei Li

I didn’t even know what it was. Then I realized this was Yunzhu’s company. I called Yunzhu. I was like, “Wow, this is your company.” Then we realized there was so much synergy.

Martin Casado

Maybe Fei could just quickly describe what Marble is.

Fei-Fei Li

Marble is the codename for the base model that World Labs has been training and iterating on. The fundamental capability of Marble right now, which is publicly released, is to take a prompt—it can be an image, a few images, or text—and turn that into a geometrically consistent world that can be represented in 3D geometry, whether it’s Gaussian splats or a mesh.

Martin Casado

The SpAItial team is trying to solve this extremely difficult problem in robotics, which is the lack of data—the lack of data in training and the lack of data in evaluation. This is very, very different from language models, where data is abundant on the internet.

Fei-Fei Li

In order for robotics to work, we have to somehow unlock the power of scaling laws. But where does that come from? This is a profound problem that everybody is battling with in robotics.

Martin Casado

It’d actually be great to talk about this synergy. You have put together a very talented team, and I’d like to understand to what extent there’s overlap and to what extent this is an extension. Maybe talk a little bit about that.

Yunzhu Li

The TL;DR is that it’s very complementary, with a shared mission. I’m 1 of the 3 technical co-founders. The other 2 are Changi Jan, another Columbia professor who has been a world-class technologist in simulation, and Sunonni, a phenomenal engineering leader.

Changi Jan has a background in VFX. He worked at Weta, he worked at Tencent, and he’s been an entrepreneur. Sunonni was also in a startup that was acquired by Amazon many years ago. He worked in many different tech stacks in the computer vision field at Amazon.

Fei-Fei Li

When we started talking more seriously, I recognized that a couple of things SpAItial has, from a talent point of view, are extremely complementary to World Labs.

One is obviously incredible thought leadership and just technical prowess in robotics—from hardware and full-stack robotics to modeling. Even when Yunzhu was my postdoc at Stanford, he already had his faculty offer, so he was there for only 1 year. I wanted him for more than 1 year, but he had to go become—have a real job.

He was a full-stack researcher in robotics, from modeling to hardware. And of course, Yunzhu and his students at SpAItial were that pool of talent World Labs hadn’t had yet.

Then, on the Changi Jan side, there’s just incredible simulation capability. He’s such a senior researcher and technologist in simulation, and what World Labs is doing is very much interfacing with the world of simulation.

I think what they don’t have, obviously, is the generative model side, as well as the computer vision and 3D reconstruction side. We’re also very strong at World Labs. That’s technology SpAItial needs. So, these 2 sides come together and make it much more complete.

Martin Casado

This is an extension and a complement to get into robotics. Having been in your situation—which is deciding when to sell a company—it would be great to hear from you on how you think about joining World Labs, the fit there, and why you made the decision to do it.

Yunzhu Li

At the very beginning, we were deciding, “Okay, we want to just keep going.” But after chatting with Fei and seeing all the synergies that were happening, it just made perfect sense for the forces to join each other.

In essence, as SpAItial, what we’ve been doing is real-to-sim-to-real: reconstructing the environment. We capture the appearance of the environment, the geometry of the environments, and also the dynamics of the environment—meaning how the environment is going to change when you apply actions.

This reconstruction right now is still a little bit on the heavier side, and what World Labs has been doing involves a lot of profound capabilities around sparse reconstruction and generation. So, we see a lot of opportunities to leverage Marble and other capabilities at World Labs in order to do very efficient reconstruction and modeling of the environments.

Martin Casado

Can we expect a foundation model for robotics from World Labs?

Fei-Fei Li

World Labs is building a foundation model. As you know, Martin, we’re building a base model, and as the technology has been evolving, some of the most exciting base models are omni-models, right? They take multimodal input and have multimodal outputs.

Yunzhu Li

And what is a foundation model for robotics? It’s very likely going to involve actions.

Martin Casado

It’s very likely going to involve the output of actions in addition to the state of the world, and we’re definitely not ruling this out.

Fei-Fei Li

Yeah. Great.

Yunzhu Li

For example, a foundation model essentially needs to be a multimodal model. It has to take into account frames, text, images, depth, and different kinds of modalities, and action is a very, very important part of those modalities.

If you think about frames and actions as inputs, that is essentially a forward simulator that is going to predict how the environment is going to change when you apply a specific action. When the action is the output, this is essentially a policy model that is trying to predict, given a specific goal, what action you should take in the real environment to get closer to that goal.

These kinds of omni models can benefit a lot and provide a huge amount of value for the robotics community in trying to understand how to model environments and, at the same time, how to act in those environments. This can also act as a backbone that you can fine-tune for specific robotic applications, making sure it really lives up to the reliability and efficiency expected by clients.

Martin Casado

You know, if you don’t mind a kind of lay-investor question, I see a lot of robotics companies, and a very popular approach right now is, “We’ll use a video model.” That’s the predominant method. This is 3D and simulation, which is a very different approach, so maybe you could contrast the popular approach of using video only with what the ambition here is.

Yunzhu Li

In order to create worlds where the robot can learn, as I mentioned, you have to capture the essential structure of the problem. One of the most important requirements for those worlds is consistency. That’s where I see very strong synergies with Marble, because what we are building is a consistent world—consistent over space, over time, over different viewpoints, and over different types of interactions.

The generated world from Marble also provides an infrastructure component of that entire world that we believe is necessary for the robot to learn. Imagine if a robot pushes an object forward and the object just magically disappears, which has been a problem with many existing video-prediction models. This won’t provide a good enough signal for the robot to know what the right thing to do is.

Obviously, there has been a lot of investigation into building better and stronger video models. We see a way that some of the infrastructure we’re building can provide initial momentum for going through this data flywheel: moving from more simulation-driven models into robot-policy models, which execute in the real environment and collect new data.

That data will come back in, and the model doesn’t necessarily have to be physics-only or learning-only, but somewhere in the middle. It will be able to capture the essential structure of the problem while also scaling and becoming better and better as you accumulate more data.

Martin Casado

You know, I’ve worked with you very closely for a while, and you’ve always had this north star that has driven all of this. You’ve articulated it variously as 3D and in a number of other ways. I’m just wondering: is there also a similar philosophical north star for you, or are you more pragmatic, like I am—build the system, do the thing?

Yunzhu Li

My north star is to make robots work.

Martin Casado

Amazing. Yeah.

Yunzhu Li

In the real environment. I’m a very practical person. I want the robot to work. One interesting thing that’s actually coming from my collaborations with Fei-Fei during my postdoc is that we’re building this kind of benchmark. We actually sent out surveys asking the general public what they want their robots to do for them.

Among the 1,000 tasks we collected, 1/3 of the tasks are about cleaning. People just don’t like to do those dull and dirty tasks. Those are the scenarios where we really want to make sure we have robotic solutions to deal with them.

Fei-Fei Li

One thing I really like about SpAItial, Martin, especially continuing your question, is that there are a lot of robotics companies building models and all that. One thing I truly like about SpAItial is that Yunzhu and his co-founders have such an incredibly pragmatic approach to robotics.

They especially come from academia—well, one of them doesn’t, but the other two come from academia. Their first instinct, though, is to work with design partners and customers in real industry, whether it’s industry labs, warehouses, or electronics assembly. That is such a refreshing way of approaching robotics, and that really made me very excited to work with them.

Martin Casado

Maybe this is for you, but this is just personal curiosity. It seems to me that, for robotics, you have to be pretty exact—not perfect, but pretty close. But for the creative use cases that World Labs has done a lot of, you kind of don’t need to be, because sometimes being wrong is stylistic or intentional or whatever.

From a technical perspective, what is the challenge of reconciling these two things? Or do they never get reconciled? Will there always be two points in the design space?

Yunzhu Li

They will be reconciled in the long term, of course. Modeling the environments doesn’t have to be perfect. The model doesn’t have to be perfect in robotics.

Martin Casado

And, by the way, is there a more formal way to say that? This is pure curiosity, but what does it mean not to be perfect? It has to be pretty close.

Yunzhu Li

Let me put it this way. For the development of all different kinds of robotic applications, models have been a very important cornerstone. If you look at all the existing robotic applications—planes, drones, Roomba, quadruped robots, or even bipedal robots—models have been the way for them to actually work and be able to transfer from simulation to the real environment.

Martin Casado

I see.

Yunzhu Li

But if you look at locomotion robots, like quadruped robots and bipedal robots, they can walk on snow and they can walk on bushes. You don’t need to have a simulator that can simulate all the bushes and snow very precisely. You need to have a simulation that captures the essential structure of the problem and does a whole range of different kinds of randomizations inside the digital environment.

That is what we’re aiming for. Basically, with SpAItial and together with World Labs, we’re trying to investigate what level of fidelity we need to model the massive worlds, in addition to the robots, so we’ll be able to transfer robotic systems trained in the simulated environment—the digital worlds—back into real scenarios.

Martin Casado

As an investor, I’ve heard other researchers, like Sergey Levine, say that simulation will always eventually deviate from the physical world, and real-world data collection is absolutely critical. Maybe talk a little bit about the viability of this approach, where simulation is a cornerstone, as opposed to some other approach.

Yunzhu Li

They don’t contradict each other. If you think about simulation, simulation is essentially trying to predict how the environment is going to change when you apply actions. This is essentially a model of the world. It doesn’t necessarily have to be pure physics; it can be a combination of physics and learning.

We are collecting real-world data, and we will be using that real-world data. It’s just that it comes in at different stages of the data flywheel. Maybe at the very beginning we place a stronger emphasis on physics to make sure we have the right consistency and the right structure for us to learn the world and train the robot policies.

But as we accumulate more and more data, both through data collection and through collaboration with our clients, we’ll have data that moves us toward more learning-based modeling of the environments. This kind of transition and data flow is really an enabling factor for getting the best of both physics, geometry, and consistency, as well as all the power and magic from data and compute.

Fei-Fei Li

I want to add to this and be slightly philosophical. There isn’t a binary choice between simulation and no simulation. All of this comes together to make robotics work.

Think about human intelligence. We do a lot of simulation in our heads. There’s a very important role simulation plays that real-world data doesn’t, which is counterfactual reasoning. You play out events that haven’t happened, or cannot happen, or for which you don’t have enough data to make them happen in the real world. While you play them out, you learn how to act in those situations. Humans do this all the time.

Martin Casado

I know you were at World Cups.

Fei-Fei Li

I was at the World Cup.

Martin Casado

Congratulations to Spain for winning. I’m sure that, in the planning of every game, there is simulation, whether it’s digital, on a whiteboard, or whatever. The role simulation plays is counterfactual reasoning, and that’s really important in robotics because we simply cannot possibly have enough real-world data for that.

Here’s a real-life example: the self-driving-car industry. Waymo has officially said they use billions of hours of simulation.

Yunzhu Li

And actually, Waymo is more simulation-heavy than just real-world-data-heavy.

Fei-Fei Li

So these are real examples, and, as you know, autonomous cars are the simplest kind of robots.

Martin Casado

Yeah.

Yunzhu Li

Yeah. So clearly, simulation plays a huge role in robotic learning. I also want to add to that. More specifically, simulation can provide 2 levels of benefits. The first one is reliability, and the second one is efficiency.

For reliability, if you're thinking about a robotic system working reliably in real environments, you need data to provide systematic coverage of all the state space and the variations that robots might encounter. That's how you can learn how to make the system robust. With simulation, you can do systematic randomizations and control the variations in lighting, friction, geometries, object types, and all different kinds of physical parameters to make sure you have sufficient coverage of the state space. This is what can give robotic systems reliability.

The second is efficiency. Right now, many people are doing teleoperation, and if you look at many of the teleoperation devices, imagine all the actual exoskeletons you're using: you're collecting data at a speed that's actually slower than a human doing the task.

Martin Casado

But for many of our clients, human speed is not good enough for them. They want faster-than-human speeds.

Yunzhu Li

So for a robot to move faster, it's not as simple as just driving the robot faster, because gravity doesn't change. But in simulation, you can systematically speed up the robot's behaviors to train the robot such that it considers all the dynamics changes of the environments. This is what can give our clients efficiency.

For both reliability and efficiency, there are some very unique values that simulation can provide.

Martin Casado

You've talked about the technology and the platform and what it does. Maybe talk about the specific use cases people use it for.

Yunzhu Li

There are essentially 2 specific use cases, especially around both training and evaluations.

Martin Casado

Okay.

Yunzhu Li

Starting with evaluations, evaluation is something people often overlook in robotics. But if you're tuning robotic models, you have to know how well they work, and that is the only source of information for you to iterate.

Martin Casado

A lot of AI people really understand what evals are and use them all the time. For non-AI people, it often means something a little different, so maybe it's even worth describing specifically what you mean by evaluation.

Yunzhu Li

Okay. What I mean by evaluation is that you'll be able to understand, for a specific checkpoint, how well it performs. Does it perform, for example, 95% of the time or 99.9% of the time?

The key criterion people use in industry is: How long does it take? How much wall-clock time does it take for you to distinguish between a checkpoint that's 90% accurate and a checkpoint that's 92% accurate? If you only do that in the real environment, it just takes so long for you to make that distinction.

If you really think about the robotic evaluations people are doing right now in real environments, the iteration speed is multiple orders of magnitude slower than the iteration speed of those language models.

Martin Casado

Yeah.

Yunzhu Li

Not only are robotic tasks very varied and very diverse—

Martin Casado

Oh, yeah, because you actually have to do the thing. The atoms have to move through space. [laughter]

Yunzhu Li

Exactly.

Martin Casado

The laws of physics have to be obeyed. Have you watched those robotics videos? Every video has 10× or 8× speed-up because it moves so slowly.

Yunzhu Li

Exactly. So not only is it slow, it's dangerous and costly, but at the same time, the speed is also multiple orders of magnitude slower.

Some of our clients actually need a digital environment that can be used to evaluate their robotic systems, and our digital environment has proven alignment with the real world. Whatever happens in the simulation is also likely to happen in the real environment. If a checkpoint is working better in the simulation, it's also highly likely to work better in the real environment, as we've also discussed in the blog post.

That actually gives our clients very strong confidence in using the data and the signal from the digital environment to do scalable, safe, and much faster evaluations of their robotic systems.

Martin Casado

Great. So that's the evaluation. Then, on the training—

Yunzhu Li

On the training side, as I also mentioned, it's about controllability. You want to control all the different possible variations of states, parameters, lighting, friction, physical parameters, and even object geometry and object types. You want to make sure you have sufficient coverage of all different kinds of scenarios, such that you'll be able to generate informative data for your robots to be robust.

This is just going to be so hard to do in real environments, as we discussed. If you do teleoperation, the speed at which you're collecting data is slow. You're also limited by how many robots you have and how many teleoperation devices you have. There are all different kinds of challenges around the data operations involved.

But in simulation, everything can be controllable, everything can be systematic, and everything can be understood at a level where you know exactly what distribution you have covered. You can develop confidence that, within that distribution, the robot will work. That confidence, efficiency, and scalability are things our clients also value when they use our digital worlds for training robotic systems.

Fei-Fei Li

Here's the crazy thing: even before Yunzhu and I were talking, our inbound customers for Marble were already showing this kind of demand. We just couldn't serve these customers. But we were already getting a lot of phone calls from early-stage robotics companies, all the way to downstream, very pragmatic use cases, and we were seeing these needs.

When people hear you're going into robotics, what they're going to envision is that you're pulling out a 3D printer, making hardware, programming the brain of a robot, sticking it in the robot, and then you've got a robot. I don't think that's what you guys are talking about here. So maybe talk about where this fits in the life cycle of creating a robot, where you will end, and where the rest of the ecosystem will begin.

Yunzhu Li

What we've been building, you can imagine, is infrastructure, with the software around this infrastructure, for people to build worlds such that robots can learn and be evaluated. This infrastructure is naturally model-agnostic and embodiment-agnostic.

Martin Casado

I just want to be very clear, because this is actually a subtle point. It's obvious to you, but it's a subtle point: from what you said, you're not building a robot. You're building an environment in which another company can place its robot brain—

Yunzhu Li

Exactly.

Martin Casado

—to navigate and learn.

Yunzhu Li

Yeah. So our customers right now have all different kinds of robots. Some are using a single robot arm, some are using bimanual robots, some are using a fixed arm, and some are using mobile manipulators. Some are using grippers, and some are using more elaborate versions of end effectors.

Our platform is naturally embodiment-agnostic. We can very easily integrate different kinds of robotic embodiments and put them into the worlds we generate with Marble, such that we'll be able to give those individual robots the capability to do the right tasks at the right levels of reliability and efficiency in real environments.

We're also model-agnostic. We can use the data generated by our worlds to train different models, either from scratch or through post-training of existing foundation models, like vision-language-action models or world-action models. To us, it doesn't matter. We just want to make sure we have the infrastructure and all the worlds such that the robot can work reliably in the real environment.

Martin Casado

You've told me that you think a lot of the predictions around humanoids were a little bit aggressive, and we're likely to see more constrained rollouts, like warehouses or whatever. Can you talk a little bit about that and how it impacts what you're going to be tackling here at World Labs?

Yunzhu Li

That's a very good question. If you look at the progress of robotic applications in real environments, it has always followed the trend of going from fully structured environments into semistructured environments and then into unstructured environments.

Martin Casado

For fully structured environments, what do we mean? That you have knowledge and control over all the configurations within the environments?

Yunzhu Li

Like factories—

Martin Casado

Like factories.

Yunzhu Li

—or car manufacturing. Those have been automated for decades.

Martin Casado

Yeah. Yeah. Yeah.

Yunzhu Li

Then you have semistructured environments, in which you have certain control over the environment—for example, Amazon warehouses, restaurants, and hotels. You have certain control over the environment to make the task easier for your robots, but there are obviously many other objects—for example, clothes—that you don't control.

For unstructured environments, it's your home and my house.

Martin Casado

Those are, I would say, the grand challenge. Especially my house, trust me: 3 dogs. [laughter]

Fei-Fei Li

5-year-old.

Speaker 1

Yes, dogs.

Speaker 2

Exactly.

Yunzhu Li

If you're thinking about where robustness comes from, robustness comes from sufficient coverage of the scenarios that robots might encounter. It's so much easier and more approachable, at least right now, to focus more on semistructured environments before we move on to fully unstructured environments. We will move in that direction; we just want to take a more sustainable and realistic approach toward it.

Martin Casado

I think your point here is that humanoids mimic the human body, and evolution has optimized the human body for unstructured environments. Our fingers and legs are not the best apparatus for doing one thing. For example, if our only goal as a species were to climb trees, we wouldn't have this body necessarily, right? We'd have different kinds of fingers. But what humans ended up evolving into is this body shape that can be very general, but not necessarily the best at everything. That is for survival in unstructured environments.

From a business point of view, and from a pragmatic technology point of view, this unstructured environment and a generalized body are actually the hardest problems to solve. It's not necessarily even the right way to solve the problem. We specialize, so we take a more specialized body to solve a narrower problem. But the challenge for World Labs is to be more body-agnostic, so that its infrastructure can serve different bodies and different semistructured environments.

You know, a common lens to look at exactly this question is an economic lens, right? You compare it to generative LLMs, where they can create prose or code 10,000 times faster than a human being and a bunch cheaper than a human being. So the economic case makes sense because our brains aren't very efficient at that. However, our brains and our bodies are very efficient at 3D navigation—moving through the world or picking things up.

This is just a prediction question, but do you believe we'll ever be able to build robots, at least in the foreseeable future, that have the power efficiency of a human being when it comes to menial tasks? Let's say minimum wage or something like that. How far away are we from this? Is this 5 years, or is this never?

Yunzhu Li

I think it's going to take a very long time. If you really think about robots in the real environment, in the end it will always be a system. Every working robot in the real environment is a system. You need to be very mindful and thoughtful about how the system comes together: the hardware, the software, the brain, and even details such as the friction coefficients of your fingers. There are a lot of things you have to consider to make these things a reality, and it will take iterations.

What I am excited about is that I have always been at the state of the art of robot learning and trying to push the state of the art forward. The state of the art is always moving faster than I expected. What I'm focusing on and trying to investigate right now is very different from what I was focusing on when I started my PhD. This speaks to how fast the whole ecosystem has been evolving and how all the moving pieces are starting to come together to build these robotic systems. But we also have to be calibrated about our predictions. We will see a lot of progress, but to achieve, for example, human-level efficiency and capabilities, it will take longer.

Fei-Fei Li

Martin, the hardest thing in today's AI is to have the right measured optimism. [laughter]

Martin Casado

Right? I mean, even LLMs do not have human-brain efficiency. The human brain operates on 30 watts.

Fei-Fei Li

Yeah, that's true.

Martin Casado

But performance-to-power may be close, right, in narrow tasks like software engineering, like generating an image or software engineering. I don't think we're anywhere close when it comes to robotics.

Does this change how you think about, strategically, the level of ambition that your team can go after? Has it changed that, or is it still very much in line with what you expected to do when you started?

Yunzhu Li

It definitely changed the trajectories in a very profound manner. We see a lot of unlocks in being able to do this whole process—the modeling of environments—much more efficiently and at a much more scalable level, especially in partnership with World Labs.

I also want to add that, if you think about the current state of language models, those are models with incredible capabilities. But you don't just blindly trust them to book your flight tickets or make your hotel reservations. Hopefully, there is still a person who reads the output from those language models.

Martin Casado

Yeah.

Yunzhu Li

But that is very different from how people will be using robotic models, because robotic models have to work reliably in the real environment out of the box. We don't even have the data or all the necessary infrastructure around robots for them to work reliably out of the box in real environments.

For that reason, being able to create these digital worlds—scalable digital worlds where the robot can learn and evaluate within them—is going to unlock so much more potential by replacing costly and unsafe data in real environments with data generated from these worlds, allowing robots to learn and evaluate at scale.

Martin Casado

I've seen many of these kinds of integrations. They actually work very well at this stage when they have this much alignment, which is great. But there's always this question of whether you integrate now into what's happening now, or whether you keep things quite separate and provide a long-term trajectory that will be realized in a year-long time frame.

How are you thinking about this, Fei? Is this something that integrates right away, or is this a separate, longer-term effort?

Fei-Fei Li

This is a great question. I think at this point Chang, Sonny, Justin, Ben, and I have been talking about this. We are going to take it thoughtfully. We're not rushing to integrate everything, from codebases to teams, because SpAItial does have a very well-thought-out—and I wouldn't call it completely standalone, but fairly contained—tech stack, as well as its customers and the products it's building.

We're going to take time. We definitely will integrate; we're already starting to talk on the simulation side, as well as on the potential base-model and action-conditioned-model side. They are also using Marble as an internal customer. We will be integrating, but we're not rushing to blend the team into a full salad bowl.

Martin Casado

Yeah. How are you thinking about geographies with SpAItial's move? Is it going to stay in the same place?

Fei-Fei Li

Yunzhu is going to move.

Martin Casado

Oh, well, welcome here. [laughter] Moving to San Francisco. Florence and the Renaissance. Perfect.

Fei-Fei Li

I think World Labs is officially becoming a bicoastal company, where the headquarters is in San Francisco. I've—you know, I live in Palo Alto. I feel like I'm in a different state. [laughter]

We're actually excited that we're going to have an office in New York that can help us attract talent on the East Coast. We've also been talking about making sure that in both offices we set up the robots, so that we can basically test out and mature our engineering stack and work with robots remotely, because we have to do that for our customers anyway.

Martin Casado

So maybe, just to be very concrete, Fei, let's pencil out what the perfect success case is in 2 years. What product do you have? Who's engaging with it? How do they use it? Just the crisp version—what is it?

Fei-Fei Li

We're very happy that the SpAItial team and World Labs team will have validated customers in a small number of important vertical use cases where our system and infrastructure have proven to be truly beneficial to their automation needs. These customers will become our lighthouse examples to scale our business.

Martin Casado

And how early—let's say someone listening to this is running a robotics company—at what stage do they engage with World Labs? Is it really early on, or is it somewhere in the middle?

Fei-Fei Li

Right now, for our customers, we're building these real-to-sim-to-real pipelines, where the simulation is essentially the world we're going to provide as the training and evaluation grounds. Some customers need only the real-to-sim part. They want to digitalize the tasks they care about and be able to evaluate their robotic systems.

Some customers need this entire real-to-sim-to-real pipeline, so they will be able to have policies running on their hardware. Our platform is designed in a way that is flexible, depending on what our clients need.

At the same time, the clients we are working with are actually pretty close to the deployment stage. Basically, they are working on very practical tasks—tasks that, when replaced with robotic solutions, can create value immediately—and they have at least dozens or hundreds of these kinds of situations they are thinking about automating.

Together with World Labs, we'll be able to develop reliable solutions for those scenarios. As we've already shown, we have a number of scenarios instantiated in our blog post, and we'll be able to further our investigation to see how they can actually solve the key requirements and constraints faced by real-world deployments.

Martin Casado

Great. I want to be very specific about this. Is it ever too late or too early to call World Labs if you’re a robotics company?

Fei-Fei Li

No. We want everybody to call us. We want to learn about your use case.

Martin Casado

Wonderful. If you’re listening to this and you’re anywhere close to a robotics project or robotics company, please contact World Labs.

Yunzhu Li

Yes. Thank you.

Martin Casado

Definitely open for business.

Fei-Fei Li

We are open for business.

Yunzhu Li

Not too early.

Martin Casado

All right. If you’re doing robotics, call World Labs. Thank you both very much for coming.