[BidClub_]
Latent Space · · 72 分钟

150亿美元的Physical AI公司:仿真、Autonomy OS、Neural Sim与1000名工程师——Applied Intuition

Alessio FanelliswyxQasar YounisPeter Ludwig

YouTube
TL;DR
  • Applied Intuition 将自己定位为Physical AI的横向技术供应商,而不是另一家机器制造商。 Qasar Younis 描述了一套覆盖仿真、操作系统和Autonomy的技术栈,服务于汽车、卡车、农业、建筑、采矿和国防;主持人称,全球非中国汽车制造商前20名中有18家是其客户,Peter Ludwig 则表示 Applied 正在日本运营L4级无人驾驶卡车。其核心逻辑很直接:向机器制造商和政府出售让机器“变聪明”所需的技术。

  • 操作系统层可能是整套技术栈里最不性感、但战略价值最高的部分。 Qasar 将今天碎片化的机器软件比作Google在Android之前面对的大约50套手机操作系统:在这种碎片化被整合之前,现代AI无法一致地部署。Applied 的开放式、跨芯片OS负责实时控制、安全回退、内存、网络和可靠更新,包括过去必须到经销商处才能更新的安全关键模块。

  • 即使底层AI技术栈不断换代,Applied 的护城河仍会跨层复利。 公司大约每两年重建一次技术,至今经历了约4次重大演进;与此同时,仿真、操作系统、工具链和模型工作持续积累。公司拥有超过30款产品,83%的员工在工程岗位,公开披露的1000名工程师规模也已经被称为过时;按Qasar的说法,它像是技术供应商版的NVIDIA或AMD,“但我们就是不做芯片”。

  • 对Physical AI而言,模型智能未必是真正的瓶颈,部署才是。 车载系统必须在毫秒内给出答案,同时满足严格的功耗、成本、散热和可靠性约束,因此“每一毫秒的零头都至关重要”。Gemma 2B规模的模型可以在设备端运行,但核心Autonomy仍由公司完全内部开发且高度专用;通用模型更适合语音等通用交互。

  • 安全验证正从二元测试用例,转向衡量学习系统能维持“多少个9的可靠性”的统计性判断。 Qasar表示,更好的模型让故障更难被发现,从而提升了评测、神经仿真和人工复核的重要性;他强调,对AI编写的安全关键软件进行人工验证仍然“100%关键”。监管机构很重要,但Applied称,推动这些方法的首要原因是让自己和客户放心。Raquel Urtasun补充说,监管往往只是最低公分母,因此好的产品必须大幅超越监管要求。

  • 世界模型可以扩展仿真,但创始人拒绝相信合成经验能够取代真实世界测试。 纯粹依靠世界模型部署,很可能会在“破产之前”就失败;仿真器必须反复与现实校准,Alessio则提出了一条经济分界线:虚拟测试需要足够有信息量,同时仍然比实体测试便宜。他们最好的例子都很具体:把执行器温度纳入建模,可以让人形机器人学会避免过热;视觉线索也能教会车辆在出现水滑迹象时减速,而无需显式理解这一概念。

  • 最大的执行风险,位于令人印象深刻的Demo与持续维护的生产车队之间。 Qasar区分了基础研究、先进工程和生产运营;人形机器人的脆弱性、Peter谨慎提及的中国机器人马拉松案例、DARPA Grand Challenge以及24 Hours of Le Mans,都说明可靠性必须通过高强度测试不断逼出来。对创始人而言,Qasar的建议很明确:商业范围要窄、执行要深、战略要与阶段匹配。Physical AI的复利空间巨大,但“每一步都在流血”,许多公司会在等到回报前耗尽资本。

摘要 · 为研究而整理的核心内容

1. Applied Intuition向机器制造商出售技术,而不是向消费者出售机器

  • Qasar将使命定义为打造“让世界更安全、更繁荣”的Physical AI,覆盖汽车、卡车、建筑与采矿设备、农业和国防,横跨陆地、空中和海上。Applied向制造商和政府销售或授权技术,而不是亲自成为制造商。

  • Peter补充了一个风险更高、也更有说服力的案例:Applied“就在此时此刻”于日本运营L4级无人驾驶卡车,学习系统犯错不能像聊天机器人答错一样被轻轻带过。

  • 主持人称,全球非中国汽车制造商前20名中有18家是Applied的客户。Qasar否认早期将公司与Scale AI相提并论主要是因为两者都属于YC生态:Scale本质上是一家标注和服务公司,而Applied从开发者工具起步,至今仍然销售技术。客户既可以只购买开发套件中的一个部分,也可以购买“全套”。

  • 公司最初服务机器人出租车客户,提供仿真和数据基础设施,随后扩展到超过30款产品。端到端Autonomy模型让跨形态泛化变得更可信,也推动创始人最初“把软件部署到实体机器上”的抱负向前发展。

2. 技术栈覆盖仿真、操作系统和机器智能

  • Peter将产品组合分为3个主要层次:仿真及其开发基础设施;真正的操作系统,包括调度、内存管理、中间件、消息传递和可靠网络;以及覆盖世界模型和车载Autonomy的基础AI工作。

  • 仿真同时支持虚拟开发和真实世界测试,二者需要反复校准。强化学习已经成为核心方法,但其价值取决于能否“足够快、足够便宜地”生成足够多的相关仿真经验,从而改进已部署的模型。

  • Qasar还补充了纯Autonomy之外的人机协作层:语音、座舱感知,以及在需要人工介入或关键决策前持续运行的Agent。他用农场作类比:1个人监督多台机器,这种安排类似于仍有人参与的SAE L2++。

3. 即使最终从生产系统中消失,Lidar仍是有价值的脚手架

  • Applied不制造芯片或传感器。Qasar表示,公司会支持一套偏好的传感器组合,但客户也可以提出替代方案;最终配置取决于机器、环境、成本目标和可靠性要求。

  • 对于摄像头与Lidar之争,Qasar给出的答案是有条件的,而非意识形态式的:Lidar“毫无疑问是有用的传感器”,尤其适合研发和数据采集,因为它能提供逐像素深度。这些深度信息可以监督摄像头训练,之后量产车辆可能移除昂贵的Lidar,仅靠摄像头推断深度。

  • 国防场景会再次改变取舍:夜间运行的机器可能需要红外线,而通过Lidar或雷达主动发射能量又可能并不理想。不同领域追求的不是一套标准化传感器组合,而是适合自身运行条件、成本最低且可靠性最高的量产系统。

4. 机器软件必须先完成整合,AI才能规模化

  • 车辆OS远不只是驾驶员看到的那个卡顿仪表盘界面。它必须控制电机、发动机、转向、执行器、冗余系统和实时传感器流,并严格管理延迟、内存、故障安全机制,甚至防范“一束宇宙射线把某个位翻转”。

  • Qasar称,可靠更新是Applied最重要的贡献之一。Tesla可能每月更新,但许多制造商要么完全不更新,要么只能更新HMI,且必须到经销商处操作;Applied的目标是更新任意模块,同时避免意外让一辆昂贵的汽车变砖。

  • 他的Android类比承载着平台化论点:Google当年面对大约50套互不兼容的手机操作系统,并创建了一个通用平台,以便大规模分发自己的产品。实体机器仍然高度碎片化,因此在一致部署现代AI应用之前,“首先必须整合操作系统”。

  • 这个平台仍然保持开放:客户可以把Applied的OS与其Autonomy栈一并授权,也可以将其与其他公司的系统组合使用。核心技术高度可复用,但必须覆盖多种芯片架构;实现主要使用C++,部分工作使用Rust,在更底层的实时需求上则会用到汇编。

5. Coding Agent正在改变产品,也在重新定义工程师

  • 在Claude Code“接管主导地位”之前,Cursor曾是Applied内部最受欢迎的工具,公司还通过排行榜鼓励员工采用。创始人也借鉴Coding Agent的交互方式:配置实体机器,最终应当更接近用自然语言指定一个应用。

  • Sensor Studio体现了产品形态的转变。它最初是一个高度依赖GUI、类似CAD的工具,用于布置传感器和评估取舍;如今则开放底层API,让Agent能够根据文字配置传感器组合,并且“很可能得到比人工操作更好的结果”。

  • 招聘如今更看重问题选择、编排能力和判断力,而不是机械式实现。Qasar观察到,候选人正呈现双峰分布:一类人深度使用AI工具,另一类人没有投入相应时间;即使面试变得更难,“生产力差距仍然大得惊人”。

  • Swyx提到,他过去认为嵌入式、OS和数据库工程师是普遍采用AI之外最明确的例外。Qasar说,6个月前他也这么认为,但现在这被证明是“苦涩的教训”的一个版本:即使GPU Shader和嵌入式工作也能从AI中受益,前提是对安全关键输出进行严格的人工验证。

6. 学习型Autonomy把验证变成寻找罕见故障

  • Qasar称,评测可能是“当下最难的问题”:模型变好后,缺陷更难找到,而漏掉一个罕见边界情况仍然不可接受。因此,更容易生成代码反而增加了评测负担,而不是消除它。

  • 端到端Autonomy如今直接把传感器数据映射为控制信号,改变了系统训练和测试的方式。强化学习需要仿真的传感器数据,这推动Applied走向“神经仿真”——一种被描述为Gaussian splatting与扩散方法混合体的技术,并将性能视为一等约束。

  • Qasar解释说,传统的Euro NCAP等项目会询问汽车是否通过了定义好的场景,例如儿童突然跑到车前时能否自动紧急制动。学习型系统则把问题改写成统计验证:能够证明“多少个9的可靠性”,以及故障之间的平均时间是多少。

  • Qasar表示,Applied正在向美国、欧洲和日本政府介绍如何建立无人驾驶安全标准,但政府并没有要求采用这套方法。更强的驱动力来自内部:道路安全是公司的第一核心价值。Raquel Urtasun补充说,监管往往只是最低公分母,因此产品必须大幅超越监管要求。

7. 统计安全无法消除单起事故带来的政治冲击

  • Swyx对Cruise的质疑是:即使Autonomy按统计方法评估,单起事故也可能让一家公司终结。Qasar起初称Cruise案例“不是技术失败”,转而指出公司处理监管机构的方式及其行为才是问题不断累积的原因。Shawn Wang反驳说,“这绝对是技术失败”,但也同意后续行为让情况恶化了很多。

  • Qasar随后将这起事故描述为长期问题链条中的“最后一根稻草”。这场交锋把系统可靠性与机构信任区分开来:统计验证很重要,但消费者是否采用,以及公司在事故后的行为,仍然是决定其能否存续的独立因素。

  • 主持人认为,社会最终可能必须接受一些人类不会造成的机器事故,因为总体安全性更高。Peter则怀疑AI系统会制造比人类已经造成的灾难更严重的后果:疲劳驾驶和酒驾司机没有Autonomy系统所具备的冗余和回退机制,后者需要“很多很多事情”同时失败才会出问题。

8. 仿真只有经过现实校准后才有效

  • Peter否定了仿真器一开始就能代表现实的前提:团队必须把真实世界结果反馈进参数,反复缩小sim-to-real差距,之后才有可能信任结论。完成全面校准后仍然可能出现意外结果,但那将是难得多的失败案例。

  • 他用人形机器人举例说明这种抽象如何落地。如果仿真器没有纳入执行器温度,强化学习可能选择看起来成功、却会让真实机器人过热的动作;加入温度变量后,策略就能在电机开始升温时调整动作。

  • Shawn Wang提出的冷冻室反例揭示了挑战:在一种温度条件下学到的策略,可能让机器人在冷冻室里表现更差。世界模型之所以有帮助,是因为仿真可以通过更多真实世界数据扩展,而不只是依赖工程师继续添加教材式方程。

  • Alessio描述了一条经济分界线:超过这条线后,在虚拟环境中复现现实将变得不经济。他粗略估算,车辆测试中95%是传统软件流程,4%是在没有轮胎和车身的部件测试台上完成,只有1%在实体车辆上进行;智能系统也需要类似的组合。

9. 世界模型无需理解人类概念,也能学会有用行为

  • Peter直言,完全依靠世界模型部署到真实世界的团队,“可能在破产之前都做不出能用的东西”。传统工程、其他开发方法、真实数据和实体测试仍然是让系统真正运行起来的必要条件。

  • 世界模型必须捕捉因果关系,而不只是画面:挖掘机移动土方后,物料所在位置发生变化,下一步可执行的动作也随之改变。这种状态转移远比生成一个看似合理的后续画面更难。

  • 水滑现象说明了隐式学习的可能性。模型可以把道路弯曲、路面排水、积水以及交通速度变化关联起来,并在这些线索出现时减速;正如主持人所说,“它不需要知道什么是水滑,也知道自己应该开慢一点”。

10. 车载推理让毫秒、功耗和连接能力成为决定性因素

  • Applied将运行环境分为offboard和onboard。Offboard模型可以耗时1秒或10秒,使用分布式算力并运行在大型GPU上;Onboard模型则只有固定数量的毫秒时间,机器随后就必须采取行动。

  • 因此,车载问题本质上是蒸馏和极限效率:模型可以源自更大的离线系统,但必须在保留性能的同时缩小。Qasar更广泛的结论是,Physical AI“目前真正受限的并不是模型智能”,而是如何部署到现有硬件上。

  • Peter表示,Gemma 2B模型可以在设备端运行,但需要定制。Applied的Autonomy“100%由内部完成”,包括数据、仿真和模型;通用模型则更适合语音助手等通用功能。

  • Applied的大多数系统仍然需要在本地设备端运行。Qasar还提到,美国许多矿区仍然具备网络覆盖。过去的采矿和农业系统使用昂贵的RTK定位,精度可达1–2厘米,但只能沿固定路径运行;现代模型则在动态世界中加入感知和行动能力。

11. Physical AI的回报来自研究、先进工程和生产运营之后

  • Applied维持多元化的技术押注,因为不同行业和地区面临的风险不同;大型Transformer方案只是其中一个方向,而非教条。规划方式也有所不同:驾驶可能接近轨迹预测,而采矿可能需要大约100个步骤,前面挖取的土料会改变后续状态。

  • 创始人认为,“世界上可能几乎所有东西”都可以还原为下一个Token的预测,前提是Token代表动作、轨迹或状态变化。一次采矿挖斗可以成为一个Token序列,改变上下文,并影响下一次预测。

  • Qasar将基础研究与先进工程、生产运营区分开来。人形机器人仍然很脆弱;Peter谨慎提到,他认为中国即将举行一场26英里的人形机器人马拉松,而DARPA Grand Challenge和24 Hours of Le Mans则展示了围绕可靠性的高难度测试和竞赛。

  • 经过近10年,创始人声称自己积累的是模式识别能力,而非天才:看到一个Demo后,他们可以列出“接下来20个问题”,并预测可能尝试的修复方案,因为他们已经经历了大量失败。这个机会如今变得可信,不是因为机器人概念变新了,而是因为“这些东西现在真的能工作了”。

12. 硬科技创始人需要足够长的商业约束,让技术持续复利

  • Qasar给硬科技创始人的建议是,为一个硬技术问题设定一个商业边界。资本充裕可能掩盖聚焦的必要性,但Applied的OS、工具链和模型只有在公司存活足够久的情况下才能复利;他以Waymo为例:在他的表述中,Waymo曾长期有趣,但直到复利效应显现之前,并不值“1260亿美元”。

  • 他仍然更倾向于在一个较小的问题空间里“做得非常深”,而不是广泛铺开、浅尝辄止。照搬Apple的纵向整合战略,却不具备Apple同等的成熟度,是一种类别错误:2007年的Apple并不是1978年或1982年的Apple。

  • 隐身策略同样取决于公司所处阶段。Applied的创始人拥有Google人脉,公司前400名员工中多数曾在Google工作;这套招聘基础让他们的路径不同于一个没有类似资源的创始人。答案在于创始人能力、早期人才和客户证据之间是否匹配第一性原理。

  • Qasar表示,2014年的YC建议不能直接支配2026年:Floodgate的一项分析显示,种子基金中估值低于100万美元的基金数量,曾经只有个位数,后来接近350家,甚至多到无法继续计数。Applied寻找的是擅长模型效率、延迟、评测以及软硬件边界的工程师——他们拥有“极强的好奇心”,会不断向更底层追问,直到软件最终落在硬件上。

Before we get into today's episode, I just have a small message for listeners. Thank you. We will not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis, but fortunately enough of you actually subscribe to us to keep all this sustainable without ads. And we want to keep it that way. But I just have one favor to ask all of you. The single most powerful completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you and it means absolutely everything to me and my team that works so hard to bring the AI in space to you each and every week. If you do it, I promise you we'll never stop working to make this show even better. Now, let's get into it.

Qasar Younis

Physical machines today are more akin to the state of the phone market before Android and iOS existed. Part of the reason that Larry at Google decided to get into Android was that they wanted to run Google products on a bunch of phones. They bought all of these phones from the industry, and it turned out they had around 50 different operating systems on these phones. It was virtually impossible for Google to make their app run on all 50 devices equally well.

The solution was, “What if they created a really great operating system and made it attractive to all of these phone makers?” That was sort of the genesis of what Android was and why Android existed. It was a way for Google to get their products onto a really wide diversity of devices. The state of the physical industry right now is a little bit like that: so many different operating systems, and it’s so fragmented. To actually get a modern AI application to run on these vehicles, you first have to consolidate the operating system. That’s why we’ve done that.

Alessio Fanelli

Everyone, welcome to the Latent Space podcast. This is Alessio, founder of Kernel Labs, and I'm joined by swyx, editor of Latent Space.

swyx

We’re very honored to have the founders of Applied Intuition, Qasar and Peter. Welcome.

You guys really know how to turn it on to podcast mode. You were just joking around right before this, and then you flipped it pretty quickly. You guys are real pros at this.

Alessio Fanelli

Oh, yeah, it’s good to have you guys. Maybe you just want to introduce yourselves so people know the voices on the mic.

Peter Ludwig

Sure. I’m Peter Ludwig. I’m the co-founder and CTO of Applied Intuition.

Qasar Younis

My name is Qasar Younis. I’m the CEO and co-founder with Peter.

Alessio Fanelli

Nice. Can you guys give the high-level overview of what Applied Intuition is? I was reading through some of the Congress files when you went out there, Peter, and 18 of the top 20 global non-Chinese automakers use you guys. You have customers in agriculture, defense, and construction. I think most people have heard of Applied Intuition tied to YC when it was first started, and then you were kind of in stealth for a long time. Maybe just give people the high-level overview of what it is today, and then we’ll dive into the different pieces.

Qasar Younis

Applied Intuition’s mission is to build physical AI for a safer, more prosperous world. We work on physical AI for all different types of moving systems, everything from cars and trucks to construction and mining equipment and defense technologies. We’re a true technology company, so we build and sell the technology. We sell it to the companies that make the machines, and we sell it to the government—really, anyone that wants to buy technology to make machines smart.

Peter Ludwig

In the broader AI landscape, a lot of the focus, rightfully so, in the last 3 years has been on large language models and everything that fits in a screen, whether it’s code-completion products or things like that. What’s different about us is that we’re deploying intelligence onto a lot of things that don’t have screens. They’re physical machines.

There are sometimes screens within the cabin of, for example, a car or a truck, but most of the value we provide is putting intelligence into safety-critical environments. Those two words are really important, because learned systems can make mistakes. You can ask something like, “Tell me about these podcast hosts that I’m about to go meet,” but you obviously can’t do that when you’re running, for example, driverless trucks in Japan right now, as we speak. You can’t have errors. Those are L4 trucks.

swyx

Was that always the mission? I remember that initially people put you and Scale AI in a similar category on the data-infrastructure side of things. What was the evolution of the company?

From the very beginning, we always wanted to be a technology company that helped generally push forward the industrial sector. We started off working on autonomy. Our very first customers were robotaxi companies, and we started off doing a lot of work in simulation and data infrastructure.

Over the years, we’ve expanded our portfolio. Now we have over 30 products, and it’s a pretty broad technology play within the landscape of physical AI.

Qasar Younis

I think the Scale comparison is because we’re all YC-universe companies, but it was a very different company. Scale was more of a services company, fundamentally a data-labeling company. We started, and still are, doing a lot of tooling.

Developer tooling is now in vogue again thanks to the AI boom, but honestly, 10 years ago it was out of vogue. Doing a tooling company in 2016 or 2017 was not the thing to do, because the general view among VCs was, “The tooling is just workflows, and workflows ultimately are not really interesting.” We’ve come full circle with that.

When we started the company, this was kind of in the periphery of what the company wanted to be. From our earliest days, we wanted to deploy software on physical machines, on cars and trucks and things like that. Obviously, we didn’t know that the Transformer boom was going to happen, and we didn’t know that autonomy systems would become end to end. Those things we didn’t know.

What’s important about autonomy systems going end to end is that those models can now be generalized to multiple form factors. So, 9 or 10 years ago, tooling was a great way—and still is a great way—to build the technology and sell technology to our end customers, many of whom want to build the stuff themselves. We offer a spectrum of solutions, from using just one part of a development suite of tools all the way to buying the full thing.

The way to think about the company, or at least the way we think about the company, is as a technology provider. It’s kind of like what NVIDIA or AMD does, but we just don’t do chips. We don’t do silicon. We’re fundamentally a technology provider.

When we started the company, we used to joke that we weren’t the guys to build something like Instagram. That was just not us in the most fundamental way.

Peter Ludwig

What was clear 10 years ago was that there was so much more that was possible with software and AI in vehicles, and that was generally the space that we started in 10 years ago. The precise path that we’ve taken over the years has been strategic, and we’ve adjusted to make sure that we’re actually building stuff that’s valuable to the market. The technology has changed so much. Our own technology stack has completely changed, I’d say, roughly every 2 years.

We’ve probably done around 4 complete evolutions of our own technology stack, and I see that cadence roughly keeping up. The way we think about engineering is almost on this 2-year horizon. We’re preparing ourselves to invest the appropriate amount, but then also to be very dynamic as the research gets published, as our research team figures out new advancements, and as we adapt to that.

Qasar Younis

One thing that has been consistent is the type of people we’ve recruited, frankly speaking. It’s engineers who fall into sometimes very traditional groups, like Google and Gen Z, but who are way different from those at other companies. We’re hiring folks who really know the intersection of hardware and software, who know really low-level systems, as well as traditional ML researchers and folks who actually put ML systems into production.

That’s been pretty consistent. If you look at the mix of our engineering, 83% of the company is engineering. So it’s a giant—it’s a lot of engineers.

swyx

Which, by the way, 1,000 engineers.

Alessio Fanelli

I mean, that’s on your website, so I imagine it’s out of date.

Qasar Younis

It is, and it is out of date. Yes.

Alessio Fanelli

Okay. And then 40-plus founders.

Qasar Younis

Yeah, I mean, we would say this was more luck than strategy. But we recruited a lot of ex-founders. It's been a great place for YC and non-YC founders because, obviously, I know a lot of the YC folks.

It's kind of like how we recruit a lot of Google people for them to exercise both their technical and non-technical skills, because we're on the applied side. We have a research team that does fundamental research, we publish, and we've had great traction there. But fundamentally, the business wants to take this intelligence and put it into the world. There's a certain type of person who's more interested in that.

Alessio Fanelli

Yeah. You mentioned the tech stack, Peter, so I just wanted to give you some free rein to go into it. I'm interested in where Applied Intuition starts and ends, in some sense. What won't you do? What do you do that's common among all the verticals that you cover?

Peter Ludwig

There's a few buckets of work that we do, and we've been at this for almost 10 years now. The technology is pretty broad, but we have 1,000 engineers. You could work on lots of stuff.

We got our start in simulation and simulation tooling and infrastructure. Generally, if you're trying to build a very complex software system that involves moving machines, you need to test that. The best way to test it is as a combination of virtual development and simulation, and then also real-world testing. There's a very careful process of correlating the simulation results with the real-world results and ensuring that the simulator is, in fact, accurate to that.

Simulation is a very deep topic. We have a whole suite of products in that, and we could talk for many hours about that specifically. But that is one part of what we do as a company. Reinforcement learning as a subpart of that is also super critical. I think a lot of the best advancements happening in a lot of these AI systems right now in some way relate to reinforcement learning, and now we have lots of compute, so you can do tons of interesting things with reinforcement learning.

The second bucket of work that we do is operating systems technology—true operating systems. Think about schedulers, memory management, middleware, message passing, highly reliable networking, and data links. The reality is, if you want to deploy AI onto vehicles, you need a really good operating system.

Qasar Younis

When we were getting deeper into that space, there wasn't really anything that we were happy with. Things existed, absolutely, and we were using what was available in the market. As an engineering organization, we realized these things weren't great. We thought we could do this better, so we decided to build something. That was the moment of inspiration that started our operating systems business, which is now a very real business for us. In order to write and run great AI, you need a great operating system, and that's what got us into that.

The third bucket that we work on is true fundamental AI technology. We have a lot of work in models, as I mentioned: the foundational research, but also the world models and actual autonomy models that are running on these physical machines. That's across cars, trucks, mining, construction, agriculture, and defense. That's land, air, and sea.

And also, a smaller subsector of that third bucket is the interaction of humans with those machines. That's a multimodal experience. Historically, if you're moving a dirt mover or any of these machines, there are buttons you press, whether they're actual physical, tactile buttons or something like a touchscreen. That's fundamentally changing to where you're just talking to the machine, and the machine and you are teaming.

Alessio Fanelli

Voice?

Qasar Younis

Yeah, voice absolutely. Also, the machine just being aware of who's in the cabin and what their state is. You can think, from a safety-systems perspective, the simplest version of this is that the driver is tired. If you get this alert when you're driving your car, it's like, "Maybe take a coffee break." Take that times a couple of orders of magnitude.

But this concept of teaming man and machine is important. When you think about running agents or just running different instances of Claude and doing work for you in the background, you can take that analogy and almost copy and paste it into a farm, where you have a farmer who's running a number of machines. You interact with the machine when there's maybe a critical decision, a disengagement, or something.

Generally speaking, the agent on the physical machine is running and making decisions on behalf of the farmer until there's something critical. That's also what we work on. It's not pure autonomy; it's a little bit of a mix, but it falls under autonomy in the automotive sense, which is typically defined in SAE levels as an L2++ system with the human in the loop. We just take that idea to other verticals.

Alessio Fanelli

Yeah. You've not mentioned hardware at all, like sensors. Obviously, you mentioned you don't do chips. I think even in AV there's a big cameras-versus-lidars debate. In your space, what are some of those design decisions that you made? Are they driven by the OEMs' ability to put things on the machinery? How much influence do you guys have on co-designing those?

Qasar Younis

We don't make sensors. We're not a manufacturer. Obviously, we use a lot of sensors in our autonomy products. In terms of what actually goes on the vehicles, we have a preferred set of sensors that we fully support, and our customers can choose from those. If there's a very strong opinion on supporting something else, we will add that to the platform as well.

The lidar question is, at this point, the age-old topic in autonomy. The state of the industry right now is that lidar is hands-down a useful sensor, specifically for data collection and the R&D phase of autonomy development. If you see, for example, a Tesla R&D vehicle, it actually has lidar on it to this day. In the Bay Area, you'll see Model Ys or Cybertrucks that have lidars on them just driving around.

It's useful because it gives you per-pixel depth information. If you compare a lidar with a camera, you can say, "This camera is looking this direction, and this lidar is looking this direction." Now, for each pixel of the camera, I can see how far away that pixel is. You can use that as part of your model training, and that depth information then becomes a learned state of the camera data.

When you're doing the production system, you can remove the lidar, and you can actually get depth with just the camera. That difference between a highly sensorized R&D vehicle and the cost-down production vehicle is something we use across our whole portfolio of products. Of course, the end goal is that you want something super low-cost and super reliable.

In certain use cases, you have more bespoke things. In defense, as an example, you do things at night often, so you care about sensors like infrared more. You don't want to be putting energy out, so you don't want to use lidar or radar. But you still need to be able to see at nighttime, so we work with the whole gamut.

Alessio Fanelli

Cool. That's kind of at the hardware level. Then, at the OS level, what does that look like? What's unique? My gripe is, I drive a Tesla. Whenever I drive some other car that has a screen, it always sucks. It's some cheap Android tablet; it's laggy and all of that. What does the OS of the autonomy feature look like?

Qasar Younis

When most people think about operating systems in a vehicle, you're thinking about what you just described: the HMI, the human-machine interface. Absolutely, that's an important part of it, but that's actually only one thin layer on top.

When we talk about operating systems for AI in vehicles, there are many layers that go deep into the safety-critical realm and embedded systems. You're talking about the real-time control of, let's say, the electric motors or the engine and the actuators, and you have different redundancies for different things, like the steering actuation in the vehicle. All of these things need very core support in the operating system.

For autonomy, you have real-time sensor data streaming in, and the latencies there are really important. If you try to imagine running Microsoft Windows while you're streaming your sensor data or controlling the vehicle, the latencies are going to be absurd. You can never do that.

What's special about what we do is that we really have this system-level thinking. We care about every performance characteristic of the entire system. Because we're doing a lot of the software—all of that software—we can fine-tune and control all those things. We can very carefully tune the latencies for every aspect of the system, tune the memory management, and have the right fail-safes and fallbacks for different things.

You have to account for what happens if there is a critical failure. What if there's a cosmic ray that flips a bit in the middle of the processor and causes some malfunction? You have to have a fail-safe for all of that.

And so the core operating system is a part of that. The last thing, which is a lot less exciting but is actually a very big topic, is the reliability of updates.

I have a Tesla, and you get updates pretty frequently—once a month. Most companies that are making vehicles are basically never doing updates, and even if they are doing updates, they're usually only updating maybe one module. Maybe they're updating the HMI module, but they're not able to update, let's say, the safety-critical parts of the system. You have to go into the dealer for that.

With our operating system, we can now enable highly reliable updates of any system in the vehicle. That's way easier said than done. There's lots of technically deep stuff in the tech stack to do that in a way that you're not going to accidentally brick a vehicle. If you imagine bricking a car, it's very expensive.

And honestly, across the industry, maybe one of the most sheer-impactful things that we've done is that we're now enabling the industry to actually do software updates.

swyx

Just to clarify as well, who is the customer for this? I assume a lot of hardware manufacturers have their own firmware, and I'm sure some of them would just have you write it for them because you're experts, while others would have their own. Who pays for this? Who invites you into the house? Is it the end user, or is it the manufacturer?

Yeah. Let me make an analogy, firstly, on the fragmentation of software. Physical machines today are more akin to the state of the phone market before Android and iOS existed. I worked on Android at Google, by the way, many, many years ago. Part of the reason that Larry at Google decided to get into Android was that they wanted to run Google products on a bunch of phones.

They bought all of these phones from the industry, and it turned out they had like 50 different operating systems on these phones. It was virtually impossible for Google to make their app run on all 50 devices equally well. The solution was, what if they created a really great operating system and made it attractive to all of these phone makers?

That was sort of the genesis of what Android was and why Android existed. It was a way for Google to get their products onto a really wide diversity of devices. The state of the physical industry right now is a little bit like that. These companies have firmware, but they have so many different operating systems. It's so fragmented.

To actually get a modern AI application to run on these vehicles, you first have to consolidate the operating system. That's why we've done that. Your specific question was, “Who are our customers?” Generally, it's the companies that are making these machines. We're selling our technology to them to really simplify the architecture and then enable these applications to run on them.

swyx

How much is reusable? Do you have one OS that's just configured for everything, or is there more customization that's needed?

Yeah, highly reusable. The fundamental technology is quite universal. Things we have to think about, though, are chipset support.

If you're coding, let's say, an LLM, and you start with an assumption that you're going to use CUDA and run it on an NVIDIA chip, then you don't really have to think about the hardware in that sense. You're just in the CUDA/NVIDIA ecosystem and you're going to use that.

But the hardware, especially in safety-critical systems, is a lot more diverse. There aren't one or two flavors. There's a bunch of different chipsets that we have to support. Our operating system doesn't just run on the equivalent of x86. It has to run on a number of different architectures, from chips made by a bunch of different companies.

We've been working on this for a long time now, so we have support for all of those chipsets. When you want to run the AI applications, we can do that reliably across a variety of providers.

Qasar Younis

I think that is heavily inspired by Android, right? Android has a huge suite of testing, and I think it's a reliable operating system that runs on thousands of devices. We think we can do the same in all these physical moving machines, with the difference that we're really in a safety-critical realm. Android isn't.

Qasar Younis

No. Yeah, it's totally open. Our philosophy is that we are a technology company, and so we license our technology to customers to use how they want. If a customer wants to license our autonomy tech and our operating system, then great, we'll license those. If they just want to license the operating system and then use different autonomy tech, that's fine also. We have great documentation.

swyx

What if you use developer tooling?

Qasar Younis

Yeah, exactly.

Shawn Wang

It's better together, obviously, if they're together. Is it all C++, I assume? Are there different compile targets?

Qasar Younis

We use a lot of C++. Rust is the hot new kid on the block for a bunch of things as well. But when you get to the lower level, especially when you get to real-time constraints, you hit C++ at some point. At some point, maybe you work your way into assembly when needed.

swyx

Damn. I'm curious about the coding-agent adoption, especially since you're mentioning more esoteric languages. What's the adoption internally? What have you learned?

Qasar Younis

Yeah, we use everything. Cursor was, I think, the hottest tool in the company for a good while. Now Claude Code has taken the reins on that. We have an internal leaderboard that we use just to encourage adoption within the company, and they're phenomenally useful.

Honestly, we take inspiration from some of those tools in how we're adapting some of that mindset to the physical realm. If it's so easy to build an app for this or that thing that lives just on a screen, we're taking a lot of the same ideas and applying them to the physical realm. If you wanted a physical machine to do something, how easy can we make that using our own tooling and platform as well?

swyx

Are you changing any of the OS architecture, kind of the way you expose services, to be more AI-friendly?

Qasar Younis

Yeah, absolutely. In the early days of our tools and infrastructure work, it was a lot about having engineers who were experts in certain topics. But the things that you're dealing with are often more mathematical or more abstract, where GUI tools are very useful for certain things.

As an example, we have a product we call Sensor Studio, which helps you design the sensor suite for your autonomous vehicle. It could be a car, a drone, mining equipment, or a robot. You place sensors in different places, and there's a library where you can understand the trade-offs you're making in the design of that system. That was a very GUI-intensive thing because it's a little bit more like a CAD tool in that sense, if you've seen CAD tools.

Nowadays, though, we expose all of the underlying APIs for that. Using AI agents, you can actually configure a sensor suite with just text and likely reach a better result than you could have through the GUI in the past. We're taking that thinking now through the whole product portfolio.

swyx

Another thing I was thinking about is, in terms of AI adoption, does that change your hiring at least a little bit? How do you manage engineers differently?

Qasar Younis

Yeah, absolutely it does. I think, like every company in the Valley right now, we're evolving our hiring practices because the skills required to be effective are changing so fast.

You used to really select for just rote implementation ability, and now it's more the AI engineer skill set. You know how to implement, but banging out code is no longer the core job. It's actually knowing what questions to ask and knowing how to tie together these different AI tools.

The interviews that we give now are way harder than they've ever been, but we also allow selective use of AI tools to solve the problems. In that, you start to see more of a bimodal distribution of engineers. You start to see this subset of people where you're like, “Wow, they really get it.” They're all in, and they've clearly invested the hours needed to learn these tools and how to be effective.

Then there's the group of people that haven't done that, and the productivity gap is just enormous. We're obviously trying to select for the people that are really, really into this.

swyx

I first wrote my AI engineer piece 3 years ago. When I first wrote about it, I was like, “Actually, not everyone should be an AI engineer,” because I think there's an extremist stance where, well, every software engineer is an AI engineer.

And my actual example of people who should not be adopting AI was embedded systems, operating systems, and database people. Are they adopting AI?

Qasar Younis

I think it’s the classic Bitter Lesson topic. 6 months ago, I would have said the same thing, but it’s becoming super useful for every domain, I’m sure. Six months ago, or maybe a year ago, if you tried to use, let’s say, the latest Claude model for writing GPU shaders, the results were probably underwhelming. If you use the latest model now to do that kind of task, you’re a little bit blown away, like, “Wow, that actually worked. That’s amazing.” We see the same thing in the embedded realm.

No question, though, especially when you get into safety-critical systems, human validation is 100% key. You’re not going to trust your life to AI-written software that hasn’t been very carefully checked by humans. I think now the challenge is really about that appropriate level of human validation for these safety-critical systems.

swyx

How do you think about—touching on the simulation side—I think verifiable rewards and reinforcement learning is the hottest thing. What have you done internally to build around that? What makes you sleep at night? If somebody is just coding something or wants to try something, do you have a good-enough system? I think the opposite is also true: if it’s super easy to write anything—

Qasar Younis

Mm-hmm.

Shawn Wang

—then it puts a lot of work on the verifiable side of it. What does that look like for people?

Qasar Younis

Yeah. Verifiability is a broader bucket of evaluations, right? How do you evaluate the results that you’re getting? I think this is probably the hardest problem right now, because as the models get better, it can be harder and harder to find the faults in the system. The problem of doing proper evals to find those faults also keeps getting harder as the models get better, but it’s no less important than it’s ever been, right? There are still going to be edge cases that are not met and whatnot, so it’s a big area of investment for us.

On the reinforcement learning topic, the key thing is that there are all of these new requirements that come with the latest generation of these technologies. For example, end-to-end is the big thing right now in autonomy and physical AI. You can now train these models to effectively take sensor data in, put control signals out, and get really good results from that. But the way that you train and improve those models is really different from previous generations.

To do reinforcement learning on an end-to-end model, you now need to actually simulate all the sensor data, right? We call our work in this NeuralSim, but think of it like a hybrid of Gaussian splatting and diffusion methods. We really care about performance. Performance is everything. If you can’t do enough simulation fast enough and cheap enough, you actually can’t get results that are worthwhile in the end. It also gets to a lot of our work in embedded systems, which is performance-critical work. That performance optimization and performance criticality carries over to a lot of the model training work, because the only way to make it affordable is for it to be really fast.

Qasar Younis

I think it’s worth a few minutes talking about our own evolving thoughts on verification and validation within traditional simulators—which are, you know, vehicle dynamics or something that would just take textbook formulas and put them into software—versus now, this neural sim/world-model universe. I think that’s an interesting topic.

Qasar Younis

Yeah. In more traditional development, you oftentimes would have more black-and-white answers to questions. In Europe, as an example, there’s a regulatory system called Euro NCAP, the European New Car Assessment Programme. As part of that, vehicles have to pass a bunch of tests. Those tests actually include safety systems, such as automatic emergency braking for a child that runs in front of a car or, let’s say, an included child that runs out and you hit it. You end up with these binary answers: Did the car under test pass this specific test?

There’s a very well-known set of test cases that the vehicle has to pass, and that was how the industry worked until about 10 years ago. But what’s changed now is that, with these models, everything is statistics, right? You no longer have a black-and-white answer. It’s, like, how many orders of magnitude, or how many nines of reliability, can I get in the system, and how can I prove that to be true?

The big unlock for physical AI as an industry is that these models are just becoming much more reliable. Things actually work a lot better. The number of nines you can get out of these systems is now good enough that it becomes cost-effective to really deploy them. The big shift in verification and validation has been from a more strictly requirements-based approach—are you meeting them or not?—to more of a statistical verification and validation case, where it’s all about how many nines of reliability you have and mean time between failures, that sort of thing.

swyx

Is the target audience regulators or even the customers? I imagine the customers are bought in, and it’s mostly regulators that need to be satisfied.

Qasar Younis

We do work with the U.S. government, the European governments, and the government of Japan. The government is not an AI lab by any means.

swyx

They just care about the outcome.

Qasar Younis

They care about the outcome, and so we do education in that regard—teaching, “Hey, this is how we think validation should be done, and this is an approach that we think is reasonable, and how to think about when a driverless system is actually safe enough to go on the roads,” and that sort of thing. But I wouldn’t say that the government is asking for it. We’re more teaching the government in that sense.

It’s honestly more so for our own comfort, because we want to build very safe systems. Of course, our customers care deeply about that as well, but in that context, we’re also typically educating our customers. Our first core value is on-road safety, so I think we can’t underline enough that us also verifying and validating that the systems we’re deploying are safe to us is probably as important as some regulator or a customer saying—

Shawn Wang

Of course, yeah. You have to certify yourselves.

Raquel Urtasun

Yeah. As a whole, across the world, regulation oftentimes is almost a lowest common denominator, but you really have to substantially exceed what the regulators are expecting to make good products.

swyx

One thing I often talk about, and I try to make this relatable to the audience as well, is Cruise, where they had an accident that basically ended the company. I wonder if people overreact to a single incident, because incidents are going to happen regardless, right? It’s just a statistical thing. I don’t know if regulators understand that you can’t extrapolate from a single incident, but we do, because that’s all we have to go on. Your sample sizes are necessarily going to be lower than consumer driving.

Qasar Younis

Yeah, I think the Cruise example wasn’t a technology failure. The real compounding issue there was just how the company talked to the regulators and what its behavior was, and I think that became more of the issue.

Shawn Wang

It definitely was a technology failure, but it was made much worse—

Raquel Urtasun

Yeah, yeah. Let me put it another way. There is a version where Cruise still exists.

Shawn Wang

Right, right.

Qasar Younis

It was the last straw. It was a long chain of—

Shawn Wang

Right. So if Uber ATG had that horrific accident, or someone had actually died—because, you know, that was a homeless person crossing the street—

Qasar Younis

Yeah, I think we can’t understate enough that, ultimately, statistical validation of something is one part of it, but it’s not the only part. Consumer and, let’s say, mainstream adoption of these technologies is also going to be part of that conversation.

I think companies like Waymo are doing a lot of positive service to the industry, in the sense that they’re setting a high benchmark and showing, in a very responsible way, how to deal with these. There have been Waymo incidents as well; they’ve just not been as significant as the Cruise one you mentioned. I think you’ll continue to see that. The long-term question is really going to be around the fact that it is very clear humans are way worse drivers statistically.

swyx

Yeah. There’s no debate. At what point do we accept that? But we’re emotional animals. Yeah, so my thing is that we have to get to a point as a society where we accept horrific accidents that would never happen with a human because, statistically, we understand that it is safer overall.

In the same way, planes are safer. I think they're the safest mode of transport that we have.

Alessio Fanelli

Yeah, I mean, it's more dangerous to drive to the airport than it is to get on a flight. So if you're ever getting nervous about getting on a plane, just think, “I just have to get to the airport,” and you're—

Shawn Wang

If I get to the airport, I'll be good.

Alessio Fanelli

But planes also concentrate terrorists.

Peter Ludwig

I don't think we honestly have to worry about there ever being accidents from these systems that are much worse than what humans would cause, because humans do terrible things. People fall asleep at the wheel all the time.

Alessio Fanelli

Yeah, I've been a drowsy driver.

Peter Ludwig

And drunk drivers. That's the extreme end of the example, but these AI systems have redundancies and fallbacks. Many, many things have to go wrong for there to actually be something catastrophic, because there are so many fallbacks that these systems have.

Shawn Wang

Yeah.

Alessio Fanelli

I mean, your simulation is so vast because there are so many use cases. What are maybe some things that worked in a simulation, and then you put it out and it's like, “This just did not work at all?”

Peter Ludwig

Yeah, so maybe there's a bit of a misconception about simulation there. Let me go a little more technical on this. At first go, no simulation is going to represent the real world. There's always a process of this sim-to-real matching where you need the real-world feedback to feed into the parameters that are being used in the simulator. You have to do that—it's like this validation flow—a number of times until you can get some confidence that, “Oh, I think the simulator is now accurately representing what's going to happen in the real world.”

Now, if you have a situation where you've done that full validation and you thought that it was accurate, and then there's something different, those are much trickier cases. That absolutely can happen, but I think the validation process is really important. You can never skip the simulation validation process, where you're actually ensuring that the sim-to-real gap here is small enough that you can trust these simulation results.

Alessio Fanelli

Right.

Peter Ludwig

There are so many fun things you can do when you get into it. I'll give one fun example that came up recently. In these humanoid robotic systems, overheating actuators is a real problem, right? Obviously, the demos are phenomenal.

They're the most amazing. I love watching robots do acrobatics, like everybody. But these systems actually overheat, right? One of the ways you can use simulation is to have the temperature of those actuators be one of the parameters represented in the simulation. Then, if you're doing reinforcement learning over a certain task, the robot can actually adjust its motions in the simulation to account for the fact that it knows that, as it's moving, it's beginning to overheat this motor.

But if you didn't have that parameter—let's say, the heat of that motor—represented in the simulation initially, then your RL policy would disregard that. Now you run that on the robot, and the robot will overheat and fail.

Shawn Wang

I guess the question is: how do you have all of these parameters taken care of while also understanding the deployment environment? Temperature is a good example, right? Why did you make my robot worse when it runs in, like, a freezer? So it actually shouldn't worry about that. How do you design these simulations?

Peter Ludwig

This is honestly what makes simulation so hard, right? Simulation is fundamentally about optimizing the development of a system. How can I build the system faster, better, and cheaper? What are all the levers that I have to actually accomplish that?

Because simulation is just a software program, you can change it a lot more easily than you can hardware systems. What's particularly awesome about, let's say, world models and using them as a part of simulation is that now the simulation doesn't just scale with adding new math equations. We can actually scale the simulation environment with additional real-world data, and that also unlocks a whole new field of robotics.

Alessio Fanelli

There is a demarcation line where you cross over and doing real-world testing is still better. There's an inherent sim-to-real gap; you can reproduce reality at exceedingly expensive costs, and, as they say, nothing is free. So you have to find that line where you're getting great performance and great feedback, whether it's on the training side or on the eval side, but it's way cheaper than doing it in the real world.

At some point, that doesn't make sense. Even from our earliest days in autonomy, our view was that you're still going to do real-world testing. There's not this magical land where you're not going to do that.

Maybe an even more nuanced version of this is traditional software development. Most of your testing for software in a vehicle—95% of that—can be traditional CSED kind of flows that you'd have in traditional web development. But once you have, let's say, a truck, then you can do 4% of those in a rig that has all the electrical and electronic components of a truck, but it doesn't have the tires. Then you have the 1% that is actually the vehicle.

There's a similar analogy in terms of using simulation for intelligent systems. You do a lot in a simulator and using world models, but ultimately, it's physical AI, so you're going to deploy it on physical machines, and the freezer example comes to light.

Shawn Wang

The world model thing has been, to me, the hardest thing to wrap my head around.

Alessio Fanelli

We've been doing a small series on another company, genuine intuition, as well. And I mean, we've done lots of coverage on NeRFs.

Shawn Wang

Yeah, it feels like we talked about the heliocentric system, right? In a world model, if you just feed it visual data, the model might learn that the sun spins around the Earth. It makes sense, right? And it's like, well, not really.

What are some of these other things? Hydroplaning is one thing I think about. Can our world model understand hydroplaning and what amount of water causes it to happen? To me, it's like, I don't understand how you guys do it.

I guess the real thing is, when you're doing both cars on the highway in Japan versus an excavator in a mine in Arizona, wherever you're deploying them, how much are you relying on world models to generate the simulations for you and then try to close the gap afterward, versus giving world models as a tool to your engineers to curate the simulations? If that makes sense.

Peter Ludwig

Yeah, totally. At a pure engineering level, I think if you're hoping to do real-world deployments and you're purely relying on a world-model approach, you probably won't get to something that works before you go bankrupt. There is a very practical mindset: world models are amazing and extremely useful for a lot of use cases, but there are a lot of other things you need to do to actually get something started, deployed, and working.

Most fundamentally, world models are all about understanding the world, but also understanding what's going to happen. It's the cause-and-effect relationship, right? If you have some sort of construction tool, and that construction tool is going to be doing some work on the earth in some way—it's going to be moving earth—the world model needs to understand that cause-and-effect relationship.

When I take this material from here and put it over there, now I have things that are over here and not over there anymore. That's that cause-and-effect relationship. Data, obviously, is a big problem.

The hydroplaning example is actually quite non-obvious sometimes. It's raining, and this road has the appropriate curvature so that the water is running off the road, and cars are driving faster here. Then you approach a road that's very flat, and water is puddling on that road. All of a sudden, cars are driving slower because when they were driving faster, they were starting to lose control.

There are a lot of visual, very nuanced cues in the scene, and I do think in the world model concept, there's a good chance the model actually would learn that you should just drive slower when these visual cues exist. That's obviously the beauty of these kinds of models: they learn these non-obvious things.

swyx

It doesn't need to know about hydroplaning to know that it needs to drive slower, I guess.

Yeah, yeah.

swyx

I want to ask questions about deploying models as well. I presume you use a lot of these world models for training data and simulation, but what about deploying them onto the systems in production? Presumably, you have GPUs on-device, but they're—I keep saying “on-device.”

swyx

What's the right term for this?

It's on-machine or embedded, yeah.

swyx

Yeah. What is the embedded world like? For people who are not used to that world, this is very alien.

Yeah. Actually, we call it onboard and offboard software. The great thing about offboard software is you don't have to care about time, and you can run really large models, right? You can say, "I don't care if it takes 1 second for it to give me a result or 10 seconds for it to give me a result," because we have time. The models can be really big, and they can run in a data center or on a huge GPU, and you can obviously have distributed compute, et cetera.

But onboard, you don't have any of those benefits. I have this many milliseconds where I need an answer from this model. A lot more of the energy is about distillation, and it's truly about efficiency—literally, every fraction of a millisecond counts. You can't have a situation where the model takes too long, because then the vehicle can't actually function.

swyx

Yeah.

Peter Ludwig

Even still, you use a lot of the same techniques. The models themselves, you can think of as a derivative of larger models that you can run offline. You're trying to get a model that still performs really well, but is a small enough version that you can then run on this embedded system, where you care about latency and power.

swyx

Yeah. Yeah, and I think the broader point, which maybe is not obvious but is worth saying, is that in the physical AI world, we're not really constrained right now by the intelligence of the models. It's actually what Peter's talking about: deploying them.

The hardware they give you.

swyx

Yeah, on the hardware they give you. There's just a reality, as these are safety-critical systems. Those end up being your limiting factors, rather than, let's say, a limiting factor for a foundation model company, which is going to be capital or researchers. For us, as people who kind of come in that realm, it's very interesting. Those constraints force creativity.

I imagine nobody was deploying or giving you the hardware for transformers back in 2018 or whatever, but now they are. What's the evolution like? Peel back the curtains a little bit.

Yeah. Transformers, first off, I think the paper was originally published in 2017.

swyx

So there's no time.

But I'm just saying, embedded ML systems usually have a lot fewer parameters and a lot less compute, and now they have orders of magnitude more.

Yeah, absolutely. What I was going to say, though, is I think in the original paper, maybe in the last paragraph somewhere in the paper, they talk about how this technique might be useful for images and videos as well. It took a few years for that impact to really hit, but now we're seeing transformers everywhere.

swyx

Yeah, using transformers.

Yeah, and the computers keep getting better and better. But you do have this fundamental trade-off, right? You have power, cost, and performance. Getting the right mix of those things in an embedded package that can also be shaken and baked in all the conditions that these things have to operate in is difficult.

But I think they're only going to keep getting better. We also have to plan our strategy understanding that we know the rate of improvement of these systems.

swyx

Yeah, so Google just released the Gemma 2B model. If you have an effective 2B model, is that useful to you guys, or is it too big?

You can run that model on an embedded system, definitely.

swyx

Yeah.

Peter Ludwig

So, yes, it's useful in that regard. The bigger question is, what do you use it for in an embedded system? You actually need to customize it quite a bit to make it useful for something. But, yeah, you could run a 2-billion-parameter model, definitely.

swyx

It's also interesting: what percentage is a custom ML model that only does that thing versus a generalist LLM, which probably is not that useful actually for your context?

You can imagine different use cases, right?

swyx

The voice stuff, yes.

Totally. For the actual autonomy elements, that's 100% in-house. We do every bit of that: the data, simulation, the model, everything. But when you get into the more generic use cases, like voice or a voice assistant kind of thing, that's where these more generalist models, like Gemma, can actually be quite useful.

swyx

Yeah, and then there's also obviously a trade-off between what percentage you must do on-machine versus just calling home.

Yeah, it's all about latency. It's all about latency, yeah.

swyx

Yeah. Well, I think actually in a lot of contexts, especially in the US, you can just have a connection to the web.

Yeah, I think, though, most of our universe—everything—has to be fairly embedded locally, because of the nature of it. Even in the US, there are a lot of mines that still have coverage, right?

If you look at the old world of autonomy within mining, which was long before transformers and neural networks in the CNN kind of universe, they were really just hand-coded systems. They were just like, "This machine is going to run to that place."

swyx

That was RTK, as I recall. RTK GPS, yeah.

Yeah. So that worked for 20 years. Why would we actually need to use transformers or more modern end-to-end systems? Mainly because you can only really run a path and run backwards. That provided a lot of value, but not as much as you get when the machine is actually intelligent. It's seeing, it's perceiving, and it's acting in a dynamic world.

swyx

I looked up RTK: real-time kinematic, 1 to 2 cm accuracy.

Yeah.

swyx

Fantastic, but fantastic in faraway lands where there's not going to be cell phone coverage.

Peter Ludwig

Yeah, that's why they use it on the legacy mining and agricultural autonomy systems today. For example, a combine that can be precise within 1 or 2 cm as it's driving down the field uses RTK. But it's expensive.

swyx

Yeah, and it's autonomy, but it's not intelligent in the way that I think all of us, if in 2026, would be talking about intelligence.

In one of your blog posts, you mentioned research on large-scale transformers that are similar to those doing modern generative AI. What are the big differences? Other than, "You're absolutely right, I should steer the car so I don't..." You really want to remove that.

We have a diversified bet strategy internally. The reason we've done that is because we now operate in a bunch of industries and geographies, and each of the approaches honestly has different risks associated with them. We're not going to put all our eggs in a single basket for a single approach, because that approach may not work out.

That's one of the bets that we have. It has certain advantages in certain scenarios, and the way that these things play out in practice is that it has certain benefits and also certain drawbacks. The research team then tries to work on the situations where that's actually worse than these other approaches, to ultimately arrive at a really great solution for all these things.

swyx

Is there a plan mode for physical autonomy? Do you have a planning step and then an action step?

The short answer is yes, right? Just like you can use cloud code to plan out some complex coding task and get some almost-specification written out, similar approaches absolutely can be applied to physical systems.

Imagine you're trying to accomplish some task. The easiest thing to think about is a robotaxi, but I think things get more interesting, let's say, in the defense context or in the mining context. You actually do have to think about many steps in advance. It's not just this one thing, but to accomplish the goal, there are 100 steps. This concept of a plan mode is very applicable.

swyx

Yeah, I was going to say, to me, driving feels like a great next-token prediction thing because you're kind of on a path, and it doesn't really matter what you've done before—you can always turn around.

Planning, yeah, yeah.

swyx

Yeah, versus mining, where it's like, "Oh man, I took a scoop out of this thing. Now we can't really—I can't really go there anymore." Is there a huge difference? I guess, do you have a taxonomy of these different types? There's driving, excavating, flying. How do you do it?

The interesting thing is, I think probably everything in the world can actually be boiled down to a next-token prediction problem. In any workflow, anything can be thought of almost as a sequence of steps or a sequence of trajectories, or whatever you want to call it, and it can actually be boiled down to that sort of thing.

In the mining case, you can imagine taking that scoop.

Okay, that was that set of tokens, and now the model is understanding that the state space is different. The next time I do token prediction, it’s going to be modified by that. But yeah, the remarkable thing about these techniques is just how universally applicable they are, right? I mean, it truly is incredible.

swyx

What else is underrated about what you guys are building on the physical side? We were talking about it before the episode. There are a lot of humanoid companies that do these great demos, and then I can’t buy it, so obviously it can’t all be there. In your case, you’re in production on real streets with a lot of customers. What are the things people are underestimating, in the same way that the Waymo demos 7 years ago were great and then it took 7 years to actually get them on the street? Can you share maybe about the last 1% that was really hard to get done technically?

Yeah, so certainly productionizing stuff is really challenging, no matter what. I would split the answer into research and production. First, on the production side, there are just so many problems that you find when you actually get the stuff to go in the real world.

The classic problem in humanoids right now is that these systems are actually pretty brittle. I’m not talking about any one company here, but just as an industry, the systems are pretty brittle. Interestingly, I saw this thing the other day that I think China is doing a marathon with humanoids.

swyx

Uh-huh.

Peter Ludwig

Yeah. In government—and not China specifically, but in any government—there’s a concept called prize policy. There are different ways of influencing an industry to go in a certain direction. You can regulate it, you can do mandates, or you can actually just do these competitions.

The U.S. version of this was the DARPA Grand Challenge.

swyx

And that worked.

But it really worked.

swyx

That’s right.

It really hurt the industry. But I think China is literally doing this marathon because they know that reliability of these humanoids is a problem. What cooler way to solve that than to have a competition where humanoids need to run 26 miles, right?

swyx

Are we there? Can robots run a marathon?

I think it’s happening any day now. So we’re there.

swyx

By the way, automotive has a version of this, which is 24 Hours of Le Mans, right? Porsche wins 24 Hours of Le Mans. Shoot them out there. It literally puts those products into production.

I would actually break it down. You talk about research, and you talk about production. There’s actually a step in the middle, which is advanced engineering. I think a lot of the industry is moving into advanced engineering, where it’s not fundamental research, like we’re coming up with novel techniques. It really is advanced engineering for production.

What are the subcomponents that are going to limit getting into production? Once you’re in production, you’re dealing with another set of problems, which are the deployment and maintenance of those machines that exist. I’d say, at least in our field, we’re mostly in advanced engineering, in automotive parlance.

swyx

Honestly, every step is hard, though. Peter, despite your worth $15 million, you still bleed.

Yeah, you bleed every step. And I think it’s fun. I find it really enjoyable.

We’ve been doing this now for almost 10 years, and we’ve seen so much bad times. Right now, we can look at any company in the space, get a demo, and write down a list of exactly the next 20 problems they’re going to hit.

swyx

Yeah, and I can guess what they’re going to try to solve each of those with, and I can guess which ones are actually going to work.

Yeah, it’s not like they were particularly genius solutions to the same stuff. We’ve seen enough of this stuff. We’ve lived enough of this stuff. Our own mental models of the world as leads in the company have been shaped by it. We’ve tried so many things. We’re talking about the wins here.

swyx

Right. There are plenty of losses among that many people doing that many different things. Does that get baked into your mental model of the world?

Yeah. But I’d say, in general, I get more excited about robotics, for sure.

swyx

Yeah—the massive opportunity.

And what’s happening now in the industry is that none of these concepts are new, right? What’s new is that the stuff is actually working now. People have wanted to use neural nets for robotics for a long time, but now we have the data sets and the simulation technologies where stuff is actually starting to really work. We want to be part of that. We’re going to be part of that, for sure.

swyx

Do you have requests for startups, or advice against starting certain startups? There are a lot of scale-up robotics companies. What do you think are the things people should be focusing on?

I think my biggest advice in this almost commercialization of technology is that the commercial constraint is often overlooked. We talked about hardware constraints, and there are also constraints on the commercial side: we’re only going to do things that fit in this box.

That is, I think, very good for founders. The reason it’s not often focused on is because you have plenty of access to capital, and the technical problems are so hard that you’re thinking, “I already have a constraint, which is just getting this technical problem solved.”

The venture community, generally speaking, tends not to be very technical. For them, if you just say, “If we solve this thing, this is going to be a lot of money,” that’s kind of enough. But as a founder, I’m not giving you advice on how to pitch VCs. That’ll work for VCs. You still have to run a sustainable business.

I think that ties into the question you asked earlier about what’s maybe not obvious about our company. This is truly compounding technology. A lot of the work that we do just compounds. We don’t throw it away; it gets better. The operating system work gets better, the dev tooling gets better, and the models get better.

We’re really going to get that. You see it in Waymo, as an example. Waymo is a company that was interesting for a long time, but not worth $126 billion, right? What happens is that the human brain just doesn’t emotionally understand the compounding effects.

That’s going to happen in our universe. If you’re a founder, you’re at the beginning of that long walk. If you can put a little constraint on the commercial side, that gives you a better chance of seeing the other end of that walk. When you get to the other end, you will get the big return from compounding technology. A lot of people just don’t make it.

To summarize, think a little bit about the equation of how you use money and where you use the limited resources and limited engineers that you have. I think sometimes founders falsely take very mature companies’ strategies and apply them to their nascent companies. They’re like, “Oh, well, Steve Jobs says be completely vertical.”

In 2007, Apple was very different from 1978 and 1982. Those companies were different. They were literally just taking electronics from other manufacturers and putting them in the enclosure.

I’d just be a bit more nuanced in your commercial approach as it informs your technical approach.

swyx

Do you feel differently today? You just joined YC, right? You’ve been building this company in stealth, and now you’re like, “Well, I should probably talk about what I’m doing.” I think a lot of founders are in a similar situation where they want to raise a lot of money to signal that they’re strong, and you raised a lot of money without spending it.

To hire, yeah.

swyx

You obviously like that. Do you think it’s still possible to have a very narrow approach—“We’re kind of building a compounding thing”—without a grand vision right away?

It’s very difficult to answer very general questions that way, so maybe I’ll reframe it as: Is it possible to build a product that has a small, let’s say, problem space and hope that the problem space will grow? Maybe that’s a different way to ask the same question, but more answerable.

I think always, yes. That is the old YC: go really deep rather than very broad and shallow. Very broad and shallow, unfortunately, means there’s just too much tech, especially in hard-tech companies. There are just too many problems, and you can’t do all of them in a very mediocre way. The full product is actually fairly mediocre.

I’m still in the camp of finding a small problem space. The other question you’re asking me is tangential: Should you build in stealth and anonymity? Well, yeah, if you’re a YC CEO.

Yeah, I mean, you can be in stealth. We worked together at Google. We have a long history, and we know them, which is another way of saying we're big networks. Of our first 400 people, the majority were Googlers. The majority of the company came from this giant company we worked at, and that's just very different from a YC founder who doesn't have that experience.

So, just don't take my version of the world, or whatever other founder's version of the world—Jensen's, for example. They are different in time and space and, most importantly, their companies are in a different phase.

swyx

Yeah.

If you want to take inspiration from other really young companies, that's also bad because most of them are going to fail.

swyx

Right.

Qasar Younis

The only solution you really have is to use first-principles thinking and say, based on my skills, my co-founder's skills, the skills of my early team members, and what I'm hearing from customers, what's the product space that I should build? Does it make sense?

swyx

Yeah, it does.

Yeah, I mean, Sam—

Alessio Fanelli

Sam Altman said he'd regret a lot of the advice that he's given at YC. So, I'm always curious to ask founders like you who have been—

Qasar Younis

So, I mean—

Shawn Wang

I mean, everyone who leaves YC does the opposite.

Qasar Younis

Yeah, I accept that. We know each other so well. Sam was president of YC, and we didn't have a CEO, so we worked together extremely closely—would be an understatement—because the firm was also small. YC wasn't as big as OpenAI is.

I directionally agree with that, but I would say that's not more of a YC function; it's more of the market. It is a different world. The AI industry is different—I should say AI companies, more specifically—and how they relate to the other YC companies and the market is just so fundamentally different.

The amount of money raised is different. The number of investors—the sheer number of seed funds—is different. One of our early investors is Floodgate, and they did some analysis in the late 2000s, when there were a single-digit number of funds like Floodgate that were writing sub-$1 million first checks and were not accelerator-incubators.

Anne, who's one of the co-founders there with Mike, said that today—as in 3 or 4 years ago—they tried to do this analysis, and they lost count at around 350 funds or something like that. So, we're just in a different environment. YC advice from 2014 just would not apply in 2026.

Sam is way better at saying these things than I am. He says it in a shorter, more interesting way than I do. I can just give you the literal version: if you ask me, “What are the purposes of a car?” I'll open the owner's manual and say, “Number one, there's a steering wheel,” instead of saying, “It can change your life.”

swyx

Yes, you own them. You have freedom.

Qasar Younis

Yeah, exactly.

swyx

And then, for Peter, I was just kind of curious: are there any particular tech or research problems that you would call out as unsolved but very meaningful for you guys if they were solved? If anyone is working on them, they should get in touch with you.

Peter Ludwig

Yeah, I think generally making models very efficient, right? Because we have to run on actual vehicles, physically. I'm literally taking very large AI and making it very small and very efficient. We're constantly at that boundary of these limitations: you have a great model, but now we need to make it faster and smaller.

So, that in general is a field. I would also say that folks who are really passionate about evaluating this technology—model evals—is a hugely difficult topic, especially in safety-critical systems. We have a really great engineering team and researchers who work on this now, but it's a big area of investment.

So, yeah, folks who are passionate about performance—I guess I'd say model performance, both in terms of capability and literally latency—and then evaluation of models.

Shawn Wang

Awesome. Are there any specific engineering roles that you're hiring for? And especially, who are the people that succeed at your company as engineers? I think that's always the most important thing.

Qasar Younis

fly.co/careers. I think there are literally hundreds of roles. We're looking at all the topics we talked about, from developer tooling and physical AI to operating systems, autonomy, and AI within physical machines.

The types of engineers—that's a great question. That's actually more interesting than the roles, because we're a large enough company where—

Alessio Fanelli

We write everything.

Peter Ludwig

Yeah, we write everything. I think we're a Sunnyvale company, and with this conversation and our backgrounds, you can predict a little bit of what that means. We tend to hire fairly serious people who understand low-level systems, not just a superficial understanding of technology—engineers' engineers, almost.

We definitely hire folks who have diverse skill sets. We hire tons of specialists as well, to be very, very clear. But they've seen production, and I think that really informs how you build technology.

Shawn Wang

Yeah, I think people who really appreciate the hardware-software boundary.

Peter Ludwig

Yeah, exactly.

Shawn Wang

Definitely. In the vibe-coding era, there's a crop of engineers who don't think about hardware at all, and we don't have that luxury. We need people who are a little more passionate about going a little deeper.

Peter Ludwig

Yeah, if you were to contrast us with an AI lab or something, that's where you're going to get the biggest contrast. We're just dealing with reality. What are the other things? All the other classic stuff: you want folks who work hard, who love the technology, and who like a podcast like this.

Shawn Wang

If you made it to this part of the podcast, you probably qualify. You're interested in this.

Peter Ludwig

Yeah.

Shawn Wang

All right. Peter has said that he likes the podcast as well, which is like—

Peter Ludwig

Yeah, yeah, yeah. I'm a fan.

Shawn Wang

Specifically on the hardware-software boundary, that's something I think about in terms of our education system in the States, but also more generally. I feel like there is a retreat from classical computer science or electrical engineering education.

Qasar Younis

Computer engineering, yeah.

Shawn Wang

Yeah. There comes a point where you just do it yourself. At this point, you guys are the world experts on this. You shouldn't wait for some college system to produce them for you.

Qasar Younis

What do you mean in terms of education and upskilling?

Shawn Wang

Yeah, just grab young—

Qasar Younis

General Motors already did it. Literally.

Shawn Wang

They have their own university?

Qasar Younis

Yeah, that's where I went for undergrad. I went to the General Motors Institute.

Shawn Wang

Okay, I did not know that. That did not come out. I saw HBS.

Qasar Younis

Yeah, yeah, yeah. Everyone sees HBS. The Harvard brand allure is high.

Shawn Wang

What was General Motors Institute like?

Qasar Younis

It started 100 years ago to answer this exact question—the question you just asked, which was that there weren't enough engineers in Michigan. You're talking about the early days of the modern corporation. There's a great book, Alfred P. Sloan's My Years with General Motors, which is highly recommended and basically talks about what became the modern corporation.

Part of that is that they were basically bottlenecked on engineers. So, they started a school. Even Google, as recently as probably 10 years ago, was thinking of starting a university internally. There were discussions about it.

We definitely upskill folks as well. The amount of training we do is actually surprising. But it's a luxury you have when you're at our size. When you're at 25 engineers, you just have to survive. Again, take advice that's relevant for your company rather than immediately trying to take high school or—

Shawn Wang

You know, I did go to a class that you taught because it sounds like you can teach a lot.

Qasar Younis

Yeah, well, I think honestly one of the most amazing use cases of these large models now is education. I've taken an engineer with a very good aerospace engineering background, and in a relatively short time span he's doing very confident front-end work and very confident back-end work with the help of these models.

Not only can you do implementation with them, but you can also learn. You ask questions, and you don't feel embarrassed because the model isn't going to call you out on anything.

Shawn Wang

Yeah, I think the thing you probably need more than an engineering degree—though engineering degrees are very important—is an engineering mindset. I don't know if there's a way to shortcut fluid dynamics or heat transfer, the fundamental stuff. At least on the mechanical side, you need an engineering mindset, and sometimes not everybody actually has that. Some people are emotionally drawn toward the arts or something else, and that's completely fine.

Qasar Younis

There's no judgment there. But I think the engineering mindset, maybe in a more usable way, is wanting to understand at a lower level, and the lower level, and the lower—how do photons move?

Shawn Wang

Extreme curiosity.

Qasar Younis

Extreme curiosity. What is light? What is a radio wave? These are really fundamental questions.

Shawn Wang

Right, and if you get curious enough about software, you ultimately end up in hardware, right?

Qasar Younis

And so that's the Alan Kay quote, yeah.

swyx

Yeah, exactly. So I'm trying to make analogies and then do all these things. You're kind of a blend between New General Motors and Tesla's autonomy division for everyone else.

Qasar Younis

We do work in all these other fields. I think if you talk to our trucking customers, they wouldn't even think, per se—you know, in some sense, they're like, “Oh, you guys did some automotive stuff, but you're really helping us.”

swyx

Automotive is not trucking?

Qasar Younis

No, no, I know. It's like a whole—yeah, yeah, it's separate. There are different problems. I guess you have the general categories of on-road and off-road; I think that's what you're thinking. So there's on-road and off-road, but within on-road, there are all these subclasses of machines, especially when you talk about a delivery robot that doesn't have a human in it.

That's actually very different because now you're not concerned with the actual feeling that you have when you're in a self-driving car, because you don't have to account for that. You can brake hard, and you don't care about jerk, and all of these metrics don't become as important.

The way I think about it, honestly, is a little bit like any system that you, as a human, would need special training to operate—you can think of it a little bit differently. So the license to operate a truck is different from the license to operate a car. It's different from the license to fly a plane. It's different from—you get it, right?

swyx

Awesome, guys. Thank you for taking the time.

Peter Ludwig

Yeah, thanks for having us.

150亿美元的Physical AI公司:仿真、Autonomy OS、Neural Sim与1000名工程师——Applied Intuition — 文字稿与摘要 | BidClub