R1、OpenAI 的 o3 与 ARC-AGI 基准：Mike Knoop 做客 Gradient Dissent 播客时的洞见
可投资的变化，是从扩展预训练模型的记忆能力，转向把推理算力用于重组知识，而 ARC-AGI v1 暴露了两者之间的断点。 GPT-4 级系统得分约4%，o3 达到75%，高算力昂贵配置达到85%；2名 STEM 毕业生合计接近98%–99%。Mike Knoop 认为，这个陡峭转折证明系统获得了真正全新的“适应新颖事物的能力”，而不是又一次渐进式基准提升。
DeepSeek R1-Zero 的科学意义高于 R1，因为它在没有人工标注推理轨迹的情况下，单独展示了通过强化学习和符号验证实现推理的路径。 它的 DeepSeek-V3 基础模型仍然使用人类数据训练，这是 Lukas Biewald 强行带入讨论的限定条件；但 Knoop 认为，这可能打开一条从原始算子走向自生成知识的路径。2025年尚未解决的问题是：纯 RL 能否在没有监督微调的情况下达到 o3 级别的适应能力。
“DeepSeek 让推理便宜了几个数量级”这一标题，目前还不是干净的成本比较。 Knoop 尚未深入审计训练数据，并强调 OpenAI 商业化 o1 的价格包含利润和未来研发投入，而开放模型可以自行部署，成本更接近真实成本。部分比较还遗漏了基础模型训练成本；他的谨慎判断是，R1/R1-Zero 与 o1 的真实推理成本可能处于“相近的区间”。
企业级 Agent 只有在推理能力把一致性提升到足以让用户放弃监督时，才具有真正的经济意义，而不只是因为 token 变便宜。 GPT-3.5、GPT-4 和 GPT-4o 都没能消除随机错误，Zapier 客户反复表示：“我还不够信任它，无法放手不管。”Knoop 预计，o1、o3 和 R1 级别的规划能力，将解锁用户已经愿意按人力成本付费的工作负载。
ARC Prize 把一个原本晦涩、针对“预训练足够论”的研究异议，转化为全行业的协调节点。 2024年6月前，AI 研究者中了解 ARC 的比例或许只有10%–20%；到年底，科技行业可能已经无人不知。排行榜后期跃升、协同发表的论文，以及数千个要求测试 o1 的请求，都暴露了此前被隐藏的需求。据称已有7或8家初创公司改变研究方向，或转向 ARC。
Knoop 与 François Chollet 创办的 Ndea，是一场高集中度的反向押注：程序合成大约提供了高效 AGI 所需的一半想法。 深度学习提供快速的近似草图；程序合成则从少量样本中搜索精确、能够分布外泛化的规则。Knoop 认为，程序合成至少占整个方程的50%，即便它不占50%的算力；而这个领域仍然没有自己的“程序合成版 transformer”。
AGI 时间表仍由不可预测的阶跃式进展和一个尚未解决的效率分母决定。 Knoop 猜测 ARC-AGI v2 可能维持12–18个月，ARC-AGI v3 的设计目标是维持3年。人类觉得容易、AI 却觉得困难的差距，至少还要3–4年才可能在人类级效率上消失。这些都明确属于“大胆猜测”：ARC-AGI v1 在接近5年时间里几乎停留在零附近，却在2个月内从4%升至85%。
1. ARC-AGI 暴露能力断点，而非更平滑的扩展曲线
Knoop 将 GPT-3、GPT-3.5、GPT-4 和 GPT-4o，与更新的 o 系列和 R 系列区分开来。前者主要通过扩大预训练规模、增加人类数据和一定程度的蒸馏来提升能力；后者则加入了推理时算力。他称之为“根本性的范式转变”。
预训练 transformer 确实能够泛化，因为压缩并不等同于字面意义上的数据库查询。Knoop 更具体的判断是，它们的泛化程度受到架构限制：更多训练可以增加知识，但不会按比例扩大系统超越既有知识进行适应的范围。
Biewald 对 AlphaGo 的反驳值得保留：一个能够发现新围棋着法的系统，当然算是在推理。Knoop 承认那是领域特定的智能，但认为它的通用性来自研究人员对领域的建模。人类真正独特的优势，是能在几小时内学会一款新纸牌游戏，或在几天内学会驾驶一辆新车。
ARC-AGI v1 自2019年推出后，5年间基本无人攻克：GPT-4 级系统得分约4%。随后 o1、o1 Pro 和 o3 推动成绩近乎直线上升至75%–85%。“当你看到一个陡峭转折时，就说明某些东西发生了明显变化。”
2. 当搜索叠加在思维链之上，思维链才变得强大
2022年1月发表的思维链论文，是 Knoop 所谓的“糟了时刻”。要求模型“把思考过程说出来”，让当时的推理基准从约30%升至70%，也让他认真考虑这项技术是否可能走上 AGI 道路。
最简单的思维链形式，是反复要求模型给出下一步，累积整条轨迹，再以这条序列为最终答案提供依据。o1、R1 和 R1-Zero 大体遵循这一模式，但单链版本在 ARC 上只能达到约15%——明显高于 GPT-4 级系统4%–5%的区间，却仍然偏弱。
Knoop 基于信息做出的推测是，o1 Pro 和 o3 增加了采样与搜索：先生成多个可能的思考步骤，选出一个，再进入下一步。这类似于推理时的程序搜索，在他的解释中，这正是成绩跃升至75%、高算力配置达到85%的原因。
Biewald 对这一机制之惊人之处停顿了一下：被认为受限于压缩能力的模型，为什么通过分阶段生成更多文本，就能以某种方式获得推理能力？Knoop 也承认这并不直观；正是这一经验性跃升，让他认为这些系统“值得认真研究”。
3. ARC 将进展定义为消除“人类觉得容易、AI 却觉得困难”的任务
一道 ARC 任务会给出彩色二维网格、若干输入输出示例和一个测试案例。解题者必须推断出一条一致规则，将对称、旋转、物体检测与追踪、基础物理等先验重新组合，用于处理从未见过的配置。
这些谜题在人类看来往往近乎简单，这正是基准的力量所在。现有的人类证据显示，2名 STEM 毕业生合计可以解出 ARC-AGI v1 约98%–99%的题目；Knoop 说，聪明的人类实际上可以达到100%，但 v1 缺乏足够严谨的人类基线。
ARC-AGI v2 计划加入更强的研究，以证明每一道任务都能由人类解决。其设计原则始终是“对人类容易、对 AI 困难”，目标是将这道鸿沟压到零，而不是不断升级到人类也极难解决的问题。
Biewald 追问，经历了这么多所谓 AGI 里程碑之后，ARC 是否真的可能成为最终基准。Knoop 不会保证这个标签，但他说，如果不存在任何对人类容易、对计算机仍然困难的任务，那么否认 AGI 就会变得困难。“这由世界来决定。”
4. R1-Zero 暴露纯 RL 押注及其关键限定条件
R1-Zero 和 R1 都在训练并生成思维链，但 R1-Zero 的后训练使用强化学习，不依赖人工标注的推理示范。符号验证器可以运行代码、检查方程、测试编译结果或比对精确答案，然后返回可靠的好坏信号。
Biewald 抓住了关键限定：R1-Zero 并不是从零开始学习语言。它从 DeepSeek-V3 出发，而后者使用了人类数据训练。Knoop 认同“这不是一个已经注定的结论”，并将未来的工程选择概括为：多少知识应当放进基础模型，多少知识应当由系统在运行时生成。
更具推测性的上行空间，是递归式知识生成：推理器可能生成新知识，将其加入不断扩大的语料库，再为后续训练提供种子。极端情况下，一个系统可以只接收基础算术算子，然后通过 RL、在“没有人类参与”的情况下，尝试自行构建微积分。
证据仍不完整。Knoop 说，o3 要在 ARC 上取得强劲表现，需要通过人类监督微调来达到足够高的效率；目前还没有公开结果证明，纯 RL 训练的语言模型推理器能够达到同等水平。他预计，这将是2025年的重要研究方向。
5. DeepSeek 的价格并不能证明其成本优势达到数量级
对于 DeepSeek 备受讨论的经济性，Knoop 给出了一个“诚实的非答案”：“我还没有深入看过。”他区分了商业定价与底层成本，因为开源 R1 可以在本地基础设施上运行，而 OpenAI 必须把利润、研究人员和未来研发投入计入价格。
他的判断性推测是，o1 的商业推理价格包含了相当可观的利润，真实成本可能比用户想象的更接近。他会把 R 系列和 o 系列推理系统放在大致相同的成本区间，同时强调自己无法核实精确比较。
训练成本标题又制造了一个分母问题。被引用的 R1 或 R1-Zero 后训练成本，可能遗漏了创建 DeepSeek-V3 所需的投入；但 Knoop 声称自己并不知道这个数字，也指出他同样不知道 GPT-4 的完整成本。
6. Zapier 的资本纪律塑造了 Knoop 对技术押注的方式
Zapier 让非技术用户连接 Gmail、Slack 和 Salesforce 等产品，在无需工程师介入的情况下，通过自动化工作流传输数据并执行决策。Knoop 估计，它服务于美国300万–400万家企业，同时拥有规模可观的国际客户群。
3名来自美国中西部的创始人最初认为，一家公司应该销售有用的产品，并用收入继续投入，因为2010年的美国中西部确实不存在真正的风险投资市场。在 Y Combinator，Sam Altman 的约束问题揭示了瓶颈：Knoop、Wade Foster 和 Bryan Helmig 每天忙于客户支持直到中午，而不是改进产品。
他们筹集了约100万美元，迅速招聘支持人员，但收入在新员工拿到第一笔工资前就追了上来。Knoop 开玩笑说，最初那笔资本可能“沿着血脉一路追溯”到现在；它真正的价值，是提供了全职投入的启动能量。
Zapier 从2011年起也一直采用全球远程办公。当时 Knoop 只能找出 Automattic 和 37signals 作为可比案例。这段运营经历对他的 AI 叙事很重要：它让他贴近真实工作流，也让他看到，决定产品能否被采用的是可靠性，而不是基准测试的轰动效果。
7. 早期 Agent 部署暴露了预训练无法消除的上限
2022年思维链结果出来后，Knoop 让 Foster 接回自己的全部职责，负责产品和工程，这样他就能专门研究 AI。在6–12个月里，他与 Zapier 的 CTO 持续写代码，开发思维树原型，并在 ChatGPT 发布前几个月做出了内部版 ChatGPT。
工具调用成为 Zapier 最自然的切入口：冻结权重的语言模型可以访问平台的操作和搜索接口，从而影响业务软件，而不只是返回文本。因此，Zapier 在采访前约2年就开始部署类似 Agent 的系统。
数百次客户交流都给出了相同的反馈，覆盖 GPT-3.5、GPT-4 和 GPT-4o：“承诺确实存在”，但“我还不够信任它，无法放手不管”。自动化是在服务器上无人值守运行的，偶发错误不像聊天回复那样可以随手审计。
这种始终不变的反馈，与2023–24年围绕扩展的热情发生了冲突。François Chollet 2019年的论文《On the Measure of Intelligence》，为 Knoop 提供了一个既保留预训练成果、又解释其局限性的框架。他最终认为，ARC 是一个异常具体的证据，表明单纯扩展记忆能力无法解决适应问题。
8. 一个误发 Slack 频道的错误，体现了可靠性的商业 stakes
Zapier 早期的一个 Agent 需要选择 Slack 频道，并根据 HubSpot 信息和销售转录内容撰写潜客信息。尽管提示中指定了测试频道，它有时仍会选中一个共享的 OpenAI 合作频道，因为该频道名称中也包含“testing”，并将客户信息发送过去。
团队立即停用并清理了这条工作流。Knoop 的反应是：“把这东西关掉。离我10英尺远。我受不了了。它正在把我的生意置于危险之中。”一次随机错误，就可能让用户的业务和一项重要合作关系陷入风险。
Zapier 的应对方式是设置硬性保证：用户默认选择目标频道，也可以允许 Agent 自由猜测，或者将其限制在3个获批频道之内。这种架构承认，在底层规划能力可靠之前，模型自主性必须与确定性控制并存。
Knoop 对 ARC 式适应能力的实际转译是“一致性”：一个更可靠地抵达同一可控答案的系统，会更容易被引导。人们可能愿意为这种可靠性支付最高相当于人工劳动的价格，因此 o1、o3 和 R1 即使在推理成本尚未下降之前，也可能解锁 Agent 应用。
9. ARC Prize 将被忽视的基准变成了研究市场
在 ARC Prize 出现之前，Knoop 接触的研究者中或许只有10%–20%了解相关 ARC 基准；很多人把它与另一个已经被语言模型攻克的旧版 Allen Institute 基准混淆。对他而言，ARC 是反驳“扩展预训练就足够”的直接、具体证据。
Biewald 将他介绍给 Chollet 后，Knoop 飞往西雅图，提出提高 ARC 关注度的想法，同时追问为什么知名度如此低、Chollet 为什么没有投入更多精力。Chollet 对这些问题给出了很好的答案；Knoop 在看到 Nat Friedman 和 Daniel Gross 如何为 Vesuvius Challenge 聚拢关注后，继续推进 ARC Prize 的构想。
2024年的比赛要求获奖者开源进展，从而重置社区基线。截止前一周，领先团队以接近55%的成绩整个夏季占据主导，却一度犹豫是否公开成果。72小时内，2支约排名第10的队伍冲了上来，另一支挑战者紧随领先者之后，3篇论文几乎同时出现。
o1 发布后，需求再次涌现：数千人要求 ARC Prize 对其进行测试。到2024年底，Knoop 认为科技行业可能已经人人知道 ARC；更出乎意料的是，有7或8家 AI 初创公司告诉他，它们已经转向或重新定向研究，投入这一基准。
10. 竞赛暴露了2条可信的测试时适应路径
经典 ARC 求解器使用人类设计的领域特定语言，其中包含小型变换函数，再暴力枚举组合，直到某个组合与示例匹配。它们的通用性存在于研究人员的设计选择中，而组合搜索的效率极其糟糕。2024年 ARC 比赛中，更新的归纳和测试时训练方法达到了约40%–50%。
Ryan Greenblatt 早期的程序归纳方法，是让语言模型根据谜题生成 Python 程序，在已知示例上运行这些程序，再选出输出匹配的程序。它达到了约40%，证明了这一方向的潜力，尽管它可能需要生成数亿个程序。
测试时训练提供了另一条主要路径。团队通过改变颜色或镜像网格来扩充每个私有任务，但不改变其语义规则；随后生成数千或数万个相邻示例，在本地微调模型，再进行推理；一个领先系统达到了约50%。
因此，Knoop 看到2类有证据支撑的适应路径：他认为 o1 Pro 和 o3 使用的思维链搜索，以及测试时训练——先处理当前情境，生成相邻数据，再将其反馈给模型。ARC 本质上不是视觉测试；它的网格只是以视觉形式呈现的二维程序合成问题，用于调动人类直觉。
11. Ndea 押注高效 AGI 需要直觉与精确搜索的融合
程序合成的目标，是寻找一个能将少数输入示例映射到输出的程序。深度学习需要大量数据，擅长快速给出近似的分布内判断；正确合成的规则则可以泛化到任意输入，但搜索所有可能程序的成本会呈指数增长。
Ndea 提出的融合方式模拟人类解题：深度学习快速勾勒出几条可能规则；符号执行对其进行精确检查；失败结果再把搜索送回直觉模块，以生成更好的候选。人类不会枚举数千个 Python 程序，高效的人工系统也不应如此。
Knoop 将 o3 的推理链理解为一种自然语言程序，将其采样理解为程序重组。他认为，大多数前沿实验室仍把这看作带有辅助搜索的深度学习；Ndea 则将程序合成视为“整个方程中至少50%的部分”，即使它不占50%的算力。
他与 Chollet 认为，o3 在 ARC 上的结果可能是类似 AlexNet 的起点，开启一段5–10年的扩展旅程。程序合成领域远小于深度学习，至今仍没有自己的“transformer”。因此，Ndea 是一家研究实验室，近期没有产品计划，而 AGI 是第一步。
更长期的目标是自主科学创新，而不只是解决今天的工作流。Knoop 把 AGI 比作印刷术：15世纪的人可以预见知识会更广泛地传播，却无法想象 Wikipedia、在 Wikipedia 上训练的 AI，或人们可以与之交谈的机器。真正吸引人的地方，“几乎更像是一场冒险，而不是一道待解决的问题”。
效率仍然是 AGI 定义的一部分，因为现在可以通过增加推理算力购买更高准确率。Knoop 暂时倾向于使用美元来比较机器工作与人类劳动：FLOPs 无法直接转换，而并行计算也会扭曲实际耗时。
他明确表示不确定的预测是，ARC-AGI v2 将持续12–18个月，而旨在测试不同能力的 ARC-AGI v3，目标是持续约3年。他预计，人类觉得容易、AI 却觉得困难的差距，至少还要3–4年才会在人类级效率上消失，但也警告阶跃式进展无法用平滑曲线外推：“你不知道那些想法是否已经存在。”
You’re listening to Gradient Descent, a show about making machine learning work in the real world, and I’m your host, Lukas Biewald. This is a conversation with Mike Knoop, who is both an AI researcher and an incredibly successful entrepreneur. He started a company called Zapier about 15 years ago with a very small amount of funding and grew it into a very large business. He then got up to speed on the frontier of AI research and recently started an organization called Ndea, which is one of the new research labs working on the forefront of AI. This is a really interesting conversation. We go into business and how AI fits into business, and then also some of the details on how the new R1 and R1-Zero models work, especially on the ARC-AGI Prize, which he funded and made popular. I really hope you enjoy this conversation.
Why don’t we start with R1 and work backward?
Sounds good. Basically, 2 actual models were released: R1-Zero and R1. They’re very similar in nature to what I would call OpenAI’s o1 model. They’re reasoning models, and they were trained in a similar way. We know for sure how R1-Zero and R1 were trained because DeepSeek chose to open-source its training methodology.
For R1-Zero and R1, they really did move the science forward. I saw a public comment from Mark Chen, who leads research at OpenAI, where he shared that he thinks o1 was trained in a similar spirit as well. I think there’s pretty good agreement between the ideas that went into creating both of these systems.
Fundamentally, the o-series from OpenAI and the R-series from DeepSeek represent a paradigm shift from the types of AI systems we’ve seen in the past. For example, OpenAI’s GPT series—from GPT-3 to GPT-3.5, GPT-4, and GPT-4o—has all followed the same broad paradigm of scaling up pretraining. We’re trying to make the models more intelligent by feeding them more data and making the models bigger.
It’s also the case that o1 was probably a little smaller. There was probably some distillation from GPT-4 to make it more efficient, but roughly, we’re still in the broad paradigm of making models smarter by giving them more human data. They’re effectively memorizing answers, which means they really have no ability to adapt to novelty.
1. Building Zapier: The early days and scaling automation
What ARC-AGI does is assess an AI system’s ability to solve problems that it hasn’t seen before. The data is highly resistant to simply memorizing the answers. Even if you’re given the training set for ARC, you can’t just memorize the training set and solve the test set.
This is another underappreciated point: ARC-AGI is legitimately very hard. That’s why version 1 went unbeaten for about 5 years. In December, we had this big news moment with OpenAI’s o3. They had a 75% score on ARC-AGI v1, and an 85% score with a really expensive, high-compute performance version of it.
This showed that this new type of reasoning system has a fundamental capability we haven’t had in computers before: the ability to adapt to novelty. That actually has implications beyond this quirky fact. It will lead to more robust and reliable AI systems, too, which we can touch on. It matters a lot for agents.
Before we go down that path, let me take us immediately off track, because I love it. When you talk about novelty, how do you define that? There’s a lot of debate about whether these systems are just memorizing or not. They’re obviously not simply memorizing text and regurgitating exactly the same text. It’s quite clear that there’s some adaptation.
There’s some generalization that comes from compression. It’s not literally a database, which would be a lookup table.
Exactly. If you had 1 parameter for every fact, you could just have a database.
These GPT-style systems do compression, and that’s where you get some of the interesting generalization. The claim I would make is that the amount of generalization they’re capable of doing is fixed because the architecture is effectively fixed.
You’ve got the Transformer underlying architecture in every language model up until last year. That means your level of intelligence is fixed. You can memorize more, but the amount of generalization you have—the amount of adaptability you can apply from your training data to a new situation—has been fixed. That didn’t change until o1.
o1 was the first prototype we saw last September of a system that legitimately had an increased amount of intelligence in its ability to be given a fixed amount of input information and do more things accurately farther away from its training data.
Would you say, for example, that AlphaGo couldn’t generalize? It seems like, within the domain of Go, it could reason pretty well and adapt to novel situations.
No. The same is true for ARC. ARC-AGI has been around for 5 years, and there are a lot of state-of-the-art systems on ARC. From a pure solver standpoint, disregarding the front-end language models, the best systems were getting around 50% in the 2024 ARC Prize contest.
But those are very domain-specific solvers. All of their generality is being put into the system that solves ARC during the contest. That comes from the researcher’s brain. The researcher thinks about the problem, models it, and says, “Here’s what I’m trying to get this computer to do. I’m going to encode my understanding of the problem into the system and the architecture in order to get this thing to work.”
2. How AI is transforming business automation
It ends up being fairly constrained in domain, and it doesn’t generalize well. The same is true for AlphaGo. We’ve had AI systems for years that can be superhuman at games, but the fact has historically remained that I could sit you down, teach you a new card game or board game in a couple of hours, and get you up to human-level proficiency.
I could take you into a totally new domain and teach you how to drive a car you’ve never driven before. You’d probably be proficient in a couple of days. That ability to adapt on the fly to a situation or type of problem that you’ve never seen before and never trained on has historically been very unique and special to humans relative to the AI systems we’ve had.
It’s probably worth saying that I don’t know how ARC is best described in words. It’s better to look at it. We’ll overlay a puzzle here somewhere.
Totally. It’s definitely better to overlay a puzzle. For someone just listening, what’s astonishing about ARC is how easy the puzzles seem to be, and how these systems fail on these easy-looking puzzles.
Maybe you could describe a little more of what it looks like.
It looks like an IQ test. It’s a grid of colors, a 2D grid. You’re given some examples of inputs and outputs, and your goal is to find the rule or pattern between them—the common, consistent rule between those inputs and outputs—and then apply it to a test case.
It’s easier to describe visually than verbally. ARC challenges you to recompose, on the fly, knowledge that you’ve acquired throughout your life—what we call core-knowledge priors. These include symmetry and rotation, object detection and tracking, and basic understandings of physics.
ARC requires you to abstract and compose those core-knowledge priors on the fly for a task that you’ve never seen before. This is what the historical language-model scaling paradigm was never very good at. GPT-4, for example, scored around 4% on the ARC dataset, in contrast to the impressive performance we started seeing from the o-series systems.
What changed from 4% to these o-series systems that are doing well? What do they do differently to make it work better?
Fundamentally, the strongest thing I can say is that there’s a very strong capability assertion, and then I have some informed speculation about how that capability was achieved.
You can see it in the score. If you look at ARC-AGI v1, over a 5-year period—from when it was introduced in 2019 through last fall—the best GPT-4-class systems got around 4%. Then o1 came out, o1 Pro came out, o3 came out, and the score rapidly went from 4% all the way up to 75% and 85% on the extreme high-end performance version of o3.
It looks like a straight line. It’s pretty remarkable. That’s actually a good thing to see in a benchmark because it means there’s more signal in the benchmark. It’s truly making a capability assertion. It’s harder to understand capabilities by looking at benchmark scores that monotonically increase smoothly over time. When you see a sharp bend, something has distinctly changed.
3. The ARC Prize: Why AI reasoning is the next frontier
I think that’s the case with these reasoning systems. Specifically, they added the ability to recompose knowledge that they had been trained on in their foundation models at test time. OpenAI calls this test-time compute. The broad paradigm is that we want to use more compute at test time to think before jumping to an answer.
GPT-4 starts spitting out tokens in about 500 milliseconds, and that’s its answer. The intuition here is that we want to allow these systems more time to think up front before they render an answer. They think using what’s called a chain of thought.
You might have heard the prompt, “Let’s think step by step.” That came from a paper published about 3 years ago. The paper was very special to me and was one of the things that first got me all-in on AI. This chain-of-thought moment happened in January 2022, and we’re still downstream of it, trying to figure out how to apply the paradigm of having models think out loud.
You say, “What’s the next step to solve this problem?” The model takes that step, then you ask, “What’s the next step?” It does that again, in a big chain. The model can use that entire thought trajectory to ground its final answer. That’s what o1 does, what R1 does, and what R1-Zero does.
Before we get there, maybe let’s take a moment to be astonished that this works. You’re saying these language models are compression algorithms that can’t really reason; they can only do limited generalization. Then they start generating text, and the chain of thought has them generating text in steps. Suddenly you claim they can reason. That doesn’t seem obvious.
I don’t think it is obvious. It’s one of the reasons I went all-in on AI back in 2022. Up until January 2022, I had founded Zapier 15 years earlier, and I was an executive at the time, running about half the company and building our new product initiatives. I had been paying attention to AI, and I thought I had a good perspective on what language models could and couldn’t do.
Then the chain-of-thought paper came out. Just by asking the models to think out loud, you saw performance scores on reasoning benchmarks spike. They grew from around 30% to around 70%. That was my “Oh, shoot” moment: are we on track for AGI with this technology?
I felt it was important to know this from a Zapier standpoint—should we start using this in our products?—but also as a human. This is some of the most important technology in the history of the world, and I wanted to understand the paradigm.
One interesting thing about that old-school version of chain of thought is that at Zapier, we were proudly using chain of thought in the fall of 2022 with a very similar paradigm. You ask the model to think out loud, and it has a very low degree of ability to adapt.
4. The shift from deep learning to reasoning models
We see this even with o1 and R1. The scores on ARC are only around 15%. That’s a big step up from the GPT-4-class systems in the 4% to 5% range, but it’s still a relatively weak amount of adaptation.
What has made these systems work with o1 Pro and o3—and this is informed speculation—is that they’ve added search and sampling on top of the chain-of-thought generation process. They’re not just asking for a single chain of thought and then saying, “Now give me the answer.” They generate multiple chains of thought in parallel, ask which one is best, use that one, and then proceed to the next step.
They’re effectively doing program synthesis or program search at inference time. That insight has enabled a significantly higher degree of adaptation, which is what gets them to 75% or 85%.
This is astonishing. These systems demand serious study, and that’s why I’m excited to see R1 and R1-Zero open-source. It will allow more people to do the science.
Before we get into R1 and R1-Zero, what is human performance on ARC? Where are the best models today? When you say 85%, how does that compare with human-level performance?
Smart humans can get basically 100% on ARC-AGI v1. The data we have is that if you take 2 STEM graduate students and put them in front of the benchmark, they’ll get 100%. The actual data is around 98% or 99% across 2 people.
That’s been one of the flaws of v1: we haven’t had strong assertions of human capability. That’s something we’re fixing in v2. We’ve been working on v2 for years. We put a lot of effort into building it last summer, and we’re putting the final touches on it now. We’re going to launch it with the 2025 ARC Prize.
One of the things we have with v2 is strong human-study baseline testing. We want to confidently assert that every puzzle in the benchmark is solvable by humans, which justifies the “easy for humans” claim.
I think that’s the guiding principle for ARC in the future. This is true for v1, and it will be true for v2, v3, and all future versions: ARC should represent things that are easy for humans and hard for AI. The gap we’re driving toward is zero.
If you could get that gap to zero, I think it would be hard for anyone to claim we don’t have AGI. If you can’t find a single thing that humans find easy but computers find hard, that’s a reasonable goal to set and a target to shoot for.
This contrasts with how a lot of other frontier benchmarks work. Dan Hendrycks’s Humanity’s Last Exam and Epoch AI’s FrontierMath are making ever more difficult benchmarks. I don’t discount that effort at all; I think it’s useful. But I think there’s something more important to understand from a capability-assertion standpoint: what do humans still find easy? That’s more revealing of what we’re still missing in these systems.
I totally agree. It’s funny, though. Throughout my lifetime, and especially in the last year or 2, there’s been this line that a benchmark represents human-level performance, and that once we get there, we’ll have AGI. Then, over and over and over, the systems beat the benchmarks.
It’s hard to make this claim with certainty, but do you feel like ARC could be the last benchmark of this genre? Is there something else out there that AI could solve, leaving some other benchmark that humans can do easily but AI can’t?
What I can say more confidently is that we’re going to drive the gap between easy for humans and hard for AI to zero. Whether that constitutes AGI is up to the world to decide. I personally think it does.
That’s the design philosophy we have for v2, and it will be the design philosophy for v3 and future versions. I’ll say that v2 will probably look fairly similar to v1 in terms of domain because we’ve been working on it for a long time. It will be harder for computers but still easy for humans.
5. How OpenAI’s O-series and DeepSeek are redefining AI capabilities
V3 will likely look very different from what we’ve done so far. I think it will still look like ARC, but it will test different capabilities that humans find quite easy and that current benchmarks don’t test. It will also likely include a formal way of measuring efficiency, which I think will be important for determining when we’ve reached AGI.
Can you describe what R1 and R1-Zero are and why they do well on ARC?
R1-Zero and R1 are trained on chains of thought and then generate them. They basically generate a single chain of thought and then give us an answer.
The important thing to understand is why there are 2 models and why R1-Zero and R1 are separate. DeepSeek didn’t have to release R1-Zero. In my view, R1-Zero is the more important system for understanding R1.
What does the Zero stand for?
The Zero stands for no human data in the training loop, like AlphaZero. They’re training it purely with reinforcement learning, using domains such as math and coding to create verifiers. DeepSeek-V3 generates a chain of thought, and then they can use a formal domain—literally running a computer program—to provide feedback.
Before we go down that path, they obviously had an earlier training step in which there was human data, right? How does it learn language otherwise?
There’s a foundation model, DeepSeek-V3, and it was trained on human data. But this isn’t a foregone conclusion. I think it’s going to be an engineering tradeoff that AI system developers have to make in the future: how much knowledge do we put in the foundation model we start from, and how much do we have the system generate itself at runtime?
My expectation is that you’ll use these reasoning-style systems to generate new knowledge and add it back into an ever-growing corpus of knowledge. You’ll probably use that corpus as the starting point for future training runs and inference systems.
So yes, R1-Zero was trained on human data through DeepSeek-V3. But that’s probably not going to remain the permanent case for these systems. If R1-Zero can bootstrap itself up in knowledge from first principles—math, for example—give it the basic arithmetic operators and let it bootstrap up calculus, that’s something a system like R1-Zero could potentially do using reinforcement learning with no humans in the loop.
They train a foundation model on human data. What’s the next step they take with R1-Zero specifically?
They generate a single chain of thought, step by step, and then use the entire chain of thought to ground a final response from the same DeepSeek-V3 model.
How is the feedback incorporated? Is it done offline at training time?
What I just described is test time. A user inputs a query, the model generates one very long chain of thought, and then it gives you a final answer.
The feedback loops happen offline at training time. The developers did that months ago. They took the DeepSeek-V3 model and had it generate lots of chains of thought. Then they had another computer program look at those chains and provide the training-loop feedback: was that a good thought or a bad thought?
That reinforcement-learning signal was used to fine-tune the ultimate R1-Zero model. In the case of R1, they also allowed human experts to label the chains of thought.
How do they have the original model that knows whether a thought is good or bad? How do you get started?
That’s what differs between R1-Zero and o1. o1 uses a grader model—a separate neural model that provides feedback and says things like “good job” or “bad job.”
R1 doesn’t do that. It uses a purely symbolic verifier. It takes the potential code the model has output, tries to run it symbolically, and uses that to provide a feedback signal about whether it’s good or bad.
That’s generally what you’re seeing in the reinforcement-learning space. You want to get a final answer, then symbolically verify it and make a 100% reliable assertion about whether it was right or wrong. You can do that in domains where you can run a computer program to verify the result: a Python program, a math equation, or a piece of code. Does the code compile? Does it produce the answer you expect?
This differs from some of the o1 work, where they’re using a process reward model—a separate, neural, trained model. Again, this is all informed speculation. DeepSeek hasn’t shared all of these details, so I’m trying to understand the systems based on what they’ve released.
My understanding is that o1 has a separate language model providing feedback during training, whereas R1 uses a symbolic verifier.
6. AI generalization vs. memorization: What’s changing?
Is it possible that there are domains, such as computer programming, where it’s easy to do chain of thought and verify the result, but other domains where verification is harder? Would that cause these models to specialize in domains where it’s easier to tell whether the result is accurate?
That’s the bet. Can R1-Zero scale up without adding humans to the loop? The evidence we have is that o3, which does much better on ARC, required supervised fine-tuning from humans to work at a high enough degree of efficiency to make it tractable.
We don’t yet have evidence that you can bootstrap a purely reinforcement-learning-based language-model chain-of-thought system to get there. That’s probably what you’ll see happen in 2025. I would expect that to be a major area of work.
Another obvious thing people have noticed is that these models were much cheaper to train—by orders of magnitude. Do you have a thought about why that is?
I haven’t looked deeply into this. The only commentary I can offer is that people are often comparing the sticker price—the commercial pricing rate of o1 versus DeepSeek’s commercial rate for hosting its model.
Because R1 is open-source, you can run it on your own infrastructure. You don’t have to pay the commercial rate. DeepSeek is giving you something that’s close to the actual cost, whereas OpenAI has a margin. OpenAI has researchers to support and future research to invest in. They’re building a business.
My informed speculation is that there’s quite a bit of margin built into o1’s inference costs right now in order to fund future R&D. The true costs are probably more comparable than most people expect, at least for R1 and R1-Zero versus o1.
There’s also a debate about how much DeepSeek-V3 cost to train. People see the headline that it cost $5 million to train R1-Zero and assume that number obscures the fact that a lot of money went into training the foundation model. That’s accurate. I have no idea how much it cost to train the foundation model, and I don’t know what GPT-4’s chain-of-thought cost was either.
I’d make this single comment about inference costs: I’d broadly put the R systems and the o systems in a similar bucket from a cost basis.
Let’s take a step back. I first met you more than a decade ago. You’re the founder of a company called Zapier, which I’ve long admired. Some people may not have heard of it. Do you want to talk about what Zapier does?
Zapier is an automation company. We try to deliver automation software that’s very easy for nontechnical people to use, primarily to automate parts of their businesses.
We’re used predominantly by individuals, people automating their own lives, small teams, and individual teams within large organizations. It’s designed to be very easy to use, in contrast with the automation systems that historically existed.
We’re used today by 3 or 4 million businesses in the United States, and we have a substantial international customer base as well. The core idea is that you can connect apps and use them together. Businesses use software such as Gmail, Slack, Salesforce, or Google Workspace, and they build internal workflows around those processes.
They often have humans who have to shepherd data between different applications and make decisions. Zapier can automate that completely, and it’s easy enough for line-of-business users that you don’t have to involve an engineer.
7. The role of program synthesis in AI’s future
One notable thing about Zapier is that you raised a tiny amount of money from venture capitalists, then got off the VC train. Almost nobody does that, and you grew to incredible scale. That’s been an amazing return for your initial investors. You were even unusual for raising money in the first place.
We debated whether to raise money in the first place. The 3 founders—me, Bryan Helmig, and Wade Foster—were all from the Midwest. I grew up in St. Louis, and there really wasn’t a venture-capital market in the Midwest in 2010.
The question was, how do you build the business? You build a useful service or product, sell it, and use the proceeds to invest back into the business. That was our basic approach.
We went through Y Combinator, and we were debating whether to raise money. Everyone else was going to raise money after Demo Day. We had office hours with Sam Altman, and we asked, “Should we raise or not?”
The question he asked us was, “What’s the constraint on the business?” It was a good question because the real constraint at Zapier was that Bryan and I were waking up and doing support until noon every day. That took time away from making the product better and reducing the amount of support we needed in the first place.
We realized we should hire a support person, but we didn’t have enough cash on hand to do that. So we decided to raise a small round to hire someone as soon as possible and give ourselves more product time.
The funny, or sad, anecdote is that by the time we raised the round, got the money in the bank, found the person, hired them, started payroll, and they received their first paycheck, revenue had caught up enough that we could have paid them directly.
I’m pretty sure you could trace the million dollars we raised through its lineage all the way to today. It did allow us to get started, though, and I think that was honestly the biggest value we got out of Y Combinator: the activation moment to go full-time on the company.
You were also remote-first from the beginning, right?
Yes. After Y Combinator, we were a globally remote team and have been since 2011. The only other companies at the time that I knew of that were fully remote were Automattic, the company behind WordPress, and 37signals.
The third interesting thing about Zapier, from my perspective, is that you were very early to use language models. You were one of the first companies I saw with real LLM use cases. Can you describe what those were and what that experience was like?
This goes back to the chain-of-thought paper that came out in January 2022. My background is engineering. I studied mechanical engineering and did optimization research, which turns out to use essentially the same math as a lot of this machine-learning work.
I didn’t figure that out until 2017, but as soon as I did, I thought, “I know how this works.” It demystified the technology for me, and I started paying more attention to the research side.
Zapier was growing, though, and I had other priorities. We had to grow the company, introduce new products, and so on, so I wasn’t paying that much attention. I read the GPT-2 paper—or the release—played with it, read the GPT-3 paper, and gave a presentation to the company about it. Then I thought, “Cool technology. We can do some basic things around the margin, but that’s it.”
Then the chain-of-thought paper came out in January 2022, and that made me think this might be extremely relevant to what Zapier’s customers were trying to do. I went to Wade, who was CEO, and said, “Wade, I need you to take back all of my responsibilities. You need to run product and engineering because I need to go do AI research and figure out what this means for our business and our customers.”
For 6 to 12 months, Bryan, our CTO, and I coded all day long and tried to understand what this technology could and couldn’t do. By the summer of 2022—4 or 5 months before ChatGPT came out—we had built a lot of the early prototypes.
8. Reliability as the key challenge in AI automation
We had built tree-of-thoughts prototypes and an internal version of ChatGPT using this technology. We had essentially prototyped all the foundational pieces in 3 or 4 months and identified the most obvious place Zapier could start: tool use.
Could we equip language models with the tools on Zapier’s platform—all the actions and search endpoints—and allow them to do more? The models had frozen weights and couldn’t take action in the world, but Zapier could provide them with tools.
That activated us very early to start building and delivering AI products. It also gave me an early perspective on the limitations of this paradigm. I spent time talking to hundreds of Zapier customers who were trying to deploy this AI technology in the middle of their automations.
Zapier has been deploying AI agents for 2 years now, so I’ve heard what people want from the technology and where it doesn’t work. The number-one problem they consistently tell me is that the promise is there and they understand what it could do for their business, but they don’t trust it enough to go hands-off.
Zapier is an automation product, which is different from ChatGPT. With ChatGPT, you type on a keyboard and get a response that you can audit. Zapier is running on a server online. You’re not watching it; you’re monitoring it.
The feedback was, “I don’t trust it because the reliability isn’t high enough for me to take a human out of the loop.” That feedback was consistent from GPT-3.5 through GPT-4 and GPT-4o. It didn’t match the scaling-law hype that was so strong in 2023 and 2024, and it didn’t match my lived reality.
I had 2 sets of facts that didn’t fit together, and I was trying to understand why. That’s when I rediscovered François Chollet through his appearance on the Lex Fridman Podcast. I had been thinking about ARC-AGI, but I really dug into it, read the paper “On the Measure of Intelligence,” which you published in 2019, and had my aha moment.
I thought that paper did a good job articulating the promise of this technology and why it’s impressive, while also explaining the fundamental limits we would hit by simply scaling up pretraining and memorization. That explained what I was hearing from customers.
Once I reached that conclusion, I thought ARC-AGI was clearly the most important benchmark in the world. More people should know about it. Going into last summer, it was still relatively obscure.
You created a prize for the benchmark. Do you want to talk about that?
That was the phase where you introduced me to François, so thank you for doing that. I hope the world gets a lot of leverage out of that introduction.
I had been surveying AI researchers in the Bay Area for 6 months to a year, asking, “Have you heard of ARC-AGI? I think this is the most important benchmark.” The awareness rate was relatively low. Maybe 10% to 20% of the people I met had heard of it.
Many of the people who thought they had heard of it had actually heard about an older version from the Allen Institute that had long been beaten by language models. ARC-AGI was a relatively obscure benchmark.
François and the ARC Prize team had been running a small version of a contest for several years, so there was evidence that it was a robust benchmark. It wasn’t completely obscure, but awareness in the AI industry was low.
I thought this was direct, concrete evidence that there were fundamental limits to scaling up pretraining. Every other benchmark in the world was saturating faster and faster, while ARC-AGI wasn’t. I concluded that awareness was the problem.
After your introduction, I flew to Seattle and had lunch with François. I pitched him some ideas about how to raise awareness. It was a fun conversation, but I also had some critical questions about why he thought awareness was so low and why he wasn’t working on it more. He had good answers to all of them.
9. The cost and efficiency of training frontier AI models
At the very end of my notes, I had written, “Pitch ARC Prize.” I had recently seen Nat Friedman and Daniel Gross run the Vesuvius Challenge, which had been very successful at raising awareness around an obscure problem and increasing interest in it.
I thought we could do something similar. That’s where the ARC Prize came from.
What happened when you launched the ARC Prize? Was it successful in getting new engagement?
Going into June 2024, maybe 10% of AI researchers had heard about ARC-AGI. By the end of 2024, everyone in technology had probably heard about it, so I think we solved the awareness problem.
My honest reaction is that I’ve been continually surprised by how much energy there has been around the benchmark. Here’s a concrete example. The contest ended in early November, and we had a number of teams on the leaderboard.
One of the requirements for winning the cash prize was that teams open-source their progress. The goal was to rebaseline progress in the community each year. The number-one team was a runaway winner. They had been at the top of the leaderboard all summer with a score around 55%.
The week before the contest ended, they emailed us and said, “We’re not sure we want to open-source our solution.” I thought, “Oh no. Is this going to work? Did we get something fundamentally wrong about how to structure the contest so that it helps the community make progress?”
Then, in the 72 hours before the contest closed, 2 other teams shot up the leaderboard from around 10th place to 2nd place. The number-two team was suddenly right behind the number-one team. There were also 3 papers published within 24 hours of the contest ending. They had timed their ARC research papers to drop at the same time and enter the paper contest.
There was a phenomenal amount of energy at the end of the contest that we hadn’t seen building. Something similar happened when OpenAI’s o1 model came out. There was an incredible outpouring of demand for us to test o1 on ARC-AGI. Thousands of people were asking us to test it, so we did.
It was important that we did, but these moments have surprised me in terms of the benchmark’s relevance and how much awareness we’ve been able to create around it.
What insights have come out of wrestling with the prize or the challenge? In AI, what approaches have worked? What have we learned about building intelligent systems?
10. Closing thoughts on AI’s trajectory
The classic way people tried to beat ARC for 4 years was through pure program synthesis. You build a domain-specific language—little Python functions or transformations—and have a human look at the puzzles and guess what the transformations could be.
Then you do a brute-force search over all possible combinations of those transformations to find one that matches the input and output. You apply it to the test case. That was the classic approach until this year.
It doesn’t work well. It’s slow, inefficient, and brute-force because all the generality is built into the domain-specific language that the human creates. At ARC 2024, there were a couple of new approaches that were interesting to see.
11. NDIA: Mike’s new AI research lab and its mission
The first was an induction-based approach. You generate lots and lots of Python programs using language models, then search over them. You also guide and inform the language model’s program generation using the input from the puzzle.
Ryan Greenblatt got an early score of around 40% using a technique like this. It was very inefficient—you had to generate hundreds of millions of Python programs—but it showed promise for this program-induction approach.
How do you pick among the programs? You generate them, but how do you pick which one looks promising?
You run them. The dataset has the answer, so you can evaluate whether a program produces the correct output.
This is true of math, coding, and ARC-AGI. It’s where o1, R1, and o3 dominate: in domains that are easy to verify. If a computer can check the answer quickly, with strong and exact correctness, these techniques work.
The other major approach was test-time training. The ARC dataset has a public set and a private set. The private set is what the Kaggle contest and the cash prize are attached to. Very few people have seen it, and you aren’t allowed to see it when you submit your test.
That gives us strong guarantees around the system’s ability to adapt and reduces the chance of cheating or fitting to the private set. What people figured out was that they could take the private dataset inside Kaggle and use it as a starting point to generate lots of similar data through data augmentation.
They might change the colors or mirror the grid—things that don’t change the semantic rule but generate lots of nearby permutations. They then fine-tune a model using several thousand or tens of thousands of those examples.
That worked. I think it was the approach that got one of the top systems to around 50%. Now we have 2 broad sets of evidence about how to use language models to adapt to novelty.
One is chain-of-thought search, which is what o1 Pro and o3 do. They perform lots of sampling and search at each thought step. The other is test-time training, where you take the situation and create data augmentations around it, feed those back into the model, and perform inference.
That’s a form of knowledge recomposition.
If ARC were broader and not just in this domain of pixels and changing colors, would that break the approach? It seems like you’ve constrained the domain by choosing that format.
I remember reading Douglas Hofstadter’s book as a kid, where he makes sequences of numbers and you try to guess the next number. That seems like a variant of an ARC reasoning challenge in a different structure.
That’s an underappreciated aspect of ARC, or at least a common misconception. ARC is not really a visual benchmark. I understand why people think that, because we render it visually. The intuition is that AI systems aren’t good at ARC because they aren’t good at visual domains yet.
ARC should be thought of more as a program-synthesis benchmark than as a visual benchmark. Classic program synthesis is exactly what you described: given a sequence of integers and an output sequence, figure out the rule that maps one sequence to the other.
There’s even a database called the On-Line Encyclopedia of Integer Sequences. Researchers have created hundreds of thousands of sequences that can be used for this kind of research.
You’re given an input sequence and an output sequence, and your goal as an engineer is to create a computer program that automatically figures out the rule. You might be surprised, but this is a very hard challenge.
Depending on the complexity of the program, how much hidden state there is, and how many hidden variables there are, programs can be long or complex. The longer the program gets, the more you run into the problem of having to enumerate possibilities.
Classically, researchers try to brute-force it. They search over every possible program that could potentially exist and check each one: “Let me plug in the input and see whether I get the output.” It’s extremely inefficient. The search scales exponentially, so it’s never really going to work in a reasonable amount of time.
That’s the rough form of the problem: you’re trying to create programs. You can map this onto o1 and o3 because you can get a better feel for how those systems work. You give the model a question, it thinks for a while, builds a large chain of thought, and gives you an answer.
One way to think about that chain of thought is as a program. It’s a natural-language program, but it is a program. It has individual steps, and each step transforms the latent space from the thought before it to the thought after it.
You’re constructing a program. What o1 Pro and o3 do is program recombination. They search over the space of possible programs, similar to classic program synthesis, trying to find the right one.
The big difference between program synthesis and deep learning is that program synthesis can learn out of distribution. It can find programs that generalize to data outside the training distribution. Deep learning can’t do that in the same way.
Deep learning is a paradigm where you give the model a lot of data, embed it on a high-dimensional manifold, and make quick, approximate judgments about new data near that manifold. You won’t get exact guarantees. You need a lot of data, and you get in-distribution accuracy.
With program synthesis, you’re looking for a program. You might need only a couple of examples. You don’t need a million or 100,000; you may need 3 examples to find the contours of the rule. Once you find the rule and have the program, it works for any input, no matter what input you give it.
It requires very little data and generalizes out of domain, but the problem is combinatorial explosion. You can’t search through all possible programs efficiently. In o1’s case, the system searches through possible reasoning chains.
How do you do this well?
The insight is to use the pros and cons of both sides and merge them together. You want to use the upside of deep learning—making quick, approximate sketches—to inform a search process so that the search isn’t brute-force.
Humans don’t sit here and think through 1,000 Python programs to solve an ARC puzzle. We use our intuition to generate a couple of possible answers, then symbolically verify them in our heads. We run through the steps and ask, “Is this right? Is this right? Is this right?”
If it isn’t right, we go back to the deep-learning part of our brain and say, “Give me some more ideas.” There’s a smooth back-and-forth between the 2 systems. We think that’s the fundamental substrate we can construct.
My co-founder Sean has done a lot of work on SWE-bench, and he got the highest score there. It’s a very different domain, but he’s using o3 and o1 to generate programs, run them, and try to figure out which one is best. The runtimes are expensive here, though.
I think cost is going to be a major factor. This is something the AI research field hasn’t fully reckoned with yet.
When we released the o3 results, we had to report them on a 2D graph where the x-axis was effectively token cost or cost per task. We’re entering a paradigm where you can spend more money to get higher accuracy and reliability. The curve is still logarithmic, so it isn’t linear, but you can spend more money to get a better answer in many domains.
That means you can’t report a single benchmark number anymore. ARC still needs to fix this for next year. We need a leaderboard with some kind of efficiency dimension because you can’t simply say, “That system got 75% on the benchmark.” You also have to ask how long it took and how much it cost.
Those are the questions we’ll need to answer in order to guide research toward efficiency. There are still a couple of human bottlenecks in these reasoning systems. Once we get past those, efficiency will be the main thing we have to figure out.
If you can use compute to get better answers, compute costs reliably go down. The shape of that curve matters too, doesn’t it?
I’m not sure I follow.
The slope, even on a log scale, of the tradeoff between cost and performance matters. I’d love to spend more money to get better answers in many domains.
There are several use cases where that’s going to start working this year. One of the major ones is agents.
Coming back to the Zapier anecdote, the number-one blocker for deploying automation with agents is that reliability and trust aren’t high enough. What does increasing the ability to adapt to novelty mean in practice? It’s not just a cute benchmark result.
It means you can more consistently get the same answer—not necessarily the right answer, but a consistent answer—from these systems. That gives humans the ability to steer them and control their behavior more precisely than before. It raises the reliability bar.
A lot of use cases where people want agents weren’t blocked by cost. People were willing to pay up to the rate of human labor. They were blocked because the systems didn’t work well enough. We’ll start unlocking those use cases by plugging systems like o1, o1 Pro, o3, and R1 into the planning steps of agent systems.
One thing in the back of my head during this interview is that we’re taking for granted that this kind of reasoning is important. I wonder if we’re losing people by showing these toy problems. What does this actually translate to in the real world?
I’ll show you. I have a funny story that led me down this AI-benchmarking path a couple of years ago.
When we were first building some of our AI agents at Zapier, we had one that I set up. Zapier has been a strong, longtime partner of OpenAI. I’ve been part of 3 major OpenAI launches, which has been fun. We use Slack, and we had a shared Slack channel with OpenAI.
When we were building some of our early agent prototypes, we built a system where we wanted the agent to automatically send a Slack message based on lead-management information coming from HubSpot. Our sales team was doing transcriptions, and those were feeding into the process.
The agent had 2 main functions. It needed to choose a Slack channel to send the customer information to, and it needed to write the body of the Slack message. We had the agent guess both based on the inbound message.
Our system let us give the agent hints about how to fill in those 2 fields. We could say, “For the Slack channel, use the testing channel. For the message, grab the lead’s first name, last name, phone number, and email, and build a little lead widget.”
When we first turned it on, we had a Slack channel called #testing. The name of the channel we had with OpenAI was something like #openai-partner-testing. The agent picked the partnership channel a couple of times and started sending customer information there.
That wasn’t good. This was a production system, and it was an important partnership. We didn’t want customer information shared there. We immediately turned it off, scrubbed the messages, and fixed the situation.
That was my first realization that reliability, overrides, and control would be extremely important for building trust. The first time you see something like that, you say, “Turn this thing off. Get it 10 feet away from me. I can’t deal with this. It’s putting my business in jeopardy.”
It made us realize early on that, in order to deploy AI agents, we needed to offer users hardcoded control over what the system was allowed to guess. With Zapier AI agents, you can allow the agent to guess if you want to, but the default is for you to choose a channel.
That gives you some certainty and guarantees. Or you can say, “Here are the 3 channels I want you to choose from.” It allows you to build in a hard guarantee.
All of this was an early insight that reliability matters enormously for deploying agents that automate tasks businesses and users care about. That is the kind of problem we’ll start to see solved with o1, o3, and R1.
If you build an agent and tell it to send something to the testing channel, and it works the first 3 times, you’ll expect it to keep working. That will become the truth. In the past, with the stochastic nature of pure language-model systems, you couldn’t make that guarantee without adding guardrails on top.
Why did you start Ndea as its own organization rather than work with OpenAI, given your close relationship with them? Why did you feel it needed to be a new organization?
The reality is that, if you look at the frontier of what most of these companies are working on, I don’t think they share our view about the importance of program synthesis.
I’ve spoken with people at the frontier labs, including OpenAI, and shown them the o3 results. I think the view is that this is still a deep-learning paradigm with some special search layered on top, and that the search isn’t the important part.
There are people at these companies who understand the importance of program synthesis, but I think the broad industry view is still that deep learning is scaling. We fundamentally disagree with that.
Program synthesis is at least 50% of the equation. Maybe that isn’t 50% from a compute-budget standpoint, but if you measure where the ideas for future, highly efficient AGI systems come from, half of them will come from program synthesis in some form.
AGI is such an important technology that anyone with a unique, differentiated idea should try it. That’s why we launched the ARC Prize: to inspire more people to try new things and return to the kind of AI research environment we had in 2018 and 2019, when everyone was trying different ideas.
I do think the ARC Prize has helped shift the Overton window. We’re starting to see more fundamental innovation from small startups. One of the biggest surprises for me was that I expected individual researchers and perhaps some large labs to work on ARC.
Instead, around 7 or 8 startups came to me during or after the contest and said they had changed their research direction or pivoted to work on ARC. That was exciting because it suggested the prize was having the impact we hoped for: getting people to explore more and increasing the overall probability that we figure this out.
We also have long-term ambitions for what we want to do with the technology, and those differ from the ambitions of many of the major players. But, purely from a technology standpoint, we started Ndea because we have a differentiated view that we think has a good chance of success and could increase the probability of getting to AGI quickly.
Are you also oriented toward making a product and making money?
I have a strong view on that. Ndea is a research lab. We launched it a few weeks ago. François and I started it as an intelligent science lab. Our view is that reaching AGI requires combining deep learning and program synthesis. We think o3 beating ARC-AGI v1 may mark the start of a 5-to-10-year scaling journey on program synthesis, similar to AlexNet beating ImageNet in 2012. We don’t yet have the Transformer for program synthesis, and the field is much smaller than deep learning, so we need to grow it. We don’t have a product offering in the near term. Our goal is to get to AGI by our definition: reducing the gap between things that are easy for humans and hard for computers to zero.
You shouldn’t expect us to make products before that. We may use some of our prototypes to advance the frontiers of science, though. Building AGI is step 1 for us. What really excites me is leveraging the technology to accelerate the pace of innovation at the frontier of different scientific fields.
AGI is going to solve a lot of problems. It’s already solving problems for Zapier customers, and it isn’t even AGI yet. People are going to use this technology to solve many problems, and that’s great. I fully support it.
What excites me more is accelerating into an unknown future. Consider the printing press. When it was introduced in the 1400s, about 600 years ago, there was fear and skepticism, but also excitement about the proliferation of knowledge, sharing knowledge globally, and exchanging ideas freely.
If you had asked people then to predict what 2025 would look like, they would probably have had a hard time imagining Wikipedia, AI trained on Wikipedia, or the fact that we can talk to computers. We’re so far down the technology tree from those initial moments that catalyzed everything.
That’s what excites me. It’s almost more of an adventure than a problem-solving exercise. There will be exciting things in the future that I can’t even tell you about yet.
To get there faster, the main constraint is that we need computers that can work autonomously to innovate. That’s the bottleneck on creating AGI, and I’m excited to help accelerate it.
Do you have a point of view on the timeline for reaching the definition you described—getting rid of all the gaps between what’s easy for humans and what’s hard for computers?
There’s another part of that definition worth considering: efficiency. I think we’ll probably reach that definition before we reach it at human efficiency.
I’d add an asterisk: we’ll have to say there are no remaining tasks that are easy for humans and hard for AI, and that we’re doing them at human-level efficiency.
What does human-level efficiency mean? Can you put a dollar amount on it? Is it your salary?
That’s literally something we’re debating right now: how to measure efficiency. I don’t think the industry knows. ARC Prize is trying to figure this out too.
You want something you can compare between humans and computers. FLOPs aren’t a great measure. You could use dollars because you can pay a computer to do a certain amount of work for a certain amount of money, and you can pay a human to do the same amount of work.
Time is interesting from a wall-clock perspective because computers can make time faster through parallelization, which is something they’re good at. Honestly, I don’t know the answer.
I’d probably argue that dollars are the best measure today. Everything fits into dollars: how much you pay a human for their labor rate and how much the compute costs. Dollars also track the price-performance improvements as compute becomes more efficient.
Let’s say human-level efficiency means that we’re paying humans to do these easy tasks that computers find hard, and paying computers to do the same tasks.
Something like that. I’ll add that asterisk.
My expectation, if I had to make some wild guesses, is that ARC-AGI v2 will be durable for 12 to 18 months. That’s my best guess based on how it’s performing on all the frontier systems we’ve tested.
It won’t be another 5-year benchmark, as v1 was, because v2 is in the same domain. It’s raising the bar for computers without raising the bar for humans. It should still be interesting and useful as a gradient—a tool that points research in a direction.
What I’m excited about is v3, which we’ve started designing and prototyping for next year. Our design goal for v3 is that it will be durable for 3 years. That’s our hope and goal, although we’ll obviously have to compare it with reality.
My expectation is that we won’t have solved “easy for humans, hard for AI” at human-level efficiency for at least the next 3 or 4 years.
One of the hardest things about making predictions is that it’s easy to make predictions around smooth scaling curves. It’s extremely hard to make predictions around step-function changes in capability.
If you have a smooth curve, you can track it and make future guesses. But if you look at ARC-AGI v1, it went from zero to 4% over 5 years, and then from 4% to 85% in 2 months. That’s extremely hard to predict.
There are 2 reasons. You don’t know whether the technology to do it exists in the world yet, and you don’t know whether the ideas exist yet. Even if they do exist, you don’t know whether someone has put them together into a system that can demonstrate the capability.
There are a lot of unknown variables when you try to predict when step-function capability jumps will happen. I don’t believe we’re in a pure scaling regime where you can draw a smooth curve over system sizes and project it into the future.
I don’t think we’ll have a job at that point. I treat this much more empirically. You can make informed guesses about when step functions might happen, but I think it’s in the public interest to understand that this is the reality: we’re still looking for step-function capability leaps, and we don’t know when they’ll appear.
Awesome. I think that’s a good stopping point for this interview. We covered a lot of content and a lot of wide-ranging topics. Thanks so much for listening to this episode of Gradient Descent. Please stay tuned for future episodes.