AutoGrad 改变了一切（不是 Transformers）[Dr. Jeff Beck]
Jeff Beck 的核心判断是：仅靠扩展预测模型，本身不会产生类人智能。 AutoGrad 和超大规模扩展把神经网络研究变成了工程学科，但结果仍然只是“非常好的函数逼近器”；创造力和新问题求解需要明确的、以对象为中心的因果模型，并通过系统工程重新组合。真正具有投资意义的区别在于架构，而不只是参数量：“仅靠函数逼近不会带来智能。”
他提出的替代方案，是对“大量小模型”进行超大规模近似贝叶斯推断。 与其让一个单体模型隐式编码一切，不如让智能体只实例化当前环境相关的对象和交互规则，保留不确定性，并持续学习。Beck 对算力的判断是节俭：“你可以有100万个小模型，但在任何时刻真正需要的只是其中极小一部分。”
机器人是最清晰的商业试验场，因为今天的系统往往只是在模仿轨迹，并没有学会物理规律。 Beck 认为，仓储和家用机器人只有在内部模型接近现实世界稀疏、宏观的结构，且模拟器优化的是准确物理而非“看起来合理”的物理时，才能实现泛化。目标不是继续“模仿人类行为、同时别把鸡蛋捏碎”，而是让智能体理解物体为何会这样运动。
仓库机器人遇到一只陌生猫的场景，体现了这种架构潜在的运营优势。 意外度会陡升，机器人随即停下，向远程候选模型库发起查询，测试约7或8个假设，保留猫模型并丢弃其余模型。能够“知道自己不知道什么”，可能让部署中的系统比永久加载通用模型更安全、更具适应性，也更省算力。
Beck 将语言落地视为一种方便的接口选择，而非认知的正确基础。 语言和自我报告并不能可靠描述现实，也不能可靠描述产生行为的过程；人类是在宏观物体、关系、力量和可供性构成的环境中进化而来。因此，他的设计原则是物理落地：“思想的原子元素……就是我们所生活的物理世界的模型。”
对齐问题仍未解决，因为从观测到的行动中，数学上无法将智能体的信念与价值观分离开来。 手写奖励函数，例如撞到松鼠扣10分、撞到猫扣50分，本质上都是任意设定；目标设错，就像向“恶毒精灵”许愿。只做预测的神谕系统更安全，但会牺牲人们想要的自动化；自主系统要么需要人在环，要么需要一种像人类那样拆分信念与偏好的方法。
主动推断用了约20年证明其广度，但Beck认为下一阶段必须证明深度和规模。 过去约8年的进展，包括归一化流、自然梯度、快速与条件采样、约束方法等，让近似贝叶斯推断更易处理。真正的技术和公司构建机会，是停止展示玩具级应用，“开始用这些方法解决真正困难的问题”。
1. 贝叶斯推断把科学探究变成明确算法
Beck 说，自己的信念源于一次听 Zoubin Ghahramani 讲解 Dirichlet process prior 和 Chinese restaurant process：新数据与旧数据比较，相似观察被归为一组，理论由此建立并接受检验。这听起来像是“总结……科学方法实际如何运作的算法”。
他的规范性判断非常明确：贝叶斯推断是“思考世界如何运作的唯一正确方式”，因为它迫使人们提出明确假设，并构建以这些假设为条件的生成模型。经验研究因此变成有纪律的模型比较，而不是非正式的模式匹配。
不过，Beck 走向“贝叶斯大脑”这一命题的路径仍然是经验性的，而非哲学性的。行为实验显示，人类和动物会根据不同线索在每次试验中的可靠性，对多个线索进行组合，表现得仿佛它们表示了不确定性，并以相对最优的方式整合可用证据。
2. 大脑之所以相对最优，是因为它过滤信息而不是记录信息
Tim Scarfe 将线索组合的结果进一步追问：人类是否高效到只有贝叶斯分析才能解释？Beck 对此加以限定——表现是“相对最优”的，因为从刺激到大脑的过程中信息已经损失，但行为仍然类似于对剩余信息进行最优融合。
Scarfe 所说“大脑做的90%是决定忽略什么”，概括了他的容量约束框架；他用雾天驾驶的例子说明不确定性同样会影响行动。Beck 随后同意，大脑处理的信息多于最终显露在行为中的信息。
Beck 提到，要维持低层视觉统计特征，就需要持续输入：闭上眼睛5年，系统的保真度就会下降。大脑使用的信息多于行为所暴露的内容，包括那些并未被直接感知的低层信息，但它仍然不会利用所有现存信息。
3. 模型靠便利性、预测和干预证明自身价值
Beck 认为，“大脑是一台计算机”和“大脑是一台预测机器”，都继承自更早的技术隐喻——先是杠杆和滑轮，后来是体液学说，如今是计算机。“我们解释大脑如何运作，会类比我们拥有的最先进技术。”
自回归说明了有用形式主义与现实之间的区别。大多数、但不是所有语言模型都按序列进行预测，因为这种方式能简洁地把过去转化为对未来的预测；Beck 怀疑，这并不意味着语言本身真的按照自回归方式运作。
动量提供了一个物理类比：动量无法被直接观察，但引入动量后，机械模型就变成了马尔可夫模型，能够轻松沿时间推进。Beck 对其本体论保持开放——也许宇宙确实具有这种结构，也许科学家只是选择了“一个计算上方便的模型”，而它恰好奏效。
4. 因果性压缩计算，并指向行动
Beck 认为因果模型的实际价值有两点：减少需要跟踪的变量数量，并找出干预会改变结果的环节。一个紧凑的因果状态可以向前迭代，再据此选择最可能实现目标的行动序列。
识别因果关系仍然需要干预。Beck 回忆说，过去人们曾错误地认为酗酒与肺癌有关：健康状况不佳和更高的吸烟量混淆了两者关系，这说明仅凭观察无法确定真正起作用的原因。
重要的因果尺度取决于智能体能够施加的可供性。微观原因在智识上有价值，但“除非你有非常小的镊子”，它们并不能指导行动；技术之所以重要，是因为工具可以扩大人类的可供性，核能就是一个例子。
5. 向下因果性为宏观对象提供依据
按照 Beck 的工具主义视角，人类会优先采用自己能够操纵的尺度上的因果描述。理解更小的尺度，只有在它能启发扩大“影响范围”的工具时才有价值，而不是因为微观解释天然就取代了有用的宏观结构。
他判断是否可以把一圈微观元素划在一起并称为一个对象，标准就是向下因果性。当宏观描述足以很好地总结系统、使微观细节对后续预测变得无关紧要时，这个圈才是“一个好的圈”。
压力、体积和温度符合这一标准，因为它们的聚合动力学可以被写出，并以相对确定的方式预测。意向性则更难：它具有解释力，但如果没有意向如何演化的模型，就可能沦为静态的人格判断，而不是具有预测力的宏观变量。
6. 主动推断建立了共同语言，但仍然不够深入
Beck 认为，Karl Friston 将信息论与统计物理连接起来，再加上 Markov blanket 结构，创造了一种几乎普遍适用的对象与智能体语言。它从流行病学、物理学延伸到社会科学，既解释了 Friston 为什么不断对同一框架做变体，也解释了这一社群为何异常多元。
Scarfe 注意到，如今关于主动推断的讨论已经覆盖加密货币、基督教、现象学和心理学。Beck 看到其中的收益与代价：研究者可以用共同数学讨论截然不同的领域，但也“经常最终各说各话”。
Beck 的批评是，20年的布道带来了“大量广度，却没有多少深度”。玩具级演示证明了这一框架可以描述许多心理、认知和神经网络现象，但这个社群基本回避了大规模、困难的应用。
7. AutoGrad 比 Transformer 单独发挥了更大作用
Beck 认为，近期 AI 爆发有3个原因：AutoGrad、Transformers 和前所未有的规模扩展，但 Transformer 要加一个星号。Mamba——他称之为“基本上是一个开了强化模式的卷积滤波器”——通过扩展规模实现了相近功能，尽管采用的是不同架构。
AutoGrad 的决定性作用，在于它用快速的架构实验取代了手工推导学习规则。研究者可以不断调整网络、非线性函数、记忆机制和训练技巧，直到所谓的瓶颈——包括梯度消失，以及反向传播带来的实际困难——都变成拥有可行解法的工程问题。
历史教训是，反向传播曾因生物学上不可信、技术上无法扩展而被否定。Beck 接受第一个 objection——大脑确实不使用反向传播——但认为实验已经推翻了技术上注定失败的宿命论：“我们不可能靠摆弄方程式发现它们。”
这场成功消除的是智能与函数逼近之间的区别。Beck 说，领先公司的专家正在收回 AGI 言论，因为单纯预测正在暴露自身局限；他的答案是，在不放弃规模扩展经验的前提下，把认知结构重新引入系统。
8. 类人智能需要稀疏且具身的世界结构
Beck 的项目，是建立一套能够大规模运行、受认知启发的模型框架：近似贝叶斯、仿照大脑组织方式，并围绕世界本身的运作方式构建。他不喜欢依赖 AGI 这个词，更愿意使用“下一个层级”这一具体目标。
具身并不只是把一个模型接到机器人上。机器人需要一个以对象为中心、动态且大体因果的物理世界表征——包括椅子能够支撑人，以及物体通过特定机制相互作用等关系性知识。
Transformers 会反复将每个 token 与其他所有 token 比较，而视觉-语言-行动模型往往运行在微观像素空间。它们之所以有效，必然已经获得了一些隐式宏观知识，但 Beck 认为，它们缺少现实世界那种明确、稀疏的因果组织。
Scarfe 提出规模扩展的反例：只要拥有足够多的正确数据，超大模型或许能够恢复出基本正确的隐式世界表征。Beck 承认这对预测可能成立，但否认它足以支撑发明，因为智能体必须操纵明确的组成部分及其关系。
9. 系统工程才是真正创造力的基准
Beck 用飞机作类比说明这一机制：理解机翼如何产生升力、喷气发动机如何产生推力，再把两者组合成一个新系统。这不同于从既有答案中挖掘内容，再移动词语，直到输出“看起来大致正确”。
一个面向系统工程的 AI，需要表示对象、子系统及其关系，然后探索新的组合。Beck 的强判断是，没有这种结构，预测系统只能改造既有方案——甚至把它描述成“改造”都可能过于慷慨。
程序合成在更抽象的层面遵循同一模式：把一个程序接到另一个程序上，直到组合能够执行有用功能。尚未解决的问题是可读性，因为合成程序可能变成一团对人类有效、却无法理解的规则“糊状物”。
10. 物理落地胜过语言，成为认知的共同货币
Beck 认为，“落地模型”这个说法很含糊，因为每个模型都落地于它获得的数据。Scarfe 则把视觉-语言模型描述为落地在语言空间中，使组件能够通过语言交流，也为人类提供了方便的对话接口。
方便并不能决定正确的落地领域。Beck 认为，语言“对我们的思维过程和现实都是极差的描述”；实验参与者经常会为自己的选择给出听起来合理的解释，但行为模型显示，实际决策过程并非如此。
他的替代方案从生存出发：细胞落地于周围汤液的化学环境，而哺乳动物落地于一个由物体、其他智能体和关系构成的宏观世界。想要像人类一样思考的 AI，应当继承这些“思想的原子元素”。
11. 近似与模块化可能让贝叶斯系统实现规模化
Beck 指出，过去约8年的进展让贝叶斯推断更易处理，包括高斯过程研究、归一化流、自然梯度、快速与条件采样，以及约束方法。自然梯度允许在参数空间进行大步移动，有时还能用坐标下降取代梯度推断。
他喜欢这个领域近年出现的新缩写，包括 Bayesian online natural gradient——简称“BONG”。但实质问题是速度。主动推断社群对“纯正老派消息传递”的坚持限制了规模；放松必须做到最大程度贝叶斯化的要求，就打开了更多近似路径。
Scarfe 将这一转变类比为约束满足：后来的经验方法让曾被认为不可处理的方法重新焕发生机。Beck 接受这一宽泛类比：现在工具已经具备，问题是要测试主动推断能否解决困难问题，而不是仅仅重新描述玩具问题。
12. “大量小模型”支持复用与持续学习
Beck 将一个单一的家庭模型与数千个更小的模型进行对比，后者分别描述书、架子、家具及其关系。在房屋中学到的书模型，可以与在公园和长椅上学到的模型组合，前提是交互语言保持结构一致。
Scarfe 提出文化歧义：一群人可能把两个东西划分成一个对象，也可能以不同方式理解时间。Beck 的回答不是否认歧义，而是用关系约束划分——对象必须参与可发现的交互类别，类似广义力，而不是固定的牛顿向量。
一个只见过书放在架子上的模型，应当保留关于重量和碰撞行为的误差条。当保龄球撞上书时，智能体应当修正后验，而不是把未见过的交互视为不可能。
这要求持续学习。Beck 将其与当代部署方式对比：公司花数百万训练一个模型，随后基本关闭学习。对他而言，智能体应保留不确定性，在新交互发生后快速更新，并在使用过程中始终保持持续、交互式学习。
13. 类似电子游戏的引擎可从原始像素中提取稀疏认知
在模拟中，Beck 的架构类似游戏引擎：拥有包含形状、颜色、质量和交互规则的大型资产库；对于水或沙子，则在粒子模拟成本过高时采用宏观处理。不同之处在于，其中的广义力必须由模型学习，而不是预先提供。
一个智能体可能拥有1,000,000个对象模型，但只实例化当前场景相关的10,000个。这样的稀疏性让世界模型库在计算上变得可行，也让认知像局部模拟一样运行，而不是激活一个通用表征。
Scarfe 称这个类比是在“作弊”，因为游戏引擎会直接暴露实体之间的力，而真实机器人无法向现实查询真值。Beck 接受更困难的版本：从像素出发，不提供预先分割的图像，自底向上发现资产、属性和有效力。
14. 更好的物理模型可能弥合机器人从模拟到现实的鸿沟
Scarfe 复述了这个实验：给黑箱智能体输入像素、移动和交互控制，以及一个游戏环境；让它学习稀疏模型库，再测试这些模型能否迁移到实体机器人上。Beck 确认：“这就是想法。”
Beck 认为，迁移效果差可能源于模拟器过于贫乏，但更大程度上来自两个错配：模拟器对现实世界建模不准确，智能体的内部表征也与其所处环境的结构不同。电子游戏追求视觉上的可信和趣味，而非忠实物理，往往还会针对僵硬的碰撞动力学加入各种补丁。
如果模拟器具备足够准确的物理模型，同时智能体能够学习其中明确的结构，那么它应当更好地泛化到现实。Beck 认为，这对机器人来说是关键能力，而不是外观层面的改进。
当前的杂货处理系统往往学习专家轨迹——模仿人类，以免把鸡蛋捏碎——却没有学会背后的物理规律。跨领域、跨任务泛化，需要用明确的对象和交互模型取代对轨迹模仿的依赖。
15. 仓库里的猫展示了不确定性如何变成产品特性
Beck 的“仓库里的猫问题”从一个经过箱子、叉车和工人训练的智能体开始。一只陌生的猫进入场景，意外度急剧上升，系统因此停下而不是盲目继续：“别把猫碾过去。”
智能体拍下异常照片，然后“打电话给朋友”——一台装有大型模型库的服务器。它收到7或8个合理候选模型，观察这只动物，进行假设检验，保留猫模型，再把未使用的候选模型发回去。
这种架构将无知变成明确且模块化的状态。仓库智能体不必携带每种可能动物的参数；它需要的是本地能力、不确定性追踪，以及访问远程专业能力的权限。Beck 称之为“巨大的算力优势”。
16. 深度学习在服务概率推理时仍然有用
Scarfe 问，视觉 Transformer 是否可以从一张“300像素见方之类”的图像出发，引导贝叶斯学习。Beck 的回答很务实：使用“适合目标”的深度学习工具，并确保它们与概率框架兼容。
归一化流是他偏好的例子。它学习从高斯等易处理分布到图像等复杂观察之间的映射，在保留可处理概率分布的同时，支持复杂的似然函数。
Beck 颇具挑衅性地把扩散模型重新描述为通过扩散协议训练的归一化流：噪声将图像变成高斯分布，模型再学习逆向变换。他会说“扩散模型”，但前提是“你非要让我翻白眼”。
17. 任意奖励函数掩盖信念与价值观，导致对齐失败
Beck 开场时开玩笑说，他的对齐方法是讨论人们的信念和价值观，然后说服他们接受自己的信念。严肃的问题在于，人工智能体的信念和奖励函数并不等同于人类的信念和奖励函数。
围棋或国际象棋有清晰的胜负奖励，但复杂环境没有。撞到松鼠扣10分、撞到猫扣50分，只是把任意的道德选择伪装成数字精确性：“这些数字是从哪里来的？”
设错的奖励就像向“恶毒精灵”许愿：要求 Skynet 终结世界饥饿，退化解可能就是“杀光人类”。Beck 看不到选择正确奖励函数的规范性规则，“除非诉诸神的干预”。
观测到的行动混合了信念与价值观；没有额外信息，就无法在数学上将二者分离。人类通过交换决策背后的事实和信念形成机制来解决分歧；只有在信念对齐之后，才能识别真正的价值冲突。
18. 只做预测的神谕系统更安全，但会放弃自动化
Beck 称现有 AI 系统既不可解释又缺乏道德：“我们完全不知道如何把道德放进它们。”最安全的回应，是移除自主决策功能，把模型当作神谕，预测采取候选行动后会发生什么。
这种分离可以防止系统强加自己的奖励函数，但也会阻断人们真正想要的自动化——让智能体代表自己行动。Beck 认为，选择要么是持续的人类监督，要么是提出一种像人类那样建模信念与价值观的对齐方法。
Scarfe 问，明确的认知模型是否是赋予信念或意向的前提。Beck 指出其中的讽刺：他部分因为不了解其他人的内部机制、又假设他们与自己相似，所以会把意向归于其他人；但当他了解预测算法后，就更难对它们进行拟人化。
19. 当宏观规则取代微观模拟时，涌现才有意义
Beck 不接受“凡是我没预测到的东西都是涌现”这种基于无知的定义。他更愿意接受通过模拟才能发现的定义，但更偏好基于向下因果性的务实解释，因为它既提供判据，也提供实际的建模捷径。
在 Lenia 中，类似变形虫的游动体撞上障碍物后会变形，仿佛失去自身身份，随后又重新形成。Beck 的物理发现系统“或多或少”捕捉到了这一转变：游动体在变形过程中成为另一类对象，之后又恢复原有身份。
目标不是找回每一条像素级规则，而是在关注的尺度上发现支配宏观漂浮体和障碍物的有效力。对 Beck 而言，真正有吸引力的问题不只是简单规则如何产生复杂性，而是“最终形成的大尺度对象具有什么性质”。
20. 程序合成需要面向可理解组合的学习先验
Scarfe 认为，元胞自动机提供了一座“神奇的桥梁”：简单的局部编程可以产生复杂一个数量级的行为。Beck 带着讽刺回答，普通计算机本来就由逻辑门组成；把局部规则调到目标方向，仍然是“一种拐弯抹角的编程”。
尽管如此，Beck 仍然认为程序合成很有前景，也与他的系统工程议程相容。两者都在寻找模块化元素的有用组合，只是现有系统经常生成过于纠缠、难以被人类理解的程序。
Scarfe 建议，来自高质量软件仓库的数据可以改善搜索，就像编程智能体会从这类样本中受益。他提到 Tony Zador 关于神经架构紧凑“遗传”编码的工作，并认为类似先验可以复用成功的结构模式，再针对新问题进行合理变异；Beck 同意，程序合成确实很有前景。
Dr. Jeff Beck
So my PhD is in mathematics from Northwestern University. I studied pattern formation in complex systems, in particular combustion synthesis, which is all about burning things that don't ever enter the gaseous phase.
Bayesian inference provides us with a normative approach to empirical inquiry and encapsulates the scientific method at large, right? I just believe it's the right way to think about the empirical world. I remember I was at a talk many years ago by Zoubin Ghahramani, and he was explaining the Dirichlet process prior. This is when the Chinese restaurant process and all that stuff was relatively new.
His explanation of it resonated with me in terms of, “Oh my gosh, this is the algorithm that summarizes how the scientific method actually works.” You get some data, then you get some new data, and you say, “How is it like the old data?” If it's similar enough, then you lump them together, and then you sort and build theories and properly test hypotheses in that fashion. That's the essence of the Bayesian approach: it's about explicit hypothesis testing and explicit models, in particular generative models of the world conditioned on those hypotheses.
I believe it is the only right way to think about how the world works, and it encapsulates the structure of the scientific method. If I'm being perfectly honest, what actually convinced me the brain was Bayesian had a lot more to do with behavioral experiments done by other people. My principal focus was, well, how does the brain actually do this? I'm referring to experiments showing that humans and animals do optimal cue combination. We're surprisingly efficient in terms of using the information that comes into our brains with regard to these low-level sensorimotor tasks.
Oh, interesting. So it's almost like we're so efficient that the only explanation that makes sense is that we must be doing Bayesian analysis.
Dr. Jeff Beck
Yeah, more or less. I mean, it's a bit more precise than that. It's not just efficiency. The cue combination experiments, I think, are really compelling.
The idea behind a cue combination experiment is that I give you 2 pieces of information about the same thing. One piece of information is more reliable than the other, and the degree of reliability changes on a trial-by-trial basis. So you never know a priori that, say, the visual cue, as opposed to the auditory cue, is going to be the more reliable thing.
And yet, nonetheless, when people combine those 2 pieces of information, they take into account the relative reliability on a trial-by-trial basis. That means that they're optimal in a sense.
Now, we have to be super careful with our words. They're relatively optimal because they're not actually using 100% of the information that the computer provided. You don't use 100% of the visual information that the computer provided you.
Right, but there is some loss between the computer screen and your brain. The system behaves as if it has optimally combined those 2 cues. It has taken uncertainty into account. This is also because of how we really do think about the world: we take uncertainty into account all the time in our decisions. You know this if you've ever driven in the fog; you're aware of this.
Ninety percent of what the brain does is decide what to ignore, right? Because if we didn't, we'd be screwed. We receive an insane amount of information, most of which we don't even bother to process.
Dr. Jeff Beck
Yeah.
Yeah. Is that definitely the case, though? Do you think that we could actually be processing more information than we know?
Dr. Jeff Beck
We are definitely processing more information than comes out in behavior. A lot of that is because we are continually learning, and learning requires input. You close your eyes for 5 years and your visual system decays. You lose fidelity; it forgets. It requires constant input simply to maintain this understanding of the low-level statistics of the visual world.
So the question is, is that using all the information, or is it just using the low-level information? It's information that we don't directly perceive, but it is still definitely being used, in a sense.
What is it being used for? It's being used to track these low-level statistics that we sometimes need but don't always need. This is why I say that, when we say context matters, you can think of that in terms of being able to flexibly switch between tasks. That means having a lot of resources maintained and having them still be in good working order just in case we need them, right?
This is why these self-supervised or unsupervised learning approaches that are ubiquitous for pretraining, getting your LLM to give you a reasonable prior over language, are the sort of stuff that your brain is definitely doing. So, in a sense, it is using everything.
But it's not really using all of the information that's present, right? That's sort of the argument that I want to make. The idea of having to traffic in squishy people in order to make our systems go is not immediately appealing. Let's put it that way.
This episode is sponsored by Prolific.
Let's get few quality examples in. Let's get the right humans in to get the right quality of human feedback in. So we're trying to make human data or human feedback, we treat it as an infrastructure problem. We try to make it accessible. We're making it cheaper. We effectively democratize access to this data.
What do you think about these broad, sort of metaphorical idealizations? The big one is that the brain is a computer. The probably more popular one is that the brain is a prediction machine.
Dr. Jeff Beck
It will always be the case that our explanation for how the brain works will be by analogy to the most sophisticated technology that we have. How's that for a non-answer, right?
A couple thousand years ago, how did the brain work? It was levers and pulleys, man. I mean, duh. Don't be ridiculous. At some point in the Middle Ages, it became humors, right? Because fluid dynamics was the most advanced technology that we had, or technology that took advantage of water power was the most advanced technology that we had. Now the most advanced technology is computers. So, duh, that's exactly how the brain works.
Philosophers used to think that the universe was a machine.
Dr. Jeff Beck
Mhm.
We interviewed Chomsky about this as well, because he talks about the ghost in the machine, and the ghost is all of the bits in the machine that we don't understand. Do you think now that we can think of the universe as a machine?
Dr. Jeff Beck
I think that that is a very convenient way to think of the universe. When we model the universe as having causal structure, do we do so because it actually has causal structure, or because that's a really convenient class of models with which to work? I think that it has causal structure, but it's also a convenient class of models.
A good example is large language models. Most, but not all, are autoregressive in terms of their predictions. Why? Why are they autoregressive? Oh, it's because it's mathematically convenient. It's a compact way to take the past and make a prediction about the future.
Does it mean that that's actually the way language works? No, I don't think it's actually the way language works, but it's a computationally convenient model.
In physics, momentum is a good example. Why do we need momentum in order to describe things? We don't observe momentum directly. If you're just looking at videos, you're just looking at the position of the ball. You want to infer the velocity; you take the difference between 2 adjacent positions, and that gives you the velocity. But you don't ever directly observe momentum in a mechanical setting.
Why did we choose momentum? We chose momentum because that's the variable that, if we knew what it was, would make everything Markovian. Everything is now described by a simple causal model that explains how the world works. We picked that model—picked that particular hidden variable—because it's what rendered the model causal.
Does that mean that's how the universe works, or was that just a computationally convenient choice? I'm going to stay agnostic on that one. But I do like that it ended up working out computationally.
And just quickly, riff on the benefits of having models that prefer causal relationships.
Dr. Jeff Beck
The nice thing about causal relationships is that when you have a causal relationship, it reduces the number of variables you have to worry about and track. That's the beauty of having a cause. It's like a Markov model.
It's the same argument with momentum and Markov models. We chose to have that hidden variable because it's the thing that made the model simpler, right? It made the calculations easy. Now we can just go forward in time and make predictions in a totally iterative fashion. That's what makes causal models great.
The other thing that makes causal models great is that, if you do ever intend to act or behave, then you still need to be able to predict the consequences of your action. The more tightly linked your actions or your affordances are to the things that causally impact the world, the more effective those actions are with respect to your model, but hopefully also with respect to reality.
And so we prefer causal models, in part because they are, relatively speaking, simpler to execute in a simulation form, but also because they point directly to where I should intervene. Where should I go in, and how should I choose my series of actions that will lead me to the desired conclusion or goal?
What's the difference between microcausation and macrocausation? I think the difference between micro and macro is a single letter.
Dr. Jeff Beck
No.
We could just model the light cone at the particle level.
Dr. Jeff Beck
Oh, yeah.
Yeah, I mean, that's the way physicists see the world, and we see the world in terms of populations and people and all these macroscopic things. We still reasonably do experiments, interventions, and randomization.
Dr. Jeff Beck
To truly identify a causal relationship, you have to do an intervention, right? The classic example is also in lung cancer. I forget how long ago this was, but at one point there was this belief that alcoholism caused lung cancer. But it was actually because they were in poor health because they were alcoholics, and they smoked a lot more than the rest of the population. So you do need to do that kind of intervention to discover a causal relationship.
However, the causal relationships that we care about are the ones that mesh with our affordances. If identifying a microscopic causal relationship is super—well, that's great, but unless you have really tiny tweezers, it's not very helpful, right? What you need to do is identify the causal relationships that are present in the domain in which you are capable of acting.
We care about causal relationships at the macroscopic level because that is where we live. We live at the macroscopic level, and most of our actions are at the macroscopic level.
One of the best things about humans is our ability to extend the domain of our affordances with technology, right? We have nuclear power because what we did was acquire the ability to take tweezers at that scale and make these things happen. We figured out how to take advantage of causal relationships at that level, not because we have those abilities, but because we were able to create the tools that gave us access to that space.
It all depends on what the problem is that you're trying to solve. The causal relationships that you care about will always be the ones that are related to the actions that you are capable of performing. Now, that said, there's clearly a great advantage in understanding the microscopic causal relationships, if for no other reason than that it might lead us to discover a way to expand our affordances into another aspect of the microscopic domain.
Is this just instrumental? Is this just something that's a little bit like when we say that agents have intentions and representations? It's just a great way of understanding things, but for all intents and purposes, it's not actually how it works.
Well, I think that sentence ended on a rather definitive statement, with which I don't think I would agree, but the rest of it is—you're asking the scientific anti-realist if it's all instrumental. So, yeah. Yeah, it's all instrumental, right?
The things that we care about are, again, back to affordances. We need to understand causal relationships at the scale that we can manipulate. That's what matters most, because that allows us to have effective actions in the world in which we actually live. To the extent that we care about other scales, it is simply because we wish to expand our domain of influence.
The mind is quite an interesting example. So let's say I want to move my hand and my mind willed it. So it's top-down causation. Now I can't act in the world of my mind, but it seems macroscopically intelligible. We think about our minds. So maybe the mind is a special case. I don't know.
Well, the mind is a special case. I'll agree with that. I think of downward causation—well, I guess, from an instrumentalist perspective, I'm not saying downward causation is the thing. I'm saying that downward causation is one way that it all works. I would take it more from the perspective that, if you discovered downward causation, it is what justified your macroscopic assumption.
What do I mean by that? Suppose I'm in the following situation: I've got a bunch of microscopic elements, and they're all doing stuff, and I'd like to draw a circle around them and call that a macroscopic object. Now, I am justified in doing so if that particular description at the macroscopic level has the downward-causation property. It's a way of saying, "Oh, that was a good circle you drew. That was a good circle," because it summarized the behavior of the system as a whole in a way that rendered the microscopic behavior irrelevant to further consideration.
Yes, I can think of some situations where we do this. We might identify an aspect of culture or a meme, and we might say that is responsible for violence or something like that. You still have to show that it has that property, right?
I think intentionality is a tough one, because it's a variable that has a lot of explanatory power, but it's not one that evolves. When I think of a good macroscopic variable, it's one that I understand how it evolves over time. That's what makes it a good macroscopic variable. I can just write down a simple equation and it says, "Pressure, volume, temperature—they are going to do this over time," and taking any little microscopic measurement becomes totally irrelevant, right?
But what made it useful wasn't just that the microscopic measurements are irrelevant. It's that I had an equation that describes how it would have behaved, and that's also fairly accurate. So I have a nice, relatively deterministic model at the macroscopic level, right?
When we talk about intentionality, yes, it can be used as an explanatory variable, but it's only good to the extent that we understand how that intentionality changes over time. It's a long-term prediction. This is why the jurisprudence example made me really uncomfortable, because it's sort of like saying, "Well, what you're kind of doing is you're saying this is a bad person," right?
I don't know how we would necessarily identify that intentionality except in a very indirect way. They're stuck with it. But it's only good as a macroscopic variable if we can make predictions about how that variable changes over time, and we're not doing that. We're saying, "You're stuck with it," right? That's why it makes me a little uncomfortable.
I did actually notice that the active inference community has quite a ragtag character. It's very diverse.
Yeah.
So, in a way, you see people rubbing up against each other that you normally wouldn't, and that can create arguments, I suppose.
Yeah. Well, I think this was Karl's influence. What did Karl actually discover? He's got this link between information theory and statistical physics that, in some way, gives you this sort of uniform mathematical framework that's widely applicable to a huge number of situations. A lot of things that are baked into how we think about the world are kind of baked into it, and so it can be applied in a whole bunch of different areas.
Karl spent a lot of time basically evangelizing various aspects of the scientific community. "Look, you can apply this to epidemiology. You can apply this to the social sciences. You can apply this to physics." He just wrote variations on the same paper, but applied in different domains. This was intentional, because he wanted to show that this is a universally applicable mathematical framework. I think he's largely right about that.
As a result, there are all these people from all these different communities who have been pulled into his sphere and who think about the world very differently. It makes for some very entertaining conversations at the pub.
Yes, even in our Discord server, we've got people thinking about it in terms of crypto, Christianity, phenomenology, and psychology. It's really interesting.
But that's the beauty of constructing a nearly universally applicable mathematical framework, right? Exactly. You suddenly get—this is one of the things I love about the community, in fact—a relatively common language to discuss a huge variety of different things.
Yeah.
Now, of course, that means we often end up talking at cross-purposes, but that's half the fun, right? I often ask people in the business what changed—why we had this massive explosion in AI development over the last several years—and I get 3 common responses. I agree with every single one of them: autograd, the transformer, and the amazing ability to scale things up in a manner that we haven't really seen before.
I actually put an asterisk next to “transformer,” because many of the things that transformers have, which people believe the transformer enabled, I think resulted more from scaling. The evidence I like to cite is Mamba, which is a state-space model. It's basically a convolutional filter, but on steroids: they scaled it way up, and now Mistral has its very nice coding agent, and it works pretty darn well, right? They got a lot of the same functionality with a completely different architecture simply by virtue of scaling. So transformers get an asterisk.
I think the biggest thing was autograd. Autograd turned the development of artificial intelligence from something that was done by carefully constructing your neural networks, writing down your learning rules, and going through all that painful process that took forever, into an engineering problem. It made it possible to experiment with different architectures, different networks, different nonlinearities, different structures, and different ways of getting memory in there. All of this allowed people to start trying things out in a way that we couldn't do before.
And then what did we do? We suddenly discovered, “Oh, it turns out backprop does work.” When I was a young man, backprop was considered a nonstarter for 2 reasons. One is that it's not brain-like, which is true: the brain does not use backprop. The other one was the vanishing-gradient problem: “Oh, you'll never solve the vanishing-gradients problem. It'll always be unstable.”
And yet, nonetheless, once we turned it into an engineering problem and started playing around with tricks and hacks, certain kinds of knowledge, and ReLUs, we discovered that there are ways around this. We just weren't going to discover them by playing with equations; we had to actually start experimenting. As soon as it got turned into an engineering problem, that's what enabled hyperscaling, which led to all of these great developments over the last several years.
What got lost in the shuffle, though, was the notion that there's more to artificial intelligence than just function approximation. We got really good function approximators, but that's not the only thing you need to develop proper AI, right? You need models that are structured like the brain is structured. You need models that are structured like how we conceive the world to be structured, certainly if you want to have models that think the way we think.
That got lost in the shuffle, and we're starting to see the limitations, faults, and flaws of these approaches. We're starting to see them not living up to the hype. I don't know if you read the other day, but according to the experts in the field at the top of the best companies in the business, AGI is no longer a huge priority, and they're dialing back the rhetoric surrounding it. In part, I think they've begun to realize that just function approximation isn't going to deliver—or that was just hype, right?
We do need to do something different. We do need to start bringing in what we know about how the brain works if we're ever going to get to something that is a human-like intelligence. That was the starting point for us about a year or so ago: “Yes, let's do the same thing for cognitive models. Let's take what we know about how the brain actually works, take what we know about how people actually think about the world in which they live, and start building an artificial intelligence that thinks like we do by incorporating these principles.”
This means basically creating a modeling and coding framework for building brain-like models at scale. That's the critical element, because obviously scaling was a big part of the solution. Right now, most of the work in the active-inference space, as I'm sure you're aware, is not at scale. There's very little active-inference work that is active inference at scale; most of the models are relatively small, toy, grid-world-type models.
Part of the reason for that is that it is, in fact, difficult to scale Bayesian methods. That has also begun to change. We now have a lot of great mathematical tools and frameworks for approximating Bayesian inference. You'll never do it exactly, but we're approximating Bayesian inference, which I believe is how the brain works—Bayesian brain and all that. That allows us to build these structured models that are structured both according to how the brain is structured and how the world we live in is actually structured.
Hence this notion that what we need to get to the next level of AGI—and I don't like that term and don't intend to use it very often—is this framework that allows us to build the kinds of models that we know people actually use, and then make them bigger and more sophisticated, and so on. Hyperscaling Bayesian inference is part of it, but so is constructing models of the world as it actually works.
The way the world actually works provides us with the structure of our own thinking. The atomic elements of thought, as I like to phrase it, are models of the physical world in which we live. The physical world in which we live is a world of macroscopic objects that have specific relations and interact in certain ways that we understand. I'm looking around the room for a good example: you sit on a chair, right? That's an example of a relationship. It holds you up and all that fun stuff.
That understanding of the physical world was necessary for us to have in order to survive. Dogs have it too, right? Language isn't what makes it all that special—well, it's actually quite special—but those are the models that form our understanding of the world in which we live. That's where we get the models that form the atomic elements of our thoughts, out of which we have composed more sophisticated models that have allowed us to do all this great systems engineering and build this great technology that we've got.
So that's what we want to do. We're focused on building cognitively inspired models based on our understanding of the way the world in which we live actually works, because we believe intelligence must be embodied. We're building a framework for putting those models together and experimenting with them at scale, all in an approximately Bayesian way, because we believe that's how the brain works.
It's not just about putting your AI into a robot. It's about giving that robot a model of the world that is like our model of the world: a model that is object-centered, dynamic, and largely causal. That's the big difference. I think sparse, structured models are another key differentiating component.
When you think about how a transformer and an LLM work, a transformer takes every word in the document and says, “How does this word relate to every other word?” It does that many, many times. It's very much the same thing with your generative vision-language-action models: they operate in pixel space. They are microscopic models.
Now, yes, do they have an implicit notion of the macroscopic? Yes, they must, because they work. But it's implicit, and it's not implemented with the kind of sparse structure that actually exists in the real world and in our conceptualization of it. That's the thing that we're saying: “No, no, no. Look, if we want an AI that thinks like us, then we're going to build models that are structured like the real world is structured. They have this sparse, causal, macroscopic structure to them, and so should our models.”
The only way to do that is not just to put a robot in the real world, but to put a robot with a model that is structured in that fashion into the real world. No one's using xLSTM. Not many people are using Mamba, because why? All you need to do is scale the transformer as much as possible.
Many people just really think you magically get these things for free, right?
Dr. Jeff Beck
I think you could argue that with enough data, if it's the right kind of data, one of these really big, superscaled models will obtain an implicit representation of the world that is more or less correct.
Having an implicit representation is great if your only goal is to represent the world, if your only goal is to predict what's going to happen. But it turns out people do something which is very different. People are creative. People can solve novel problems. They can't just mine old problems and figure out where to move some words around to get an answer that looks more or less right, right? We actually are capable of creating. We're capable of inventing new things.
The way that we invent, I think, is exemplified by systems engineering, right? How does systems engineering work? I know how an airfoil works to create lift. I know how a jet engine works to create thrust, right? And I can take those 2 bits of information to invent something brand new, which is an airplane, right?
That kind of systems engineering was predicated upon having this sort of model of the world that was relational, right? Here's the wing. I can put a jet on it. I don't know—you don't staple it on. I'm sure you use rivets or something, right? I know how to put things together. I know how to construct new relationships and new objects.
An AI that is designed for systems, that is designed to do systems engineering, will have an object-centered or system-centered understanding of the world and will know how all of the objects relate so that it can start experimenting with different ways to combine them. Without that, the only thing you will ever be able to do is retool solutions for new purposes, and even that, I think, is a generous interpretation of what a purely predictive model is going to do, right?
So this is how I like to think about the principal advantage of taking this object-centered approach, right? It enables systems engineering.
What is a grounded world model?
Dr. Jeff Beck
That's a trick question. I actually had this conversation with one of my friends and co-conspirators, Maxi, the other day. In some sense, every model is grounded. It's grounded in the data that it was given. That's a true statement, but that's not what we want. When we use the phrase “a grounded world model,” we say that it's grounded in something, and that something is not just the data that it saw.
So, for example, vision-language models. A vision-language model is a way of grounding the visual model in the linguistic space, and this is the approach that we're taking. This is what LangChain does, right? It's all about taking models and everything becomes a language model—vision, whatever; everything becomes a language model.
What you're doing when you do that is saying that you're grounding all of your models in a common linguistic space so that they can communicate with one another via language, right?
Dr. Jeff Beck
Now, why did we choose language? We chose language because, honestly, I think it's because we wanted models that we could talk to, right? We wanted a model that—it was really all about making the interface convenient for us, which is great. That's totally something you want. But it begs the question: what's the right domain in which to ground your models?
We also use the phrase “ground truth,” and of course, ground truth is the thing you made up a priori and said was ground truth, right? So what's ground truth? What is the right domain in which to ground models in order to get them to think like we do? That's the relevant question.
My view is that if you want AI that thinks like we do, you need to have it grounded in the same domain in which we are grounded, and we are grounded in this domain, right? This is why the embodied bit is such an important thing. We want models that are grounded in the physical world in which we evolved.
The reason for this is because that is the world that provides us with these atomic elements of thought. A single cell lives in a soup, right? Whatever model it has of the world, to the extent that it has one—or it behaves as if it has one—that model is the model of its environment, right? If it didn't understand the environment in which it lived to some extent, then it wouldn't be able to continue to exist and function in that environment.
So you can sort of say that a cell has a model that's grounded in chemistry, in the chemistry of the soup in which it lives. We talk about that as a prerequisite for its survival.
Now we talk about mammals and bigger animals and things that live in the macroscopic world that includes other animals, right? What's that model? What's the world? Well, at the very least, we can say that whatever models we have, a significant subset of them are grounded in that world.
That world has properties that we can understand. It is object-centered, it's relational, it's all this stuff. The grounded bit is more about being properly grounded—grounded in the domain in which we are grounded—as a route to creating AI models that, in fact, think like we think, right? That's the grounding that we're particularly focused on.
If you had to choose the domain in which to ground your models, what would you choose? I don't think language is the right one. Language is an incredibly poor description of both our thought processes and reality.
I tell the story all the time, right? You ask any cognitive scientist or psychologist who's done some experimental work with humans. You put them in a chair, make them do some tasks, carefully monitor their behavior, and look at what they did. Then you have a nice way of informing your theory of that behavior, or however that works.
If you do the experiment well, you have a very good model of how they made whatever decisions they made throughout the course of the experiment. Then you go back and ask them, “Why did you do what you did?” and they give you an explanation. It sounds totally reasonable. It also is completely inconsistent with an accurate model of their behavior.
Self-report is the least reliable form of data that one gets out of a cognitive or psychological experiment, right? We don't want to rely on that. We don't want to ground our models in what we know is an unreliable representation of both the world and our thought processes, right?
We want to ground it in something that's a good model of our world. That's why we've chosen to focus on models that are grounded in the domain of macroscopic physics as opposed to language.
Can you speak a little bit more to the limitations of current active inference?
Dr. Jeff Beck
Active inference is a nearly uniformly applicable information-theoretic framework for describing objects and agents, right? It's really inspired by statistical physics and its links to information theory. When you take those 2 mathematical structures, throw in a little Markov blanket thing so you can talk about macroscopic objects, you have a very generic, widely applicable mathematical framework that you can throw at many problems.
A lot of what has gone on in the active inference community over much of the last 20 years has been demonstrating that it's uniformly applicable. So there's been a lot of breadth and not a lot of depth, right?
Of course, that's appropriate, given that if you really want to make the argument that everyone should be using this, you show, “See, in this domain it works on your toy examples.” But the people doing that—the active inference community—have had this habit of showing, “See, I can handle this. I can model this psychological phenomenon. I can model this cognitive phenomenon.” Oh, and look, it's a good post hoc description of this neural network's behavior and things like that.
They've been showing that, but they've never really sat down and tried to tackle any really big, really hard problem, because the emphasis has been on evangelism. You couple that with the fact that there's this strong bias within the active inference community toward being as Bayesian as possible. Of course, they also shun the really hard problems because Bayesian inference has historically been challenging to scale.
There have been a lot of developments over the last few years that have come out of the machine learning community as well, but mostly out of the Bayesian machine learning community, that have really made it possible to start scaling Bayesian inference in ways that we really weren't able to do before.
You couple that with a desire to stop evangelizing and start solving really hard problems with these methods, and you've got a way to prove that active inference really can live up to its promises.
Yeah, it was a similar thing with constraint satisfaction.
You know, in the 1970s there was the Lighthill report, and people said symbolic AI would never work and wrote it off. Apparently, there are all these empirical methods that have been discovered in the last 20 years that make it massively more scalable and tractable. Is it the same thing here? Are there some specific techniques that have dramatically improved the tractability of active inference?
Dr. Jeff Beck
I would lump it all into the Bayesian inference category. There have been a number of developments over the last, I would say, 8 years or so that have made Bayesian inference significantly more tractable than it used to be. Some of it had to do with work in the Gaussian process space. My current favorite trick is normalizing flows, which are a great way of ensuring that you have access to sophisticated likelihoods but nonetheless result in tractable probability distributions.
I've been using natural-gradient methods for a very long time, which allow you to massively speed up gradient-based inference and, in some situations, completely eliminate the need to do gradient inference and instead do coordinate descent. That allows you to take massive jumps in parameter space without actually losing the ability to learn sophisticated models in a sophisticated modeling scenario.
I also like the fact that the natural-gradient work has been getting some great acronyms recently, like Bayesian Online Natural Gradient, or BONG for short. I just think these guys get me every time. I wish I was that clever, honestly, but there have been a lot of developments in that space as well.
Additionally, there have been a lot of developments in rapid-sampling methods, conditional-sampling methods, and constraint methods that have really improved things. I think one of the problems with the active-inference community historically, which I think is now starting to change, has been a hesitance to use these sorts of approximate methods. There has been this focus on straight-up, old-school message passing.
As soon as you relax the desire to be as Bayesian as possible, it opens up a lot more possibilities for scaling this stuff up when we're talking about agents that are interacting with the world around them. And that still presumably needs a lot of data.
Dr. Jeff Beck
We've got a couple of tricks. One of the nice things about taking an explicitly object-centered approach is that you don't have to train all of your models. You don't have to train just one model at a time. This is my favorite trick, and I think this is one of those things we're going to be seeing a lot more of in the near future.
If you want to train a vision model to understand YouTube videos or something really complicated like that, you basically take one big model and train it on a ton of data. You just keep training, keep training, keep training, and eventually it gains this implicit, sort of object-centered understanding. Another way to go is to train objects in specific domains. These are smaller data sets. I'm only going to worry about the Zillow problem—the inside of people's houses—and that's going to have a much smaller set of objects that it has to learn an implicit distribution over.
You can do this with one big neural network. There's a really great Gaussian Splatting paper where they trained a massive neural network that is able to make predictions about what's going on inside people's houses, along with some nice language models. But obviously, its understanding is limited to a house and the objects that are inside a house.
If you have an explicitly object-centered model, then you end up not just with one model that understands a house. You end up with one model that's actually thousands and thousands of little models, each of which explains a single object or object class within the house. So you've got a book model: all books come in different shapes and colors, but there's just one book model.
The beauty of doing this is that you have to be a little clever about how you structure the interactions between these things. If you're a little bit clever about how you describe the relationships between objects within this modeling framework, you gain the ability to train a model just on the insides of houses and a model just on parks and park benches. You can take the objects that were discovered in this space and the objects that were discovered in that space and put them into a combined environment that has objects of both kinds, and it still works. That's the advantage of taking an object-centered approach, or what I like to refer to as the lots-of-little-models approach.
Some of these things are a little bit weird. Some cultures—maybe one culture—don't have the notion of time, and some cultures might see 2 objects as 1. Is there a potential problem here in that there's some ambiguity that we need to overcome?
Dr. Jeff Beck
I'm not going to say that there's not the potential for ambiguity that we need to overcome. What I will say instead is that the additional constraint we're imposing is not just about objects; it's also about their relationships.
Now think about physics. This is why the physics-discovery work is such a big part of it. In physics, and in particular Newtonian mechanics, let's pretend we're living in a world of rigid bodies. All I need to worry about is the weight and shape of things, and that defines a particular object type. But I also need to know how they interact.
In Newtonian mechanics, we can take these objects, watch them bouncing off each other and doing all these sorts of things, and quickly infer that their interactions are all governed by a single language: the language of forces and force vectors. That language of interaction is really what makes it work. Otherwise, we'd just have pictures of things. That's all we would have.
What you're empirically discovering is a generalized notion of forces that describes the relationships between things. The constraint that you place in order to avoid the problem of things being too brittle is that they all have to use the same class of forces in order to interact. We're stuck with that. But by being flexible about our definition of what a force is, and having the ability to discover new kinds of forces—not just literal force vectors—it gives us the ability to generalize without becoming too brittle.
You're talking about this interaction dynamic. So there's a graph of interactions, which might possibly represent affordances in the macroscopic domain. By doing analysis on the interaction graph and simplifying the analysis as much as possible, you get a principled way to partition the world up.
That's right. It's all about having interactions and interaction classes. It's not like there's just one adjacency matrix; there's an adjacency matrix that also specifies the type of interaction. There's one for every type of interaction that's possible. That's what gives you the additional flexibility.
The other thing that gives you additional flexibility is being a little bit Bayesian about things. It may very well have been that all of your observations of this object when it was in a house were really simple. It was all just, “It sits on a shelf.” So what do you know? You know that the object sits on a shelf, which is one kind of interaction. It has a force pushing down, and there's a force pushing up. You don't know anything about its weight, and you have priors.
But if you keep error bars around that, and around the other kinds of interactions that you have seen, while remaining agnostic about the specific details for this particular object, it gives you the flexibility to say, “I'm going to put it in this environment, and I can make some predictions about how it's going to behave.” If I throw a bowling ball at it, I'm going to be making some assumptions about how it might behave. But once the bowling ball hits it, I might have to revise those assumptions.
This is the other critical element of the approach we're taking: you have to have some kind of continual-learning element. This is something that really doesn't exist in contemporary AI. When you build your big model, you've spent millions of dollars training it, and then you're done. Yes, someone else can come along and fine-tune it a bit for a particular task, which is great, but at the end of the day, when you're at the deployment phase, you turn learning off.
Whereas in this approach, we're saying, “No, no.” One of the critical aspects of the way we think about the world and the way we learn about the world is that it's continual and interactive. That needs to be true of the objects that we're discovering as well.
We've learned classes of interactions, but just because we haven't seen a particular class of interactions previously doesn't mean we say the others never happen, right? We still allow for that possibility. Then we do continual learning quickly, with rapid updates when we see something happen—when we see a new interaction.
What makes that work is the fact that you've specified that there's a certain set of kinds of interactions, some of which you've previously observed and some of which you still don't know about and might observe soon. Then you can update your posterior beliefs about whether or not that object interacts in that way.
What would the architecture of such a system look like? I'm imagining it'll be distributed, right? We have all these different agents, and then we have the consistency problem, because maybe this agent has empirically learned that these 2 things are a book, but the agent over there just thinks this 1 thing is a book. Then there's the question of how many objects there are. Would it become intractable? Realistically, I don't know.
From a simulation perspective, the way that this gets simulated is remarkably like the way a video game engine simulates the world. The only difference is this abstract notion of forces. How does a video game represent the world? You have all these assets, right? Each asset is basically a shape, maybe a texture or a color. Something like a fork is an asset, or a little 3-legged stool is an asset.
It has a bunch of properties associated with its shape, color, mass, and all of this stuff. Then it has a set of interaction rules, which are like Newtonian forces, force vectors. Then you've got other things like water and sand that have special rules for them, because otherwise the compute would be insane. You need a macroscopic rule to describe them.
It's very similar to that, right? When you take this lots-of-little-models approach, what you end up with is the moral equivalent of a giant list of video game assets. When it comes to modeling a particular environment, when you find the agent that you're talking about, which has this lots-of-little-models model in its head, what it does is look at the scene and say, "Oh, okay. I need to worry about these 10,000 little models right now, and that's it. I don't need the rest of it."
Then it operates in that space, running something that looks a lot like a video game simulation. That sparsity is what makes this lots-of-little-models approach work. You may have 1,000,000 little models, but at any given time you only need a tiny fraction of them, and you just instantiate those.
The thought occurs, though, that in a game engine, all of these particles are in the engine. I can say, "What are the forces between these 2 particles?"
Yeah. It's called cheating.
Well, yeah. Because when you deploy an agent in the real world, you can't just ask, "What's the force vector between Jeff and the light?"
That's right. You have to learn those. If you take a video game engine as ground truth, are we capable of discovering the video game assets and their properties that were in that game engine?
What would your input be? Would it just be the pixels?
Yeah, why not make it hard? It would be cheating to start out with something that already segments the image for you.
Right? If you can't solve the hard problem from the bottom up, then it's not a hard problem. Why'd you do it?
If I understand correctly, a successful implementation of the technology you're talking about would be: let's start with a game engine, and we almost treat the AI like a black box. It has input—you know, I can move left, I can move right, pan up, down, interact with objects—and then maybe there's some kind of score function. I'm not sure, but it can learn inside the game engine, and it will build up this internal model library that represents things in the world, in the game engine.
If it's learned a sparse, robust model library, you could, in principle, take the same learned model and apply it to a robot in the real world, and it would generalize.
That's the idea, and that's the problem that we're trying to solve. This is one of the critical missing elements in the robotics space. Training models in simulated environments does not translate really very well to real-world environments.
This could be a result of a situation where the simulated environment is just too impoverished, but it could also be a result of a situation where the artificial environment isn't actually a very accurate representation of the real world. I think it's largely the latter: these environments, coupled with the fact that the artificial agent's internal model is not structured like the world that it's being trained to function in. I think those are the 2 biggest problems.
What do you need in order to address those? One is that you need a good model for the robot's brain that has the structure of the world in which it lives. The other thing is that you need a mapping from real-world data to simulated data.
Right now, what we're typically using is video game engines. Video game engines are great. I certainly enjoy them on a 10-hour-a-week basis. The problem with them, though, is that they weren't designed for realistic physics, right? Most of them were designed to be plausible. They were designed to look good to the user.
Part of this has to do with the fact that there are a lot of tricks and hacks thrown in to deal with the fact that the equations of Newtonian mechanics are very stiff, right? When collisions happen, if you're just a little bit wrong about that, weird stuff can happen, and non-physically realistic things can occur.
If you had the ability to construct an environment that had good enough physics to accurately represent the real world, and trained your robots in that domain where they have these models in their heads, so they're actually capable of learning the ground truth that you've implemented in the simulated world, then I believe that they will generalize better to functioning in the real world.
This is absolutely critical for robotics going forward, if for no other reason than right now, the way that we're currently training robots to put your groceries away and things like that is all by training them to mimic human behavior. It's expert trajectory learning. They're not really learning the physics of their environment. They're learning to mimic human behavior without crushing the eggs, right?
If you want them to be able to generalize across domains and across tasks, you need to get rid of reliance on expert trajectory learning. That only happens when you move to something that is explicitly model-based, with a model that accurately represents the world in which they live.
Once you've got a core set of models that work in the world, is that the value of the AI?
Yeah. Once you have a core set, then you have the ability to deploy your agent out there in the real world, and it can handle situations that it couldn't previously handle.
One of my co-conspirators likes to talk about the cat-in-a-warehouse problem. What do we have? We've got an AI agent that has been trained to manage a warehouse, right? It understands things like forklifts, boxes, and workers, hopefully, and it knows how to manage all of that.
Then one day something comes along that it has never seen before. It's a cat, right? Cats don't belong anywhere. A cat comes along, and the model has never seen a cat before because that's the environment in which it was trained.
This is one of the beauties of this approach. The cat comes into the warehouse and it's like, "What the hell is this?" It's screwing with my system. Because we're taking this free-energy-based approach, one of the critical elements is tracking surprisal.
When a cat comes along and the system doesn't know what a cat is, the surprisal signal goes crazy. Then it says, "Okay, stop. Don't run over the cat, right? Let's figure out what's going on."
What it can do is take a picture of the cat and fire it off to a server somewhere that has a huge bank of models and has been pretrained on model selection to a small extent. It says, "What the hell is this?"
Then the big bank of models says, "I think it's one of these 7 or 8 things." It could be different kinds of cats. Maybe there's a dog thrown in, whatever. Then it ports those little models over to the warehouse model and does some proper hypothesis testing, watching the cat behave for a little bit.
Ah, it’s a cat. It sends the other models back because it doesn’t need them anymore, right? It’s figured out that this is what it is. And now it’s incorporated an understanding of the cat into the system.
This is another beauty of taking an explicitly object-centered approach. It gives the model the ability to know what it doesn’t know. That comes from the active inference component: knowing what it doesn’t know. When it doesn’t know something, it can go phone a friend. That’s another way to describe it.
The friend will respond by saying, “Oh, it’s a cat.” Then it can take the model of a cat and incorporate it into its warehouse model, right? Now it understands that. This is really great from a compute perspective. There’s a huge compute advantage to this, right?
If we had started with one big model that already knew what a cat was, think of how many parameters it would have. It’d be huge. This model is very frugal in the sense that it only needs to know two things: what it needs to know about the environment in which it exists, right, and what to do when it sees something it doesn’t know. Then it can just go pull one.
That’s the idea: you have this massive bank of models. But when you instantiate a particular model for a particular use case, you don’t need them all.
Yeah.
Right. You just need the ones that are relevant to that environment. But these models are continuously tracking surprise or uncertainty. When they see something they haven’t seen before, they’re smart enough to say, “I don’t know what that is.”
How and when should deep learning be combined with this? My naive perception of Bayesian inference is that, right now, if you have a photograph from a camera and it’s 300 pixels squared or something, that would be a challenge for Bayesian inference. Could you just use a vision-language transformer or something and use that as part of the Bayesian framework? Or could you even use deep learning models as a way of bootstrapping knowledge acquisition in the Bayesian framework?
The reason why I mentioned normalizing flows is because that’s technically a deep learning tool. It just happens to be a deep learning tool that takes in an image and turns it into something that is easy to deal with from a probabilistic reasoning perspective.
Are we going to use deep learning tools? Yes, the ones that are fit for purpose, for sure. That’s a great example of one where we’re taking something and saying, “Why wouldn’t we use this if it’s compatible with our framework?”
Many folks in the audience won’t know what a normalizing flow is. Can you just give us a quick update on that?
We’ve got a pretty good handle on how diffusion models work these days, right? You take your image, add a bunch of noise to it, make it Gaussian, and then learn an inverse transformation. It’s the same thing, right? What you’re doing is learning a mapping from a probability distribution that is easy to deal with, like a Gaussian distribution, onto the thing you’re actually observing—the thing you care about. In this case, it could be an image.
In fact, I actually don’t think we should call them diffusion models. It’s a normalizing flow. The diffusion—it should be referred to as a diffusion training protocol for a normalizing flow. To some extent, we will be using some of those tricks as well. You could say, “Yeah, we’re going to use diffusion models,” if you’re going to make me roll my eyes and say that.
Jeff, what is your approach to alignment?
I typically like to talk to people about their beliefs and values, figure out how they came to form them, and then try to convince them to adopt my values.
The beliefs that these artificial systems have are not the same as our beliefs, and the reward functions that we specify for these artificial agents are definitely not the same as our reward functions. There are a few exceptions, like Go and chess, right? Any game where you either win or lose, the reward function is obvious. But in general, in complicated situations, it’s not so obvious what the reward function should actually be.
I know there’s this definite belief that reward is all you need, and there’s some truth to that. But the question is: Where did your reward function come from? From a philosophical perspective, there is no normative solution to the problem of reward-function selection, barring divine intervention. That’s just another fancy way of saying that your values and my values might be different, and it’s really difficult to say whose are better, right?
From a practical perspective, a situation that I like to point out is that, if you’re talking about self-driving cars, obviously you’d like to penalize your self-driving car if it drives over a squirrel, right? But if it had to choose between a squirrel and a cat, probably most people would want it to choose the squirrel. The way you would do that in an RL model is to say, “Minus 10 points for a squirrel, minus 50 for a cat.” Where did those numbers come from? It’s completely ambiguous, right? They’re relatively arbitrary. They’re kind of made up.
Relying on arbitrarily selected reward functions seems like a terrible idea. We also know that things can go horribly wrong. I know everyone’s sick of this example, but when you rely on reward, you’re effectively making wishes from a malevolent genie, or you run the risk of saying, “Hey, Skynet, end world hunger.” And it’s like, “No problem. Kill all humans,” right? If you don’t specify your reward functions very carefully, you can get very degenerate behavior.
The goal of alignment in an RL setting would be to somehow get my reward function, or perhaps humanity’s collective reward function, into the AI agent. This is really, really, really hard. It’s really, really, really hard because measuring reward functions is really, really challenging.
The approach that we’re taking is to ask: How do people actually do this? How do we, as humans, construct alignment? The first thing we do is try to figure out what other people’s reward functions are.
The problem of reward-function identification is confounded by the fact that people have different beliefs. Action, which is what we can observe other people doing, is a combination of their beliefs and their reward function, or their values. The problem, of course, is that you only observe people’s actions. There’s a difference of opinion about what to do, right? You want to figure out why. It could be because your beliefs are different, or it could be because your values are different.
But it’s ambiguous. Mathematically, it’s not even possible to separate these two. Belief and value are fundamentally conflated when all you observe is action or decision.
The way that we solve this problem as people is that we talk about our beliefs. I ask you, “Why do you think this is the action?” And then you tell me, “Well, it’s because this fact, this fact, and this fact suggest that if I do this, then this will happen,” right? Then I can say, “Ah, I see. Maybe the reason for the disagreement in our beliefs, or in our decision, is because you’re not aware of this fact and I’d forgotten about this fact.”
So what we do is say, “Let’s incorporate all of these things together,” and then you would still say, “Well, I still think we should do X,” and I’m like, “No, it’s still definitely Y.” We continue this conversation until each of us has a very reasonable model of the belief-formation mechanism that the other person has, at which point the only cause for disagreement is a disagreement about the reward function.
AI systems are completely illegible, and that’s almost a good thing, because if we actually understood how flawed they were, they would be banned, right? They’re amoral. We have no idea how to put morality into them.
The smart, safe thing to do is to remove decision-making from their capabilities and simply use them as oracles or prediction engines, right? Then we can just say, “Hey, what would happen if I did X, Y, and Z?” It tells you, “Well, this is the ultimate outcome,” and then we’re like, “Oh, okay, well, then maybe A, B, and C were better choices,” right, and things like that.
That prevents them from participating in the valuation of actions. That prevents them from using their reward function, right? You can get that just by training them to do good prediction. That’s totally great, but it doesn’t give us the kind of automation that we really want, right?
What we really want are decision-making artificial agents that can act on our behalf. So it’s either going to be a human in the loop, or it’s going to be something like what I propose, where we figure out how to solve the alignment problem in that fashion.
But Jeff, you’re an old-school cognitive guy.
So for someone like you, would you always think that, in the absence or in the light of explicit cognitive models, we would never be able to say that these things actually had beliefs or intentions?
I think that what allows us to currently say that they don't have beliefs or intentions actually stems a lot from our knowledge of how they actually work. I, for example, have no problem concluding that you have beliefs and intentions, though it may very well be that that conclusion is drawn from the fact that I really don't know how you work. I have an intuitive feel for it. I assume you work the way I work—I have beliefs and intentions. That's my perspective of myself, and so I conclude the same about you.
It's kind of like emergence, right? There's this whole class, this whole branch of the emergence literature, that defines an emergent phenomenon as anything that I didn't predict, right? Which is a remarkably anthropocentric and, I would argue, ignorance-based definition of emergence, and I don't like it for those reasons. The same sort of thing goes with this. I think the converse of that is what's going on here: we know that these algorithms do not have the capability to do anything other than predict, and so we don't believe they have intentions. But something like strong emergence usually means causal irreducibility.
Whatever definition of emergence you end up going with, it shouldn't be ignorance-based. It shouldn't be based on, “The only way I could have discovered this was by simulating it; therefore, it is an emergent phenomenon.” I don't even like that. I'm more sympathetic to that, but I prefer definitions of emergence that are more pragmatic, right? They're more like, “An emergent phenomenon—this is why I like downward causation as a fundamental feature of emergent behavior.” Mostly because downward causation is not only a nice explanation of when you can say a phenomenon is emergent—a nice, fairly rigorous definition of when you can say a phenomenon is emergent—but it also comes with a practical tool. It tells you that you don't need to model the microscopic phenomenon.
Last time we spoke about Lenia and Game of Life, didn't we?
Yeah. I'm still playing with that, by the way.
Oh, yeah. Some of the Lenia—one of my favorite Lenia simulations. This is not Particle Lenia; this is the traditional Lenia. What they do is, they have a field, and there are obstructions, squares and circles and things like that. Then they have these little creatures that are sort of like little amoeba-like swimmers—they've got fins in the back and everything—and they swim. They'll hit one of these obstructions, which will cause them to deform and kind of look like, “Oh, it's going to die. Hey, that's so sad.” Then it reforms and becomes itself again.
We thought of this as a really nice abstract environment in which to test some of the properties of the physics-discovery algorithm, because one of the nice things about the approach we've taken is that, as the little swimmer goes and hits something, it's possible that it loses its identity when it deforms into something new and then reforms into itself. We wanted to see if the approach we've taken captures that, and it more or less does. It hits the obstruction; it changes its identity into an object of a different type, then reforms and comes out the other side, and regains its identity.
Fascinating. Quick aside: we spoke last time about Alexander Mordvintsev, and he had this convolutional cellular automaton with the gecko, the self-healing gecko. He has now written a new paper with his friends at Google, and it's using logic gates. So it's an emergentist logic-gate thing that draws a Google logo. I haven't read it in detail, but it looks amazing, so definitely look at that. Now you're taking your system and applying it in something like Game of Life, basically, but you still expect it to work?
Yeah. Well, there are forces in Lenia, right? There's the rule that causes the pixels to change. It has a few properties: it's radially symmetric, and it can flip sign, but any radial symmetry can work. So it has a polarity. You can think of that as a force in a sense, and it's even a force that's kind of like real forces—a weird kind of charged-particle thing. I still think the approach we're taking is basically discovering the effective forces between things. We're not worried about the microscopic forces; we don't care. That's the whole point of macroscopic physics. There are microscopic forces that govern the behavior of the system as a whole, but what you're interested in are the things that make predictions on the scale you care about. So you're discovering the effective rules that describe the interactions between not just the particles or pixels that make up the little floater or whatever flyer, but the rules that govern its interactions with other floaters or physical objects like obstructions they put in the domain and things like that.
Keith really loves cellular automata because they are Turing complete and they have this miraculous ability to arbitrarily expand their memory. You can have a grid size that's this big and just add more memory and add more memory, and you don't have to train your thing from scratch. Using some of these approaches we've just been talking about, you can actually train the update rules—learn the update rules—of stochastic gradient descent. So do you think in the future we might actually have an AI system which is running inside a cellular automaton?
That is a very good question. The snarky response is to say, “Don't we all? Don't we already?” I mean, we've got it running on a computer, and at the end of the day, a computer is just a whole bunch of logic gates. So isn't it already a cellular automaton?
Well, it's in the same class of algorithms. But there seems to be a cellular automaton that has this emergentist thing: what it does is not how it's programmed. It feels like there's a trick—the way it's programmed is an order of magnitude less complicated than the thing it does, right? So it feels like a magical bridge to do stuff that's more complicated than we could explicitly program or learn.
I agree, but that also sounds a lot like a computer.
It's like, well, what can you do with a lot of—now, I guess the difference between a computer and a cellular automaton is that, with a computer, you program it—you tell it exactly; you specify something. Whereas in a cellular automaton, if you're training it to do something in particular—for example, find a bunch of discrete objects that go in a certain direction—you're allowed to tweak the rules that govern the local interactions until you get something that more or less does that. That's just programming in a sort of backhanded way.
Yeah. I think those systems are very interesting because it is remarkable that really dumb, simple rules can lead to really interesting, sophisticated behavior. But the thing that I find interesting isn't the fact that complicated stuff can result from simple local rules. What I find interesting—what I'm more interested in, rather—is: What are the properties of the resulting large-scale objects? How is that related to the small-scale objects? What's the mathematical description of those big things, the things that have emerged? I'm less interested in how they precisely emerge. This is probably because of my bias toward taking a human cognitive approach.
Most people don't, when you look at the Game of Life. When most people think, “Oh, that's really cool. Look at these pretty pictures and all these little creatures, and they're doing fun things,” they don't really care about the low-level rules. The thing that captures their imagination is the macroscopic, the high-level behavior of these things, though it is cool that you can get them from simple rules.
Yes. Yes. No, as you say, we can program computers, but there's the legibility ceiling. We can do program synthesis; it doesn't work very well.
It will. They're going to. I have confidence that that's not one of those things that I'm going to outright poo-poo. I do have confidence that it's a rich, relatively new area, and it has a lot of promise. To some extent, the approach we're taking is compatible with program synthesis, right? We're taking this object-centered description of the world.
The reason we're doing that is because we want to automate systems engineering. Well, what's systems engineering? That's like taking this object and attaching it to this one, attaching it to this one until you get something that does something really cool, right? Program synthesis is an abstract way of doing that.
You start with one program, attach it to another program, attach it to another program, and so on and so forth. There is this problem of just understanding the program, though. Going back to DreamCoder—and I'm sure Kevin and Josh have put other ones out more recently—some of the programs that are learned are just really complicated. They had examples of, I think, drawing towers and drawing graphs and stuff like that, and you just saw this huge confection of rules being composed together. It's great. It has many good properties in that it's a program, but it doesn't really make sense to us.
Yeah. To a large extent, I suspect that there are ways around that that are related to how it is that your AI coding agent actually works. For example, when they're doing this program synthesis, what they don't currently have access to is the kind of data set that GitHub has access to. They don't have access to a whole bunch of really well-written programs that do exactly what they were intended to do.
There was a paper in Nature. This was actually one of those situations where neuroscience is making interesting statements about machine learning, from Tony Zador. What he had done was take a whole bunch of neural networks that did a variety of different things, and then they came up with a way of genetically encoding them for the purposes of seeing what their structure was.
They had to have a layer that did this, and then a layer that did this. Then what they were going to do was compactly represent the weights in each layer and come up with a representation of that. Then I'm just going to look at a whole bunch of different neural networks and solve a whole bunch of different problems. Say, are there any patterns that are present in these neural networks such that, when I have a new problem I'm interested in, I can just take something that understands this genetic code, maybe mutate it a little as a way of sensibly traversing the space of possible neural networks until I find the best one? Program synthesis could in principle exploit the same trick. They just need the data set to do it.
Yeah. Yep. What are humans in a world where everything can be done by a robot?
Yeah.