[百万 AI 工程师特别篇] 斯坦福如何教授 AI 原生工程——Mihail Eric,Monaco
- AI 原生招聘正在把入门门槛抬向资深工程师的判断力。 Mihail Eric 告诉斯坦福学生:“每家公司都需要你,但门槛高得多了。”实现工作正越来越多地交给智能体,架构设计、系统权衡和理解力则仍然值钱。他的应对办法是让学生通过真实的开源贡献接受学徒式训练:维护者教会他们判断力,而智能体光靠生成代码给不了这些。
- 代码生成提速后,验证能力和意图的持续维护成了瓶颈。 当每天有“30多条 PR”时,传统审查会让人类成为吞吐量上限,但不加甄别地合并又背离了工程的意义。Eric 支持以可验证性为基础,有选择地自动批准;主持人则追问一个尚无答案的问题:如何让一项功能积累下来的要求,贯穿许多单看都合理的改动。
- 简单的智能体循环很容易复现;让它在大型代码库里可靠工作则难得多。 Eric 用200行代码讲清一套实用的编码 harness,再通过拦截请求展示生产环境中的提示词。真正难的是实现前先做研究、在多个智能体间分配上下文,以及随着能力沉淀进模型权重而删减指令——同时不能让自动化取代理解。
- Monaco 的差异化不只在智能体 harness,也在分销、数据增值和销售专长。 这家 B 轮公司瞄准尚未深度绑定 Salesforce 的种子轮至 A 轮客户,希望伴随客户一同成长。Eric 介绍了其经过增值处理、涵盖公司、客户账户和联系人的“世界数据库”。主持人用销售副总裁跳槽说明买家时机;Eric 谈的是如何找到合适的买家与时机,并未表示 Monaco 依赖这一具体信号。
- 垂直智能体公司可以先赢得用户,再补齐模型经济性。 Monaco 初期用前沿模型探索能力边界,如今正研究开放权重模型、后训练和 RL 环境,以降低成本、提升速度。其工程原则是“买来学习,自己建来留用”;不过,Devin 在公司内部自然获得采用,也说明好用的外部产品可以持续占据一席之地。
- 安全自动化在发现漏洞与理解实际发布内容之间留下了缺口。 依赖项之争有两面:重建组件能减少外部暴露,却会把尚未发现的错误集中到私有代码里。主持人质疑“模型变强就能解决安全问题”的假设,强调敏感数据、计费、认证、授权,以及代码生成速度超过审查后带来的认知缺失。
- 网关整合在架构上说得通,但买家仍可能保留单点解决方案。 Eric 认同 MCP 门户和 LLM 网关处理的是相互重叠的治理问题;Swyx 用 OpenRouter 的成功反驳了自己偏好的捆绑思路:专注模型路由也能做成生意。“你买的是你的组织架构”——企业内部的归属和采购边界,可能比统一平台的吸引力更有决定性。
- 软件工厂必须靠产品证明价值,而不是靠编排本身制造动静。 对话将人类仍需掌握的控制权概括为“意图”和“审查”,主持人则批评那些主要产出更多自用基础设施的系统。他们建议学生提高目标,尝试构建浏览器这样的项目,或扎实做好基准测试和评测;复杂的智能体活动本身并不是进步的可靠衡量标准。
1. 初级工程师的门槛正在向资深判断力靠拢
Eric 在2024年11月左右构思了斯坦福 CS 146S《现代软件开发者》课程,之后花了大约9到10个月设计课程方案和教学内容。课程于2025年秋季首次开设,当时还在“Opus 4.5、4.6 days”之前。第二版课程反映出自主编码智能体自首轮开课以来对工作方式带来的巨大变化。
第一版课程覆盖软件开发生命周期中的智能体应用:规格制定、实现、审查和 SRE。新一版更强调 Eric 认为更具长期价值的原则,尤其是上下文工程、架构,以及在生产代码库中工作。他把教学和实践结合起来:既在斯坦福授课,也担任 Monaco 的 AI 负责人;客座讲师则带来超出他自身经验的前沿实践。
这门课明确不是编程入门课,也不是计算机科学的替代品。算法等基础课程是先修要求。Eric 希望学生先理解系统如何运作、其中有哪些权衡,再把更多实现细节交给智能体。他对就业前景的判断乐观,但要求也更高:“每家公司都需要你,但门槛高得多了。”
Eric 认为,初级工程师不能再默认达到 Google L3 水平就够了;学生应当向接近资深工程师的能力靠拢。Swyx 故意用挑衅性的说法概括这一分化:大多数学生“没戏了”,但少数人会变得极其抢手。主持人也承认,斯坦福学生的就业前景并不能说明更广泛劳动力市场会发生什么。
2. 开源维护者成为缺失的学徒制
去年的课程由9个作业和一个开放式毕业项目组成。Eric 举的例子是给学生一个代码库,让他们使用 AI 审查工具,再评估工具的输出。这类练习很快变得太容易,学生可能“一次提示就做完”。Eric 在课堂上非正式调查发现,所有人都在用编码智能体;他说,现在基本没人手写代码了。
新课程把学生直接带进大型开源项目,包括编码智能体及服务智能体的工具。贡献代码成了课程作业,而非可选的课外活动。关键变化是让学生面对已有系统和真实约束:他们必须读懂陌生代码、做出有用改动,并满足那些对接受或拒绝实现方案有充分理由的维护者。
Eric 认为,导师制可以弥合资深工程师借助智能体获得巨大杠杆、而初级工程师缺乏底层心智模型之间的差距。维护者审查贡献,并解释什么是“好软件”,通过真实工作传授设计判断力。表现出色的贡献者还有机会被聘用。教育和招聘在这里相互衔接:学生直接向评估其代码的人证明能力。
算力不应成为限制条件。Eric 已与 Factory、Cognition 和 OpenAI 等参与方协调,为学生争取额外额度,超出常规教育优惠。他不希望“每月20美元的订阅费”阻止学生探索能力边界。目标是让学生在有助于实验时,能够“把 token 用到极限”。
3. 200行 harness 让智能体循环一目了然
Eric 把编码智能体拆解为模型在工具间循环调用的机制。一个能编辑文件的实用版本大约200行代码就能实现,所需知识不超过斯坦福入门算法课 CS 106B。它没有子智能体、MCP 等功能,但足以消除一种印象:核心机制依赖外人无法获得的“秘方”。
接下来,学生会从这个迷你实现转向检查生产系统。Eric 使用 mitmproxy 拦截请求和响应,展示系统提示词、规划提示词和压缩提示词。他说自己用的是“Opus 5.5”,当时上线才两天。这个学习路径很重要:学生先作为工具使用者,再搭建基础机制,最后检查真实 harness 在其上增加了什么。
主持人承认,自己早期低估了 Claude Code:一次查询大约要50美分,他们当时问:“谁会用这玩意儿?”Eric 记得 Boris 曾提醒他,要为模型半年后的能力做开发。此后模型持续进步,解锁了长时间自主运行,也大幅改变了课程应该教什么。
Harness 的复杂度会不断变化。主持人谈到约20,000 tokens 的系统提示词、一次缩减80%,以及之后又扩张再收缩的过程。Eric 强调,Git 和任务处理等指令正从提示词迁移为模型自身能力。较弱的模型可能需要示例,较强的模型则可能被示例过度限制。他早期偏好的设计原则是“少量工具承担更多任务”,使用 Bash、SQL 这类强力工具。
4. 上下文工程需要研究,也需要有选择地记忆
Eric 发现,能超越基础提示词、“帮我把这事做了”的从业者出奇地少。遇上陌生的百万行代码库时,这种办法就会失灵。他的工作流程会先做研究、提出备选方案,由人决定方向,然后再制定计划或规格,最后实现和测试。研究阶段的目的,是在智能体开始改代码之前,先确认它需要了解什么。
Swyx 提出的未解问题是如何在多个智能体间分配上下文。子智能体应继承整段对话,还是从头开始?返回结果前要压缩多少发现?审查智能体可能需要广泛了解所有人的工作。“我不喜欢我现在做的任何一种办法,”他说;单靠一份大规格说明解决不了协作问题。
上下文越多,也会带来新的错误。主持人描述了反复扩充、收缩智能体上下文的过程:一开始知道得太少,之后又可能过度依赖对话早先的一句随口评论。“它们把那句话当成了铁律。”Eric 认为,这仍是艺术多于科学的领域,需要有意识地整合、刷新信息,并留意上下文膨胀。
主持人建议读一些易读的研究,了解系统提示词变化及上下文管理的下游影响,但并不假装每篇论文都能给出定论。Eric 的实用建议更简单:每天花10分钟做实验。用新工具重新解决一个熟悉问题,往往就足以缩小很大一部分认知差距。
5. Skills 应保留理解,同时自动化流程
Eric 区分了工具调用和流程。MCP 可以接入外部函数或服务;skill 则把可重复的一系列操作写成类似 SKILL.md 的文件。他希望学生亲自把流程形式化,而不只是找到另一批现成提示词来消费。
他偏好的教学示例包括 Matt Peacock 的“Grill Me”和 Jeffrey Blitz 的“Explain Diff”。后者会对改动做可视化解释,再就改动内容考问开发者。最后这一步才是重点:扫一眼漂亮的总结,不等于证明自己理解了发生什么。skill 让用户参与学习,而不是只接收生成结果。
主持人不认同产出量等于生产力。如今一句话的要求就可能生成数万行代码,让作者与系统脱节。他们反复强调一个原则:任何情况下都不该“关掉大脑”。好用的 skills 往往会提出更多问题、迫使用户思考,正因为实现代码已经变得更容易。
6. Skills 本身也带来维护问题
Swyx 认为,自己写 skills 比收集网红推荐更重要。简单版本往往质量不佳,包括默认 skill 创建工具生成的内容。他维护着一个“skill cutter”,用来删掉反复出现的问题和冗余内容。难点在于,有用的指令可能夹在冗长输出中:编辑时要保留真正能改变行为的部分,而不是不加区分地一味删短。
最糟糕的失效方式,可能是 skill 在无关项目中意外触发。描述写得太宽泛,就会让原本为某个旧任务准备的流程被调用。Eugene Yan 每隔几周清理未使用 skills 的做法,是一种应对方式。Swyx 猜测 OpenAI 和 Anthropic 有些人可能希望 skills 消失;他明确表示这只是猜测,并称堆积成山的 Markdown 文件是个笨拙的终点。
另一位主持人的反驳是,易用性很重要:用户甚至不用打开文件,也能从中获得相当大的价值,只需说:“看看我们上次5次是怎么做的,给我做个 skill。”Eric 认为 Monaco 有一个可能更经得起时间考验的用途:把组织知识编码进 Devin 的审查流程,包括公司偏好的 API 构建方式和服务边界。这不同于保存一条随手写下的重复提示词。
7. Monaco 选择在 Salesforce 还不难替换时切入
Monaco 是一家总部位于旧金山的 B 轮公司,正在打造一体化的 AI 原生销售平台。其早期客户是种子轮到 A 轮为主的初创公司。Eric 的切入点,是在团队还用 Google 表格追踪销售管线时就服务他们,再伴随客户一起成长。要让一家已经成熟的企业离开 Salesforce,难度要大得多。
公司花了一年多时间隐身开发,目标是打造一体化产品,而不是从一个狭窄的销售单点工具起步。Eric 说,过去一年半里他们对一体化模式有了很多认识,现在正向更大型客户拓展。他把这种分销模式比作 Brex 等公司:从新兴企业起步,随着客户发展而扩大业务。
Monaco 自有的 harness 支持多种销售工作流,包括 TAM 和 ICP 评分、ICP 定义、销售管线变化、会议准备和摘要。其经过数据增值处理的“世界数据库”覆盖公司、客户账户和联系人。主持人用销售副总裁换工作举例,说明这可能是推荐销售平台的好时机。Eric 说,需求来自找到合适的买家和合适的时机,但没有说 Monaco 使用了这一具体的跳槽信号。
公司使用 Parallel 和 Exa,Eric 不愿对两者排序,同时也自行做数据增值。他认为另一项差异化来自前线部署客户经理,即 FDAE:这类销售人员帮助客户完成入驻并搭建工作流。Monaco 正把自家 GTM 团队的经验编码进产品,其中包括曾在 Brex 工作的人。Eric 承认,没有这些领域知识,产品会更趋同质化。
8. 垂直智能体可以逐步补齐模型经济性
Eric 对垂直 AI 的起点判断是把前沿模型当作外部服务,先探索可能性。产品成熟后,成本和速度的重要性上升,于是需要考虑开放权重模型、后训练和 RL 环境。Monaco 目前正处于这一阶段。他提到围绕 Harvey 毛利率为+50%还是-50%的争论,借此引出对经济性的审视,同时明确暂且不讨论这些数字是否准确。
Swyx 把这套做法称为“智能体实验室打法”:先拥有用户和产品,需要什么模型就用什么模型,然后“再把模型侧补上”。模型实验室的路径正好相反,从预训练起步,再寻找用户。两者之间的关系依然复杂,因为基础模型供应商也可能与它服务的垂直应用竞争。
9. Monaco 先买来学习,再围绕自有基础设施建设
Monaco 的后台智能体平台 Monocoder 基于 coder.com 的 Coder,并按内部需求做了改造。Eric 介绍,它支持隔离环境、多个智能体和长周期编码任务,角色类似 Ramp Inspect。他选择它,是因为与 Monaco 现有云基础设施契合,而不是坚持所有工程工具都必须自研。
Monaco 的 Slack 智能体 Hal 覆盖 AI SRE、客户成功和 QA。它能协助调试问题并提交修复 PR,销售人员也会用它了解产品行为。这种广泛用途带来一个实际设计问题:同一底层系统在回答开发者和向销售人员解释问题时,需要展现不同的“个性”。
Eric 更正自己的说法,明确了总原则:“买来学习,自己建来留用。”外部工具能帮助团队发现自身需求;当现有产品还不成熟或不合适时,内部系统才更有吸引力。他强调,Monaco 仍愿意重新评估供应商。代码审查是一个重要例外:他们至今仍觉得外部软件有用。
第一次评估审查工具时,他们考察真实 PR、评论类别、冗长程度和实用性,没有供应商达到门槛。后来试用 Devin 后,工程师主动申请访问权限,产品便逐渐铺开。主持人质疑这会不会只是“凭感觉”;Eric 区分了之前的结构化分析和之后的自然采用。第一次评估失败后,他不想再强推一次部署。
10. 智能体就绪度始于常规工程基础
Eric 把智能体就绪度视为并行智能体、后台系统和软件工厂的前提。团队在启动数十个智能体前,需要有一个自己能理解和验证的代码库:结构合理、边界清晰、有文档,也有反馈机制。他认为 Factory 很早就把这一类问题讲清楚了。报告或仪表盘固然有用,但真正让后续自动化成为可能的是底层工程条件。
有些公司的瓶颈没那么光鲜:合并队列积压、CI/CD 太慢,或者搭建一个应用要花太多时间。“答案也不总是多上几个智能体。”再加一个 SRE 机器人,并不能解决系统性的基础设施问题。
当系统有庞大的依赖图、Kubernetes 基础设施和散落各处的 Helm 文件时,AI SRE 的价值就更明显。Eric 称赞 Resolve 和 Night Shift,并用福特或沃尔玛级别的企业说明复杂度。主持人提到,有的组织拥有数万个代码仓库,有时仓库数量甚至超过开发者人数;已经离职的作者留下的代码仍然“承重”。规模较小的系统则可能仍然足够容易管理,适合自建辅助工具。
Resolve 曾为课程提供一个包含多个服务并植入缺陷的沙箱,让学生扮演值班工程师,借助其智能体排查问题。更大的目标是闭环:智能体应能检查构建日志,理想情况下还要接入生产分析。主持人指出,许多复杂的触发机制最终不过是通过 cron job 运行,但这并不影响运行时反馈的重要性。
11. 每天30多条 PR 让审查成为吞吐量瓶颈
智能体每天能产出“30多条 PR”后,传统审查会让开发者变成瓶颈。Eric 既不接受每项改动都必须由自己把关,也不接受什么都直接合并。问题在于,如何在保持合理信任的同时,让生成能力带来的效率提升传导到系统其他环节。否则,“大规模生成的意义是什么?”
Eric 认为,清晰的架构和可验证性可以让某些类别的改动适合自动批准,但并非所有改动都符合条件。对于其余部分,团队必须决定人类该检查每一行代码、规格,还是其他表达形式。他大体认同 Aviator 的 Ankit 关于取消传统审查的观点,同时指出还有另一个瓶颈:决定应该构建什么。
Monaco 会投入精力编写详尽的 PR 描述,把它们作为浓缩版规格说明,包含文件级定位信息和改动边界。一个段落的摘要远远不够。Eric 将这份实现记录与 issue 阶段更早的规格区分开来。主持人还指出,审查活动并没有随着合并 PR 数量同步增长,这进一步加大了重新定义审查方式的压力。
12. 局部 PR 规格无法保留整个系统的意图
主持人提出的反驳是:一条 PR 在内部逻辑上自洽,也仍可能违反长期积累的要求。功能是多次改动的总和,而不只是最新实现说明。模型可能推翻5周前有意加入的内容。人类成了记得它为何存在的“规格之神”;对主持人来说,单靠指示模型维护更广泛的理解,并没有可靠解决问题。
一位主持人以会计公司 Basis 为例,说明可以针对不同用途对不同产物进行版本管理。一种记录未来预期行为;另一种记录“代码的规范状态”,包括原本隐含的知识。Eric 认可这个例子,但主持人只是把它视作一种可能有用的做法,并非证明业界已经解决如何共同维护规格和实现的问题。
Swyx 介绍了一个探索中的“程序数据库”,但不确定会不会真的发布。它会记录执行轨迹,让规格成为持久对象,管理附加的 skills,并支持比 Git 锁定模式更高的并发。他质疑的是表达方式:当假设、约束、意图和多个智能体的工作都需要协调时,只把代码当作事实来源过于狭窄。
他认为 Linear 可能胜出,因为它已经承载任务和意图。Eric 把可能性扩展到其他领域:收集客户反馈或分析数据的 B2B 产品,也可能通过把问题闭环到产品改动而转型为编码智能体公司。Linear、Amplitude、Decagon 或 CRM 都可能把观察到的问题变成行动:“这里出了问题,去把它解决。”
13. 去掉依赖项,是把外部暴露换成私有错误
在安全讨论中,一位主持人提到成熟安全公司 Semgrep 正在为智能体扩展产品,并强调依赖项追踪和供应链纪律。实操建议很基础,但后果重大:依赖项应按需锁定,不要随意运行 npm install,同时假定整条依赖链都可信。自主操作和大量生成代码扩大了需要关注的攻击面。
Eric 提出一种可能:最安全的代码库没有任何依赖项。主持人表示赞同,并进一步推演:代码变得便宜后,内部从头构建一切似乎更安全。Eric 随即反驳,这种说法很危险,因为团队可能只是自己犯错,而不是依赖别人发现错误。一个有100,000名用户的包,会有很多人有机会发现漏洞;而公司自己的用户可能永远不会发现或报告同类问题。
Eric 补充说,开源本身也可能发生变化。他认为,大多数开源仓库,“如果不是全部”,最终都会变成由智能体运行的软件工厂,并以 Warp 的终端为例说明这一趋势。如果每项贡献被人检查的机会减少,传统上依靠广泛人工审查来论证开源安全的逻辑就不再那么直接。他也不确定这样的代码库最终会更安全还是更脆弱。
主持人讲到 Vercel 曾提出将广泛依赖的 C 和 C++ 软件移植到 Rust,预算可能约为600万美元,希望做出更快、更安全的实现。他们并不把这当成模型能消除安全问题的证据。主持人的质疑仍然成立:如果生成式重写引入了另一种漏洞呢?能力增强会改变失效方式,但并不能证明失效会消失。
14. 更会找漏洞,不代表清楚自己发布了什么
讨论“Daybreak”和“Glasswing”时,话题转向给予防御方优先访问权并延迟发布,是否能解决安全问题。主持人质疑其中的时间逻辑:即使给防御者3个月,攻击者之后仍可能花6个月试探。把网络安全模型交给100家重点机构,也会新增100个可被攻破的目标。这些反驳针对的是一种假设:发布顺序能够解决根本的不对称性。
另一位主持人强调,关键不在平均漏洞率:模型平均可能少犯错,但“你甚至不知道自己发布了什么”。开发者越来越多地既不编写、也不审查自己交付的全部代码。代码扫描和安全审查依然重要,但理解力流失是另一类问题,不能等同于模型漏洞检测能力如何。
Eric 认为,数据存储、计费和认证尤其敏感,并说:“基本上,不要在这些地方写代码。”一位主持人认同这一担忧,区分认证和授权,并建议先学习 AWS IAM 之类的概念,再让智能体实现相关功能。正如这位主持人所说:“有了这套语言,你才能把要求写进提示词。”Eric 也认同,即使实现变容易,这些概念基础仍不可少。
15. 后台智能体让非技术工作也有了隐藏代码库
后台智能体不只是 IDE 里的工具,还涉及异步工作、CI/CD、代码仓库事件和运维触发机制。Eric 计划介绍 Stripe Minions、Ramp Inspect 和 Spotify 的内部系统。他也称赞 Cloudflare 作为一家同等规模的上市公司,在采用情况和公开分享经验方面都做得非常出色。共同的问题是:当没有人主动发出提示时,软件团队如何让智能体继续发挥作用?
主持人认为,编码智能体已经藏在一些表面上看似非技术的产品和工作流之下。Swyx 在 Codex 中运行 SEO 和 AEO,因为持续研究需要脚本、数据存储和可视化。营销人员可能从不查看底层代码,但长期运行、多步骤的工作流依然能从代码库中受益。可见界面不必暴露背后的工作机制。
Eric 接受非技术场景可以通过代码实现,但认为这与入口界面是两个问题:开发者是从消费级助手进入,检查它背后的计算机,还是切换到开发环境?主持人认为产品正在融合;Eric 则追问,这是否意味着所有工作都必须从同一个产品入口开始。对话没有给出明确答案。
16. 网关在技术上能整合,销售时却面对不同买家
Eric 仍然看好 MCP 作为现有标准,但主持人区分了这一点与 MCP 门户是否能成为生意。规模扩大后,Eric 认为组织需要统一入口来管理治理、安全、审计和成本优化。他的判断有前提:当许多工具和服务需要接受共同管控时,价值才更加明显。
Swyx 认为,门户可以并入 LLM 网关,连同授权和计费一并处理。这些本质上是同一个架构层的不同组成部分,也就是在多个用户和多个服务间发挥中介作用的“上帝盒子”。Eric 接受整合思路,并解释说在课程里分开讲这些概念,部分是为了教学。把相关控制集中起来,也能让它们彼此配合、相互依赖。
OpenRouter 是 Swyx 用来反驳自己偏好架构的例子。它专注模型路由,没有同时包揽可观测性、运维和 MCP 接入,却仍然成功。“我很乐意这样被证明错了,”他说:边界明确的产品可以让工程和营销都更聚焦。架构上有所重叠,并不意味着一家供应商就应该接管所有相邻功能。
Eric 指出,企业里往往由不同团队分别负责技术栈的不同部分;要整合各个端点,需要跨越这些边界达成共识。Swyx 将其概括为“你买的是你的组织架构”,以及“如果是给所有人用的,那就没人会买”。Eric 提到,平台工程正成为越来越重要的角色,职责覆盖云基础设施、开发者体验和安全,而不是某一项应用功能。
17. 软件工厂仍离不开意图、审查和触达客户的路径
主持人把工厂里人类剩下的控制权概括为“意图”和“审查”,并认为审查可能比意图更早松动。Eric 对课程的定位也类似:追问人类还在哪里发挥作用,并点名意图和可验证系统。在他的角色类比中,技术监督者要足够了解各组成部分,才能设定方向、指标并做出调整。他提出的89%代码由智能体编写只是一个假设;他说,即便前沿工厂供应商也声称,完全由智能体驱动的贡献“也就30%多,甚至还不到”。主持人提醒,曾经的“暗工厂”支持者也公开改变过看法。
销售整座工厂是另一回事。Eric 认为,采用过程可能需要分阶段推进:先部署编码智能体,再加入后台工作或 SRE,最后把各部分连接起来。把一整座工厂直接放到福特的办公桌上不现实。主持人猜测,许多公司都想自己建设,因此基础设施供应商和内部平台之间的商业边界仍未确定。
主持人引用了一篇 OpenAI 文章:团队花5个月搭建内部工具,手写代码行数为零。初期速度变慢、之后才逐步卸载工作的过程说明,harness 工程需要学习,不会立即带来加速。Warp 与课程合作,则让学生有了另一种参与方式:为工厂做贡献并帮助维持运行,而不只是提交一个常规功能后离开。
Eric 用 CEO 作比,认为 CEO 要足够了解各组成部分,才能设定方向、指标并做出调整。技术监督者同样需要权衡产品需求和工程约束。主持人不愿断言产品和工程岗位会立刻合并:最近有人问 Anthropic 的平台产品和工程负责人是否会合并职责,得到的回答仍是“还没有”,尽管两边的工作正越来越重叠。
18. 编排若只产出编排本身,就会变成陷阱
围绕 Steve Yegge 的 Gas Town,讨论焦点是那句据回忆曾有的批评:“Gas Town 唯一做出来的有用东西就是 Gas Town。”主持人认为这说明系统没有对真实产品负责。复杂编排、不断增加的 token 消耗和看似忙碌的活动都不够;如果系统主要是在维持自身开发,就谈不上成功。
Swyx 说,他希望招聘那些经历过智能体“精神错乱”又走出来的人。他描述自己支付员工薪水和 token 费用,却几乎得不到产出,而且反馈也听不进去。Eric 猜测,Gas Town 可能也超前于时代,或者很难理解;主持人则反驳说,它的宣言和术语恰恰帮助这个概念传播开来。
他们讨论的 Boris 框架,把团队采用分为5个阶段:受控、辅助、并行、监督式自主和 AI 原生。一位主持人说,相关材料把 Anthropic 描述为处于第3阶段、试图走向第4阶段;Eric 开玩笑说,这样看来 Boris 本人就属于第4阶段。这个区分很重要:个人高级用户的工作流,不等于整个组织都以同样方式运作。
“动态工作流”提供了一种超越复制父智能体、再派生子智能体的具体进展。系统会按需编写任务专用 harness,再运行它完成工作。Eric 说 Monaco 已为 GTM 场景构建了一个版本。这种模式做的是专门化、可执行的编排,而不是假定每项委派任务都需要另一个一模一样的通用智能体。
19. 新一届学生需要更高目标和更好的衡量方式
主持人预计,PR 贡献率或合并率两年后会显得荒谬,无法再作为工程指标。但替代指标仍不清楚:收入离代码太远,客户满意度衡量的可能更多是产品决策,而不是代码。Eric 的回答是训练学生具备整体系统和产品工程判断力,把一项技术贡献放进正在构建的整体产品中理解。
Eric 用 YC 的创业需求清单作为毕业项目灵感:“去创办一家公司。”去年,一位设计师把笔记本上的草图拍下来,生成了一个能运行的 Vercel 应用;Eric 认为这可能帮她拿到了 Vercel 的工作。主持人认可这个成就,同时也指出标准已经提高:把照片变成应用,现在远不如写一个浏览器有挑战性。
Eric 说,支持一次性实验可以动用1000多个智能体;提供额度,是检验学生抱负、识别优秀开发者的一种方式。与此同时,许多公司还停留在单次 ChatGPT 调用或刚开始使用 Cursor。Eric 强调,这种差距比硅谷想象的要大得多,也给理解前沿方法的学生留下了机会,让他们可以在其他地方发挥实质作用。
主持人提出“第11周”的直接建议:“做基准测试。做基准测试和评测。”扎实的工程文章和有用的评测工作,能为人打开进入前沿模型公司和智能体实验室的大门,也可能成为一门生意。他们以 SWE-bench 为例:这是普林斯顿本科生做的项目。建议是深入研究一个重要问题,做到其他人能够评估并使用你的成果。
完整逐字稿
本集逐字稿暂无译文,当前显示原文。
Okay, this is a pretty special episode because we’re celebrating one million subscribers on the AI Engineer YouTube channel. Go over there and subscribe if you haven’t heard about it already. Our guest is Mihail Eric, who is teaching The Modern Software Developer at Stanford, which is going viral for covering AI-native development best practices, and is also Head of AI at Monaco. He’s not only a teacher, but a practitioner. Those of you coming to AIU Code Summit next week will be celebrating all the extreme progress in coding over the last year. I’m excited about the themes of fighting slop, system one models, and increasing your ambition for major rewrites of basically everything you’ve seen in code. There’s never been a more exciting time to be an AI engineer, and with one million AI engineers, I think it’s just getting started. So let’s get into the interview.
Okay, we’re in the studio with Mihail Eric, adjunct professor at Stanford and head of AI at Monaco. Welcome.
Thank you. I’m glad to be here.
You caught our attention because of your course, which you’re refreshing for the third time—third year?
Second time.
Second time?
Yep.
CS 146S, The Modern Software Developer, at Stanford. This particular generation of junior developers and junior engineers—the people who are now entering the workforce—will, I think, be the first generation of that new shift. I looked at the syllabus and thought, “This is everything we cover.” Every now and then, I like to invite people like yourself, who cover a broad range of topics, so we can go topic by topic and give people the state of the art on each of these things, which I assume you’re doing with all your guests anyway.
For sure.
Tell us the history of the course. What made you want to spin out CS 146S?
This is the second time it’s being taught, and I taught it for the first time last fall, in fall 2025. This was before the Opus 4.5 and 4.6 days. Even before fall 2025, the course had been about a year in the making. The genesis of the class was around November 2024, when I had this idea that we were seeing all this stuff happening with coding agents. They were an earlier version of coding agents, but already it felt like something was drastically changing in how developers were doing their work and how quickly we were getting work done.
At the time, there were scattered blog posts and things you could find online if you were trying to get into the methodology of how to do these things. I had the idea: “Hey, is there a way to structure a curriculum around this that’s a little more formal?” My alma mater is Stanford, and I’d always had this dream of going back to teach something and pay it forward. I’ve always been very deep into education.
I talked to Chip Hieatt, who also taught a class and is one of my good friends.
Chip Hieatt?
Yeah, Chip Hieatt, exactly. I spoke to her. She had taught, I guess, Designing Machine Learning Systems, so I asked her what that process was like and took a lot of inspiration from how she designed the curriculum and thought about getting something through the school. I put it together and spent about 9 to 10 months on it, from fall 2024 through the middle to late part of 2025. I got it through the committee and made the case for the class to exist. I had to put together a proposal and everything.
That took me to fall 2025, when I taught it for the first time, and it was very well received. Students were really interested in it. There was even a Business Insider article about the class last year. This year, a lot of things changed. Coding agents—autonomous coding agents—have really picked up in a way we hadn’t seen before, so it felt like it was time to do this again, refresh things, and here we are.
Meanwhile, you’re also head of AI at Monaco.
Yeah.
Which is also a very buzzy startup.
Yes, yes. We’re in the airwaves right now and having a fun time doing it.
I do like that you’re also a practitioner. You’re obviously implementing all this stuff at Monaco. Any principles or observations about what you’ve changed from last year to this year at a high level?
A ton. When I reflect on last year’s version of the class, coding agents had been around, but we were still in a phase where companies were emerging to tackle every part of the SDLC. There was a coding agent for code review, one for spec design, and one for SRE. Of course, some of those companies still exist.
Last year, it was a lot more about a broad tour of everywhere you could use coding agents. A lot of the curriculum was geared toward exposure—a kind of odyssey through everything you knew about the SDLC changing because of AI. This year, I think some principles are solidifying: careful context engineering, and really thinking through design and architecture as a way of handling complex tasks.
My class isn’t intended for the beginner programmer. It’s not a replacement for traditional computer science, like algorithms; a lot of those classes are prerequisites. But if you have a good foundation, applying it and using those principles in production environments and the real world is now something students can do. At Monaco, we deal with massive code bases. I also help a lot of companies think through their massive code base designs and things like that. So it’s more of a practitioner’s course for students who have graduated or are entering the workforce: how to be the most productive members of the workforce.
You said you’re passionate about education. I feel like students in that phase of college are in a pretty interesting spot. Any advice or takes? Should people study computer science right now? Is Claude Code and Codex all we need? I want to hear your thoughts.
There’s a lot of anxiety from students, as you can probably imagine. A lot of what I’m trying to communicate this time around is that the anxiety is still here, and there was anxiety last year as well. But I want to make the case that a well-trained AI-native engineer—and I know that’s a buzzy word, but we can dive into what aspects of that are most interesting—can be one of the most useful assets at any company.
This is the first week of class, and in the first lecture I say, “Hey, I really believe every company needs you, but the floor is much higher.” You’re entering your first job out of school, but it’s not sufficient to be an L3, in Google terms. You generally have to be almost at the senior level going into this. So how do we get there more quickly? I’m trying to say that there’s still a core bit of foundational software engineering that’s important: understanding how software systems are built and the trade-offs of those systems. Beyond that, humans should still be valuable, and you can defer more of the implementation details.
This is something I’ve said before, and it’s almost the kind of thing you say to be spicy, but I genuinely kind of believe it: most students are cooked, and some of them are going to be super valuable. I don’t think it’s a popular thing.
This is a delicate topic.
Stanford’s kids are going to be fine, right? They’re from Stanford.
Sure.
But what happens to the rest of America?
Right, right. That is an open question. It’s a weird situation.
Yeah, because I feel like a lot of education is slow to move. I’m curious: how do you determine where to put the end date for the curriculum in a course on AI?
Oh my God.
You’re teaching a Stanford course on modern software development. That’s changing every year.
Ah.
You cheat. You just get people from the coding agent companies to come and speak, right?
Oh, yeah.
Like, yeah, have them be the—
Yeah, that’s funny—
The old guest lecturers.
It’s funny. Well, yeah. On the guest lecturers point, that was also the case last year. That was by design, because I came in saying, “I’m a practitioner,” as you pointed out, Swyx.
I’m a practitioner. I do this in my day-to-day; I design these systems. We build these things at Monaco, and I’ve seen that many times before. That being said, I don’t want to claim I have all the answers today. It would be hubris to make that claim.
The structure of the class was always: here’s one practitioner’s aggregated set of insights from working in the field and talking to a lot of people in the field. But there are people literally on the frontier, seeing things even beyond that. I invite them to put their proposals and their view of the world forward, and that’s why the guest lectures are there.
No, I think it’s really impactful for students to see big names, people actually doing it, hear from them directly, and be able to ask them questions. Of course, they want to intern and get jobs. It’s good to have guest lecturers.
Yeah, no, I think what he’s saying is—let me rephrase it. We’ve also interviewed Ange, who teaches the Coachella course, right? That’s like, here are all the names. It’s only the names, right? Whereas he’s also producing things like Claude Code and 200 lines of code.
Yes.
No one’s telling you to do that. No one’s going to come to you to do that. But when you want to teach fundamentals in a framework you can use, and a working understanding, this is the kind of thing you do.
That’s right.
That’s right. Yeah, exactly.
But I do think there’s a balance for this kind of material, to your point. It’s not 100% solidified, by design. That’s a feature, and maybe a bug, of some kind. A big portion of the curriculum did change from last year, so there’s a dynamism to the whole development of the curriculum.
I’m trying to put forward that there are some things I believe are sticking around. Prompting coding agents, for example: they’ve gotten smarter, so what does that mean for the new ways of dealing with them? But Boris is also going to know something I don’t know. So let’s have him come to the table and share his learnings.
One of the fun things I noticed this time around is that a good bit of the course is on open-source contributions. What’s the thinking behind that?
1. Open Source As Apprenticeship
This is one of the things I’m honestly most proud of and most excited about. We’re about to be in week two of the class. These are the very gracious open-source partners I spent the greater part of this summer talking to and getting excited about this offering.
Last year, when I taught the class, we had nine assignments and a capstone open-ended project. The nine assignments were pedagogical exercises I came up with, like, “Here’s a repo. Use an AI code review tool to post a code review, and then see why it’s good or not good.” They were relatively pedagogical and, honestly, pretty easy. I think a lot of folks could have one-shot them, and I thought, “Okay, this is not that helpful.”
At this point, agents are around. Every student I surveyed on Tuesday was using coding agents; not a single person really types by hand anymore. They’re all using coding agents. These are sophomores and juniors at Stanford.
Do you survey them on market share? What tools they use?
What tools, yeah.
Okay.
I haven’t yet, but that’s a good call.
You should do a survey. It’s fun.
Yeah, yeah. I’ll do that.
You could be spicy. Stanford prefers—
No, no. I think through the university, most of the providers have educational programs. They’re very generous with plans for students. By default, even before the class, there’s probably a Claude license, a Cursor license, and a Cognition license as well. I don’t know; you might know them better than I do.
What I was also able to accrue on top of that was additional credits for students. I didn’t want people to be bottlenecked by a $20-a-month subscription. If you want to token-max something just to see what’s possible, then you should. Factory is getting involved, the Cognition folks are getting involved, and OpenAI as well.
For the open-source partners, one of the things I’m most concerned about when it comes to educating the next generation of students is that if everyone is saying, to Swyx’s point, “They’re cooked,” what’s going to happen to them? There’s almost this sense of them being left behind. I think mentorship is a really important part of upleveling everyone and making sure there isn’t a massive gap between the senior people who were already in the industry before coding agents, know what’s up, adopt coding agents, and become amazing, and the juniors who are just asking, “Okay, now what do we do?”
One of the goals was to put students in situations where they’re dealing with complex, large codebases at scale, because coding agents are really good. We’re going to work with what I think are some of the best AI-native repos: projects that do AI-focused things, are coding agents themselves, or build tools in service of agents in the modern world. I want students to get involved with these projects and have part of their assignments be actual contributions to them.
In those projects, they’ll talk to the maintainers, who will review their code and help them build mental models of good software. That way, there’s a passing on of the craft of good software design in a way I don’t think most students get the opportunity to experience. And, of course, I’m sure some of them will get picked up as a result of doing well in those assignments.
One of the interesting things, for what it’s worth—and I’m going to get a bunch of hate if I don’t say this—is I think you say “cooked” versus “cracked” for dramatic effect. It’s nice to state the extremes.
Sure, sure.
Maybe we should go into the details of the curriculum, or is there anything else you want to explore at a high level?
Yeah, that’s all there.
I have high-level questions about students using tools and best practices. Are you in favor of individual tutors? I’m assuming this will also be picked up by students, so before we dive straight into the course, do you have any guidance there?
You said you’re passionate about education. There’s also a stigma around how teachers and professors use these tools, and how students can use them best. It’s not just throwing your assignments in so a tool can one-shot them. How do you still learn? Do you have any guidance there?
I still have some assignments in there that aren’t intended to be the hardest things in the world, but are meant to be more conceptual. What does a skill mean? These are more about the tactics of some of these tools: skills, agents, and agent-ready codebases like AGENTS.md. There’s a certain amount of, “If you haven’t heard these terms, get up to speed on them.”
I think most of the learning from putting the tools to the test will happen both in the open-source projects and in the final project, which is very open-ended. They’ll have to explore something interesting.
Cool. Maybe let’s go through the course from one to ten, breadth-first, and then we’ll dive in deeper.
Sure.
We start with the internals of coding agents. You teach people how to write their own Claude clone. It’s very helpful that it’s now open source. And then?
2. Inside The Ten Week Curriculum
Week one is all about the internals of harnesses: building a coding agent from scratch. Week two is about how to think about context engineering at scale, beyond vanilla prompting. Coupled with that is thinking about tool design and MCP. Then people…
I think for a while MCP was getting a lot of hate. I’m still pro-MCP. I see it all the time, and at this point, it’s a standard that’s existed and I think will continue to exist. So, talking about that and effective tool design, that’s week two.
Week three is more about skills and what it means to extend beyond simple, single-purpose tools to procedures. That’s where we’ll talk about skills, which starts leading into the automation of agents.
Week four is about foundational ways to customize an agent: agent config files like AGENTS.md and CLAUDE.md, hooks, and subagents. There are a lot of mechanisms in modern harnesses that you can use for different purposes, so I want to expose students to those.
Agent-ready code bases are probably one of my favorite topics. This is especially relevant for folks going into industry and thinking about code at scale. What does it mean to have a repo that’s ready to be ingested and consumed effectively by an agent? We’ll think about the trade-offs of designing surface-level boundaries, as well as docs, verifiability, and things of that nature. That’s week five.
Week six is agentic code review. Extending from week five, which is about agent-ready code bases, week six is about how code review is becoming a bottleneck. I don’t know that students have seen this unless they’ve had a lot of experience; some of them are master’s students who’ve been in the industry, but a lot of them haven’t. Most people using code agents today are experiencing a new bottleneck in code review. How do we unblock that and ease the bottleneck? What mechanisms are available? We’ll look at verifiability and how we can extend it into the code review process.
Week seven is security. This was in last year’s class as well, and I wanted to reinforce it. With any autonomous system taking a lot of actions, we’re seeing a lot of things get hacked this year, especially. Some of the core pieces of software, from Vercel to LLM gateways, have been affected. I want to spend time talking about what new attack vectors exist and how we can mitigate some of them.
Week eight is background agents. I want to start touching on how you run agents in many different product and service areas, not just in what was traditionally an IDE. You can also have things based on triggers in your code base or repo, during CI/CD, or triggered by events, like SRE-style bots. How do you build a system like that? This past year, a lot of companies started releasing posts about things like Ramp Inspect. Spotify has its own version, and I think there’s Stripe Minions. These folks have been doing really cool write-ups about the internal infrastructure they’re building for this. I want to touch on that and expose people to what it is.
Week nine is more about what happens when you’re on a team in the modern world and the core pieces of team-level infrastructure. That includes everything from LLM gateways to MCP portals, which is MCP at scale, and how to think about evolving roles. The vanilla, maybe not-super-nuanced thing to say is, “Everyone’s a builder,” but I want to go deeper than that. What’s actually getting subsumed? What roles are evolving or merging that didn’t exist before? What should students think about in terms of a defensible skill set that maybe wasn’t true before? There are a lot more team-dynamic considerations.
Then it all builds up toward the factory, which I think is top of mind for a lot of folks today. Once we’re building factories, where do humans still exist? What are the new choke points that are intentionally there around human involvement? Things like intent and setting up verifiable systems. What happens in the limit when maybe 89% of code is fully driven by agents? We’ll get into all of that.
Nice, solid 10 weeks. I was going to say, some of these are more established patterns than others, and I think people need to know that not all of this is settled science.
Sure. I’ll endorse all of these as things that people talk about and that you should know. We know the open problems; we may not know the answers.
That’s absolutely right. I think even with the software factory, are we talking about dark factories or half-lights-on factories? I think there’s a spectrum there.
Yeah, and there are people on the very frontier, like Warp and Factory, building factories and selling them as services. Even they’ll only claim 30-some percent, if even that, of fully agentically driven contributions.
Yeah. What does that mean in the limit? It’s hard to say. There are people who went very, very far toward a dark factory and then came out saying, “I was wrong, guys.” We covered that on AIE. Just be aware that they probably believe the things they’re saying at the time, but they have the right to change their minds, and they do. Even Boris recently tweeted, “Yeah, I am often wrong.” Everyone is treating his word as gospel because he created Claude Code, but sometimes he disagrees with himself.
Totally. And that’s okay. This isn’t politics. We’re still scientists and engineers. I don’t claim to be dogmatic, but I want people to experiment. I think that’s a principle as well—you asked what principles earlier. You just have to be ready to experiment.
Oftentimes, when I give lectures or talk to people about this stuff, they ask, “How do I stay on top of this? There’s so much going on.” It can be as simple as experimenting for 10 minutes a day. Spend 10 minutes doing something you didn’t do before. If there’s a new tool or framework you want to try, maybe do the exact same problem with that new thing. That alone will cover most of the gap.
I’ll call out that Cognition is on the week where you’re talking about code review. I actually worked with Silas on Devin Review. It’s pretty good, I would say. It’s hard to benchmark these things. There is a review benchmark, but no one uses it. Qualitatively, Devin Review is pretty good.
Well, we use you guys.
Oh, really?
Yeah. Monica does use Devin. Silas actually spoke last year, and he’s great—super smart. We use you guys, and among the providers of code review tools that we sampled, we converged on Devin for a lot of our use cases.
Silas is actually more on the model side these days. Walden is the person you want.
Yeah, I was talking to Walden as well.
I’ll also shout out Factory. They’ve been all about agent readiness, and they build signals and tooling for it. Do you guys use that?
Yeah, yeah.
Any opinions?
We do use agent readiness of our own kind. Eno is awesome as well. I think they were super early on this, and when he codified it in his talk at AI Engineer Europe, it was a fantastic talk and ahead of its time. There was only one person I could see as the most appropriate person to talk about this. I really wanted to give him the spotlight, and they were super excited.
Oh, yeah. It was funny. I was on a call, and we were talking about this stuff. We said, “Okay, we need to form some opinions about it.” Then Factory shipped—
Yeah.
—a full product on it that week. I was like, “Okay, you won.”
Yeah. Like, obviously.
Yeah. I mean, everyone kind of agrees on the same things.
Sure, sure, sure.
It’s just a question of whether you want to turn it into a dashboard.
Yeah, yeah, yeah.
Yeah, because it’s a dashboard.
Right, right, right. But there’s value in it, and I think they did well from a marketing perspective by jumping on the readiness report. I regularly pitch it to other people and say, “You should run readiness reports of your own kind.”
Yeah.
Do you want to go piece by piece, or do you have anything else?
I can go piece by piece.
Yeah. Okay. So the first one: you focus on Claude Code, but then you also have Pi as an open-source sponsor. What’s up?
Okay, this one is maybe a little bit more marketing-y, but one thing I do want to highlight—and I made this clear last year, too—is that I’m not endorsing any tool. I think that’s the wrong way to approach something like this. Tools come and go, and, transparently, when I taught the class last year, Claude Code wasn’t as popular or as readily accepted. Boris had only 4,000 Twitter followers, and now he has, whatever, a million. I don’t even know if he does, but he has a lot of followers.
Claude Code has really emerged on the scene.
I think we also underestimated Claude Code when we had Boris and Cat on the podcast.
Right, right.
It was two months old. I hadn’t really used it—I used it once. A query cost me 50 cents, and I was like, “Who’s gonna use this shit?”
Totally. It was still in beta for a while.
Yeah.
But I think it’s good on them for how they built it before the model was ready.
Yes.
You hear so much about building for where the model will go.
Totally.
It was super ambitious. The models weren’t that good; you couldn’t feel it. And then we see where it came from. It’s funny, because when he gave his talk last year, I remember it was a very high-level set of slides. One of them was literally, “Built for where the model is going to be six months from now.” This was October of last year, and I can see why he would have made that point if he knew where Opus would be in six months. I’d argue that was one of the reasons the curriculum changed as much as it did: there was suddenly this huge arms race around true autonomy in long-running, agentic coding tasks.
So, back to your point, it’s less about endorsing Claude Code. I use Claude Code, of course, and other things as well. I’m more interested in what a modern coding harness looks like. I wrote a blog post earlier this year where I broke down some similar principles. The TL;DR is that with about an introductory-algorithms level of understanding, you can define an agentic loop that does the bulk of what you’d call a coding agent. There are bells and whistles, of course: no subagents, no MCP, none of that. But if you want to make something you can talk to that will edit your file, and do some pretty sophisticated things, it doesn’t take more than 200 lines.
3. Inside The Coding Harness
A lot of what I try to do in education is peel away the stuff that seems like the secret sauce and show that, at its core, it’s not that complicated. These are still students, and you want to teach conceptual principles. There’s nothing more to it than Stanford’s CS106B, an introduction to algorithms. You don’t need more than CS106B to build something like this. And there you go.
Given that you’ve already taught this class, did you get any pushback or questions? What did the students not get, or what kinds of questions did they ask?
Not really. I haven’t received any pushback on this idea. I think most of them are consumers; most students end up using the tools. They don’t get much exposure in their coursework, if any, to peeling that onion back.
This is the week of the harness. Tuesday’s first lecture was logistics, but the first real bit of curriculum was the 200 lines of a harness. Yesterday, in the follow-up, we took Claude Code and used proxy tools to intercept the requests.
mitmproxy?
Yes, exactly. We used that tool to intercept the requests and responses, look under the hood, and see what was happening. I said, “I did say you can build Claude Code in 200 lines, but here’s actual Claude Code.” I took Opus 5.5, a model that was 2 days old, intercepted those requests, and showed them exactly what was going on: the prompt, the system prompt, the plan prompt, and the compaction prompt. Then we went through all of it and showed it to the students. Now they have a state-of-the-art harness they can understand.
It’s nice to get those 3 levels: students are very much consumers; here’s the baseline agent in a loop; and then the real one. I like that you have a video walkthrough of your blog post, too. Do you see students at the extremes, where some people are power users who consume a lot, and others might find this straightforward if they know how agents work? How do you balance that in the classroom?
I think it’s a bit self-selecting, because the curriculum is all here. It’s been available for a few weeks. I announced it a few weeks ago, but it had been in the works for at least 1 or 2 months, so if you looked for it, you could probably find it. The reality is, if students aren’t interested in this, they just won’t show up.
What I find interesting is that students do show up, but there are also a lot of people from across Stanford, including postdocs in other departments. There’s a lot of interdisciplinary interest from people in pathology, medicinal fields, and other engineering fields that aren’t pure computer science. A lot of GSB students and MBAs are also interested in learning how not to have a technical co-founder, or whatever their motivation is.
By the way, if anyone hasn’t done the mitmproxy exercise, you should do it.
Yeah.
You’ll be shocked. Claude Code has the longest system prompt of all the coding agents, something like 20,000 tokens.
Yeah, although that’s true—they’ve also reduced the size. They cut it by 80%.
Yeah, I know, but then it grew again, and then they cut it and grew it again. It’s a seesaw. It’s interesting. You’ll be surprised what goes in there that you sometimes don’t use. It can also hint at the internal workings of some of the tools. Anything you found there that would be cool to talk about?
Yesterday, when I was talking about this, I mentioned one of the things I found most interesting about Claude Code back in the day, and I think part of why it really blew up in popularity: their focus on simplicity. In the early days, they were among the first to lean into the idea that fewer tools can do more. They leaned into SQL, Bash, and other powerful, Swiss Army knife tools, and made that a big driver of capability.
I think they did a fantastic job leaning into that. I think that’s why they were as popular as they were. When I presented something like the system prompt, it was like, “Well, there was a different system prompt 2 months ago.” They’ve reduced it in complexity, and things that were once part of the system prompt, like how to use Git or set up tasks, are now expected to be captured in the model weights themselves. They’ve baked in a lot of those learnings at the model level.
That raises interesting questions about the model versus the harness. How do you think through that trade-off? What do companies invest in? What should you invest in to push the frontier of your own capabilities? As a community, we still seesaw between the two a lot.
I think you don’t want to fault them for growing a system prompt and then cutting it.
Yeah. At some level, that’s also how the model was. With weaker, older models, you need to give them examples of what’s good and what’s bad. Then, obviously, you cut those away at some point. That’s just part of learning what has been done.
You can apply the same idea if you’re using a local coding agent with a less capable model. You might need to add those examples.
That’s right. Exactly.
And then, of course, there are interesting AI learnings from that. As the models get better, when are you over-constraining them by adding those examples?
Yeah.
And how do you pull away the over-constraining? That’s part of the learning as well.
Okay, we’re going to talk about context engineering. RPI—I haven’t seen that, but there are many versions of spec-driven development. What would you highlight to a broader audience about what you’re putting in here that people should know?
This is the one-hour version of the course.
Exactly. Please condense everything.
That’s fair.
It’s also a trailer, right? If you want more, go to the source. If you’re signing up next year or next fall for courses, this is your one hour. It’s almost like the Quibi of the class: too long to be a trailer, too short to be the class.
Is this what people will double-click on?
And you know the best part? If it’s a trailer, by the next time you teach it, it’ll change again.
That’s true. That’s true. A good callout on that, Andrew. The main thing I want to reinforce is that I see this even when I’m interviewing people: very few people in industry progress beyond vanilla prompting. There’s a vanilla form: “Hey, do me the thing.” As the models get better, that can carry you a certain distance. But inevitably, anyone I speak to who comes from Google or somewhere with a massive codebase has to be much more thoughtful about how they’re ingesting context.
Thinking through the principles and techniques for that is important. The RPI framework I’m describing is functionally similar to Dex’s RPI: you have a research phase, then a proposal or spec, then implementation, followed by testing and so on. In this framework, after the research, you propose a few alternative ways to solve the problem. That’s where I invite the human into the loop to decide on an approach. Then you move into the plan, which is the spec.
The TL;DR is that there’s a way to do this that scales to codebases that are millions of lines long. It’s not just a vanilla prompt. If you’re going to work in these big codebases, you should use something more involved.
Anything to add on context engineering?
The weirdest thing about how I think about context right now is when you split out to a sub-agent. Sub-agents can inherit your entire main-thread context, or they can start without it. There’s a lot of disagreement about that, and sometimes you want to condense what the sub-agent does and put its research back onto the main thread. It’s very hacky. I don’t like anything I’m doing, so I don’t have any answers.
All I know is that when you want to fan out work, it’s not just about having a big spec and doing RPI or whatever. It’s about fanning out the work and making sure all the agents have the right context and guidance. If you have a review agent reviewing everyone else’s work, you have to make sure it has everything.
You always go through this phase of expansion and contraction. You think, “Oh my God, these agents don’t have enough context,” so you give them more. Then they overfit to some random, one-off comment you made a while ago and make stupid mistakes because you didn’t mean it that way, but they took it as law. I don’t know what the word for it is. It’s multi-agent management for context.
Yeah, that’s absolutely right. There’s some aspect of this that’s more art than science. To your point, there’s the fan-out for exploration and subtasks, and then the consolidation or compaction, whatever you want to call it. You have to be thoughtful about refreshing the context and understanding when it’s getting too bloated.
Dex had this concept of the “dumb zone,” and we should be more thoughtful about it.
He’s good at those little diagrams.
Oh, yeah. I think he’s—
The little diagrams.
Yeah. He definitely informed a lot of the curriculum here.
Why didn’t we have him on?
Well, he’s fantastic to think with on all this. The diagrams are huge; they’re useful.
I’d add some breadcrumbs for people looking into this. It’s good to see best practices for context engineering, like whether your sub-agent inherits the main context. Claude-managed agents have a version where sessions are separate and the context isn’t inherited.
This used to be a big topic before reasoning models, when agents didn’t run as long and we had shorter context windows. Everyone was carefully managing that precious context window. LangChain used to write a lot of blogs about this and good context engineering. People should know all this.
In the student context, you can expect the average student taking your class to know these things, so you should know them even better. This is a fun space for research, too. One of the research papers presented at AI Engineer looked at the changes to the Claude Code system prompt, plotted a bunch of changes in performance, and tried to reverse-engineer it.
There was another paper that looked at context-window management and how it affects downstream performance. They weren’t the best papers, but the research is accessible and generates good blog posts. I recommend it. It’s a good area you can easily poke into.
Yeah, that’s a good shout-out.
Agent skills, CLI.
Yeah, yeah. This is an extension of a single “function tool” call, like MCP, for interacting with third-party external services. It extends into what you might call procedures: how to automate a set of actions over and over, then formalize them in something like a SKILL.md file.
I want to give people exercises in codifying some of those procedures. I’ll also include a few fun skills that I think are good to use. Matt Peacock’s “Grill Me”—
Yes.
It’s a good one. There are others I like from more of an educational perspective, like Jeffrey Blitz’s Explain Diff. I think it becomes especially interesting when you start talking about code review, because the whole point is to understand the code—
Understand the code.
Understand the code, exactly. You’ve got to understand the code. I don’t know if you’ve used it, but effectively, you run it on the diff you’ve made, and at the end you get a visual representation of what’s happened, so you can skim it and understand what’s at the core of it. What I really liked was the quiz at the end, where you go through and reinforce whether you understand what’s happening. That level of interactive back-and-forth is quite nice.
There are a couple of blog posts in this domain that I really like to bring up. One I saw recently makes the point that there’s no situation in which turning your brain off is a good idea. I think these skills force you to turn your brain on. With coding, people want to turn their brains off and say, “Make me an app. Read my mind and guess everything correctly without me telling you about it.”
That’s right.
A lot of these skills are in the genre of, “No, no, ask me more questions, please.”
Yeah, 100%. I think there’s a lot more output than people are used to. I can type in a one-line prompt and get tens of thousands of lines of code and a whole structure. It’s like, “Look at my productivity, guys.”
More effectively, you should know what you’re working on. You can really be put out of touch, out of the loop. You should spend more time thinking about what you’re actually doing.
Mm. Doing the work and getting the code written is much easier than it was.
Yeah, yeah. There’s no excuse for not understanding it.
For sure. Sorry, what is a web skill?
Skills that interact with the web itself.
With the web?
Yeah, with the web itself.
Computer use?
Computer use, yeah, things like that.
Anything else?
Mainly things that involve pulling information from the web. Are there best practices for skills that interact with the web?
I think this is more of a survey or demo of things I found interesting. I’m trying to remember where it was. Was it Vercel? I’d have to look back at the reading materials. There was one site I found—maybe it was browser-based. They might have—
Ah. Do you remember this?
Well, it would have been them that coined this thing.
Could be. I was just like, “Did I miss something? This is a big part of the skills community.”
No. Yeah, this is good. He’s being educated.
I think they made some kind of website showing some of these interesting capabilities. They like browsers.
Seems legit. CLI is a fun one. Do you have a lot of students for whom this is their first time in a terminal or a CLI, or is it the norm now?
No, I don’t think so. Most of them have interacted with Claude Code at this point.
I’m curious about the split of people who are now just using the apps, right? Claude Code also has a desktop app. So does Codex. Codex is primarily an app now. You’re not in the CLI, not in the terminal. It puts a lot of average people off, but just a random note.
I’d say there’s another thing about skills. There are skills that are bundles of skills, where they cross-reference each other.
Yeah, like plugins, right?
Like plugins. And then there’s also the art of writing a skill, which is something I’ve focused on for AIE.
I think a lot of these discussions focus on skill consumption. You have all these influencers like Matt Pocock saying, “This skill will solve all your problems.” But no, you should write your skills yourself.
Right.
And actually, when you try to write them naively, they’re usually bad. It takes a little bit of skill to write a skill.
That’s absolutely right.
What are Swyx’s best tips on writing skills?
I have a skill— I have a problem with skills, people. I went through a phase of psychosis where I wanted a skill for everything.
No, no, no. How do you write good skills?
Oh. I have a skill for cutting skills.
Okay, okay.
Claude Code and Codex both have skill-creation skills, right? I have a skill-cutter skill that cuts them down because the default skills they create document every way those skills are bad.
So you always run this after a skill is created to cut it down?
Right. It’s really annoying because sometimes there’s slop, but there’s good in the slop. You have to whittle down what’s there.
Oh, interesting. So even the slop is still in the language of the agent, and it means something?
Yeah, exactly. It’s fun. Everyone has their own version. I use a simple vibe check: you call a skill, run it a few times, and distill your preferences into a skill that says, “This is what I like, this is what I don’t like.” It adjusts over time and can automate a lot.
Yeah, well, the problem is that the worst part is when a skill triggers when you’re not expecting it to. You wrote it to fix one thing, and now you’re working on a completely different thing and it triggers. Why? Because it’s badly written. The trigger says, “Call me whenever blah, blah, blah happens.” It looks good, and then you start a different project and it triggers. You’re like, “What the hell is happening?”
Some people keep adding skills whenever they mess up. I know people like Eugene Yan are pretty particular about how many skills they have. They prune them and clean them out every few weeks.
Yeah. This is a huge problem if you’re in the skills world. I suspect there are very smart people at OpenAI and Anthropic who want skills to die.
Hmm.
They won’t say this publicly, but it just feels sloppy. I can see people in my own skills community being sloppy. It doesn’t feel AI-native. It’s not what AGI was supposed to look like. It’s just everyone having a giant bunch of Markdown files.
I mean, I think part of it is—
Yeah, there’s a gardening problem. A lot of people don’t look at their Markdown files. The UX of just chatting about what you want, creating a skill in chat, using it, and seeing the output has a lot of value. Don’t overcomplicate it for yourself. You can get a lot of value by saying, “Look at the last five times we did this and make me a skill,” then iterating on it in chat.
There’s another way to look at it. A skill can just be a procedure the agent runs whenever it needs to do something like a code review. But in the case of something like Devin, we’ve codified skills as a way of representing our best practices and exposing them to the agent or the code review system. That lets us implement APIs the way we like to define them and enforce service-level boundaries the way we define them.
That’s an alternative way of representing organizational knowledge, rather than just an arbitrary thing you want the agent to do so you don’t have to write the prompt each time. I like that positioning. That purpose might exist longer than these random procedures.
I just Googled “Monaco AI skills,” and it’s giving me the country.
Hey, maybe the country is— One day, one day, we’re gonna become—
Well, we’re gonna beat them on AEO, that’s for sure. Can you just buy the country somehow?
Here’s the thing: we were about to buy— No, we’re not about to buy the country. But we did get a little bit of flak when we bought the monaco.com domain. I will say that. We did get sued by them.
Ew. What?
Yeah, there’s a whole public thing about this. They didn’t like that, but here we are. And we won. We won.
Okay, then we’re keeping it in. I was about to say this is spicy, but you won. Is that why we see the flags flying around in here? Because it’s like a victory, because you won?
It’s our victory.
That’s funny. For those in SF, just look up.
Yeah, yeah—you’ll see.
For those listening, we’re showing this on YouTube. You fly a plane around. I think you’re still one of the few that actually do this on a regular basis. Sam Blan came up with it: $6,000 a day. I don’t necessarily think it directly causes me to Google what Monaco is, but all of Monaco’s ads are just Monaco and a dollar sign.
Right, right. Oh, yeah, we stand for money. It’s also a dollar sign and the next number or whatever it says.
Right. The whole point of the ad is, “What the hell is Monaco?”
Yeah. And you could argue that’s a debate to be had, but in some sense, what is the purpose of marketing if not to get attention? Maybe there’s one level beyond this. I’ll go to events, and people are like, “Oh, you guys are the ones with the planes. You guys are the ones with the big number on the billboard—193 million, whatever it is.” But then they’re like, “What do you do?” And maybe that’s the whole point: I just needed someone to come and use it as an icebreaker.
Oh, God. I will say it’s what I said as you walked in, too. I was like, “Ah, the planes.”
Yeah. And now it’s, “What do you guys do?” “Well, let me tell you about what we do.”
As someone who’s been in those meetings deciding what we do on the Devin billboards, we spend so much time on the words. Maybe we should just be like, “Devin.”
I think we do it.
Which is so dumb.
Look, I think Sam had another post this past week where he was talking about it. Sam is effectively our marketing mastermind here. There’s a little bit of history behind the planes, too, but the TL;DR is his whole shtick is, “I don’t have all the answers. You just gotta try things.”
It’s good, yeah.
A lot of this is just trying things, and if it sticks, great. Did we know this was going to blow up as much as it did? Not really. But now that it did, it’s—
You had an idea.
Yeah, exactly.
And the mental model is: start small, go big instead of start big, go small.
Right. It’s way easier to start small and go big.
Exactly. Do this post.
That’s right.
Also, it’s $6,000 a day. It’s very much just trying things.
Yeah. I think people—
Cheaper than a billboard, for sure.
Yeah. And it’s fun. Not much harm done. But the simplicity here that’s worth mentioning is, when we’re trying to one-up the plane, we’re like, “You know what’s better than one plane? Two planes.”
Ooh. Two planes. What? My mind went to a rocket.
I also had that idea.
I was gonna say, I’ve seen the memes.
I’d like for that to happen at some point, too, but we’ll see.
This is the next SpaceX launch. We just see Monaco—
Yeah.
And they—
There you go.
There you go. Gotta warm up to our SpaceX friends.
Well, I mean, we’re on Monaco. Is it worth talking a little bit about Monaco? Do we leave it there?
Yeah, yeah. We mentioned Monaco, right? Let’s give a little more. I think it’s a little annoying to mention “What the hell is Monaco?” and then not answer it. Can you answer it?
Sure. Well, I hope so.
Whoa. This is the country. It’s a small little, I guess—
Oh, no, dog poop.
Oh, no. There you go.
There we go. There you go.
Okay, this is Monaco. This is the Monaco that I know and love.
4. Monaco Builds An AI Sales Platform
Yeah, we’re a Series B company based here in SF. We’re building an all-in-one sales platform. It’s fully AI-native, with agents doing all the fun stuff—or at least the stuff humans don’t like to do. Agents do like to do it because they don’t really have opinions, and they do it in a less error-prone fashion, so salespeople and sales teams can focus on what humans are good at, which is closing deals.
So presumably Marc Benioff doesn’t like you guys.
Probably not. But he’s also honestly a very AI-native person.
Mm-hmm. Very aggressive with Agentforce and everything. What are you doing that he doesn’t do or can’t do?
We didn’t go after enterprise from the get-go. We said that if our goal is to build the all-in-one, we need to build it iteratively with a group of customers and grow with them. It’s much harder to pull someone away from Salesforce than to meet them when they’re still tracking their pipeline in a Google spreadsheet, show them a more AI-pilled answer to the problem, and grow with them. In a way, it follows the same model as Brex or other YC companies: you sell to your batch mates, grow with them, and then go upmarket. I’d argue we’re among the best, if not the best, startup-focused sales platforms. All our customers are early-stage startups. We’ve primarily focused on seed through Series A, and now we’re going more upmarket. We’ve learned a lot about the all-in-one over the last year and a half.
And you are the head of AI?
Yeah, that’s right.
What are some cool AI things you guys have built?
We’ve built our own agentic harness for sales use cases. To call back to coding harnesses, we’ve taken a lot of inspiration from systems like Codex and Claude and built bespoke tools and workflows focused on sales and geared toward GTM use cases. Our version powers pretty much everything in the Monaco platform, from scoring TAMs and ICPs, and defining ICPs, through to pipeline evolution and all the mundane things a salesperson may not want to do, like updating pre-meeting notes and summaries. We have agents powering all those workflows, and it comes out of the box. We have long-horizon agents and, in some cases, short-horizon agents for GTM use cases. It’s pretty domain-specific, built on foundation models.
Would you do any of your own models for this?
That’s a good callout. So far, I think this is generally true of a lot of vertical AI companies: you start by using frontier models as third-party services to see what’s possible, because they’re at the frontier. We’ve seen this in the past week with the whole Harvey discussion about 50%-plus and 50%-minus margins. Irrespective of the numbers, what’s interesting is that at a certain point, things become interesting from a cost perspective. You start to wonder whether you can do it faster on open-weight models. A lot of the purpose of the AI pieces is then to post-train those models and build environments to post-train using RL and things of that nature. We’re kind of in that phase right now.
Ooh, fun. I’ve been calling this the agent lab playbook, but that’s my coinage: agents start with the user, then use whatever models they need to build products, and then backfill the model side once they have the user base. Model labs go the other way. They start—
Yeah.
—with the big pre-trains, and then they try to find users. Hopefully, it works as a split. Monaco would be on the revenue GTM side.
That’s right, yeah.
I think it’s interesting that they’d want to serve you while also trying to compete with you—
Mm-hmm.
—as model labs.
That is correct.
One of the hard problems I’ve run into—I’ve dabbled in some of this—is enrichment. I spend a lot of my time doing that for SEO, research, and what have you, and I imagine for you it’s a core part of the business. Do you find that models are good at it? Are there vendors you like working with and want to give a shout-out to?
Yeah, absolutely. We were early adopters of Parallel, and I think they’re very strong.
Okay, so just general search?
Yeah, I mean, they’re—
Not like ZoomInfo or one of those guys?
No, no, no.
Yeah.
We use Parallel, and we also use Exa, so we’ll give a shout-out to both of those folks.
Can you compare them?
No chance, brother. They’re both friends.
Okay.
They’re both friends. They’re both folks we like to work with.
They’d be very interested if you liked one over the other.
Yeah. Look, I like them. I’ll keep it neutral. I’ll stay Switzerland on this today. But, of course, we do our own version of enrichment as well. One of the most valuable pieces of software we’ve built, or one of our most valuable systems, is our world database: the database of companies, enriched companies, enriched accounts, and enriched contacts. When we’re constructing ICPs for people, we have the most up-to-date information about who they should reach out to at which company, at that point in time.
I’d say where we’re world-class, hands down, is demand. We’re a very good demand company. We help companies increase the number of meetings they book.
I see. So, find the buyer at the right time, right?
It’s a combination of those two pieces.
What do you mean by “at the right time”? Say you’re targeting VPs of sales. The right time to reach out isn’t when they’ve been at the same place for 5 years. A good signal is when they’ve gone from one job to a new job. They’re entering a new place where they’re more likely to purchase, or be open to purchasing, a new sales platform because they’re in the middle of a job transition. So, for a VP of sales moving from Company A to Company B, a good signal that you should reach out is detecting that their LinkedIn has changed or something like that.
How much of this is your expertise put into a model and harnessed, versus models being good at generic best practices? Is it the glue, making it reliable and doing it all in one platform? It seems like there’s also domain expertise baked in that we wouldn’t know about.
Without the former, it would be fully commoditized.
Yeah.
I mean, not fully commoditized, because there’s still a lot of value in the all-in-one. I think that’s something we definitely pioneered. Most companies in the sales space started as point solutions. They tried more of the YC approach: land and expand. You pick a niche and then expand from there. From the get-go, we knew we wanted to be all-in-one. That’s why we stayed in stealth for the greater part of a year and some months: we were building.
That alone is powerful and meaningful. But I think the real power is that Monaco has a world-class go-to-market team selling Monaco. The same go-to-market people who sell Monaco are also what we call forward-deployed account executives, or FDAEs. They’re effectively human salespeople, consultants of sorts, who help onboard our customers, get them building agentic workflows, and guide them through every part of their sales go-to-market lifecycle.
So, yes, it’s the all-in-one, but it’s also the fact that we have these people. We’ve brought on a lot of former Sam’s team members who were some of the best people in GTM at Brex and other places. We’re codifying all of that in the platform.
Yeah, interesting. You talk very differently from a normal head of AI I might talk to, because you’re also a former founder, right?
Mm-hmm.
You’re also used to thinking about these things in this way, and it’s interesting to—
Right.
—it’s important to have that—
Right.
—in your role.
Yeah, cool.
I think you’ve introduced Monaco enough there, and as the Monaco story develops, we can revisit.
Sure, sure, sure.
Maybe the bridge back is: what is coding, or agent-native coding, or modern software development like inside Monaco? When you’re teaching, you might be trying to give people the lay of the land, but maybe inside Monaco you’re more opinionated about one thing versus another.
Yeah. We made investments in coding agents basically from day one, and we’ve learned a lot. We have our own scars and learnings, and I think we’ve been getting much deeper into sharing some of them. On our blog, we’ve increasingly been posting some of our learnings. Maybe not the one at the very top—that one is more of a product—but I’d point to “When AI Gets Desperate,” “Before Scaling Contributions,” “Build or Buy,” and “SRE.”
We spent a lot of time looking at different providers and vendors, and we ended up building a lot of things in-house. For us, something like a background agent platform: we have our own version of Ramp Inspect that we call Monocoder. It’s built on the Coder platform, and it’s what we use to scale up our engineering productivity.
Coder, like Coder.com?
Coder.com, yeah. We’ve adapted it to some of our use cases. It’s a really well-designed piece of software for isolating development environments, scaling out to multiple agents, and running long-horizon agentic coding tasks. So, Monocoder—shout-out to John Tracy and co. for building that out.
We’ve also built our own version of a Slack bot that we call Hal. Hal is part AI SRE, part customer success, and part QA engineer. It lives in our Slack and helps us debug things and put up PRs when things are broken. Our salespeople also use it to find out how the product works and get to the bottom of things. We’ve had interesting questions to answer around the different personalities Hal should have, depending on whether it’s talking to a salesperson or a developer. We’ve built out a lot of systems like this over the last year or so. Shout-out to Yunja for Hal, and to John Tracy as well.
So, you’re building a lot in-house. You also use other tools. What’s the line? “Build or buy”—was that the idea?
Yeah, “build or buy.” It’s a good question. There’s a general principle I usually go by: you build to learn and you buy to keep. Or, sorry, you buy to learn and you build to keep.
Yeah, the other way around.
So, you buy to learn and you build to keep. I think that’s generally true here. In the early days, when we were still trying to figure things out and get our feet under us a little bit, we…
Code review is one example. We spent a lot of time looking at different code review platforms. Today, we still use Devin because we’ve found a lot of value in how the system has been built, so shout-out to the Cognition team. Along the way, we tried some SRE tools and some off-the-shelf, more background-agent platforms. Given our infrastructure and how we’d set up our own cloud platform, it made more sense to build this out internally. Of course, we’re not dogmatic; we’re open to revisiting them if something comes out. I think we usually ended up building in-house when we felt there wasn’t a mature offering out there.
Code review is an interesting example because we did a whole survey of code review tools, maybe over a year ago. At the time—
I’ll shout out CodeRabbit, right?
Yeah, CodeRabbit and others. There was also Graphite.
Yeah.
Graphite, yeah. We used Graphite for a while before they got acquired. There was Ellipsis, which was a YC company at the time. There were a bunch of folks we tried. They were okay, but I don’t think they crossed the threshold of being immediately useful yet, so we put it aside. This was well over a year ago. Then we went back to it organically. Shout-out again to the Devin team and the Cognition folks: we tried it again. Somebody had told me, “Hey, you should try this. It seems like it’s gotten a lot better.” I’d actually been at Silas’s talk during the class last year, and it was good. It was picked organically.
So purely vibes, right? There’s no way you know this.
Okay, here’s what I’ll say. When we did that first evaluation and didn’t pick a vendor because there wasn’t one we were really happy with, I analyzed our PRs. I made pie charts showing what the tools were solving and whether we were happy with the trade-off between verbosity, how often they posted, and their utility. So it was more statistically driven than just vibes. The first time around, I was pushing for us to adopt an AI code tool. The second time, I wasn’t going to push hard on it again. But then people organically started saying, “Hey, Devin is good. Can you add me to this?” Today, people are getting added organically. It’s one case where we’re still leaning toward the vendor. We’ll see what that looks like in the long term, but outside of that, we didn’t feel the maturity was there.
And it’s not painful enough that you would—
Yeah, we’ve involved Devin right here.
That’s right. That’s interesting. Good talk for Devin, who doesn’t focus on code review.
Yeah, I didn’t even know. For the record, I’m only an advisor. I’m not that involved anymore, but it’s interesting to hear about it externally.
Mm-hmm.
Internally, there’s a certain bubble. On Slack, you only hear positive things because people only want to circulate positive things.
Right. But you have no dog in the fight; you just choose the best tool. If you want to get us free credit, that’d be cool too. I don’t think anyone would say no to that.
I think Silas will hook you up.
Okay, so that informs all your teaching work.
Yes, that’s right.
I think that’s really good. We can bring in more of this as you like, because this lived experience is very valuable.
Sure. I’m happy to go back to agent-ready codebases or more.
For sure. Weeks four and five kind of blend together. Whatever you want—whatever strong beliefs you’ve introduced in Monaco and recommend people start with.
For sure. Weeks four and five blend a little bit. Week four is more about conceptual foundations and ways to do this, because there are a lot of ways to customize agents. Week five is where I really lean into applying this in practice on a real codebase. I’ll give a shout-out to the Factory team and highlight the value of something like a readiness report. The main point I want to reinforce is that agent-ready codebases are a precursor and a building block for everything that comes afterward. People have been—and I think still are—really hyped about multi-agent systems, with agents working in parallel and tens of agents doing all kinds of things. You need to lay a foundation to do that well: how verifiable your codebase is, how well-structured it is, and how understandable it is to an agent. Agent readiness is almost the first piece of the next four or five weeks, up to around week eight or nine, when I spend more time talking broadly about AI software teams and organizational structure.
That’s definitely building up the building blocks for background agents and the software factory. I think this is where it all gets interesting. One thing I noticed you talking about for Monaco, but that isn’t here, is the SRE side. When you close the loop, you want the agent to read your build logs and, ideally, your analytics from live production.
That’s right. It’s not explicit, but that’s a really good call-out. I’m going to weave it in in two places: the background agents week and the software factory week. This idea of closing the loop based on triggers or events is now one of the two or three entry points people talk about when they discuss agents kicking off tasks in the software factory. That’s absolutely right; it’ll come in there.
It’s interesting that people call it trigger-based or event-based. I guess that’s the actual technical implementation, but it seems almost too low-level. The trigger is usually a cron job, right? Let’s be real. Event-based is usually a cron job that surfaces an event, and then it’s event-based. It’s all crons.
Sure, sure.
Or a webhook or something like that.
That’s absolutely right. At the end of the day, being a CS person by trade, I fall back on that terminology.
Have any AI SRE companies made an impression on you?
I think the Resolve AI folks are doing well there. Night Shift is also doing quite a good job with background agents. I’m a big fan of what they’re doing. They’re Greylock-backed, and I think they’re doing really well in this area. SRE is an interesting use case, and this goes back to the build-versus-buy question. Where does an AI SRE see the most value? Typically, it’s when you have a tremendous number of service dependencies, massive systems, and tons of Kubernetes and Helm files strewn everywhere. I think that’s why Resolve AI has done really well in the enterprise. You go and sell to Ford or Walmart and say, “Your service dependencies are crazy.” You see these diagrams of how everything is set up, and then they go in and say, “We’ll try to make sense of all this.” At smaller scales, it’s usually still manageable to roll your own. But for the enterprise, I think they’ve done a good job.
Yeah. For people who have never worked at this scale, because it took me a while to see this, it’s on the order of—let’s call it tens of thousands of repos. More repos than developers. These are situations where you don’t know what code is running—
That’s exactly right.
—anywhere in the company—
That’s right.
—because probably the person who wrote it is no longer here, right?
That’s absolutely right.
And, by the way, it’s still load-bearing.
That’s absolutely—
That’s the scariest thing.
That’s absolutely right. It’s like—
And then you’re the person on call, interacting with a system that goes down, and the person who knows it still isn’t there. There are old, out-of-date runbooks, and you’re like, “What the hell is going on here?”
It’s a scary, scary thing. And yes, at the enterprise level, it’s a whole different beast.
Yeah, so the college hacker, with one person per repo, has no appreciation for this. But I think if you drop them into a simulation and say, “Okay, day one on your job, this happens. What now?”
Yeah.
Yeah.
So another shout-out I’ll give here: the Resolve folks made an assignment for the class last year. They created a sandbox with many services doing various things, and there were even bugs in those services. As part of the assignment, students had to use the Resolve agent to probe and play the human on-call person, probing the system and discovering what was happening.
Wow, that’s—
They’ve got an interesting—
Yeah, no, it’s an interesting website. It’s been a while since I’ve seen this. It’s very interactive. Okay, they’re being a PagerDuty now.
Yeah.
Yeah, that checks out. For sure. But also, this video—I was—
Well, you know, you were shouting “Adam.”
Interesting. Interesting. Sure. But okay, we never get a chance to talk about this side of the world except for conversations like this. Any other segments? I think AI SRE might be too broad or even too narrow, depending on how you look at it. Too broad meaning there are several interesting subdivisions of AI SRE, or too narrow meaning that’s not even the biggest problem we have.
Do you mean specifically for Monaco or broadly, for organizations thinking about this?
Broadly, because that’s our audience. Take it however you like.
I think it’s a function of how we scale. For the enterprise, there are a number of issues that aren’t always solved by more agents. Since we’re still on week five, “Agent-Ready Codebases,” when I talk to companies at a certain scale, the things holding them back are often lower-level DevOps and CI/CD issues. Their merge queues are getting backed up. It takes too long to—
Okay.
—stand up an app. There are foundational things that have almost nothing to do with adding one more SRE bot. A bot wouldn’t be the answer, because there are more systemic things they need to figure out.
So that brings us to week six.
Yeah, exactly right. That’s where we get into—
To the merge queue and beyond.
5. Code Review Becomes The Bottleneck
Yeah, this is a love letter of sorts to modern agentic code review. I think the biggest thing that happened between last year and this year was that it suddenly became very clear that code review is a bottleneck. Once people got really AI-pilled, the velocity of generation became so high that you could, in theory, put up 30-plus PRs a day. Then what? At what point do you decide to put yourself in the middle to gate and review things the traditional way? Then you’re literally the constraint, the choke point for velocity, and that’s not good. What was the point of generating at scale?
Or you ask, “How do we get the best of both worlds?” We want enough oversight to trust what’s getting merged, but we also want to merge at scale without freewheeling and letting anything go. A lot of what I want to touch on is that middle ground. If you’ve thoughtfully designed your agent readiness and verifiability, there’s a class of problems you can feel comfortable auto-approving some percentage of the time. What needs to be true from a design or architecture perspective for you to auto-approve? When do you have to insert yourself and say, “I still want to review this carefully”? And even then, are you doing line-by-line review or something else?
One of our hottest topics this year: we have a guest writer program for Latent Space, and this is from Ankit at Aviator.
Nice, nice.
He has a whole thing on killing the code review. That may be the final boss, because it’s a bottleneck. He has a few gates by which he does this. I don’t know if you’ve talked to Ankit or—
Yeah, I’ve read this post.
Okay. I haven’t spoken to him personally, but I’ve definitely read it. Was he right? Any criticisms?
No, I think that was broadly right. I guess you mentioned the last boss. Maybe it’s not the final boss. I think there are 2 choke points: the review and probably the intent, or the thing at the top. Like, this is the—
Yeah, and there are some stats I like to pull up. Everyone’s stats are different, for sure, but clearly more people are merging. The number of reviews isn’t increasing in line with merged PRs.
Yeah, that’s right. If you look at any modern company that’s really invested in this and thought about it carefully, some people have already written posts about it, and I’m sure you’ll write about it too. Amplitude, Cloudflare, and Augment have all written good posts about the design of their AI code review systems.
Mm.
These are a bunch of really good posts. If you want to share some of those links, they’re fantastically written.
Shout-out to Cloudflare, because I think that, in a sense, they’re a legacy or bigger company, but they’ve done a fantastic job with their content around the AI SDLC. The short version is, yes, this is true. We’re still finding the sweet spot: what’s auto-approvable and what isn’t? And if something isn’t auto-approvable, what do we review? Are we still doing line-by-line review, or some combination of the spec and something else?
Are people keeping the spec up to date at all times? The spec is in the PR, right?
Yeah, no. There are a few different places. There’s a spec in a Linear issue, and there’s a spec in the context of a PR description. You can have the PR description explain more or less the entirety of the implementation in a condensed format. As long as it’s the most up-to-date representation of the codebase at that point in time, I think that’s sufficient. We’re thoughtful about our PR descriptions. They’re not just one paragraph; they’re effectively a spec of what happened in the PR, with file-level pointers to what changed. That gives you both the high-level view and a sense of the file structure and surface-level boundaries you’re touching. That’s where we’ve invested quite a bit.
So I push back on that because, to me, of course, at bare minimum it should be internally consistent with the PR spec.
Sure, yeah. No, what I meant is the spec for the feature, the codebase, because that is the summation of all PRs. If your PR spec got it wrong or missed something, it’s going to be the delta between what the whole codebase, or the whole history of the codebase, is supposed to do and what your PR is trying to do.
Mm-hmm. Yeah. I don’t expect you to have an answer. I really stress about this myself. I try to instruct models to do it, but they’re bad at it. That’s it. [laughs] The humans are the spec gods: “Five weeks ago, I did this other thing, and now you’re reversing it.”
Have you seen the post from Basis? They had a tweet article, and I thought they did a great job talking about the different file artifacts they track via Git. Some are the specs—the intent of what is to be—and others are what they call the canonical state of the code, which is a little more like what you’re describing: implicit knowledge, implicit intent that also has to be codified somewhere. That part is all version-controlled, and I think they found that works for them, so that might be another thing to share as well.
Yeah. Vibhu was trying to pull that up.
This is Basis—the accounting company.
Exactly right.
We’re having Mitchell at my New York conference soon.
Amazing.
Talk to him about that. I like that they’re coming up with this. My solution for this is something that replaces Git in order to capture all of this. I don’t know how ready I am to talk about it, because I may never ship it, but I call it a program database. It captures all traces and reifies specs into something we can keep and refer to. It also controls the skills attached to it.
That’s cool.
I think Git is only focused on code, and you need a source of truth for everything else outside the code, including the implicit assumptions, the intent, and the other stuff. It also needs to be highly concurrent, because Git, by design, introduces mutexes everywhere.
Mm. Yeah.
So you need super concurrency. Have you seen GitButler?
Yes, yes.
The guy who used to be CTO at GitHub—I think his name is Ian. I forget his name, but, yeah, Scott. Scott. There are a lot of people going after this, and the question is what the ideal representation of tasks is. I think the winner probably isn’t me; it’s Linear.
Really?
[laughs] That’s where the tasks are.
Yeah, where we’re going.
And they’ve clearly decided to become a coding agent company. They have all the source of truth there, as long as you buy into Linear.
Yeah. That is true. I have this somewhat-held belief that a lot of companies are going to evolve into some version of a coding agent company. Any company that interacts with B2B software, where you’re collecting something about the user or some product insight, can close the loop. Ultimately, the purpose of a coding agent is to act on some event or something happening where a user or customer is involved, and use that to fix some aspect of the product.
I think what Linear is doing is something Amplitude could do as well, because they’re also getting insights about user analytics and user problems. If you stretch it, even a company like Decagon or a CRM could do something similar: “Here’s what’s broken. Go close the loop.”
Mm. Mm-hmm.
All right, so set all those things up for an agent-ready codebase and set up code review. Security is week seven, not week one. [laughs] Students: “What do you mean security’s not important?”
No, I make fun of it because I always say that at Amazon, “Job Zero is security,” and they always say this.
Yes, yes.
Anyway, security review at the end.
Okay. I was at Amazon, and I don’t remember that, but I believe you.
This is a thing Andy Jassy says.
Okay, okay, okay. I see.
Yeah. I was there partly during Jeff’s time and partly during the transition to Andy. This was a topic last year as well, and it’s one this year too. Isaac also spoke last year. I find him to be an incredibly thoughtful leader in this space.
Semgrep is an interesting company because they were around even before agents. Their open-source repo was around well before the traditional SAST defense mechanisms. They’ve done a good job augmenting their offering for this space. It’s an interesting time; there’s a lot going on, and these attack vectors are coming to the surface. There’s a lot more slop and AI-generated code. One of the core insights I want to drive is dependency tracking and supply management—supply chain, basically. You have to be careful about your dependencies, pin things appropriately, and not just arbitrarily hit “npm install” and expect the chain to work.
So potentially, the most secure codebase is one with no dependencies.
Yes.
Right? You look at a dependency and say, “Cool implementation, bro, but I’m just going to make my own.”
That’s absolutely right. It’s an interesting insight because, taken to the limit, if you believe code is now commoditized and easy and cheap, why not just build everything in-house? Of course, that would be the safest thing: you’ve seen everything and control everything, instead of depending on Axios or whatever. We’ll see how many people actually act on that, but that is the safest thing. I agree with you.
No, that statement is dangerous, because then you’d just make your own mistakes instead of relying on other people to discover mistakes. The whole point of open source is that—what is it?—sunlight is the best disinfectant. It’s like Linus Torvalds’s thing. You’ll be slow to discover mistakes in your own custom software dependencies, because you only have your users, and they won’t report it to you. But if something has 100,000 users worldwide, someone will find something, and then you can depend on it more.
Yeah. This is—there’s no… I don’t want to be—I’m trying to be a voice of reason.
Right, right. You can’t be absolutist in both directions. I kind of trapped you there. I talked you into the talk track of vendoring everything.
Right, right. And there’s a bug there.
Yeah, it’s an interesting one. I wonder how much that’s impacted by the fact that everything—I’ve always believed that most open-source repos, if not all of them, will just become software factories that are effectively agent-run in the future. What happens then? Even if you look at Warp, they’re also selling a software factory product, but they’re running their open-source Warp terminal as a software factory. That suggests humans will still be involved, but increasingly less so. Certainly, not every single person is reviewing every single thing that comes in at the same level we did in the past. Does that make this codebase more or less vulnerable to security? It’s hard to say. That’s where there’s still a little nuance, and even the people I know who work in open source are trying to figure out exactly what that stable point is.
There are a couple of related arguments, and I’m happy to keep going. Security is a nonstop, huge surface-area problem. There’s one argument from Malte Ubl of Vercel: “Okay, guys, let’s take every piece of highly dependent software written in C or C++ and port it to Rust. We’ll be faster and more secure. It’ll probably take $6 million, but you’ll get it done.”
There’s another argument, and this also links to “Pacing the Frontier,” from the frontier model labs—
Mm-hmm.
The model labs are like, “Okay, let’s slow down. We’ll do Daybreak, we’ll do Glasswing, we’ll make sure everyone’s patched, and then we can let loose on rampant acceleration.” I think there’s an inherent belief that the models will fix all security.
I think that’s unproven. What if they introduce new bugs, just different bugs?
That’s fair. That’s fair.
I’m looking for pushback. I’m looking for someone to say, “No, you’re an idiot. Obviously models are way better at this.”
Yeah. I guess that’s almost the question in the limit: Do we believe a model will create perfect software? If a model’s creating—
Yeah.
—perfect software, then you’re done. Is there such a thing? Then we’re done here altogether. We’re just going to go home and chill on a beach for the rest of our lives.
I think there’s also the question of whether it’s good enough to use for security first, so people can patch, and then just delay the release and do it afterward.
Hmm.
Or is the problem that it’ll be much better at finding vulnerabilities, so you can really test it for longer even if you’ve patched? How long, right?
Yeah. You can give it 3 months. Someone can try to exploit it for 6 months. The problem’s not solved, right?
Right. Right. Yeah. And my whole problem with this—and this is a whole rant we don’t need to go into—is that you name 100 select organizations and say, “Okay, these are priority. They have access to a cyber model you’ve just created 100 attack vectors for me to use to compromise them.” And, by the way, they’re pretty easy to compromise.
Yeah. Mm-hmm.
I don’t know. Anyway, it’s above our pay grade.
It’s always fun to get into security.
Yeah, security is very juicy right now.
Yeah. I think the security folks love it because they usually don’t get that much attention, and now they have a ton of attention. Is that true?
What do you mean?
I mean, I find that, at least from a VC perspective, people love investing in security companies. A company like Wiz has done very well.
Wiz and Socket—
I literally had this conversation with a VC 2 days ago. Apparently, Wiz and Socket are the only security companies in the last 5 years that have worked.
What about Abnormal?
I don’t know. What do you mean by “worked,” from a VC point of view?
I see. From an economics perspective.
Yeah.
Hmm. Interesting.
Yeah. Interesting. Not for me to say. Obviously, there are many great players in that field, but it’s a tough industry.
I think it’s not controversial to say the best time for any security company is right after a high-profile incident has happened.
Yeah. Obviously.
Then you’re going to sell a bunch of security.
Totally. I think that’s fair. People should be aware; people should know the basic mitigations. I do think code scanning is important, but I want people to understand that models introduce bugs you wouldn’t make as a human. If you’re not checking those bugs, then—
It’s not even that they make bugs. It’s that you have less awareness of what’s being written, right?
Yeah, because you’re not even reviewing everything.
Yeah. Forget bugs. Models are pretty good; on average, I’d say they’ll have fewer bugs. The problem is that you don’t even know what you’re putting out.
Yeah. I think we have to stay on top of mitigations, and there’s also the combination of security and security as part of code review. Doing those checks is important.
Yeah. One more thing—or just check my intuition on this—is that not all parts of the codebase are created equal. The parts of the codebase where security is very important are data storage and billing. Let’s call it data, billing, and auth. Auth is obviously related to all those things. Basically, don’t write code there, and everywhere else, go nuts.
That’s absolutely right. It’s really funny: if you don’t know the difference between auth and authz, please learn it. If you’ve never looked at AWS IAM, which you have, please learn it. These are basic concepts to understand, and you can prompt for them. Once you have the language, you can prompt for it. If you don’t have the language, you’re screwed.
Totally. I’d even extend that to certain cloud-platform things. There was a huge incident at one company—I forget which—where a DNS issue brought things down. Do you remember this?
There have been many, many DNS issues.
You’re going to have to narrow it down.
I know. There was a particularly prominent one. Was it—?
There have also been many prominent DNS issues.
I don’t remember now. List them off. Let’s go down the list.
It’s always DNS.
Yeah. DNS is more of a reliability and uptime issue.
Sure. Security is more like, “Did you lose money? Did you leak customer information?”
Yeah. I’d probably put that above DNS, for sure. DNS is about general uptime, SLAs, and stuff like that.
Background agents—now you’ve got your buddy from Cloudflare here. But your guests aren’t necessarily correlated with that week, right?
Yeah, no. Week 9/1 was a bit of a scheduling thing. I’ve thrown in Elad and Amjad. I guess Amjad is definitely week 9. I was trying to get Elad at the end, but timing-wise, it couldn’t work out.
So it’s a bit of a weird one where I’m not lecturing at all in week 9, but we’re coming back to those topics in week 10, so it’s a little staggered in a bizarre way. Week 8 is Rajesh, who’s been leading a lot of the infra stuff at Cloudflare—cloud infrastructure, DevX, and beyond. As I said earlier, I think they’re doing a great job. Their adoption, for a company of their size—a public company of their size—is fantastic, and their content is really good, too.
This is fundamentally about all the ways you can use asynchronous background agents. I’m also going to spend quite a bit of time talking about Stripe Minions, Spotify’s version of this—I forget the name—the Ramp Inspect, and things of that nature that show what it looks like to scale this out.
Cloudflare. Instacart? Grokbot? Muse?
Yeah. Muse. I guess those are less about consumer use and more about software teams using them at scale.
They’re all coding agents under the hood.
It’s true. Everything is a coding agent.
Grokbot is one of those interesting things that’s very much presented as consumer-y. It’s Grok under the hood. It’s Cursor under the hood.
Yeah, yeah, yeah. It’s the Cursor team.
No, no, for sure. I’ve seen Cursor working in the data docs, but they don’t have a codebase integration, right?
They do.
Oh, okay. I haven’t used that.
I’ve only been using it for OpenTable reservations.
Exactly.
I think that’s the point of what they’re trying to do: you can use it for consumer stuff, but there’s extra power if you know how to use it well, because it’s actually Cursor under the hood. I think this applies to all coding agents: Claude Code, Claude Design, and people using Codex without knowing they’re using Codex. A random code-interpreter window pops up, and you’re just asking ChatGPT a question. At some level, there’s a coding agent under the hood.
Yeah.
I’ll give you another practical example. I run all my SEO and AEO inside Codex, and it has a codebase, even though a traditional marketer would just check Ahrefs or something. I write all these scripts for doing all sorts of things. I also need to store data and visualize it. Pretty soon, if you’re doing anything on an ongoing basis that has a lot of steps or a lot of data, you need a codebase.
Hmm, that’s an interesting take. I think models also like building codebases. But that could be a temporary thing, right? I guess my point was more about the interface: are you going so far as to say that a Grokbot will have a place where a power user can see the code under the hood? Is that what you’re saying?
Well, there is a computer attached, right? Maybe they’ll just jump to Cursor, I guess. Is that what you’re saying?
Exactly, because otherwise you basically cannibalize Cursor. If, in the limit, what you’re saying is that Grokbot is Cursor, then you don’t even need Cursor if everything under the hood is just becoming—
It’s the same team, so they—
Maybe they don’t care right now. But, yeah, it’s—
Yeah, it’s all Grokbot.
Yeah, I buy that. I do accept that even things that seem outwardly like nontechnical use cases are technical under the hood, and can be written, designed, and implemented in code. That much is absolutely true. The question is more: for a developer, is that their single point of entry?
Yeah. I think it’s not even developers, right? Think of a regular person doing marketing who’s using Claude Code, or just Claude in the app. They’re working in a codebase they would never know or touch, but there’s a codebase under the hood.
Mm-hmm. Yeah, Claude Code is a good example of background agents. That’s a big part of how Anthropic works. You should see how they work.
Yeah, yeah. Check it out.
Okay, AI-native team. We talked a lot about these topics, like gateways. We just put a routing episode out today, and MCP portals. I’d say MCP portals are the least established.
You said you’re bullish on MCPs.
I’m bullish on MCP. It’s not bullish or bearish; it just is.
Mm-hmm. It’s just part of the landscape now.
That’s a statement.
It is. There was a whole “MCP is so back, MCP is dead” thing. People like extremes. But, yeah, portals are an unproven business. I thought about starting one. I actually pitched a guy at Anthropic on potentially starting one, and then I lost interest. I lose interest in a lot of things.
What’s in there?
What I’m thinking is, if this is more geared toward scaling things out, and what happens when you take some of these principles, a lot of this was informed by conversations I had with people leading AI-native teams at various companies. I think something like an MCP portal becomes interesting once you’re at a certain scale, where you effectively need a single point of entry for governance, security, auditing, and cost optimization. I’d be curious to hear what you see as the future—or maybe lack thereof—of something like this. If you believe in something like OpenRouter, a single point of entry for many things to connect to and be governed, I feel like there’s a world where this could easily exist.
My only tweak is that this folds into the LLM gateway. They’re the exact same platform.
I’ll give you that. I can accept that. And then, by the way, it also folds into the other thing—what did I say it was? AuthZ, permissioning, and billing. Ultimately, all infrastructure converges on the same problems: there’s a box in the middle, and then you have boxes on two sides. The God box will solve everything for the enterprise, and that’s true only if you understand the job of the God box. Separating MCP portals and LLM gateways means you don’t fully accept that they’re the same layer problem.
In a very similar way, I find that people don’t understand database querying. They separate filtering from ranking, or from general recommendation, but they’re the same problem. When you combine all these things in the same layer, you can make them depend on one another in useful ways. If they’re different layers, they can’t.
Yeah, I actually won’t disagree with this. As a slightly pedagogical point, I think this breaking up into separate terms versus one is absolutely correct. The underlying architecture is the God box, where one thing goes in and there’s a bunch of other stuff—third-party, internal, and first-party—on the other side.
I have a spicy tweet about this. Where is the God box tweet? Oh, yeah. This is the M×N to M+N architecture. Let me bring it up.
Sure. Sure.
Anyone working in infrastructure needs to understand this: congratulations, you now have a job in infrastructure, or you want to be an infrastructure engineer. Your entire job is making God boxes and then taking them apart. Either you don’t have a good God box, or you don’t like someone else’s God box, so you have to take it apart and make your own.
That’s funny. I like that. That’s very funny. Into the heat death of the universe. Wow.
This is a restatement of the idea that there are only 2 ways to make software: you bundle things or unbundle them. But what are you bundling? The things that go together, hopefully. Over time, people’s opinions shift about what goes together and what shouldn’t. Maybe Monaco should be more of a point solution; maybe Monaco should own everything. In some ways, you’re bundling for the startup segment you serve. In other ways, you’re unbundling from Salesforce, and that’s how you win market share from Salesforce.
That’s interesting. It’s an even broader statement, connecting it to vertical AI. Bundling and unbundling—that’s the whole Craigslist and Airbnb thing.
Yes, and Tinder and all those things are separate.
Literally separate. Tinder is basically a personals site.
Yeah, yeah. Never thought about it that way. One of those things: we just shipped the OpenRouter episode today, right? OpenRouter is a huge success. Lots of people tried to build gateways, but none of them were as successful as OpenRouter. OpenRouter only builds gateways. They disagree with me that the router layer should also do ops.
Because Arize is doing ops and observability. They don’t have an MCP gateway; they only handle model routing. I love being wrong like that. They’re saying, “We are the point solution, and we only do this one thing really, really well.” That’s their engineering focus, which is good, but it’s also their marketing focus, which is nice.
I think what you’ve described—the one-stop shop for all the things—is ideologically what makes sense. But in practice, especially in the enterprise, this is broadly under infrastructure, under platform, and there are separate teams owning different parts of the stack. For that consolidation to happen at enterprise scale, someone has to come together and say, “Now there’s a single endpoint you’re hitting for both of these things.”
And that’s fundamentally a hard thing to get everyone on board with.
Yeah. So often it’s more of a political or people problem than a pure software design problem. From a software perspective, this makes a ton of sense.
Yeah. But software systems are sometimes bottlenecked by people problems more than pure software problems.
Right. My framing is usually: you ship your org chart. I forget the name of the principle, but Conway’s Law says you ship your org chart. You buy your org chart. When you start a startup or make a tool, who in the company are your best friends?
Hmm. Hmm.
And if it’s for everyone, it’s for no one.
That’s interesting. So a conscious part of Latent Space and AI Engineer is, “Okay, there’s this category of software engineer that really cares a lot about AI. Is it going to become their day job? Let’s only make it for them.”
Right. Not a bundling of engineering. It’s never going to be the dev tool for everyone.
We talk about databases and security, but there’s an AI angle to it.
Mm-hmm.
We’ll see. That makes sense. Cool. Software factory. One thing I’ll shout out, related to this point, is that I think a role of the future that’s increasingly important is platform engineer. This is a huge one.
More lore: at Temporal, before all this, we wanted to own platform engineering. If I hadn’t started AI Engineer, I would’ve become a platform engineer. That’s the role that decides all this mission-critical stuff, and then the kids over in applications can do whatever they want on top of our platform. The big boys do platform.
And I think nowadays they own cloud, DevEx, security, SecOps. It’s all the things where you can actually lose the company or make or break it. It’s platform people. Product people can adjust the CSS.
That’s how you’re starting a fight here.
I’m not the CSS adjuster, but I appreciate everyone.
It’s a fun time. What’s the future? I feel like all the guys can adjust all the things.
Yeah. So there are a few aspects here. There’s a little bit of spillover from the previous week: coding agents and big teams, some of those concepts. But we’ll touch on the software factory. I want people to start thinking about this world of the machine that builds the machine, and where, if at all, people are still in the loop. Today, I think the answer is some combination of intent and review: intent, where you’re defining the specs and what needs to be built, and review, where you’re doing some amount of your own checking of the system. I think that bottleneck is going to loosen up before even the intent piece does. For now, those are the 2 choke points in a fully automated system. I want to give people exposure to that.
For one of the open-source partners, Warp, they’re doing this as part of their contribution to the open-source program. They’re going to make a version of Warp that runs through the factory, and students will be building and contributing to the factory and keeping it running.
Yeah.
Work in our factory.
Exactly.
Then they’ll find things they find interesting and pull them back in. This is one of those things where I’m not sure. Obviously, Factory itself is very keen on this. I feel like everyone wants to build their own.
Yeah.
That’s a good question. They’re entitled to try to sell the products they make. I think it’s a hard system to own in its entirety. It’s a lot. “Decide everything for me, please.” It is a lot. I haven’t gone too deep in understanding how they approach their go-to-market here, but I suspect you’d have to do it piecemeal. You’d say, “Hey, we first do your coding agent. Oh, then also…” You basically land and expand. We talked about this bundling and unbundling piece. You don’t start by saying, “Here’s your whole factory.” There’s no way you could get a Ford on board with that, just plopping the whole thing on their desk. At least, that’s how I’d think about it: first be their coding agent of choice, build up some background agent for them, own some of their SRE pieces, and then say, “Now we can weave it all together if you define everything in code or configs.”
There was a fun blog post about this from OpenAI. Our buddy Ryan—we had Ryan on the podcast. They spent 5 months making an internal tool with zero lines of manually written code. They walk through it. It’s called “Harness Engineering,” so you can learn about harness engineering. One of the learnings was that there was a slump where it was slower at first, and then you offload, and you have peer review. It’s from February, but it’s a vision of a software factory: fully committing to letting agents write everything, and how to do that well.
Absolutely. Some version of this is going to exist. I do wonder about the sales piece here, the actual business of this. That’s an interesting question.
From a principles perspective, I think this has been the vision of the future. Beyond that, I think there’s going to be some discussion about what it means to be a developer in the future. People talk about taste, understanding systems and architecture, and software taste. I want to go a little deeper and start putting some answers to that question. So this is your answer to where AI software engineering goes next?
Yeah. The factory is tactical, but beyond that there’s a question around the role. If the factory is running full steam ahead and the conveyor belt is churning out code, what do we do? You and I had this discussion with Jerry at the dinner a few weeks back with Dex and Jerry, around the factory. One of the points there that I’m pretty much on board with is increasing levels of abstraction, and that somehow someone, even a CEO, still has to have enough oversight and understanding of the different constituent parts to make a judgment on direction, course-correct, and set metrics.
There’s an analogy that translates to someone who oversees the technical side. We call it a manager of agents, or use different terms. They’re all buzzwordy, but in some respects, it’s someone who understands product requirements and constraints, technical constraints, loops through all of those, and knows how to adjudicate.
In the chair before you, we just had Anthropic’s head of product and head of engineering for the platform. I asked, “When are your jobs going to combine?” They said, “Not yet.” If Anthropic isn’t there yet, you shouldn’t feel like it’s imminent.
Good. Did they say how long, or was it just “not yet”?
Not yet, but who knows? There was definitely a lot of, “Okay, yeah, I’m doing her job. She’s doing my job.”
Yeah, I think it’s also an interesting perspective for students, right? You said there are a lot of sophomore undergrads. They’ve got 2 or 3 years to plan out what they want to be doing.
Yeah, there are different ways to look at it. You’re not straight in industry. The thing I care about, and think about a lot, is the loop stuff. You start writing prompts, and then you no longer write prompts. You write prompts that write prompts.
That’s right.
And then you just keep going up. I think the recursive, concentric-circle nature of going outward but also inward is something that builds the factory. I think you build up to the factory, not build down.
That’s right.
But I think a lot of people will eventually start building down. You start from the top level—what is my goal?—and then figure out the systems. That currently creates a ton of slop, and you shouldn’t do that. But someday you will.
Mm-hmm.
So that would be my question to prompt you: What’s the prime question that people exist to solve? That’s your thing, right? Let’s say an engineer today: you’re measured by your PR merge rate.
Right.
PR contribution rate.
Well, 2 years from now, that’s going to be ridiculous. Okay, what’s your main thing?
Right, I don’t know. Money?
Yeah. People always go to money too quickly. It’s something in between money and—
Right, PRs.
Customer satisfaction—
CSAT scores from your features. Even that’s a product decision, not really a code decision, right?
Mm-hmm. I don’t know. But if engineering is evolving into more of a PM engineer, a product engineer, a hybrid, then—
Then I think that’s my answer to the question of what the role evolves into. I think it goes without saying that I would recommend students today spend more time thinking about holistic systems—not just their single part of the engineering puzzle, but how that ties in with everything else.
In practice, this is what ends up happening. For their capstone projects, I encourage students to go out and build something. In fact, the seeds of ideas I give them for final projects are the Y Combinator Requests for Startups. I’m almost like, “Go build a company.” Go build something interesting.
What did people build last year?
A variety of things. One person was in the E-school, studying EE, and built a translator for parsing manuals that have memory addresses in C, then converting that into code they could interpret and visualize. Stuff like that is specific to their use case.
Someone was a designer and is now at Vercel. She had an idea where you take a picture of your design notebook and generate an app based on it. For her demo, she snapped a picture of some random scribbles she’d made, and got a fully fleshed-out Vercel app. I think that probably helped her get a job at Vercel doing some version of that.
Wow, nice. That’s a high bar, actually. Vercel is very known for its design engineers.
Yeah, and I think she’s now working on Eve, from what I remember.
Ah. No v0 in your—
Eve is an OSS partner. One of the ways Vercel is partnering with us is by having students build with Eve.
Yeah. So this is the concept of meta-harnesses, right? Harnesses of harnesses. You can always go one level deeper.
I do think it makes sense, at least for me. In the same way that model routers wrap model APIs, you need to wrap harnesses as well if you’re going to swap between them.
Yeah, that makes sense.
Agreed. I think students should also stay grounded. There’s a lot happening at the edge, but there are also a lot of companies still using a single ChatGPT call in production, just playing around and not having tried much else. There’s still time for it to work.
Your point about taste is good. Have taste, try stuff. Whether each company has its own version of a software factory or you’re building tools to help companies do it, stay on top of it, use the tools, play with them, and you’ll figure it out along the way.
Yeah, and I think—100% agree with all that. First, to your point, a lot of companies are still prompting ChatGPT or some version of it. Maybe they just got a Cursor license and are now getting people onboarded. That is way more true than we believe, certainly here in the Valley. I can say that with 100% confidence.
But for students, being exposed to this is important. The fact that they know this and are on the edge will eventually let them be some of the more interesting and valuable contributors when they go to places that maybe aren’t up to snuff or on the frontier.
One progression-type talk that’s still very influential, from about a year ago, is Steve Yegge talking about the levels of psychosis. He’s all the way through level 10 now.
Where he’s—
He’s given up and moved to a farm. I’m making fun of him. He built Gas Town.
Yeah, I remember.
He moved to Gas City. Now he’s burned down the city and said, “I’m building my game. No more Gas Town.”
I thought that— I remember talking to him at the AI Engineer World’s Fair earlier this year. Wasn’t his whole thing that the only useful thing built at Gas Town was Gas Town?
Yeah.
Okay. That’s a horrible failure.
Totally. People should be aware that psychosis is real. If you’re not holding yourself accountable to delivering real products, you’re just screwing around with LLMs.
I wonder how much of it is relevant to this, but I played a little with Gas Town. It’s a very involved system, maybe a little ahead of its time. I’d also wonder if it was bottlenecked a little by the marketing, because it was difficult to grok from a terminology perspective. There’s just a lot going on there.
I think the Gas Town marketing also helped propel it. What was right before Gas Town?
Beads.
Yeah, Beads. There was OpenClaw, something else, and then Gas Town. The manifesto also really helped spur the concept. People who would never use Gas Town learned about it because they could read the manifesto. There’s a marketing angle to it.
I think Steve isn’t necessarily someone to take literally, but directionally, he’s correct. If you’re at the idea stage, it’s time to move into the power-user stage. If you’re using a single agent, it’s probably time to move to multi-agent, and you probably should try to build your own orchestrator. That’s all part of it.
Arguably, this is already out of date. There’s more to the software factory beyond this, and this article predates the software factory. As an employer, I have this problem in my company today: I don’t want to hire anyone who hasn’t been through psychosis and come out the other end, because I’m currently paying for somebody to go through psychosis. It’s very expensive in salary and tokens, and I’m getting nothing out of it. My feedback doesn’t get through.
It’s almost like I would encourage your students to have psychosis and then come out and ask, “What was wrong with me?” That’s what Steve just went through: a very public 2-year journey.
It’s weird on the student side. There’s a lot of industry where you think you’re being productive, but you’re not. That’s probably the student version. You’re making Notion charts and all these optimizations and personal apps, and then it’s like, “Dang, I went through the psychosis, but nothing has changed.”
Right. So go through it.
Go through it.
I haven’t seen this. I’ve seen a version of this from Boris, actually. It was more geared toward Claude use cases, but I remember he made a Claude artifact that showed the progression, I think more at the team level. He had 4 stages, not 8, but it roughly followed the same progression, from no adoption through to—
Only 4 is good. This is too many levels.
We’ll see if people can pull it up. Other than that, zooming out, have we discussed all the things—
That’s the one.
Oh, that’s the one? Okay. It’s the Boris—
Yeah, it is an artifact.
Okay, so it’s gated, assisted, parallel, supervised autonomy, and AI-native.
It’s zero-indexed, so it’s 5 levels. Things start at zero.
Wow. Actually, it’s pretty good. I need to read through this in more depth. And use AI-native.
Yeah. I’d give a shout-out to dynamic workflows. This is a sub-agent pattern that the OpenAI guys are also exploring, and it clearly fits. Basically, you’re writing a harness on demand for the thing you’re trying to do.
A lot of the sub-agent conversation is just copying the parent agent, but you actually should be writing way more code and then running that as a sub-agent. That’s what the Anthropic people have chosen to call dynamic workflows.
We built a version of this as well.
Yeah, it’s a good pattern. For Monaco?
Mm-hmm.
Yeah. What does it do?
On the fly, sort of like harness use cases for GTM.
Ah, yeah. That’s great.
I’ll note that when Boris wrote this, it was for team adoption, for people who are listening. He also says, “Anthropic is at stage 3. We’re still trying to get to stage 4.”
Wow. He’s like, “I’m at stage 4. I’m the guy.”
But it’s not as simple as Steve’s. Steve is very good at “do this, do this, do this.” This is much more for running a team and wanting to AI-pill your team, not just be a user of one.
Yeah. Anthropic is still at stage 3. Don’t be like—
Yeah.
Yeah, yeah.
Okay, I mean, you should teach this. This is great. This has got all this stuff—this is everything we talked about.
Yeah. I can’t believe I haven’t come across this.
Good resource, yeah. Very, very good. Okay.
That was a wild tour. One semester of Stanford in 1 hour—or 2 hours. Thank you for doing this. Any parting thoughts? Any call to action?
No, thanks. I mean, this has been a ton of fun. It’s been 9 months in the making, and I’m excited. This is the end of week 1. There’s a lot to do and a lot to learn. I’m excited to see what happens, especially with the OSS partners and getting students deep in the code of some complex codebases and having them build some stuff. It’ll be fun. I’ll keep you guys posted.
You’re doing another version next year? If OSS people want to get involved, they need credits. Can they reach out?
Sure. Sure, yeah.
Do credits matter? Do they want them? Do they actually want the credits, like the students, or—
Yeah, they want credits. You do have a credit card. I don’t think a lot of students want to spend 5 bucks. I work in AI, but—
Yeah, no, I think—
Look, I remember there was a certain number of credits for sure, but on top of that, he was like, “Look, if they want to go hog wild with this, let me know, and we’ll do these one-off experiments where they can scale up to 1,000-plus agents.” And so—
Great. This is an email from Vercel.
Yeah, an email from Vercel.
Yeah. Those kinds of people are valuable. So this is Wilson from Cursor, right? Who’s like, “I’m going to write a browser,” right?
Yeah. That’s right.
Those people who are ambitious and can actually execute are very valuable, and that’s basically a recruiting call.
That’s right.
But—
Yeah, that’s a good point. I do think students should understand how to be ambitious because snapping a photo and turning it into an app is not that ambitious in this day and age. Writing a browser is. And there are a lot more projects like that.
Yeah. I think that’s right.
It’s all about raising the floor. Last year, credit’s due to the students of last year, but this year, obviously, we’re going to expect more.
Yeah, that’s the floor now. We’re going to expect more.
But that’s the whole point, right? We’re leveling up. Last year it was like, “Hey, have you heard of prompting? Here’s what it is.” I spent some time talking about K-shot prompting as one of the lessons, and I don’t even think you guys think about that now.
I guess there’s a version where you stick out in your cohort of classes, and then there’s also a version where you’re in early engineering roles with a bunch of people in industry, and people are still trying to get jobs. Not to call out one demo, but you can do a lot to stick out.
For sure. I’ll give students a freebie for week 11: make benchmarks. Make benchmarks and evals. This will get you a good one. It’s going to get you in the door of every frontier lab and every top agent lab.
Good engineering blog posts with some rigor on anything interesting. There’s a lot of low-hanging fruit there, but engineering blog posts do really well, right? You went deep on one thing.
Yeah. There’s also a lot of money to be made in that, so it’s not just about getting in the door. If you turn that into a business, you can really cash in on it.
People don’t know, SWE-bench was an undergrad project.
Yes, yes, yes. From Princeton, right?
Yes.
Yeah, yeah, yeah.
Great. Okay. Well—
Awesome. Fantastic. That’s it. Thank you so much.
Thanks a lot.
Yeah, thank you, guys.