第042期 - GDPx：前沿 AI 的基准 | Andrew Megalaa, Jordan Nanos, Max Kan
Jordan NanosMax KanAndrew Megalaa
- GDPX瞄准的是其创作者认为最先带来1万亿美元AI收入的领域。Max Kan的判断是，这笔收入将高度集中在少数几个领域：更强的模型能在这些领域持续创造高回报，包括芯片设计、生物学和网络安全。他预计，进入2027年后，大多数常规任务仍不需要前沿智能；因此，GDPX衡量的是经过刻意筛选的一部分经济活动。
- 基准的核心区别，在于任务本身够难，还是评测设计有缺陷。Max将其概括为“难在任务，还是评测不公”（“difficult versus unfair”）：正确答案也可能被判错，因为验证器只认另一种同样成立的解法。SemiAnalysis从14家专业数据供应商处征集任务，淘汰的供应商多于最终纳入的数量，并将大部分任务保密，以减少针对基准优化的空间。
- 材料科学最清楚地揭示了模型能力与成本之间的意外落差。Astra在一个子任务上超过了已发表的业界最佳结果；贡献者Brian则发现，部分看似困难的工作流之所以失败，是因为他“提示方式不对”（“prompting it wrong”）或提供的上下文不足。Jordan Nanos指出，材料科学任务成本最高达到40至50美元，而有些模型约花10美元就能完成任务。
- 要可靠地自主完成真实的金融和法律工作，模型仍有明显差距。金融环境还原了一笔涉及一家零食公司的交易：以略高于8亿美元收购62%股权，文件约有4,000份。按二元评分，金融任务基本全军覆没，只有Carbon零星答对；Opus的部分得分也只有50%出头，法律任务的最佳部分得分则在30%或40%区间。
- 安全限制会实质改变模型的实用性，即使其底层能力可能更强。Andrew认为，过于严格的安全分类器理应受到一定惩罚；Astra因拒绝执行任务，在网络安全上得分为0%。Muse Wonder和Muse Wonderful被评为网络安全领域的“顶尖选择”，也印证了Jordan的实际体验；不过，Muse Wonderful当时无法通过公开API使用。
- Carbon提升了Google的竞争位置，但在Jordan偏好的比较口径下，仍未追上领先实验室。排除生物学、网络安全和对齐任务，并采用部分评分后，Gemini 4 Carbon仍落后于Astra和Opus。Jordan认为Carbon是Google的转折点，称其为“第三名选手”；他也提到，OpenAI和Anthropic预计很快会推出新的旗舰模型。
- 就业争论仍无定论，而初级岗位承压的程度可能远超简单的生产力叙事。Max认为，人的判断力眼下让优秀工程师更有价值，同时也承认未来可能只需要少得多的主管。Andrew则质疑，被取代的员工会因此变得更有野心这一假设：很多人只想“上完自己的班，然后回家”。
1. 难基准，首先得是公平基准
Max强调的核心区别是“难在任务，还是评测不公”：模型可能通过模块B正确修复了代码错误，验证器却只接受模块A。此时，失败反映的是评分器与提示要求不匹配。他认为，常用基准里有很多这类问题；要识别出来，必须检查模型的实际执行过程，判断模型为何失败，而不是只看任务描述。Andrew补充说，模型可能用意料之外的方式解题，因此团队会审阅交互记录，有时还会借助AI探查结果是否正确、评测是否公平。
SemiAnalysis没有假定自己能独立设计出200个优质任务。团队招募了14家专业数据供应商，统一负责质量审核，最终淘汰的供应商多于纳入的数量。入选合作方也经历了大量迭代。Max的重点是，要让任务同时贴近现实、公平且足够困难，既需要领域专业知识，也需要反复检视模型的真实表现。
这套基准为团队提供了一种可重复的方法，用来评估他们无法亲自大量试用的新模型。最终排名对团队而言确实陌生：任务周四才定稿，周日上线；完整排行榜则在发布前3至4小时才出炉。模型运行加上评分，轻易就会耗时24小时。Carbon就是团队此前几乎没接触过的模型之一；他们周四晚才获得访问权限，当时才第一次开始测试。
2. 前沿收入取决于那些能持续从智能中获益的工作
Max的核心判断是，“AI收入的第一个1万亿美元”将集中在少数几个领域：这些领域能持续吸收更智能的模型，并带来高投资回报。他认同大多数任务不需要前沿智能，而且预计进入2027年后这一点会更加明显。因此，GDPX瞄准的是前沿模型供应商未来的收入引擎，而不是所有知识工作的代表性切片。
仪表盘列出的领域包括递归自我改进、芯片设计、网络安全、生物学、金融、法律、量化交易、硬件设计、材料科学、对齐和软件工程。选题重点是可验证性，以及领域吸收智能的能力。对齐也反映了团队更广泛的评测重点：他们认为对齐对未来模型至关重要，而Fleet的专业能力也让将其纳入评测成为现实。
材料科学家Brian的经历展现了他看法的变化。他起初没想到，Astra在自己提出的某些工作流上会表现得这么好。实际设计任务后，他意识到自己“提示方式不对”（“prompting it wrong”），或是提供的上下文不足，于是需要设计更难的任务集。Max说，Brian看到Astra完成的工作，足以让一位人类材料科学家超过同行中位数。
Max表示，当Astra在某项材料科学子任务上超过已发表的业界最佳结果时，拿人类水平作比较就变得困难；有些芯片设计任务甚至需要公司团队协作完成。Jordan则给出了成本对比：材料科学中最贵的任务每个花费40至50美元，而有些模型约花10美元就能完成任务。
3. 安全过滤和专长差异重塑排行榜
总榜纳入了生物学、网络安全和对齐任务，但安全分类器有时会压低模型表现。Astra因拒绝执行任务，在网络安全上的得分为0%。Andrew认为，过于严格的安全分类器在一定程度上理应受到惩罚。
Jordan自己也发现Muse Wonder在网络安全上很强；Muse Wonderful的结果进一步印证了这一点。团队虽然无法通过公开API使用Muse Wonderful，仍设法拿到了它的结果。Jordan开玩笑说：“你们把我的超额收益全抖出来了”（“You guys gave away all my alpha there”）。他强调，比较结果时必须考虑用户实际要做的工作，以及实际能用到的模型。Andrew说，团队拿到的Anthropic权限只有最低档，或倒数第二档，但这仍会触发许多任务的限制。
Andrew很高兴自己对Opus 5.5的主观体验得到了验证，同时也承认Astra在材料科学和部分工程工作上表现格外出色。不同实验室各有强项。他的实际建议是，针对手头的工作使用表现最好的模型，而不是认定某一个模型在所有任务上都能胜出。
Jordan偏好的同类比较排除了生物学、网络安全和对齐任务，并对金融和法律采用部分评分。按这一口径，Gemini 4 Carbon仍落后于Astra和Opus。Jordan称Carbon是Google的转折点，认为它重新站稳了“第三名”的位置，同时也处在Tier 2阵营的头部。他表示，Google与OpenAI、Anthropic之间仍有明显差距，并提到后两家预计很快会推出旗舰模型。Max补充说，这一比较针对的是公开模型；OpenAI和Anthropic未必拿出了最强模型。
4. 一笔真实收购暴露金融任务的上下文难题
Max对比了两种做法：让银行家用两小时编出一个真实任务，或者直接还原一笔真实交易。GDPX的金融合作方使用了一笔私募股权收购案例：一家家族经营的零食公司以略高于8亿美元的价格出售62%股权。评测环境包含全部4,000份真实文件。对Max来说，这与让专家合成一份近似其工作的任务“有本质区别”。
文件来自私募股权公司、投资银行、会计师事务所，以及卖方的数据室；原有的文件组织方式和混乱程度都得到保留。任务取材于人们真正做过的工作，包括会计合伙人通过邮件交代助理的事项。Jordan尤其看重一点：团队可以还原任务分配时的数据室状态，而不是让每道任务都访问最终版本的全部文件。
有一道任务涉及一封要求将数据调整至Q4的邮件。模型漏掉这封邮件后，尽管其他工作看起来合理，最终数字仍然错误。Jordan向金融从业者朋友核实过，对方告诉他，助理绝对应该发现这封邮件。用Jordan的话说，在真实银行业务环境里，漏掉这一点是“不可接受的表现”。
5. 部分得分能体现进步，却不代表工作簿可以直接使用
金融任务的最终评分采用二元制，但工作簿的各个组成部分会单独计分。Max的理由很实际：一份盈利质量分析工作簿如果有一半数字错了，就不能算合格交付。主管不会说“干得好，小吉米”，然后自己把工作做完。部分评分衡量的是能力；最终结果则看分配的工作是否成功完成。
按二元评分，金融任务基本全军覆没，只有Carbon答对了一道。Jordan说，Opus的部分得分在50%出头。法律任务的二元评分全部为零；Jordan称最佳部分得分在30%或40%区间。这些差距值得关注，因为它们表明能力缺口不小，不只是被某个边缘要求卡住，导致其他工作功亏一篑。
Max承认，二元评分可能掩盖任务过于简单或评测不公的问题：如果二元得分为25%，部分得分却有97%，差的那3%可能只是模型本应满足的一项冷门要求。Jordan补充说，两种得分差距很大，也可能意味着任务不公平或设计有误。GDPX在大多数领域采用连续评分。例如，芯片设计中的PPA优化，可以根据进展在朴素基线的0分与理论上限的1分之间计分。
金融领域未来的token支出有利润空间支撑，能够承受相当高的成本；Max说，金融在2026年已经是一个重要支出领域，他预计2027年还会有大量投入。法律领域还存在“PvP”机制：如果更强的AI能改善诉讼结果或合同谈判，对手使用AI就可能迫使另一方购买更强的智能。
6. 更强的模型让部分员工更有价值，也威胁初级岗位
Andrew以前认为，人类加AI会一直显著强于AI单独工作，自己也会继续手写代码。他说，Sonnet 3.7、Sonnet 4、Sonnet 5，尤其是Opus 4.5和4.6，改变了他的看法。他仍认为人类加AI胜过单独使用AI，因为人能带来模型本身不具备的“些许熵”（“little pieces of entropy”）。
Max将基准结果与失业预测区分开来：GDPX刻意不评测客户服务等容易受影响的工作。优秀工程师眼下能提供“品味、监督和判断力”，但两年后，一个主管能否取代一家公司的大多数工程师，仍未可知。当模型能处理定义明确的问题，而管理实习生本身又会增加开销时，安排实习生任务也已经更难说服人。
Jordan认为，能力提升也可能带动需求扩张：网络安全需要更多专业人才，材料发现也可能催生更多工作。Max则反驳说，一位能力极强的材料科学家，仍可能让博士助理的工作自动化。转录中一位未具名发言者提出了乐观的可能：过去做基础工作的员工，可以用上“随叫随到的无限量博士级科研助理”（“infinite PhD grunt workers on tap”），转而探索更有野心的科学问题。
Andrew明确反对把这种乐观判断当作普遍预期：他认为，许多美国人只想上完自己的班、回家，而不是不断提升职业目标。Max则认为，AI可能迫使人们更在意自身价值，否则就无法创造价值。但最终结果仍未确定。双方争论的是人如何适应，而不是工具有没有进步。
7. 保持任务保密，才能守住基准的初衷
Jordan谈到公开任务的两难：公开能让外部专家检查质量，却也会让针对基准刷分变得更容易。基准可能不再衡量模型能力，转而衡量哪家实验室买到了最相似的训练数据。GDPX的首要目标是衡量“模型的真实能力”，而不是让每道任务都对公众开放。
折中方案是每家合作方公开一道任务及其模型执行过程，同时提供详细方法说明。佐证包括真实的金融交易数据室，以及网络安全合作方把自己的零日漏洞改造成任务。Max说，团队用自己的智能体做质量检查；他提到Bench 4曾判定约50%的任务有问题，而GDPX的任务之后交由每家合作方逐一测试，未达标的任务会被剔除。团队希望通过具体的构建细节和样例表现建立可信度，同时将大部分评测材料保密。
下一步目标是让GDPX v1出现在主要模型发布说明页上，之后再推出能应对任务饱和、纳入新兴高价值领域的版本。Max和Andrew希望引入更多强力合作方、增加各领域的任务数量，并覆盖所有推理水平的评测。这个基准应随着前沿模型能完成的工作而演进，而不是永久冻结在今天的任务组合上。
完整逐字稿
GDPX taking the world by storm.
Something like that.
1. Launch Week Chaos
Okay, nice job, guys. Your benchmark matches your—what did you say?
Our subjective impression of the models.
Yes. You’re pleasantly surprised that the benchmark matches your subjective impression of the models.
Yeah. To be clear, we had no idea what the final leaderboard was going to look like until about 2 hours before publishing—maybe 4 hours. We had all the partners calibrate difficulty on 4 models. They ran Astra, Opus, 3.8 Flash, and some Argon, because we were working on the benchmark before Argon came out, and then also GLM-5.3.
We only had results for those 4 models until about 3 or 4 hours before the benchmark launched. Andrew switched to polyphasic sleep over the past week so he could build all the infrastructure. He set things up so Claude would wake him if something broke and he needed to provide input. Then he set up all the infrastructure and started running the rollouts. These are incredibly long rollouts; doing the task plus grading could easily take 24 hours.
We didn’t have all the final results until literally 3 or 4 hours before we published. When we saw his leaderboard, we were like, “Yo, I think you might have, dude.” Some of these models we hadn’t used or even seen out in the wild. We got access to Carbon on Thursday night or something; we hadn’t used it for anything. It was the first time we were trying it out, and it did well. It was surprising. Same with Muse Wonderful. It isn’t available in the API, but we got access to it, and it did great on Cyber.
Yeah. When did you fix the tasks? When did those get finalized?
Thursday night. We launched on Sunday, and the tasks were finalized Thursday night because we were doing a lot of back-and-forth iteration with our partners. Our QA would flag something and say, “Hey, take a look at this.” Then they’d say, “Oh, these two are false positives, but this one’s for real,” and we’d get updates.
It was really last-minute. I’m very happy we got it out in time. It was very hard, though. I’m also glad to be taking a break now. It was a crazy process.
I think the work starts now, man. I don’t know.
The work? Okay, I’m going to take a couple of chill days.
Yeah.
And then we can go through the process of, yeah, I guess, reaping the rewards of our effort—
Maintaining the dashboard as new models get released. Did you impose this deadline on yourself? What was the motivation for that?
It was mostly self-imposed. We were pretty confident that a new OpenAI and Anthropic flagship were probably both coming out this week, and you obviously can’t launch your benchmark on the same day as Fable 5.5.
Especially if your benchmark doesn’t have it.
Yeah. I think we were pretty confident that Monday wouldn’t be the release date for these models. Maybe it would be Tuesday or Thursday, but if there was any risk of a model coming out on Monday, we were like, “Fuck it, we’re just going to push Sunday instead.” There was some fear that Sunday night people wouldn’t be as online, so you wouldn’t get the virality component. But I think we were very happy with how the launch turned out.
2. Broken Benchmarks
Yeah, generally happy.
Take a step back and explain GDPX from scratch for people who maybe haven’t seen the article or are just hearing about it for the first time—
Or didn’t read it, like many people commenting about it.
Yeah. Maybe taking a step all the way back, I think it’s important to understand that a lot of the really popular benchmarks out there are kind of trash. This is largely because they’re made by people who don’t have expertise in making eval environments or training data in general. It’s very easy to make a task that seems good on the surface but is actually unfair in some way, making it a poor test of model capabilities.
A lot of people make their tasks harder by making them unfair rather than actually making the task difficult. It’s very hard to figure this out unless you run 10 rollouts, look at the transcripts, see why the model failed, and manually determine whether that was a legitimate failure.
3. Difficult vs. Unfair
Yeah. Can you explain the distinction between difficult and impossible here? You wrote about this in the article. It has to do with both the prompt and the grader.
Yeah. Maybe “impossible” isn’t quite the right word. It’s more like difficult versus unfair. A classic example of an unfair task would be a coding problem that asks a model to fix a bug in an open-source repository. An expert human software engineer might look at it and say, “There are 2 ways to fix this bug. You can use either module A or module B. I recognize that both are equally fair and valid. You should be able to pick whichever one you want.”
But when we’re making these benchmarks, you also need to include a verifier for each task so you can check whether the model succeeded. It’s totally possible that someone might make a verifier scoped only to the module A solution. Then an AI might rightfully pick module B instead, and the verifier would mark that as a failure. The AI would perform worse on the task, even though it technically solved the problem. That’s an example of the task being unfair: your grader isn’t faithful to the prompt.
It turns out that a lot of popular benchmarks, like Terminal-Bench 4, SWE-bench Pro, DeepSuite, and others, have tons of issues of this type. People have started noticing this over the past month or so. There have been a couple of big blog posts about it, because data companies have agents look at all the transcripts and prompts, and run the tasks as part of their own QA process. They’ll see that 60% of the Terminal-Bench 4 tasks are broken or something.
4. Building GDPx
The funny thing is, I feel like everybody knew it. All the labs definitely know it. Anyone who looks at the data knew the benchmarks were broken. It’s just that no one spends the time to write about it, because to most people in the industry, it’s not new information. It’s unsurprising.
5. Public vs. Private Tasks
Yeah. So benchmarks are bad. That’s one thing to know. We at SemiAnalysis knew we wanted to make a benchmark ourselves, because it’s kind of the only way you can consistently say interesting things about new model releases. The team isn’t going to spend meaningful time and effort trying out the new MiniAX model when it comes out to see if it’s actually good. No disrespect to MiniMax, but we’re just not going to do that. You really want an eval, and we’re not going to do—no one tested GPG 6.1 Terra either, right?
If you have a really high-quality eval that you can regularly run models on, it can go a long way toward tracking their performance. But at the same time, we recognized that a lot of public benchmarks are bad—not because people aren’t putting in effort. It’s also just really hard to make a task that’s fair, sufficiently difficult, interesting, and reflective of real work.
It would have been naive to think SemiAnalysis could come up with 200 really high-quality tasks ourselves. Our strategy was to partner with 14 specialized data companies that used a lot of their expertise to supply tasks for each domain. We were responsible for QAing all of them. We dropped more data providers than we included in the final benchmark because a lot of them failed QA. We were also heavily involved in task iteration with the providers who passed QA, as we mentioned at the start of the episode.
6. Frontier Domains
That was the philosophy and methodology for making the benchmark. In terms of what it covers, that’s another big question we needed to figure out, because you can pick any domain. AI is affecting everything these days. One thing we strongly believe on the Tech Economics team is that the first trillion dollars of AI revenue will be highly concentrated within a few domains that can absorb increasingly smarter frontier intelligence and earn a high ROI.
I think a lot of people have recently seen these token-budgeting trends, or heard people say, “You actually don’t need frontier intelligence for the majority of tasks,” and taken that as bearish for the AI trade. We totally agree that many tasks won’t need frontier intelligence. But we think there are a lot of domains, like chip design, biology, and cybersecurity, where we’re going to need increasingly smarter frontier models, and you’ll continue to get high ROI from Fable 6, Fable 7, Fable 8, whatever.
So we decided to pick domains for our benchmark, GDPX, that match those domains that can absorb frontier intelligence. That was our guiding philosophy.
7. The 11 Domains
Sweet. Okay. Maybe we can talk through the specific domains in a little more detail. For those who don’t know, there’s a website up, gdpx.semianalysis.com. I’ll share it on screen for a second. As we scroll through the domains, we have recursive self-improvement, or RSI; chip design; cybersecurity; biology; finance; legal; quant trading; hardware design; materials science; alignment; and software engineering.
I don’t think this is really a breakdown that a lot of people have on the tip of their tongue when they’re talking about the domains AI is coming to. Even just introducing these one by one would be interesting. Maybe you can start by saying how you arrived at these categories broadly, and then how you found the specific data vendors you went with in the categories you focused on.
Yeah. Arriving at the domains involved a lot of brainstorming and trying to pick domains that are verifiable and seem like they have the capacity to absorb a lot of intelligence. After we had our initial list, we talked to a bunch of people at the labs and people working on similar problems. We went through that process for a few weeks until we arrived at our final seven domains.
In terms of how we picked the partners, we’re fortunately pretty well connected in the data space, and we happen to know a lot of the environment companies and what each of their specialties are. We reached out to a bunch of people working in the various domains, got sample tasks from them, and QA’d them. We dropped the ones that were bad. In cases where we had more than 1 good person who wanted to work exclusively with us in a particular domain, we’d break ties based on things like our tastes, how interesting we thought the problems were, and how representative they were of their domain.
That was basically the whole process. Eventually, people started hearing about GDPX on their own and reaching out to us to be included in the eval, which was kind of interesting. It spread around the data space.
Yeah. Some of them were more opportunistic. Alignment is a good example where—
It’s not something you’d probably immediately reach for if you were trying to make a benchmark that measures agent capabilities. But I really do think it’s one of the most important things you need to evaluate future models on, so it deserves a place in any comprehensive benchmark.
At the same time, Fleet is one of the biggest, most well-respected data companies today. I have a few friends who work there, and when we were brainstorming domains we could potentially include, Fleet for alignment ended up making a ton of sense.
Yeah, makes sense. Maybe another way to think about it, when I was reading through some of these descriptions, is that these are clearly domains where people using frontier AI today are burning a ton of tokens.
8. Domain Experts
Absolutely. The domains are really interesting, though, because they vary so much. Some have very established data companies that have been selling data for a few years, while some, like materials science, are just one PhD who’s really good at materials science and delivered these insane tasks that are difficult even for Astra. I found that super cool, because they used papers they wrote, or papers written by people they know. Shout-out to them. Those are pretty cool evals.
9. Models vs. Experts
Yeah, my favorite part of the article is going through and reading about these pretty esoteric domains. It was so clear from the task details that the person who wrote them was a domain expert. They certainly knew more than me.
10. The Leaderboard
Mhm. And maybe you can talk a little bit about the reaction from the people you’ve worked with since the article came out, and how it seems like a lot of people were surprised by the results. I know Brian was specifically saying he was shocked that some of the tasks were solved by the—
Yeah, I think Brian initially had a brief moment where he didn’t expect Astra to be so good at his tasks when we first asked him for them. Some of the workflows he initially thought Astra wasn’t good at, once he sat down and made tasks, he realized that actually he was just prompting it wrong or not providing enough context, and it ended up solving them.
So he had to make another set that was actually difficult and frontier-level, because I think a lot of people don’t realize how good the models are. He even said this: materials scientists don’t know how good the models are, and they think the models can’t do certain things, but really they can.
Well, you should talk about the experience you had with software engineering at the start of the year and how that relates to Brian.
Yeah.
11. AI and Jobs
Yeah. I think I went through a similar thing. It was actually a year ago at this point. After I left my job to start an RL company with Max here, at first I still thought humans plus AI was just a lot better. I thought I couldn’t really take a software engineering job; we’d still be coding by hand, while AI would automate the easy things. But then Sonnet 3.7 was really good, Sonnet 4 came out, Sonnet 5, and then Opus 4.6. That Opus 4.6 was a moment. That sounds weird.
Opus 4.5 was the first one.
Opus 4.5 was really good, but Opus 4.6 was also really, really good.
Like, those 2 models were just—
12. Cyber and Safety
Yeah, I think if you give it a few more iterations, there’ll be no need for me to look at a lot of this code. It was a very clear trend, and I had to reconcile that myself because I really liked coding at the time. That was my thing. I was just good at coding and systems. But I had to figure out, “Oh, actually, I have to take on the higher-level things now. This is no longer a job where I’ll always be writing code by hand.”
I still think humans plus AI are better than AI alone, just because there are little pieces of entropy that you can provide that the models don’t have themselves. And so—
Still valuable.
Yeah. But funny enough, the reason why I wanted Andrew to talk about that is I think Brian had a very similar moment for materials science when he made the task for the benchmark. He was like, “Holy shit. If any human materials scientist could do what Astra just did, they’re probably a top— they’re definitely better than the median materials scientist already.” He watched Astra do this, and I think we’re probably going to have similar moments in a lot of other domains, like chip design, materials science, and the other GDPX domains, in the imminent future.
That’s something we also hope to see in actual benchmark results as we’re able to evaluate Fable 5.5, Astra 6.1, 6.2, or whatever. Someone asked whether GDPval had human baselines for how long tasks take, to put things in context. That was really funny, because I responded that it doesn’t even make sense to have human baselines anymore. In the materials science tasks, Astra solved a subtask better than a published state-of-the-art reference. When models are essentially past expert level, how do you even set a human baseline? Some of these chip design tasks are so long-horizon and would take teams at companies to do. There’s no human baseline there. How do you even put a number on it?
It’s a lot of money. Whatever the hourly equivalent is in human work hours, it’s a lot of money. I just threw up on screen the cost per task for the materials science work. The biggest ones were $40 or $50 per task, and many of the models are solving some of these tasks for $10, which I think is a pretty unbelievable statement.
13. Model Behavior
Maybe you can talk about reading some of the rollouts in more detail.
What were some of the pieces of model behavior that surprised you the most?
The models find very creative ways to solve the tasks. This goes back to the whole unfair-verifier thing: people don’t expect the way models solve tasks because models aren’t human. They solve things in very weird ways. Maybe they use Module B instead of Module A, for example, but sometimes they just have completely different taste in how things are done. Sometimes it’s correct and sometimes it’s not, which is why we have to go through these transcripts. It can be hard to go through them because we’re not human experts, but luckily you can benefit a lot from chatting with AI about certain rollouts to figure out whether the stuff is correct. The models are so smart on their own that, with human probing, you can figure out whether something is fair or not.
Yeah.
It’s a very interesting experience, for sure.
14. Frontier vs. Day-to-Day
Maybe we can talk about the results. I’m realizing I haven’t even shared the flagship leaderboard screenshot at the top of the article yet. You guys can talk about what you learned about the models themselves. We’ve been going back and forth, trying to explain the difference between the flagship leaderboard across all domains and the next one, where you filter out biology, cybersecurity, and alignment. In those domains, safety classifiers can prevent models from demonstrating performance that’s in the weights somewhere.
Yeah, I think that’s a really good point worth emphasizing because we went back and forth on which screenshot should be the headline result: the one where you include all the domains or the one where you exclude cybersecurity, biology, and alignment. Those were the 3 domains where safety classifiers sometimes kicked in and sandbagged the score for some models. If you look at Astra on cyber, for example, the score is 0% because it just refused to do the cyber task. Ultimately, we decided the one including all domains is better, but I think it’s fair to penalize overly strict safety classifiers to some degree.
I know you’ve experienced this yourself, Jordan, doing your day-to-day work at SemiAnalysis and then getting hit with a cyber safety classifier from Opus Rational. You’re like, “Well, I guess I’ve got to use GLM-5.3 now or whatever.”
No, Jordan had a very interesting insight. He figured out that Muse Wonder is a top-tier cyber model independently, and he was right. Muse Wonder and Muse Wonderful were just top-tier for what you can get out of the box for a cyber model.
Yeah.
That was a very surprising result for us because Jordan already knew, but we didn’t have the Muse Wonderful number until 2 hours before launch. That was pretty cool. We were like, “Wow, Muse Wonderful just crushes it.”
Yeah, you guys gave away all my alpha there. Using the models yourself for something where you think you’re pushing the frontier absolutely gives you insight into which models are good and which models are bad. But you have to actually have that work to do, and I had that work to do in one of the 11 domains we’re talking about here.
Yeah, I could figure that out. I do think probably the biggest benefit of this benchmark is the ability to filter the results as you see fit and interpret them as relevant to you. People with Daybreak access or CVP access from Anthropic—which we sometimes have—
Well, there are 3 or 4 tiers, and we only had the bottom tier, or maybe the second-to-worst tier. That tier is more or less useless because you still get flagged doing basically everything. I think they both have very confusing access programs for the models today, which hopefully they can improve.
Yeah, yeah.
The crazy thing is, Muse Wonderful does that well in cyber. I wonder what red-team access on Astra would do.
No one can use it.
Yeah.
Unless you work there, I guess.
We’ll get there.
But maybe the better question is: can you emphasize again that this benchmark is meant to measure model performance at the frontier rather than on day-to-day tasks?
Yeah, for sure. Our core thesis is that day-to-day tasks just don’t actually matter, right?
They’re more or less solved.
I think the majority of them are more or less solved. Heading into 2027, you’re not going to need frontier models for most of these. We think they’re poor tasks to use to measure frontier model capabilities, and we also think they probably won’t make up the majority of revenue for OpenAI, Anthropic, and the other frontier-model companies. The things we’re most interested in tracking with this benchmark are where frontier-model capabilities are going, and whether you can do any TAM or revenue analysis based on that. So you very explicitly focus on the frontier domains.
Yeah. When we say day-to-day tasks, that doesn’t even exclude some of the domains people want to use these models for. It was surprising to see how bad the models are in finance and legal, given how much people talk about automating those fields. When we looked at some of the tasks we got from Living Context and Chakra—
We went through those rollouts ourselves because we were confused. I actually thought the models were better at these.
Yeah, Andrew is a tech-bro supremacist. He was like, “Dude, the models have definitely saturated finance already.” That was his prior coming into this.
Yeah, and we actually had some of our finance friends look at these tasks. When we were choosing partners for these, we had different—
Rollouts and tasks, and some of them were like, “Yeah, these tasks are pretty bad.”
Yeah, there were some tasks from potential finance partners where we were like, “Whoa, this isn’t realistic of actual finance work.” The only reason it’s sufficiently difficult for Opus and Astra is that it lacks realism and is unfair in these ways. After you correct the unfairness, it’s like, “Oh, look, Opus is saturated.”
That would make you think those tasks are solved, but then we got the really high-quality tasks, and our finance friends looked at them and said, “Yeah, this is totally realistic of something I would have done in banking and PE.” The models just failed.
15. The Finance Task
Yeah. Actually, I want to talk more about the finance task in particular because it’s probably the most understandable domain to the average listener. I think it would be useful in giving people a sense of why GDPX is such a high-quality benchmark. Historically, for something like GDPval from OpenAI, they would go to a company like Mercor, which has this vast network of finance experts, and ask a finance professional to sit down for 2 hours and come up with a task representative of something they’d actually do in their job, then provide the gold solution. That’s the task they’d give your model. Obviously, there are a ton of biases when you tell someone, “Make a task for my benchmark and please try your best to make it realistic.”
What our finance partner did was take a real private-equity transaction where a private-equity firm bought a family-owned snack company. I think they paid a little over $800 million for 62% of the company. They had this actual transaction, with all 4,000 files corresponding to it, separated into the deal rooms for the private-equity firm, the investment bank, the accounting firm, and the snack company’s own VDR, because they were selling themselves.
And it was organized—or disorganized—exactly how it would be.
Yeah, it was the real files for this real transaction.
Anybody who’s worked in a VDR—a data room for an M&A transaction—has seen exactly this.
They had the real files, and obviously they de-anonymized them. Along with the files, they had all the real tasks that analysts, associates, and other people working on the deal had to do. They essentially converted this actual deal room into an environment where they could evaluate the models.
And then they found the emails where an accounting partner told an associate what to do as part of the task, and had the model do it. I think this is categorically different in terms of realism, interestingness, and quality from just taking some random banker off the street and asking them to come up with a task.
Okay. One more thing about this, Max, is the life cycle. I thought this was fascinating: they could effectively replay what the data room looked like when that task was assigned, right? So, like—
You’re going to add these 1,000 files into the data room over time, but not every task is assigned at the same time in the data room.
I thought that was fascinating.
Yeah, obviously super fascinating that this is—
—the depth at which people can go to make this task realistic. But some of the failure modes are so esoteric when I looked at them. I had to ask my finance friends, and they told me, “Yeah, you would totally be expected to have done this.” It might be one email from someone saying, “No, actually adjust these numbers to Q4,” or something. If the model didn’t find that email, it would fail because it had to find it to evaluate all the context relevant to the task. Sometimes the models didn’t look hard enough, missed it, and sent out the wrong numbers.
And if the human associate failed, they would have gotten yelled at.
Yeah, apparently that’s exactly how it would have worked. If you were an analyst and didn’t read that one email, if you weren’t on top of it, you would have gotten yelled at for sure.
They would have said, “Yeah, this is unacceptable behavior in a real banking environment.”
16. Binary vs. Partial Grading
I know some people on the Tokenomics team who might be listening to this podcast and having some flashbacks. Can you talk about the grading in this case? I think a lot of people would be confused about the partial-credit versus binary-grading approach you have to apply to some of these things that don’t necessarily have a binary outcome.
Yeah. Our finance tasks use binary grading, so the model either gets a zero or a one. But there’s also a continuous scoring component, because the final deliverable might be an Excel workbook, like a quality-of-earnings book an accountant had to make for a private equity deal. That workbook can be decomposed into a bunch of different parts: different cells where the model has to find the right number. You can assign partial credit for each of those atomic criteria.
But we still use binary grading at the end because if you were a real accounting associate and gave your boss a workbook where half the numbers were right and half were wrong, they’re not going to say—
“Oh man, great job, little Jimmy. You got half of it right. I’ll take it from here.”
17. Google's Comeback
“What the fuck, man? I’m firing your ass right now.” So I think it makes sense to use binary grading in those cases, even though there’s a partial-scoring component. Jordan can show that finance is basically at 0% with binary grading, except for Carbon, which got one task right. But if you switch to partial grading, you get much more signal about what the models actually did.
A lot of people on Twitter were criticizing us for using binary grading, because other benchmarks sometimes use it to hide the fact that their tasks are quite easy. I won’t name any particular names, but there are popular benchmarks where you look at binary grading and it’s like, “Oh, 25% on Opus; this is probably pretty hard.” But then you switch to partial grading, and it’s like, “97% on Opus.” Then it’s like, okay, that last 3% is probably some esoteric requirement that there’s no reason you shouldn’t have expected.
Yeah. Or your task is just unfair and broken in some way. I agree: if our finance tasks were at 95% on Opus with partial grading, they’d probably be bad tasks. But you saw they were in the 50-something-percent range. It’s the same with legal. Legal is all zeros, but even with partial grading, the best model is in the 30s or 40s. Most of our other domains use continuous grading, because that makes sense for them. You might have a chip-design task where the model is asked to do some PPA optimization, with zero as a naive solution and one as the maximum theoretically achievable solution. You give the model partial credit based on how much it can optimize the chip design. We chose the grading strategy that makes the most sense for each domain; we didn’t use binary grading for everything just to artificially lower the score.
Yeah. I think this goes back to the comment about frontier intelligence versus day-to-day tasks. I don’t know that “frontier” in finance necessarily represents the same thing as it does in science, where it’s clear you’re pursuing a new discovery or something like that. But in finance, the frontier is being accurate.
Yeah. One thing to note about finance is that, at first glance, you might think it’s different from chip design or materials science. But we think it’s going to be one of the domains that spends very heavily on tokens in 2027. It’s already one of the main domains people spend tokens on in 2026, largely because the businesses’ margin structures support it. They can afford to spend a ton of money on tokens.
But there are also some PvP components—and this is even more true for legal. If AI could do a better job winning a lawsuit or redlining a contract, that has implications, especially if your opponent in the scenario isn’t using superintelligent AI to do those things. I think finance and legal are domains that will be able to absorb a lot of frontier intelligence, whereas real estate or government might not.
Makes sense. And just to expand on that a little, I think I know the backdrop. You’re talking specifically about why you chose to focus on these domains rather than taking a cross-section of all knowledge work in the economy and making tasks for every single thing people do, right?
Yeah.
I think what listeners are probably thinking about right now is—
—the concept of AI-driven job loss.
If people are using frontier intelligence and gaining productivity, making more money, or winning more things, how does running a benchmark like this inform your view on whether AI is going to cause job loss in the next year, 2 years, or even a 5-year time horizon?
I actually think they’re orthogonal in some sense, because we purposely choose not to cover a lot of the white-collar knowledge-work domains where people are probably most afraid of job loss, like customer service.
As for whether we’ll have massive job loss from AI, it’s really hard to say. I think a lot of the doomers have probably been pleasantly surprised by how AI plus human co-working has evolved over the past year. Any software engineer who has their eyes on the ball will probably tell you that, given a well-specified problem, Opus is just a better software engineer than me, period. But at the same time, good software engineers are more valuable today than they’ve ever been.
Maybe we’re in a Goldilocks zone where a lot of humans can still provide the taste, supervision, and judgment that make them plus the model way more effective than the model alone. But maybe 2 years from now you’ll only need one dude at each company supervising all the software engineering, and basically everyone else will have been fired. It’s possible that’s the world we end up in, but it’s also possible it isn’t. We have to pay close attention to model capabilities to see. I’ll say it’s definitely raised the bar for the kind of person you’re looking for to work on things like this. When you’re looking at software engineering, for example—
I feel bad for the interns.
Yes. Essentially, you have to be really good to provide more value than Opus at this point as a software engineer.
Yeah. A lot of people viewed interns as people you give well-specified, well-defined problems you don’t want to do, and they go do them for you. But obviously now you can just tell Claude to do that, and all the additional overhead of managing an intern often becomes unbearable in those cases. I know one of the big quant trading firms had a post about this. They said their interns finished their projects in the first few weeks this year, compared to previous years.
Really?
Yeah.
Hey, hey, hey. At least interns know how to use AI. That’s good.
Yeah. I think the typical response is that you need to get more ambitious. It makes sense. Going back to cybersecurity, I’ve been doing some work recently. We saw this recently with the increase in CVEs. I posted this on October 1. It’s been a month, and I don’t have an update. There’s a lot more than just this at this point.
Um.
Dude, is the whole internet just broken now? What’s going to happen? What are you going to do, Jordan?
Well, I think it’s a—yeah, I can—
I can talk about that in a lot more detail. Let’s save that for another time. But let’s use it to make the point that I think there’s a lot more demand for cybersecurity experience right now, even when using the models.
Yeah, for sure.
And I think that’s one of the first domains, as a product of software engineering, that’s getting knocked down. But if you’re pushing the frontiers of materials science, for example, you’re probably not automating away the jobs of materials scientists. You’re probably starting to discover a whole bunch of new materials that need to go through the whole discovery phase.
Well, no. I think it is very possible that you’ve automated the job of your PhD candidate assistant if you’re a cracked materials scientist.
I think this is exactly what we’ve heard from—
Yeah. If they have no ambition to go beyond what they were doing today or last year—
Yeah.
—then there’s nothing useful for them to do. But if they have ambition to accomplish more with the tools that are available, then the person who would have been forced to be a PhD candidate grunt worker two years ago can now leverage their infinite PhD grunt workers on tap to elevate themselves and do much more interesting materials science. That’s great. I’m generally very optimistic about human agency and people being able to take their lives into their own hands and do whatever makes them happy. I’m bullish on people using these tools to uplevel themselves. We’ll have to see what actually happens.
It was interesting. I actually completely disagree with that.
Andrew’s a hater. I think most people—
Well, when you look at it, most people in America just want to go in, put their hours in, and go home.
They don’t care that much about their careers. Maybe this is what forces them to care, because otherwise you won’t be able to provide value. We’ll see.
18. Quick Takes
Yeah, interesting. Okay, let me go back and get some quick reactions to results that we kind of glossed over. Andrew, you love Opus 5.5. Were you super happy to see it come out on top on a lot of these leaderboard charts?
Justified. Where did you get that insight?
Obviously, I’m glad that my subjective experience was validated. Though in some domains I’m not particularly involved in, it was very clear that Astra was ridiculously strong on materials science and some of the engineering stuff. Maybe that’s just because we don’t have Fable 5.5 yet, but we’ll see. It’s good to see that different labs have different strengths, because there’s a spike here and there, so you can leverage whatever is best for what you’re actually doing.
Yeah, makes sense. How about Gemini 4 Carbon? Quick take there. We saw Sundar quote-tweet everything there. You guys are excited about that?
Yeah, that was sick.
Shout out, Sundar.
I mean, we’re excited about it, too. More frontier models are better for all of us. We definitely want better models from everybody. It only benefits the consumer as there’s more competition.
Yeah. I do, however, think the right apples-to-apples comparison for where the frontier labs currently sit is to exclude cyber, bio, and alignment, and look at that GDP score. Maybe even switch to partial scoring, too, because that gives you more signal on legal and finance, where no one’s smart enough to do binary grading. At least you get rewarded for binary grading. If you do that ranking, you see that Carbon is still kind of behind Astra and Opus. That being said, it’s certainly an inflection in the trend for where Google’s model releases were going. At one point, it felt like they were going to fall farther and farther behind even Meta, potentially xAI, and potentially some of the Chinese open-source guys. But they’ve really reestablished themselves as the third-place player with Carbon.
How much of this is a timing thing, then? Sorry to interrupt, but Fable 5.5—you guys are teasing that it might come out soon. Obviously, Opus and Sonnet are on the list.
Yeah. I think the gap between Anthropic and OpenAI versus Google today is likely still very big, because we’re going to have new flagship model releases from them quite soon. Muse Wonderful came out pretty recently, right? So I think you’re evaluating Google’s position versus Meta and xAI. I think there’s a pretty strong argument that they’re at the top of that Tier 2 cohort. But I definitely still believe there’s a pretty meaningful gap between OpenAI and Anthropic versus—
And, you know, OpenAI and Anthropic obviously aren’t releasing their best models. So it’s just a comparison between the public models. With Gemini and Meta, they’re releasing the best they have so far.
Anthropic could just be on Mythos 4 at this point. Who knows?
Yeah, true. We’re going to get Fable plus one, because I think Anthropic is still kind of rolling with Fable-one-level models, essentially—or Mythos 1, I guess it’s called Fable 5. Internally, they’re probably at least two generations above that, and then we’re finally going to get one generation above that, hopefully this week. So I think the gap between the actual best model each lab has, including internal models, is very wide across OpenAI, Anthropic, Google, Meta, and SpaceX.
Yeah. Well, maybe the biggest takeaway for me there is just how relentless the pace of model releases is. The trend is for it to get faster and faster as they improve post-training and the data mixes.
RSI, bro. It’s RSI.
Yeah. Can you talk about next steps? One thing we didn’t talk about at all is the concept of releasing one task and one rollout for each category—or, well, for each of the data vendors, so multiple from some of the categories. Can you talk about the approach you took there, with the mix of partially public and partially private, and how other benchmarks approach it by comparison?
The trade-off you have to make for any benchmark is that the more public you make it, the more experts can QA everything themselves and verify that your stuff is actually good. At the same time, if you make it public, it becomes way too easy to hill-climb on the benchmark, and often your benchmark no longer measures model capability. It becomes more a measure of which lab bought more data that looks just like the tasks in my benchmark. I’m not going to name any particular names, but I think a lot of public benchmarks have already fit into that category. It’s really just a measure of which lab spent the most effort buying the most data of that shape.
Our priority is that people can’t hill-climb the benchmark. We want it to be an accurate measure of real model capabilities. At the same time, we know that establishing credibility is important, so releasing the rollouts for one task per partner seemed like a good trade-off. People can get some visibility into each partner and how confident they can be that the tasks are good. At the same time, we go pretty in-depth into the actual methodology for each partner, explaining how our finance guys actually use this real deal room, or how our cyber guys found their own zero-days and used them to task.
19. What's Next
I think binding all those details together is another way we can instill confidence in the quality of the benchmark. Also, giving details about our QA process helps. We say, “We ran our own agent to QA it.” if you take Bench 4, it literally flags about 50% of the tasks as broken, but then we ran it by every partner and made sure that if a task couldn’t pass this bar, it just wasn’t included in the benchmark. Stuff like that allows us to give people confidence in the quality of the benchmark without making all the tasks public.
As for what’s next, we really hope we can get Version 1 on some major new model release cards when they come out, because we do think it’s the best benchmark for measuring frontier capabilities in the world today. At the same time, we’re definitely just getting started, and we plan to do future versions as models become increasingly capable and saturate the tasks in GDPX v1. There are more domains that get unlocked that can absorb lots of frontier tokens, be super valuable, and earn higher ROI.
We definitely want to release future versions with more data partners, always using the highest-quality data partners at any given point in time and picking the most important domains. Hopefully, we can have more partners, more tasks per domain, and run all the models at every reasoning level now that we have more clout and traction with the benchmark.
Awesome, man. Okay, I think you answered the final question about where we go from here. Andrew, any closing thoughts from you about the future, next steps, or your initial reaction to this stuff? Pretty fun week, huh?
Yeah, I definitely agree with Max. I want this to be the benchmark everyone thinks about when they think about which model has which capabilities and where they stand. This is only the first iteration. We’re going to keep working on it and hopefully get more data partners. If anyone listening to this is interested in evals and maintaining this, please reach out. We are hiring on this team, of course—just email gdpxian.com.
Awesome. Nice way to end off. Thanks, guys, for joining. Appreciate it.
Thank you, Jordan.
Thank you, Jordan.