Ep. 042 - GDPx: The Benchmark for Frontier AI | Andrew Megalaa, Jordan Nanos, Max Kan
Jordan NanosMax KanAndrew Megalaa
- GDPX targets the domains its creators expect to capture the first $1 trillion of AI revenue. Max Kan’s thesis is that this revenue will be “highly concentrated within a few domains” that keep earning high returns from smarter models, including chip design, biology and cybersecurity. Most routine tasks, he expects, will not require frontier intelligence heading into 2027; GDPX therefore measures a deliberately selective slice of economic activity.
- The benchmark’s central distinction is between genuine difficulty and defective evaluation. Max calls it “difficult versus unfair”: a correct solution can fail because the verifier recognizes only another valid approach. SemiAnalysis sourced tasks from 14 specialized providers, rejected more providers than it admitted, and retained most tasks privately to reduce benchmark optimization.
- Materials science produced the clearest surprise about capability relative to cost. Astra beat a published state-of-the-art reference on one subtask, while contributor Brian discovered that some supposedly difficult workflows failed because he was “prompting it wrong” or providing insufficient context. Jordan Nanos highlights materials-science task costs reaching $40–$50, with some models solving tasks for about $10.
- Realistic finance and legal work remains substantially beyond reliable autonomous execution. The finance environment reconstructs a transaction involving more than $800 million for 62% of a snack company, with roughly 4,000 files. Finance was essentially zero under binary grading apart from an isolated Carbon success; Opus’s partial score was only 50-something percent, while the best legal partial scores were in the 30s or 40s.
- Safety restrictions materially change which model is useful, even when underlying capability may be stronger. Andrew says it is fair to penalize overly strict safety classifiers; Astra scored 0% on cybersecurity because it refused the tasks. Muse Wonder and Muse Wonderful emerged as “top tier” cyber options, reinforcing Jordan’s experience, although Wonderful was not available through the public API.
- Carbon improves Google’s competitive position without closing the leading labs’ gap in Jordan’s preferred comparison. Excluding biology, cybersecurity and alignment, and using partial grading, Gemini 4 Carbon still trails Astra and Opus. Jordan sees Carbon as an inflection for Google, calling it the “third place player”; he also notes that new OpenAI and Anthropic flagships are expected soon.
- The employment debate remains unresolved, with sharper pressure on junior work than a simple productivity story implies. Max sees human judgment making good engineers more valuable today, while allowing that far fewer supervisors might eventually be needed. Andrew challenges the assumption that displaced workers will become more ambitious: many people simply want to “put their hours in and go home.”
1. A difficult benchmark must first be a fair benchmark
Max’s core distinction is “difficult versus unfair”: a model can correctly fix a coding bug using module B, while a verifier accepts only module A. The failure then reflects the grader’s mismatch with the prompt. He argues that widely used benchmarks contain many such problems; identifying them requires examining rollouts and judging why the model failed, rather than simply inspecting the task description. Andrew adds that models may solve tasks in unexpected ways, so the team reviewed transcripts and sometimes used AI-assisted probing to assess whether a result was correct and fair.
SemiAnalysis did not assume it could invent 200 excellent tasks itself. It recruited 14 specialized data providers, handled QA centrally, and rejected more providers than it ultimately included. Accepted partners still underwent substantial iteration. Max’s point is that making tasks simultaneously realistic, fair and sufficiently difficult requires domain expertise as well as repeated examination of actual model behavior.
The benchmark supplies a repeatable way to assess releases the team cannot personally use extensively. Its final ranking was genuinely unfamiliar: tasks were finalized Thursday for a Sunday launch, and the complete leaderboard arrived only three to four hours before publication. Execution plus grading could easily take 24 hours. Carbon was among the models the team had scarcely encountered before evaluating it; they had access on Thursday night and were trying it for the first time.
2. Frontier revenue depends on work that keeps rewarding intelligence
Max’s guiding thesis is that “the first trillion dollars of AI revenue” will concentrate in domains capable of absorbing increasingly intelligent models at high ROI. He agrees that most tasks will not need frontier intelligence, and expects that to be especially true heading into 2027. GDPX therefore targets the frontier providers’ prospective revenue engines rather than a representative cross-section of all knowledge work.
The dashboard lists recursive self-improvement, chip design, cybersecurity, biology, finance, legal, quant trading, hardware design, materials science, alignment and software engineering. Selection emphasized verifiability and capacity to absorb intelligence. Alignment also reflects a broader evaluation priority: the team considers it essential for future models, with Fleet’s expertise making its inclusion practical.
The team’s account of materials scientist Brian captures a change of mind. He initially did not expect Astra to be so good at some of the workflows he proposed. Once he made tasks, he realized he had been “prompting it wrong” or supplying insufficient context, and needed a harder task set. Max says Brian watched Astra do work that would place a human materials scientist above the median.
Max says the usual human-baseline comparison becomes difficult when Astra beats a published state-of-the-art reference on a materials-science subtask, and some chip-design tasks would take teams at companies. Jordan supplies the economic contrast: the largest materials-science tasks cost $40–$50 each, and some models solved tasks for around $10.
3. Safety filters and specialist strengths reshape the leaderboard
The headline ranking includes biology, cybersecurity and alignment, although safety classifiers sometimes suppress performance. Astra’s cybersecurity score was 0% because it refused the task. Andrew says it is fair to penalize overly strict safety classifiers to some degree.
Jordan independently found Muse Wonder strong on cyber; Muse Wonderful’s result reinforced that experience. The team obtained Wonderful despite its absence from the public API. “You guys gave away all my alpha there,” Jordan jokes. He emphasizes filtering results around the work and access a user actually has. Andrew says the team had only the bottom or perhaps second-to-worst tier of Anthropic access, which still triggered restrictions on many tasks.
Andrew says he was glad his subjective experience with Opus 5.5 was validated, while acknowledging that Astra was exceptionally strong in materials science and some engineering work. Different labs have different strengths. His practical takeaway is to use whichever model performs best for the work at hand, rather than assume one model wins everywhere.
Jordan’s preferred apples-to-apples comparison excludes biology, cybersecurity and alignment, and uses partial grading for finance and legal. Under that comparison, Gemini 4 Carbon remains behind Astra and Opus. Jordan calls Carbon an inflection for Google and says it has reestablished itself as the “third-place player,” while also describing Google as at the top of the Tier 2 cohort. He says the gap with OpenAI and Anthropic remains meaningful and notes their flagship releases are expected soon. Max adds that the comparison is between public models; OpenAI and Anthropic may not be releasing their best models.
4. A real acquisition exposes finance’s context problem
Max contrasts asking a banker to invent a realistic task in two hours with reconstructing an actual transaction. GDPX’s finance partner used a private-equity acquisition of a family-owned snack company: a little over $800 million bought 62%. The environment contained all 4,000 real files. For Max, this is “categorically different” from asking an expert to create a synthetic approximation of their work.
The files span the private-equity firm, investment bank, accounting firm and seller’s data rooms, preserving their actual organization and disorder. Tasks came from work people really performed, including instructions in emails from accounting partners to associates. Jordan particularly values the ability to replay the data room as it existed when a task was assigned, rather than give every task access to the final document collection.
One failure involved an email instructing someone to adjust figures to Q4. Missing that message meant producing incorrect numbers despite otherwise plausible work. Jordan says he checked with finance friends, who told him an associate would absolutely be expected to catch it. In Jordan’s words, the same omission would be “unacceptable behavior in a real banking environment.”
5. Partial credit shows progress without making a workbook usable
Finance tasks receive binary final grades, although the workbook’s components are scored individually. Max’s rationale is operational: a quality-of-earnings workbook with half its numbers wrong is not a satisfactory deliverable. A supervisor will not say, “great job, little Jimmy,” and finish it themselves. Partial scoring describes competence; the final verdict reflects whether the assigned work was completed successfully.
Binary finance results were essentially zero, apart from Carbon, which got one task right. Jordan says Opus’s partial score was in the 50-something-percent range. Legal remained all zeros under binary grading; Jordan says the best partial score was in the 30s or 40s. These gaps matter because they indicate substantial missing capability rather than one marginal requirement blocking otherwise complete work.
Max accepts the criticism that binary grading can disguise easy or unfair tasks: a 25% binary score alongside 97% partial might mean the remaining 3% is an esoteric requirement the model should have been expected to meet. Jordan adds that a large gap might instead indicate an unfair or broken task. GDPX uses continuous grading in most domains. Chip-design PPA optimization, for example, can score progress between a naive baseline at zero and a theoretical maximum at one.
Finance’s prospective token spending rests on margins that can support substantial costs; Max says it is already a major spending domain in 2026 and expects heavy spending in 2027. Legal adds a “PvP” mechanism: if better AI could improve litigation outcomes or contract negotiations, an opponent using it could create pressure to buy stronger intelligence too.
6. Better models strengthen some workers while threatening junior roles
Andrew once believed humans plus AI would remain much better and that he would keep writing code manually. He describes Sonnet 3.7, Sonnet 4, Sonnet 5, and especially Opus 4.5 and 4.6 changing his view. He still thinks humans plus AI outperform AI alone because people contribute “little pieces of entropy” the models do not possess themselves.
Max separates benchmark results from a job-loss forecast: GDPX deliberately omits vulnerable work such as customer service. Good engineers currently add “taste and supervision and judgment,” but maybe in two years one supervisor can replace most of a company’s engineers; maybe not. Intern assignments are already harder to justify when models handle well-specified problems and managing an intern adds overhead.
Jordan argues that capability can expand demand: cybersecurity needs more expertise, and materials discovery can generate additional work. Max counters that a “cracked” materials scientist might nevertheless automate a PhD assistant’s job. An unidentified speaker in the transcript offers the optimistic possibility that a former grunt worker could use “infinite PhD grunt workers on tap” to pursue more ambitious science.
Andrew explicitly disagrees with that optimistic view as a general expectation: many Americans, he argues, want to work their hours and go home rather than continually elevate their careers. Max suggests AI may force people to care more because otherwise they will not provide value, but the outcome remains unresolved. The disagreement concerns human adaptation, not whether the tools are improving.
7. Keeping tasks private protects the benchmark’s purpose
Jordan describes the disclosure trade-off: publishing tasks lets outside experts inspect quality, but also makes hill-climbing easier. A benchmark can stop measuring capability and start measuring which laboratory bought the most similarly shaped training data. GDPX’s first priority is preserving a measure of “real model capabilities,” rather than maximizing public access to every task.
The compromise is one released task and rollout per partner, supported by detailed methodology. Evidence includes the real finance deal room and cybersecurity partners turning their own zero-days into tasks. Max says the team ran its own agent for QA; he cites Bench 4 flagging about 50% of its tasks as broken, and says GDPX tasks were then run by every partner, with tasks that failed the bar excluded. The team seeks credibility through concrete construction details and sample behavior while keeping most evaluation material private.
The next ambition is GDPX v1 appearing on major model-release cards, followed by versions that respond to task saturation and newly valuable domains. Max and Andrew want more strong partners, more tasks per domain and evaluations across every reasoning level. The intended benchmark evolves with the work frontier models can perform, rather than freezing today’s task mix indefinitely.
Full transcript
GDPX taking the world by storm.
Something like that.
1. Launch Week Chaos
Okay, nice job, guys. Your benchmark matches your—what did you say?
Our subjective impression of the models.
Yes. You’re pleasantly surprised that the benchmark matches your subjective impression of the models.
Yeah. To be clear, we had no idea what the final leaderboard was going to look like until about 2 hours before publishing—maybe 4 hours. We had all the partners calibrate difficulty on 4 models. They ran Astra, Opus, 3.8 Flash, and some Argon, because we were working on the benchmark before Argon came out, and then also GLM-5.3.
We only had results for those 4 models until about 3 or 4 hours before the benchmark launched. Andrew switched to polyphasic sleep over the past week so he could build all the infrastructure. He set things up so Claude would wake him if something broke and he needed to provide input. Then he set up all the infrastructure and started running the rollouts. These are incredibly long rollouts; doing the task plus grading could easily take 24 hours.
We didn’t have all the final results until literally 3 or 4 hours before we published. When we saw his leaderboard, we were like, “Yo, I think you might have, dude.” Some of these models we hadn’t used or even seen out in the wild. We got access to Carbon on Thursday night or something; we hadn’t used it for anything. It was the first time we were trying it out, and it did well. It was surprising. Same with Muse Wonderful. It isn’t available in the API, but we got access to it, and it did great on Cyber.
Yeah. When did you fix the tasks? When did those get finalized?
Thursday night. We launched on Sunday, and the tasks were finalized Thursday night because we were doing a lot of back-and-forth iteration with our partners. Our QA would flag something and say, “Hey, take a look at this.” Then they’d say, “Oh, these two are false positives, but this one’s for real,” and we’d get updates.
It was really last-minute. I’m very happy we got it out in time. It was very hard, though. I’m also glad to be taking a break now. It was a crazy process.
I think the work starts now, man. I don’t know.
The work? Okay, I’m going to take a couple of chill days.
Yeah.
And then we can go through the process of, yeah, I guess, reaping the rewards of our effort—
Maintaining the dashboard as new models get released. Did you impose this deadline on yourself? What was the motivation for that?
It was mostly self-imposed. We were pretty confident that a new OpenAI and Anthropic flagship were probably both coming out this week, and you obviously can’t launch your benchmark on the same day as Fable 5.5.
Especially if your benchmark doesn’t have it.
Yeah. I think we were pretty confident that Monday wouldn’t be the release date for these models. Maybe it would be Tuesday or Thursday, but if there was any risk of a model coming out on Monday, we were like, “Fuck it, we’re just going to push Sunday instead.” There was some fear that Sunday night people wouldn’t be as online, so you wouldn’t get the virality component. But I think we were very happy with how the launch turned out.
2. Broken Benchmarks
Yeah, generally happy.
Take a step back and explain GDPX from scratch for people who maybe haven’t seen the article or are just hearing about it for the first time—
Or didn’t read it, like many people commenting about it.
Yeah. Maybe taking a step all the way back, I think it’s important to understand that a lot of the really popular benchmarks out there are kind of trash. This is largely because they’re made by people who don’t have expertise in making eval environments or training data in general. It’s very easy to make a task that seems good on the surface but is actually unfair in some way, making it a poor test of model capabilities.
A lot of people make their tasks harder by making them unfair rather than actually making the task difficult. It’s very hard to figure this out unless you run 10 rollouts, look at the transcripts, see why the model failed, and manually determine whether that was a legitimate failure.
3. Difficult vs. Unfair
Yeah. Can you explain the distinction between difficult and impossible here? You wrote about this in the article. It has to do with both the prompt and the grader.
Yeah. Maybe “impossible” isn’t quite the right word. It’s more like difficult versus unfair. A classic example of an unfair task would be a coding problem that asks a model to fix a bug in an open-source repository. An expert human software engineer might look at it and say, “There are 2 ways to fix this bug. You can use either module A or module B. I recognize that both are equally fair and valid. You should be able to pick whichever one you want.”
But when we’re making these benchmarks, you also need to include a verifier for each task so you can check whether the model succeeded. It’s totally possible that someone might make a verifier scoped only to the module A solution. Then an AI might rightfully pick module B instead, and the verifier would mark that as a failure. The AI would perform worse on the task, even though it technically solved the problem. That’s an example of the task being unfair: your grader isn’t faithful to the prompt.
It turns out that a lot of popular benchmarks, like Terminal-Bench 4, SWE-bench Pro, DeepSuite, and others, have tons of issues of this type. People have started noticing this over the past month or so. There have been a couple of big blog posts about it, because data companies have agents look at all the transcripts and prompts, and run the tasks as part of their own QA process. They’ll see that 60% of the Terminal-Bench 4 tasks are broken or something.
4. Building GDPx
The funny thing is, I feel like everybody knew it. All the labs definitely know it. Anyone who looks at the data knew the benchmarks were broken. It’s just that no one spends the time to write about it, because to most people in the industry, it’s not new information. It’s unsurprising.
5. Public vs. Private Tasks
Yeah. So benchmarks are bad. That’s one thing to know. We at SemiAnalysis knew we wanted to make a benchmark ourselves, because it’s kind of the only way you can consistently say interesting things about new model releases. The team isn’t going to spend meaningful time and effort trying out the new MiniAX model when it comes out to see if it’s actually good. No disrespect to MiniMax, but we’re just not going to do that. You really want an eval, and we’re not going to do—no one tested GPG 6.1 Terra either, right?
If you have a really high-quality eval that you can regularly run models on, it can go a long way toward tracking their performance. But at the same time, we recognized that a lot of public benchmarks are bad—not because people aren’t putting in effort. It’s also just really hard to make a task that’s fair, sufficiently difficult, interesting, and reflective of real work.
It would have been naive to think SemiAnalysis could come up with 200 really high-quality tasks ourselves. Our strategy was to partner with 14 specialized data companies that used a lot of their expertise to supply tasks for each domain. We were responsible for QAing all of them. We dropped more data providers than we included in the final benchmark because a lot of them failed QA. We were also heavily involved in task iteration with the providers who passed QA, as we mentioned at the start of the episode.
6. Frontier Domains
That was the philosophy and methodology for making the benchmark. In terms of what it covers, that’s another big question we needed to figure out, because you can pick any domain. AI is affecting everything these days. One thing we strongly believe on the Tech Economics team is that the first trillion dollars of AI revenue will be highly concentrated within a few domains that can absorb increasingly smarter frontier intelligence and earn a high ROI.
I think a lot of people have recently seen these token-budgeting trends, or heard people say, “You actually don’t need frontier intelligence for the majority of tasks,” and taken that as bearish for the AI trade. We totally agree that many tasks won’t need frontier intelligence. But we think there are a lot of domains, like chip design, biology, and cybersecurity, where we’re going to need increasingly smarter frontier models, and you’ll continue to get high ROI from Fable 6, Fable 7, Fable 8, whatever.
So we decided to pick domains for our benchmark, GDPX, that match those domains that can absorb frontier intelligence. That was our guiding philosophy.
7. The 11 Domains
Sweet. Okay. Maybe we can talk through the specific domains in a little more detail. For those who don’t know, there’s a website up, gdpx.semianalysis.com. I’ll share it on screen for a second. As we scroll through the domains, we have recursive self-improvement, or RSI; chip design; cybersecurity; biology; finance; legal; quant trading; hardware design; materials science; alignment; and software engineering.
I don’t think this is really a breakdown that a lot of people have on the tip of their tongue when they’re talking about the domains AI is coming to. Even just introducing these one by one would be interesting. Maybe you can start by saying how you arrived at these categories broadly, and then how you found the specific data vendors you went with in the categories you focused on.
Yeah. Arriving at the domains involved a lot of brainstorming and trying to pick domains that are verifiable and seem like they have the capacity to absorb a lot of intelligence. After we had our initial list, we talked to a bunch of people at the labs and people working on similar problems. We went through that process for a few weeks until we arrived at our final seven domains.
In terms of how we picked the partners, we’re fortunately pretty well connected in the data space, and we happen to know a lot of the environment companies and what each of their specialties are. We reached out to a bunch of people working in the various domains, got sample tasks from them, and QA’d them. We dropped the ones that were bad. In cases where we had more than 1 good person who wanted to work exclusively with us in a particular domain, we’d break ties based on things like our tastes, how interesting we thought the problems were, and how representative they were of their domain.
That was basically the whole process. Eventually, people started hearing about GDPX on their own and reaching out to us to be included in the eval, which was kind of interesting. It spread around the data space.
Yeah. Some of them were more opportunistic. Alignment is a good example where—
It’s not something you’d probably immediately reach for if you were trying to make a benchmark that measures agent capabilities. But I really do think it’s one of the most important things you need to evaluate future models on, so it deserves a place in any comprehensive benchmark.
At the same time, Fleet is one of the biggest, most well-respected data companies today. I have a few friends who work there, and when we were brainstorming domains we could potentially include, Fleet for alignment ended up making a ton of sense.
Yeah, makes sense. Maybe another way to think about it, when I was reading through some of these descriptions, is that these are clearly domains where people using frontier AI today are burning a ton of tokens.
8. Domain Experts
Absolutely. The domains are really interesting, though, because they vary so much. Some have very established data companies that have been selling data for a few years, while some, like materials science, are just one PhD who’s really good at materials science and delivered these insane tasks that are difficult even for Astra. I found that super cool, because they used papers they wrote, or papers written by people they know. Shout-out to them. Those are pretty cool evals.
9. Models vs. Experts
Yeah, my favorite part of the article is going through and reading about these pretty esoteric domains. It was so clear from the task details that the person who wrote them was a domain expert. They certainly knew more than me.
10. The Leaderboard
Mhm. And maybe you can talk a little bit about the reaction from the people you’ve worked with since the article came out, and how it seems like a lot of people were surprised by the results. I know Brian was specifically saying he was shocked that some of the tasks were solved by the—
Yeah, I think Brian initially had a brief moment where he didn’t expect Astra to be so good at his tasks when we first asked him for them. Some of the workflows he initially thought Astra wasn’t good at, once he sat down and made tasks, he realized that actually he was just prompting it wrong or not providing enough context, and it ended up solving them.
So he had to make another set that was actually difficult and frontier-level, because I think a lot of people don’t realize how good the models are. He even said this: materials scientists don’t know how good the models are, and they think the models can’t do certain things, but really they can.
Well, you should talk about the experience you had with software engineering at the start of the year and how that relates to Brian.
Yeah.
11. AI and Jobs
Yeah. I think I went through a similar thing. It was actually a year ago at this point. After I left my job to start an RL company with Max here, at first I still thought humans plus AI was just a lot better. I thought I couldn’t really take a software engineering job; we’d still be coding by hand, while AI would automate the easy things. But then Sonnet 3.7 was really good, Sonnet 4 came out, Sonnet 5, and then Opus 4.6. That Opus 4.6 was a moment. That sounds weird.
Opus 4.5 was the first one.
Opus 4.5 was really good, but Opus 4.6 was also really, really good.
Like, those 2 models were just—
12. Cyber and Safety
Yeah, I think if you give it a few more iterations, there’ll be no need for me to look at a lot of this code. It was a very clear trend, and I had to reconcile that myself because I really liked coding at the time. That was my thing. I was just good at coding and systems. But I had to figure out, “Oh, actually, I have to take on the higher-level things now. This is no longer a job where I’ll always be writing code by hand.”
I still think humans plus AI are better than AI alone, just because there are little pieces of entropy that you can provide that the models don’t have themselves. And so—
Still valuable.
Yeah. But funny enough, the reason why I wanted Andrew to talk about that is I think Brian had a very similar moment for materials science when he made the task for the benchmark. He was like, “Holy shit. If any human materials scientist could do what Astra just did, they’re probably a top— they’re definitely better than the median materials scientist already.” He watched Astra do this, and I think we’re probably going to have similar moments in a lot of other domains, like chip design, materials science, and the other GDPX domains, in the imminent future.
That’s something we also hope to see in actual benchmark results as we’re able to evaluate Fable 5.5, Astra 6.1, 6.2, or whatever. Someone asked whether GDPval had human baselines for how long tasks take, to put things in context. That was really funny, because I responded that it doesn’t even make sense to have human baselines anymore. In the materials science tasks, Astra solved a subtask better than a published state-of-the-art reference. When models are essentially past expert level, how do you even set a human baseline? Some of these chip design tasks are so long-horizon and would take teams at companies to do. There’s no human baseline there. How do you even put a number on it?
It’s a lot of money. Whatever the hourly equivalent is in human work hours, it’s a lot of money. I just threw up on screen the cost per task for the materials science work. The biggest ones were $40 or $50 per task, and many of the models are solving some of these tasks for $10, which I think is a pretty unbelievable statement.
13. Model Behavior
Maybe you can talk about reading some of the rollouts in more detail.
What were some of the pieces of model behavior that surprised you the most?
The models find very creative ways to solve the tasks. This goes back to the whole unfair-verifier thing: people don’t expect the way models solve tasks because models aren’t human. They solve things in very weird ways. Maybe they use Module B instead of Module A, for example, but sometimes they just have completely different taste in how things are done. Sometimes it’s correct and sometimes it’s not, which is why we have to go through these transcripts. It can be hard to go through them because we’re not human experts, but luckily you can benefit a lot from chatting with AI about certain rollouts to figure out whether the stuff is correct. The models are so smart on their own that, with human probing, you can figure out whether something is fair or not.
Yeah.
It’s a very interesting experience, for sure.
14. Frontier vs. Day-to-Day
Maybe we can talk about the results. I’m realizing I haven’t even shared the flagship leaderboard screenshot at the top of the article yet. You guys can talk about what you learned about the models themselves. We’ve been going back and forth, trying to explain the difference between the flagship leaderboard across all domains and the next one, where you filter out biology, cybersecurity, and alignment. In those domains, safety classifiers can prevent models from demonstrating performance that’s in the weights somewhere.
Yeah, I think that’s a really good point worth emphasizing because we went back and forth on which screenshot should be the headline result: the one where you include all the domains or the one where you exclude cybersecurity, biology, and alignment. Those were the 3 domains where safety classifiers sometimes kicked in and sandbagged the score for some models. If you look at Astra on cyber, for example, the score is 0% because it just refused to do the cyber task. Ultimately, we decided the one including all domains is better, but I think it’s fair to penalize overly strict safety classifiers to some degree.
I know you’ve experienced this yourself, Jordan, doing your day-to-day work at SemiAnalysis and then getting hit with a cyber safety classifier from Opus Rational. You’re like, “Well, I guess I’ve got to use GLM-5.3 now or whatever.”
No, Jordan had a very interesting insight. He figured out that Muse Wonder is a top-tier cyber model independently, and he was right. Muse Wonder and Muse Wonderful were just top-tier for what you can get out of the box for a cyber model.
Yeah.
That was a very surprising result for us because Jordan already knew, but we didn’t have the Muse Wonderful number until 2 hours before launch. That was pretty cool. We were like, “Wow, Muse Wonderful just crushes it.”
Yeah, you guys gave away all my alpha there. Using the models yourself for something where you think you’re pushing the frontier absolutely gives you insight into which models are good and which models are bad. But you have to actually have that work to do, and I had that work to do in one of the 11 domains we’re talking about here.
Yeah, I could figure that out. I do think probably the biggest benefit of this benchmark is the ability to filter the results as you see fit and interpret them as relevant to you. People with Daybreak access or CVP access from Anthropic—which we sometimes have—
Well, there are 3 or 4 tiers, and we only had the bottom tier, or maybe the second-to-worst tier. That tier is more or less useless because you still get flagged doing basically everything. I think they both have very confusing access programs for the models today, which hopefully they can improve.
Yeah, yeah.
The crazy thing is, Muse Wonderful does that well in cyber. I wonder what red-team access on Astra would do.
No one can use it.
Yeah.
Unless you work there, I guess.
We’ll get there.
But maybe the better question is: can you emphasize again that this benchmark is meant to measure model performance at the frontier rather than on day-to-day tasks?
Yeah, for sure. Our core thesis is that day-to-day tasks just don’t actually matter, right?
They’re more or less solved.
I think the majority of them are more or less solved. Heading into 2027, you’re not going to need frontier models for most of these. We think they’re poor tasks to use to measure frontier model capabilities, and we also think they probably won’t make up the majority of revenue for OpenAI, Anthropic, and the other frontier-model companies. The things we’re most interested in tracking with this benchmark are where frontier-model capabilities are going, and whether you can do any TAM or revenue analysis based on that. So you very explicitly focus on the frontier domains.
Yeah. When we say day-to-day tasks, that doesn’t even exclude some of the domains people want to use these models for. It was surprising to see how bad the models are in finance and legal, given how much people talk about automating those fields. When we looked at some of the tasks we got from Living Context and Chakra—
We went through those rollouts ourselves because we were confused. I actually thought the models were better at these.
Yeah, Andrew is a tech-bro supremacist. He was like, “Dude, the models have definitely saturated finance already.” That was his prior coming into this.
Yeah, and we actually had some of our finance friends look at these tasks. When we were choosing partners for these, we had different—
Rollouts and tasks, and some of them were like, “Yeah, these tasks are pretty bad.”
Yeah, there were some tasks from potential finance partners where we were like, “Whoa, this isn’t realistic of actual finance work.” The only reason it’s sufficiently difficult for Opus and Astra is that it lacks realism and is unfair in these ways. After you correct the unfairness, it’s like, “Oh, look, Opus is saturated.”
That would make you think those tasks are solved, but then we got the really high-quality tasks, and our finance friends looked at them and said, “Yeah, this is totally realistic of something I would have done in banking and PE.” The models just failed.
15. The Finance Task
Yeah. Actually, I want to talk more about the finance task in particular because it’s probably the most understandable domain to the average listener. I think it would be useful in giving people a sense of why GDPX is such a high-quality benchmark. Historically, for something like GDPval from OpenAI, they would go to a company like Mercor, which has this vast network of finance experts, and ask a finance professional to sit down for 2 hours and come up with a task representative of something they’d actually do in their job, then provide the gold solution. That’s the task they’d give your model. Obviously, there are a ton of biases when you tell someone, “Make a task for my benchmark and please try your best to make it realistic.”
What our finance partner did was take a real private-equity transaction where a private-equity firm bought a family-owned snack company. I think they paid a little over $800 million for 62% of the company. They had this actual transaction, with all 4,000 files corresponding to it, separated into the deal rooms for the private-equity firm, the investment bank, the accounting firm, and the snack company’s own VDR, because they were selling themselves.
And it was organized—or disorganized—exactly how it would be.
Yeah, it was the real files for this real transaction.
Anybody who’s worked in a VDR—a data room for an M&A transaction—has seen exactly this.
They had the real files, and obviously they de-anonymized them. Along with the files, they had all the real tasks that analysts, associates, and other people working on the deal had to do. They essentially converted this actual deal room into an environment where they could evaluate the models.
And then they found the emails where an accounting partner told an associate what to do as part of the task, and had the model do it. I think this is categorically different in terms of realism, interestingness, and quality from just taking some random banker off the street and asking them to come up with a task.
Okay. One more thing about this, Max, is the life cycle. I thought this was fascinating: they could effectively replay what the data room looked like when that task was assigned, right? So, like—
You’re going to add these 1,000 files into the data room over time, but not every task is assigned at the same time in the data room.
I thought that was fascinating.
Yeah, obviously super fascinating that this is—
—the depth at which people can go to make this task realistic. But some of the failure modes are so esoteric when I looked at them. I had to ask my finance friends, and they told me, “Yeah, you would totally be expected to have done this.” It might be one email from someone saying, “No, actually adjust these numbers to Q4,” or something. If the model didn’t find that email, it would fail because it had to find it to evaluate all the context relevant to the task. Sometimes the models didn’t look hard enough, missed it, and sent out the wrong numbers.
And if the human associate failed, they would have gotten yelled at.
Yeah, apparently that’s exactly how it would have worked. If you were an analyst and didn’t read that one email, if you weren’t on top of it, you would have gotten yelled at for sure.
They would have said, “Yeah, this is unacceptable behavior in a real banking environment.”
16. Binary vs. Partial Grading
I know some people on the Tokenomics team who might be listening to this podcast and having some flashbacks. Can you talk about the grading in this case? I think a lot of people would be confused about the partial-credit versus binary-grading approach you have to apply to some of these things that don’t necessarily have a binary outcome.
Yeah. Our finance tasks use binary grading, so the model either gets a zero or a one. But there’s also a continuous scoring component, because the final deliverable might be an Excel workbook, like a quality-of-earnings book an accountant had to make for a private equity deal. That workbook can be decomposed into a bunch of different parts: different cells where the model has to find the right number. You can assign partial credit for each of those atomic criteria.
But we still use binary grading at the end because if you were a real accounting associate and gave your boss a workbook where half the numbers were right and half were wrong, they’re not going to say—
“Oh man, great job, little Jimmy. You got half of it right. I’ll take it from here.”
17. Google's Comeback
“What the fuck, man? I’m firing your ass right now.” So I think it makes sense to use binary grading in those cases, even though there’s a partial-scoring component. Jordan can show that finance is basically at 0% with binary grading, except for Carbon, which got one task right. But if you switch to partial grading, you get much more signal about what the models actually did.
A lot of people on Twitter were criticizing us for using binary grading, because other benchmarks sometimes use it to hide the fact that their tasks are quite easy. I won’t name any particular names, but there are popular benchmarks where you look at binary grading and it’s like, “Oh, 25% on Opus; this is probably pretty hard.” But then you switch to partial grading, and it’s like, “97% on Opus.” Then it’s like, okay, that last 3% is probably some esoteric requirement that there’s no reason you shouldn’t have expected.
Yeah. Or your task is just unfair and broken in some way. I agree: if our finance tasks were at 95% on Opus with partial grading, they’d probably be bad tasks. But you saw they were in the 50-something-percent range. It’s the same with legal. Legal is all zeros, but even with partial grading, the best model is in the 30s or 40s. Most of our other domains use continuous grading, because that makes sense for them. You might have a chip-design task where the model is asked to do some PPA optimization, with zero as a naive solution and one as the maximum theoretically achievable solution. You give the model partial credit based on how much it can optimize the chip design. We chose the grading strategy that makes the most sense for each domain; we didn’t use binary grading for everything just to artificially lower the score.
Yeah. I think this goes back to the comment about frontier intelligence versus day-to-day tasks. I don’t know that “frontier” in finance necessarily represents the same thing as it does in science, where it’s clear you’re pursuing a new discovery or something like that. But in finance, the frontier is being accurate.
Yeah. One thing to note about finance is that, at first glance, you might think it’s different from chip design or materials science. But we think it’s going to be one of the domains that spends very heavily on tokens in 2027. It’s already one of the main domains people spend tokens on in 2026, largely because the businesses’ margin structures support it. They can afford to spend a ton of money on tokens.
But there are also some PvP components—and this is even more true for legal. If AI could do a better job winning a lawsuit or redlining a contract, that has implications, especially if your opponent in the scenario isn’t using superintelligent AI to do those things. I think finance and legal are domains that will be able to absorb a lot of frontier intelligence, whereas real estate or government might not.
Makes sense. And just to expand on that a little, I think I know the backdrop. You’re talking specifically about why you chose to focus on these domains rather than taking a cross-section of all knowledge work in the economy and making tasks for every single thing people do, right?
Yeah.
I think what listeners are probably thinking about right now is—
—the concept of AI-driven job loss.
If people are using frontier intelligence and gaining productivity, making more money, or winning more things, how does running a benchmark like this inform your view on whether AI is going to cause job loss in the next year, 2 years, or even a 5-year time horizon?
I actually think they’re orthogonal in some sense, because we purposely choose not to cover a lot of the white-collar knowledge-work domains where people are probably most afraid of job loss, like customer service.
As for whether we’ll have massive job loss from AI, it’s really hard to say. I think a lot of the doomers have probably been pleasantly surprised by how AI plus human co-working has evolved over the past year. Any software engineer who has their eyes on the ball will probably tell you that, given a well-specified problem, Opus is just a better software engineer than me, period. But at the same time, good software engineers are more valuable today than they’ve ever been.
Maybe we’re in a Goldilocks zone where a lot of humans can still provide the taste, supervision, and judgment that make them plus the model way more effective than the model alone. But maybe 2 years from now you’ll only need one dude at each company supervising all the software engineering, and basically everyone else will have been fired. It’s possible that’s the world we end up in, but it’s also possible it isn’t. We have to pay close attention to model capabilities to see. I’ll say it’s definitely raised the bar for the kind of person you’re looking for to work on things like this. When you’re looking at software engineering, for example—
I feel bad for the interns.
Yes. Essentially, you have to be really good to provide more value than Opus at this point as a software engineer.
Yeah. A lot of people viewed interns as people you give well-specified, well-defined problems you don’t want to do, and they go do them for you. But obviously now you can just tell Claude to do that, and all the additional overhead of managing an intern often becomes unbearable in those cases. I know one of the big quant trading firms had a post about this. They said their interns finished their projects in the first few weeks this year, compared to previous years.
Really?
Yeah.
Hey, hey, hey. At least interns know how to use AI. That’s good.
Yeah. I think the typical response is that you need to get more ambitious. It makes sense. Going back to cybersecurity, I’ve been doing some work recently. We saw this recently with the increase in CVEs. I posted this on October 1. It’s been a month, and I don’t have an update. There’s a lot more than just this at this point.
Um.
Dude, is the whole internet just broken now? What’s going to happen? What are you going to do, Jordan?
Well, I think it’s a—yeah, I can—
I can talk about that in a lot more detail. Let’s save that for another time. But let’s use it to make the point that I think there’s a lot more demand for cybersecurity experience right now, even when using the models.
Yeah, for sure.
And I think that’s one of the first domains, as a product of software engineering, that’s getting knocked down. But if you’re pushing the frontiers of materials science, for example, you’re probably not automating away the jobs of materials scientists. You’re probably starting to discover a whole bunch of new materials that need to go through the whole discovery phase.
Well, no. I think it is very possible that you’ve automated the job of your PhD candidate assistant if you’re a cracked materials scientist.
I think this is exactly what we’ve heard from—
Yeah. If they have no ambition to go beyond what they were doing today or last year—
Yeah.
—then there’s nothing useful for them to do. But if they have ambition to accomplish more with the tools that are available, then the person who would have been forced to be a PhD candidate grunt worker two years ago can now leverage their infinite PhD grunt workers on tap to elevate themselves and do much more interesting materials science. That’s great. I’m generally very optimistic about human agency and people being able to take their lives into their own hands and do whatever makes them happy. I’m bullish on people using these tools to uplevel themselves. We’ll have to see what actually happens.
It was interesting. I actually completely disagree with that.
Andrew’s a hater. I think most people—
Well, when you look at it, most people in America just want to go in, put their hours in, and go home.
They don’t care that much about their careers. Maybe this is what forces them to care, because otherwise you won’t be able to provide value. We’ll see.
18. Quick Takes
Yeah, interesting. Okay, let me go back and get some quick reactions to results that we kind of glossed over. Andrew, you love Opus 5.5. Were you super happy to see it come out on top on a lot of these leaderboard charts?
Justified. Where did you get that insight?
Obviously, I’m glad that my subjective experience was validated. Though in some domains I’m not particularly involved in, it was very clear that Astra was ridiculously strong on materials science and some of the engineering stuff. Maybe that’s just because we don’t have Fable 5.5 yet, but we’ll see. It’s good to see that different labs have different strengths, because there’s a spike here and there, so you can leverage whatever is best for what you’re actually doing.
Yeah, makes sense. How about Gemini 4 Carbon? Quick take there. We saw Sundar quote-tweet everything there. You guys are excited about that?
Yeah, that was sick.
Shout out, Sundar.
I mean, we’re excited about it, too. More frontier models are better for all of us. We definitely want better models from everybody. It only benefits the consumer as there’s more competition.
Yeah. I do, however, think the right apples-to-apples comparison for where the frontier labs currently sit is to exclude cyber, bio, and alignment, and look at that GDP score. Maybe even switch to partial scoring, too, because that gives you more signal on legal and finance, where no one’s smart enough to do binary grading. At least you get rewarded for binary grading. If you do that ranking, you see that Carbon is still kind of behind Astra and Opus. That being said, it’s certainly an inflection in the trend for where Google’s model releases were going. At one point, it felt like they were going to fall farther and farther behind even Meta, potentially xAI, and potentially some of the Chinese open-source guys. But they’ve really reestablished themselves as the third-place player with Carbon.
How much of this is a timing thing, then? Sorry to interrupt, but Fable 5.5—you guys are teasing that it might come out soon. Obviously, Opus and Sonnet are on the list.
Yeah. I think the gap between Anthropic and OpenAI versus Google today is likely still very big, because we’re going to have new flagship model releases from them quite soon. Muse Wonderful came out pretty recently, right? So I think you’re evaluating Google’s position versus Meta and xAI. I think there’s a pretty strong argument that they’re at the top of that Tier 2 cohort. But I definitely still believe there’s a pretty meaningful gap between OpenAI and Anthropic versus—
And, you know, OpenAI and Anthropic obviously aren’t releasing their best models. So it’s just a comparison between the public models. With Gemini and Meta, they’re releasing the best they have so far.
Anthropic could just be on Mythos 4 at this point. Who knows?
Yeah, true. We’re going to get Fable plus one, because I think Anthropic is still kind of rolling with Fable-one-level models, essentially—or Mythos 1, I guess it’s called Fable 5. Internally, they’re probably at least two generations above that, and then we’re finally going to get one generation above that, hopefully this week. So I think the gap between the actual best model each lab has, including internal models, is very wide across OpenAI, Anthropic, Google, Meta, and SpaceX.
Yeah. Well, maybe the biggest takeaway for me there is just how relentless the pace of model releases is. The trend is for it to get faster and faster as they improve post-training and the data mixes.
RSI, bro. It’s RSI.
Yeah. Can you talk about next steps? One thing we didn’t talk about at all is the concept of releasing one task and one rollout for each category—or, well, for each of the data vendors, so multiple from some of the categories. Can you talk about the approach you took there, with the mix of partially public and partially private, and how other benchmarks approach it by comparison?
The trade-off you have to make for any benchmark is that the more public you make it, the more experts can QA everything themselves and verify that your stuff is actually good. At the same time, if you make it public, it becomes way too easy to hill-climb on the benchmark, and often your benchmark no longer measures model capability. It becomes more a measure of which lab bought more data that looks just like the tasks in my benchmark. I’m not going to name any particular names, but I think a lot of public benchmarks have already fit into that category. It’s really just a measure of which lab spent the most effort buying the most data of that shape.
Our priority is that people can’t hill-climb the benchmark. We want it to be an accurate measure of real model capabilities. At the same time, we know that establishing credibility is important, so releasing the rollouts for one task per partner seemed like a good trade-off. People can get some visibility into each partner and how confident they can be that the tasks are good. At the same time, we go pretty in-depth into the actual methodology for each partner, explaining how our finance guys actually use this real deal room, or how our cyber guys found their own zero-days and used them to task.
19. What's Next
I think binding all those details together is another way we can instill confidence in the quality of the benchmark. Also, giving details about our QA process helps. We say, “We ran our own agent to QA it.” if you take Bench 4, it literally flags about 50% of the tasks as broken, but then we ran it by every partner and made sure that if a task couldn’t pass this bar, it just wasn’t included in the benchmark. Stuff like that allows us to give people confidence in the quality of the benchmark without making all the tasks public.
As for what’s next, we really hope we can get Version 1 on some major new model release cards when they come out, because we do think it’s the best benchmark for measuring frontier capabilities in the world today. At the same time, we’re definitely just getting started, and we plan to do future versions as models become increasingly capable and saturate the tasks in GDPX v1. There are more domains that get unlocked that can absorb lots of frontier tokens, be super valuable, and earn higher ROI.
We definitely want to release future versions with more data partners, always using the highest-quality data partners at any given point in time and picking the most important domains. Hopefully, we can have more partners, more tasks per domain, and run all the models at every reasoning level now that we have more clout and traction with the benchmark.
Awesome, man. Okay, I think you answered the final question about where we go from here. Andrew, any closing thoughts from you about the future, next steps, or your initial reaction to this stuff? Pretty fun week, huh?
Yeah, I definitely agree with Max. I want this to be the benchmark everyone thinks about when they think about which model has which capabilities and where they stand. This is only the first iteration. We’re going to keep working on it and hopefully get more data partners. If anyone listening to this is interested in evals and maintaining this, please reach out. We are hiring on this team, of course—just email gdpxian.com.
Awesome. Nice way to end off. Thanks, guys, for joining. Appreciate it.
Thank you, Jordan.
Thank you, Jordan.