Artificial Analysis：独立 LLM 分析机构——与 George Cameron 和 Micah Hill-Smith 对谈

This is kind of a full-circle moment for us because the first time Artificial Analysis got mentioned on a podcast was when you and Alessio talked about it on Latent Space.

Amazing, which was January 2024. I don't even remember doing that, but yeah. It was very influential to me.

I'm looking at the AI news for January 17 or January 16, 2024. I said, “This gem of an AI model and host comparison site was just launched.” Then I put in a few screenshots and said, “It's an independent third party. It clearly outlines the quality-versus-throughput trade-off.” It breaks things out by model and hosting provider.

I did give you [__] for missing Fireworks. How do you have a model benchmarking thing without Fireworks? You had Together and Perplexity. I think we just started chatting there. Welcome, George and Micah, to Latent Space. I've been following your progress. Congrats on an amazing year. You guys have really come together to be the presumptive new Gartner of AI.

Okay, how do I pay you?

Let's get right into that. How do you make money?

We're very happy to talk about that. It's been a big journey over the last couple of years. Artificial Analysis is going to be 2 years old in January 2026, which is pretty soon now.

We first ran the website for free, obviously, and gave away a ton of data to help developers and companies navigate AI and make decisions about models, providers, and technologies across the AI stack for building things. We're very committed to doing that and intend to keep doing that. Along the way, we've built a business that's working out pretty sustainably. We've got just over 20 people now and 2 main customer groups.

We want to be who enterprises look to for data and insights on AI. We want to help them with their decisions about models and technologies for building things. On the other side, we do private benchmarking for companies throughout the AI stack that build AI stuff.

No one pays to be on the website. We've been very clear about that from the start because there's no use doing what we do unless it's independent AI benchmarking. It turns out a bunch of our stuff can be pretty useful to companies building AI stuff.

Is it like I'm a Fortune 500 company, I need advisors on objective analysis, and I call you guys? You pull up a custom report for me, come into my office, and give me a workshop? What kind of engagement is that?

We have a benchmarking insights subscription, which looks like standardized reports that cover key topics or key challenges enterprises face when looking to understand AI and choose between all the technologies.

For instance, one of the reports is a model deployment report. How do you think about choosing between serverless inference, managed deployment solutions, or leasing chips and running inference yourself? That's an example of the kind of decision that big enterprises face, and it's hard to reason through. This AI stuff is really new to everybody, so we try to help companies navigate that with our reports and insights subscription.

We also do custom private benchmarking. That's very different from the public benchmarking that we publicize, and there's no commercial model around that. For private benchmarking, we'll sometimes create and run benchmarks to specifications that enterprises want. We'll also do that sometimes for AI companies that have built things, helping them understand what they've built with private benchmarking through the expertise we've developed by trying to support everybody publicly with our public benchmarks.

Yeah. Let's talk about the tech stack behind that. But first, I'm going to rewind all the way to when you guys started this project. Were you all the way in Sydney?

Sydney, Australia for me.

George was in San Francisco, but he's Australian. He moved to Uruguay.

I remember I had the Zoom call with you. What was the impetus for starting Artificial Analysis in the first place? You started with the public benchmarks, so let's start there and then go to the private stuff.

Why don't we even go back a little bit to why we thought it was needed?

The story kind of begins in 2022 or 2023. Both George and I have been into AI stuff for quite a while.

In 2023 specifically, I was trying to build a legal AI research assistant. It actually worked pretty well for its era, I would say. But I was finding that the more you go into building something using LLMs, the more each bit of what you're doing ends up being a benchmarking problem.

I had this multistage algorithm trying to figure out what the minimum viable model for each bit was, trying to optimize every part of it. As you build that out, you're trying to think about accuracy, a bunch of other metrics, performance, and cost.

Mostly, no one was doing anything to independently evaluate all the models, and certainly not looking at the trade-offs for speed and cost. We basically set out to build a thing that developers could look at to see the trade-offs between all of those things, measured independently across all the models and providers.

Honestly, it was probably meant to be a side project when we first started doing it. We didn't get together and say, “Hey, we're going to stop working on this stuff, and this is going to be our main thing.”

When I first called you, I think you hadn't decided on starting a company yet.

That's actually true. I don't even think we'd paused. George and I would have dropped it and quit working on my own legal AI thing. It was genuinely a side project.

We built it because we needed it as people building in the space and thought, “Other people might find it useful, too.” So we bought a domain and linked it to the Vercel deployment that we had, and tweeted about it.

Very quickly, it started getting attention. Thank you, Swyx, for doing an initial retweet and spotlighting this project that we released.

It was useful to others, but very quickly, it became more useful as the number of models released accelerated. We had Mixture of Experts 7B, and it was a key one.

Yeah, an open-source model that really changed the landscape and opened up people's eyes to other serverless inference providers, as well as to thinking about speed and cost. It became more useful quite quickly.

What I love about talking to people like you who sit across the ecosystem is that I have theories about what people want, but you have data. That's obviously more relevant.

I want to stay on the origin story a little bit more. When you started out, I would say the status quo at the time was that every paper would come out and report its numbers versus competitor numbers, and that's basically it.

I did the legwork. I think everyone has some version of an Excel sheet or Google Sheet where you just copy and paste the numbers from every paper and post it up there. Sometimes they don't line up because they're independently run, so your numbers are going to look better, or your reproductions of other people's numbers are going to look worse because you don't prompt their models correctly, or whatever the excuse is.

I think Stanford HELM, Percy Liang's project, would also have some of these numbers. I don't know if there's any other source that you can cite. If I were to start Artificial Analysis at the same time you guys started, I would have used EleutherAI's evaluation framework, the LM Evaluation Harness.

That was some cool stuff. At the end of the day, running these evaluations is simple. If it's a basic Q&A evaluation, all you're doing is asking a list of questions and checking whether the answers are right, which shouldn't be that crazy.

But it turns out there are an enormous number of things that you've got to control for. Back when we started the website, one of the reasons we realized that we had to run the evaluations ourselves and couldn't just take results from the labs was that they would all prompt the models differently.

When you're competing over a few points, you can put the answer into the model.

For example, back when Google had Gemini 1.0 Ultra and needed a number that was better than GPT-4, it constructed, I think, never-published chain-of-thought examples—32 of them in every topic in MMLU—to run it and get the score.

There are so many things that you need to control for. We were pretty sure that we needed to run the evaluations ourselves and run them in the same way across all the models. We also said from the start that you couldn't look at those in isolation. You needed to look at them alongside the cost and performance data.

They never shipped Ultra, right?

That's one I never even heard of.

I mean, I'm sure it existed.

We were pretty sure that we needed to run the evaluations ourselves and just run them in the same way across all the models. We also said from the start that you couldn't look at those in isolation. You needed to look at them alongside the cost and performance stuff.

A couple of technical questions. I also thought about this, and I didn't do it because of the cost. Did you not worry about cost? Were you funded already? Clearly not, but you know.

We definitely weren't at the start. We were paying for it personally at the start. The numbers weren't nearly as bad a couple of years ago, so we certainly incurred some cost, but we were probably in the order of hundreds of dollars of spend across all the benchmarking that we were doing at the start.

It was kind of fine.

Yeah, yeah. These days, that’s gone up an enormous amount for a bunch of reasons that we can talk about. But it wasn’t that bad, because the number of models we were dealing with was hardly any, and the complexity of what we wanted to do to evaluate them was a lot less. We were just asking some Q&A-type questions.

One specific thing was that, for a lot of evals initially, we were just sampling an answer directly without letting the models think. We weren’t even doing chain-of-thought stuff initially, and that was the most useful way to get some results initially.

And so, for people who haven’t done this work, literally parsing the responses is a whole thing, right? The models can answer any way they feel fit. Sometimes they actually do have the right answer, but they return the wrong format, and they’ll get a zero for that unless you work it into your parser. That involves more work. There’s an open question about whether you should give it points for not following your instructions on the format.

It depends what you’re looking at, right? If you’re trying to see whether it can solve a particular type of reasoning problem, and you don’t want to test its ability to do answer formatting at the same time, then you might want to use an LLM-as-answer-extractor approach to make sure that you get the answer out no matter how it answered.

These days, it’s mostly less of a problem. If you instruct a model and give it examples of what the answer should look like, it can get the answers in your format, and then you can do a simple regex.

And then there are other questions. Sometimes, if you have a multiple-choice question, there’s a bias toward the first answer, so you have to randomize the responses. Once you dig into benchmarks, you’re like, “I don’t know how anyone believes the numbers on all these things,” because it’s such dark magic.

You’ve also got different degrees of variance in different benchmarks, right? If you run a 4-question multiple-choice eval on a modern reasoning model at the temperature suggested by the labs for their own models, the variance that you can see is pretty enormous if you only do a single run, especially with a small number of questions.

One of the things that we do is run an enormous number of all our evals when we’re developing new ones and doing upgrades to the Intelligence Index, so that we can dial in the right number of repeats. That way, we can get to the 95% confidence intervals that we’re comfortable with, so that we can get the Intelligence Index to at least as tight as plus or minus 1 at 95% confidence.

Yeah. Again, that just adds a straight multiple to the cost.

Yeah. So that’s one of the many reasons that cost has gone up a lot more than linearly over the last couple of years. We report a cost to run the Artificial Analysis Intelligence Index on our website, and currently that’s assuming 1 repeat in terms of how we report it, because we want to reflect a bit about the weighting of the index. But our cost is actually a lot higher than what we report there because of the repeats.

And probably this is true, but just checking: you don’t have any special deals with the labs? They don’t discount it? You just pay out of pocket or out of your customer funds?

There is a mix. The issue is that sometimes they may give you a special endpoint which—

Ah, 100%. Yeah, yeah, yeah. Exactly.

So we laser-focus everything we do on having the best independent metrics and making sure that no one can manipulate them in any way. There are quite a lot of processes we’ve developed over the last couple of years to make that true.

With the point you bring up right here, if we’re working with a lab and they’re giving us a private endpoint to evaluate a model, it’s totally possible that what’s sitting behind that black box is not the same as what they serve on a public endpoint. We’re very aware of that.

We have what we call a mystery shopper policy. We’re totally transparent with all the labs we work with about this: we will register accounts not on our own domain and run both intelligence evals and performance benchmarks without them being able to identify it. No one’s ever had a problem with that, because one thing that turns out to be quite a good factor in the industry is that they all want to believe that none of their competitors could manipulate what we’re doing either.

That’s true. I never thought about that. I’ve been in the database industry before, and there are a lot of shenanigans around benchmarking, so I’m going through the mental laundry list. Did I miss anything else in this category of shenanigans? It’s interesting shenanigans.

[laughter]

I mean, okay, the biggest one I’ll bring up is more of a conceptual one, actually, than direct shenanigans. It’s that the things that get measured become things that get targeted, because that’s what they’re trying to build, right?

Exactly.

So that doesn’t mean anything that we should really call shenanigans. I’m not talking about training on the test set. But if you know that you’re going to be graded on a particular thing, and you’re a researcher, there are a whole bunch of things that you can do to try to get better at that thing. Preferably, those things are going to be helpful for a wider range of how actual users want to use what you’re building, but they won’t necessarily do that.

For instance, the models are exceptional now at answering competition math problems.

[laughter]

There is some relevance of that type of reasoning, that type of work, to how we might use modern coding agents and things like that. But it’s clearly not one-for-one.

The thing that we have to be aware of is that once an eval becomes the thing that everyone’s looking at, the scores can get better on it without that being a reflection of the overall generalized intelligence of these models getting better. That has been true for the last couple of years, and it’ll be true for the next couple of years. There’s no silver bullet to defeat that, other than building new stuff to stay relevant and measure the capabilities that matter most to real users.

Yeah. And we’ll cover some of the new stuff that you guys are building as well, which is cool. You used to just run other people’s evals, but now you’re coming up with your own. I think that’s obviously a necessary path once you’re at the frontier and you’ve exhausted all the existing one-on-ones.

I think the next point in history that I have for you is AI Grant, which you guys decided to join and move here. What was it like? I think you’re in batch 2?

Batch 4.

I mean, it was great. Nat and Daniel are obviously great, and it’s a really cool group of companies that we were in AI Grant alongside. It was really great to get Nat and Daniel on board. Obviously, they’ve done a whole lot of great work in the space with a lot of leading companies, and we’re extremely aligned with the mission of what we were trying to do.

We’re not quite typical of a lot of the other AI startups that they’ve invested in, and they very much had the mission of what we wanted to do.

Did they say any advice that really affected you in some way, or was one of the events very impactful?

That’s an interesting question. I remember fondly a bunch of the speakers who came into the fireside chats at AI Grant, which is also a crazy list.

There was something about speaking to Nat and Daniel about the challenges of working for a startup, just working through the questions that don’t have clear answers, and how to work through those methodically. It was about working through the hard decisions, and they’ve been great mentors to us as we’ve built Artificial Analysis.

Another benefit for us was that other companies in the batch, and other companies in AI Grant, are pushing the capabilities of what AI can do at this time. Being in contact with them and making sure that Artificial Analysis is useful to them has been fantastic for supporting us in working out how we should build out Artificial Analysis to continue being useful to those building on AI.

I think to some extent I’m of mixed opinion on that one, because your target audience is not people in AI Grant who are obviously at the frontier.

Yeah, to some extent. But then—

[laughter]

A lot of what the AI Grant companies are doing is taking capabilities coming out of the labs and trying to push the limits of what they can do across the entire stack for building great applications. That actually makes some of them pretty archetypical power users of Artificial Analysis.

Some of them have the strongest opinions about what we’re doing well, what we’re not doing well, and what they all want to see next from us. When you’re building any kind of AI application now, chances are you’re using a whole bunch of different models. You’re maybe switching reasonably frequently between different models in different parts of your application to optimize what you’re able to do with them at an accuracy level and to get better speed and cost characteristics.

So, for many of them—not commercial customers of ours, because we don’t charge for that data on our website—they are absolutely some of our power users.

So let's talk about the evals as well. You guys started out from the general MMLU and GPQA stuff. What's next? How do you build up to the overall index? What was in V1, and how did you evolve it?

Okay. So, just for background, we're talking about the Artificial Analysis Intelligence Index, which is our synthesis metric. We currently pull it together from 10 different eval datasets to give what we're pretty confident is the best single number to look at for how smart the models are. Obviously, it doesn't tell the whole story. That's why we publish the whole website of charts to let you dive into every part of it and look at the trade-offs. But it's the best single number.

Right now, it includes a bunch of Q&A-type datasets that have been very important to the industry, including a couple that you just mentioned. It's also got a couple of agentic datasets, our own long-context reasoning dataset, and some other use-case-focused stuff.

As time goes on, the things we're most interested in—the things that are going to be important for capabilities becoming more important for AI and for what developers are caring about—are going to be, first, around agentic capabilities. Surprise, surprise: we're all loving our coding agents, and how the models are going to perform in that context, and then doing similar things for different types of work, are really important to us. Linking to use cases—to economically valuable use cases—is extremely important to us.

We've also got some of these things that the models still struggle with, like working really well over long contexts. Those aren't going to go away as specific capabilities and use cases, so we need to keep evaluating them.

Mhm. But I guess what I was driving at was the V1 versus the V2, and how that evolved over time—how you changed the index to get to where we are.

Yeah, I think that reflects the change in the industry well. That's an easy way to tell that story. V1 would be completely saturated right now by almost every model coming out, because doing things like writing the Python functions in HumanEval is now pretty trivial.

It's easy to forget how much progress has been made in the last 2 years. We obviously play the game constantly of today's version versus last week's version and the week before, along with all the small changes and the horse race between the current frontier and whoever has the best smaller-than-10B model right now, this week. That's very important to a lot of developers and people, especially in this particular city of San Francisco.

But when you zoom out to a couple of years ago, literally most of what we were doing to evaluate the models then would all be 100% solved by even pretty small models today. That's been one of the key things, by the way, that's driven down the cost of intelligence at every tier of intelligence. We can talk about that more in a bit.

So, V1, V2, V3: we made things harder, covered a wider range of use cases, and tried to get closer to things developers care about, as opposed to just the Q&A-type stuff that MMLU and GPQA represented.

Yeah, I don't know if you have anything to add there, or we could just go right into showing people the benchmark, clicking around, and asking some questions about it. Yeah, let's do it.

Okay. This will be a pretty good way to chat about a few of the new things we've launched recently. I think it's also a little bit about the direction we want to take it and how we want to push benchmarking.

Currently, the Intelligence Index and evals focus a lot on raw intelligence, but we want to diversify how we think about intelligence. We can talk about it, but the new evals that we've built and partnered on focus on topics like hallucination. We've got a lot of topics that I think aren't covered by the current eval set but should be. We want to bring them forth.

And so, for listeners, just to timestamp this: right now, number 1 is Gemini 3 Pro High, followed by Claude Opus 4.5, GPT-5.1 High—you don't have 5.2 yet—and Kimi K2 Thinking. Wow, still hanging in there. Those are the top 4. That will date this podcast quickly.

Yeah, yeah. I love it.

No, 100%. Look back this time next year and go, "How cute." Yep, totally.

A quick view of that is okay—there's a lot. I love this chart. This is such a favorite, right? Almost every talk that George and I give at conferences and stuff, we always put this one up first just to talk about situating where we are in this moment in history.

This is the visual version of what I was saying before about zooming out and remembering how much progress there's been. If we go back to just over a year ago, before o1 and before Claude 3.5 Sonnet, we didn't have reasoning models or coding agents as a thing, and the game was very different.

If we go back even a little bit before then, we're in the era where, when you look at this chart, OpenAI was untouchable for well over a year. You would remember that time period well: there were very open questions about whether or not AI was going to be competitive, full stop—whether or not OpenAI would just run away with it, whether we would have a few frontier labs, and no one else would really be able to do anything other than consume their APIs.

I am quite happy overall that the world we've ended up in is one where multi-model is absolutely and strictly more competitive every quarter over the last 2 years.

This year has been insane. You can see it. This chart, with everything added, is hard to read currently. There are so many dots on it, but I think it reflects a little bit of what we've felt—how crazy it's been.

Why 14 as the default? Is that a manual choice? You have ServiceNow in there, which is a less traditional name.

Yeah, it's the models that we're highlighting by default in our charts and our Intelligence Index.

Okay, so you just have a manually created list of stuff?

Yeah, that's right. But something that I don't actually think every artificial intelligence user knows is that you can customize our charts and choose which models are added. If we take off a few names, it gets a little easier to read.

Yeah, a little easier to read. I love that you can see the old one curve. Look at that. This is September 2024—and the DeepSeek jump.

Yeah, which got close to OpenAI's leadership.

So close. I think we remember that moment, around this time last year, actually.

Yeah, agreed.

Yeah, well, a couple of weeks. It was Boxing Day in New Zealand when DeepSeek V3 came out. We'd been tracking DeepSeek and a bunch of the other global players that were less known over the second half of 2024, and I'd run evals on the earlier ones and stuff.

I very distinctly remember Boxing Day in New Zealand. I was with family for Christmas and stuff, running the evals and getting back results, one by one, on DeepSeek V3. This was the first of their V3 architecture, the 61.1B MoE, and we were very impressed.

That was the moment where we were sure that DeepSeek was no longer just one of many players, but had jumped up to be a thing. The world really noticed when they followed that up with the RL work on top of V3 and R1 succeeding a few weeks later, but the groundwork was absolutely laid with an extremely strong base model, completely open weights, that we had as the best open-weights model on Boxing Day last year.

Boxing Day is the day after Christmas, for those who don't—

I'm from Singapore. A lot of us remember Boxing Day for a different reason: the tsunami that happened.

Of course. Yeah, that was a long time ago.

So, yeah, this is the rough picture of the Artificial Analysis Intelligence Index—AAII, or is it AAI? AI.

Once upon a time, we did call it Quality Index, and we would talk about quality, performance, and price, but we changed it to Intelligence Index. There have been a few naming changes. We added hardware benchmarking to the site, and we benchmark that at a system level. Then we changed our throughput metric to what we now call output speed.

Throughput makes sense at a system level.

Take me through more charts. What should people know? Obviously, the way you look at the site is probably different from how a beginner might look at it.

Yeah, that's fair. There's a lot of fun stuff to dive into. We have lots and lots of evals and stuff, but the interesting ones to talk about today would be a few of our recent things that probably not many people are familiar with yet.

The first one is our Omniscience Index. This one is a little bit different from most of the intelligence evals that we run. We built it specifically to look at the embedded knowledge in the models and to test hallucination by looking at what happens when the model doesn't know the answer or isn't able to get it correct: what's its probability of saying, "I don't know," or giving an incorrect answer?

The metric we use for omniscience goes from -100 to +100, because we're simply taking off a point if you give an incorrect answer to the question. We're pretty convinced that this is an example of where it makes the most sense to do that, because it's strictly more helpful to say, "I don't know," instead of giving a wrong answer to a factual knowledge question.

And one of our goals is to shift the incentive that evals create for models and the labs creating them to get higher scores. Almost every eval across all of AI up until this point has been graded by simple percentage correct as the main metric, the main thing that gets hyped. So, you should take a shot at everything. There’s no incentive to say, “I don’t know.” We did that for this one here.

I think there’s a general field of calibration as well, like the confidence in your answer versus the rightness of the answer.

Yeah, we completely agree. One reason that we didn’t do that, or put that into this index, is that we think the way to do that is not to ask the models how confident they are.

I don’t know. Maybe it might be, though. You give it a JSON field, say “confidence,” and maybe it spits out something. We’ve done a few evals podcasts over the years, and we did one with Clementine Fourrier of Hugging Face, who maintains a leaderboard. This was one of her top requests: some kind of hallucination or lack-of-confidence calibration metric. So, hey, this is one of them.

And like anything that we do, it’s not a perfect metric or the whole story of everything that you think about as hallucination, but it’s pretty useful and has some interesting results. One of the things that we saw in the hallucination rate is that Anthropic’s Claude models are at the very left-hand side here, with the lowest hallucination rates out of the models that we’ve evaluated on this AA Omniscience one.

That is an interesting fact. I think it probably correlates with a lot of the previously unmeasured vibes that people like about some of the Claude models. Is the data set public, or is there a held-out set?

There’s a held-out set for this one. We’ve published a public test set, but we’ve only published 10% of it. The reason is that, for this one specifically, it would be very easy to have data contamination because it is just factual knowledge questions. We will update it over time to also prevent that, but we’ve kept most of it held out so that we can keep it reliable for a long time.

It lets us do a bunch of really cool things, including breaking down quite granularly by topic. We’ve got some of that disclosed on the website publicly right now, and there’s lots more coming in terms of our ability to break out very specific topics.

I would be interested—let’s dwell a little bit on this hallucination one. I noticed that Haiku hallucinates less than Sonnet, which hallucinates less than Opus. Would that be the other way around in a normal capability environment? I don’t know. What do you make of that?

One interesting aspect is that we’ve found that there’s not really a—well, not a strong—correlation between intelligence and hallucination rate. The smarter the models are in a general sense isn’t correlated with their ability to, when they don’t know something, say that they don’t know.

It’s interesting that Gemini 3 Pro Preview was a big leap over Gemini 2.5 Flash and 2.5 Pro. If I add Pro quickly here, I bet Pro’s really good. Actually, no, I meant the GPT Pros.

Oh, yeah, because GPT Pros are rumored—we don’t know for a fact—that it’s like 8 runs and then with the LLM judge on top.

Yeah. So, we saw a big jump in accuracy. This is just the percentage that they get correct, and Gemini 3 Pro knew a lot more than the other models. There was a big jump in accuracy, but relatively no change between the Google Gemini models between releases in the hallucination rate.

Exactly.

It’s likely due to just a different post-training recipe between the Claude models. That’s driven this. You can partially blame us in how we define intelligence, having until now not defined hallucination as negative in the way that we think about intelligence. That’s what we’re changing. I know many smart people who are confidently incorrect.

No, look at that—that is very human. Very true. There’s a time and a place for that, I think.

Our view is that hallucination rate makes sense in this context, where it’s around knowledge, but in many cases people want the models to hallucinate, to have a go. Often that’s the case in coding or when you’re trying to generate newer ideas.

One eval that we added to Artificial Analysis is Critical Point, and it’s really hard physics problems. Is it sort of like a human eval type or something different? Or like a FrontierMath type?

It’s not dissimilar to FrontierMath. These are research questions that academics in the physics world would be able to answer, but models really struggle to answer. The top score here is now 9%. When the people who created this—like Minwei and, actually, Ofir Press, who was kind of behind SWE-bench—

What organization is this?

Oh, is this Princeton? It’s a range of academics from different academic institutions—really smart people. They talked about how they turn the models up in terms of the temperature, as high as they can, when they’re trying to explore new ideas in physics as a thought partner, just because they want the models to hallucinate.

Sometimes it’s a good feature.

Yeah, exactly. So, it’s not right in every situation, but I think it makes sense to test hallucination in scenarios where it makes sense.

The obvious question is: this is one of many. Every lab has a system card that shows some kind of hallucination number, and you’ve chosen not to endorse that and have made your own. In some sense, the rest of Artificial Analysis is public benchmarks that other people can independently rerun. You provide it as a service. Here, you have to fight the question, “Who are we to do this?” Your answer is that you have a lot of customers, but how do you converge the industry on one number that everyone actually agrees is the rate? You have your numbers, they have their numbers. Never the twain shall meet.

I think for hallucination specifically, there are a bunch of different things that you might reasonably care about and that you’d measure quite differently. We’ve called this the AA Omniscience hallucination rate, not trying to declare that it’s humanity’s last word on hallucination.

The biggest-picture answer to that, something that I actually wanted to mention just as George was explaining Critical Point as well, is that as we go forward, we are building evals internally. We’re partnering with academia and partnering with AI companies to build great evals. We have pretty strong views, in various ways for different parts of the AI stack, about where there are things that are not being measured well or things that developers care about that should be measured more and better, and we intend to be doing that.

We’re not obsessed with everything we do having to be done entirely within our own team. Critical Point is a cool example of where we were a launch partner for it, working with academia. We’ve got some partnerships coming up with a couple of leading companies. Those ones obviously require us to be careful with some of the independent stuff, but with the right disclosure, we’re completely comfortable with that.

A lot of the labs have released great data sets in the past that we’ve used to great success independently. So, between all of those techniques, we’re going to be releasing more stuff in the future.

Cool. Let’s cover the last couple, and then I want to talk about your trends analysis stuff.

Totally. Before that, I have one little factoid on AA Omniscience. If you go back up to accuracy on AA Omniscience, an interesting thing about this accuracy metric is that it tracks more closely than anything else that we measure the total parameter count of models.

Ooh. Makes a lot of sense intuitively, right?

Because this is a knowledge eval. This is the pure knowledge metric. We’re not looking at the index and the hallucination rate stuff that we think is much more about how the models are trained. This is just: what facts do they recall? It tracks parameter count extremely closely.

Okay, what’s the rumored size of GPT 3 Pro?

To be clear, it’s not confirmed by any official source—just rumors.

Rumors do fly around. I hear all sorts of numbers. I don’t know what to trust.

If you draw the line on AA Omniscience accuracy versus total parameters, we’ve got all the open-weights models. You can squint and see that likely the leading frontier models right now are quite a lot bigger than the 1 trillion parameters that the open-weights models cap out at and the ones that we’re looking at here.

There’s an interesting extra data point that Elon Musk revealed recently about xAI: Grok 3 and 4 are 3 trillion parameters, and Grok 5 is 6 trillion. But that’s not out yet.

Take those together and have a look. You might reasonably form a view that there’s a pretty good chance that Gemini 3 Pro is bigger than that, that it could be in the 5- to 10-trillion-parameter range. To be clear, I have absolutely no idea, but just based on this chart, that’s where you would land if you had a look at it.

Yeah, and to some extent I discourage people from guessing too much, because what does it really matter? As long as they can serve it at a sustainable cost, that’s about it.

Yeah, totally. They've also got different incentives in play compared to open-weights models that are thinking about supporting others in self-deployment. For the labs that are doing inference at scale, it's, I think, less about total parameters in many cases when you think about inference cost, and more about the number of active parameters. And so there's a bit of an incentive toward larger, sparser models. Agreed. Understood. Yeah.

Right. Obviously, if you're a developer or company using these things, it's not about exactly—as you say, it doesn't matter. You should be looking at all the different ways that we measure intelligence. You should be looking at our cost-to-run index number and the different ways of thinking about token efficiency and cost efficiency based on the prices, because that's all that matters. It's not as good for the content-creator rumor mill, where I can say, "Oh, GPT-4 is this small circle. Look at GPT-5—it's this big circle." And that used to be a thing for a while. Yeah.

But that is actually a very interesting one, right? Is it? Well, no—just purely that chances are the last couple of years haven't seen a dramatic scaling-up in the total size of these models. And so there's a lot of room to go up, probably, in total size of the models, especially with the upcoming hardware generations. Yes. So, taking off my shitposting face for a minute.

Yes, yes. At the same time, I do feel like, especially coming back from NeurIPS, people do feel like Ilya is probably right that the paradigm doesn't have many more orders of magnitude to scale up, and therefore we need to start exploring at least a different path. GDPval, I think, is only about a month or so old. I was also very positive when it first came out. I actually talked to Tejas, who was the lead researcher on that.

Oh, cool. And you have your own version. It's a fantastic data set. Maybe a little recap for people who are still out of it?

It's 44 tasks that, based on some kind of GDP cutoff, are meant to represent broad white-collar work that is not just coding. Each of the tasks has a whole bunch of detailed instructions and some input files for a lot of them. Within the 44, I would estimate there are around 225 subtasks at the level that we run through the agentic harness.

They're really interesting. I will say that it doesn't necessarily capture all the stuff that people do at work. No eval is perfect. There's always going to be more things to look at, largely because, in order to make the tasks well enough that you can run them, they need to have only a handful of input files and very specific instructions for that task.

I think the easiest way to think about them is that they're quite hard take-home exam tasks that you might do in an interview process.

For listeners, it is no longer just a long prompt. It is, "Here's a ZIP file with a spreadsheet or a PowerPoint deck or a PDF. Go nuts and answer this question."

OpenAI released a great data set, and they released a good paper that looks at performance across the different web chatbots on the data set. It's a great paper. I encourage people to read it.

What we've done is take that data set and turn it into an eval that can be run on any model. We created a reference agentic harness that can run the models on the data set, and then we developed an evaluator approach to compare outputs. That's AI-enabled, so it uses Gemini 3 Pro Preview to compare results, which we tested pretty comprehensively to ensure that it's aligned to human preferences.

One data point there is that, even as the evaluator, Gemini 3 Pro interestingly doesn't do that well in GDPval-AA.

The thing that you have to watch out for with LLM judges is self-preference. Models usually prefer their own output. In this case, there was not.

Totally. I think the way that we're thinking about the places where it makes sense to use an LLM-as-judge approach now is quite different from some of the early LLM-as-judge work a couple of years ago, because some of that—and MT-Bench was a great project—was about judging conversations and a lot of style-type stuff.

Here, the task that the grader and grading model are doing is quite different from the task of taking the test. When you're taking the test, you've got all of the agentic tools. You're working with the code interpreter, web search, and the file system through many, many turns to try to create the documents.

Then, on the other side, when we're grading it, we're running it through a pipeline to extract visual and text versions of the files and provide that to Gemini. We're providing the criteria for the task and getting it to pick which one more effectively meets the criteria of the task out of 2 potential outcomes.

It turns out that it's very good at getting that right match with human preference a lot of the time. I think it's got the raw intelligence, but that's combined with the correct representation of the outputs, the fact that the outputs were created with an agentic task that is quite different from the way the grading model works, and the fact that we're comparing against criteria—not just trying to ask the model, zero-shot, to pick which one is better.

Got it. Why is this an Elo and not a percentage like GDPval?

The outputs look like documents, and there are video outputs or audio outputs from some of the tasks.

The user has to make a video?

Yeah, for some of the tasks.

What task is that?

It's a marketing video.

Wow. What?

The model has to go find clips on the internet and try to put them together. The models are not that good at doing that one for now, to be clear. It's pretty hard to do that with the code interpreter. The computer-use setup doesn't work quite well enough, and so on.

There's no ground truth necessarily to compare against to work out a percentage correct. It's hard to come up with correct or incorrect there. So it's on a relative basis, and we use an Elo approach to compare outputs from each of the models between tasks.

You know what you should do? You should pay a contracted human to do the same task and then give it an Elo. Then you have human performance.

There is. I think what's helpful about the OpenAI GDPval one is that 50% is meant to be normal human performance.

And maybe a domain expert is higher than that. But if 50% was the bar for, "Well, if you've crossed 50, you are superhuman"—

We haven't granted the score in that exact way. I agree that it can be helpful, but we wanted to generalize this to a very large number of models. That's one of the reasons that presenting it as Elo is quite helpful. It allows us to add models, and it'll stay relevant for quite a long time.

I also think it can be tricky looking at these exact tasks compared to human performance, because the way that you would go about it as a human is quite different from how the models would go about it.

I also like that you included Llama 4 Maverick in there. Is that just one last—

No, no, no, no, no. It is the best model released by Meta, and so it makes it into the homepage default set. Still, for now.

Another inclusion that's quite interesting is that we also ran it across the latest versions of the web chatbots.

Oh, that's right. Sorry, I completely missed that.

Not at all. That has a checkered pattern.

So that is their harness, not yours?

Exactly. What's really interesting is that if you compare, for instance, Claude 4.5 Opus using the Claude web chatbot, it performs worse than the model in our agentic harness. In every case, the model performs better in our agentic harness than its web-chatbot counterpart—the harness that they created.

My backwards explanation for that would be that it's meant for consumer use cases, and here you're pushing it harder. The constraints that everyone has and the amount of freedom that you can give the model are different. Also, you have a cost goal.

We let the models work as long as they want, basically.

Do you copy and paste manually into the chatbot?

Yep. Yeah, that was how we got the chatbot reference. We're not going to be keeping those updated at quite the same scale as hundreds of models on the harness.

I don't know. Talk to Browserbase. They'll automate it for you, you know?

True. We should. I have thought about turning these chatbot versions into an API because they are legitimately different agents in themselves, right?

Yeah. And that's created a huge amount of velocity, right? The tools that are available have actually diverged, in my opinion, a fair bit across the major chatbot apps. The amount of data sources that you can connect them to has gone up a lot, meaning that your experience and the way you're using the model is more different than ever.

What tools and what data connections come to mind when you say what's interesting? What's notable work that you've worked on?

My favorite example is that, until very recently, I would argue that it was basically impossible to get an LLM to draft an email for me in any useful way, because most times that you're sending an email, you're not just writing something for the sake of writing it. Chances are, the context required is a whole bunch of historical emails.

Maybe it's notes that you've made. Maybe it's meeting notes. Maybe it's pulling something from wherever you keep your stuff. For me, it's Google Drive, OneDrive, and our Supabase databases if we need to do some analysis on some data or something. Preferably, the model can be plugged into all of those things and can go do some useful work based on it. The things that I find most impressive currently, and that I am somewhat surprised work really well in late 2025, are that I can have models use Supabase MCP to query—read-only, of course—and run a whole bunch of SQL queries to do pretty significant data analysis, make charts and stuff, and read my Gmail and my Notion.

Okay, you actually use that. That's good. Is that a Claude thing?

To various degrees, it's supported on both ChatGPT and Claude right now. I would say that the stuff barely works, in fairness, right now.

Because people are actually going to try this after they hear it. An email from Micah—odds are it wasn't written by a chatbot. [laughter]

So, yeah, I think it is true that I have never actually sent anyone an email drafted by a chatbot yet. But you can feel it, right?

And this time next year, we'll come back and see where it's going. Totally. Supabase shout-out, another famous Kiwi. I don't know if you've had any conversations with him about anything in particular on AI building and AI infrastructure.

We've had Twitter DMs with him because we're quite big Supabase users and power users, and we probably do some things more manually than we should in Supabase.

So he's just the support line because you're curious?

A little bit, yeah. Being super friendly. [laughter]

One extra point regarding GDPval-AA is that, on the basis of the overperformance of the models compared to the chatbots, we realized that our reference harness actually works quite well on generalist agentic tasks. This proves it, in a sense. The agent harness is very minimalist. I think it follows some of the ideas that are in Claude Code, and all that we give it is context-management capabilities, a web search, a web-browsing tool, and a code-execution environment.

Anything else? We can equip it with more tools, but by default, yeah, that's it. We give it, for GDPval, a tool to view an image specifically because the models can just use a terminal to pull stuff in text format into context, but to pull visual stuff into context, we had to give them a custom tool.

Yeah, exactly. You can explain the next part.

No, so we turned out that we created a good generalist agentic harness, and so we released that on GitHub yesterday. It's called Stirrup. If you want to check it out, it's a great base for building a generalist agent. It is kind of cool.

For more specific tasks, I'd say the best way to use it is to git clone it and then have your favorite coding agent make changes to it to do whatever you want, because it's not that many lines of code and the coding agents can work with it super well.

Well, that's nice for the community to explore and share and hack on it. I think maybe in other similar environments, the Terminal-Bench guys have started Harbor, and so it's a bundle of, well, we need our minimal harness, which for them is Terminus, and we also need the RL environment or Docker deployment thing to run independently. So, I don't know if you've looked into Harbor at all. Is that a standard that people want to adopt?

Yeah, we've looked at it from an evals perspective, and we love Terminal-Bench and host benchmarks of Terminal-Bench on Artificial Analysis. We've looked at it from a coding-agent perspective, but I could see it being a great basis for any kind of agents.

I think where we're getting to is that these models have gotten smart enough, and they've gotten better tools, that they can perform better when just given a minimalist set of tools and let them run. Let the model control the agentic workflow rather than using another framework that's a bit more built out and tries to dictate the flow.

Awesome. Let's cover the Openness Index, and then let's go into the report stuff. So that's the last of the proprietary AA numbers, I guess. I don't know how you classify all these. So that's AALCR for people. Yeah, yeah, yeah, yeah. But let's talk about the Openness Index. This here is, call it, a new way to think about how open models are.

We have for a long time tracked where the models are open weights and what the licenses on them are, and that's pretty useful. That tells you what you're allowed to do with the weights of a model, but there is this whole other dimension to how open models are that is pretty important that we haven't tracked until now, and that's how much is disclosed about how it was made.

So, transparency about pre-training data and post-training data, whether you're allowed to use that data, and transparency about methodology and training code. Basically, those are the components. We bring them together to score an Openness Index for models, so that you can, in one place, get this full picture of how open a model is.

I feel like I've seen a couple of other people try to do this, but they're not maintained. I do think this does matter. I don't know what the numbers mean apart from—is there a max number? Is this out of 20?

It's out of 18 currently. We've got an Openness Index page, but essentially these are points. You get points for being more open across these different categories, and the maximum you can achieve is 18. So AI2, with their extremely open OLMo 3 32B Think model, is the leader, in a sense.

Is it Hugging Face?

Oh, with their small model, it's coming soon. I think we need to get the intelligence benchmarks running to get it on the site.

You can't have an Openness Index and not include Hugging Face.

We love Hugging Face. We'll have that on our version.

I mean, you know, FineWeb and all that stuff—it's amazing.

FineWeb.

FineWeb, yeah, yeah, no, totally.

One of the reasons this is cool is that if you're trying to understand the holistic picture of models and what you can do with all the stuff the company is contributing, this gives you that picture. We are going to keep it up to date alongside all the models that we do an Intelligence Index on across the site, and it's just an extra view to understand.

Can you scroll down to the—I'll start—yeah, yeah, that one. This really matters, right? Obviously, because you can be super open but dumb. [laughter] This line obviously goes the wrong way here, right? A lot of people would like to see labs hill-climb on the entire game.

This is the axis to hill-climb, yeah. Unfortunately, it might be fundamentally true that the slope will always go this direction, because once you open something up, then everyone else can get to the level of what you opened up.

Well, let me tweak your point system, right? You have these numbers in the point system and it goes up to 18, but just because I have a little bit of open data doesn't mean I'm necessarily that much better than someone who put a lot of effort into their open weights, who is smarter. So, I might just mess with the point system to make sure that I'm accurately representing the contribution to openness.

It is hard to weight for the materiality of the contribution to open source. We tried to make it so that it is quite well-defined and no one can disagree about which category things should be in, and so we're not saying this was a big contribution or a small contribution in terms of impact on the industry or anything. It's just how much of your data did you release?

I would say that it is still valid to say that we trained a model that's not that smart, maybe even not at the frontier for a particular size category, but we chose to open up all the data and all the training code. That is a very useful exercise for the industry, and we want to recognize that even if it isn't the smartest model in the category.

Yeah, and also, special shout-out to NVIDIA Nemotron, which doesn't get enough credit for the amount of stuff that they do. Honestly, it's a sales-enablement effort for NVIDIA as well—the fact that they can do this as a side project.

Totally. [laughter] But it is true that NVIDIA has actually put an enormous amount of effort over the last year, especially, into the Nemotron models.

Yeah, and so many people actually use it for synthetic data and stuff. It's a pretty interesting secret of the industry that NVIDIA holds up all these guys.

I mean, it's in their interest for there to be more AI.

So, obviously, I think you want to push openness as having an index. Every index that you push has encoded in it some kind of opinion or value. One of the openness questions for this year was people messing with the license, and so Llama had this: if you have 700 million daily active users, you're not allowed to use our model, or you have to talk to us, something like that.

So, basically, what are your customers telling you about the kind of licensing worries that they have? Obviously, most people will never hit 700 million users.

We have a detailed breakdown of that in the Openness Index, and that was actually one of the initial questions that took us down the road of wanting to do this.

The simplest thing, in our opinion, is that there is a lot of advantage to having an official OSI license, like MIT or Apache 2, because then the box is just checked. You don't even need to read it because it's just Apache 2, and you can do whatever you want, and it's fine.

There are often very good reasons that companies don't want to release language models with those completely open licenses. The index tells you. So, if you get the top category, that's one of those licenses, and you're totally good. Then we've got some lower categories for when attribution is required and when commercial use is not allowed. That's the Openness Index.

Thank you for doing all that work. Let's talk a little bit—or at least end the pod—on the trend reports that you guys do, which are a bit of the bread and butter of how you make money. How do you encourage everyone to see George's talk at World's Fair, which gives a little bit of a preview, and you were very excited about talking about the smiling curve, or I don't know what you call it.

Yeah. Let's talk about that one. Let's explain it for people.

Yeah, we've got a copy of it. A copy of the slide would be excellent. It's important for people to have it in their heads because people only get the marketing message from the labs that we're cutting costs all the time.

Yeah, but it's true. It's just that it's not the whole picture. So, the big trends that we track at Artificial Analysis over time, and that we're always showing charts of on the Trends page in these reports, are these.

One is that the cost of intelligence has been falling dramatically over the last couple of years. The best way to think about that is that the cost for each tier of intelligence has been dropping. One fact is that you can get intelligence at the level of GPT-4 for over 100 times cheaper than GPT-4 was at launch.

I think my number's 1,000, actually. If you look at the Amazon Nova models, which are very, very cheap.

Yeah, my conservative statement is normally 100 times, but in fairness, this slide that we were actually talking about for the podcast might be 6 months old now. It's conceptually still correct, but we could probably update the exact numbers because the market's moving so quickly.

If you feel like kicking it off, we'll have this chart. Let's introduce the context that makes you make something like this.

There are 2 trends that seem not to make sense together, both of which we talk a lot about at Artificial Analysis and are very important to developers building things in AI. The first is that the cost of intelligence for each level of intelligence has been dropping dramatically over the last couple of years.

We track the cost to run models at each bucket of scores on the Artificial Analysis Intelligence Index, and in each bucket, you just see the line go down really, really quickly. It actually goes down more quickly for each new level of intelligence that's been achieved over the last couple of years. So, the rate of that cost function has actually been going up.

We've got that being true, and yet it is clearly possible to spend quite a lot more on AI inference now than it was a couple of years ago.

NVIDIA stock goes up. It's going really up. I just heard from a friend's startup that just went to the Asia-Pacific. They're spending $5,000 per employee on coding-agent spend alone. That's an impressive number. We need to get our numbers up.

It's so high that I'm like, "Are you doing something wrong?" There are a lot of efficiency questions along the way, but you can make AI inference useful to that level in a bunch of ways that I can imagine, right?

Yeah, I don't think that's that nuts. Basically, the reason we made this slide is to ask the question. The crazy thing is that it is actually true. We've had this 100x to 1,000x decline in the cost of GPT-4-level intelligence on the left-hand side, and yet on the right-hand side, because the multipliers are so big, even though small models can do GPT-4-level tasks now, we still want to use big models, and probably bigger-than-ever models, to do frontier-level intelligence.

We've got reasoning models using tokens, and then we're throwing them into these agentic workflows, where they're consuming enormous numbers of input tokens and making enormous numbers of output tokens, working for a really long time. Those 2 things taken together get you back to being able to spend enormously more today than we could a couple of years ago.

Yeah, I think that's right. There are a number of drivers at play, and we outline 6 key ones here. AI is changing quickly; all of these have changed very dramatically in the last 12 months.

Let's pick on hardware efficiency, since you also track hardware stuff. I think the general assumption, or the message, is that the efficiency from next-generation NVIDIA chips is actually not 4x. Is it 3x or 4x? You have 3x in here, and it's like 2x, maybe, or it's more of a power story rather than a share of compute-token efficiency story. What's going on in hardware?

The answer, unfortunately, is that it depends, and it depends massively on so many things across a bunch of different types of workloads and ways to think about it.

One of the simplest ways to think about this is to take a single relevant model, think about serving it at speeds that are realistic for what you actually might want to hit and can afford to hit, and then think about the throughput per GPU that you can achieve serving the model at those speeds. One of the reasons that's important is that there's a trade-off between the throughput per GPU that you can achieve and the per-user speed that you can achieve. It costs more to serve things fast to users.

When you run all of that for especially big sparse models, you can get a lot better than a 2x or 3x gain going from the Hopper to the Blackwell generation of NVIDIA. This shouldn't be too controversial to say, but I'm pretty confident that Blackwell has delivered pretty enormous gains, and that the next couple of years of NVIDIA's road map are going to continue to deliver quite enormous gains.

Those will actually come through as lower total cost per token to the companies that are running models on them. They will allow bigger models and way more tokens to be made for lower cost, and that's going to continue. These things also stack on all of the software and model improvements being made. So, basically, my prediction across both sides of that small chart is that we're going to see the left-hand side continue to be true, probably for another order of magnitude, and the right-hand side continue to be true for another order of magnitude, and that's going to enable a whole lot of things.

Okay. Well, I'll push on the small chart. We've gone a long way on sparsity. DeepSeek was a major pusher of fine-grained experts, let's call it. I have a mental number for sparsity in terms of active parameters versus total parameters, and that number went from 25%, let's say, down to 15%.

You obviously can't really go below—I don't know—5%. Is there a lower limit to sparsity? That's what I'm saying.

I don't know that that's that obvious, actually. There must be a limit somewhere, right? But we've got numbers in the wild that are quite a lot lower than that right now. The GPT-OSS models, the big ones, are about 5% active. Can we get to 2%? Is it 3% active? I think so; I'm pretty sure. I've looked at those numbers. I calculated them. I don't remember, but I remember thinking, "This must be it."

Your 5% is exactly around the ballpark for the open-weight models of what's released today. I think one interesting thing that gives me pause when thinking that the sparsity won't go higher, or that the percentage of active parameters won't go lower, is that in our benchmarks, we see a lot of performance correlated more with total parameters than active parameters, and not that correlated with how sparse the models are.

Our accuracy benchmark is part of AA-Omniscience. It's very correlated with total parameters. It's not correlated with active parameters at all, which I think is very interesting. So, I think there could be quite a bit to go here.

Awesome. We don't have that much time, but I did want to leave some room to cover reasoning and non-reasoning models and token efficiency. Let's do that one.

At a super-high level, people have to classify this binary thing of reasoning versus non-reasoning. People who are insiders have some discomfort with that because, basically, you just have to think: tag or no tag. How have you guys decided to approach this? And how is that laid out over the course of the year, where we have things like GPT-5, which is a model router?

Let's say GPT-5 and ChatGPT: the consumer experience is a model router.

When you're hitting the API, you can pick the different versions, and you can pick the reasoning strength of the different versions. But that goes to why this is now such a complex thing.

Earlier this year, and probably when you and George last spoke for the AI Engineer World's Fair, we had this great slide that was super easy. We would show that the average reasoning model was using 10 times the number of tokens per query in our Intelligence Index as the average non-reasoning model. There was a moment where that was a pretty clear distinction and extremely useful to look at just like that.

That's definitely no longer the case, not least because you can think about reasoning strength for a bunch of these different models, but particularly because different models have wildly different token efficiency now—more than an order of magnitude of difference. That means that the way you probably need to think about cost for any application is to use something like our Cost to Run Intelligence Index metric as the starting point for what it's going to look like for these different models, these different reasoning strengths, and this continuous spectrum from non-reasoning to reasoning.

We'll still show reasoning and non-reasoning, and define reasoning as when there is that separated chain of thought that you're getting at a different parameter in the API, normally. But it doesn't necessarily anymore mean that that model is actually going to have longer end-to-end latency or use more tokens than something else.

That's true. I think it was GPT-5.1, was it? And then GPT-5.1-Codex had this chart, which was super nice, of, let's say, the bottom 10% of queries being faster but the top 10% being longer. That's the kind of efficiency chart you want to see, right?

That's a really important extra thing, though, right? You've got not just the average number of tokens being used by the model, which we cover really well right now, but the behavior that you want in the model: it should use more tokens when it needs more tokens, and not use more tokens when it doesn't need more tokens.

That's what OpenAI was basically claiming that GPT-5.1-Codex is better at. We don't actually publish anything on this right now, but we've tracked it a bunch internally in our analytics on evals across all the models that we run, where we look at the difficulty of the questions and the correlation between token usage and difficulty. Net net, surprise, surprise, models have gotten better at doing that over the course of this year.

I think going into next year, that's going to be really important, especially as you multiply it by the number of steps in an agentic workflow that a model has to take to get to an answer. We're going to care a lot about token efficiency and number-of-turns efficiency for getting to what we want.

Which would you rather have: token efficiency or number-of-turns efficiency? Which is more important to work on?

It depends on the application, and both are going to be really important.

Because your total cost is really a trade-off between the two.

Interestingly, in τ²-bench Telecom, it's cheaper to run, on a per-token basis, more expensive models like GPT-5 compared to some smaller open-source models. Some of the GPT-5 runs, for instance, got to the answer faster and were able to resolve the customer's query faster in fewer turns.

Maybe it used more tokens per turn, but it certainly cost more per token. So you would always rather use GPT-5 in that scenario. I think that's where we're getting to. I think number of turns is going to be a metric that we're going to be talking about a lot more, and it'll be something that people want to really start to think about a lot more.

There's a trade-off in benchmarking here where most benchmarks need to be one turn—to be autonomous, to be parallelized, and all that. But a lot of real-life use cases need to be multi-turn, and especially quick multi-turns, so you can align.

Yeah. I would say that historically benchmarks have been single-turn, but I wouldn't say they need to be at all in the future. We have a couple of agentic benchmarks in the index right now, and in GDPval, which we were talking about, we let the models do up to 100 turns. Our Stir agent kind of asked us to do that as well.

It definitely is hard. You've got all kinds of infrastructure problems to run that and, exactly as you say, parallelize it, because we need to run that on hundreds of models and we want to do that really fast when new models come out and when labs want us to run it on their models. But you can do it. We're putting in the work to build that stuff, and it's going to be great.

Okay, so we've covered—I mean, there's a lot more to cover, and we haven't even touched on multimodal. That's huge. You also do speech benchmarking, image benchmarking, video benchmarking, and hardware.

I like the way that you've done it because it's very smart. Video takes a long time, so you pre-generate, right? Then people just pick their preferences and can see the overall arena results. You also avoid any sensitivity issues around unsafe content that's being generated.

Yeah. You can see it as a good thing or a bad thing, depending on what your view is, but it means that we have quite an active creative-direction approach to trying to understand what creative professionals and users want to do with those image and video models. That way, we can direct the arenas and our categories toward gathering votes on what people care about.

One call-out actually to listeners: if you are using our arenas, you can submit requests to us for things that we should cover.

I didn't know that.

Understudied categories, areas where you think the models are bad and the labs don't focus on them enough. If you want something solved, one of the levers that you have is to send us a couple of prompts on it. We might be able to get a category going on it.

This is the thing that we were talking about earlier, right? Once things get measured, they can get targeted. You can make that work for you.

For me, as a content creator: infographics. Very needed. I took the latest DeepSeek paper—they had some descriptions of their search agents and their coding agents—and I put it in and created an infographic.

I just think this is an industrial use case that doesn't require a lot of, I guess, design taste, but just requires some—you need to conform to some preset references, which is something that's increasingly important, especially in the Nano Banana series.

OpenAI is releasing image tools soon, which is going to have that. I think it's all of a kind where people need to incentivize workhorse use cases and not just art. I don't know.

Totally.

What are we going to be talking about next year? What's emerging that you're seeing, maybe not in the discussion?

The first answer that I'll give to that is the boring answer: on most of our charts, the lines go in a particular direction, and our overall prediction is that the lines are going to keep going in that direction. We're going to do a lot to be as useful as possible to developers and companies to measure what's important on every one of those lines.

But I think we're going to talk about similar stuff. It's just that we're going to have continued on this trajectory for another year, and things are going to feel pretty different because of that happening. I know this is the boring answer to that question.

No, no. I'm a fan of truths that don't change because you can build and plan for that. I think in media in general—in the podcast business, newsletter business, and Twitter business—people are addicted to change. “Oh, everything's breaking, everything's…” No. There are some truths that are just constants that you can plan on and build.

I think one of the truths is that the demand for AI intelligence and smarter AI intelligence is going to be insatiable. Some people disagree that once we reach certain thresholds, you don't need more intelligence. To that, I ask people: have they ever worked with or managed someone in a work environment and not pressed a button to make them smarter or better at their job? Or would they never press that for themselves? I'm not sure that's the case.

I think for Artificial Analysis, we'll keep benchmarking raw intelligence, but we also want to think about and explore models more deeply across other axes as well. I think hallucinations are the start of that, but we're getting into wanting to support people in understanding the behavior and personalities of the models to help people make more nuanced decisions.

Are you going to have a personality benchmark?

That is a direction that OpenAI is leaning into a lot. So if you manage to solve that, you should definitely talk to Fidji and Rune.

Oh, okay. Yeah, so what is going to be included in, let's say, a V3 of the Intelligence Index? Obviously, you're going to saturate in March. Why don't we break it now? How soon is the podcast going to come out?

Whenever you want. Okay, so we're at V3 right now. The version that's going inside of it is V3. V4 is what we're going to call the next major update.

Surprise, surprise, we're going to be adding several of the things that we've actually talked about today that we've launched over the last few weeks.

So it’s not going to be wildly shocking, but some of the things that are most exciting are adding GDPval, which is going to give us this general agentic performance in a really strong way in the Intelligence Index. Adding CritPt, the physics eval George was talking about, similar to FrontierMath, gives us a completely new view with a brand-new data set of very, very hard research problems. We are going to be using AA-Omniscience, and we are going to be using hallucination rate. The exact ways that all those are going to come together—the weighting is going to be hard because the numbers are different.

Yeah. Yeah. We’re going to make sure that we don’t do anything to cause odd distortions and stuff that could be misleading.

Every time you version it, you have a one-time reset.

Exactly. That’s exactly how we think about it. We will make sure that within each version number, there’s no drift in any of the scores so that people can rely on them and reference them. You just have to watch out for that version number. Once it’s V4.1, those numbers won’t be comparable with V4.

Of course. There is a little bit of debate over the accuracy of τ-bench. I don’t know if you’re clued in to what’s going on.

Apparently, a very high number of τ-bench tasks have been impossible. Potentially, for the earlier versions, τ-bench Telecom, we’re pretty convinced, is pretty good. If anything, the only issue there is that models have got very good at doing it.

Yeah. Yeah. On we go. Okay, well, thank you so much for providing such a great service to the industry. I’m glad to at least know you guys from before you got famous, and now you are famous. [laughter]

Oh, look, our pleasure, and we really appreciate your support along the way. I wasn’t kidding at the start, right? It was quite a material moment for us when Artificial Analysis was covered on Latent Space. Some random guy in San Francisco mentions you, and I was a fan of Latent Space for like a year before you mentioned us. So I’ve been listening. I don’t think I was familiar with you personally yet at that point, but I listened to your voice for probably many, many hours. [laughter]

And so once you mentioned it, got to know you, and met you for the first time nearly a couple of years ago, it was really cool, honestly. So, yeah. And thanks for being such a great member of the community and spotlighting projects which don’t have attention and bringing them to your audience.

Yeah. Well, actually, it wasn’t me, right? Someone in the Discord dropped it in our Discord, and I rely on our community. It kind of feeds itself, right?

Nice.

So someone brought it to my attention. I don’t know who; we should probably go back and check. But once I saw it, I was like, “This looks good. This is something I always wanted. I wanted to build it. I was too shy or dumb or lazy to build it, and you guys did, and now it’s the whole thing.” You built some really cool other stuff, like this button. [laughter]

Yeah, yeah. Totally. So thank you. That’s it. Great. Cool. Thanks.