Ep. 002 - InferenceX 2.0 Release (Technical Staff) | Cam Quilici, Bryan Shan, Doug O'Laughlin, Jordan Nanos

All right. Welcome back to the SemiAnalysis Weekly Podcast, No. 2. Today, Doug and I are joined by Cam Quilici and Bryan Shan. We're going to talk about InferenceX. Guys, welcome to the show.

Hey, what's up, Jordan?

Nice to be here.

We're covering, I think, all 3 major time zones right now in the U.S., right?

Yep.

We're missing Mountain Time, maybe, but we've got Pacific, Eastern, and Central representing. Doug, welcome back, man. You just got off the train?

Yeah. I just came down from Boston, took the train, and took a lot of calls on the train. Now here we are, having our weekly podcast.

How long is the train ride from Boston to New York?

It's like 4 hours, dude.

It's very long.

Honestly, and this is completely off topic already, it's just nice that you don't have to go through airport security and stuff like that. You can usually work on the train. That's my favorite part. The Wi-Fi is okay.

Do you buy the first class on Metro—

Yeah.

—or whatever?

Yeah, dude. It's business class, bro. Pretty expensive. So, yeah, but let's go.

Cam and Bryan, and also Jordan, are all part of the team that does the beautiful work at InferenceX, not Max. I think that's what we're going to go over today, right? Am I—

Yep.

Am I off base?

1. InferenceX Version 2 Expands The Benchmark

Yeah, absolutely. So, obviously, Version 2 of the InferenceX article came out earlier this week, and in the previous edition, there have been a lot of updates. First of all, there's the name change from InferenceMax to InferenceX. Shout-out to HBO Max for that one.

Perfect.

The other, more interesting changes are really focused on 2 things. One is the dashboard itself, which anybody can log into and see. You can go to inferencemax.ai or inferencex.semianalysis.com.

InferenceX.ai.

InferenceX.com, not AI, right?

Oh, .com, right. Yeah.

Any of them work. You can actually see the results for yourself. We can talk about what these results actually are.

The second part is the code. In other words, it's an open repo where anybody can see how to run this stuff, fork it, and use it for themselves. Guys, can you start by giving more background on why we're putting out this 2.0 release, and what the major changes are from the original InferenceMax to now moving to InferenceX?

Yeah, definitely. Originally, with InferenceMax 1.0, the goal was to benchmark a ton of different GPU SKUs on various production workloads. But for the last 3 months, we've mainly been benchmarking just single-node deployments of DeepSeek, GPT-OSS, and a few Llama 70B models.

In reality, big labs, neoclouds, and serverless providers are using more state-of-the-art techniques for inference, such as disaggregated prefill, wide expert parallelism, and speculative decoding. In order to have a more nuanced and robust benchmark, we wanted to include those.

Since the 1.0 release, I think that's been the primary focus: going at it and trying to add all the disagg, all the YDP benchmarks. The results really show a better picture of what frontier inference is like in production.

Okay. Specifically, that means the DeepSeek-R1 model was added, right?

Right. DeepSeek was there in Version 1, but it was only single-node. Now it's multi-node. We're testing the entire NVLink domain, as well as multiple nodes of MI355X with their Pollara NICs. It's pretty cool.

Yeah, and explain that as well, because there's a whole section on it. The first part is the models and the software configuration on the runtime to enable features like speculative decoding and prefill-decode disaggregation. But the second part is the hardware.

The first release did not include GB300, and it didn't include B300, either, which are both new GPUs from NVIDIA being included in 2.0. The major result, maybe, is showing how big an improvement GB200 or GB300 is actually bringing—not with the Grace CPU, but with the NVLink domain, right?

Right. Specifically with DeepSeek, you can run expert parallelism, which is effectively a form of parallelism for LLM inference. For DeepSeek, there are 256 experts, and you can basically allocate or disperse these experts across a bunch of different GPUs.

When you're doing this on one node, you're running EP8, so you have 256 divided by 8, or 32, experts on each GPU. When you're running that across, say, 8 nodes of B200, you're dispersing over 72 GPUs, but it's going over InfiniBand for scale-out because—

When you're running on GB200, you have an NVLink domain, and it's like 72 GPUs acting as one, as they say. Expert parallelism is just so fast, and we can go into those results later, but it's very clear how that frame-mogs all of the—

Frame-mog it. Yeah, yeah.

2. Frontier GPUs Transform Inference

Let me actually go into the results right now, because I think this is probably the most key thing. For those watching, I've got the InferenceX Version 2 article on screen. This is the first major screenshot, and I think it's the biggest takeaway from these results: in a direct comparison of the max performance for DeepSeek that you can get out of an H100 with all the tunings and improvements that have come, compared directly with GB200 and GB300, at low interactivity levels—so, low speed per user—you get 20 times more throughput. In other words, you can support 20 times more users per GPU, or more requests per GPU.

At high interactivity levels, it's doing things that the H100 can't even do. So, 80 to 100 times more throughput per GPU at an interactivity level of 100 tokens per second, which, to be clear, we're seeing used today, right? There are models available, like Codex Spark from OpenAI, as well as—

I thought Codex Spark, by the way, was—is it on Cerebras or no? I mean—

Codex Spark is on Cerebras. We don't know exactly what Opus 4.6 Fast is being served on right now. We have our suspicions about Trainium and TPU because that's what Anthropic uses for inference compute.

But they have GPUs too, so—

Yeah. But the point is that those quote-unquote “fast models” are being served at interactivity levels above 100 tokens per second per user, from what we can see right now.

Yeah.

Whereas the standard models are more around 40. So, it's a real requirement for people who want to serve models at 100 tokens per second per user or per request. The fact that even at that level of interactivity, the GB200 or GB300 is showing 20 to 80 times more throughput is such an obvious statement that there have been incredible improvements generation over generation here.

So—

Yeah.

I'm sorry. To summarize: frame-mog, you know? Oh my God.

3. The Benchmark Keeps Improving

Other than the awesome work that the InferenceX team has done, I guess, since SemiAnalysis has a lot more people than just Dylan or me or even Jordan, I'd love to talk about the whole process of doing all this. I just don't feel like people appreciate the scale of what's happening and the tests that we're doing every single night.

I think Cam and Bryan can definitely offer a really cool insight into it. I'd like to hear what it's like making the spaghetti yourself, I guess.

Yeah. Bryan, I'll let you take that one, man.

Wait, did Jordan just disappear?

Oh.

Yeah, Jordan will come back when he comes back. Don't worry about him.

Oh, sure. Yeah. So, actually, we work quite closely with AMD and NVIDIA engineers who help us find recipes for these kinds of setups. Optimizing such a task is not easy. There are many environmental variables, and there are many settings to tweak.

We work very closely with both companies to try and showcase their hardware to the best of its potential, basically.

Mm-hmm.

Yeah.

And I definitely think something that people accuse us of is favoritism. But more than anything else, we really just try to make sure this benchmark is a living, breathing benchmark.

What's shown each night is effectively the best results, right? They work with us and put up numbers, right? These benchmarks have massively improved, partially because people at AMD and people at NVIDIA—both teams—have been working really hard on increasing throughput. I don't think there's favoritism; it's more that we just want to be the best benchmark and see the most token throughput.

Yes, exactly. And that's why we are currently trying to do day-zero support for some of the new Chinese models: Qwen 3.5, Minimax, Kimi 2.5. We really want to track how AMD and NVIDIA actually improve their implementations for these models over time.

Yeah, I think that's a good point. A lot of the benchmarks that currently exist are kind of just 1-point-in-time things, and not only 1 point in time, but 1 single point on the Pareto frontier, right? As Bryan said, we're really focused on trying to show that this is a very hard problem to solve, and a lot of people work very hard to improve it and push the frontier forward iteratively, week by week. So I think that's one of our North Star goals.

What was before this, Cam? Because I know you have experience doing inference benchmarking before InferenceX came along. So when you're saying this runs nightly, what was it like before that? Leading question.

Yeah. In my previous experience, I think Bryan has some experience with other consortiums of inference benchmarks, too. I won't name names, but benchmark maxing is a word that comes to mind, right? It's important to understand, for those who don't have the background, that I'm going to share my screen real quick, if I can. Okay, cool.

The idea here is, if I look at GB200, this is an entire curve of different configurations that you can tune and serve at. I don't know if Claude Code wants a really fast mode, right? We can talk about this in more detail, but they have lower total system throughput while serving users at 140 tokens per second, right? So each user receives tokens very fast.

But if you want to make the most out of your system and have total system throughput that's very high, you're going to be over here or somewhere in the middle. Really, there's no one-size-fits-all solution, and I think other benchmarks for inference kind of assume that as a prior: We just want to maximize the total throughput and advertise that as a single number. The problem of inference is so much more nuanced than that, so I think that's the difference there.

Another difference is that, as you mentioned, we run nightly or weekly, at a cadence that keeps pace with what vLLM, SGLang, TensorRT-LLM—all of the serving runtimes—release, instead of every 6 months or every quarter, whatever. The release cadence is just constant.

And that's the second chart in the article, showing the improvement in performance for ROCm on the DeepSeek model, right? It almost doubles in the span of, I think, 3 months or something.

Yeah. We can look at this right now, right? You're talking about MI355X multi-node here, right?

Yeah.

You can see that one of the first public recipes AMD has ever released, to my knowledge, was from January 8th. The most recent one is from January 17th. You can see that they're continuously improving. The square is FP8 and the circle is FP4, but let's just show FP8 for now. You can see that the curve is being pushed forward multiple times in a span of 1 or 2 months, which is impressive in and of itself, because this goes to show that it's not just hardware improvements. Hardware is released every 2 years, maybe every year and a half.

Yeah.

These are just engineering challenges, right?

Re-share your screen here, I think, because it would help for the recording.

Oh, I'm not sharing?

No, you were sharing, but it was frozen. I can bring it up on the other screen. There you go. Yeah, that's much better. For those viewing, or those just listening, basically Cam's got a graph from the InferenceX dashboard with 4 dates on it, January 8th through February 17th, and it shows an improvement from about 1,000 tokens per second of throughput per GPU at a 20-token-per-second interactivity level, all the way up to 3,000. So that's roughly triple the performance in 1 month through software-level optimization.

Yep.

To go back to the original motivation for answering this question, when there's a benchmark that's tied to a specific interactivity level, only accepts submissions every 3 to 6 months, and does not get updated over time, you can't actually see these improvements for a long time. You can't see them 3 times within a month, and it's hard to give feedback or get feedback when you're doing development at that speed, let's say.

Yeah. And one more thing on this note of the difference is that we really enjoy our partnerships with AMD and NVIDIA on an engineering level, and we want to show them in the best light always, for both companies. But with other benchmarks, it's much more supplier- or company-centric, right? So if a company submits a benchmark and sees that they're losing to someone, they can unsubmit the results, right?

Here, we don't really allow that. We are an independent entity here. We're just performing the benchmarks with their help, so we're going to show the results no matter what. Once they're up, they're not going to be taken down, right? I think that's an element of transparency that we have over other existing benchmarks.

Yeah. Awesome. How about that process of working together, right? There's been huge improvements in the software stack tied to the new hardware, right? Maybe you could walk through the improvements that you've seen, roughly 1 at a time, and how that comes to be.

Let's take prefill-decode disaggregation to start, right? If there's a framework that's developed, or a paper—really, a blog post—like DistServe that explains the theory of how to do this, it takes a while for that to actually show up in software that anybody can download and run. There's only a limited set of people in the world who have a GB200 NVL72 rack to test these optimizations on, right?

Right. Yeah.

How do you go through that? How do you go through doing that testing and working with NVIDIA or AMD to get the latest software?

Yeah. Bryan, do you want to talk about this?

You can go ahead first.

I mean, the fact of the matter is that most of the inference-level optimizations being done are by those AMD and NVIDIA engineers, right? All credit to them where credit is due. The Dynamo team, the TensorRT-LLM team, and the AMD Distributed Inference team are amazing, and they are extremely smart—smarter than I am. They're making these kernel-level improvements every day, and I think this is where these iterative improvements come from, right?

They have their own testing clusters and their own testing environments. We try to help them understand their optimizations, but all credit to them in that regard, I think.

Yeah, especially in the early stages of recently released models, we also try to work with them to discover new flags, search for recipes online that other people have found, or test recipes that other people have published.

Yeah. Bryan knows this: it's kind of like a puzzle, because you have these engines—vLLM, SGLang, and TensorRT-LLM—that have just blown up in size over the last few years, right? Every time you add a new feature, maybe there's a new feature flag, like an environment variable that you set for a backend for an MoE model or whatever.

Now there are 1,000 different environment variables you can set, and finding the right combination of which to use for what deployment is really a challenge that requires a very deep understanding of all these different frameworks.

Yeah. Another issue is that some of these optimizations work by themselves, but when you combine them, the effect isn't really replicated.

Yeah.

It's not as simple as just combining good flags.

It’s about finding flags that work well in combination.

Yeah, Bryan, I think you should talk more about the specific—

The AMD issue.

…problem. Yeah, AMD’s composability issues.

Yeah. So that’s one issue that we discovered with AMD: They have many flags that work well, and they say work well by themselves. They say, “This flag will boost performance by this much.” But when we actually combine them, they don’t really work well together. I don’t remember the exact flags that do this, but that’s one big issue that AMD has, specifically.

Yeah. At a high level, this can look like when you have FP8 inference, right? Plain, single-node FP8 inference is actually on par with that of the B200, its counterpart. But then when you go to FP4, FP4 inference is significantly worse than B200, and then when you add disaggregation and MTP, it’s significantly worse than even the last step—way worse than the last step, right? So it’s just that this composability of software is kind of an issue for AMD at the moment, but I think they can do better.

Yeah. Can you talk about some of the analysis? We’ve talked a little bit about prefill-decode disaggregation and MTP. These are relatively complicated to understand because I think you need to understand a little bit about how a transformer model works to be able to understand the concept. But one that’s probably pretty simple is MTP, or multi-token prediction, here, and the chart that shows how much that can reduce cost is a pretty clear, obvious one—that horizontal bar chart that you made, Bryan. So maybe you’d be willing to walk through what sort of improvements MTP has brought and, just as an example of one of the optimizations that you could turn on, explain what MTP is.

Of course.

Yeah, sure.

4. MTP Cuts Inference Costs

MTP is actually a recent type of speculative decoding. Previously, we used smaller draft models to produce these tokens, but DeepSeek has actually popularized this method of using MTP heads within the model that are pretrained along with the model for this multi-token prediction. Basically, these optimizations in general exploit this memory-bound issue with decoding. Because during decoding, you’re actually producing logits for previous tokens as well, not just for the next token, speculative decoding makes use of these logits to verify past tokens.

There’s actually no drop in evaluation performance, as our charts on GSM8K show. Basically, this is almost a free optimization in the sense that it provides a lot of performance gains at almost no cost to accuracy.

Yeah. Just to be clear, GSM8K is a benchmark for grade-school math. It’s simple math questions, and you validate that the model can still perform on all those reference examples—it can still hit the reference level of performance on that evaluation.

Yep.

One of the things that SemiAnalysis has spent a lot of time doing is showing the analysis beyond this, right? MTP makes things faster, but what does this actually mean for users? The result is clearly cheaper tokens. Either the provider can increase their margins because they can serve the tokens for cheaper at the same price, or they can pass those savings on to their customers, right?

Exactly.

Yeah.

Also, maybe a dumb question: How would we know how the realized cost differences are going to be seen? I’ve been wondering this. For example, is this just going to look like a better cache hit rate, or is this truly just going to be that they’re going to eat the margin and your end customer will never really see the difference? I would love to hear what you guys think. How do you think MTP will work in practice? And also, I guess the question is: Are these changes even in production today? Because it’s relatively new.

Yeah. MTP is specific to DeepSeek, and yes, it’s definitely in production. Other speculative decoding methods are also used in production. I mean, like Bryan said, it’s a bit more nuanced than this, but it’s basically a free optimization in terms of accuracy because of the way the probability distributions work. So, yeah, they’re definitely being used in production.

And then I really have no idea how the savings will be used, right? I’m not in finance; I don’t know the economics of this. But, as Jordan was just showing on his screen, it’s significantly cheaper in some scenarios, like by 50% or more. So I’m assuming some of those savings will make it to the end user, but I’m not entirely sure.

Yeah. The chart that I was just showing on screen showed that the B300 with Dynamo and the TensorRT runtime was serving the DeepSeek model at 68 tokens per second per user, a reasonable speed, at a price of 25 cents per million tokens, which many people can compare to an existing price that they might spend on a model, right?

People will be familiar with a model that might cost $1 per million tokens or $15 per million tokens. Opus is at that level for the 4.6 model. It depends on your blend of input/output tokens, but we’re making an assumption here—I think a 3:1 input-to-output ratio—and we just price it that way for input and output, combined. And so when you turn MTP on, it goes from 25 cents per million tokens to 5.7 cents per million tokens—an 80% decrease in the example shown, right?

Right.

And so you’re talking about a free optimization, for sure. But I think the key point to tease out about why it’s cool that InferenceX does this is that you can just see the results for both.

Right.

You want to see it with MTP on, and you want to see it with MTP off. You want to see the Y-axis with throughput per GPU or cost per million tokens, or throughput per GPU per all-in utility megawatt, if you care about power. We have all of that analysis done based on our data in terms of how much servers cost, how much power costs, and things like that.

So maybe we can talk about more of the metrics or analysis that you guys did beyond just running it and seeing the output on throughput. It’s about actually drawing the conclusions of how much it’s going to cost people.

Yeah. I can share my screen, and Bryan and I can just talk about this. I can hear myself in the background. I’m not sure if that’s my fault or someone’s—

My connection froze with what we’re using to record here.

Yeah. While Bryan is sharing his screen, I just want to say that DeepSeek has really popularized MTP, and basically everyone is using it now. The recent GLM-5, as well as Qwen3.5, also support MTP.

Yeah. So, okay. One example here of the metrics that a lot of people care about is obviously cost, right? So if I want to compare B200 to MI355 distributed inference, I’ll select MI355, and I’ll select B200, and we’ll compare SGLang to both so we’re apples to apples.

Here you can see in this scenario it’s pretty close: B200 and MI355 are, for all intents and purposes, equal in terms of total throughput. But our amazing analysts also have a TCO model where we count all the components that go into owning a GPU and calculate the cost per million tokens using that. Here you have all the TCO.

You can see it becomes clearer that at these more realistic interactivity levels, since MI355 is slightly cheaper than B200 by about 50 cents, MI355 is actually more economical by, I don’t know, 20 cents or so. It doesn’t seem like a lot, but when margins are pretty thin, that can make a big difference. And then—

25% change, right? The first graph you had was just the raw performance, but it doesn’t consider how much the GPUs cost.

Right. Exactly.

When you consider how much the GPUs cost, the performance per dollar is actually going to be better on the MI355.

Right.

Yeah.

Guys, why do we make all these changes? Am I crazy? This was not working a few days ago. Am I crazy as fuck?

Yeah. We hired a great front-end engineer who’s been super locked in, so this is great.

Yeah, I was going to say, man. I literally think when we did the InferenceX launch, none of this was here. Or it was like—

Yeah.

…yeah.

5. InferenceX Builds Real World Benchmarks

I mean, another thing I want to mention is the next steps for InferenceX. Right now, you have these 8K/1K, 1K/1K, 1K/8K, and 8K/8K workloads, right? This would be just basic QA. This would be long decode, so maybe coding-agent output.

Sorry, a little more explanation: 8K means 8,000 input tokens, and 1K means—

Right.

1,000 output tokens, and—

Yeah.

Vice versa for 1K/8K, and then 1K/1K being balanced.

Right. Right. Right. So, right now, InferenceX uses random data—random tokens—for these benchmarks. We're pre-filling between 80% and 100% of the input and decoding between 80% and 100% of the output, such that we have variable sequence lengths.

The point is, this is 8K random tokens, random words as input and 1K random tokens as output. That leads to a scenario where you don't have any prefix caching. For those of you who don't know what that means, when you're using ChatGPT and you ask something, then ask another question and another question, the whole conversation gets sent each time. But the prompts are being appended to some KV cache each time, so they don't have to be recomputed on each turn of the conversation.

When you're using random data, you don't get that because they're just single-turn conversations of random data. So, this data is a baseline because it's assuming no prefix caching and completely random data, so the speculative decoding distributions and acceptance rates won't be super high. In a way, that's good, because you can think of this current data as the worst it would be. Without any prefix caching or anything, this is the worst it would be. It would only get better from here.

Moving forward, we want to do more realistic benchmarks with real data, multi-turn data, prefix caching, SSD KV-cache offloading, and DRAM KV-cache offloading. This is just, frame-of-reference-wise, the best way to think about it: this is the worst result you can get from this GPU.

Yeah. You guys have big plans.

Big plans. Big plans coming soon. Honestly, it's pretty bullish in terms of inference margins. Effectively, if we're not seeing massive cost reductions in inference every month or so, margins should go up quite a bit. We're talking about multi-node, multi-turn, MTP, all this stuff. Assuming this is the worst case, the best case does seem like 50% price decreases.

I think that may be underappreciated because we've done quite a bit of work internally on the tokenomics stuff, specifically focused on how many tokens you do a month and where your break-even is on Max. In the beginning, we were like, “Yeah, these guys are losing money.” But all of these inference optimizations, specifically some of the ones you mentioned, make you think, “Wait, that actually means that Max might not be as unprofitable as you think.”

I don't think our first stabs at the estimates, Jordan, really encompass a full, all-in, high-cache-rate scenario, all that kind of stuff.

Yeah. I mean, Cam is talking—there's a way in which this offsets. I agree with you. DeepSeek is likely a smaller model than what the frontier labs are running. In some cases, these guys are implementing some of the latest DeepEP and PD disaggregation optimizations pretty quickly after they're released, so it's not clear that that's running the whole fleet of all the models everywhere.

A lot of these benefits are applying to the latest and greatest GPUs, and we know that they're running big fleets of old GPUs, serving both inference and training still. You need optimizations on all this stuff, not just the GB300 NVL72, because that's not your entire inference fleet right now.

So what Jordan is saying is, instead of adding TPUs and Trainium like people want us to, we should be adding A100s, you know? That's the real frontier that's going—

Absolutely not what I'm saying. Yeah, yeah, yeah. We need to focus on the L40S for the next round, guys.

No, I think in some ways these things offset, right? You can think that InferenceX is always going to be a little lagging behind what the absolute frontier is in terms of model size and optimizations on the runtimes, because we're getting what's open source, not what's closed source. But on the other side, we're pushing to show the cost basis of the latest GPUs, which is really attractive.

I think the other thing I'd consider on InferenceX is how many improvements you guys have made since the launch. Doug, you commented on this a second ago, but it's both on the system side and on the datasets you were just talking about, Cam, as well as the models. Can you talk a little bit about the roadmap—new GPUs, new models, new benchmarks and datasets? We're just shipping even more from here, right?

Yeah. From a V3 standpoint, some of our big goals are to begin implementing Trainium and TPU, which is something a lot of people want to see. This is going to be awesome when we do it. I don't know if it's going to be by V3, but V3, V4—it's a long-term goal.

More immediately, though, as I just mentioned, we want to develop a state-of-the-art, real-world, multi-turn, long-prefix-cache benchmark and dataset that we can use to have the most real-world performance benchmark we can possibly curate. I think that's something to look forward to.

In terms of models, I'll let Bryan talk a bit about that. I think he's been doing some cool work.

Yeah. Of course, everyone's waiting for DeepSeek V4, so we are aiming for day-zero support. It's not just about the models; we're also looking at different inference optimizations.

One thing we know is that multimodal LLMs are quite important, especially with agentic work. Browser Use, Playwright—all of these require some sort of vision for the models. There's a new paradigm called EPD, or Encoder-Prefill-Decode, where we separate the encoder onto different nodes as well, like we do with prefill and decode.

We're also trying to bring this into V3, where we actually start to benchmark images as well, not just text, because of this importance in agentic workflows.

Where are you going to get that data? All of Doug's Claude Code traces?

Yeah. We're going to ask Doug to make a tarball of 15 petabytes of Claude Code sessions and just send them over.

Dude, at this point, I know OpenAI was like, “How are you guys using it for non-coding use cases?” They're going to lock me in a room and be like, “Generate data all day, dude. Make the fucking training data.”

You're like the infinite-monkey thing, bro. You're just making a monkey type Shakespeare.

But all jokes aside, this is actually a problem that we've been running into. We're like, “We want to make better benchmarks with multimodal data, multi-turn data, agentic coding, and so on.” I've been doing a lot of research on datasets for this, and they kind of don't exist because these are very valuable datasets for training and benchmarking.

I think we're going to have to make our own, actually. We'll see how it plays out. You can't find many online. The best one online right now is probably WildChat, and that's still about 3 years old. It's GPT-4o, with people asking GPT to be their therapist and stuff. I don't know, but we'll get there. They're very hard to come by.

I have a really good query dataset if you want a benchmark. There's an old AOL query dataset from around when AOL was coming out in terms of search. I think it came from Yahoo! It's 500 randomized queries from 1999. That's my personal dataset I've been using whenever I do random, weird benchmarks, but it's not multi-turn. That's the real issue: the multi-turn.

My most consistent multi-turn, multimodal use case is probably, “This is broken. Here's a screenshot. Can you fix it?” That one is really hard to generate.

Yeah.

Yeah.

Yeah. Something we've been talking about for agentic benchmark datasets is literally spinning up a bunch of the models—DeepSeek, whatever, frontier models—on these GPUs that we have, having a bunch of OpenCode instances, starting with an initial prompt, and telling them to simulate an environment where they would be coding. Then we could capture the traces from that.

Yeah. What we're doing is not necessarily building a golden dataset that's going to be used for all benchmarks into the future, as if it's the future shared GPT or something. I think what we've talked about is the process by which we can gather data and use it in our benchmarks, because we're just going to trace—

Yeah.

All the guys are doing Claude Code for the future, and then there’s going to be agentic stuff, and there’s going to be multimodal stuff. We just need a process internally to capture all that data and use it in our benchmarking, which is turning SemiAnalysis into the Data Lab. We’re going to be better, Doug, with your Excel benchmark, and—

Yeah.

Next, scale it up.

I, I do want to do the Excel benchmark. We’ll have Ablos do the PowerPoint benchmark. I mean, the PowerPoint benchmark already feels saturated, unfortunately. And then Bryan had a really good little benchmark for the agent stuff.

6. Agentic Models Defy Simple Rankings

I would love to have this conversation because I know I saw Cam say this. We’ve been shit on on the internet because we’re Opus lovers, but how was your experience using 5.3, out of curiosity? I’ve just completely—

Me or Bryan?

Cam or Bryan?

Yeah. Yesterday, Doug and I were talking about this. I had a problem where I was using FFmpeg, and Opus just couldn’t solve this issue. I was like, “Okay, fuck it. I’m using Codex 5.3.” I used it, and it did solve the problem. I don’t know if I was helping it, but I think Codex is better for targeted issues, and Opus is better for general vibes and creativity. I think that’s the general consensus.

That was the first time I used Codex, and immediately, within 15 minutes of using it, I was like, “Dude, this is better,” I think.

Yeah, no, I definitely agree. I’m not a coder, is the problem, so it’s just not useful for me because I like the general vibes of 4.6 doing broad stuff and being like, “Hey, read this stupid report and summarize it for me,” and then also, you know, make it into a scraper or something like that.

But I just want to put it out there because I know Rune on Twitter keeps dragging us for 180-ing. No, Codex 5.3 is actually good. It’s also kind of weird because this is back to the agentic benchmarking stuff. We are maybe past the linear benchmarks, for lack of a better word, but there definitely continues to be this massively jagged frontier.

That’s a perfect example. How do we benchmark the fact that Codex is really, really, really good at doing a one-shot targeted ask, but it’s very bad at being like, “Make the whole project from scratch. Make no mistakes,” right? I have no idea. That’s becoming, I think, the most interesting problem to solve or understand: the agentic benchmarks, for lack of a better phrasing.

But we’re also like, yeah, I don’t know. It’s fucking wild. It’s been fun times, though.

I think the idea that having different models contribute to different tasks is a good thing, and we’re hopefully going to continue to motivate the people signing the checks to have a subscription to everything. Use everything, man. I have that—

Are you saying—

—a multi-model evolution—

—a multicloud future is possible?

It’s a multipolar world, Doug.

Yeah. Multipolar.

I think you can tell Rune that we’re interested in updating our minds, or changing our minds, when new information presents itself, and we don’t like people who take the opposite approach. When new information is presented, we change our mind. What do you do?

Yeah. We change our minds a lot. It’s actually kind of getting tiring. Every week seems to be different, dude.

What do you mean you change your mind every week? What are you talking about?

Dude, I feel like I’m changing my mind. Time isn’t real, though, to be clear. I saw this tweet today where I was like, “Crazy, but you just actually rooflined agentic coding at 100% CPU for the human brain.” That’s how I feel, at least, in my code psychosis. Sorry.

Bryan, when was the last time you changed your mind?

I change my mind all the time, particularly on my general viewpoint of AI and its utility. Some days I’m like, “Dude, this is the stupidest thing ever. I just want to write code by hand and go back to 1985.” Then some days I’m like, “This just one-shotted my entire job.” I don’t know. What about you, Bryan?

Yeah, exactly. It’s like that graph of, “We are so back. AGI is here.” Then the second day it goes, “Oh, man, this thing sucks.”

Are you guys tired? One of my friends was telling me that he’s basically up all night coding now because he has these models. I don’t have that exact same vibe, but I am definitely tired from how much work I’ve been doing with AI now.

That’s why I’m telling you: the agentic human, the human CPU, actually rooflines at 100%.

With the current capabilities of Opus and Codex, do you believe that you are the current bottleneck?

Dude, it’s a hard question because you’ll ask Opus to do a task in the Inference Max front end, and it’ll just completely shit itself. Then you reprompt it 100 times, and it’s like—

Not quite there yet.

You know what? I’m not sure, because I think it’s a bit more nuanced. It’s weird. I feel like there’s some meta-level thing going on within the models. I have no idea. But if it goes down the wrong path, bro, it’s over.

Yeah.

Like, hard—

For most of the simple stuff I’m doing right now, I feel like I am the bottleneck, only because I can’t manage the 9 different tabs I’ve got open, trying to remind it to pull from main and merge the conflicts and stuff.

But the general feeling is that the model wants to code. Previously, the models wanted to learn. Now, man, the models want to code, and I just need to be able to prompt it. It’s always waiting for me to give it some feedback. Like—

Yeah. Yeah, I guess—

Yeah, exactly. The human brain can’t run in parallel, but we can run concurrently. I’m attending to this Claude, then the second I’m attending to Codex, et cetera.

Dude, you know what I was thinking? I was listening to the Dario Amodei episode of the Dwarkesh Podcast, and I was thinking about all of these podcasts I watch with these CEOs and engineers. The assumption is just like, okay, we’re assuming that these models are going to get better infinitely, forever and ever, and these harnesses are going to get better forever and ever.

I just had this fleeting thought: “Shit, man, what if that just isn’t the case? What if that’s not a good prior to have? What if they just stop getting better? Then what? Then what happens?” Then we’re fucked, man.

Cam the bear. Cam’s the senior bear.

I’m trying to be balanced, bro.

I think it’s good to always have your priors as, what is this, “strong beliefs, weakly held,” or something like that.

Yeah.

Looking at the history of technological innovation, blah, blah, blah. When you get really screwed is when you’re like, “Actually, I’m going to throw away all my skepticism, and I’m just going to blindly believe in God,” effectively. Then every—what happens is there are no incremental believers, and then it’s over.

I do think the models will get better. I have to ask myself, though: What if the models didn’t improve at the rate they’ve been improving? What if it really chills out? Do you think it’s enough to change our lives forever? That’s my question, and I think yes.

Yeah, I’d say probably. The genie is out of the bottle.

Just to wrap this up, InferenceMAX shows you that maybe models aren’t getting smarter, although we do have evaluations, but they are getting cheaper. Even if they never got any smarter from today, diffusion would be much more widespread because they’re getting cheaper, faster, and more performant. So, if you want to take a deep dive into all this stuff, everyone listening should check out the new Inference Max article and also go to the site, play around—it’s all free, with beautiful visualizations. If you have recommendations, go to the GitHub, go to the issues, and make an issue recommendation. But check it out for yourself. It’s super cool.

Yes. Appreciate it. Thanks, guys, for coming on. Nice job, Cam.

Yeah. All right. See you guys.