[LIVE] Anthropic Distillation & How Models Cheat (SWE-Bench Dead) | Nathan Lambert & Sebastian Raschka

Alessio Fanelli

Thanks for coming to Sail Live number six. This is a very exciting one. I think the topics are always fun with these, whatever the topic of the day is for our little rat-racing minds trying to keep up with AI. We're welcoming the latest writer joining the Sail Coalition, so I think this just means more content for Sail. I've been a fan of swyx and a friend for a while at this point, so I'm very happy to have his content join this. I think you've been doing great stuff recently and continuing to evolve this, so welcome to the team.

swyx

Thank you, sir.

I just think this is my friends and colleagues in the AI media space, and it's great to be able to support people and keep that network closer. Welcome.

Yeah. Well, thanks for joining us. It's a real pleasure to have you on here, Shawn, or swyx. I coincidentally listened to your podcast about the SWE-bench benchmark. Small world. Awesome to have you here.

swyx

Thanks for having me. I'm glad to be on and chat. I've never done one of these Substack Live things, so I'm curious how it works. I always think about Substack as a newsletter platform, but they want to go multimedia.

I think the live thing, before we get to technical content, is actually good because it gives it a different edge. It's just a little bit sharper when you know you're live. I think we've all done a lot of podcasts, even podcasts that are unedited and put out later, but I think the live thing is a different element that can be tapped into nicely.

Why don't we just dive into it? We're going to start with distillation. I put “How Models Cheat” at the top so we can talk about benchmarks. I think Anthropic posted this pretty spicy blog post this week. It was essentially detailing how they found distributed distillation—quote-unquote, “attacks”—on their services from prominent Chinese labs. I'm very unsurprised with Anthropic calling it an attack. I think that fits with a lot of their branding.

Anthropic is detailing how they found distributed accounts across multiple Chinese labs building shaded LLMs. They described what they were doing and why Anthropic is concerned about this in its worldview of AI geopolitics. I think it's very interesting because I'm of the opinion that the Chinese labs obviously should do this. They're at a massive GPU shortage, and using APIs is much easier than generating synthetic data on their own.

Mm-hmm.

I think there's a lot of—

Nathan, if I may interrupt you here, maybe we should, just for the general audience, define distillation before we dive into the details.

Yeah.

Distillation is a broader concept. It's not a new concept that came up with LLMs; it's an older concept in machine learning in general. The idea is that you have a larger model, let it generate outputs, and train a smaller model on those outputs. The idea is that you can train the smaller model more efficiently using the larger model.

Originally, what you would do is train on the logits. Old-school machine-learning people might remember that, from deep neural networks: the logits, the outputs of the last layer, are what you usually work with to compute the loss function, the cross-entropy term. You would train on this signal.

Nowadays, in the context of LLMs, it's a bit looser, so it doesn't have to be logits that you train on. It could just be output data—synthetic data, like Nathan just said. For example, it's actually a very common practice. In DeepSeek-R1, as described in the paper, and in other companies, they would train the flagship model, the largest model, the R1 model with 6.71 billion parameters. Then they would create smaller variants—I forgot the numbers, but 1 billion and 3 billion models, in a smaller range, like these models you can run locally—and train them on the outputs of their own larger models.

This is very common practice. Everyone does that when they're producing smaller model variants. The question Nathan brought up is, what happens if you're a company and you generate this synthetic data from another company's LLM, then train your own model on it? Sorry, that was just a little interruption. Distillation, in short, is training a smaller model on the outputs of a larger model.

Yeah. I think this is even possible at the frontier. People distill from something like Claude Opus to build clones on it. They're generally doing very similar things internally. They have different access to tools and richer tools.

All of these large labs have had terms of service for years saying that you effectively cannot use the outputs from these APIs to train something like a competitive AI model. They're vague terms. Terms of service are not a contract, essentially; you're using a service, and if the provider finds that you violate the terms, they can cut off your access. That's just a basic thing.

These terms have not been enforced much within the US. I think there was one case, maybe ByteDance a year or two ago, where OpenAI cut off their API access. But this was discussed so much right after ChatGPT, when people were building the first open models, like Alpaca and things. People wondered, “Is OpenAI going to come after us for doing these research models?” It totally died down. People were worried about this for over a year; it was kind of an inseparable discussion. So nothing really happened, and this is the first prominent reemergence of the discussion. I think it's because people are far more worried about AI competitiveness.

Mm-hmm. Yeah, yeah.

I'm curious what you guys think.

Can we talk for a second about how they would even detect that? You said in the beginning something about a distillation attack. You didn't say that specifically, but you implicitly put quotation marks around “attack.”

How would you even detect that? Distillation in that context means literally just letting ChatGPT or Claude generate synthetic data, then collecting that synthetic data and training your own model with supervised learning—supervised fine-tuning on it. But how would a company know that this is a distillation attack versus just an evaluation? Right now, I'm actually distilling for chapter 8 of my book, but I'm doing it with open-weight models, so no worries, Anthropic. Please don't worry about it.

I distill from API models for my job.

Yeah. I use OpenRouter right now and just distill from the DeepSeek V3.2 model, which I think these folks are okay with.

What I wanted to say is that when I'm evaluating models, I use basically almost the same script. When you're evaluating a model, you have the question and let the model generate the answer, right? You generate the response to your benchmark question. In my benchmarks, I have data sets from MATH-500—500 examples—and I have a bigger MATH data set of 12,000 examples. You're basically just running an API in a loop to let it generate these questions and—sorry, the answers.

But how would a company know that someone is just evaluating versus saving that data and training their own model? You see what I'm saying? It's the same process.

I think it's a scale thing. When you're evaluating, at least with the basic evaluations, you're going to do it once and not keep doing it. There's some amount where you're not going to—I think most of it is quantity, and then they're going to look at patterns across similar accounts. That's what they're saying here.

Sure, sure.

I think they're going to see really repetitive stuff.

Yes. I think the interesting point this leads to is that you can do evaluation at a large scale. If you're a big company and you want to know whether your LLM performs well, you have a large suite of benchmarks you're going to run.

You said maybe they're looking for patterns. One way would be, “Okay, this is a familiar question. It comes up in the benchmark, so this person is maybe not stealing our answers; they're just using it for benchmark purposes.” But then it means they're looking at what you're generating there, which is, of course, nothing is private when you're using LLMs on the internet.

Alessio Fanelli

The data is somewhere, you know, intermediately stored, but then it almost implies that they are checking what you use the LLM for and what you generate, which is kind of a sensitive topic, almost privacy-wise, right? That's an interesting point because, of course, you mentioned the terms of service, which say that you are not allowed to distill, but you're not distilling.

The point I'm trying to make is that you're not distilling live when you're on the platform. You're doing it somewhere later. You're just letting the LLM generate answers, and I find it kind of interesting that a company would look at that, even at the scale, and call you out like, “Hey, you're generating too many answers here. That's not cool,” or something. That's kind of a weird thing.

Yeah. I wanted to respond to something a couple of sentences back, but actually, Anthropic blocked US companies first, before the Chinese companies. It has blocked both OpenAI and xAI from using the models, and I think it maybe explicitly accused xAI of distilling stuff. I don't know, but definitely not in a full blog post like this.

So this is definitely the most high-profile case. I do think it's actually pretty hard to distinguish from, “Hey, I'm just running my internal benchmark, man.” Of course, it's going to be a very high volume of a lot of the same stuff because, especially with some benchmarks, you have to run them 3–5 times—the exact same questions, right?

Mm.

I do think that once you get into the tens of thousands or hundreds of thousands, then you're like, “Okay, you're not just running benchmarks. You're distilling this thing.”

There is a good point in the chat: how would the distribution of questions look if you were distilling? Related to your point, at a certain point, when you have a certain magnitude of answers generated, it might look suspicious.

But there are a lot of legitimate use cases. If a company uses, let's say, OpenAI or Claude API as its own chatbot and has a lot of customers, naturally a lot of answers are generated. They would probably look at distributions. Maybe you would expect a very broad distribution when you are distilling because you want to cover pretty much everything, whereas when you are running benchmarks, it's maybe more specific.

If you're running a math benchmark, it's just math. If you have a customer chatbot, it's more like customer answers. I think they would maybe analyze your distribution. I feel like this is kind of a weird thing to do. I don't know. If you're a company and you're looking into your customer privacy—the data generated—you have to expect that it's not private, of course, but it's still kind of weird that they essentially do that.

swyx

Okay, what else do we have to talk about? I think this is interesting. One thing I did was throw it into Nano Banana, which is kind of a decent visual.

Throw it into Nano Banana 2. It's a Nano Banana 2 live pod. It just released 5 minutes ago.

This is actually Nano Banana 2. Because I'm in the early-access program, they cut you over to the new Nano Banana, and then I couldn't access the old one. I was trying to do a diff, but I couldn't because I only had access to the new Nano Banana.

Classic. That is classic early-tester-program shit. Look at the pain we have to deal with here.

Is it interesting that DeepSeek used so much less than MiniMax? I think, Nathan, in your write-up, you had a little bit of a comment about that.

This is a political blog post in a way. Maybe not political, but they're trying to make a point that's more about making a point than the details. The DeepSeek thing is definitely way smaller-scale.

Okay.

I think most of the labs will experiment with all the APIs they can get access to. Data is just so important, and you're going to have a pipeline where you could sub in any API and then run an ablation to see if it gives you performance.

Yeah.

The API is kind of free. Just do it.

Yeah.

The millions of exchanges are a bit more of a bet. You can measure that over a longer period, and it takes a lot longer to get the data out of the API, especially when you have to spread it across a ton of accounts. These accounts are all rate-limited and have other problems. That takes longer, but this tiny one is so fast.

That was generally my point: it made it clear that Anthropic is trying to use the DeepSeek name as the only Chinese AI name that people in the US know.

Mm. Marketing-wise, to make it stick.

Actually, you mentioned the different APIs. I'm not sponsored by them and have no affiliation—I’ve never talked to anyone from that company—but OpenRouter, for example, is a good example. I've been using it a lot for the open-weight models because the bigger ones are too big to run locally.

What's nice is that they also offer this: it's basically just routing you through other companies' APIs, and they automatically select which one is cheapest at that point. I sometimes get some failures. I think when it switches, it might sometimes crash, but maybe that's something I have to fix in my script.

Even then, if you're distilling, you can do that from multiple providers. Of course, if you want something from ChatGPT or Claude, it's always going to go through the official one, and then it gets, I guess, suspicious. But you could also technically distill a bit through OpenRouter, through their account rather than your direct account. You can make multiple accounts, and yeah, it's kind of interesting that they track all that.

Different topic now that you called out that they call out DeepSeek, which is quite interesting.

swyx

For what it's worth, OpenRouter seems not to be using DeepSeek in most of these.

These are free models. DeepSeek isn't free.

Yeah.

I see. Yeah. I'm using the paid API, I should also say. It's also nice that they show you how much it costs and the tokens per second for different providers.

If you go to the search at the top, you can look at the different DeepSeek models. This one is an older model, so maybe it only has 1 provider. But if you go to DeepSeek R1 or something, or even the normal DeepSeek V3.2, there should be multiple providers. If you scroll down, you can see different providers, different tokens per second, and different costs.

I just like that website because it's quick to use the API, and they have an OpenAI-like API. It's not sponsored or anything; I just find it generally useful. But, yeah, just a side note.

Do you want to go back to the comparison? Did you have a high-level point to make there, swyx?

swyx

Okay, just a couple. First, I think the timing—after Moonshot released their stuff, after MiniMax released their stuff, but before DeepSeek V4—was strategic. I think that may also have factored into why MiniMax was detected more, or had a higher number.

When you collect data is very important. They found MiniMax during the training of MiniMax 2.5, which we will confirm later if we do end up doing the call with them. Obviously, the number is going to be very high because they were actively looking for it, and then they banned the MiniMax accounts and MiniMax changed its behavior.

Actually, I don't think that's exactly what happened. Sorry, let me correct myself. While MiniMax was distilling, Anthropic released Opus 4.6, and they said that they redirected nearly half their traffic. So I'm like, “Okay, very clearly, this is them, right?” This is the exact same traffic switched to a new model the moment a new model was released.

DeepSeek maybe wasn't doing that because they hadn't been working on their stuff actively. I don't know. It could be a different thing. Or DeepSeek is just way more efficient: I get all I need from 150K. You guys are so inefficient.

Yeah.

You guys—

It would be so interesting if we knew the timeframe of this. Are all of these API requests within the last 4 weeks? Are they within the last 6 months? That's such a different nature of what is going on.

Exactly. That's what I'm saying. DeepSeek was training DeepSeek V3.1 and V3.2 about a year ago.

Yeah. Or, I don't know, DeepSeek OCR. I guess they said what it is, but it's not that.

Alessio Fanelli

Yeah.

But, scale-wise, I do think MiniMax is 3 times smaller. It's just a faster model. They don't use MLA, and they don't use DeepSeek Sparse Attention. I think it's just grouped-query attention, but it is still a pretty snappy model, so I think it's attractive to use.

And the other one, off the top of my head, I don't know—maybe they had some free tier or something like that. When the models come out, they sometimes offer free usage, and that was a more recent model than DeepSeek. I think the last one was from December, the V3.2.

Yeah. So maybe this is an irrelevant point because they were training before, and before would have the same amount of traffic, or they're just way more efficient, right? It does bring to mind, okay—

The efficiency thing is not it. I can guarantee it. There's a small chance that they got the right research idea early and found the right data to use, but it's not that they're going to be 10× or 3× more efficient.

Okay. So it's a timing thing, or they just don't actually use it that much. I mean, you play this out: I was like, "Okay, why don't they share? They're all buddies, right? What?" And it does come to a point where, okay, let's have all of China distribute it to every citizen, and then—

I can talk about this a little bit. There's not a lot of research, but there are a few research projects trying to understand how you use distillation data. I think SFT is the cleanest example, where you're doing this autoregressive loss on Q&A pairs.

But the strongest model is not necessarily the best teacher, and most of us in this area think it's because you have to match the probabilities of the tokens to the base model. What's happening is that Qwen dense models are the best teachers for a lot of open-weight models, and I think that's because a lot of open-weight models are either Qwen or have been Qwen-like for a while. So OLMo learned really well from Qwen, and obviously other Qwen models did.

But scaling these pipelines up to use, say, GLM-4.7, a bigger DeepSeek model, or a more recent big Qwen MoE—it's a lot harder to just generate the data from the same prompts with the right sampling settings, then do SFT on them, and actually make the numbers go up.

Interestingly, GPT-OSS is a pretty good teacher. But there's a huge gap there: just because you have this data does not mean it's actually going to make your model better. So you have to do the research to be like, "Oh, we learned that we get signal out of Claude. We need to get 100 billion tokens ASAP because it's going to just immediately make our model better."

That's not a common place to be in modeling because of this weird teacher-student dynamic going on. So I could see that being different across labs.

Yeah, I think it also has something to do with that. I noticed that if you're distilling the smaller model from the same model family, it performs better. And I think it's to your point that if you have a very, very strong model, it might also be too different, or the style might be too different, and then it's too much of a leap for your model to adapt. It's too different from the Q&A answers during pre-training, and so—

Ah.

You make a bigger leap. Another thing I wanted to say: you mentioned OLMo, and it's been a while since I read the paper, but you might know way better than I do. I think you also trained on the logits, and maybe—

We didn't do technical distillation. We just took the tokens.

Oh, I see. Okay, then that was probably a different paper. I think Google does that for their Gemma models.

swyx

Yeah, they do.

Because there's also the distinction: you can only do that for open-weight models. If you do that for Claude or OpenAI, it would not work with the logits because they don't provide them. They only provide them for some tokens, like the top 100 or 1,000 tokens.

And so, in a sense, if you want to do the real, in quotation marks, "distillation," it's even easier to do that from open-weight models because you can control it. But then, like you said, we need 100 billion tokens ASAP. That's not an easy thing to do because even at 40 tokens per second or something for these large language models, when you generate answers, getting that many billions of tokens takes time, right?

So it's almost easier to start distilling from a medium model. It's the question of more data versus higher-quality data, right? It's also a sweet spot, an experiment in itself, in an ablation study, right?

Yeah. I like that Nathan had to call it technical distillation because it is no longer the default, even though it was the first.

Also, another fun fact: I did my Jeff Dean interview recently, and I tried to get it out of him, but he sort of dodged it a little bit. Remember, there were actually 3 sizes of Gemini models? There were Nano, Pro, and Ultra, and I was like, "Where is Ultra?"

They keep it in a basement and distill from it, right?

Interesting. Yeah. Maybe it's also to safeguard yourself so no one can make a copy, or for the price too, but probably both.

Yeah. I mean, I think this is how I always think of it: the model you deploy is never the model you train.

Mm-hmm.

Because you train the dense and then you deploy the MoE, right?

Mm.

You basically always do it.

Mm.

Say more. Do you think they're really distilling from dense models?

I mean, I think that is the full thing. When you have unlimited resources, don't care about inference, just care about maxing intelligence—why not?

Yeah. I'm not 100% sure. I think the MoEs just give you a FLOP. I don't know if that's actually how I think of the gains of MoE when you have a really good MoE architecture. But I do think that they have bigger models that they distill from.

And they train internal models differently from external ones because the external models have been getting a lot smaller, which is kind of a weird thing. We don't have a good way to measure it. Maybe Dylan will reverse-engineer it in Inference Max, whatever the heck.

But I'm always suspicious of these things. It's really a capacity thing too: how many people use the model at the same time, the hardware, how much is allocated. It's a rule of thumb, but it's really tricky, I think. It's really hard to say anything from these numbers.

I do think they might start restricting models that will only be in products and not be in the API. I think the whole API business is brutally competitive, and I don't have a good sense of what its defensibility is.

I think it makes sense for something like Google, Azure, or any existing cloud businesses to have APIs, so that's kind of a more natural transition. But the Anthropic and OpenAI API—the transition from their products, which are their big differentiation, whether it's ChatGPT, Claude Code, or Codex—you don't get people to go use the API from that.

And I think you get a lot of people who are already spending on cloud who then go to use the APIs, which is why Lambda and Nebius are going to have these API products. But isn't it that if Claude's really worried about distillation, they should put the model release in Claude Code ASAP and then just not bother with the API?

I don't know when that'll happen, but it could.

I do think, though, it's a big customer base, the API customer base. Any type of product that's built with LLMs, like customer-chatbot types of things. But more generally, I do think the problem is—I don't know exactly how the plans work in Claude, but you would reach a token max where you can only get so much with your subscription. You can, I think, buy more tokens, but I think it's just easier with the API at a certain scale.

Also, the whole OpenClaw customer base, right? They don’t allow the plan anymore in the OpenClaw context, so you have to use the API. Given how many tokens OpenClaw generates, it’s actually not a bad business if you don’t lose money on these tokens. If you sell it at a nonsubsidized price, I do think the API is actually not a bad business model.

Yeah.

Do you want to take a side? Do you want to try to break the tie? I’m obviously being provocative.

Yeah.

I don’t really know.

Yeah, yeah.

But I could see it. Anthropic gives me Apple vibes.

I mean, Anthropic has a higher chance of doing this, yes. OpenAI, just because I’ve talked to the people so much, I just don’t super believe that they will have locked models to products, only out of, I guess, idealism and sort of principles rather than economic incentive. Economic incentive would agree with you that they should have private models for products.

Recently, they’ve done this, right? The last 3 GPT-5s all had Codex variants that were 2 to 4 weeks ahead, released only inside Codex rather than as an API.

Hmm.

But, just constitutionally, I don’t think the people who run these things believe in locking things behind APIs because they have such a huge market anyway. They kind of don’t care, and if you’re genuinely sort of a zealot—if you’re not trying to maximize the value of your company and are genuinely just trying to spread AGI everywhere—then you release the API because you just don’t know what people are going to build with it.

Yeah.

One more thing, though, with the Codex thing. We’ll have to see next time, because I think this time it might also be a bit biased toward releasing it in Codex because they almost released it simultaneously with the app that they want to promote at the moment. So it could have been more like they did that so that anyone checks out the app. But we’ll see next time.

Yeah. But there’s always a 2- to 4-week exclusive window.

Yeah.

And that’s their right.

Yeah, sure.

If you want to promote Codex, it’s pretty effective.

Yep.

We have a bunch of questions in the chat about other topics. Do we want to cover benchmarks and then this thing, or—

Go right ahead.

What do you want to do? It’s your Substack. I don’t know.

Oh, no. Oh, man. It’s a collective.

It’s a collective.

You should just dive into what you’re interested in. Go for it.

Sebastian was interested in the SWE-bench stuff. This past week, SWE-bench Verified died—officially.

Wait. What do you mean by this?

Yeah, let’s define SWE-bench first, maybe.

Okay. I happen to have the post on this. Let me remember.

So the broader topic, the umbrella topic here, is how we compare which LLM is currently the best LLM. One of the ways would be SWE-bench, basically. But I’ll let you explain, because you had this brilliant podcast—or article.

Where do you want me to start? Should we just define SWE-bench, I guess?

I guess, yeah.

All right.

So basically, it is a coding benchmark.

Yes.

SWE-bench is a popular way to compare the capabilities of LLMs, and then there is SWE-bench Verified. Maybe you should talk a bit more about SWE-bench first.

SWE-bench was a paper out of Princeton, from Ofir Press’s group. They do a lot of good code-benchmarking work. They drew thousands of example open-source issues and the pull requests that closed those issues. There’s a bit of selection bias here because they only focus on popular open source, and only a small number of popular open-source projects, but a large number of issues from those projects.

They dredged up some passing tests and some failing tests that you need to make pass in order to get the score. When it launched, it was kind of obscure. Devin was actually the first one to pick it as a benchmark to report. I think at launch it was 13%, and now everyone’s at 80% or something like that. SWE-bench, because it was done on a student budget, was very—let’s call it—sloppy.

Terminal-Bench is like this now, too. They’re just aggregating. It’s hard to do a benchmark that is well-calibrated across topics and difficulty.

Yeah. It is hard. For the small group that is watching, I’m actually working on it with Cognition to launch a new benchmark here.

OpenAI was like, “Okay, guys, SWE-bench is taking off. We’re going to adopt this, but we refuse to abide by the full SWE-bench. We’re just going to curate a 500-example subset of the original SWE-bench.” They actually hired humans to vet each task. I think it’s somewhere inside this blog post, but basically, they hired 3 humans for every task to vet whether the task was high quality or not, because there’s a lot of slop in there. They said, “Okay, this is the 500 that we’re going to endorse.”

So it’s a curated subset of SWE-bench with 500, let’s say, challenging problems that are supposedly well-defined.

Yeah. What’s really funny is that at launch—this was launched in 2024—OpenAI could not run all 500 of its own tasks. For a while, there were a few releases from OpenAI that reported on a subset of the subset because they couldn’t run it on their eval infrastructure. Their numbers were higher because their denominator was lower, which is very funny.

Maybe in that context, we should say what SWE-bench looks like. I think it’s basically code that has bugs in it, and usually the task for the LLM is to fix the bug in the code, right?

It’s right here. The whole thing’s open, which becomes a problem in the future. But right now, you can see the whole thing, right? You can see the reports from the issue ID and the problem statements, and then you also have the tests that you’re supposed to pass and fail. It’s all here on Hugging Face, and you can see that it says 500. Anyway, I think we don’t need to get too lost in the details.

I just wanted to define the context: this is a coding benchmark with essentially 500 examples that are available on the internet.

Yes. If you want a bit more historical context, this is a step up from HumanEval, which is more about completions, right? This was, in my mind, the first proper agentic benchmark, I guess, apart from τ-bench, where they give you the problem and the end result, and they don’t really specify how you’re supposed to get there.

A lot of previous benchmarks, like MMLU and HumanEval—which, in the coding domain, was also released by OpenAI—were very much, “Here’s the problem statement, and then give me the right answer immediately after,” without that many extra files or anything that you’re supposed to run. The other ones are more autocomplete; this one is more agentic. It’s all a spectrum, obviously, because you can use agents to solve autocomplete, but that’s not what HumanEval was testing.

Anyway, I wanted to make sure people understand that OpenAI actually invested a lot of money and effort into making SWE-bench Verified.

How much money do you think this cost?

Oh my God, don’t do this to me.

Millions. I would guess on the order of a couple of million. It could even be a few million.

I’d say a couple million. Basically, you do the first filter pass, and then it’s 500 times 3 because they had 3—

Per person?

Individuals.

Yeah.

Yeah. 3 people per task, and then maybe a couple more verification passes or whatever, right? So then they were like, “Oh, well, not only is it saturated because everyone just takes turns incrementing by 0.1 every time they release a new model.” It’s bullshit. It’s obviously bullshit.

The inherent noise in just running these models varies by 0.5 to 1 every time you run it. You just choose the highest one every time.

Mm.

A little nitpick. I don’t think it can be 0.1%, because—

Sure, sure.

Is it 500?

Because what you said before—it’s 500 examples. I think the smallest increment is 0.2%, if I—

Okay.

But it’s a little detail. Yeah, sorry.

A little.

Yeah.

I think as we progress to the next era of benchmarking, the N—so the N here is 500, right?—doesn’t directly correlate to the percentage points because you get sub-points as well from the rubric.

Ah, yeah. Good point. Good point.

So it’s like Terminal-Bench. Even though it has 90-something tasks, you can get subdivisions less than 1%.

Anyway, not only do they have this, they actually audited their own benchmark. They were like, “Okay, how come everyone is saturating at 80%? What’s up with the remaining 20%? How come everyone is failing at it?” And they were like, “Oh, actually, we looked. We paid even more people—6 people per task now—with an extra team if any sort of positive identification is found.”

They found that 59% of them cannot even be solved at all because the original benchmark was still slop. Stuff got through that was not solvable. I actually tried to illustrate this in my post.

Here’s an impossible test, right? This is an example of a SWE-bench Verified task that passed the first round of human verification. Here’s the task: we want to implement Python type hints or something, and we want to see the expected behavior. I want to see a string in the output.

If you were given this, you would never pass it because the test said, “I am looking for something called `get_annotation`, and if you don’t give me this magic string, `get_annotation`, you will fail this test.”

Why?

Yeah, it’s way too—

It’s way too specific.

It’s like a coding interview.

Yep, yep. Yeah.

This is just a bad task that somehow escaped validation in the first run of verification.

So the only way—

You could kind of solve it is if you’re memorizing the answer.

Yeah, exactly. Exactly. I think every benchmark should include stuff like this.

Like a honeypot.

If you solve this, you’re like, “Oh, shit.” This is a canary, right? I mean, you’re definitely cheating.

Like a sanity check. Yeah, yeah, yeah.

Yeah, yeah.

That’s actually a really nice point, yeah.

I just think it’s a beautiful illustration of how hard it is to make evals. There were multiple rounds: there was the original SWE-bench, where the Princeton kids did the initial first pass. Then there was a second pass, with OpenAI doing SWE-bench Verified, and every single person who ran SWE-bench Verified for the next 1.5 years did not call this out—until OpenAI was like, “Hey, let’s look at the data.”

I think it’s really interesting. While they were looking at this, they had a second thing. They looked at the chain of thought, and inside the chain of thought, they found GPT-5’s own chain of thought starting to include information from the future.

Because the problem is open source, and because the model was trained on information from GitHub, it would use advanced knowledge of future versions of the Django version they were using to solve the problem. It knew how to—

Mm.

—

Definitely seen stuff like this in the real world, where the models will hallucinate the new version of the API even if your script isn’t on it. I think a lot of the Hugging Face stuff is the worst with this, where the models just produce total gobbledygook. They’ve seen all the versions, and the API has changed too much over time, so they just fucking throw something out there.

Mm. Yep.

Yeah. I mean, I think there’s a lot of this, right? There’s the ethical question: you can blame things like, “Oh, you should not have released the full dataset in public, because obviously people can train on a full dataset.” But it’s not like the researchers are trying to do this.

Because these things are also open source, any dataset that touches GitHub—any training corpus that touches GitHub—is eventually going to absorb this.

Yeah, yeah. And it’s not even this website or the repository directly. It’s a clone of this repository, or someone else who has that develops their own open-source library and has it in the unit tests or something. It’s not even intentional or malicious or anything; by accident, you’ve already absorbed that.

Yes.

Yeah.

Or a new feature that releases this—an edit-only feature—it gets written up in a blog post or a conference talk or something, and then it just makes it in, right? It’s really funny.

To me, OpenAI could have stopped there and said, “Okay, we’re done.” They did one more extra thing, which is kind of funny. They also then ran Gemini Flash and Claude Opus, and in this case it was even more egregious.

They just gave the task ID and said, “Repeat the SWE-bench task to me.” From the task ID, the model could just vomit out the whole problem statement and the solution.

Mm.

—

These are crazy. The stuff that’s in these models when you zoom in deep is really incredible, because these are models that are really well done, but there’s just so much complexity in all the pieces of the puzzle that get put in the recipe.

There are just so many weird corners.

Yes.

I also still find it fascinating that, of course, it’s kind of by design when you’re training that you memorize things, because that’s literally next-token prediction. But given how big a model is and how much data it sees—and usually it sees the data only once—it still has enough capacity to memorize.

Usually I would think, “Okay, I would have to train multiple epochs to be able to memorize,” but no. It may be enough to include it once or twice in the training corpus, and it can produce a perfect rendition or a perfect recap of what’s in there. It’s kind of fascinating, even for people who don’t want that. It’s crazy.

Yeah. Labs got good at this. There’s essentially a duplication level that you need at each stage of training, and it’s not easy to measure.

If you do too much at pretraining, your model forgets basic facts. At post-training, it’s probably closer to these abilities, and I think that’s something that is not well reflected in evals of your knowledge tank.

Yeah, I mean—

But I think this is an art that they have probably gotten good at.

Yeah. Continued pretraining does also require some revisiting of old data; otherwise, like you said, you have forgetting. But it’s still fascinating to me that with such a small fraction—because you usually use 1% or 2.5% for continued pretraining—it’s enough to have the model memorize almost everything. That’s fascinating.

Yeah.

I don’t know. It’s still fascinating after all these years.

I think there’s—one of the pet topics that I pursue 2 or 3 times a year in my work is the information theory of LLMs. I still think it’s super understudied. How come you can memorize from one pass?

Mm. Yeah, exactly right.

People also forget superposition, which is Anthropic’s original mech interp work. It basically stuffs information inside smaller bits that then get forgotten. But how does superposition actually work?

I don't think I've seen a convincing study on that.

Yeah.

Okay. Anyway, I don't know. I'm done on my SWE-bench right now. I don't know if you have thoughts or questions or whatever, but I do think this is an example of the model unintentionally cheating, and benchmarks are hard to make. We need new ones. If this happens to SWE-bench Verified—which I think is the most scrutinized benchmark in the world—

**swyx**

In my recent post...

**Nathan Lambert**

What else is there?

**swyx**

I had a bar plot where I showed the SWE-bench Verified numbers for most models, and, like you said, they were all 80-something percent—literally between 81 and 89, let's say—where there's almost zero variation. Even something like MiniMax M2.5, which I do think is worse than GPT-5.2—no offense, it's a smaller model, it's a cheaper model. For my usage—

**Sebastian Raschka**

M2.5 is an open model. It's a little bit worse, but on this particular benchmark, it's the same. I don't think what I'm saying is that M2.5 should get a lower score on SWE-bench, but I think other models should get a higher score. But, like you said, the problems are just impossible to solve.

One point I think we didn't bring up is that we said SWE-bench Verified has issues. So what do we do about it? I think there is a SWE-bench Pro now, which is kind of—

**Nathan Lambert**

Yes.

**Sebastian Raschka**

It's a—I would say Verified tried to fix the regular SWE-bench, and Pro tries to fix Verified. But I haven't looked into this. Is it another subset, or is it a completely different set of problems?

**Nathan Lambert**

Yeah, it's a new set. SWE-bench draws from roughly a 2022–2023 era of problems. There are a few things you do, right? One, you do private/public splits, right?

**Sebastian Raschka**

Mm-hmm.

**Nathan Lambert**

That's super obvious. Two, you update the dates from which you draw, and three, you diversify the repos and the languages, right? These are all very, very basic fixes, and then obviously trying to fix the testing. They're super-basic fixes to the original SWE-bench. It doesn't take a genius to figure out, but they did the hard work and—

**Sebastian Raschka**

But it is, in a sense, also what Verified meant to do. So it's not—let's say people looked at this again, but there's no guarantee that it doesn't also still have issues that might be discovered later on, right? I mean, it's—

**Nathan Lambert**

No. So SWE-bench Verified was an intentional subset, right? These guys were like, “No, no, no. We need to have a superset.” Not even a superset.

**Sebastian Raschka**

A different. Yeah, yeah.

**Nathan Lambert**

Yeah.

**Sebastian Raschka**

But what I was trying to say is, when SWE-bench Verified was developed, there were 3 people per task making sure the task was well-defined and everything.

**Nathan Lambert**

Mm-hmm. Mm-hmm.

**Sebastian Raschka**

But then 2 years later, it turns out, no, no, this was not the case for everything.

**Nathan Lambert**

Yeah.

**Sebastian Raschka**

And what I'm trying to say is, it could be that SWE-bench Pro is better, but it might still have issues that might not be obvious right now. Maybe in 1 to 2 years, once we revisit this and see some of the failure cases, maybe we'll discover, okay, this still has some issues. So it's not a guaranteed perfect set, is what I'm saying. I don't know, but it's just a suspicion here.

**Nathan Lambert**

Totally, totally. I do think Scale AI has a professional interest in making sure this is good.

**Sebastian Raschka**

Yeah, no, but what I was trying to say is SWE-bench Verified also had a professional interest in making sure, just by accident.

**Alessio Fanelli**

Oh, very different incentives. I guess they all have very different incentives.

**Nathan Lambert**

This one has an unlimited budget. This one has basically an unlimited budget because it's literally existential to Scale AI that they have good data.

**Sebastian Raschka**

Sure.

**Nathan Lambert**

But I also think it's really nice that OpenAI's evals team keeps endorsing Opus. It is kind of funny. So, yeah, they deprecate SWE-bench Verified, and then they were like, “We're gonna report SWE-bench Pro now.” And GPT-5 is, you know, number 1, 2, 3.

**Sebastian Raschka**

Maybe—do you know, if I wanted to evaluate on the private dataset, how would I do that? Is there an API call I have to make against Scale AI, or—

**swyx**

I don't know.

**Nathan Lambert**

You get an API key and have to agree, because if you don't have an agreement, then you just have to keep the data. You have to jump through special hoops to make sure that you don't steal the private eval.

**Sebastian Raschka**

Mm-hmm. Yeah, my question was basically: do they even let you download the data, or is it more like you send the answer to them and they do the evaluation on their backend, so that you don't even get to download the data? Because otherwise, like you said, you could—

**Nathan Lambert**

Scale AI.

**Sebastian Raschka**

Yeah, yeah. So basically, you only provide the answer. You have your LLM generate an answer, and you submit the answers, and then they have some process to evaluate on their end, so that their private data never leaves their server, is my guess, because otherwise someone might upload it or something.

**Nathan Lambert**

Yeah, I don't know. I haven't tried it, so I don't really know. I'm sure you can reach out to them to figure it out. Yeah. Anyway.

**Alessio Fanelli**

I think this is good unless people have more comments that they want to add.

**Nathan Lambert**

Yeah. But this is only coding, right? Every other domain needs this.

**Alessio Fanelli**

But coding is the hot thing right now. Yeah.

**Nathan Lambert**

The frontier is even more expensive.

**swyx**

I think the frontier evals are even more expensive, which is like the APEX eval from Mercor. Evals are going to cost millions. They're going to cost tens of millions and hundreds of millions of dollars at the frontier, which is just a very strange dynamic. Whereas so much of the ecosystem is forking between frontier models and then research and other things, and trying to follow that dynamic and explain it to people is going to take a lot of work.

**Sebastian Raschka**

But, yeah, coding is—I do think—really interesting because that's what most people use LLMs for these days, but it is also easier to evaluate. I think once you leave coding and math, it becomes a bit obscure. How do you measure the quality of the answer you get back? You get back to, let's say, preferences, I guess, which is more of a subjective thing, whereas coding is more objective. So it's not a bad thing to do.

I think the other day, though, Anthropic acquired another company that does UI-type stuff on the computer, and I think that is something where—

**swyx**

Minor. Doesn't really matter.

**Sebastian Raschka**

No, I mean, that's—I'm not trying to say this is a big thing to talk about. What I'm trying to say is, this is another interesting point for evaluating LLMs on those tasks, because I think a lot of people want an LLM to control the computer and do various things, but those are harder to measure. So maybe in 2 years we'll have something more like benchmarks for that. It's harder to specify. It's kind of like—what is it called? In programming, there's unit testing and then system testing, basically, like UI testing and stuff like that.

**Nathan Lambert**

Yes.

**Sebastian Raschka**

Yeah, so I think that is the next—

**swyx**

Maybe it's going to be the next big thing.

**Sebastian Raschka**

Basically, that thing. Yeah.

**swyx**

Yeah, yeah.

**Sebastian Raschka**

GDPval is usually the thing that gets brought up here. So I'll just leave it there. I think we've sort of beaten the—

**swyx**

Yep.

**Sebastian Raschka**

The dead horse.

**swyx**

Yeah, the benchmarks. Yeah.

**Sebastian Raschka**

Benchmarking. But definitely GDPval is sort of here. I'll put it that way. Okay.

**swyx**

Yeah, but the big topics are essentially distillation and the benchmarks this week, yeah.

**Sebastian Raschka**

Cool.

**swyx**

And welcoming Max to our coalition of—

**Sebastian Raschka**

Yeah.

**swyx**

GDPvals, whatever that means formally.

**Sebastian Raschka**

It just means I get to hang out with you guys, which is what I want anyway.

**swyx**

I mean, it's ultimately a media vehicle, and brands and vehicles for media are actually very influential today. I think you see many companies investing in that, and I think it's important to have people that you respect and are aligned with able to amplify each other.

**Sebastian Raschka**

Yeah. It's also nice to talk to humans because I noticed the last couple of weeks, if you go to social media, I think 50% are lobsters, like OpenClaw clients nowadays. I get a lot of emails, but also notifications or responses that look AI-generated, so it's nice to have this human connection and actually talk to an expert about things, yeah.

**Sebastian Raschka**

Cool. There are a bunch of comments. I don't know if you want to do quick hits, or if you're kind of checked out.

**swyx**

I have to go to a meeting.

**Sebastian Raschka**

Okay.

**swyx**

That's why I'm trying to wrap this up.

**Sebastian Raschka**

I see. Okay. Well, you know, time is yours. What do you want to do?

**swyx**

Yeah. Okay. Thanks, everybody. We'll see you next week.

**Sebastian Raschka**

Yeah. Thanks, everyone, for joining. It was a nice spontaneous discussion, I guess. It always feels nice to talk about things, and it's too bad we didn't get to discuss these chat questions because I probably need glasses at some point. My screen is pretty far away, and I can just barely read them.

But yeah, thanks, everyone, for commenting. It's nice to see so many people excited about these topics.

**swyx**

Peace out. Hopefully, see you later.

**Sebastian Raschka**

Yeah. Have a good rest of the day.

**swyx**

Bye.

**Sebastian Raschka**

Bye.