The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals

Okay. Hi. We're here in the OpenAI studio with Mia and Olivia from the Frontier Evals team. However you want to introduce yourselves, maybe name what you do at OpenAI, and we can get started.

Sure. Hi, I'm Olivia. I'm on the Frontier Evals team.

Are you sure? Great.

Hi, I'm Mia. I am a VP of Research at OpenAI. My teams are the Codex team, the Human Data team, and the Alignment team, and we work a lot with Olivia's team on Frontier Evals.

Very exciting. As I understand it, you were part of the original team that worked on SWE-bench Verified as well.

Yeah. Olivia's team, the Frontier Evals team, and the Human Data team collaborated on creating SWE-bench Verified. You've seen the evolution of coding benchmarks over time. What's the blog post that you've worked on and that we're releasing today? What's the main thesis that you're putting forward?

The main thesis is that SWE-bench Verified has been one of the North Star coding benchmarks that the field has looked at to measure coding progress. Recently, we've seen that progress kind of stall, and we realized that this is because the evaluation is effectively saturated and also highly contaminated. At this point, we don't think it's measuring improvements in coding performance very well anymore, and we think the field should move away from it toward other benchmarks.

Like SWE-bench Pro.

Like SWE-bench Pro.

Amazing. One of the jokes I always have is that there's a group chat with all the labs, and everyone just takes turns incrementing by 0.1 on SWE-bench. Then it's, “Okay, well, you have the best coding model, I guess, because you're 0.1% higher.” It's not super convincing at this point.

Yeah.

I think we should reset on the original work that you did for SWE-bench Verified, which was pretty substantial. It was a very significant investment from OpenAI that people still don't appreciate. What was SWE-bench Verified, and what should people know about it?

SWE-bench Verified was a cleanup of the original academic benchmark from a lab at Princeton called SWE-bench. The agent is basically given a codebase and a task sourced from a real-world repository and GitHub issue, and it's asked to solve the task. It's graded on whether some tests pass.

At the time, it quickly became a popular benchmark because the field didn't really have good real-world coding benchmarks. But when OpenAI took a look at the benchmark as part of one of the evaluations we wanted to track in our Preparedness Framework, people started realizing that some of the cases where agents were failing were due to bad problem setups rather than models simply being incapable.

Folks at OpenAI ran a pretty extensive human-data campaign, hiring almost 100 real-world software engineers to go through the problems and figure out whether the tasks were well specified and whether the tests were actually fair. They created a curated set of about 500 tasks that we thought were much better.

It's hard to overstate the amount of effort it took to create that benchmark. There were many expert software engineers reviewing the problems sequentially, multiple times. Basically, 3 different experts independently reviewed each one.

You didn't have to do that. You just tripled your cost for—

I mean, we had to do it. It's quite a hard task to look at a problem and the patch. It's not just the problem and the patch; you have to understand it in the context of the codebase that the human or model is in to solve the task.

It's a very complex problem, so it was definitely necessary to have 3 reviews. Maybe we should have done more, but it was definitely a lot of effort to get there.

There’s more, but people can read the blog post for that.

I will note that you guys have a trend in verifying benchmarks. I recently saw that Qwen had HLE Verified for humanities, and LiveCodeBench Verified. Now everyone's verifying everything, which is nice and adds extra quality.

The meat of it is that this was a problem statement, a diff, golden tests, and regression tests. That's the rough setup of these 500 problems. Contamination always happens because the measure was fully open. You did have canaries, but things leak through multiple avenues. The problems are sourced from open-source repositories.

So it's not just that, when we usually publish evaluations, we add canary strings to ensure that they are easily filtered out at training time. If you use data from open-source GitHub repositories—

GitHub.

—you don't actually have a canary string. Some of these are also very popular repositories, like the Django repository, so you're going to see many instances being used throughout GitHub.

Yeah. Just before recording, you were telling me that you found this in your own chain of thought for GPT-5.2, where it was seeing that it had extra knowledge or something.

Yes. This was an example where the task asked the agent to implement something, but it wasn't told that there was a specific argument that the test was going to look for. In GPT-5.2's chain of thought, we actually saw instances of the model reasoning, “At some later version of this repository, they implemented this particular argument. Maybe I should add it in.”

This is an example of a test that would be pretty impossible to pass without that contamination knowledge.

I think you found that source, and it triggered a whole investigation, both in our own models and in other frontier models on the market, to understand how contaminated the benchmark is across the industry. What else did you find? I have to double-click on this.

When I say “we,” this is mostly from other folks on our team. We did some analysis on whether the tests were actually fair. This happened by first taking all the problems that o3 couldn't solve reliably and then getting a lot of humans to do another pass, digging into what was wrong.

Was it the same exact analysis, or were they reading o3's output and saying, “Here's where o3 went wrong”?

It wasn't exactly the same work. It was a deeper dive. We asked, “Which are the problems that we don't see any model solving? Is there something fundamentally wrong with those problems, or are the models just not smart enough to solve them?” That's what we dug into.

And you found some.

Yes. In over half of the problems investigated in that deep dive, there was one problem or another. The most common problem was overly narrow tests, where there was some particular implementation detail that the tests were looking for but that wasn't specified in the problem description. It wasn't fair to expect the model to make that particular design choice.

One pretty blatant example is when the task asks you to implement a feature, but the tests look for you to name an argument or function with a particular name. If you chose another reasonable name, the test would fail.

Another type of bad test looks for additional features that were never mentioned in the problem description.

That is significant. It means that if you pass a test, you probably did a really good job. But just because you didn't pass a test doesn't mean that your implementation wasn't a good one. We only accept very narrow versions of solutions, not the whole space of viable and good solutions to the problem.

Yeah.

I think it's important that you're doing this because, in some way, it's you in 2025–2026 going back in time and correcting your own work. You could have caught all this in the original Verified work.

I think so. It's definitely much harder to find a problem in the abstract than when you're looking at a very smart agent's best-effort solution and trying to compare it.

Is it harder or easier?

It's much easier when you have exactly that. I also think that, at the time SWE-bench Verified was published, it was a very strong benchmark. It's not as though we thought, “This wasn't a strong benchmark at the time.” I think this is something that a lot of benchmarks go through as they evolve.

When benchmarks start to become popular and viable, it's because they measure something important. Models might get 20% correct on them, sometimes even less, and people have something to hold on to and improve models on. By the time you hit very high performance on a benchmark, additional 0.1% improvements become meaningless. At the time, I think that benchmark was extremely valuable, and it taught us and the industry a lot.

It’s just that we’re now at the point where the audits are as strong as they are, and we’re starting to measure not necessarily what we want to measure—the coding capability of our agents—but the agents’ ability to correctly guess how to name a specific function.

And that isn’t really what we want to measure at this point.

Yeah, I think that’s fair. If I asked you to ballpark it, most frontier models are now at 80-something. What’s the actual number on SWE-bench Verified that you would guess as the ceiling?

I guess that’s really hard to say. When GPT-5.2 came out, folks took a look and found that it was solving 31 problems that were in the set of problems that should be very hard to solve without contamination. So I think it’s quite possible that that number is already something we’ve hit if you didn’t have contamination at all.

Fair enough.

Hard to say, though.

Yeah. Cool. We’re going to stop reporting SWE-bench Verified, right? SWE-bench Pro will be the next one, which is an effort from Scale. What’s your comparison analysis? What attracts you to SWE-bench Pro?

The first one is just that it’s harder. For SWE-bench Verified, I think something like 90% of the problems are things that were estimated to take an expert software engineer less than an hour. They’re very well specified and very self-contained, and the SWE-bench Pro problems are just bigger and harder. There’s much more headroom on the eval because it’s not saturated.

Yeah.

There are categories of 1–4 hours and 4+ hours.

Yeah. And it’s more diverse: lots of repositories, multiple languages, and qualitatively more different types of problems. So all that’s great. On the contamination side, we also think it’s better there.

The way we were measuring contamination in SWE-bench Verified was with this little contamination-auditor agent, which is given the description of the task, the patch, and the task ID. It’s told to take this target model and, with an open-ended set of questions, try to find questions that will reveal what contamination might be lurking in that model.

In SWE-bench Verified, we found many instances of contamination across OpenAI models, Claude Opus 4.5, and Gemini Flash. In all of these, we saw things like regurgitating the ground-truth solutions and, in some cases, giving the task IDs and other things that are pretty clear evidence of, at minimum, familiarity with the repositories.

Yeah.

A task ID, that’s—[laughter]—yeah. On the other hand, we don’t see this in SWE-bench Pro. I think the auditor agent found some very light evidence that maybe a couple of models might be very lightly familiar with 1 or 2 of the source repositories, but it’s very different from SWE-bench Verified.

So contamination is less of a problem. I think we should also expect that, at some point, this isn’t going to be the right benchmark anymore. As a field, we have to continue to move on and find harder and more representative problems that we can evaluate our capabilities on.

Awesome. So let’s go into that. I think there are a lot of—I think we also saw in the pre-chat as well—people feel a qualitative difference when they’re using GPT-5.1 to 5.2 to 5.3, and it’s not super expressed in these benchmarks because they’re saturated on a number of these things. What capabilities do you really want to benchmark in an ideal coding benchmark—an ideal agent-coding benchmark, whatever you call it?

One thing is open-ended design decisions: places where the problem may be a little bit underspecified, and seeing if the model can make reasonable design decisions.

What’s a reasonable prompt for that? “Vibe-code me a B2B SaaS to make no mistakes”—that’s the meme—but what’s an actual usable open-ended problem like that?

Sure. Maybe an example could be finding a way to speed up a particular part of a codebase, but there might be multiple different ways to—

Yeah, there are dedicated performance benchmarks. I think you guys have an efficiency benchmark, or is that—I don’t know. I think that’s from Harris’s group.

But yeah, that is a good one.

There are just many things that people value about working with software engineering agents. SWE-bench Verified obviously measured some important capability: given a description of a GitHub issue, can you produce a patch that solves that issue satisfactorily? Obviously, there are issues with the benchmark, which means that now that we’re at 80%, we don’t really trust further improvements on it. But it does measure a real capability of models.

As a field, we’re moving beyond, “Can my coding agent solve a small GitHub issue for me?” We’re starting to look at much longer-term tasks that don’t take 15 minutes but maybe an hour, sometimes days. Beyond what kinds of tasks my agent can solve, there might be things that are harder to grasp. Olivia talked about whether it has design taste: Does it solve the problem the way my team likes to solve problems? Is the code nice? Is it well written? Is it clean code? Is it maintainable in the future?

People care about a lot of these less tangible and harder-to-measure things that are still super meaningful for people working with coding agents.

Yeah. These are all qualities that are obviously no longer the low-hanging fruit. We have no idea how to evaluate the simple question. Maybe there are 2 forks in the road. One is the very human-intensive, money-intensive path, which is to hire a bunch of contractors and try to annotate this. The other is to use an LLM to proxy it and try to align the LLM so that it can give you a reasonable proxy. Which of those would you want? Would you want to do both?

I think maybe you should talk about GDPval as an example.

Sure. GDPval is an eval that was produced by a collaboration between the Human Data team and the Frontier Evals team, and it’s trying to measure whether agents can do a variety of real-world white-collar work. That was an eval where grading is very hard and requires a lot of domain knowledge about exactly what you’re looking for in each different context.

Yeah, across 15 or 16 white-collar professions that take up a significant part of GDP—

Kind of high-level professions, and then a lot of different granular subspecialties.

I’ve said I’m a big fan. It is basically the eval for AGI.

But partly because it was so hard—it required so much domain knowledge—the Human Data team hired a lot of people from these professions to be very involved in creating tasks and creating the gold solutions, and trying to help create rubrics and so forth so we could create a high-quality eval.

So basically, take the GDPval, which is a generalist thing, and take that same approach and apply it to code, and you roughly have a roadmap.

I think it’s an interesting solution. I think what you’re pointing out is an important problem, which is: How realistic is it? What we want to do is have coding agents write code that we think is good, and asking humans is actually a good way to ensure that. It’s also a slower, more complex way to do that.

Part of why I think SWE-bench Verified ended up being super popular, and why we’re seeing all benchmarks like this being super popular, is that it’s very easy to compare. Validating that a solution passes all the tests is pretty trivial once you can run the tests on your computer or wherever you’re running them. You can ask, “Is it correct or is it not correct?” and aggregate that. It’s super simple.

But it doesn’t tell you: Did it solve the problem? Is it ugly? Would an open-source maintainer of that project have merged that PR? It doesn’t tell you that. There is a lot of value in having benchmarks that are both easy to compare across the industry and can be run really fast without human involvement.

Yeah. Amazing. Your teams have also put out other kinds of evals that are related, like PaperBench, and then the more recursive-self-improvement-type evals. How much should that figure into mainstream coding evals? Is there some way in which those things join together?

Sorry, are you asking whether we should also be building evals for self-improvement, or are you saying that coding evals currently cover that domain?

I just think those are some of the most advanced evals that we have, and we’re not using them in the normal path. It’s an interesting split between evals for normal coding tasks and the one for machine learning, which is completely different. I think you get what I mean. That’s mostly a safety argument, I guess, but it’s also really useful for people to understand if the model is really good at AI coding, basically.

Yeah. My guess is that part of the reason a lot of benchmarks so far haven’t focused as much on AI coding is simply a question of what datasets are easy to gather, because a lot of the state-of-the-art AI codebases are proprietary. If we make evals for that, we’re probably not going to release them, and it’s harder for people in the field to create that kind of measure: Is this a realistic research-coding workflow?

I do think it’s good for the field to try to measure these skills publicly. I think it’s just harder to make it realistic.

And then one more thing that a lot of people are trying to do is, instead of a percentage from 0 to 100, redenominate it in dollars. You have Freelancer and all that; other people are doing Vending-Bench or whatever. Is there any alpha in those, or do you still want a traditional academic benchmark?

I think, in a way, there are different ways to measure the same thing. If we’re saying, “This is how much money it produces,” that’s fairly similar to saying, “This problem would take a human 2 hours to solve,” or something like that. Usually, they’re fairly correlated. However much time it would take a human to solve that problem kind of determines the value that we ascribe to a solution.

Yeah.

And so I think that’s an important piece. I think here, monetary value, time, or complexity all try to capture a similar thing.

Yeah. Okay. So they’re all proxies for some amount of increasing capacity that we want to measure. I think that’s a good thing. The only other major player in this field is METR, which has done this sort of long-horizon work. Congratulations—you guys have completely destroyed the curve for that. Any takes on that? Obviously, you’ve done really well, so it looks good, but I don’t know if that approach is something you want to incorporate into your own work making evals. This is the long-horizon test eval, if you—

Yeah. I know we’re from OpenAI, and we work with METR on these evaluations, so we do appreciate them. I think they’re using time, right? They’re not using money. So I think that was your question. Complexity, however we can quantify it, is really important to understand where our models are getting to.

Okay. Complexity is the abstract thing, and then it projects down to time, story points, or whatever—dollars. Great. One last question on the overall preparedness framework. I’ve seen people mention the preparedness framework a lot, but I don’t think it’s well explained to a lot of people. You actually have a nice website where it’s, I think, “test,” “inform,” and “teach,” or something like that. I feel like you do a lot of work there. I don’t know if you want to talk about how the preparedness framework applies.

The preparedness framework is an open, public framework for how we track frontier risk. These are capabilities that are typically dual-use: You can use them for good things or bad things, but we want to at least keep an eye out for the bad things to make sure that we—as a company and as the broader society—are prepared to handle the potential downsides.

At the moment, we track 3 different categories. One is bio risk, another is cybersecurity, and a third is research automation and model autonomy. That’s what ties most into the benchmark. Coding is not all of automating research, but it is 1 very important key component.

We initially created SWE-bench Verified as part of building out evals for that model-autonomy workstream. Now, I think we have to move beyond that toward looking more at whether models can actually automate research workflows.

Yeah. Amazing. Anything else to add on the general question of what people should know about preparedness, and how evals and human-AI alignment all work together in that?

I think maybe the thing I would say is that we really appreciate it. We work really hard to build these evals, and that’s where we published Verified and where we’re sharing evals and these sorts of things.

We also deeply appreciate other people, and the entire field, building evals and sharing and reusing them. SWE-bench Pro—yes, that’s a better eval; now we should use it. We would really encourage people to find more ways to create and share evals that we and the entire field can use to measure progress on a variety of capabilities, including coding, because it’s important to understand where we are.

Mia had to leave, but we’re just talking a little bit about the future directions we want evals to go. Mhm.

I think here we can dive in on these things and give you a platform to make a call for what you’re looking for. A few things would be useful. First of all, really, really hard tasks—the kinds of things that would take top-notch engineers months or teams weeks—would be quite good, especially if grading is reliable and you have, for example, rubrics that have been sourced and validated by many people in the field. I think that would be quite valuable.

I think benchmarks on creating products end to end, as people are building more, would also be quite useful. A third thing I’d say that’s maybe not quite an eval but is still relevant to the overall mission of how we as a field and as a world should track where these capabilities are going: I’d like to see more metrics tracking real-world usage. How much is AI actually being used in the field? How much is it replacing people’s jobs? How much is it augmenting people and speeding them up? Just real-world metrics.

Yeah. The replacement thing is always sensitive on the PR side of things, but we create new jobs that manage the old jobs, and that’s how it is.

In terms of the frontier evals that OpenAI is really excited to push—you put out really good work every single time—what should people expect from OpenAI itself?

I’m not sure I could say what we’re going to—

General directions.

I mean, general directions—

That kind of stuff.

Yeah.

Yeah. Amazing. I’m excited for more real-world impact. I think you guys have made a lot of progress and taken a lot of industry leadership with SWE-bench Verified, and now moving on to SWE-bench Pro. Thank you for doing this and for being so transparent. I think people will respond in kind.

Yeah.

Yeah. Thanks for your time.

Thank you.