[State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang

We're here at NeurIPS with John Yang of SWE-bench and many other things. Welcome.

Thanks so much for having me. I'm really happy to be here.

Last year I talked to Oier and Carlos, one of your co-authors. How's SWE-bench doing? Generally, the project is about 1.5 years old.

Yeah. I think it's about 1.5 years old in terms of when it became useful. We put it out in October 2023, and people didn't really touch it too much. Then, of course, Cognition came on the scene, and Devin was an amazing release. I think after that, it kicked off the arms race.

Did they tell you beforehand, or did they just show up?

I got an email about 2 weeks ago. I think it was from Walden. He was like, "Hey, we have a good number on it." I was like, "Wow, congrats. Thanks for using it." Then the release was mind-blowing. I was like, "Wow, these guys did an excellent job."

Amazing. SWE-bench Verified was maybe last year.

That's right.

Catch us up on this year. You have other languages, and there are a whole bunch of varieties of SWE-bench now. What should people know?

For sure. I think there are a couple of extensions happening. One is more SWE-benches: SWE-bench Pro and SWE-bench Live.

Oh, SWE-bench Pro. Was that with you guys? It looks independent. It has different authors.

It's completely independent.

So they just called it SWE-bench Pro without your blessing?

Yeah. I think we're okay with it. When it came out, we were like, "Oh, cool. Interesting." It would have been fun to be part of it, but congrats to them. It's a great benchmark.

Multimodal?

Yeah, we did multimodal and multilingual.

Is the multilingual version JavaScript? What else?

Multilingual is 9 languages across about 40 repositories. You've got languages like JavaScript, Rust, Java, C, and Ruby.

And SWE-bench itself: a lot of people talk about the Django focus. How do we move past Django?

For sure. It's cool to see a lot of the newer benchmarks really try to diversify the repositories. In the 2 follow-ups we did with multimodal and multilingual, we made a point of doing that.

But you can also just put out SWE-bench 2025.

That is true: do a new distribution. It's been cool to see the follow-ups. Quietly, it's an open question for me, and I'm excited to see how people curate the next sets. It's interesting to see in the literature or in blog posts how people justify creating their separate splits. The easier ones are like, "Oh, more languages, more repositories." Now people are saying, "Ours is more difficult because of this curation technique." I'm excited to see how long that lasts and where we're going to guide evaluations.

More recently, you're working on CodeClash.

Yes, that's right.

You've already done other podcasts about it. I refer people to your chat with Andy, but give people a 1- or 2-sentence explanation.

Happy to do it, especially on your podcast. It's an honor.

Basically, I don't like unit tests as a form of verification. I also think there's an issue with SWE-bench in that all of the task instances are independent of each other. The moment a model submits it and says, "Oh, it's done," that's the end of the story—the end of the episode.

With CodeClash, we're trying to evaluate long-horizon development on a codebase that is consequential and conditioned on what a model did to that codebase before. The general idea is that you have 2 or more language models, and they play a programming tournament. Each model maintains its own codebase, and in each round of the tournament, it first gets to edit and improve its codebase however it sees fit. It's very self-determined.

Then, in the competition phase, those 2 codebases are pitted against each other. The codebases are run in an arena—we have a lot of diverse arenas—and the arena determines whether codebase A is better than codebase B. Then you repeat that across multiple rounds.

As determined by an LLM judge?

Yeah. An LLM judge is definitely one of the mechanisms. We started with some pretty simple programming games. One of the cooler ones is Halite, which—

Oh, yeah, I played it for Jane Street.

Yes, that's right. That's awesome. Halite 1, 2, and 3—Michael Truell of Cursor wrote this game.

Two Sigma. Jane Street.

Yes—oh, Two Sigma.

I worked at Two Sigma.

Oh, there you go.

This was too long ago.

There you go. Yeah, 2016 at this point, but we're bringing it back.

Halite is fun. I would say, if you've never done a programmatic competition where you have to control fleets of ships, attack and defend things, and collect resources—

Yeah, it's like playing StarCraft, but you can code, right?

Yeah, exactly. A lot of games.

Yeah.

Are there non-games, or are you focused on games?

I think that's an excellent point. For the initial release, for scientific purposes, we used existing programming games. The current ongoing effort is to build economically valuable arenas. That's the popular word these days.

Yeah, SWE-bench is a big one this year.

Yeah, awesome. I think the big selling point of Terminal-Bench and SWE-bench, and these evaluations, is that they're really close to real-world utility. I think that's resolvable for CodeClash, and that's what we're working on.

You're part of a group?

Yes.

The other students have also been putting out a lot of other things. What would you highlight?

Oier is such a prolific mentor when it comes to benchmarking. SWE-fficiency is one I really like, in the line of performance optimization.

Performance optimization. What's the one?

Yeah, for sure. SWE-fficiency was written by a PhD student called Jeffrey Ma, who happened to be my high school classmate. The idea is that you take a codebase and make modifications that literally make the code run faster. This includes parallelization and similar techniques.

So, no behavior change—just faster?

Exactly. Keep the unit tests passing, but get better runtime.

And then there's AlgoTune, which is in line with that.

Yeah. Then there's also the scientific coding domain.

Yeah, exactly. SciCode, too, is awesome. They did a quick—

For people, the way I explain SciCode is that it's HumanEval but better.

Exactly. I think there's a lot of good stuff these days. That's the way to go.

SWE-bench is expensive to run. Any agentic benchmark is expensive to run. You do need some completion benchmarks that just—

Complete. Exactly. You can do well on those first and then graduate to the expensive, multi-turn stuff.

Broadly, what other work in the field in 2025 has caught your eye in terms of coding evaluations?

Obviously, shout-out to METR. They use VBench, and they have a very interesting human-hours-worked number. They have the x-axis as the runtime and the y-axis as completion. We can do more long-running tasks. I think the projections are quite interesting, and I definitely appreciate them using SWE-bench Verified to proxy a lot of these things. They're great.

Any other work that caught your eye?

Within Terminal-Bench—

Critical Point was kind of cool.

Critical Point is a very new benchmark that Oier did, and I think it's related to physics.

There's also SecBench, which is related to cybersecurity.

Yeah, exactly. SecBench. It's cool to see people really dive into different coding domains.

Stepping a little bit outside of coding, I personally think it's quite interesting to think about user-simulator work, such as τ-bench, τ²-bench, and Vending-Bench.

I have mixed feelings.

I'm interested.

Well, I mean, it's like sampling 1 path. I don't know how realistic it is, to be honest. It's just the elements, but it is cool.

No, for sure. I agree. I think it's a good initial effort. To me, it's super cool to see companies like METR and others focusing on building environments for code beyond code. It might be interesting to have WorkGym-style stuff.

This is stuff that my adviser, Diyi Yang, at Stanford thinks about a lot. So, yeah.

Yeah.

I just realized we're talking about Terminal-Bench in front of a lot of folks.

Yeah. Yeah.

You know, really, really, really good work. Overall, let's talk about τ-bench, because you mentioned τ-bench.

Yes. Yes. There's some discussion—some people are saying that τ-bench is impossible to get a high score on because some of the tasks are underspecified or just impossible.

I don't know if you're up to speed on that.

I'm a little bit spicy. Yeah, it's a bit spicy. I think I saw—I worked with Shunyu and Karthik back at Princeton very closely. I think Karthik just posted a tweet rebutting some of these claims.

I get the concern, but I think it also brings up interesting research problems to solve. Why is it impossible? Is it the ambiguity? Is it the user simulator that has issues? I think generally we all agree that we'll improve on these things over time for agents.

So I actually really like benchmarks that intentionally include impossible tasks as a flag. Like, “Hey, you're cheating.”

Yes. Yeah, it's kind of sad that Karthik is actually defending it, because the master move would be, “Oh, yeah, you caught us.” Everyone reporting above 75 on τ-bench Retail would be cheating.

Yeah. Oh, interesting. That would be cool. You'll have to ask the τ-bench authors, but that's fun. I think ImpossibleBench was a recent benchmark, maybe from Anthropic. They basically took SWE-bench Verified and changed the issues to make them impossible, then checked how often the models would say, “I actually just can't do this. I don't know what's going on.”

Oh, like for refusals.

Yes. Yes. Yes. So—

Oh, how did they do?

I thought that was interesting. I think all the models were attempting the tasks and saying, “Oh, I did it.” So maybe not great.

That's cool. No, that's an important one.

Yeah.

How do coding evals evolve next year?

Wow, that's a great question. Honestly, I think people will make more SWE-bench variants. I think Terminal-Bench has really got something going where, with SWE-bench, you're confined in some sense to the domain of issues and pull requests that already exist. That has the benefits of being close to reality and natural, but with Terminal-Bench, there's a lot of creativity that you can infuse into it.

I would personally be really excited—Terminal-Bench 2.0 was really excellent—and I'd be super excited to see 3.0 and 4.0 because of the environments.

Yeah, I mean, the environments—bringing more people into the fold. Correct me if I'm wrong, Mike, but early on you had PhD students and very smart computer science people adding tasks. What does that look like when you add more coding environments for non-coding tasks, non-coding environments in general, and ask people to make things there? That's pretty cool.

And then, of course, for myself, I think this long-running SWE-agent kind of thing feels very compelling. The vision of, “Hey, I tell it a goal. I don't have to be super specific about my task. I have a decent verifier that proxies what I want”—something literally like, “Make the codebase that makes the most money in this setting.” That's my verifier.

I walk away for 5 hours. The thing is just running while I'm hanging out with you and talking to my friends. I come back, and it gives me literally a solid codebase for that task. I think that would be super cool. Okay, I'll push back. We're part-time.

Yes.

And we are emphasizing a lot of interactivity—

Because the point is that you're going to underspecify, right?

Right. Right. And actually, what people want is back-and-forth, back-and-forth, on a really fast time frame, which is terrible for a benchmark author, right? Because how do you do that?

But realistic.

Yeah. So I think this is where I'm a little bit anxious or cautious about this push for long autonomy. Let's say this time next year, 5 hours is pessimistic—it'll be 24—

Long. Yeah. Right. Days.

But I don't know if that actually materially changes the industry.

So we'll push it as an eval. You know, we have people who make evals here.

Yeah. Yeah. We push the industry in ways that we want to push it, but I don't know if that's a productive way, because it's more of a stunt. It's a proof of concept, an existence proof that it can be done.

Yeah.

But will you use it in real life?

Yeah. Yeah. Honestly, I think there's potentially room for growth, so I would agree with your take here. With my lab at Stanford, with Diyi, her emphasis is on human-AI collaboration, so I definitely don't believe in this idea of just getting rid of the human.

Maybe it's about finding the balance. The developer ecosystem is so diverse, and there are so many participants who want different things out of it. We should enable different levels of abstraction. It depends on the task. There are settings where you want to be more involved and more hands-on, so you want to use Windsurf for that.

But maybe there's a general data-processing task that's just a lot of JSON parsing. You don't really care about it, and that's the one you want to walk away from and just let it figure it out. So, yeah, I would generally agree with you.

Yeah. Yeah. Amazing. Any calls to action? What do you want help on? How can people find more of your work?

Definitely. For the call to action, I'm super jealous of all the great data that Cognition and Cursor get. That user-interaction data is fascinating from an academic standpoint.

It feels like there are 2 difficult approaches to resolving that. Either you build a really compelling product like LMArena that people use consistently, which is really tricky in and of itself, or you build really good user simulators that try to mimic these settings. But that's also nontrivial. I don't think it's as simple as, “Hey, ChatGPT, act like a human,” right?

Yeah. It would be really cool to get inspiration for what exactly that data looks like, or, between the 2 approaches, what the best way is to scale up evaluating human-AI interaction.

For visibility for my own work, we're pushing more arenas. For CodeClash, what I'm excited about is that the current framing is really long-running SWE agents, but you could have multi-agent settings. What happens when 2 agents work together on a codebase? What happens when you have a human and an agent work on the codebase, versus just AIs?

When the models improve and hopefully hill-climb, becoming better at digesting logs and iterating on analysis, how does human-AI interaction change with model capability? I'm trying to inspire and convince people that it's a very cool testbed where you can do a lot of different combinations of humans and AIs on different arenas—playing one arena at a time or multiple arenas at a time.

Yeah, I'm very interested in working with you on the interaction stuff.

That would be awesome. One more thing I'll add is that Cognition is going to be pushing a lot of codebase understanding, which is kind of codebase retrieval plus.

Yes.

And mostly, it is helping humans understand their own codebases better—to enable humans, or to sort of mind-meld the human with the machine, to do the highest-possible task that an LM couldn't do alone and humans couldn't do alone.

The other thing is automated context engineering for an LM. That's sort of a research subagent that we're working on.

That's so awesome.

I don't know what the benchmark would be, because how do you benchmark understanding?

That is true.

Apart from, I guess, freezing a repo, having some manually curated answers, and then posing trivia questions. That's very easy to saturate, so I don't know how else to—

Yeah, I think Swyx tweeted a while ago about something like the wiki—the code wikis. That's incredible. I mean, I use—

Google actually just came out with its own version.

Oh, yeah, with the Antigravity people.

No, no, no. This is a separate team.

Gotcha. Gotcha.

But cool. That's the state of code.

Yep.