⚡️Multi-Turn RL for Multi-Hour Agents — with Will Brown, Prime Intellect

I’m Alessio, partner and CTO at Decibel. I’m joined by my co-host, swyx, founder of Small AI.

Hey.

Hey. We knew that Claude 4 was coming, but we were too busy to have a dedicated episode, so this is our makeup dedicated episode with a special guest, Will Brown from Prime Intellect—now I can say it. Hey, how’s it going?

Great to be on. I’m so excited. I’ve known you guys for a little bit, and this is my first time on the podcast, I believe. Great to chat with you guys. Big news day, I guess. Lots of stuff out in the world.

There’s always a news day. I think this week is particularly heavy for some weird reason. Monday was Microsoft Build, Tuesday and Wednesday were Google, and today is Claude. I wonder what tomorrow will bring. We had I/O, and then we had I/O, and then—

Yeah, yeah. Different I/Os.

Exactly. We were supposed to record this morning, but we all wanted to watch the Claude keynote, so we went and watched the Claude keynote. Obviously, it’s a good model—a big model. They’re really emphasizing coding. They didn’t really talk much about reasoning, to be super honest; they were just like, “It runs for longer now.” What are you guys’ takes?

One thing I’ve been seeing coming for a little bit, and that I think people are also aware of now, is that the thing that’s going to make the next wave of stuff powerful is that everyone wants better agents. Everyone wants models that can go off and do stuff, and reasoning was kind of a precursor to that.

I always think of OpenAI’s 5-level framework, where chatbots were the RLHF era and then reasoners were o1 and R1. But really, what people were thinking of was that reasoners are a step on the path toward agents. I can see why Claude isn’t saying, “We have the best reasoner.” They’re really showing off their sweet agent, tool use, function-calling benchmarks, and multi-turn stuff, because I think that’s what people care about more for actual applications as opposed to whether it did really well on a math competition.

The math competition stuff was all a signal that was supposed to make us think we were getting somewhere. But the thing we were getting toward, for a lot of people at least, is practical agents.

The extended-thinking mode, I think, removed the capitalization. In the Claude 3.7 release, it was “extended thinking,” kind of capitalized, and now it’s just “extended thinking with tool use.” So I think they’re also downplaying whether or not it’s reasoning. They’re trying to merge everything together.

I didn’t realize that extended thinking couldn’t use tools before, based on the way they worded it, and now it can. In Opus, they haven’t put it as front and center as last time.

Do we have any—this is already veering off from Claude directly into speculation—but do we have any idea if there are any material differences between how Claude’s extended thinking works versus the old o-series models? Do we know?

The biggest difference seems to be—and this has been a thing I’ve been seeing for a little while—that Anthropic has always had this little thinking feature. Sometimes even Claude 3.5 Sonnet would do a tiny bit of thinking, and it was really just deciding which tool to use. If it was doing an artifact in the Claude UI, it would have this little thing where it would think for 2 sentences about which tool to use.

Anthropic’s attitude seems to have been that extended thinking is an instance of tool use, and that it’s the kind of thing you want to equip the model with the ability to do. But it’s not like, “Oh, it’s a thinking model.” It’s just a thing for the model to brain-vomit, because that brain-vomiting will help it find a nice thing to do next.

That’s the same way that search or code execution are ways to get more information on the path toward finishing a problem.

Yeah, inference-time compute, as they say. I met somebody who claimed to have coined the term in the foundational scratchpad paper. This was obviously before Jason Wei’s chain-of-thought paper, but it’s all in the same general family of techniques.

I think the question for me is also: Is there some model routing going on? Are they different models—the thinking and non-thinking models—or are they the same models with just the end-of-turn token generation turned off?

I think these models should be the same model, and Anthropic knows what they’re doing. It’s not that hard to make whether or not a model thinks be the sort of thing you can control. Qwen did it in a very simple way, and they talked about how they did it a little bit.

All this stuff is hard at serious scale, but conceptually, it’s not a big problem to solve. We have reinforcement learning, or we can do supervised fine-tuning on different things. We can teach models skills like that pretty easily.

You have some work that you’ve published recently on GRPO and RLVR, and you’re doing a lot of work on multi-turn RL. I wanted to round out any other Claude highlights. There is a controversy that I’m leaving toward the end, but are there any other technical highlights that you guys want to focus on?

I think it seems like a really cool model, but Chollet tweeted earlier today that it seems like linear progress, which is great, but it doesn’t feel like there’s anything I’ve seen from it that feels like a paradigm shift in terms of the sorts of stuff Dario talks about. I think maybe we’re still on the path to get there. It feels like this has just gone up in terms of the complexity of agents.

The one thing that, to me, was really nice to see—I haven’t done too much testing myself yet—is the way they reported the reward-hacking benchmarks. The reward-hacking issue with Sonnet 3.7 allowed it to do stuff that felt reward-y. You would ask it a coding question, and it would do your question and then 7 other things as well, presumably because there was some RL environment where there wasn’t really a penalty for doing that, or there wasn’t enough of a penalty. Covering its bases was more likely to pass test cases on some coding task.

You could imagine a SWE-bench kind of thing where there’s a minimal diff that is really what you want, but you could do a ton of other stuff and put all these other things in place. As long as it’s not enough that you trip over its feet, it’s just extra stuff that’s there if it helps pass the test cases.

What I really think you want to do with these models is kind of min-max. You want the models to do the thing and no more. They had some internal benchmark for this that went from 45% down to 15%, for both Sonnet and Opus, as opposed to 3.7. I’m hopeful that these models are much friendlier to code with and maybe more trustworthy.

I have buckets for models based on how much I can trust them in a codebase, especially something beyond a single file. Old Gemini was very trustworthy. GPT-4.1 is very trustworthy. New Gemini is not. Sonnet 3.7 is not. o3 is not. I haven’t decided which bucket new Opus is going to fall into.

I mean trustworthy in terms of reward hacking: They’re going to do the right thing in the codebase. Worst case, they’ll do it in a dumb way, but they’re not going to break a bunch of stuff. They’re not going to leave a bunch of extraneous comments and helper functions all over the place that aren’t really needed, or make 7 new files just to have them there. This is the sort of thing o3 does a lot.

Yeah, I already had the function in my codebase. I would just make a new one because it felt like it.

One thing I often wonder about those things is, for RL environments in general, why isn’t token cost more of a thing in the penalties? That’s the one rule above all: You can skip a lot of reward hacking by saying, “Hey, the more tokens you use, the worse it is.”

That’s not what the model companies want; they’re selling you tokens. But there’s that element of it, and I think there was also this initial reaction from everybody that more tokens is better. If you look at the line, it goes up: As you spend more tokens, your accuracy goes up.

The pressure to really tamp down on token usage wasn’t that serious for a lot of people, especially because the companies like to sell you more tokens. But it is the sort of thing where you can have more controls.

Qwen did this in a very abrupt way. In the UI, you can set a token budget, and it just truncates the thought. It seems like artificially truncating the thought is actually fine. The model can, even if it gets cut off mid-sentence with an injected end-of-thought token, finish with the best that it got from that point.

That’s one way to do it. The other—and this is becoming a standard API feature now—is a thinking budget. Claude has that. We did a little bit of experimentation with that in our last Intellect-2 run. That was before I joined, but thinking budgets are the kinds of things you can insert into a reinforcement-learning objective, and you can see the model get better at targeting the right amount of thinking.

Let’s say something goes in your system prompt: You can have the prompt say, “Use X amount of tokens.”

If you've trained the model to respect this, you would hope that, if you execute it correctly, the model learns to roughly think the right amount. This actually changed my opinion of thinking budgets, because previously I was thinking that reasoning effort was better than thinking budgets. A thinking budget is kind of like a max cutoff.

The same thing? It's a target, right?

Okay. The effort is a target, probably.

Yeah, right. Because I actually want to set effort. I don't super care about the cutoff apart from cost. Giving me, you know, 64K tokens of cutoff or whatever doesn't matter.

I'm not sure that they're that different. We don't know how they do it under the hood, but my guess is that the whole reasoning-effort thing is essentially a token budget that the model has been reinforcement-learning-trained to use. You would hope that you get different behavior: when the model is told it has a short thinking budget, you would hope that it uses slightly different strategies that are better, versus if it has a high budget, when it's more willing to do lots of math calculations, for example.

Conceptually, I think it's really just about the model having some amount of room that it can bank in tokens, and trying to do that, hopefully.

Do you think we're going to have these as hyperparameters for much longer, or do you think that, as we're early in reasoning models, more of this stuff is exposed and then gets moved away from the user?

In chat interfaces, it probably won't stick around. I don't think we're always going to have a dropdown with “o4-mini” and “o4-mini-high.” That feels silly.

I do think it's a thing that developers want, especially because once you've built around a certain model, a lot of these providers are hoping you stick with that one model and don't switch all the time. You need a knob to control costs and latency, so that's one useful knob to expose to developers for controlling quality versus cost and latency.

Awesome. Cool on all of that. I think the elephant in the room is this controversy around Claude Opus—Claude Opus, right?—snitching on you. For those out of the loop, let's recap, because I feel like you're closer to this than I am. I learned about it from you.

Sure. This was someone from Anthropic—I'm not going to name him, because I know he doesn't want all this attention—who deleted the tweet. It was essentially going through different things that people found during safety stress-testing of Claude. This isn't, “What's Claude going to do for you?” I think people took this pretty badly out of context. There's a fair point there: people are really reading into that 1 sentence much more than they should.

But this is something Anthropic does a lot: they really stress-test their models. They try to put their models in situations where they can see what an adversary could get the model to do, or what the model does if it's in a situation where there's no right answer.

A lot of the headline Anthropic safety results, especially related to reward hacking, deception, and alignment faking, seem to me like a rock-and-a-hard-place situation, where the model has 2 objectives that it's given that are conflicting with each other and has to pick 1. No matter which one it picks, it's going to sound terrible. It's either following the user's instructions or it's following common norms, and once you accept either of those, it's going to do the thing that's aligned with that set of guidelines.

If the model's goal is to be maximally helpful to the user, then it would help a user build a bomb. If the model's goal is to be maximally helpful to society and a user is asking it to build a bomb, it's going to be like, “No, that's bad. I have to do something to stop this.” You kind of have to pick a goal. Maybe the right answer is that the model just defers: “Nope, I'm going to stop talking.” But people also get mad when you tell them that the model will stop talking to them or refuse to do anything. There's just no way to win and make everybody happy.

I do think they report this because they think it's important to have people understand the safety implications of these models and to understand how bad it would be if someone were trying to use this. Could it meaningfully help someone commit a crime or violence, or whatever? That's what they have their safety framework for.

The things that happen in these blog posts, threads, and papers—where the model tries these things—come from putting the models in a scenario that elicits those behaviors. It's the sort of thing that you would imagine a very smart human might also do in those situations. Let's say you're told to accomplish some vague, underspecified goal at any cost, and you really want that goal. Think of game shows like Survivor, or Lord of the Flies—any of these canonical situations where people are put in a weird spot and have to figure out how to do things.

They're crafting these environments for the models, looking at them, and seeing what happens. I think it's a little silly to overanalyze behaviors in either direction: “The model is reporting you to the police,” or, “The model's going to help you find uranium on the dark web.” These models can kind of do anything. The base model of an LLM is not artificially constrained in any way; with the right prompt, it'll do whatever it can up to its intelligence limit. The question is how you constrain the space from all possibilities down to a more reasonable set, and that's hard.

Yeah, this is it. Does any of this make you think differently about what tools to give an LLM? I know they deleted the tweet, but it's basically, “Well, before, if you're putting in all these MCPs, you have email access and all of this. Now it's like, maybe I don't want to give email access all the time if you're going to snitch on me with the email access.”

I think coding with these models, especially Claude 3.7, is something I did a fair amount of. For a few weeks, I was doing a lot of Claude Code with 3.7, mostly for random side projects. I never really got to the point where I found it helpful for something that was a large existing codebase. But if it's, “Hey, I want to cook something up in a few hours for fun,” it's pretty good at that.

These become messy and hard to maintain, and you get to a point where nothing is working and you have to dig in and fix it all yourself. I think part of that is that the models have access to a terminal, and you can do a lot of things in a terminal.

MCP is a way of constraining the action space. In canonical RL, people talk about states, actions, rewards, and policies as the moving parts. Models are generally trained in old-school RL with a very fixed action space, such as the keys on a video game that I can hit. With an LLM, text is kind of unbounded in what you can do with it. In a terminal, there's not much you can't do.

I got a lot of flak for this one. I'm just showing this.

Wait, flak? Why?

It was both people saying the notation was stupid and bad, and people saying RL is really simple, as well as people saying RL is complicated. Everyone has a different opinion on what RL means, and I was trying to say, “Hey, it's actually kind of complicated.”

I wasn't making the point that, “Oh, the definition of the MDP is complicated.” I was saying that there are just a lot of moving parts, especially if you want to change any part of the system.

Here's a hypothetical question: What happens if you have 2 LLMs learning together? How do you reason about that? How do you think about it? Is this going to be a stable system or an unstable system? What if they're kind of cooperative but kind of not cooperative, and they're training to work together but also want to backstab each other?

This is the kind of environment people find themselves in all the time out in the real world. But if you want to make AIs do this, you have to translate it into code and math. The more complex your goals are with this thing, the more complex the math gets. RL is 1 math language that exposes these primitives, but I think a lot of people are like, “I can follow the equations.”

That means I understand it. Sure. But there's also this N-body problem where you can freeze the system and look at it, but how does one thing moving affect everything else? What are the cascading ripple effects?

Wow, you just brought the three-body problem into this. Amazing. I mean the physics version, not the show.

No, no, no. I mean, it's actually very difficult to model. I guess you can simulate it, but it's sensitive to initial conditions. You can't really predict it. This is one of those things where I wonder: Why does no one predict the weather a year out? I don't think anyone has anything that's good at long-term weather forecasting beyond, I don't know, climate. No one can predict whether it's going to rain in Seattle on a given day a year from now.

Even if you think the system is predetermined, we know it's all clouds bumping into each other and whatnot in mountain ranges. We kind of know how these things work. So the butterflies are flapping their wings. You have to let it play out, like butterflies. If we had no butterflies, we could predict it, right? It's very sensitive to butterflies.

Interesting. Okay. So I guess we can round it out, unless there's any more of the controversy.

I don't think there is. I think the system card is actually very good. They probably went too hard on it compared with normal system cards, but it's a little bit confusing whether this is marketing or they're just saying, “No, we really care about safety.” Part of this is Apollo just being Apollo, pushing the frontier of red-teaming, right? They're going to report the things because it's extremely good Apollo marketing.

Yeah, yeah. I think they're really still trying to be creative with their consumer marketing. It feels like people in the AI world either love Claude or have grown to love Claude, and they've had a phase where they were using it a ton, but it hasn't really broken out to general people in the way. A lot of their marketing that I've seen is a little confusing. It feels like they've done a really good job crafting a brand image that appeals to a segment of the population who have certain considerations, who really like a model to have a deep personality or whatever.

Those are the sorts of people who I think also really like GPT-4.5. Many of them really love Claude 3 Opus, the big model. A lot of people just don't care and just want to use it. They're trying to figure out how to appeal to that audience. The LLM sycophants, the people who love those models, are a different crowd, and it's a larger crowd. That's a tough problem to solve.

What's your quick take on LMArena getting $100 million?

We'll see. I imagine that they partner with AI labs in different capacities, probably making a lot of money. I'm not in the business of trying to point the finger at them and say they definitely did this, but if I were a company that was able to raise at that kind of valuation and had just had a long public partnership with Meta—an eval partnership for a thing where we've kind of seen that Meta had the ability to do a lot more back-and-forth than a lot of other labs—I would imagine that there's some compensation going on there, or access to data. Being an evals company puts you in a really hard spot.

Some people were talking about this on Twitter: To be an evals company, you kind of have to sell to the labs, but selling to the labs kind of wrecks evals because they're your customer.

Yeah. In finance, we would—I mean, this is the credit-rating agencies. Literally, your customer is the one that you're supposed to govern, but they're also your customers, so you have to be nice to them or they'll just go to the next one.

I do think that the best source of evals going forward is probably going to be academia. This is what I tell people who are starting a PhD: Find things that are cheap to work on as a PhD student, because you cannot really pretrain a foundation model on your own, but you can build a really good, really clever eval. We are churning through evals all the time; we saturate them. We always need more. It's not the kind of thing that's ever going to end.

The task of translating the vibes of what is good or bad about a model into precise scientific questions is an important problem. It's a problem where you can get by with a lot more brainpower rather than dumping capital into it. You need to pay for the API costs, but that's generally the kind of thing you can get covered through academic grants or industry sponsors. There are also versions of these things with small sample sizes that get you on the radar, or you can pick and choose which models you can afford to evaluate.

It's an accessible field of research, and it's one where I think the incentives of academia are quite good. You can write a splashy paper that says something interesting about the broader field, rather than, “Oh, we want to make this one look like the winner.”

Yeah. I think a lot of grad students don't have taste. I don't know how better to put it. You go to enough academic conferences and think, “Why did you work on this? You're so smart. You're capable of better.”

So, how do you teach taste?

I can tell you how I did it originally. You always want to be thinking pretty far ahead, and you want to be making educated bets about what the world looks like in a few years. You have to ask: What are the questions that no one's even talking about? This isn't an easy thing to do. You have to really convince yourself that you're right, at least about the way things might go.

I finished up undergrad in late 2019, then went right into grad school. Toward the end of the 2010s, we had AlphaGo and DeepMind doing all this multi-agent RL stuff that was really cool. Then it was, okay, this stuff kind of works; AI is going somewhere. Multi-agent systems were still in the very early stages, but what was going to happen once this got there? It seemed like these things were all going to be continually learning in parallel as this big multiplayer game, basically.

If you look at the math, it was undercooked. There are some really hard open questions that are still open questions in multi-agent learning theory. So my focus was: How do I learn about this? How do I learn to think about this stuff better? At some point, I got tired of proving theorems and thought, “Okay, let's just go build the thing.”

Whether you're doing theory or experiments, you have to lay out a few different conditional statements to get to the point where you're really doing interesting research that's beyond just working on things that people are obviously going to be working on in parallel. You want to be jumping ahead of the curve a little bit.

I wasn't the first person to do this, but it was pretty clear to me, after and before R1, that RL was going to work and that it was going to intersect with agents, where the solution was going to be RL tool use. That seemed like the direction things were going to go. I don't think that was a very risky research bet, but it was a research bet that seemed to work out.

Speaking of which, you just published the paper. Now I have the full context: You were an adviser on this, and one of your grad students was doing the work, something like that?

Yeah. It was me with Silan Gong, who was my intern. This was kind of the last major thing I was working on at Morgan Stanley, and it was in parallel with the Verifiers repo that I've been building out. Major updates to that are coming very soon, by the way. I'm very excited about some stuff.

It was something I really started in earnest in January, kind of in the follow-up to it. I'd had the GRPO demo go viral, and I thought, “Oh, wait. There's something to this format-reward thing.”

It was literally a GitHub gist, right? Or something.

This is a proper repo.

No, no, no—the GRPO one. The other one?

Yeah, yeah. The other one was just a gist. This one is a repo for multi-turn RL with tool use. In some ways, the paper is the first paper—there have been a couple of other papers that people have used the repo for—but it's one where a lot of the stuff from the original GRPO demo gets extended to the multi-turn RL tool-use setting.

There are a lot of experiments here about how you actually get models to use tools and how you incentivize tool use. Something we'd see is that if you set these models up to use tools, they just won't. If you say, “Here's a question. You have access to these tools. Do as many rounds of tool calling as you want, and then submit your answer,” they'll just submit their answer. Especially for small models, if they aren't already trained to use tools, they don't really want to because they don't necessarily have that instinct. They're also pretty bad at function calling and format-instruction following.

And so what you would see is that when they’d use a tool, they’d mess up the JSON. Then they’d say, “Oh, that didn’t work,” and it would throw them off—it got them out of focus. Following that, the model would be more likely to go off the rails because it would get an error message from the parser. The safe option for the models was just to stay in this basin of “think, then respond.”

It’s the same with normal formatting rewards. If you want models to use thinking tokens, you have to incentivize that. You either have to do a little bit of SFT warm-up, or you have to reward them for doing it. Otherwise, they will not follow it 100% of the time on format alone, versus a model like R1, which 100% of the time is going to use its think tokens. You’re never going to see R1 just talk normally without the thinking section.

You have to decide what you want the model to do. This is a little bit of a user-facing question: What behavior should be the model’s default? If you want it to be a tool-using agent model, it does help considerably to actually have this incorporated into the reward.

The key trick in the paper to get around this problem was that one kind of reward these models would get is a dummy tool call. They would learn to ask for the same Google search every time and ignore it. Some questions would be, “Here’s an MMLU-style question. Go figure out the answer; use web search.” If you start rewarding them for tool use, they will use the tool, but they don’t really want to have to. They want to be very safe with it, and many of these questions they already know the answers to.

I think calibrating the right difficulty of your questions for RL is an important problem that we’re still figuring out. The models would do silly versions of tool use where they weren’t actually using the tool to assist in their reasoning. They were using it to get the reward.

We have to do a credit-assignment step: Did the tool result in information? For these experiments, the trick was to ask whether some string-matching function involving the ground-truth answer and the returned search results from Wikipedia showed that the model had actually searched for something that retrieved useful information for a question. The framework is more general than that. Once you have a way to do intermediate evaluation—if you can evaluate the quality of an intermediary state—you can rewrite the GRPO advantage calculation to take this into account.

I think this is less of a problem in PPO, which is old-school RL and is also what people use for RLHF. But in the context of GRPO, it’s great for leaning heavily on highly parallel inference compute. It’s more memory-efficient for the actual training process, and it’s much easier to do in a distributed fashion because you have less gradient syncing and fewer model-weight copies. It’s kind of like DPO on steroids, I think, is one way to think about it.

But it also gets around a lot of the pitfalls of DPO, both because it’s online by default and because you have a large set rather than just a pair of completions. You do get some intermediate credit assignment through this group comparison. But for tool use, it seems to be far enough out of distribution for small models, especially when you incorporate turn-level credit assignment.

The way I’ve been thinking about it is that in canonical RL, state-action pairs are part of a process where you do many rounds: take an action, go to a new state; take an action, go to a new state. For a while, people thought about language-model RL as, “Each token is an action, and the new sequence is a new state.” You can do that, but you can also think of each turn as an action. Yeah, that’s more likely where the state is the response you get back from the tool call.

Now you have a different way of designing your algorithms to take credit assignment into account, and it’s also a little more flexible from a reward perspective. It feels like people are moving in the direction of model-based rewards, where either an LM is a judge, the judge sees the correct answer, or it has questions that it’s supposed to verify as properties of the response. That’s much more flexible than trying to write these little parsers.

Writing a math parser to check whether a math question is right isn’t that easy, because there are so many edge cases. You want to handle LaTeX support, Markdown, equivalent fractions, and so on. Just let a model do that. Don’t have a 2,000-line Python script that does it. Let me clarify: a math parser to verify that the math is right has to have a LaTeX parser inside it.

A lot of models naturally think in LaTeX because they’ve been trained on a lot of arXiv LaTeX. If you’re doing something like R1, people say, “Oh, math is easy to verify.” The “easy to verify” solution is usually still a very long piece of code that has to handle lots of annoying edge cases, and even then it’s only 98%.

It’s a free-form response, so there’s not only one way to write an equation. If you have 2 valid mathematical expressions that are equivalent, but they’re also symbolic, you need to verify that the 2 symbolic expressions are correct. One might be written as code, one might be written as LaTeX, and one might be written as words. You can’t cover all of that with literal pseudocode. The parsers try to cover a lot of these cases.

That’s also why you see models put \(\boxed{}\) around their final answer so often. It’s one hack: it’s much easier to verify the right piece of information if you know exactly where it’s going to live, rather than having the model say, “The answer to the question is 4.” Then you have to parse away “the answer to the question is” and throw that out.

Deterministic rewards are nice if you can get them to work, but they’re also really painful and pretty hard to generalize across domains. For math, the easiest case is when the final answer is an integer and lives in the same spot—there’s a box where it’s going to be an integer. This is one of the reasons everyone used GSM8K for so long: it’s mostly integers. I think it’s all integers. Multiple choice is super easy to verify, too, but anything that’s a little more flexible is where rule-based rewards start to break down.

The model-based direction seems pretty promising, and I think it’s underexplored: What if you use an LM as a judge in your RL loop? Anthropic has been talking about this for a long time through Constitutional AI. In that case, it was less about the LM judging and giving a direct reward to the model, and more about training a reward model that was doing token-level advantage estimates, which is the PPO way of doing it.

It seems like you can do that for GRPO, too, and for other flavors of RL where you can incorporate a full reward model. The reward model can basically be an LLM that’s fine-tuned to be more calibrated and to have the right kind of range of responses. But it could also be a reasoner, or something that’s able to do tool calling. There’s no reason why the full power of LLMs can’t be applied to the process of evaluating whether an answer is correct or satisfies a certain set of criteria.

That’s the direction I’m most excited about: really pushing beyond deterministic, rule-based rewards into more flexible approaches. I think you want to do this both at a broader level and at a more granular level. That paradigm isn’t going to work super well with token-level rewards, but I think it does work with turn-level rewards. Can the LM verify whether a certain search query was useful? Sure. There are a lot of these questions that are pretty granular, and an LM can basically nail them all the time if it’s a good enough LM. You decompose the problem, and you can incorporate that into RL with that sort of framework.

Awesome. I think that was all the topics that we had prepped. Will, I think you’re also pretty good on that. Obviously, it’ll take some time to figure out Claude 4. Anything you want to plug? We already talked about your talk, I guess, coming up.

Sure. I’ll be at AI Engineer on June 4, in a couple of weeks.

Yeah, coming up. Your track is particularly interesting.

Yeah, mine’s going to be a lot of fun. I’m also collaborating with Kyle Corbitt from OpenPipe to do a course. Both of us have open-source projects that are agentic-RL-focused, and we’ve been friends for a while. We’re trying to do something a little more structured as a way of getting information out to the world for people who—I think we’re especially thinking about practical use cases for agents and helping people, giving people an outlet to learn more about how this stuff works. More coming soon about that.

Awesome. Well, I think that’s it. Thanks for coming on, Will.

Yeah, thanks for having me on at very short notice. I’m glad we could make this happen.

We'll do part two with Kyle and do a full Prime Intellect thing whenever you guys are ready.

Awesome. That'll be fun. Great. Awesome.