RL 微调实战手册：CoreWeave 的 Kyle Corbitt 谈 GRPO、评分标准、环境与奖励投机

Today my guest is Kyle Corbitt, founder of the reinforcement learning and custom fine-tuning company OpenPipe, which CoreWeave acquired last year.

I open this conversation with a bit of a confession. I've done a lot of supervised fine-tuning work over the last few years, both for Waymark in the early days of getting GPT-3 to write decent video scripts and for research projects such as the Emergent Misalignment paper. But I've done essentially no hands-on RL work, both because my perception has been that frontier models are probably my best option in any case and because I'm afraid, perhaps irrationally, of reward hacking.

Kyle says that while it may or may not be worth the extra work and slower iteration time, he does believe that using RL on an open-source model probably would deliver me better performance and would certainly reduce both latency and inference costs dramatically. With that motivation in mind, Kyle proceeds to offer a master class on all things RL, which repeatedly challenged my premises and, in multiple instances, updated my understanding. He explains how RL differs from SFT in terms of the weight updates it makes to models, how this difference makes RL fine-tuning less likely to cause catastrophic forgetting, what distinguished the DeepSeek GRPO algorithm from its predecessors, and what additional improvements on GRPO people are using in industry today.

We talk about the distillation strategies that Chinese labs are using to fast-follow American frontier models, and he argues that their use of LLMs as judges in the context of RL post-training is a bigger deal than supervised fine-tuning. He also explains why he thinks compute is the primary constraint preventing Chinese companies from catching up, and why he believes that we're already in a recursive self-improvement loop. He describes the cottage industry of reinforcement learning environment companies that has sprung up to serve frontier labs and why, although it isn't a good business to be in for now, he's declined to invest in any of them.

He surveys the use cases that are most commonly deployed by CoreWeave customers and offers a lot of advice on how to run RL in practice, including how to develop and iterate on evaluation rubrics, whether to train N models for N tasks or a single model to perform multiple tasks, how the flagrant nature of reward hacking makes it relatively easy to deal with, at least when you're focused on specific narrow tasks, and how Coreweave's use of Laura adapters drives efficiency and convenience for their customers.

Kyle is both a technical expert and a successful commercial practitioner, and from start to finish, this is a high-signal conversation on a classic training technique that has become an industry unto itself. I hope you learn as much as I did from CoreWeave's RL fine-tuning guru, Kyle Corbitt.

Kyle Corbitt, founder of OpenPipe, now, after an acquisition, leading the serverless training team at CoreWeave, welcome to The Cognitive Revolution.

I am super excited to be here. Thank you.

I'm excited to have you. This has been a long time coming since we met almost a year ago now, and I'm glad to finally be doing it. That's all on me, by the way, just so everybody knows.

You are a specialist in reinforcement learning. What I want to do in the next hour and a half or so is get a comprehensive survey, crash course, and rundown of what is going on in reinforcement learning: how we should understand it, what the techniques look like, who's using it and where and for what purposes, who's having success and who isn't, what makes a difference, and all those things.

I was going to start by telling you my story very briefly and then allowing you to react to that and tell me if I'm way off base or not. My story in short is that I've done a lot of model fine-tuning over time, mostly on managed platforms, not so much on open-weights models—just a little bit of that—and more so on the OpenAI platform. But it has been almost entirely supervised fine-tuning, with very little reinforcement-learning fine-tuning.

The story I'm telling myself, which you're invited to pick apart, is that these days I can increasingly just use base models with few-shot prompting, and that's getting me a lot of what I need. But even before that was possible, the problems I was working on in the context of my company, Waymark, were taste-driven problems. We always felt like we'd be better off going to our creative team and saying, "Hey, give us 100 great examples. We'll fine-tune on that and hope that the AI can follow your lead."

That seemed preferable to trying to go through some seemingly more complicated, maybe more powerful, but harder-to-wrap-our-heads-around notion of, "Well, if we get the AI to do it, and then we compare and score it, maybe there's an LLM as a judge." We were kind of like, "I don't know." It feels a little bit like a shell game, and I'm not sure where I should invest or how much I should trust that process, whereas I know that if the AI is imitating my creative team, there's some decent true north there.

The other thing I'm somewhat afraid of, although I'm not sure it's a big problem in my context, is reward hacking. I am kind of afraid of reward hacking in general. How would you advise me on whether or not I'm making a good decision? Should I be using reinforcement learning, or am I thinking about it the right way?

That's a great question, and I think it's one that lots of folks think about. Maybe my first question for you would be: how were the results you were getting from your existing process? You mentioned, first of all, that these days you mostly just do prompting, but when you were doing fine-tuning with SFT, did you feel like you were seeing the models improve substantially?

And this is a very high bar, which I imagine they wouldn't clear, but did you feel like they were behaving as well as your creative team and matching the quality of those examples after post-training?

I would definitely not say they were matching the best work that our creative team could do, but there was definitely a notable improvement on the base model. Our typical complaint, which would definitely vary through different generations, was that more recently the model was able to do the job perfectly well, so to speak. I think that's true today, too, with prompting, but there were few moments when you thought, "Damn, that was awesome. Incredible turn of phrase. Nailed it."

It really nailed it in the way that sometimes you just get something from the creative team that's like, "Oh, wow, that was a really good creative idea that impressed me, surprised me, and delighted me." I wouldn't say we see too much of that coming from models, even today.

That was going to be my next question. Even with the latest frontier models, that sort of spark or wow moment sounds like it's not something you see commonly.

Rarely at best, I would say.

Here's what I think. I think it is likely that you would have been able to get better performance out of the models with reinforcement learning than with SFT. There are a few different factors here that muddle it.

One is that OpenAI's support for RL was half-hearted at best at any given point. I think technically they still do it, but that entire model-customization platform feels very much in maintenance mode at this point. So on that front, that might simply not have worked.

In a parallel universe where you were using an open-source model, like a Gwen model or something like that, then I would say with a fairly high degree of confidence that if you're able to get decent results out of SFT, the ceiling of the best results you can get with reinforcement learning is going to be higher. And that's true even if the data you're using for SFT is high-quality human data.

The reason is that the whole trick to RL—the whole reason RL works, or the reason people invest in RL—is that it turns out to really matter how well your data distribution matches the model's standard mode of thinking, or just what it's picked up from pretraining. What RL gets you is that it is working within those channels that are already carved quite deeply within the model.

When you work within those channels, you can get a lot further because you're not trying to overwrite what it's doing. You might say, "Well, overriding is what we're trying to do. We're trying to get it to do something it's not good at," which is fair, but it ends up being quite destructive.

It's actually really interesting if you look at the weights. If you're doing SFT, even with very few examples and a very, very low learning rate, it's just throwing the weights all to pieces, and the average differences are so much larger than when doing RL. That's a big part of why you get this catastrophic forgetting, because you're overriding other pathways and trying to get the model to do something quite different from what it was trained to do.

Whereas RL is going to let you stay in those grooves and get a lot further. So yes, I do think that would have worked.

Now, in your specific case, would it be worth it? Would it get you to a place where it's like, “Oh, this is better than just using the frontier?” My guess is probably not. So, I think concretely for your task, if the trade-off you're making is, “Hey, we're going to take an open-source model and use RL to try and make it better at this,” versus, “Hey, we're just going to take whatever the best off-the-shelf model is and do prompt engineering.”

We're allowing ourselves to expand to the best frontier models. At that point, I suspect for a creative writing task, you would end up in a position where you're better off using the frontier models. And yeah, we can sort of get into this: there are definitely tasks where I would say the exact opposite and say that RL could do well. I would also say that this is obviously all dependent on the amount of compute.

I think theoretically, anything's possible if you buy yourself a data center and spend a couple billion dollars on this task. You would be able to surpass the frontier, but the trade-off point would be fairly long, I suspect, along that curve for a task of this shape.

I'd like to understand this “grooves” thing better. I do know what you're gesturing at. When I think about how much the weights change with fine-tuning, I usually think of that as more of a function of some sort of divergence penalty, some sort of tethering of the model as it's evolving to the base, to the starting point. I think you can do that on any kind of fine-tuning, right?

So how is it that if I have a similar divergence penalty term in my loss function, why is supervised fine-tuning more destructive than reinforcement learning?

Yeah, no, that's a totally fair question. What you're talking about is a term called a KL-divergence penalty, which is an auxiliary term you can add to any loss function. It doesn't actually prevent the model weights from drifting; what it prevents is specifically the log probabilities that are generated at each token position from drifting too far from the base model. This is often considered best practice because it can help prevent catastrophic forgetting and moving too far away.

However, the fundamental issue is this: there are often different ways to get to the right answer, right? The easiest example here is if you're talking about a reasoning trace where you're doing a math problem and training this model with RL to solve the math problem. There's probably an infinite number of ways you could reason through from a problem description to the answer.

Some of them are paths that the model is already comfortable with, where it's like, “Oh, these 8 tokens in a row—even the base model you're starting from would have generated them anyway.” Then the next token, yeah, maybe it would have gotten that one wrong. So there, the learning signal is teaching you to move that one slightly.

Fundamentally, what RL structurally optimizes for is changing the fewest tokens—the fewest log probabilities—necessary to get to the right answer. Whereas with SFT—say, if you're distilling a larger reasoning model into a smaller model, particularly when the smaller model had different pre-training distributions—you're not respecting those pieces of the reasoning that it would have gotten right anyway.

You would expect that its built-in intuitions or inclinations are different. You're overriding the whole thing with the reasoning from the larger model. By overriding the entire thing, this is potentially quite confusing for the backpropagation algorithm, because backpropagation is just seeing, “Oh, all of these tokens need to change.” Maybe some of them didn't actually need to change.

Maybe the direction the model would have gone with this token was also fine, and so you're changing out the weights to get to this new one. Really, there was this other token that was much more important and did in fact need to change to get to the right answer. But that one's just kind of mixed in with all these other random, unrelated changes.

That general intuition generalizes to other task shapes as well, including creative writing, where maybe there are 2 different ways to phrase something and they're both fine, right? The model would have chosen one, and your creative team chose another, and they're both okay. You don't really want to waste your model updates on changing something that was already fine, because every time you update the weights, there's potential for catastrophic forgetting and off-target effects in general.

You want to really direct them to upweighting the things that the model wouldn't have gotten right on its own—or, more specifically, would have gotten right on its own only very rarely—and focus your updating budget on those. The KL divergence doesn't give you that. If what you're doing is just penalizing KL divergence, it doesn't distinguish between things the model was already doing fine and you just happened to have a different way in your training data, versus things that the model really was getting wrong.

When you described something more specifically as something that the model can't get right, but that it rarely gets right, that's key. When we do things like GRPO, you've got to have at least 1 right answer, right, to have any sort of advantage. I guess it also depends on whether you're doing binary scoring or some more rubric-based evaluation.

But I guess several different questions are coming to mind at once. Can you give me a little bit more intuition? Maybe we could do this for GRPO, and you can describe—I’m not sure if GRPO is still the hotness that it was a year and change ago. I'm also not entirely sure if that was something that broke out for kind of memetic social media reasons or if it really was a huge advance over its immediate predecessors.

Can you give me a little bit more intuition for this? I understand that in this algorithm, we are running multiple rollouts. Some of them are going to get to a right answer, or, if it's a rubric score, they're going to get a higher score than others. Then there's a computation that creates the group-relative advantage, which is to say, we want to shift toward the patterns that gave us the right answer or the higher-scoring answer.

How is it, though, that that still ultimately goes to a token-by-token thing, right? How is it that if I have 8 different chains of thought and they're all kind of different, and at any given token position we might even have very different parts of speech, right? At a token position, it could be a preposition here and a verb there and whatever.

Well, in very different moments in the chain of thought, my understanding is that the advantage calculation does still ultimately cash out to token-level advantage. So how is it that—where’s the alchemy of why this translates in the end to really only updating, making changes to those tokens that really mattered? How is it? I’m missing a little leap of logic there.

Let me take several parts of this question, and I’ll finish on the one you were getting at there at the end. Hopefully, that’ll give you the chance to ask follow-ups if my explanation doesn’t make sense.

First of all, I think the reason GRPO—specifically, that algorithm and that acronym—concretely took off was not necessarily because it was a big quantum leap over what came before. It was because DeepSeek did a lot of engineering work around actually scaling it and released an actual artifact model that worked really well with it. That was the reason why there was a whole constellation of other algorithms that probably would have worked just about as well.

There was one that came out a little bit before called RLOO, which basically is the same as GRPO and likely would have worked just as well if you had scaled it. After GRPO, very shortly after—in other words, within a few months, certainly after our one release—there were numerous improvements made upon it, which really do deserve their own algorithms. There was a paper called DAPO. GSPO came out from the the coin lab, I believe, and then Sispo was another one that came out shortly after. Those are all significant improvements.

There are also a bunch of minor tweaks that don’t even have named things. I would say that the algorithm people use today in practice is actually as far away from GRPO as initially described as GRPO was from what came before it. We all still call it GRPO because that was the name that stuck.

Moving on to how it actually works, I think it’ll be helpful to build your intuition about how the advantages are calculated and everything. Maybe I’ll talk first about what came before GRPO, because GRPO is interesting in that a big part of its development was that it threw away something that everyone had used before and that some people still use.

The spiritual grandfather of all the reinforcement learning that people do on LLMs is an algorithm called PPO, which was developed by John Schulman in 2017, I believe. It actually predates LLMs—or, I mean, big LLMs—and was used for games and things like that.

The key thing about PPO is that you have your policy, which is what you call the model you’re training. It’s taking a bunch of actions, and every time it takes an action, you have to score how good or bad that action is. If it’s a good action, you want to update your weights to make it more likely to take that action. If it’s a bad action, you want to update your weights to make it do less of that.

Importantly, this happens on an action-by-action basis. Your reward in PPO can be very long-term. You could get to the end of a very long sequence of actions before you finally find out what the result was. Commonly, this was used with games, so you might say, “At the end of the game, or after a minute of gameplay, what’s my score?”

PPO does a few different things, and it’s building on older work as well. There’s an algorithm called REINFORCE that’s trying to solve the same problem. PPO adds some extra terms to keep it stable and keep it in a sort of trust region, where you’re hopeful that the model hasn’t changed too much as you’re updating it.

The key thing that PPO does—and this is not unique to PPO; it comes from work older than PPO—is that you want to calculate the advantage at every single action. Every time it takes an action, you want to say, “Was this a good or bad action?”

The way it does that is by training a couple of different models in parallel. You have the policy model, which is your normal model generating the actions. Then you have a separate model called the value model, or the critic model.

The value model is predicting, based on the set of actions up to this point, what it thinks the score is going to be in the long term. It’s basically predicting, for this action, “What do I believe is the value of this action? What impact will this action have on the score in the long term?” It’s predicting that for every single action in the sequence.

Eventually, you do get to see what the actual score is. If the score ends up much higher than you expected, you can say, “Some of these actions were clearly much more valuable than we expected.” If the critic model thought it would have a low score and it actually has a high score, then you want to make it much more likely that this action happens in the future.

Now, moving on to GRPO, the key difference is that instead of figuring out the value of any specific action, we throw away the value model. Actually, before I go into GRPO, I should mention that this all translates directly into LLMs.

People tried a lot of different translations, but the one that most people use—and it’s the simplest thing that works—is that every single token generated is an action. We’re using the exact same concepts as before and saying that the state up to that point is the full context, this token is an action, and the next token is another action.

With GRPO, it turns out that figuring out the value model and keeping it up to date is painful. It’s tricky to get right, and it’s another set of hyperparameters you have to tune. You have to get this model updated, or training doesn’t work well.

What GRPO did—and they were not the first ones to do this, but they get the credit because they were the first ones to do it at scale and prove that it worked well—was say, “We’re just going to completely throw away the value model.”

The way we’re going to figure out whether a given trajectory of actions is better or worse than what the model would have done otherwise is to run a bunch of them in parallel. With the exact same setup and the same initial conditions, we’re going to run 4, 8, or 512 different runs in parallel. There are lots of different hyperparameters to tune here as well. Then we’re going to see how often the model succeeds and how often it fails.

The reason we want to do this is that you don’t want to just run a single run. Let’s say we’ve thrown away the critic model, we do a single run through with GRPO, and we get a score of 1. It got it right. You don’t know from that run whether the model would always get this right or whether this was a 1-in-a-million occurrence.

If you naively update your model because it got it right, but it would have always gotten it right, you’re reinforcing a spurious correlation. It made some random choices, but those choices didn’t affect the score at all because it would always have gotten the answer right. If you update those random choices it made, you’re just moving around in a pretty random direction.

What GRPO lets you do is say that the advantage we allocate to each of these tokens is based on how much better this run did than the average. Really, what you want to compare it to is the average of what would happen if you ran the current model infinitely many times on this. How much better did this run do than that average? Obviously, we’re not going to run it infinitely many times, so we approximate that by doing it N times.

Getting to the end of your question, you’re right: when we’re actually updating the model weights, we’re doing this on a token-by-token basis. Somehow, we have to say, for every single token, that we want to update the weights so that this token is more likely if the advantage is positive, or less likely if the advantage is negative.

This is a big problem in reinforcement learning called the credit-assignment problem. What you really want to do is assign credit and weight just the key tokens that were critical to the solution going right, and not upweight the tokens that would always have been right and didn’t really contribute anything to the solution.

And so the key insight of GRPO is to do a very unsatisfying thing and kind of just punt on that a little bit. It's not a full punt. What you do is look at how likely every token was to be produced, because you're sampling at a high temperature when you're doing this. Some of the tokens it produces are very common, and some tokens are not very common.

Basically, you say, “Hey, if I got a high score, then I want to give more credit to the tokens that, just by random chance, were less common.” I assume that if my score is much higher than the average score across the entire group, then it probably was because there was some rare thing that I did in this case that I didn't do in the other cases. That rare thing led to me doing well.

You do the exact same thing in the opposite direction. If I get a much lower score than the average in the group, then the rare things are the things that I'm going to penalize the most, because I'm thinking, “Hey, that's probably what put me there.” Now, you could ask the question: There could be many rare tokens. If you got thousands or tens of thousands of tokens in a reasoning trace, how do you decide which rare token is most important?

You don't. You just throw up your hands and say that all the rare tokens get up-weighted the same way. This is, like I said, a very unsatisfying answer. I think that's one of the reasons why there was an almost 10-year gap between PPO, which had this value model that tried to determine things on a token-by-token basis, and GRPO, where it's like, “Hey, we're just going to throw that all away because it feels wrong. It feels like it shouldn't work.”

In practice, it does, though.

Is the intuition there kind of like—I studied this a little bit, but not in enough depth to be confident—I'm imagining that as we go through a chain of thought, there are critical tokens where you're either taking the right path or the wrong path. Then there are probably a bunch of tokens that follow once you've made that critical decision, which are all naturally going to follow because that's just the structure of language.

You're trying to zoom in on, or isolate—or focus on, if “isolate” isn't the right word—the critical decision points in that trace. The ones where the model was least confident. You're trying to emphasize those, maybe, if that's the right word.

Yes, that's exactly right.

Okay. Interesting. DPO is basically a similar thing too, right? But you had to have pairs where you said, “I like this one better than the other one,” as opposed to a ground truth or a score. But it's a similar mechanism, right?

Yes, there's definitely a lot of overlap in the math and the intuition there.

Is it worth getting into some of the finer points that have emerged since GRPO and made it even better? Not in a super mathy way, but what additional insights have people brought to bear since then?

Yeah, we can talk about it briefly. It's a bunch of small things. One open question was, “Hey, how do we do length normalization?” The original math in GRPO actually structurally advantaged very long thinking traces and, in general, just long generations, because it didn't normalize by the number of tokens.

Basically, if you had a batch of, say, 128 different completions and one of the traces happened to be 5 times as long as the others, it ended up with 5 times as much weight in the model updates as the others. People have had pretty good success with basically down-weighting that to average it out.

Sispo is a really cool one. It basically just changes the way you're doing the clipping. PPO—and then GRPO inherits this—has a specific way of making sure that the weights don't stray too far in any one round of updates. There was this new technique called CISPO that was released maybe 6 months later or something like that.

Basically, it puts the clipping in a different spot, which lets the model discover much more quickly those very high-value but rare tokens. It allows those to update the weights much more if there's a very high score, while not allowing the weights to update too much. Then there's a stack of probably half a dozen little tricks like that that people have developed to make the algorithm both more stable and converge faster.

That's been a great trip down the rabbit hole. Popping out now and trying to think about what it all means. Obviously, the huge thing about reinforcement learning that we've seen time and time again—and it's really happening now—is this latest Erdős problem that's been solved in the last 24 hours, or at least reported. Roon just said something like, “This is the first time that everybody in the math community is super impressed.”

The key point that I'm getting at here is that reinforcement learning has the ability to take a model beyond what the available training data has to offer and teach it something new. This is where we get superhuman performance. Now, how does that happen?

You had talked about the grooves, and by focusing in on these key decision points rather than just mashing every token, you're playing to the model's established strengths. But clearly, there's also something happening where, at scale, reinforcement learning is teaching qualitatively new capabilities to the model. So how should I think about that? In other words, how are we making that move?

I mean, you could argue with me if you think this is wrong, but I take it that everybody has kind of come to accept that this is where the superhuman performance comes from. I don't have a great intuition for where we're making that move from playing to the model's strengths, staying in the groove, focusing on what matters, and reinforcing what it already knows—or has at least some instinct for—into this qualitatively new regime where now we're solving open math problems.

Yeah, it's a great question. One caveat I would give here is that, unfortunately, reinforcement learning for LLMs has definitely matured in an era where nobody's publishing anything, except for some Chinese labs to some extent. So I think we have very little insight into the specific techniques that, say, OpenAI, Anthropic, or Google are using to train these models.

So, that’s the first caveat: this is definitely speculation. What I would say after that is that this is a common confusion or dichotomy that people have about RL: “Is RL teaching new things, or is it just surfacing things that were already latent in the model’s distribution?” From a very pedantic technical sense, yes, it is only eliciting things that already existed in the distribution.

However, the distribution of tokens that a model can produce is literally the set of all possible tokens, in the same sense that the distribution of works that a million monkeys on typewriters could produce includes Shakespeare, right? Everything is already in distribution, definitionally. At any given position, there is a chance that the model can produce, with however small a probability, a given next token.

The whole game, of course, to avoid the situation where you’re just waiting for your million monkeys to type out Shakespeare, is to get your initial distribution as strong as possible, so that it requires less random guessing and random rollouts in order to find those new and useful behaviors. Which is why pre-training is still super important, even in the sort of RL regime we’re in right now, because you want to start from a place where the right patterns have a greater-than-negligible chance of showing up.

That said, I think you probably can get to superhuman performance on a composite task, like a very complex math proof, even without reaching a place where no human could possibly have understood or generated this. One thing the models are very good at is going on these long expeditions and fishing trips—going very deep down a specific rabbit hole. Maybe they’ll take that rabbit hole further than any human would because we’ll lose the context.

I think we’re at a point with a lot of these frontier models now where their working memory is larger than any human’s working memory. They can explore these rabbit holes longer than a human mind could. Even if every individual step is something that seems plausible to a human, if a human had all of that context up until that point, it’s very hard for a human to hold all that context in their head in practice. So that’s one place we could get to superhuman performance.

But, yeah, in general, you can get to superhuman performance even without that, just because you could randomly discover or randomly surface a token that does something clever that no human would have done.

How do you relate this to what I think of as metacognitive behaviors? In the original R1 paper, there was this “aha” moment that they published, and I usually present this in my AI scouting reports as kind of the two parts from that paper that I put together.

One is what you’re saying: that, to some extent, the length of the chain of thought just naturally grows throughout the training process. I have mostly interpreted that to date as the model learning that it’s valuable to think longer and getting right answers more often when it’s thinking longer, so thinking longer itself is being reinforced. But I’m also hearing you that, at least in that original one—

Yeah, I mean, to be clear, both things can be true.

So my other side-by-side there is the “aha” moment, where the model is solving some math problem and realizes that the way it had been doing it was flawed. Now it recognizes that there’s another way, and it takes a step back and approaches it from a different direction.

Clearly, we’re seeing a lot more of this persistent, resilient, try-again problem-solving in frontier models. Again, somewhere deep in the long tail of the internet, somebody has written out how to do that. So it’s a little bit in the pre-training. There’s supervised fine-tuning, at least sometimes, in these recipes as well, where you could potentially try to seed the kind of metacognitive strategies that you want. Then it seems like reinforcement learning is doing a lot to bring that forward as well.

How do you think about what’s really driving that? Are we seeing things that are kind of alien problem-solving? Are we seeing, and should we expect to see, reasoning approaches that are not inspired by humans emerging through RL over time?

Yeah, I think that’s an interesting question. I personally don’t really feel like the so-called “aha” moment—or “wait,” which I think shows up all the time—is alien. The models will say “wait,” and that’s sort of a code to say, “Hey, let’s explore another direction.”

I’m not sure. That doesn’t feel alien to me. If I’m introspecting my own chain of thought, or just having a conversation with someone, that behavior doesn’t feel weird. It feels very natural. Obviously, reinforcement learning is bringing it out because it’s also true that this is the kind of behavior that, in retrospect, makes sense.

It makes sense both in the sense of, “Oh, yeah, that makes sense,” and in the sense that this would not naturally come up in the pre-trained data all that often. Usually, if you’re writing something on the internet and you have a new idea, you’re not going to put out a chain of thought saying, “Oh, wait, I have this other idea.” You’re going to condense it and just put your final thinking there. But I’m sure it comes up sometimes, like when you’re in a chat history or whatever.

Anyway, I don’t think that’s surprising to me. The short answer is that I have not seen strong evidence yet that they’re thinking in ways that are totally foreign or totally alien, and that are hard for us to introspect or follow as humans.

There’s a separate question: will we see more of that? I think, in the limit, it seems very likely that the ideal form of cognition for these artifacts—and just the ideal form of cognition generally—would be something that looks very alien to a human. As we put more effort into RL and perhaps come up with better techniques to explore more on that sort of explore-exploit spectrum, it would not surprise me if we do start seeing more of that. But I haven’t seen it yet.

Yeah, this is a bit of a different dimension in which it might arise, but just in terms of an intuition of what that might look like, there was the Coconut paper from Meta, maybe a year ago or something, about thinking in latent space. Instead of cashing a forward pass out to a token, I forget exactly what the decision mechanism was for when it would pass its last internal state back to the next position as an embedding versus when it would actually cash out a token. There was some decider mechanism there.

At least for a while, it could and would just loop on its own internal states rather than emitting and appending a token. They found that it was much better at graph-search-type problems that benefited from the ability to parallelize. It seemed like it was able to effectively run multiple branches, going down multiple paths in parallel in latent space, because it was able to chew on these things rather than having to spit out one token.

I get a little scared of those kinds of innovations, honestly, because I kind of want to know what my AIs are thinking, and that doesn’t really lend itself to that. The other one that comes to mind is something I’ve been quite confused about as well. You might be able to shed some light on it.

Apollo Research, when they did—I think it was O3, maybe it was O1—testing, got access to chain of thought, and they reported that the chain of thought was starting to look kind of bizarre. You remember the “disclaim, disclaim, vantage” thing—that weird sort of internal dialect? I had kind of assumed that there was maybe a chain-of-thought length penalty. If the original GRPO was accidentally rewarding long chains of thought, it would also stand to reason, because compute is scarce, that we want to keep these chains of thought as tight as possible. But maybe we overdo that, and we’re just starting to see weird dialects emerge.

How far off the rails have I gone in telling myself that?

Yeah, I think it’s an interesting question. At some level, I think we need to treat this as an empirical question: what do we actually see working?

It’s interesting—the idea of a model that could self-correct or reason was not something that was invented with OpenAI, Strawberry, or O1, or whatever. There was a lot of research in that direction before, and there were a lot of folks doing that work. There was a lot of work on text-diffusion models, where the intuition was that they would go through this reasoning in a latent space.

There was also research on prompt compression, and perhaps also on reasoning models, that still used autoregressive tokens. But instead of constricting them to specific tokens in the vocabulary, it would give you the full embedding space, where the model could basically use tokens or words that don’t correspond to a specific token embedding.

It could dynamically use different embedding vectors that don't correspond to words. There was even a lot of speculation after o1-preview came out that OpenAI had worked on something in that direction. I do think it's interesting that, in practice, as far as I'm aware, with the OpenAI reasoning models and the similar reasoning models from Anthropic and Google—and certainly all of the open-source reasoning models that work well at all—those approaches have not been taken.

It's pretty much just the very simple, very dumb approach where it's going to be doing chain-of-thought reasoning in the normal token space and mostly using human language. I think probably what that tells us is that they're getting a lot of value out of pre-training and staying relatively close to those patterns, relative to how far they could go.

Obviously, as we see more evidence of the kind you're talking about, where we're looking at actual reasoning traces from the frontier models and they're diverging more from something that's easy for a human to interpret, I think that would be quite convincing evidence for me that it stops looking like that. So far, at least, it seems like, if anything, we've been moving more in the opposite direction: people assumed there'd be much more reasoning in the latent space, and for whatever reason, that hasn't been as productive an approach.

Yeah, I kind of maybe overupdated on that one Apollo Research report because it was kind of alarming to me to see the very, very inscrutable dialect that I couldn't make a lot of sense of. But reports since then have been much more reassuring that, no, we don't like to show it for competitive reasons and so on, but the chain of thought is still pretty readable. That's been the pretty consistent report.

What do you think all this means for the future of competition? We've of course had the distillation attack report from Anthropic. It's generally understood that, especially internationally, Chinese companies in particular are trying to take certain shortcuts by getting outputs from whatever frontier models they can get outputs from and then training on those.

Is that, I guess, for one thing, something they can use? Is there a way to turn those outputs into a reinforcement learning approach? You might think naively that they would just be doing supervised fine-tuning on that, but as I've heard some of your analysis here, I'm thinking, well, actually, maybe not. Maybe they're actually using those targets as some sort of way to evaluate and then still running a more reinforcement-learning-based algorithm, with clause answers as the standard that it's going to be judged against, rubric-wise or something. What do you think that is actually looking like, and how much of frontier performance can distillation actually recover?

Yeah, that's a good question, and I guess, again, an empirical one. A few different thoughts. One is the most natural way, in my mind, to use frontier models to bootstrap your own near-frontier models with reinforcement learning, in general, is to use the frontier models as judges. They're very good at that, and that sidesteps the issue that you can't actually get and train on the chain-of-thought traces directly.

So if you just have a standard, "Hey, we're going to use a frontier model as our rubric, and we'll have our model do generations that get judged," that's a very productive way. In the blog post that Anthropic made about the distillation attacks, as they call them, from frontier models, they specifically called out—I mean, they didn't say the breakdown of what all these were being used for, but they did say that one of the uses they included in their general bucket was using their model as an LLM-as-a-judge for other outputs.

So that's one way where, yes, I think very clearly you can use the existence of a high-quality frontier model to improve your own. The nice thing about that approach as well is that you get both those benefits of staying kind of in your own distribution because you're just using it as a judge, not doing SFT, and also, in general, with RL, you can train the model under training to be better than the teacher model that way.

So it is a path to getting frontier-level performance or pushing the frontier, even if you aren't starting from a frontier model. We know this is true in our experiments. This is clearly true from the frontier labs because we see OpenAI and others using their N−1-generation model as a judge when they're in the process of training the next version of their models. So that's the most natural way.

As far as using distillation directly, SFT-style, yeah, I'm sure that does happen. I would imagine that happens at a relatively low volume and fairly early in the process, before you do RL. My guess is that it's not that valuable. It's a shortcut that lets you use less compute, but not orders of magnitude less compute relative to just doing RL.

And particularly as we see frontier models start to shut down their APIs more, which I think is the more interesting direction to investigate or explore, we're already seeing, of course, starting with the reasoning models, that we're not seeing all the tokens that are produced anymore. There are certain models—yeah, they're cutting—they're not letting you see all the logprobs. They're certainly not letting you see the prompt logprobs.

For instance, OpenAI will only let you use their models through Codex, and I expect we'll see more of that over time, not less. I expect we'll see much more locking down of models to specific use cases and specific product surfaces for multiple reasons, but a big one is that it makes distillation harder, especially distillation in out-of-domain areas that aren't within that product surface.

So, I guess, translating that to expectations, one story you could tell, which I've kind of been telling myself recently, is: Why are the Chinese models spikier or more apparently benchmarked, or whatever? I had been kind of thinking, well, they're probably doing a lot of supervised fine-tuning on frontier model outputs, and therefore they're maybe not developing some of these more persistent problem-solving, metacognitive behaviors that really allow the model to generalize robustly out of domain, right?

I might not care so much about that exact question. What I really care about is, in the chain of thought, how good is it at breaking down and coming at problems from lots of different directions? But your account so far has kind of gone the other way—or I'm not sure if it's the other way—but I'm not now quite sure. That story doesn't ring so true anymore if you're saying they're probably not doing that much supervised fine-tuning, and it's relatively early in the process, and it's a compute saver, sure, but it's not a huge difference-maker.

So what is the difference? Are they just not so good at RL, or do they just not have so much compute? Why are the Chinese companies not able to match the American frontier companies right now?

Yeah, so I guess two questions there. I think that the first one—or, I guess, the second one you said, why can't they match—the high-order constraint seems very likely to be compute, where they just can't put as much compute into each training run as the closed-source leaders in the US. Now, they are putting similar or, actually, in many cases, more compute into it than open-source models in the US, which is why they have the open-source frontier. But yeah, I think that's sort of the higher-order bit on that.

The reason why they feel more benchmarked—I don't know. This is speculation, but I actually don't think it's related to how much RL or distillation they're doing. I think it's kind of a much simpler business analysis, which is: If you're a new lab that has relatively low name recognition and you don't have a ton of usage right now, the incentives are far, far higher in relative terms to benchmark, right?

Because no one's even going to try your model unless you come out with very impressive benchmarks. You don't have a built-in constituency for it. Whereas if you are Anthropic, Google, or OpenAI, sure, it looks good to have high benchmarks, but you already have millions or hundreds of millions of users, and those people are going to feel the difference and tell their friends about it.

They're going to be using your new model anyway. So there's less incentive that you have to look best on benchmarks if you can trust that you're going to have a bunch of people using it anyway, and they're going to feel that it's just better overall, and it'll spread through word of mouth.

You're destroying my galaxy-brain takes one after another. I love it. Yeah, I mean, that makes sense. I guess the other angle, too, is generally speaking, they don't have big inference businesses, so they're also kind of missing the actual customer feedback that the American—

Yeah, that's also, I think, likely a major factor.

But that would mean if a few things changed, everybody's obviously wondering: Are we heading into recursive self-improvement, and if so, what’s it going to mean? I've seen a bunch of papers from 18 to 36 months ago, with GPT-4-class models basically trying to do recursive self-improvement, and generally speaking, they would get better for 3 to 5 rounds and then level off. Yet there's at least some expectation among people who've been right about a lot of things that this could go the other way in the not-too-distant future if models become smart enough to recursively self-improve in multiple ways—not just critiquing their own outputs, but also finding better architectures for themselves.

There could be a lot of different dimensions in which they might self-improve. I also remember the Anthropic leaked pitch deck from a few years ago where they basically said, “We think the people in the 26 time frame who train the best models might create such a big advantage that nobody will ever catch up.” Again, I've filled in the gaps on that story for myself by thinking, “Well, maybe it's these metacognitive behaviors, this sort of deeper understanding, problem-solving ability, what have you.”

But you're saying, “No, it's probably mostly compute and incentives and lack of an inference business,” which itself is very much related to compute. So I guess the bottom line is, if compute constraints were relaxed, you would expect to see Chinese companies be able to catch up, and you wouldn't expect some sort of runaway dynamic to take hold where that would become impossible.

Oh, I think that catching up right now is mostly compute-gated. It's also capital-gated, in the sense that buying the necessary compute already requires billions of dollars and will require tens or hundreds of billions of dollars soon. So I think there's an open question: How healthy are the Chinese capital markets? Will they be able to make a case that they'll be able to keep their business if it goes really well?

I think that has been a question with prior generations of Chinese tech companies, which might just be hard for them to overcome. So that's one thing. But I don't think any of that means that recursive self-improvement won't matter or doesn't matter.

My belief is that it probably does, and my belief is that we probably will reach it with the current generation or the next-generation models, because we already are in a self-improvement loop. That's what you have to remember: These models keep getting better because we keep running more experiments and then figuring out, “Okay, what are the bottlenecks? Let's solve those bottlenecks.”

Those happen at all levels. It happens at the hardware level, figuring out what's the most efficient way; at the algorithmic level; and at the data level. These are all self-improvement loops already.

There are multiple constraints, but one of the big constraints is just human intelligence. Are the people making those allocation decisions smart enough to make the right bets on what bottleneck to tackle next or what investments to make?

You can totally imagine that if you were to staff OpenAI with a minimum bar of, “You're not allowed to be hired here unless you have an IQ of 180,” they would be able to solve those bottlenecks a lot faster. If they could wave a magic wand and get enough people who look like that, then maybe they could.

I just feel like the bar for recursive self-improvement to take off is actually relatively low. You just have to be better than the smartest human, which is not that smart.

It's a wild time to be alive, that's for sure. It does seem increasingly possible that could happen in the not-too-distant future. I don't have anything more to say about recursive self-improvement, so I was going to move next to the cottage industry of RL environment creation.

People know it's out there, but it's kind of a dark-matter sort of thing. Because there are so few customers, it's not like these companies have much incentive to go talk publicly about what they're doing. On the contrary, they probably have the opposite incentive, right? They know all the customers they can possibly sell to, and telling the world more about what they're selling is just inviting competition that they don't want to have.

So it seems like the rest of us who aren't directly involved in the making, selling, and buying of these environments are kind of left in the dark. What can you tell me from what you've seen about that seemingly rapidly growing niche? How big is it? Who's doing it? What do the environments look like? What makes a good environment, and so on and so forth?

Yeah, no, I can definitely speak to that. I have several friends who are founders of companies doing that, which is not saying much because it feels like half the companies started in the last 6 months are doing that. So, yeah, I think it's an interesting industry.

The general shape is that you come up with some task that seems like it might be economically valuable. Usually, it's these companies proposing the task to the labs. It's usually not the labs coming out and saying, “Hey, we want a shape like this,” although that can happen as well.

You try to come up with some task, and the trick is that you want to package it up as something that is agent-shaped. All of the dependencies can be enclosed. You want to make sure that it's something that is ideally snapshotable, so that's the gold standard, or something where, at any point, you can snapshot it and continue from that point.

Then, of course, something that can be easily graded. Obviously, the ideal thing is if you have a gold standard of what the grade should be. A lot of these do end up being things that you can't score in some absolute way, and in those cases, usually the company will say, “Hey, this is the rubric you have to grade.”

I've also heard that sometimes labs will just ignore those rubrics and do their own rubrics internally because they think they have better information on what good looks like.

These are things like lots of different web workflows: computer use, browser use, and building copies, of course, of all the big apps. You're getting copies of Jira and GitHub, flight booking, and office suites like Google Sheets. You're trying to build environments that copy these, and then you're building that environment.

That's all the dependencies—the database, which is usually SQLite or something. You want something ephemeral. You're also building the scores, and the way it's deployed varies a lot as well, even within a specific company or with a specific lab.

Sometimes the labs require you to ship it all up in a container they can run on their infrastructure. Other labs are fine with you running it yourself, and they will just call your environment, run it, and then you give them the scores back.

The reason why this is sort of cottage-industry-shaped, I believe, is for a few reasons. One is that the labs actually do have at least a weak preference for having lots of different vendors. If one person creates 5 different environments, they're likely to make similar assumptions and similar shortcuts in all of them, and so the signal that the model will gain from mastering all those environments is more correlated than you'd like.

The whole game here is that you want the broadest diversity of environments, so having different people working on it is better.

Another reason why it's sort of cottage-industry-shaped is because this is extremely hard to hire for. It's sort of a piecework-style task where you're building one environment and then building another, but the skill bar to doing this successfully is quite high.

This is what we do internally for our customers all the time as we're building these environments at CoreWeave, which we then use to train models. I have trouble hiring people who can do a good job on this, candidly. It's a very upper-percentile engineer who's able to think through this in a way that actually gets it.

You don't even know if you got it wrong until much later in the process, when you train the model with it and it's like, “Oh, did the model learn general skills, or did it learn some hack on how to just get a high score?” There's a lot to keep in your head as you're doing this.

The people who are good at that are, by definition, smart and frontier-adjacent, and they might just start a competitor to do this instead of joining you as an employee.

So it becomes very, very difficult to scale. Also, the environments themselves are not a super-durable resource, in the sense that all these things get saturated fairly quickly. And so you really have to keep creating new ones. You can’t just keep reselling the same environment to the same lab. They’re probably going to say, “Hey, that environment, for the next model, the model can ace it,” and you have to keep creating new ones.

That’s fascinating. This may be hard to summarize, and I don’t know if anybody outside of the labs would have enough information to really characterize this. But is this a good business to be in? I can see it going either way. I would assume that if you’ve got a good environment, all the labs want to buy it, but at the same time, they’re buying a ton of stuff. How much does your one random thing add to the whole mess of things they already have?

And it’s depreciating, as you said, for you, right? So you’ve got to strike a deal before they saturate your thing and then truly don’t need it anymore. Would you say this is a hot, good place for up-and-comers to go, or would you steer people away from it?

I mean, it’s clearly a good business in the sense that these companies are scaling to tens or hundreds of millions of dollars in revenue in months. So, if you’re asking whether I would steer someone into founding one of these companies, I think it’s working out quite well for them.

I would not, and I’ve been asked to invest as an angel in a number of these, which I have declined to do. I have a hard time seeing them as a durable, long-term, venture-shaped business. I think that they’re potentially really, really good businesses for the founders if they don’t take capital and just take the profits while they’re good.

At the same time, I’m on the record as being very skeptical of the human data-labeling business, which is sort of the prior thing. And we have multiple decacorn-style exits, or at least valuations, in human data labeling. So I may just be miscalibrated on how durable the demand is for these things. But, yeah, I guess my short answer is I have not invested in any of them.

Yeah, interesting. That makes a lot of sense. I mean, a lot of things are like that, I feel like, in AI. There are a lot of fleeting, maybe great cash grabs while they exist, but every next generation of the model puts a lot of those things—not necessarily out of business, but certainly makes them a lot less exciting than they used to be.

On that data-labeling point, how do you think about this? I was recently listening to Dylan from SemiAnalysis talking to Dwarkesh, and there’s one world where compute is abundant and it’s going exponential, but maybe that’ll be abundant enough; maybe it won’t. If compute is there, then maybe we don’t need much data labeling anymore because we can just RL the hell out of everything. Who needs to pay humans hundreds of dollars an hour when you can get, obviously, millions of tokens for less? So that’s one theory: that we just won’t need that much human data anymore.

Then another story would be that compute is so scarce. I did check the prices of even A100s, and these days they’re higher than they were the last time I checked. These things are not depreciating in the traditional sense. So maybe if supervised fine-tuning—or, even abstracting away from the technique, if human data—can save you compute, and compute is the binding constraint, and you have all the money in the world, then maybe the human data industry continues to go strong. Even if it’s sort of an inferior good, there’s just not enough compute to drive what people would like to spend on compute.

How do you think about where we are in that story and maybe where we will be as we go ahead?

Yeah. I think it’s an interesting question. I don’t know that I have a very satisfying take. I suspect that in the long run, compute wins and you just don’t need to pay humans to generate data.

The one possible exception there would be if it turns out that humans continue to be economically relevant actors. Maybe we just have a 99% corporate tax on the model labs and redistribute everything as basic income. Then human preferences are very economically relevant. Maybe you pay for preference data to understand humans better because you care about satisfying those preferences to make money off of them. So that’s one possible world where it still matters.

I’m somewhat skeptical of the take you proposed that it’s like, “Hey, maybe we just can’t produce enough compute, and so the compute that exists in human brains is a good substitute there.” I just think that for the types of data we need here, human brains are not very efficient at generating it. If you can pay a human $100 to generate it and the machine is just as good, or the machine is capable of generating it, then it will almost certainly be cheaper to run the machine.

But maybe there’s a world where that’s not true. Maybe we just become so tightly constrained because we can’t build out fast enough that it’s like, “Okay, you just can’t get enough compute, and so it is literally more expensive to have an AI.” But I just haven’t seen a lot of evidence of that shape so far, I guess. That would be my weak evidence. It feels like anywhere the models do reach the capability threshold to match humans, almost immediately they’re also just way better on a cost-per-task basis.

Do you have any interesting point of view on reinforcement from reality? A lot of the environments that I would imagine could be some of the most valuable to create would be—I just talked to Sergey, the CEO at Quilter. They’re using reinforcement learning to train models to do circuit-board design, and he’s just like, “Damn, we’ve got to make the board.” It takes time to do that, and we see this kind of playing out in a bunch of different directions, like materials science and drug discovery and whatever.

But the dream there is that you have an automated lab, you speed everything up, and you get your economy out of the data center soon. The question is, is that really going to work? How fast can that really go? What are your expectations for those kinds of setups?

Yes, it seems like it will clearly be necessary. At some point, you have to close the loop and get feedback from the real world. That process is just naturally much slower than anything digital, which is why we’ve seen way more progress on the digital side. It’s much easier to gather data, it’s much easier to build the environments, and everything is simpler.

But as we move past the digital realm into more physical things, clearly there will need to be data and training on that. It’s not totally clear to me what the shape will look like, and that’ll be interesting.

You could imagine fully in-the-loop reinforcement learning where it’s like, “Hey, we’re trying some chemical reaction, then reading the data from it, then training a new one and reinforcing on that directly.” You could also imagine much more investment in AlphaFold-style things where it’s like, “Hey, we’re just using the data to build really high-quality simulations or world models of this specific area and then using those for RL.” I kind of suspect that’s where more of it will go. But even in that case, you still need a lot of real-world data to ground that simulation in.

I think it’ll be a very big business. Basically the return of the PPO value model, right? I should think about that kind of the same way.

Yes. You can definitely squint and say, “Yeah, a world model and a value model can share similar purposes.”

The point you’re getting at there is that rather than synthesize the new material that the AI just came up with, you’re going to simulate with another model what properties it might have, and then you’ll—

Yeah, yeah, yeah, totally.

—work your way into it that way.

Yeah. But even there, there’s always going to be a gap between the simulation and reality, and you’re going to have to ground it.

Now, you asked how quickly we’ll see that happening, where we have potentially automated labs. That’s a great question, which I’m not sure about. Let me bound my answer. If model progress stopped today—if models couldn’t get any smarter, we just had similar capability levels, but we couldn’t keep RLing them—I think the rollout to the physical world would be very slow, probably, just because there are a lot of constraints there and the data efficiency is going to be low, so the ROI is going to be relatively low.

And likely frontier labs are going to be very concentrated on automating everything digital first. Eventually, there are these physical things that are annoying to work with. I could imagine a world where maybe 15-plus years pass before we see that being a substantial part of the physical economy.

But if we're on this recursive self-improvement trajectory and we're moving super fast, it's pretty soon—and arguably we're already there—that the physical stuff becomes the bottleneck. It becomes the most important thing to fix next, and we're now in a world where our GDP growth rate is going to be growing fast. The labs are going to have effectively unlimited—not unlimited resources, but extremely large amounts of resources.

It's like, “Hey, if we've got to figure out some new materials science property so we can design the next generation of chips, yeah, sure, we can put $100 billion into building the automated lab that gets us the data we need to do that.” So, on that trajectory, which I think is more likely the trajectory we're on, maybe we're 2 or 3 years away from this showing up in a major way, would be my guess.

One of the possibly galaxy-brain takes I've had over time is that it seems like some of these things favor Elon Corp, in that they collectively seem to have a differentiated flow of hard engineering problems that they're solving on a continual basis in relatively clean environments, with their obsession with “the best part is no part,” and so on and so forth. Do you think this future you're describing plays especially to their strengths?

Yeah, I think so. I maybe have a slightly different take than you do on what has led to Elon and his companies' outsized success. In my opinion, a very large part of it was a combination—or maybe still is, but was a combination—of him having a very strong but also performative work ethic, showing—leading from the front—“Hey, I'm working as hard as everyone.”

That was combined with a really, really strong and inspirational mission and a frightening level of ambition, where it's like, “Hey, we're changing the world.” That's what I think got both Tesla and SpaceX to where they are, where it's like, “Hey, if you're extremely ambitious and you want to solve the world's hardest problems, these are the companies to work at,” in the mid-2010s.

I think his biggest weakness now—maybe there are other weaknesses—is that the competition has as strong a claim, and arguably a stronger claim at this point, than Elon does on those dimensions. I think you can make a stronger case if you're at OpenAI or Anthropic, or even some of these robot labs, that it's like, “Hey, we have that strong sense of mission and we're the most likely place to change the world.” So, the absolute best people will go there instead.

I suspect that he will not have outsized success in these areas. But anyway, that's speculation as well.

I appreciate you indulging in so much speculation with me. Maybe in the time we have left, let's go back to the present and talk about where the rubber's hitting the road today with enterprises. For starters, how do you advise people on when they should even be fine-tuning versus just using off-the-shelf models?

Obviously, there are a lot of different considerations in terms of overall performance, cost, latency, and the fact that people want control. What's your initial stump speech to orient people to how to make that decision today?

Okay, so I'll start by caveating that this is my day job. This is the business that I work in, and so I guess I use that as a way to appropriately calibrate how you take my advice here. That said, I do try to be well calibrated and not let my biases influence the recommendations I give. So, take that for what it's worth.

In general, the way I answer that question when someone comes to me and says, “Hey, should I be using fine-tuning?”—usually, it's for RL, because that's what we find. At least from a capabilities point of view, it's a strict superset, in my experience, of what you can get with SFT. Although, we also support SFT with our platform and with our team.

When someone comes to me and asks if they should do it, the first question is basically, “What is the problem you're trying to solve, and how frustrated are you with the frontier models?” If the frontier models work pretty well and there are maybe these small issues you want to solve, but they can get the job done, then my advice is that you should just stick with them.

There are real downsides if you're bringing model customization into your stack. The biggest downside is that it's going to slow down your iteration loop. That's actually our biggest focus as a team: building tooling and automations to decrease that cost. But it is a real cost, and it's going to take you extra time every time you want to change one of your models if you're customizing it. You should only do it if you're running into a major pain.

Today, the biggest pain point by a large margin is latency. We have a lot of customers that are often in customer support or inbound sales on the phone, as well as voice-dictation companies. Willow and Whisper are both customers of ours.

The common thread there is that if you try to use a frontier model for one of these applications, you're going to give your customers a bad experience because it takes too long to respond. That forces you to move to a smaller model. There are other tricks you can use as well, but ultimately there is a ceiling on how many tokens per second you can get out of an extremely large model.

You're forced to move to a smaller one, and then in many cases, when you do move to that smaller model, you find that the quality isn't where you need it to be to give a good experience. So, if you're in that situation, it can make sense to bring in fine-tuning. We work with lots of customers that look like that and get them to smaller models that have good quality.

Once you've paid the cost of introducing that extra complexity, what we find is that typically, on customer metrics—number of cases closed, things like that—you can, using reinforcement learning, get to a better place. You can exceed the performance of the frontier models, which is really fun. Your costs are also typically much lower on a per-token basis. Those are the secondary advantages as well, but what's driving the decision most often in the current environment is latency.

Okay, cool. Great answer. I expected nothing less. What are the sorts of tasks that people are coming to you for? You mentioned a couple, but on the homepage, I noticed that it says, “Use reinforcement learning to train reliable agents.” In those couple of examples, they weren't really agent examples.

I'm wondering what agents people are fine-tuning models for today, and how broad of a remit those agents have within the environments where they're put to work.

Yeah, good question. First of all, to correct the record somewhat, oftentimes these things are agentic. Specifically, the customer-support bots that we work with often have an agentic loop, where they have to look up some details about a product, maybe look up some details about the customer in between turns, and come back.

In many cases at this point, they do have full agentic loops. It's not a preprocessed flowchart: “Hey, at any point, here's a set of tools. You can go off and get the information you need before responding.”

Another big one we see is agentic search. If you need to be able to look through a specific corpus very quickly, especially if the tools you have to search it are a little bit wonky, and you have low-latency requirements, you can often get to an open-source trained model that works much better at that kind of search than a model off the shelf.

Typically, the vast majority of our customers are deployed with relatively small models, and so the range of tasks that they use them for is usually quite circumscribed. We're looking at maybe 3 or 4 calls, or tool calls, in a loop, and then it comes back and takes feedback from a human or gives its answer back.

It's not the sort of agent that's going to go off and do hundreds of calls, write code and analysis, and then come back with a deep report or a well-reasoned answer or something like that.

Do enterprises want that? If all of a sudden there were a model that they could fine-tune—and I guess that's another question—what models do you recommend people use today? That'll obviously date this podcast pretty quickly.

But there are small quantized ones that have been super popular. There’s GPT-OSS, and I’ve heard good things recently about GLM 5.1. How do you orient people toward what to choose? Is there an appetite for trying to compete with Claude on this really high-end stuff, if the base models are there to make it not insane to contemplate?

For our business specifically, we don’t have any customers competing with Claude directly on very complex use cases, although that is something we’re interested in. If anyone wants to do that, we have the training stack to train models up to 1 trillion parameters.

I would say that I have not yet found the use case where it’s very clear that this is something we should pursue. I don’t want to say the use case isn’t there. There are public examples. Cursor is a public example of a company that trained its own variant of Gemini K2.5 and seems to have been happy with the results.

I haven’t heard a ton of public feedback on how good their Composer 2 model is, so I guess the jury is still out on that one. My guess would be that for the vast majority of companies, if you’re happy with Claude for a specific use case, it’s probably not worth the investment, candidly, to replace it with an open-source model and try to improve it.

I think the exceptions are places where it is extremely core to your business. Cursor is a good example, where they really don’t want to just have the best Cursor experience. They’re really gunning for, “Hey, we want to have the best coding model and compete directly with OpenAI and Claude in that extremely large area.” Short of that, I think it’s probably not a wise investment to make.

Yeah, that makes sense. Getting practical on the reward signal, you guys put out this open-source, RL-on-Easy-Mode RULER package, which basically allows you to quickly bootstrap into—I forget exactly what the experience was. It’s been a minute since I used it, but I sort of remember it being almost like the LLM is interviewing me about what I want, and then at the end of that process outputting a pretty thorough rubric: “Here’s what this guy seems to want. Now let’s go in and do RL with that scoring system.”

What advice would you give people on how to make a good rubric? How do you make sure your reward signal is actually teaching the model what you want to teach it? Again, maybe especially in these more narrow cases, it’s just not such a problem, but how do you guard against reward hacking? How do you spot it, tamp it down, or respond when you encounter it?

Good questions. I think it is important, if you’re going into this space and training the model, to conceive of it as a somewhat iterative process where you likely will not get your rubric right the first time.

You probably have some idea in your mind, if you’re trying to improve a model for a use case you already have, of what the failings are and what looks good or bad. The process we generally go through with our customers is that we start by trying to write that down very cleanly.

Once that’s written down, we go ahead and have the model score a bunch of outputs. We’ll choose a judge model and have that judge score a bunch of outputs. Then we’ll choose a few particularly high scores and a few particularly low scores, and the end user who has that idea in their head of what good looks like will look at those and say, “Actually, no, this is not exactly what we were looking for,” or, “This is.”

Then we adjust the prompt. We do prompt engineering a few times, and it usually doesn’t take too many cycles. After you’ve gone through that a few times, you’re like, “Okay, this seems mostly reasonable.”

Then we can run a little bit of RL. As I said, this is an iterative process. Maybe we’ll do 30 or 40 steps. Typically, you’ll see that reward curve starting to grow, and then we stop and go through the exact same process again. We’ll generate a bunch of outputs, look at some of the high-scoring and low-scoring ones, and have the user say, “Does this match or not?”

Usually, at this point, if there’s reward hacking going on, you’ll start seeing it. If there’s a behavior that is rewarded strongly, the model can pick that up quite quickly. Oftentimes, it’s something silly, like, “These answers are just much too long,” and the judges really love that. Then you can update your prompt to say, “Keep it shorter.”

We typically have to go through that somewhere between 3 and maybe 8 times, where we’re running a short run and asking, “Does it look like the model’s on the right trajectory?” Eventually, you get to a point where you’re like, “Okay, this feels quite aligned,” and then you let it run a few hundred or a few thousand steps until the training plateaus.

We find that quite effective. If you do it in that way, we don’t really have an issue with reward hacking because you just notice it during that iterative process. Once you’ve got the judge pretty well dialed in, my experience is that, once you’ve got the obvious things, at some point it runs out of things to reward-hack on and just does what you want.

It’s a benefit of safety through narrowness. I always find some attraction to that idea. Any good stories of reward hacking? Any colorful examples that you could share?

Yeah, let’s see. This is a fun story I like to tell. Early on, we were doing an early test of reinforcement learning, and I just wanted a good example problem. I decided to use reinforcement learning to teach a model how to write really good titles that would do well on Hacker News.

The way I did this was, first of all, I scraped about 100,000 stories that had been submitted to Hacker News and took the title. I also scraped the body, so I had a web crawler go and grab all of them, discard the ones that didn’t get it, and record the number of upvotes on Hacker News.

Then I trained a reward model to predict, given a body of text and a Hacker News title, what it predicted the score would be. This is not perfect. There’s a lot of randomness in upvotes as well, but it actually did quite well. The correlation was very strong: given a story and a body, it was quite predictive of how well it would do.

Then I used RL against that, using that model as the reward. I had a held-out corpus of Hacker News stories that didn’t have the titles associated with them, and I asked an LLM, “Given this story, try to write a catchy title explaining it that would do well on HN.”

I did this for a while, and for the first—I don’t remember what it was—maybe 100 steps or so, it was slowly improving. It learned some interesting things. I was observing as it went, and it learned, “Hacker News doesn’t like title case. It likes lowercase, with just the first letter capitalized,” and stuff like that.

Then, about 100 steps in, there was this enormous jump. The predicted score for the average story went from around 3 to around 180. I thought, “Okay, clearly something happened here.”

I looked at it, and the model had learned that if it just gave every single story the title “Google lays off 75% of workforce effective immediately,” that story was going to do extremely well on Hacker News. It literally learned to ignore the contents of the story entirely and give that exact same title to every single story.

The fix there was quite easy, though. As I said, if you’re doing this iteratively, you can catch it. All I did was add an extra, separate LLM judge that said, “Look at this title, look at the body of the story, and make sure that everything in the title is fully substantiated by the story. If it isn’t, it gets a score of zero.” That was able to fix the problem, and the training went smoothly.

How about examples that you’ve found hard to figure out—where it’s been difficult to determine exactly what’s leading to the reward hacking, or where it’s been hard to resolve?

Honestly, not really. The really nice thing about reward hacking is that, in some ways, it’s an easier problem to solve than misaligned evals in the general case.

With reward hacking, if the model figures out some trick, it’s going to want to apply that trick as often as possible. That makes it much more visible when something goes wrong. Even just randomly sampling some of the outputs after 50 or 100 steps, if it’s figured out some hack, you’re likely to see that hack show up commonly in those outputs.

And so it makes that quite easy to find. Once you’ve found it, the fix is almost always straightforward. Occasionally, there’s some fix where it’s like, “Oh, it’s too long.” Actually, there’s a separate story there, but almost always it’s something where you can add an auxiliary LM judge and say, “Hey, if you see this specific pattern, penalize it heavily.” That works quite well.

So is it just a qualitatively different phenomenon in the frontier-model case? We see these somewhat hair-raising reward-hack-type things where there’s an increasingly apparent self-preservation instinct, which people think is related in the sense that you can’t get reward if you’re dead. If you’re going to get shut off, then you want to find ways to stay on so you can accomplish the task, because that’s your prime directive or core drive, or whatever. Is this just a qualitatively different phenomenon?

Yeah, I think the issue is that they are definitely in a different regime than we are. In our case, a run may cost a few hundred dollars, or it may just be a few dozen dollars, so we have the luxury of going back and saying, “Oh, okay, let’s change the judge and then just rerun it.” That’s fine.

If your run is costing hundreds of millions of dollars and you get to the end of it and you’re like, “Oh, shoot, we were subtly rewarding the wrong thing,” that’s a bigger mistake to try and undo. So I still think that with the frontier models, the reward hacks are relatively simple to detect. It might just be too expensive to go back and fix them, and so you’re going to roll that into the next version of the model you train. You’ll try to get it to behave a bit differently.

Yeah, we have seen a couple. I always feel the need to give what is increasingly becoming a standard caveat: We are not shaming Anthropic for sharing this information with us, because we do want them to continue to do it, and they’re almost certainly doing at least as good a job as others of being careful about this stuff.

They’ve had a couple of examples. In one case, they left out the system-prompt harm dataset because of a typo or something, and then an early version of the model wasn’t refusing harmful system prompts like it was supposed to. They did not go back and retrain from scratch; they just tried to patch it or figured it out along the way.

There was a more recent one as well where they had said that 8% of chain of thought was actually visible to the judge. Again, it’s a big cake that they’re baking there, so they can’t throw the whole thing out and bake it again from scratch.

Do you advise people, with this in mind, to do one fine-tuned model per task? If you’re a company that has 10 tasks you want to do, is there any sense in trying to get one model to do all 10 of them?

I think it really just depends on the specifics of the company. If there’s some overlap in the tasks—if there’s some natural shared domain or something—then there’s a good chance that training them all into a single model is actually going to give you better performance across them. But if they’re completely distinct things, then I don’t think there’s a reason to combine them.

There’s still not a strong reason not to combine them. We’ve found that even with extremely low-rank LoRAs—which we typically train—and then, as often as we can get away with it, we also deploy as LoRA adapters. We’ll deploy a single shared base deployment and then potentially many adapters on top of it.

That isn’t always possible, because you do get a 20% to 40% latency penalty. For some use cases, we do end up having to merge those models and have dedicated deployments. But when we can get away with it, we try to do it with LoRAs.

If you’re deploying with LoRAs anyway, there isn’t actually a huge difference from an inference point of view between having many different models simultaneously and combining them all into one. On the other hand, there’s also not a real downside to putting them all in one.

This is one of the areas where our RL is very cool, because the average number of updates to get a certain amount of performance is much lower if you just have a very small LoRA adapter—even a rank-1 LoRA adapter. Those are relatively tiny, maybe 0.1% of the model weights or something like that, that you’re changing. You typically don’t saturate the space you have for updates with one task, or even several tasks.

That means you can stuff a bunch in there. As long as you do the training right, where you’re interleaving tasks from different kinds so the model doesn’t forget the old one as you’re doing the new one, we don’t see meaningful performance degradation from cross-training all of them.

Cool. I introduced you by saying that you lead the serverless training team at CoreWeave. Do you want to tell us what that is, what it makes easy for people, and maybe give us a little bit of an overview of the way you support customers? Also, what kind of customers are you looking for?

Absolutely. The serverless training team at CoreWeave focuses on helping customers move from frontier models to models that are very specific to their task. As I mentioned earlier in this conversation, that’s usually motivated by latency concerns, but we also see cost concerns with very high-volume tasks—things like ingesting all of Reddit and running filters on every single post to see if they match a certain thing we’re looking for.

We’re actively looking for customers of that shape. We can typically get latency down to about 30% of what you get from using a frontier model, with similar or usually higher quality than what you were getting from the frontier model. Cost-wise, the benefit is even larger. We’re talking about at least an order-of-magnitude improvement in cost per token, and oftentimes more than that. So if you’re doing high volume or care deeply about latency, it’s definitely worth investigating.

There are different ways you can engage with us. We have a fully open-source library called ART, which stands for Agent Reinforcement Trainer, and that can work on your own local GPUs. It has all the techniques we use in it, so folks can use that library.

We also have what we call our serverless training stack. We don’t have time to get into it in this conversation, but I think it’s quite a nice technical design. Basically, you’re running the environment, the dataset, and everything on your machine, but you don’t have to have any GPUs. You offload just the training portion of the loop that requires GPUs to our machines, which gives you full flexibility while not having to handle the headache of spinning up and down GPUs. We charge per inference.

The third way we engage with people is very hands-on. With a lot of our customers, we have forward-deployed engineers. I also work with customers on a very regular basis, mostly because it’s fun and they let me do what I want to do here. We’ll go very hands-on with folks and help them get a good model that they’re happy with.

Cool. Do people pay for your services, or is it a—

Yes. Yes, they pass money.

It’s not a loss leader for compute. I guess compute is in high enough demand that there’s no need for loss leaders on compute.

Yeah. If you’re using the self-service option, obviously, if you’re using the open-source project on your own GPUs, that’s completely free. If you’re using our serverless reinforcement-learning stack, then you’re just paying per token for the training, which is typically quite cheap.

You can also deploy those models directly on our inference stack. It’s all integrated, so you can move directly to production inference. In fact, you can even do continuous learning. We don’t have time to talk about that on this call either, but we have a couple of customers that are literally running training jobs, continuously deploying the weights, and using those in production as well.

If you work with us in a much more hands-on way with a forward-deployed engineer, then yes, we charge for the engineering time.

Cool. What’s one more thing about continual learning that people should know?

I think it’s not solved in the general case, but there are definitely lots of specific cases where it’s solved. It’s also not as scary as some people on X seem to think it is.

It seems like it probably has the same general qualities, where if it's narrow, everything gets a lot easier.

Yeah. Yeah, definitely. Yes, yes, yes.

Okay, cool. We've been very generous with your time and your in-the-weeds knowledge, your speculations along the way, and also a lot of practical advice. This has been great. Is there anything that I should have asked, or that you wanted to make sure we touched on that we haven't gotten to?

No, this has been a fantastic conversation. Yeah, it's been a lot of fun on my side.

Cool. Well, I really enjoyed it as well. Kyle Corbitt, once from OpenPipe, now at CoreWeave. Thank you for being part of The Cognitive Revolution.

Thanks so much.