RL为何胜出——Kyle Corbitt，OpenPipe（被 CoreWeave 收购）

Hey everyone, welcome to the Latent Space podcast. This is Alessio, founder of Cornell Labs, and I'm joined by Swyx, editor of Latent Space. Hello, hello, and we're so excited to have Kyle finally in the studio. Welcome.

Hey, very excited to be here.

Kyle, you're a CEO and founder?

Co-founder and CEO, yeah.

Of OpenPipe, which started 2 years ago and recently got acquired by CoreWeave. Congrats.

Thanks.

I think you might be our first started-and-exited founder that we've had on the pod. Maybe-ish. I don't know. I'm not kidding, especially on that timeline. I don't remember if we set this up before or after we announced we were getting acquired. I specifically pinged you because I think you got acquired. You've been on my list to watch. Obviously, you've spoken 3 times at AI Engineer, and you've been on my list of, “When is it a good time to have an OpenPipe or fine-tuning/RL discussion?” Then you got acquired, and I'm like, “Okay, yeah, that's a good time to talk about it.”

Also, I think it gives us a window to talk about acquisitions, consolidation, what should be an independent company, and what maybe doesn't have to be. Anyway, we'll do this chronologically so we don't get too far ahead of ourselves. You were famously the director of Startup School. Maybe for people who don't know, what is Startup School? Did that make you fall in love with the color orange?

Yes, I'm wearing an orange shirt for those who are listening. A very bright orange shirt. This is my conference shirt. I felt like it was appropriate for the pod as well.

1. Inside Startup School

I was at Y Combinator for about 4.5 years and led the Startup School team there. Startup School has changed over the years. It meant one thing before I was there, and it means another thing now. But during the time I was at YC, Startup School was basically all of the external-facing work, a lot of the content, and certainly all of the tech.

We had a MOOC, effectively, where founders could come in and learn about how to start a company and get advice from YC founders and YC partners. We had a co-founder-matching service that we built, which actually worked really well. We got a lot of people through.

Technically, it probably doesn't matter anymore, but a very large fraction of the batches that went through YC while I was there were directly attributable to people that we found and ended up recruiting to YC through their experience at Startup School. That was kind of what we were working on.

You were kind of considered the scout program for YC—the YC before the YC.

Any notable famous people met as part of your co-founder matching? I'm always very negative on those things because it's like online dating. The chances of success are super low, but when it works, it's really nice.

That's a great question. We launched that product probably 9 months before I left, so I don't know what the long-term outcomes were of that specifically.

You left YC, spent a year in the wilderness, and went through YC S23. What's that journey like?

2. OpenPipe Finds Its Wedge

I was very excited about AI things in general. I left YC, I guess, at the beginning of 2022, and I was trying out a bunch of different things. I ended up landing on what turned into OpenPipe in early 2023.

I'd been working with my co-founder, who is my brother—my little brother—which has been a fun journey on its own. We were looking at different ideas, and one thing we realized was that we had started the company immediately after the GPT-4 launch.

What we saw as the opportunity in the market at the time—which has changed since then—was that GPT-4 was insanely expensive and extremely powerful, but there was an opportunity to distill specific workflows from GPT-4 down to much smaller, much cheaper models. There was a very clear value proposition there, given how expensive GPT-4 was.

It was hard to deploy in production, but you could take those abilities and deploy them much more cheaply. That was the first thing we built: these very managed, very clean distillation flows.

What was that process like in the beginning to get people to care? I'm assuming most people were doing experimentation but didn't really have these large production workflows that they needed to distill down. Then, once we got there, the models got cheaper and faster. What were the initial 6–9 months of the company like through the evolution of the models?

3. The Fine Tuning Squeeze

It worked; it was great. It did take us a while. We formed the company around March 2023, and by the time we launched our product, it was August, I want to say. There were some different things we were trying in between.

Actually, it wasn't hard at all to find people and get them excited. There weren't very many people in production, even in late 2023, but anyone who did have production workflows found it extremely painful. They were paying hundreds of thousands of dollars a month to OpenAI, so it was very easy to convince them to try this out.

We got our first 3 customers probably within a month of launching, and we were doing significant revenue over the next 6 months. We actually got to $1 million in ARR over about an 8-month period following that launch, by the latter part of 2024.

The initial traction was super strong, with a very clear value proposition. But then, as you were alluding to, there was this slow march of frontier-model token prices dropping over and over—by 3–5× repeatedly—which ate away at our value proposition over time.

What was the process of fine-tuning the model? Even the open models weren't that great. What were the bottlenecks? Instead of having 3 customers and getting to 30 customers, did you feel like, in the beginning, it was a matter of the market growing, the open-source models not being good enough, or the fine-tuning not being simple and efficient enough?

The pain point, repeating what I said before, was that the price was too high on the closed models. But you couldn't just drop in an open model and replace them because, as you're saying, the quality was quite bad, especially as you were moving to smaller model sizes. Larger open models weren't even available at that time.

That's kind of where the value proposition was: the closed models were too expensive, at least the ones performant enough to do the job; the open ones were not good enough; and we had a very clear managed flow.

The way the flow worked was quite simple. You simply put in our SDK. It's a drop-in replacement for the OpenAI SDK. You continue to use GPT-4 in production for a period of time, and it captures the questions and responses.

Then, at some point, you say, “Hey, I want to distill this down,” and you train on that. We provided an API that was a direct drop-in replacement. You would just change the inference URL, and you were using your own model. Your app continued working.

I think the market analysis here, because I was also exploring starting a business around that at the time—and that's why I ended up not investing—was basically that you get squeezed between the GPU providers, who also want to do fine-tuning as a service because that makes people stickier, and the labs, who keep putting out distilled versions of something—whatever, mini versions of their models.

What was the analysis on the neocloud side? You kind of also want to host the inference.

Honestly, we felt very squeezed from the frontier labs that were putting out more capable models at lower cost. I did not see the competition ever really materialize from the neoclouds, from the GPU providers.

Everybody had an offering in fine-tuning. When we talked to customers, nobody used them because they were really hard to use. So I do think that's a product thing, I guess. It's not their focus.

Yeah. Who cares? Interesting. Developer experience matters. It did. I don't know, maybe it doesn't matter anymore. Now we just have coding models to do everything for us.

No, it still does. When you have Thinking Machines Lab launching an API and people getting excited about the API, you're like, “Yeah, okay, that's pure developer experience there.”

That's fair. Yeah.

What was the Mistral 7B fine-tune—kind of one of the big inflection points in the history of the company? Like, “Okay, this is a good open model in the 7B size,” or was that just—

Yeah, Mistral and Mixtral—that was a golden period for fine-tuning startups because Mistral was a credible open-source model. They were really strong models, better than Llama 2, which they were effectively replacing.

They also had a super-open license. Licensing has become maybe less of a concern over time, at the margin, because people are getting used to it, but at the time, it was a pretty big deal that they had this fully open Apache 2.0 license. Maybe they have their own IP issues with how they trained it.

I don't know. I have no inside information there. But at least the guarantee they were making to people using their models was that they were using—

Mistral washing. As long as it's a constant, sparkling region of France called Mistral, it's okay. Don't ask about what goes into it. There's plausible deniability—an arms-length connection there.

Okay, that was this Mistral period. In January 2024, you talked about LoRAs, and there was a period of time when LoRAs became more important. I feel like they then became less important, and I don't know what the rise and fall of LoRAs has been for you as a business.

LoRAs have really attractive properties relative to doing a full fine-tune, if you're predicated on doing fine-tuning at all. If you're doing a LoRA, at training time it helps some: you're using less memory to train. But where it really helps is at inference time, because if you're doing LoRAs, then when you deploy for inference, you can multiplex basically an arbitrarily large number of LoRAs on the same GPU deployment.

That lets you do things like per-token pricing as opposed to GPU-hour pricing. It gives you much more flexibility at deployment time. I'm actually still a LoRA bull, for the record. You're talking about the rise and fall, but I think LoRAs' future is still out there. I mean, they're cool again because of Thinking Machines.

Yeah, yeah. I felt very vindicated by that blog post, for the record. For listeners, Thinking Machines put out a blog post a week or 2 ago doing quite a lot of research on the trade-offs between LoRAs and full fine-tuning in various different training regimes. I think the reason LoRAs were uncool for a while was mostly just because fine-tuning was uncool.

If you're doing fine-tuning anyway, LoRAs are still, in many cases, the way you want to do it, but not that many people are doing fine-tuning. As a marketing guy, LoRAs had bad marketing. They were just like, “You can't afford full fine-tuning? Here's the Walmart store-brand fine-tuning.”

No, that's fair. There is some of that. I don't think we had a huge issue. We've had to do some user education: “Hey, just try it.”

I think for the training runs—the types of training runs that we're interested in, where it's, “Hey, I'm doing a relatively lightweight customization of an existing model for a specific task”—there's really no downside to using a LoRA, and there are a lot of upsides from an infrastructure-simplicity point of view. I agree that there's a branding issue around that. Hopefully the Thinking Machines blog post addresses that.

Rank one. I think there are different hyperparameters of LoRAs that you can use to make yourself happy. The fact that John Schulman was like, “Nope, we're actually betting the company on this, at least for now,” is a pretty big vote of confidence. I think it's surprising that no one had done the research prior to them doing this research.

Thinking Machines had people, prior to their launch, who had come from one of the big labs. And what that research told me was, “Oh, no, everyone doing post-training research inside this big lab uses LoRAs.” I mean, not for the full run, but when they're doing experiments, they'll just use LoRAs on a base model to run the experiments, and it works fine.

For listeners of the pod, that was leaked in one of the pods that we released.

Cool. And then, at the first World's Fair, you talked about how you probably don't need fine-tuning as a fine-tuning founder. Basically, I think your talks are really good, and I would recommend people watch all of them. What I pulled out was that you had a piece of advice. Your talk title was obviously somewhat intentionally clickbaity, but your actual advice on when people should fine-tune is when it's cost, quality, or consistency that you really care about.

I mostly stand by that. I don't think it's changed. The biggest one we see today—and this is true for classical SFT, and it's also true for the RL stuff we're doing today—cross my fingers, it's not always the thing, but the main one I see that really drives fine-tuning is if you have to move to a smaller model. It's typically for latency reasons, and this is usually real-time voice.

So if you're forced into a smaller model anyway, there's a very high chance that doing some tuning on that model is going to be necessary for a successful deployment. We see that a lot from customers that have those latency requirements. There are other reasons as well. Sometimes, for whatever reason, you really have to deploy on a single GPU or within your own cloud, and you basically have to use a smaller model to do that. In the case where you're forced to use a smaller model anyway, fine-tuning it is often necessary.

I would say for 90% of use cases where you aren't forced to use a smaller model, it's still not a good ROI, and you probably shouldn't invest in it today.

How do you quantify these things? Cost could always be lower, so is there a threshold of cost to ROI? It's also hard to figure out how much it's going to cost you to do the fine-tune, because you need to get the data and all of that. Do you have a mental model of that?

This is sort of a function of the total amount of overhead required. I'd say there are 2 parts on the cost side, and then there are multiple parts on the benefit side.

On the cost side, the main things you have to think about are the upfront effort required to get an actual training system set up for your task. That can be quite variable, but I would say at a minimum, you're going to have to dedicate a couple of weeks of a fairly competent engineer's time. If you have a very complex system and you're doing RL and need to set up a whole environment, it could be a lot longer. It could be a couple of months of time. That's just a fixed cost you have to pay.

There's also an ongoing carrying cost. Once you've committed to doing fine-tuning, it does make other parts of your stack less flexible and less nimble, because whenever you're updating your prompt or adding new context or whatever, now you have to spend a few hours training a model. That's going to slow down your iterations, which is a real cost. In many cases, that's the larger cost, so you only want to do that if the benefits are large enough.

The dollar cost is basically never a factor. It's just so much less than the amount you're spending on this engineer to do the work that it's not a concern. Each of these runs is between $5 and a couple hundred dollars, and you don't have to do that many of them.

Yeah, because most of the data is first-party.

Mm-hmm. Yeah, right.

Okay. When was the switch to RL? Was it when o1 came out? Were you maybe like, “Okay, it's time to move on from SFT”?

4. OpenPipe Bets On RL

That was a big moment for us. With all the leaks before that about Strawberry and all that, a lot of people were talking about how they were doing it. We realized through that, “Okay, someone's figured out how to make RL actually work with LLMs,” which was not a thing. I mean, it was something that some people had played around with before that, but it wasn't something many people were thinking about.

Our bet at that point was, “Yes, let's figure out whether this works for task-specific things.” In this space, I think it's important to tease out different parts of the market. With the release of o1—and this has been proved out many times with releases since then—I think there's now a very strong consensus that, on the frontier, general-purpose-model side, investments in RL are paying off. I don't think most people would argue with that.

Especially as you're getting into these agentic tasks and training models to do that, it seems very clear. Obviously, the big labs are paying ridiculous amounts of money for these environments and everything, but they're actually getting really good results. The models coming out—we're seeing it especially on the coding-model side, but in other contexts as well—are working way better for these especially agentic uses because of this.

I think even in late 2024, it was pretty clear that RL was going to work in that context. The question in our mind was, “Can we apply this in a different segment of the business, which is task-specific customization?” Does that work well? How much effort does that take? Is it going to end up being unnecessary because the big labs can just train on every single task, and the base models are going to be good at everything, so there's no benefit to it?

Those were the open questions in our mind, but it seemed like there was at least a good enough bet that we wanted to try it out.

Yeah, and you had this agent reinforcement training framework, and you did the email agent, which was kind of the first proof of concept. Was it obvious to do email? Was it obvious to call it that way? What was happening behind the scenes? How should we package this?

What I told our team—and this was when we decided to go all in on RL in January 2025—was that we had done some experiments before that. We had released an RL-trained model that would generate Hacker News titles from articles, which was a fun project. So we had done a little bit before that, but January was when we said, “Hey, we’re going to bet the company on RL.” Not in the literal sense—we could have done something else later—but this was the thing we were going to spend all of our time working on for at least a few months.

What I told our team at that time was, “There’s probably a 25% chance that this is the right direction,” in the sense that a year or 2 years from now, all the companies—everyone doing inference—should be doing RL and task-specific training so that their models are much better at their tasks. It was a relatively low chance, but it was one of those big-if-true things. If it turned out that doing RL on your task was something everyone should be doing, and continually teaching these agents through experience was going to be a huge benefit, then being the first people working on that would be an awesome position to be in.

That’s how we thought about it: less than a 50% chance, but a really big outcome if so. Since that time—and I’ve been very transparent about this with our team and when I’m talking to other people—I don’t think the chance that this is the right approach is 100% yet. I think we’re still in the process of figuring it out, even after going through this and doing it, but the probabilities in my mind are going in the right direction. Today, I think there’s a 55–60% chance that everyone deploying an agent at scale should be doing RL with it, either as part of pre-deployment or continuously as it’s deployed. That’s informed by our experiments working with customers.

So, going all the way back to your question: No, it was not obvious. It was an informed bet. It’s still a bet, but one I’m feeling pretty good about right now.

One thing I think is tricky about onboarding onto this space is all the math. I remember reading the DPO paper—I think it was at NeurIPS 2023—and people were very excited about it. Some of it is just being pretentious for a paper, but some of it is actually real complexity. If you don’t have a PhD or a prior ML background, how do you come to grips with it? What were the best ways to get around it for you?

I would probably push back on that a little bit. I don’t think the math is actually that complicated. When you see the PPO equation, with all the symbols, and that’s your first introduction to it, then it feels very complicated. But if you were to show that exact same equation as code—not necessarily PyTorch code, because you also have to understand that, but a naïve implementation in Python—and show someone, “Hey, this is how we’re computing the loss here,” I think a strong engineer would find it quite grokkable.

I don’t think the barrier to entry is that high. You just have to believe you can do it and spend some time staring at it. You can read the papers and look at the equations. I think this is one area where LLMs have been super helpful. If I’m reading a new paper and I look at one of those equations and don’t understand how a new term they introduced corresponds to the other terms, I can dump all the context around it into GPT-5 and say, “Can you write this out in Python for me and show me what they’re doing differently?” That’s super helpful for my background.

Yeah. The way I put it is, I wish all these papers were published with pseudocode, or straight-up Python, instead of math. You actually just need to look at the implementation.

I know Jeremy Howard has been beating that drum for years, and I mostly agree with him. There’s a little website called Papers with Code, and people just keep not following it. I remember interviewing the DPO guys when they were at NeurIPS, and they were very obsessive about proving in-principle equivalence to PPO. It was very hard to follow. I definitely agree with that.

Now, obviously, at some point GRPO kind of took over the general consensus. It was very strange, because when DeepSeek first started talking about it, it was viewed as an optimization. They generally tend to couch everything as an optimization. But I think the later insight—which I think you touched on in one of your blog posts—was that it actually makes comparisons independent rather than global. That’s what unlocks some models, like sort of self-supervised RL.

There are real pros and cons if you’re moving from PPO, or something similar, to GRPO. One pro is operational simplicity. With PPO, there’s an extra model—a value model—that you can throw away with GRPO. That makes your life easier: You don’t have to train that model, and you don’t have to configure all the hyperparameters around it.

Another benefit is the one you’re talking about, which we’ve observed. The way GRPO works is that you have a set of different trajectories, or rollouts, all in parallel with the exact same environment and the exact same conditions. Then you score each of them, and GRPO uses the differences in those scores to promote the trajectories that did better and decrease the probability of the ones that did worse.

Because it does this in a group-relative way, it lets you be a little looser with how you score the trajectories. You don’t necessarily need a globally aware scoring function; you just need a scoring function that can distinguish between the small set of things in front of you. That’s easier for a human, too. If you ask someone which of these is better, it’s easier for them to answer than if you ask, “Is this one good or bad in absolute terms?”

5. GRPO Meets The Real World

The big downside—the huge downside—of GRPO, and I think the reason it’s likely to be a dead end and we probably won’t continue using it indefinitely, is that you need these parallel rollouts in order to train with it. That makes data generation much more complicated, because you need a fully reproducible environment to run those parallel rollouts.

It turns out that setting this up is the hardest challenge today in getting RL working. You have to design a robust, reusable environment where you can run all of this training. Sometimes that’s easy to do, but for the work we do—training agents on real codebases to operate real applications—it’s really hard to sandbox those things in a way that’s totally reproducible.

Most companies—and that’s not always true—will use an environment like that when training with PPO, because it lets you do a bunch of runs and be more data-efficient. But, at least in principle, PPO gives you the option to train purely on real production traces of people interacting with your app. You don’t have to have a simulated environment at all, which makes deployment much easier.

Can you double-click on why it’s hard to do the sandboxing? In principle, you just capture all the inputs.

You don’t just need to capture all the inputs. You need a system that reacts the same way your production system does, in many different ways.

Let’s say you’re Airbnb. I’m bringing this up because it’s an example of a company where people have gone out and built sandboxes. If you’re Airbnb and you’re trying to train an agent to operate Airbnb and book on your behalf—or you’re a company like us trying to train an agent to do that—you have to build a copy of the Airbnb website that reacts to you as the user in exactly the same way the real one does, with the same failure modes.

If you don’t include the same bugs and failure modes they have, then when one of those bugs comes up in production, your agent is going to have no idea what to do with it.

It’s just going to fall over. You also need to simulate whether this is a cooperative agent that’s getting human input as well and working with the human to get something done, which in practice is the way a lot of these are deployed.

You also need to simulate the user. You can do the naive thing and say, “We’re going to have a separate LLM with a system prompt that is the user simulator,” and we do that. But the breadth of ways a user might respond is much more diverse than the actual diversity you’ll get in practice with a simulated user.

Then it’s, “Is this environment close enough to how a real user would interact that, if a user says something different, it’s going to know what to do?” In many cases, the answer is no. If you’re purely training on an LLM user simulator, it’s going to have its own idea of the correct way to answer. The breadth of ways a human might respond in a situation is wider, and your agent may not be able to deal with that.

Do you feel like it’s hard to build the simulations as a company that needs to build the product that lets everybody do it? Or do you feel like, even for the individual companies that own the codebase and are domain experts in their own product, it’s still just a very hard infrastructure problem?

I think it’s still very hard. Ideally, all companies should have this anyway, because if you’re doing end-to-end testing, theoretically, if you’re following best practices, you would have one of those set up. When we talk to enterprises, almost universally, that’s not something that really exists.

There are some startups and some companies we’ve talked to that do have it, and we can just use that, but it’s a very small number that actually have an environment like that. I think it’s hard to do, and there are lots of weird bugs that don’t show up in environments like that. Even if they do have a testing environment, they don’t have it populated with full, realistic data, which is also important so that the agent understands how to interact.

I think in practice it’s hard in both cases. Maybe it’s easier for the company, but at the same time, depending on the quality of the company’s engineers, it might not be easy for them either.

How do you classify the types of environments? You have formal environments, like a compiler, that you can put in there so you don’t need to do any work—it just works. Then you have RL environment startups that are building a bank environment or building these things that aren’t digital twins, or whatever the term is for the actual environments, but are close to them.

On top of that, you have companies helping people build an exact replica of their environment. There’s obviously value in the formally verified ones. We verified that. Do you think there’s value in these RL environment startups that are building somewhat generic but task-specific environments? If none of those work, what do we do instead of GRPO? I guess that’s the question.

I suspect there is value in that. The folks buying those environments and training on them in the big labs would have the best knowledge of how well they work. I think they probably work okay. We’ll see, maybe with the next generation of models, how well they transfer.

I would say that so far, it seems like they don’t train well enough. If you use OpenAI’s agent interface, or if you use the computer-use products that everyone is putting out, they’re okay, but not reliable enough to actually let them go do something interesting unsupervised in the world.

I think if the environments they were training in were high-fidelity enough, then they would be good enough. Coding agents can go much further because, in that case, we do have environments that are much higher fidelity. It’s a codebase, maybe running a web browser—it’s much easier to capture the full, realistic environment in that context.

For those who are interested, when you make a reference to RL environment startups selling to the big labs, they’re selling it for a lot of money—at least 7 figures, right? That’s my understanding. I’m not a buyer, so please drop data points, because people who are not in Silicon Valley don’t know this.

It’s probably the current thing in VC: RL environment startups. Anyway, there are a lot of them. There are like 20 of them, apparently.

Yeah, but it’s a small number. I know that all the labs are buying ad hoc. In a way, it’s almost like they don’t even care. It’s not a product. They’re basically paying the company to build an environment ad hoc for them.

It’s a services business at the moment.

Exactly. But if you’re spending a billion dollars on a training run, you can specialize. You can say, “We’re the ones that do e-commerce. We’re the e-commerce experts, so come to us for e-commerce. Go to the other guys for social media.”

I’m curious about your take on how you need to get the data out to make it fit in your training run. Especially when you get to the larger labs, they have very sophisticated post-training pipelines. I don’t know if there’s a way to build a company where you just send them a CSV of data. It needs to be very integrated, but I’m curious what you’ve seen working with customers, too.

For RL, the whole way this works is that it has to be getting feedback from the real environment. I don’t see a world where it’s as simple as, “Here’s a CSV-type approach.” I guess you could have coded anything as a CSV if you try hard enough.

For RL to work, you have to be looking at real runs, ideally of your actual agent in its current state within an environment that’s as real as possible. The data format is actually super simple. It’s basically just a list of chat-completion messages; it’s effectively whatever tool calls—

Yeah, exactly. It’s whatever your agent will be seeing and doing when it’s running. Getting the data is not hard. What’s hard is that, when you’re doing one of these runs and your agent makes a tool call, that tool call has to connect somehow. It has to get data back from something, and that data has to look like it will look in real usage. Setting up that whole part of the system is the challenge.

As a reference for more people, WebArena is my first instance of this kind of thing. You literally have a Docker container with a clone of Reddit, a clone of Wikipedia, a clone of GitLab, a clone of a CMS, and a clone of an e-commerce site. Since then, there’s been Mind2Web, maybe. I don’t know if there are any other large, well-known academic environments where people are using these as benchmarks, but they’re probably also pretty useful for training.

If you want to check out those things, you can definitely look there. I think the question for you is: as someone who bet on SFT, then bet on RLFT, and now sees these guys making a lot of money, why don’t you go there? It seems to me like that definitely is a services-heavy business at the moment, as it’s presently constituted.

I’m sure these companies are all developing different kinds of secret sauce for how to do this more quickly. That’s part of it. I don’t particularly enjoy services businesses.

I also kind of feel like we’ll move toward a world where either the big labs can do it themselves, or—this is one of those businesses where the only customers right now are whatever 4 big, maybe 6 big labs that are training these models on environments.

Right. What’s the TAM?

You say the same about Scale AI and all of their competitors, which are many-billion-dollar companies that have basically the exact same customer set. So, yeah, it may work out.

And let’s say you—I don’t know if you want to do a small, shameless plug for Varys.

Oh, yeah. Varas is one of our portfolio companies. They work with the people building the agents, not with the model, on their internal tool-call loop. They observe all the internal traces and build the data to then have OpenPipe do the RFT on the thing.

In the enterprise, we’ve seen a lot of that, especially for chatbots. It’s the less sexy use case, but they work with a lot of financial services companies where their customers go there and say, “What’s my balance? When did I do this transaction?” Those are all tool calls, and they need a way to test and improve that behavior.

The models haven’t gotten that much better because these tools are badly documented and badly named. I think that’s the problem with a lot of the agent builders that aren’t AI-native companies: they just put these very generic tools in the thing, and then they expect it to work like magic.

These simulations also help them with the usual compliance requirements. Before shipping this, we tested that it doesn't give financial advice. We tested all these different things.

I'm curious to see how much the companies generalize. I think Varas has a lot of success in highly regulated environments because of different requirements. But I'm curious if you have a different way to segment the market. When you think about RL, there are environments that are low-stakes, environments that are high-stakes, and environments that have implicit rules made by the SEC or other government agencies. How do you think about it?

I don't know that segmentation is necessarily the most relevant. I'd have to think more about that segmentation and whether there's a strong difference in how useful RL is across those sectors.

Where I see the segmentation is something basically capabilities-based. If I'm trying to do something much more advanced and maybe long-horizon, then RL can probably give me much better behavior. I might almost think that, in those more compliance-oriented environments, you probably don't want your agent doing very much, because then you can't make any guarantees about what it might do. You're probably not doing these long-horizon things, and maybe RL isn't going to get you what you want. But I don't know; I haven't thought about it too much.

Yeah. I think a lot of the customers don't necessarily end up doing RL anyway. It's almost like the simulation and the environment are a way for them to understand the paths that the agent can take, and less about when you need to use that data for fine-tuning. But I think it's going to be a spectrum. What replaces GRPO?

Yeah, it's a good question. We need the alpha. I don't know is the short answer. I do think this is a fairly high-salience question in the research community. A lot of folks are trying to figure that out. Every paper has a variant.

The big question is: are we doing normalization based on grouping, or in some other way? I would claim we're just going to keep calling it GRPO as long as the normalization is done within a group, even though a lot of things probably should get their own names. A lot of things have tried to get their own names and have failed on the marketing side.

Something that doesn't require group-level normalization, which a lot of older things didn't, probably works, but I think the older things are also really finicky. There may be other kinds of simplification, and I don't know exactly what those will be.

6. Prompt Optimization Meets RL

Where do you put the prompt-optimization thing? We did a DevDay episode and mentioned GEPA, and then everybody came out of the woodwork on Twitter.

The cyber bros.

Yeah, exactly. Okay, tell me: have you or people you've talked to tried GEPA? I want to know, because I read the paper and I'm just like, “Look, the prompt-level updates are not the same as weight updates. They're comparing apples and oranges.”

I talked with a few people I respect on the RL side, and they validated that the way these grad students market their papers is their thing beats the current hot thing, and the current hot thing is GRPO. But they're just not that comparable.

I disagree with that. I actually think they are comparable, in the sense that it depends on for what purpose. If I'm a company trying to get the best performance out of my agent, I don't care if you're changing my prompt or changing my weights. If you get better performance on my agent, I'm happy. On that front, I do think they're comparable.

We've evaluated GEPA as well.

So their answer was, “You are going to do both. If you really want maximum performance, you're going to do both.”

Yeah. We evaluated everything from this benchmark, and we evaluated GEPA as well. It just doesn't work. It didn't work on the problems we tried it on. It simply didn't.

Our naive prompt with our model got maybe 50% on this benchmark, and GEPA got to 56%. We do RL and get to around 96%. It wasn't even comparable. Maybe we were holding it wrong.

Okay, that's going to be the pull quote. So both sides are claiming a skill issue, right? What they would say is, “You probably used it wrong.” And with us, the RL people are saying that the GEPA guys, when they set up the GRPO benchmark, didn't make it a very fair comparison, which is exactly what my source said. It's hard to tell. Everyone is trying to get to some version of the truth.

Yeah. What I will say is that we want it—I don't know if I'd go so far as to say we want it to work—but we certainly want to know if it works. That's actually very relevant to us, especially if it's more efficient to get there. We just haven't been able to get it working.

It's actually more credible now that you're part of a larger collective, because I think GEPA maybe makes OpenPipe less relevant.

I totally disagree with that. The level we're operating at is actually—we're not RL bros trying to figure out the use case for all RL. We're working with all these enterprises; we have all these big companies we're talking to, and we're trying to figure out how to make their stuff work better.

Personally, I'm very motivated. If something like GEPA works, then, “Okay, let's build a product around that.” That's how I think about OpenPipe, at least.

No, that's a good clarification to make. Even more so, you took a sincere look at it and concluded that there was nothing to build.

Well, maybe we were holding it wrong.

We had Shen Yu on the podcast a while ago, and I think he's been a proponent of automatic prompt optimization and this idea that you can do a lot more in prompts than you can do in weights. In principle, I'm biased—inclined to believe that something like DSPy, something like GEPA, works. I'm very surprised to hear this.

Yeah, we keep trying it. We tried the MIPROv2 stuff that was hyped before that.

Also, I should not bury the lead on the best argument for this: GEPA basically models how the big labs do their system prompts. It's genetic evolution. They incrementally evolve based on the overall evaluations that they have. It's slow because it's done by humans, but GEPA theoretically automates this.

Okay, hold on. Is the claim that the big labs have something? This is news to me.

No, no. This is philosophically the same. I'm saying that the model in the world is humanity, or smart humans, and now we're doing GEPA using dumb LLMs.

Oh, sure, but you're injecting a whole lot of human intuition and potentially out-of-band information.

The model in the world is humanity, or smart humans, and now we're doing GEPA using dumb LLMs.

Right, but humans can bring in out-of-band information that may not be captured in the actual evaluation. They can be like, “Yes, technically this did well on the eval, but it's not really what we want.” I would suspect that a lot of that ends up getting injected through the human being in the loop.

Yeah. I've always been very surprised at how these guys work on their system prompts, which are tens of thousands of words long, and there are no ablations. They just pick what seems to work and then chuck it in there. Those are the Claude system prompts.

Can't argue with success. Is GPT-5 the first model that had a prompt optimizer from one of the large labs?

I believe so, but I don't remember.

Claude Workbench had this about a year and a half ago, if you see it that way. It just wasn't fully automated, but it was extremely good for its time. I kept telling people about it, and nobody believed me. Do we know if they used it internally?

Claude Workbench, yeah.

I don't know. My experience, knowing a lot of people at these labs, is that they launch a lot of products because some team is super excited about the product. I wouldn't put that much weight on it just because they launched it.

For some measure of internal use, I'm sure. The people I talk to are biased, so I don't know if you fully explored that.

Yeah, no, I think it's interesting that it's now been acknowledged that an LLM can improve your prompt. GEPA is also a way of saying, “Okay, maybe we can do this programmatically.”

But I also think there's a long tail of people who just prompt really badly. I think there's some value there, versus once you go into RL, you already have a more sophisticated audience. Who gets to do GRPO? People who are really smart.

Who gets to do prompt optimization? Everybody's trying to do it. Yeah, that's fair.

Maybe even our baseline was pretty good.

Your naive prompt is probably in the top 10% of prompts that people put into these LLMs. I'll take it. Yeah. And then the other thing that comes to mind, as you were talking about injecting things out of band and all that, is a broader trend that I'm tracking for work for ’26, which is the move to online evals.

The way that we do evals today is probably too locked down. You're kind of fighting the war that you already know should be fought, and you're not fighting the wars that you don't know about because you didn't get it right, whatever. How can we move more online evals into our GEPA process? Maybe that's what it is. That part I'm much more bullish on.

And we can make the analogy. We can pull in some RL intuition here, which is, if you're doing GEPA on a sort of static data set of, “Oh, this is the input. This is what makes a good or bad output,” then as you're updating your prompt, the information—the data you're training on—becomes less useful, right?

Because it's generated by—because it's based on the problems you're running into before. And that's the same problem you have with RL, where you have this concept of being off-policy. As you're doing training, you really want to be training on rollouts that came from the latest version of your model.

If you train on something that came from further back, then it's sort of stale data, and it's no longer representing the current issues with your model. So if you try to correct for the issues that existed back then, it may not actually be helping you that much. And I think, for either RL or prompt optimization, that's definitely true.

I think that one way to apply that in practice is exactly what you're saying: you're using the actual data from your real evals. You have some way of saying, “Hey, either people are flagging these or an LLM is flagging these,” or some way of saying this was a good or bad output. I totally agree with you that if you're bringing that into your process, I'm much more optimistic that you're going to get good results.

Yeah. And the pipelines are not set up. This is analytics and UX people trying to be drawn into the ML process, which has never been done before. If I had to make a bet as a big theme for next year, this is going to be it.

No, I agree. I mean, I think that all of the observability platforms see that and are trying to figure out what the right shape is. I haven't seen the right shape yet, but yes, it seems like a theme for next year.

Statsig, maybe. I haven't used them, but OpenAI seems to like them. Yeah, I do think buying an experimentation platform makes sense. I think I've said before on the podcast that I'm very bullish on model routing as a feature, but less bullish on model-routing companies, because of exactly stuff like this, where it's just going to get absorbed into the model.

It's a very big part of building the process. You probably don't want to, and it's not that hard. It's not rocket science; you're just connecting pipes and making sure things are set up so that it's easy to use that data. I have a question for you, a general question: What fraction of tokens generated by, say, the end of 2026 do you think are going to come from open-source models versus proprietary models?

Oh, that's a fun question. We have an answer from Ankur at Andreessen Horowitz, where he was like, “It's 5% and going down.” I think it's going to go up because of the amount of enterprise adoption of open models that I'm seeing.

There's demand. Enterprises would much rather be on open models if they could get the performance they're looking for—for cost, for privacy, all that stuff.

And I think, basically, we may have hit, quote-unquote, AGI, in the sense that the average LLM is capable of the work of the average human. Not the best human, but the average human, sure. It's actually pretty decent at customer service, and it's actually pretty decent at, I don't know, transcribing things from PDFs, whatever.

So, yeah, I mean, totally, I think that should rise, but people who believe that it should rise to 50% are out of their minds.

I think it's a trick question. We should take coding out. Once you take coding out, I think, yeah, it can be 15% or 20%. But with coding, it's still going to be very low, because these Max plans are so subsidized and so many tokens are being generated. Anthropic gets 50% of its revenue from this.

Will coding mostly be closed models because the tokens are subsidized, or because the models are just so much better?

I mean, I'm paying $200 a month, and I'm spending thousands of dollars by accident. By accident, I paid with my credit card and spent $100 in an hour.

By the way, this is the thing that nobody wants to talk about about Anthropic. Anthropic went from $1 billion in revenue to $5 billion, and it was like, “Woohoo, yay!” And then, what are the margins? You have this goose meme going, “What's the margin?” They say it's 6%. There you are, part of the 6% that's abusing everything for everyone else.

It's not like I'm rotating accounts. I'm just using—

Yeah, you're using the product for it. Through you, people hear about Claude Code, pay their $200 a month, and then don't use it. They pay for your inputs.

Yeah. Thank you, everyone. Right, I don't want it to go away. But I don't really see—it's hard to see a world in which Qwen Coder or whatever model replaces that.

Between quality and cost, to generate this amount of tokens for $200 a month, I don't know how anybody can offer it. Fireworks can't really offer it at that price, and the quality isn't as good.

But the reason they can't offer it at that price is because of the subsidies, right? Which is not sustainable in the long term.

I mean, it's interesting because both Anthropic and OpenAI are building their own infrastructure, right? They're going to get to a place where they're going to have idle GPUs that they own. So they will also be incentivized to have 100% utilization, and they will subsidize some of it.

It's the same way that if you go on SM Compute, SF Compute, now you pay $1.40 for an H100 instead of the $2.20 listed price on AWS. So I think it will continue, but again, it depends on whether or not they actually have the $500 billion they were saying, which I think they do.

Just so we're clear, I think Stargate will go online. But once it goes online, then it's like, “Well—”

Figure out how to pay for $500 billion worth of compute, then they probably can subsidize for a while.

I think they have the $500 billion. They're going bigger. Isn't it obvious? What do we mean by “have”? At the start of this year, when they announced Stargate, people were like, “Oh, you don't even have $10 billion.” Elon was like, “You don't even have $10 billion.” Whatever.

And then Satya said, “I'm good for my $80 billion.” But now we're seeing all the money start coming in, and probably it's in the order of $200 billion to $300 billion that you could probably get raised and committed. And they're going to get the rest. It's fine. I think the plan is actually a lot bigger.

Can I just say I love this industry? It's like, “Yeah, they've got $200 billion or $300 billion. What's another couple hundred billion?” There's no other industry in the history of the world where you can say something like—

Stupid, but do you doubt it? I don't.

Yeah, that's fair. No, I literally, after last week—I think maybe 2 weeks ago—with the whole Oracle and NVIDIA, and then even the AMD deal, I'm like, “Oh, these guys have not only locked down Stargate 1, they're working on Stargate 2,” whatever that is.

Their sheer ambition is freaking crazy. There is still 1 more shoe to drop, which is the non-sovereign-wealth funding that OpenAI needs to get, which they've promised to drop by the end of this year. And my money is on them having to do a coin. I'm not a crypto guy at all, but—

You think it's going to be an OpenAI coin?

This is the one AI founder that has his own coin already. And he needs more money, and he said that they will come up with new innovative financing methods. What else is there?

They're already in the token-selling business.

But you've got—

That's a great line. Buy an OpenAI token that translates to a GPT-5 token? Sure. Plus a stablecoin.

Hmm. You'd have to get a lot of political buy-in, I think, to take that level of risk.

What about the White House—the most crypto-friendly since the dawn of time?

Well, I guess Elon's out of there now. So maybe they can make friends, yeah. I think it's doable. We'll see. Who knows.

For what it's worth, nobody's said this; this is just my theory.

I don't have any inside information. Should we go back to RULER?

Yeah, sorry. Right. Anyway, we were saying—

I think this story takes us to July 25, when you released RULER, which you call “Easy Mode for RL Rewards.” Shortly after that, you got acquired in September. Maybe you could talk through the summer: what was the vision then, and how did the acquisition come together?

7. RULER Makes Rewards Relative

Absolutely. I mentioned that my initial opinion of how likely this direction was to work was maybe 25%. We're up to 55% or so, and RULER was actually a big update on that—it got me from 25% to 50%.

For context, there are several problems you have to solve if you want to use RL successfully. Some are really basic: you have to get the infrastructure, and the libraries have all really sucked and been built by PhD students who don't know how to build reliable software. There are all these practical issues that we're working through. That's one thing, and that's what we're trying to solve with RL.

But even after you've got that solved, you've got major issues. You have to know whether your agent—or whatever system you're using RL on—is doing a good job. That's fundamental. You have to have a reward; you have to know whether it's doing well or poorly.

Sometimes that's easy to do if you're solving a math problem or something. You can come up with a dataset of math problems and the known solutions, then check whether the answer is the same. On the coding side, there's been a lot of innovative work around this. There are a lot of open data and existing test cases that you can break, then run the test case and see whether your code fixes it or not.

In a lot of other domains, it's much murkier. What is a good job versus a bad job? How do I know if I did a good job? You really need that information.

We've tried a bunch of different things. RULER is a library we released, which stands for Relative Universal LLM-Elicited Rewards.

Thank you. Yes.

The way it works basically depends on the GRPO insight we were mentioning earlier. With GRPO, it has this nice property where you don't have to have an absolute judge of the truth; you just have to judge relatively.

Simplifying a lot, it's basically an LLM as judge on a whole group. You say, “Okay, this is the task I'm trying to achieve. Here are 4 different runs of an agent trying to achieve it. Which of these did best?” Then it's stack-ranking them.

It turns out that works phenomenally well with GRPO—way better than I expected, and way better than anyone I talked to before we actually tried this expected. In the LLM you're using as a judge, it can self-ground because it's just getting these relative ranks. It doesn't have to have an omniscient view of what good or bad looks like.

That has worked on basically everything we threw at it. We've done it with a bunch of client projects and with a bunch of our own customers. It basically just works. Honestly, I kind of feel like the reward-assignment problem is fairly solved.

Yeah, it's fantastic. Is it just any LLM as judge off the shelf? You've tried it with so many things.

One of the results we published used Qwen 2.5 14B as the model we were training, and Qwen 2.5 32B as the judge. It's fine, but it's not a frontier model. It's much worse than any frontier model.

Even with that combination, we were able to get our agent performing at state-of-the-art levels—better than any frontier model on the task we tried it on—even with an extremely weak judge model. It really doesn't depend on having a great judge model in practice. It's just not something we've had to worry about since then at all.

That got checked off. It gave me a significant increase in confidence that this is actually something people can apply. It's now packaged up. People can just use RL; we open-sourced everything. You can use it off the shelf. If you stick it in your trainer run, it will probably just work.

That leaves the environment problem, which is the one big remaining piece that we don't know how to automate or remove and that requires a lot of manual work for every single task.

For listeners, this is why I refer to it as self-supervised: it removes more and more of the human judgment. The history of machine learning, all the way from the start of ImageNet and everything, is really about taking humans increasingly out of it and scaling the data you can throw in there with no supervision.

Yeah, totally. It's really awesome.

Are you bullish on dedicated LLM judge models? Have you looked at those? Bespoke Labs—we did an episode with them—and they're really trying to cover a niche there.

We've looked into it. We've trained some ourselves, and we've also used some off the shelf. There's an evaluation benchmark that the AI2 people put together called RewardBench. RewardBench is trying to benchmark models on serving as an LLM-as-judge.

Are reward models and LLM-as-judges, in your mind, the same thing?

They have mildly different meanings. It depends on the task. LLM-as-judge is usually more product-facing, and reward modeling is much more specific within a chat task.

That used to be the old meaning of reward model. I don't know—maybe the terminology has changed. I think they're pretty equivalent.

I understand that. I can see your side. Anyway, RewardBench is kind of like that, and we've tried a bunch of those.

My meta take on this is that any task that is extremely common is going to end up as a specific part of the training data for the frontier labs. LLM-as-judge is something everybody is doing in so many different contexts that you have to assume all of the frontier labs have a bunch of LLM-as-judge-style tasks that they're training their models on.

I do believe that if something makes it into their training data in more than a minor way, they're going to do at least as good a job as a dedicated model. I don't think there's probably a lot of alpha in dedicated LLM judges, just because it's something that the frontier labs are also going to be training on.

Let me caveat that. If you've got a very specific task that's weird and has weird requirements, and you have a lot of data on what's good or bad, then training a reward model for your specific task could still work. Fine-tuning an LLM as judge on your specific task could work, too.

I'm pretty bearish on a model that's trained as an LLM judge but is a generic LLM judge that can be used to judge anything. I just don't think you're going to beat the frontier labs on that.

Yeah. One other version of this that isn't quite an LLM, but that some people are thinking about—and that we're working on for a future episode—is world models. Very sexy.

They're first applied in video, as far as I can tell, with Genie 123, and now with code, and potentially with virtual cells for AI bio. Any exploration there that's interesting to you?

8. World Models Simulate Environments

We've been playing around with it a little bit. It's one of the directions that I'm fairly optimistic about for solving the environment problem specifically. If you think about it, a world model is a simulated environment. That's what its whole purpose is, right?

But in an LLM-like thing, not like a Docker?

Yes. It's like hallucinating, generating, or imagining the responses you'll get from the world. You can imagine that if you had a really great world model that you were training on, your agent would go out and make some tool call, and then this world model would generate what the tool call probably returns.

If you have a smart enough, strong enough one, it could keep its own effective internal state of the changes you've made so far and how that affects the world. We've played around with it some.

If we can get it to work really well, that could be a solution for the environment problem. You could take a bunch of production traces and use those to condition your world model, so it understands your specific system and what its failure modes are. Then you train against that world model, and the resultant agent would be able to perform in your real environment.

I do think it's a really interesting area of research.

Did you see Meta's Code World Model work?

I don't think I saw that one.

Okay, yeah, it was like 2 weeks ago. We've just confirmed the person from the AI code team for November, and it's really interesting.

Oh, sorry. You're talking about the Meta one? Okay, I missed that. Yes, I did—I saw that one. I said a lot of syllables, so it may not have parsed, but yeah, it's literally like having a debugger as the environment, as the world model, and opening up the execution trace to the model to see what's going on, see the state, and track the state as the code executes. It seems to be smart and exploits the unique situation of code environments, where we can actually do these things.

Mhm. Yeah, I think the way they envision that model being used is a little different. I think they're trying—actually, I'm curious; I'll have to see the talk. But my understanding from that paper is that the goal they're imagining is almost sort of like a pretraining step, and then, now that this model understands code really well, we can use it as basically a code-generation or coding agent of some kind.

Okay, yeah, which I think makes sense. That's almost more like a different kind of pretraining, I would say. The way I'm interested in applying world models is not as a means to something else; it's basically as its own end, right? The goal is to come out of this with something that simulates the world, which is not something you really need in code at all, because it's so easy to run code. You don't need to model what will happen if you execute this code, typically, because you can just execute the code and see what happens.

But it closely models how we think about code when we code: we kind of mentally execute the model as we type, and we go, “Is that what we really want?”

Yeah, I don't know. Anyway, it's the first model that Weights & Biases released since the MSL reorganization. We know, just based on our context, that they're very, very, very interested in code models as a path to AGI, which I'm also, of course, very interested in.

I know we kept you in here for a while. Let's wrap up on the acquisition. A lot of people say companies are not sold; they're bought. What was that process like for you? Did it just happen? What was going on behind the scenes?

9. CoreWeave Takes OpenPipe

That was driven mostly by the Weights & Biases founding team.

Lucas?

Yeah. Lucas and Shawn, particularly. They had recently been acquired by CoreWeave, and CoreWeave was looking to continue growing up the stack. They approached me and said, “Hey, no pressure, but this is an area that we think is really promising, and would you like to work here?” That's how the conversation started.

It was long and pretty painful. There were points as late as the week before we actually signed when it was unclear whether it was actually going to happen, so that part was super painful. However, we've been there a month now. We shipped a product yesterday, which I'm super excited about, and it's been fantastic working there so far.

I was very concerned. I thought, “Okay, yes, this is great. We make a lot of money by selling our company, but is the work environment going to really suck?” I thought, “Well, I guess that's just a risk I'll have to take.” It's been fantastic. It's honestly been way better than I could have imagined.

Do you go down to the office? The one down here?

I was there today. I'm based in Seattle, and they have a small office up there that we work out of. The Weights & Biases office in San Francisco is fantastic. If you have the chance, go visit. They do hackathons and co-working things.

Yeah, there's a hackathon going on in a month or so. I'm sure you can sign up for it. But do you consider yourself working for Weights & Biases or CoreWeave? Or both? And OpenPipe, too?

No. I report to the Weights & Biases founders, so we're within that organization. In the org chart, we're there. Branding-wise, they're trying to say that everything that's not being sold to big labs is kind of Weights & Biases. So the stuff we're launching is Weights & Biases-branded; it's not CoreWeave-branded as much. I don't know. They're still figuring it out.

And what's the product you launched?

We launched serverless reinforcement learning. Basically, it lets you offload all the GPU management. You don't have to worry about crashes, out-of-memory errors, or scaling up and down—we handle all that for you. You define your environment and your reward function, and then every time you run a step, you ship it back to our backend: “Hey, these are the trajectories, these are the rewards; now update my model.” We make it work for you. It makes it way easier.

Okay, yeah. Very Thinking Machines-like.

It is very Thinking Machines-like. I love the Thinking Machines launch. I think they have a really good idea. It's also very validating for what we're doing.

Why did it take so long for this to appear?

I don't know. But that's how I felt about everything. There are so many things that should clearly exist. I just think there still aren't enough smart people working in this space. Honestly, it feels like there's a lot of low-hanging fruit that nobody's picking.

One thing I saw from your post was that your North Star as the RL team at CoreWeave is to build an open world where every agent learns continually from its real-world experience. You're touching on the hot topic of the moment: continual learning. What else do we need to get there?

I super believe that, and that's basically the vision. I keep talking about these percentages—25, 50—and if we get to a world where we build that, then I think the advantages are huge and clear. Everyone should just deploy their agents that way. We want to be the team that builds the software that makes that easy to do.

I talk to a lot of engineers at our customers, and they're trying to deploy agents. It's so easy to get the initial prototype and something that kind of works well. It is so hard to get from that to something that you're confident is reliable enough to actually deploy in production. When you look at what those failure modes actually look like, it's, “Oh yeah, we know if it gets in this situation, or if it gets these kinds of inputs, it behaves funnily.”

Then it's like, “Yeah, you can update your prompt to address that, but that's not scalable, because at a certain point it's going to start breaking other things. You don't know what it's breaking.” You really want some way to just say, “Okay, look, what you did there was the wrong thing. Adjust this behavior when you get in this situation, and otherwise carry on,” right? And that's what we can do with RL. That's what we can do with continual learning. We don't have to have this concept of, “Up front, I'm trying to make the perfect model that solves everything.”

It's like, “I'm trying to make a model that's good enough that I can deploy it in production, and then when these errors come in, I'm going to say, ‘Oh, you know, exactly that.’” It's very analogous to how you train a human employee: “Oh, no, actually that's not what you should do in that situation. All right, fix that and carry on.” That's just going to make this whole process so much easier.

I think that today there is 10 times as much AI inference that could exist as currently exists, purely from projects that are sitting in the proof-of-concept stage and haven't been deployed, because there's a huge bucket of those. It's all about this reliability issue: it works in controlled circumstances, but there are areas where it doesn't work. If we can solve this problem, 90% of the addressable inference market today is just going to come online because we've solved that problem.

That's what we want to do. I'm super excited about it, and I think we have very concrete ideas about the specific pieces we need to make that work. We just have to execute against them.

Do you feel like online RL is more susceptible to reward hacking, especially as you're shortening this loop and don't spend as much time looking at the different checkpoints?

I'm not that worried about it. The reason is that reward hacking is quite easy to detect once it starts happening, because once the model has found some hack, it just starts doing it all the time. It's like, “Oh yes, this worked great. I'm just going to keep doing it.” You notice very quickly: “Whoa, it's doing this thing.”

Assuming you're using, at least in part, an LLM judge to determine which ones are good and bad, it's so easy to throw in an extra term and say, “Hey, that weird thing that you keep doing—if you do that, that's bad. Give it a low reward.” We've done this with a bunch of customers. Reward hacking does happen, but you see it, adjust your reward prompt, and it just goes away.

What's one thing from YC that guided you through your entrepreneurship journey, and what's one thing that you disagree with YC on?

That's a good question.

One thing that I really identify with, and that I've tried to do a good job of, is—I think they say, “Hold your problem tight and your solution loosely,” right?

That's what you did.

Yeah. Spend a lot of time thinking about what problem people are trying to solve, and then don't be too bought into the way you're solving it today. I think that's super important. It's very easy to get that balance wrong if you're not thinking about it very consciously.

Something I disagree with—that's a good question. I think there are lots of things I disagree with, but I don't have it cached in that direction in my brain. I definitely have disagreed with lots of specific pieces of advice, but I don't have a great answer right now.

I'll bridge it for you in case something comes up. Sam Altman said, “Everything I said as president of YC was wrong for OpenAI.” Right? Like, do B2B, and they ended up doing B2C. You should ship products often, and they ended up being stealth for 3 years.

Yeah. Actually, I think that second one does resonate with me a lot. We have tried to ship really quickly and just follow the gradient of the market. I think if I do another startup—and I don't know, maybe this is just me being beat up by the market too much—if I do another startup, I think at least at some points I probably would have done better to be heads-down and execute my vision for longer, and go for the more ambitious thing.

But that would take longer to prove value, which is definitely not the YC way. I think if you have a good vision and good taste, then that can work quite well.

Yeah, we'll see what that is whenever that comes out. But thanks for your time. This is a great overview of everything.

Thank you, guys. This has been a super fun conversation. Thanks to both of you.

Awesome.