RLVR 革命——与 Nathan Lambert（AI2、Interconnects.ai）对谈

Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO at Decibel, and I'm joined by Swyx, founder of Small AI. Hello. Hello. And we're excited to welcome back Nathan Lambert from AI2. Welcome. Thanks. Fun to be here.

I feel like I also have to say Interconnects, the Lex Fridman podcast, you, and the AI Engineer World's Fair—you've just done a lot in the last year and a half.

Not that many. I still say no to plenty of things.

Yeah.

Yeah. Your first episode with us was in January 2024, when you had just joined AI2. Then you released everything—you joined us again at NeurIPS, where you did the open models talk. Well, Luca did it, and you supported it. More recently, you were here in San Francisco for the AI Engineer World's Fair.

First of all, I wanted to congratulate you on winning the best speaker award.

Oh, yeah.

For the reasoning track. Here you go. I'm limited by emoji.

Oh, there's a nice AI-generated image. I look too zen.

I look so zen in this AI-generated image.

We had our track host take photos of you while you were speaking, and we turned them into Ghibli photos. In this one, your eyes were closed. It's funny.

We were trying to have Mochi, the reasoning Pomsky, join us, but I think she's getting very anxious and restless.

A little too crazy. Mochi—

Very restless. Okay, sure. You've been doing really good work, and honestly, I think one of the things that we wanted to establish was Tulu and RLVR. I guess, is that a good place to start?

Sure. It starts us in the recent journey. I think we can recap the story of what Tulu 3 was aiming to be and then how it got folded into the new narrative.

The goal is to do the work to compress complicated industry post-training recipes into something somewhat tractable that you can modify on your own and use for post-training at an actual state-of-the-art level. I think what we do relative to frontier labs is that we probably have a smaller number of tasks. Our post-training suite for Tulu is probably 10 to 15 tasks, but I would guess post-training at OpenAI has maybe hundreds of evals. Adding more evals is more data work, more mixing work, and making sure you have all these things.

On core evals, our suite of models—8B, 70B, and 405B, based on Llama at the time—matches or beats Meta on these core evals. I think Meta has different priorities, and Llama 3.1 was a great set of models at the time. It's just: How do we distill what is a very complicated post-training process?

You see explanations and diagrams from the likes of the Llama 3.1 report, where they have these complex feedback diagrams with many iterations, and earlier signs of that from Anthropic papers that have these multiple model variants and early Constitutional AI work over multiple years. What does that look like when you're doing large-scale instruction tuning, preference tuning, and whatever else you might add?

A lot of the core contributions of that, before we talk about this reinforcement-learning thing, are that we showed through Tulu how to scale up preference data. The academic community had been using this 1 dataset since all the way back in the Hugging Face models of Zephyr-beta, when the UltraFeedback dataset got popular, and still a year later it's the state-of-the-art dataset for open preference tuning. It's just one of those obvious things that doesn't need to be the case. It's about trying to make more mature recipes available to people.

I mentioned this either on—I think on ChinaTalk with Jordan—the origin of the RLVR idea. Realistically, when you work in the open, a lot of it is trying to match what industry has done, and we're on a different path because our infrastructure is different. Some things that OpenAI does now that work really well for long context won't work that well for OLMo because we might not have enough FLOPs in our base model. We might not have certain datasets for legal reasons, but directionally, a lot of it is just trying to reproduce things.

I've long tried to get John Schulman on the podcast. He was at OpenAI and Anthropic, and now he's at Thinking Machines. At the time, he had gotten approval to chat with me, and what he said was very confirming of a lot of the things that I had said about instruction tuning, multitask training, and preference tuning. He was like, “Oh, yeah, everyone just does RL on the outputs,” and that's how we got the RLVR idea and scaled it into something that is a general method.

There were a lot of reasonably similar works at the time, like VinePPO and Quiet-STaR, on doing these math and coding domains to get verifiable rewards. I think the RLVR thing was about doing it in general recipes.

And the naming was something that stuck.

Originally, we had—I think especially Kosta Harlan, who was a lead RL engineer at AI2 and is doing some stealth startup now. You can hear more from him on that soon. I think he's the founding engineer of something. Hamish Ivison, who's still a student at UW, and Kosta were leading most of the technical work on this.

The naming was going to be RL from ground truths, but verifiable rewards is actually a more general notion because only math questions have a ground truth. Code is verifiable, and precise instruction-following is verifiable.

I think it's a nice evolution of the name, which makes sense as you look at more domains. That's why it catches on with people. Once Jensen started using it, I was like, “Okay, that's really our goal.” That's where it took off because it was after DeepSeek, but when people like that have the acronym on their slides.

It's also very clear that RLHF is 4 letters. We want to evolve that and have a similar 4-letter acronym. It's not that much magic to it, but there's definitely intention on these—

On these little things.

RLGT may not have worked as well. I don't know why, but yeah.

Yeah. People definitely called that out, and they made that name change, which works and is fun. You did mention it, so we'll show the RLVR chart. You mostly quoted from the Tulu paper there, and you mentioned that you wanted to change it now. We'll preview a little bit of the agents discussion.

When you're introduced to RLVR, there's really just a function that checks whether the output is correct. You have a string outputted from the language model and a relatively simple function that's like, “Is this answer from the language model correct?” There's no real environment because you're just looking at the generation.

Now I need to figure out the right way to communicate what multi-hop tool use looks like for this, which is something people are definitely doing. What is the right diagram to encapsulate how o3 is trained, where they take multiple actions and the next action depends on feedback from the environment, which is some sort of information store?

When it's searching for a niche piece of information, you can't know what the next actions are without feedback from the search. Bing is what they say they use. That is a step that is very much happening.

As people try to transition to more end-to-end RL, there's a really strong notion of an environment. You're looking for a sparse signal from these multiple generations, and that's what people want to do. I think it's debatable whether people are actually doing it now.

I think the Deep Research blog post kind of hints that they do a bunch of small-scale RL and then, poof, the system works. I think what's happening is much more that people train on a bunch of small things, do some prompting, and see that when you put these pieces together—or use a couple of different fine-tunes of a model—it works.

It seems like Deep Research has some fine-tune of o3 in it, so you do that with some different domains of RL and it works, rather than Deep Research being trained on the outcome. I think that makes a lot of sense as a reason for it not working in Deep Research, because doing outcome-based RL for Deep Research would be RLHF again. You have to have 2 humans asking, “Which generated report is better?”

You can definitely do that. The whole o3 thing OpenAI showed is that they have many different reward models and reward signals in their post-training, but that's just one of them. I think a lot of the progress in making it exist is doing RL on a bunch of information-retrieval, editing, and search tasks.

We talked with Noam Brown about Deep Research and the verifiable rewards. He mentioned that this is an example of a non-verifiable thing having RL work on it. In one of your recent posts, you also talked about how the big labs have all this data that they can find long-tail things to apply RL to, and then when you put them all together, that fixes it.

Do you feel like what we're able to verify is a big bottleneck? The verifications are only done on these smaller, atomic things, so we can't really scale that.

I think my comment was mostly on the question of what agent progress will look like relative to modeling progress. We've had almost 3 years of modeling progress, and we're pretty used to the messaging around that.

And it wasn't just about being able to RL on small things, but doing any post-training to fix a weird behavior. RL is a very data-efficient way to train if you can get the right signal. But you could also just say, “It does this weird, non-verifiable thing. Let's create 100 or 1,000 instructions to include in post-training so that the model does these types of information extraction correctly, or soft extraction.” It's a space that I want to flesh out with more examples of tasks.

If you watch Claude Code running, it's like, what is it doing in the background? It's a lot of reading files, and even just compressing context—that's not, I don't think, really a verifiable thing. But that being messed up is a super crucial skill for long-context actions and longer tasks: just compressing well. That's going to take some training novelty around how you can effectively modify your training data. Instead of having all the multi-turn context, you just insert the summary, and you want to make the performance stay as good, because it's also a cost saving to have shorter context. There are just a lot of new domains like that.

But do you feel like you can figure out what these things are before you release, or do you think the labs have a big advantage because they have so much user data that they can inspect this at inference?

I think it's mostly looking at real-world data at this point. To the extent that there are clear benchmarks, you can use them in the open, but we see the industry consolidated around data in different forms. I think that's a really important touchpoint for people.

I'm curious who's still collecting reliable sources of open data that everyone uses.

There's a lot of action in the space, but it's hard to get traction.

Yeah.

Yeah. So I think for a long time, preference data has been something where people understand that it'd be very good to have large repositories of it. If you want that, you can annoy me to try to release all of it. For Tülu, we have a final dataset, but we have completions and ratings from more models. I'm talking to the student to figure out how to mark this down, because we just have so many completions and LLM-as-a-judge AI feedback data that we don't know how to clean. That's one thing.

That is something that has made it hard to grab off the shelf. For example, this UltraFeedback that I mentioned just has a lot of models in it. So most of the models that people are fine-tuning have some signal to improve on, and I don't know how long that lasts. We still don't have the answer to how important human feedback is versus AI feedback.

Every time I check in with people at frontier labs, they're like, “Yeah, we still use human preference data.” I'm like, “Okay, I don't have access to that, and I don't know how to measure how much it gives you, really.” It might be that most of the benefit is on—what's the right adjective to describe Chatbot Arena? People are down on Chatbot Arena, but it might be that the human data helps boost retention time and general preference a lot, whereas most academics were doing MT-Bench and AlpacaEval-type things. It's just not as crucial to everybody fighting in the attention economy.

Since we're there, you mentioned LMSYS and LMArena. That was one of your posts on Interconnects that I really enjoyed. Are they cooked? Is there a future for arenas? How does this play out? They got $100 million now. What are they going to do?

I don't know what the money does for them, but I think the eval is still valuable, especially at the frontier. People are very cynical, but in the compression race—what is the cheapest model you can have that does pretty well at this?—it's still so useful to a lot of people.

Chat is king.

Yeah, everyone chats with these things. That's why I use GPT-4.5. It isn't as good on Chatbot Arena. I think it's—

It's higher on Yupp, which is a new competitor to this. They have a vibe category—

Sorry, Yupp?

Yeah, Yupp. You can look it up. It's another startup and a competitor. All these companies have categories, and one of their categories is vibes. GPT-4.5 is at the top, and I'm like, “Okay, there's something this tracks.”

It's a frontier model.

Yeah. That stuff is intangibly very nice. The leaderboard is established. People still should use it. It's a focusing function for the community across different batches, from industry to academia.

Yeah.

I'm not going to try to solve their monetization problems for them, but having clear norms and things that could be hill-climbed forever is very good. Having this idea of an Elo linking models—

That you cannot saturate.

Yeah, it's a great problem. What is—

But you can game it. I think that's the issue.

Yeah, but everyone evaluates on multiple things.

Sarah Hooker came out. I've never seen her so public about any of her gripes. She has gripes, but she doesn't really go public like that.

Yeah. Artificial Analysis also has one, which I think is kind of cool. The other thing I think is relevant to this discussion is that a lot of the data is actually a single test—a single round. It's not multi-turn, and I wonder how to create proper multi-turn arenas, because you have to switch the models. That's the whole premise of LMArena.

It depends on how valuable the user data is. If the user data keeps being equally or more valuable than the inference, there's going to be a platform to keep pushing this into more and more expensive things. They're going to set up a deep research arena, because that's the data that, if I were OpenAI working on deep research, I would want. LMArena is the entity that has the marketplace to set it up right.

It's almost like how I see Scale. Scale kept climbing the edge of what AI data processing is, and because they're the name brand, they keep climbing the incremental evaluation game. A lot of them have longevity.

Yeah, that's a network effect in some ways. You mentioned Scale, which is another hot topic, but we'll put all the hot takes at the end. I do want to focus on the technical things up front. You're still writing the RLHF book. Is it an RL/RLVR book now?

I can give my spiel on it. Ultimately, RLVR is not mature enough, nor is it as interesting for a book. Those are the 2 fronts of why I don't want to rebrand. There's also some personal career strategy, but that should be independent of what is objectively a good book.

RLVR is going to be changing so much in the next 18 months. We've already seen it. There are all these new algorithms, but I think there's a lot more under the hood in how you do the right pre-training for it, what the data is, and how tool use emerges. All of this stuff is core to what RLVR will be seen as.

I'm watching to see if o3 is a niche model or becomes the path that everybody needs to follow. It's a different style of tool use that you see, particularly with search. We don't know how OpenAI did this, and these are the things that I think are core to an RLVR book that we don't have.

RLHF is more interdisciplinary, in the same way that Chatbot Arena can never be saturated. RLHF can never be solved. We know these problems of alignment and overoptimization, and we know the pipelines for getting the data that people are using. Yes, I can add more RL algorithms to the book, which is nice for me to study, but that's not really changing what reward modeling is or the different ways that people implement these today, whether it's a value function or a reward model.

I think the breadth of RLHF is nice. I would tell a lot of academics that I think RLHF problems are going to be foundational, with a much steadier study rate. We're on this massive spike of RLVR, but it might just be solved and then go back to zero academically. It's not an embellishment, but there could just be a best practice for getting 100% accuracy on any problem that you want, and then it's solved. The debate over what a preference is will go on forever.

Yeah. Because it's verifiable, there is a right answer.

Yeah.

Sorry, what do you mean about there being a lot of changes over the next 18 months? What do you foresee? Actually, let's just catch up on what's already happened in the recent history.

Yeah. So there are 2 categories of information that we have: what the models are doing and what the researchers are doing.

I think the models provide a lot of inspiration in terms of what the actual frontier is—

And that's things like o3, Gemini 2.5, and Claude 3.7 Sonnet. These are a mix of approaches. I think o3 is the most scaling-RL approach, and Claude and Gemini 2.5 are very similar hybrid reasoning models that you can turn on and off.

They rolled it out in different ways. Gemini didn't have hybrid reasoning at launch, but they've brought it in, and Claude had it at launch. One of the most important questions has got to be: Is the o3 path of just a reasoning model, or are hybrid reasoning models more useful?

Do they diverge in their methods for training them? I think the NVIDIA Llama Nemotron reasoning paper is probably the most detailed paper on a hybrid reasoning model, and DeepSeek R1 is still the canonical recipe for a reasoning-only model. Those are very different approaches, and I don't know if one will win out or not. There's also a lot of work on the data side and on RL methods. I think there's a whole list of GRPO complaints out there where the math doesn't make sense for certain things.

To me, every paper I see come out always has some fix to GRPO. It's kind of cool that people are taking variations on it, but I also don't know if DeepSeek is going to come out with R2 and just blow everyone away with whatever is next.

Yeah, I definitely don't think the algorithm tends to be the most important thing. I think I had this in my AI Engineer World's Fair talk, which was kind of a snarky take on how you train a reasoning model: You get a starting data set, incrementally improve the data set, and do that until you're running out of time or your performance starts going up. Then you try all these switches from all the papers, or you do a whole bunch of binary tests of all these various algorithmic changes, and do a grid search to see what works.

Candidly, that's why I dismissed GRPO when it first came out: it was sold as an efficiency thing.

Yeah. And I was like, “Okay, fine.” But I've been trained not to care about efficiency because it's just a matter of resources.

Yeah. The GRPO advantage estimate is very well-suited to verifiable rewards.

Right?

But the other thing is kind of an intangible: it works better on the infrastructure-type argument. When it came out for DeepSeekMath, which was well before the RLVR phase, it was really marketed as that.

When you talk about hybrid models, how do you reconcile that with OpenAI saying they want to move away from the model selector to just having a unified interface? Do you feel like they feel pressure to say, “Hey, look, when I have all these different classes, we want to route them to the right thing,” or do you think there's something else?

I would think that OpenAI wants to have a model that knows how hard the problem is. I think that has to be the north star for most people working on reasoning: The model will just spend the right amount of tokens on it. And if you look at a compute-level discussion, it seems like that's what inference-time scaling means.

I think in plenty of ways, hybrid reasoners might just be aged out except for niche applications, because quality is so much more important than having 100× fewer inference tokens. You just pay for the compute, and that'll get better. I think that's really what Jensen Huang said in his most recent interview. I think Stratechery highlighted it or had the interview with him, and it was like, “Yeah, everything's going to be a reasoning model because it's going to get so cheap and they're better.” I was like, that's why the hybrid reasoning thing is a little bit weird.

I always just turn reasoning on unless it's a really silly query, like, “What is this thing?” So in 2 years, that kind of tracks. I think o3 is also just burning money and searching 80 websites for me when I ask what paper something is. That's a lot of tokens, but directionally, if that's the thing that works, that'll be the default.

At least in all of these high-end applications, most of the people that we talk to, whether it's coding or the very high-end information economy, think the value is there.

I wanted to double-click on something that you seem to be coming back to a lot. You seem to assert that o3 does something very different by using search a lot more than basically everyone else.

Yeah.

Do all models come with a search engine now? Is that a must-have?

It depends on your use case. If you're doing general information retrieval or understanding, yeah. There are old papers that we could try to find links to—I don't know if Sam was talking about it—but there's this RETRO paper from DeepMind and other architectures that people have been pulling into the discussion again. You have a very small model with a very big context length and a very big retrieval store.

I'm not one to bet against the transformer architecture and just figuring out long context and things like this, but those are ideas that people are bringing back: Search is better. You look at all the evals from reasoning models, and one of the trends is that simple QA numbers all drop. DeepSeek R1 to the new R1 goes down. Qwen 2.5 to Qwen 3 goes down, at least when you're evaluating these without tools.

SimpleQA is considered to be a very nice, fairly numerically robust long-tail knowledge evaluation. The raw models are all going down, but it just makes more sense for long-tail information to have this search behavior.

Okay, the counterargument for this—and I've been through this journey too—is, why don't you make a model that doesn't know anything but searches? You can search up anything that you want to learn just in time. But the problem is, you need to know what the search terms are. You need some baseline intelligence to make all of this work.

Yeah, that makes sense. That's a good way to put it. I think it's important because there's this thesis of LMs becoming just online LMs permanently, and it hasn't been super-pursued. Perplexity was one of the first to put it on my radar: They were like, “We'll attach the search engine to the LM, and that's what you get now.” More and more people are starting to offer it as part of their default services. Gemini has a search-grounding feature as well.

I mean, it's what people say is a big limitation of Anthropic, because it uses Brave Search, which returns a bunch more SEO slop.

Is that proven? Because I don't know. I thought they had their own index.

Okay. I haven't done a detailed look, so I'm dealing with rumors. But I think they'll all end up doing their own index, and they should. It's one of these things where Google should have an advantage again, but who knows if they do.

I also hinted at this in my post, but Hamish had tried to set this up—the same student from RLVR playing with search and an RL model. It's very easy to get the model to use tools if you prompt it to, but it's very hard to get the RL model to learn that the tool is useful. That's why it goes through these things where it makes 80 failed tool uses and still gets it, or it stops and gets it on the 81st.

It's just an RL behavior that feels emergent from having a very nice way of getting the model to learn to use the tool. You can't SFT this model to do this. It really feels like they set up the environment, and it plugs into this Deep Research kind of line of work that they did. They broke down the problem into these sub-RL tasks, and then it kind of lets it do this thing.

Interesting.

I don't want to be an OpenAI shill all the time, but I just tell people to play with o3 all the time because it's weird.

It's excellent. I would say the amount of work you're imputing on the Deep Research team, when as far as I know it was 3 people who did it, is a lot. It was Isa and the 2 other collaborators that she had. I don't know if they did that much on top of o3. Every indication I've had from OpenAI is that Deep Research is more or less a thin wrapper over just o3.

Yeah, it's probably 1 or 2 small things. They're like, “Oh, we can make Deep Research work by adding this small amount of data to the training, and then it just works.”

That is how I would describe it.

I mean, what is it, Gwern, the anonymous person? He replied to my Q* post on Twitter the other day, and he was like, “Why was this all wrong?”

It's obviously that simple things don't scale. There's a lot of complexity because there were a lot of other exciting things in the AI field at the time, and OpenAI sends out a lot of things that confuse people. But this would fit into that: Deep Research is a minor change from an existing RL trajectory of what was probably o3. They had already figured out that search was going to be better, and then were like, “Okay, we can repackage this.” It's a simple thing that makes a big difference.

Most things are like that once you have traction. I think getting the initial takeoff on the sigmoid is the hard Q* thing, but then once it's like this, a lot of things in the middle feel obvious. That's why I describe one of the things that we work on for OLMo. A lot of it is just having the motivation to do things that feel somewhat obvious, but they're still hard.

It's hard to get different recipes, or it's hard to get a full reasoning recipe off the ground. It's just a huge change because you have all this inertia on this eval suite, and then you have to figure out whether you branch your recipe or start from scratch. Do we just take Open-Reasoner-Zero and start from scratch? It's a whole other headache of things.

It’s just hard to move these projects that are anywhere above 5 to 10 people with enough inertia to get stuff done. But then, once you’re hill climbing, things can seem really obvious.

Yeah. Okay. You covered a lot there. Before my next question, just to close the Brave thing, our friend Simon Willison wrote a post that Anthropic added Brave Search as one of the subprocessors in their product. Yes, so that’s where the thing came from. Now, to what extent it gets used, we don’t know.

We don’t know. I would just comment on a couple of things that he said, and then we’ll go on to your question. There’s a very good post on the retrospective of Q*.

There’s a very good post that you had, which I want to send people to, which is: Was o1 a scam? That does imply the question of, if o1 was a scam, what else could be scams now?

Yeah, there are definitely scams out there. I mean, the whole inference-time scaling plot is such a scam. You put these two things next to each other with an x-axis, and it just looks like it’s easy to control. Whenever you see an x-axis, you think it’s easy to control it.

Whereas, for training, the left one was training. Training makes a lot of sense. So, especially if you go to really old RL papers, RL learning curves are usually on a non-log x-axis, and they look like this. They look like a logarithmic or exponential rise, and if you take one of these and make it a log x-axis, it’s a straight line. So that side is like, “Oh, okay, we’ve seen this before with RL.” But with inference-time scaling, it being an x-axis is why people are like, “Oh, there’s a knob I can turn—search up a lot.” That’s what breeds all these weird ideas.

I think the most interesting—well, you have a lot of interesting things in your blogs—but one that stood out was about RL and tool use. You said that it’s easy in an RL experiment to tell the model to try searching, but then, if it doesn’t get results with the tool, it’s going to stop using the tool very rapidly. Can we unpack that?

Can there be a good tool that the model doesn’t know how to use, and then it kind of fails and stops using it? Can there be a bad tool that should be improved before giving up on it? How should people think about designing the tool, improving the model, and where to intervene?

This is definitely on the newer side of the things that I want to work on or have worked on. I think, particularly in 2026, especially on the open side, all the infrastructure and models will have caught up a lot. I want to go deeper on this in terms of Deep Research-style things, which are very inference-heavy and require multiple calls.

To answer your question, there definitely can be bad tools, and there definitely can be cases where the model is just using them wrong. Something that I would want to see in a model is not necessarily creativity, but an openness: it doesn’t know exactly what it’ll get out of all of its tools, and it has this uncertainty that leads it to just try a few different things. That almost seems like classical RL behavior.

But if you think about what a language model does, they’re always very confident—not necessarily confident, but they have a path and a direction in their answer. Whereas that’s a big change in these reasoning tokens: to have the notion of backtracking and things like that, which is some sort of openness to the tools having things that are unknown in them. It seems like a really nice thing for the model to have: “Oh, what if I try this? What does it get?”

Especially on the open-model side, if this is going to work—where people want to use open models with tools—it’s going to be because people have private data stores and stuff. So, if you were to train an open model that is going to be a good reasoner like o3, but on private records of some sort that will never get sent to the cloud, it needs to be thinking, “I can try some things with this to get a sense for it,” before saying that it has to give up.

If you look at tool use, much of it right now seems more similar to code execution, where it’s just a part of a sequential path that you need to get to. It’s like, “I have a plan, and if it fails at a certain step, I might have a backup,” but it’s not this iterative process of, “I need to fiddle with the environment in order to come up with my plan.”

It’s something that people are probably going to have to train into these models. You might just tell it, “You don’t know what is in this, but your answer might be in it,” which is a very odd prompt, but maybe it’ll help.

Yeah. When we had Eric Schluntz from Anthropic, who worked on the Claude agent before Claude Code, he mentioned they spent basically the majority of the time on the tool design to give to the model, and then you just learn how to do it.

I don’t know how much you’ve worked on this actual stuff, but are you putting the tools in one by one in the RL process? Do you think that helps, or do you usually give all the tools and let the model explore?

I don’t really know. We haven’t gotten this to work. I would say it would probably depend on the model and your starting point. If your starting point is already good at tools, it can probably generalize more.

But if you’re doing this weird base-model RL and you have to have this curriculum long enough, you’re going to need a curriculum of things getting harder, and that’s pretty obvious. So, in that case, it might be that tools get added when things become too hard for it to solve certain questions, which sounds very intuitive but is also just really hard to manage in practice, because what is your automated signal on your training run that it’s time to do that?

That’s why video games are so good: they’re designed to unlock things as you progress. But I think with things like search, it’s like, if you’re given access to a small data store or you’re given access to all knowledge on the internet—

Good feedback for the ARC-AGI people for the V3 benchmark would be to have things where the language model needs to learn to use new actuators in the world after a certain threshold.

That would be ARC-AGI 4 then.

Yeah, I don’t know. They’re cranking them out.

They’re cranking them out. They’re actually doing a launch party, I think, in a couple of weeks. So, it’s fun to play ARC-AGI. I don’t know if you’ve tried it.

Oh, I haven’t.

It’s pretty fun. These are IQ tests. I used to be like, “Oh, they weren’t that relevant,” but now that we have a gradient where LLMs are actually significantly climbing them, it’s much more interesting to compare your own intelligence.

I’m with Noam on no harnesses.

No harnesses. Yeah.

Yeah. I mean, harnesses are cool, but they’re a handicap that’s changing the learning dynamic substantially. So, they’re good demos, but I feel like the core thrust has to be no harnesses.

I mean, is it wrong to say that these are just inductive biases, right? They’re not in the model. Sure, but anything where you’re just looking at the results contaminates this; it’s a different task. I think—or, I mean, I’ve talked with Greg about this at ARC-AGI, where I told him, “Do harness and no harness. You just have both different categories.”

Just like, you’re trying to be transparent and build targets for frontier labs, just do both. I don’t think it dilutes that much. The no-harness one is obviously going to be harder, and then you just get more bang for your buck on your benchmark.

Mhm. Yeah. It’s the same data set.

Staying on the topic of tools while we’re at it, you had a really good summary of recent work in multi-tool RL, which had Loop, ReTool, ToRL, and all these other things. I think this is just an area that’s super rich for research right now. I just wanted to give you the space to highlight what your favorites are. What do you think that people should explore?

I could share what my moderate-ambition, what-would-be-fun research project ideas are. You want to create some sort of competitive dynamic or an eval, and it has to be so much narrower than what industry is doing.

I told you this at lunch: deep research, but only arXiv papers. You don’t have to do a full index; you have a limited domain. You have to figure out how to measure it or something. I think it’s good for academics to work on academic tools because they have very high domain expertise. They already know what’s going on, and you just have to figure out how to make that either very useful to users, if it’s going to be good enough at that, or something you could climb on.

I don’t know if this is brainstorming on the fly, but take related works out of papers, just look at the text, track all the links, and make an eval that involves filling in hundreds of related works with arXiv links. That’s a fun deep-research-style idea. See if you could do it with open models on a data store with tools.

AI2 has gone through a lot of discussions about this. If you’re trying to have an impact in AI right now as an academic, you have to level up out of papers to artifacts: models, data sets, eval data sets, and evals are easier for people to have an impact on.

Then the next thing is, what do people actually use in AI2, especially in this Semantic Scholar team that’s now working on information agents of different types? That’s another thing that I’m distancing myself from, so I don’t have all the names, but it’s: can we make open models do that science thing better?

It's like, can you make something that people actually care about? That's a whole level of impact that's much higher if you have actual users. It's hard for academics and small institutions to do that.

But if you're working on agents, dogfooding is viable. It's like, can we make ourselves a good Slack summary bot that we like or something, and just make these agents really tractable? I mean, that's one direction. The other direction is just hill-climbing on Humanity's Last Exam with tools.

I just think it's kind of unlikely that we're going to win as an academic on a state-of-the-art number, because they're going to start spending millions of tokens per query, and it's a lot of compute burn. Beating that on the FLOP equivalents is going to be so hard. Unstructured thoughts are something that I'm mostly like, “Okay, I'll get to this.” I have more things to figure out on the modeling and what I call the skills level, which is just: how do you do reasoning to induce inference-time scaling and get high eval numbers? And once you know you can do that—

—you can take your knowledge with you to do it in more specific domains. There's skill, and there's skill acquisition. I think the ARC-AGI definition of AGI quoted it. What is it? It's like efficient skill acquisition—efficiency, because I described it as 3 words.

Right?

Yeah.

Your emphasis on skills in the recent talks you've given—do you want to reiterate that thesis for people to pick up on?

Yeah. So, I've been thinking about this mostly because I'm trying to get ahead of what OpenAI, et cetera, are doing. Probably now, if it's not already in their models, and with all the agents, it seems that planning is a very critical task. So, how do you come up with a taxonomy for the different types of things you need to train into reasoning models, for when they'll be a bottleneck? And so I came up with 4.

The foundational one was skills, which is what I would say we have already done with o1 and R1: you do a lot of RL, you show that inference-time scaling works, and you get really high benchmark numbers. The next 3 are kind of what comes next, and most of them are around planning.

What I had as the other things on my list were abstraction and strategy, which is trying not to use “planning” because planning is a word that people already use a lot. Strategy would be the direction the model should go in and, technically, what the steps of its plan are. Abstraction is how it breaks things down into things it can actually solve.

And then the 4th and last thing is calibration, which is just not wasting compute and knowing when to give up and ask the user things, because overthinking is obviously a problem. It's easy to keep getting your eval scores to go higher by using more inference-time scaling, but eventually that's not what people want in their models. They want a smarter training regime where the model is actually getting proportionately better for its training.

There are a lot of papers on overthinking and stuff like this, which I think OpenAI wants because they have to foot the GPU bill. If o3 just infinite-loops itself for a bunch of people, that's not good.

Does it actually?

I don't know, but it might. These reasoning methods definitely can make the models kind of unstable and yap. It's also the GPT-5 idea, which is: how do you get a model that just routes the question to the right—maybe not necessarily a router—but just knows if it needs to do a plan or if it can just answer?

If you look at DeepSeek R1 and you ask it a hard math question, it's not like, “Here's my plan of attack.” It just starts. Having a model that knows when to be like, “Okay, here's my plan of attack. I might need to make myself a memory store. I might need to take a Claude Code approach for this query. I'm going to build a memory store and spin up some parallel searchers and then come back.” Conceivably, this is all something you can train into a model, because the searches or the parallel models could be tools.

In that case, the simple way to describe it is that we have something like thinking tokens and then answer tokens, and the model should be able to optionally have plan tokens before thinking or before using tools. It's like, okay, here are the table stakes: I need to do these things, and these sorts of tasks will be harder versus easier. It seems more tractable than some far-out ideas for AI.

It's like a language model can write a good plan, and it just needs to be asked to do so. I would bet that Claude Code and Deep Research are doing this: you get a user prompt, and first the model is like, “Yeah, there's a plan tool in Claude Code,” and then they break it down. That is something they've trained into the models. I don't think DeepSeek has it built in, but it probably could do it. I'm just thinking about that interface: if the model needs to be able to do the task end to end on its own, can it do that sort of thing?

I think my challenge with reconciling this approach with the no-harnesses thing is that a lot of the way that people, especially engineers, want to model it is that the plans and the memories are tools. There are no special plan tokens and no special memory tokens; it's just context, or it's just whatever, specifically for planning, because then you can fan out to other agents for tool calls and stuff. So it doesn't have to be sequential, but I'm wondering: is this a fork in the road? Do we have to make a real choice here as to whether we outsource things to tools or keep them native within the model's tokens?

I don't think it's a subjective difference. I think mostly the planning idea is to make the point that people don't get things for free, and the planning improvements might be kind of mundane.

It's just like we were prompting Claude and its plans were bad in this way. Let's give it some data where its plans are more detailed or break things down into more steps so that it's easier for it to do.

Yeah.

Because it's in a black box, effectively. So, if it hasn't been targeted, it's unclear what the performance will be.

Or, on the open-model side, it might just be the idea of having different models for different parts of it. Then you're really training a model to just be good at planning, and that's data that you need to come up with. You only use that model for that one part of it. Does it feel like plans are much more reusable and should maybe not be generated every time? I feel like, especially in coding for certain sets of tasks, you want to have similar types of plans.

So maybe it's not the right way to ask the model to regenerate a plan every time. There should almost be plan blueprints as tools, and then the model fills them in. Where do you think the balance should be?

I think they're reasonable. A plan is obviously an intermediate goal. I just think it's likely that there are failures on this kind of planning level. The same thing goes for these rubrics that are popular, where a lot of the technique for so-called rubric-based things is that you have a prompt and you have a language model generate a rubric for that prompt, which is a few specific things that it needs to get right.

That's conceptually very similar to making a plan for every task. I think whether or not it's grading, you're going to have a different type of abstraction than executing. But what people are seeing is that it's cheaper relative to the effectiveness to just generate it.

So, plans are not super long, and they probably don't have that many tokens. It's probably just kind of like, “Okay, we do this.” Putting it in my taxonomy might be overselling it: it just needs to be a prompt, and you need to make sure that your model's not too weird at that prompting stage.

I think your taxonomy is super useful, by the way. Skills, calibration, strategy, abstraction. I feel like maybe abstraction might be the most underrated one or hardest to solve. The way that you introduced it was different from how you wrote it in your blog post. You said it was basically not to overthink.

That's calibration. Abstraction is about breaking things down.

Yeah, I think both strategy and abstraction make the most sense on the hardest tasks that we don't know if the model can do them.

Right?

So, if you're assigning a task to a model that you don't know if it can implement, the strategy is very important because it needs to be very specific and narrow. If it's doing mundane code or Deep Research, the plan is actually not that interesting of a thing.

But when you're at the frontier of it—I don't know, some GPU-implementing thing—you could buy into the OpenAI narrative: “Help me implement this research idea in our complex distributed GPU thing.” My God, it's like this is a task that's hard for a human, and for an AI to come up with the right plan to debug and do this is a very narrow path.

Therefore, the strategy is pretty important: does it start with certain tests, and how does it actually build this out to complexity? It's obvious that I need to come up with better examples for this, but I think as you push it, it's more natural to see that there are only a few plans that actually get it done. Abstraction is just important as your task becomes so big.

It's like a prompt-engineering thing almost.

Yeah. And you only have 100K tokens you can generate. You need to make sure the model breaks it down so it's not just spawning a ton of infinite processes under itself, which I do agree is an interesting part of abstraction, especially when you start to think about these models that could call in other models to do subtasks for it, or parts that can be parallelized with multiple searches or just more compute.

I think that kind of folds into abstraction, which is just: how do you approach a certain nugget of the problem? I definitely don't have experience building this. It just feels like, if you're going to visualize AI doing the hardest software or other tasks, it's something that humans are very good at.

So it's like, how do you come up with a research plan in 10 weeks? How do you prioritize which experiments to do?

There are a lot of inductive biases that go into that, and I don't think a language model would do well at that right now. Probably memory would be helpful there. The way we do this in real life is we accumulate experience.

One thing I did want to dive in on was parallelism in general. There's one case where, with o1 and sort of the Q* ideas, it was overhyped in some sense, but now it's coming back with o1 Pro and Deep Think. The theory is, at least—you correct me if I'm wrong—that they run o1 8 times, then have a reward model rate it and give you the best of the 8. Yeah.

Something like that.

Something like that. Deep Think is also the same. We don't know any details beyond that. I think there's a lot of people exploring that, at least on the inference-provider side—how do we parallelize search and planning and all that? I'm worried about getting too hyped about it.

I think it makes a lot of logical sense, and this is one of those things where MCTS also made a lot of logical sense and we were fooled. I don't think we're using parallel compute in a way to search over low-probability tokens. We're using it to get robustness. o1 Pro was so nice because it just had a very predictable depth to it, even on niche topics where sometimes models just fail out.

Yeah, you had some numbers that it went from 10% to 95% or something.

I don't remember the exact numbers, but that's what it feels like. It doesn't feel like you turn on o3 Pro to make it 10 times more likely to find some niche piece of information. Maybe it'll be a bit more likely, but we're not getting that type of searchy notion of getting more breadth or depth into our tree.

I think there's value to using this parallelism on either the most important tokens that we're generating, or saying, "I know this part is crucial. Let's just spend a bit more so that those tokens are better." But it's not a transformative thing. The part that's potentially interesting on the transformative side is if you can get much better verifiers.

I think verifiers are changing the slope of inference-time scaling. You spend more tokens at inference, and the better verifier you have—if you're doing parallelism—it can extract a rare occurrence. Right now, if our verifiers are only good at human preference, we don't need to crank that up very much. But if we are doing really diverse generations and our verifier is better, it'll do better.

You could look at the extreme between a reward model and an oracle. The oracle is such that the more you search, eventually it works, so the slope is good. But a reward model has a really capped signal, at least if you're doing this preference type of thing. The slope is pretty minor, and it has diminishing returns.

If you could fill that with more interesting verifiers, there's potentially more to get out of parallel compute, but I don't think it is as transformative right now. My outlook is more that parallel agents make sense as a throughput engine if our tasks are taking a long time, rather than as a peak-performance engine.

Okay, yeah.

Which kind of fits with the whole agent-versus-model thing, where agents are much more about getting it done at all—being robust and being fast—whereas a model is one generation. It's like, can you get the answer right?

Yeah. We'll spend a little bit more time on this, and I'm happy to move on. My pushback, or counter to this, is that it's a way to pull forward a hypothetical future model that you can then distill from.

Yeah.

Which is nice.

Well, I bet people will surely use these for synthetic data. The marginal gain on synthetic data is always very high. Amanda Askell will say that better prompting will effectively make it seem like you have the next-generation model, where most people don't put effort into their prompts.

Oh my god. Okay.

Or she said something along those lines in one of her Anthropic interviews, which is that if you can really figure out how to get into certain states of the model.

Yeah, yeah. Well, anyway, that's my pitch for why this is worth doing at all. I have a science-fiction story that I want to write about quantum models in a world where we could explore multiple universes cheaply, then pull forward the right one. That would work.

This sounds too science-fiction-y, but I feel like in a world where we could control quantum computing well enough to explore this and scale it up enough, it could be kind of cool. It also could be that parallel compute is grounds for interesting types of innovation. I don't know what it means to have parallel compute with diffusion language models that generate all their tokens at once. Does that meaningfully change some sort of application? I don't really know.

I think a diffusion language model would be fun if it works. You have much more control over inference-time scaling. Gemini has one, but it's hard to suss out what it changes. Once we have all these knobs, I'm hopeful that it helps build some interesting types of innovation, because the parallel stuff is new and architectures can change. We'll see.

I've been using the Codex-based thing, and I feel like most of the generations are 5% different from each other.

Because you use Ruby.

No, no, no. I had a JavaScript one. I have a JavaScript one, so I should be good at that. I don't know if it's just how the RL training works. One thing I've noticed is that these models always want to use if statements when there's a missing variable, so that it doesn't fail when it runs. To me, that's a symptom of the RL.

Yeah, the code is terrible. You should not write code like that. It shouldn't silently fail if there's a missing variable; it should just raise an error.

But I feel like RL is pushing the code in this direction. All the generations have the same pattern. I generate 4 things, and all of them use the if statement, just in different pieces.

Yeah, that was something where the labs are trading off massive gains in performance for small detriments in usability.

And it's like, do you ship that model? You just ship it and deal with it later, but I'm sure they can fix it. I'm sure that's a fixable thing.

I think the question is that you talk about gains in pieces of the thing but not in the full trajectory sometimes. Do you feel like these are examples of that? Or do you feel like, as we get better, if we did a longer trajectory where instead of just writing this piece of code, you have to think about how you're going to maintain it later and how it's going to run, that's going to fix it? It's hard for me to grasp.

Yeah, the software stuff is not easy because maintainability almost feels like a human-preference-type issue again.

Where somebody could look at it and be like, "Yeah, that's not as good."

But adding the heuristic and training seems very messy.

Yeah.

So maybe it is. I don't know. There's a lot more to dig into there. This is what Anthropic says they're doing, and what are the actual frontiers in making— they said they're working on code only, and what does that actually mean?

A bunch of it is going to be design trade-offs, like how much autonomy the model has versus these potential side effects from training longer that we don't know how to get rid of.

That definitely could be the sort of behavior that I would say is a simple thing to remove. It might just be obsessed with some code format that fails when you revisit it, or something, even if it's like everyone has seen it with just bypassing test cases. I think they'll be a bit more nuanced than that, but they could probably be super simple.

This topic has a similar semantic content, at least for me, as overoptimization, which is something that you've written about. It is overoptimization with a different reward function.

I know. Okay. Well, I made that link, and I want to verify that we are thinking on the same wavelength. I just wanted to go over specific topics that you've spent some time thinking about.

You write that there are 3 types of overoptimization. The first was RL for control. The second was RLHF, and the third is RLVR. They always happen. Obviously, RL is no stranger to reward hacking, but do you want to elaborate on how things are evolving in terms of how we're learning as an industry?

Yeah. That 3-things breakdown is for people to put the pieces together for what has happened historically. All of these overoptimizations are a result of the model optimizer being strong enough that it can manipulate the agent with respect to the environment, or manipulate the environment in a way that's useful to its target signal.

Also, for context, I think what we're doing with language models in RL in general is that if there's something that can move its reward signal up, it'll move the easiest, most direct thing to move that signal up.

So that’s part of the story I mentioned on sycophancy: this reward model for user feedback was probably so obvious that humans just like to like stuff. People press that thumbs-up button when—

Filled bullet points.

Yeah. All those things have just been really easy for the model to extract. Once they added it, the model changed a lot, the score went up a lot, and it was easy for RL to find that and control it.

In older RL, the environment is normally a fixed simulator. There’s no feedback, so the overoptimization looks like unphysical and nonsensical behaviors. There’s the motorboat example going in circles. There’s an example from a project I was a middle author on that was effectively overoptimizing HalfCheetah, which is this MuJoCo thing. Instead of running, it took cartwheels off into the sunset and got infinite reward. It’s obviously not the intended purpose; it looks like a glitch.

It’s just manipulating the agent interface with the environment. RLHF is a classic case where the model will break down because the reward model is imperfect. The environment is really imperfect in the RLHF case, where—

It’s so sparse. It’s very artificial.

Yeah, it’s a very artificial environment. It makes sense that these actions, which are generated tokens, will do things like degenerate into just repeating one token over again. One of the early examples we had when playing with this at Hugging Face was that the model would just say “JavaScript.” It would be “JavaScript, JavaScript, JavaScript,” on some toy dataset, and it’s very obvious when you see it.

It’s probably harder to see when you’re at the top and making decent decisions about when to stop training if you’re doing a lot of RLHF. But that was the phase that people went through, and now we’re in the RLVR phase, where we’re giving the model a reward when it does something “right” for math.

It’s a bit harder to overoptimize, I think, unless you have tools and the model learns to search and cheat instead of learning math. I’m sure somebody could seek that out in the world. The model is like, “Oh, you’re training me on Stanford’s problem set for CS whatever, which I’ve seen a thousand times. I’ll just go get the solution manual.” I’m sure there’s an example where that has surely happened.

With code, and maybe information retrieval, it’s easier to fudge. The easiest way to get a unit test to pass is just to put a `pass` in it, and it’s not too surprising that a model can learn how to do that. For code, you need more reward design, which I think would be a nice area for substantial academic work: what is reward design in code for balancing understanding, overoptimization of test cases, avoiding failures, or something like this? It’s not necessarily going to be a controlled environment because these models are complicated, but I would guess you can reproduce that in some ways.

Just to double-click, reward design means, for example, giving partial credit for partially correct work.

Yes, or giving the model a slight penalty for doing the unit-test thing if you can detect it—for cheating.

Yeah, because it adds a lot of complexity to training these models compared to math, where if the answer is right, you can look at the GRPO math. Partial credit is weird in that because it’s kind of normalized per batch. I don’t know if I have a whole spiel already on it for that, but it also becomes very complicated if you’re mixing domains. Is partial credit in code better than partial credit in math? All these things mean that reward design becomes very complicated, and that’s what you’re incentivizing the models to do differently.

Is there any literature or hypothesis about mixing these things? Let’s say you have one verifier for code, one for math, and whatever other verifiers you can come up with. Individually, they work. Do they conflict?

I think part of the intuition of RLVR is that the model is good at knowing which prompt area it’s in, which is why models don’t get worse on knowledge benchmarks if you’re training on just math or precise instruction following. The model develops an intuition for where the different prompts are in space.

The gradient updates will be different depending on your batches, which is partially why people say to use big batches: a lot of the model is activated, and you have a less noisy signal with RL. A lot of the intuition is that the model just handles that.

There are interesting questions around sequencing. Do you do large-scale math and code RL to get the sequence length, and then add in more general stuff? That’s something DeepSeek mentioned. One thing in the DeepSeek report is going from math and code to more general RL.

There’s also a question of where you add tools if you’re going to do code execution and search within this. I don’t know if that’s interwoven or if it’s a second stage. It’s surprising how much is not known, and you just need a lot of compute for ablations.

The high-inference-length generations definitely break all the infrastructure because it’s just so many tokens. There’s more opportunity for out-of-memory errors or other things to go wrong. By default, all of your training jobs need way more GPUs just for the memory of inference.

Sure.

And it’s like training, but it just makes it more of a pain. Yeah, that’s a cost thing. One of the maybe controversial takeaways from the No Priors pod, which you listened to, was that there’s also just the wall-clock time of getting feedback from the environment, whatever that is, especially if it’s a real-world thing.

Yeah, we have to find a way to accelerate things internally within the training time, faster than the passage of time in the actual universe. I’m not worried about that problem, but I agree with you in principle.

Right. I’m stretching this all too far. I get it. As we start wrapping up, what are other interesting ideas that people should pursue? In your AI Engineer talk, you said that what you’re thinking about for scaling RL includes big multidomain datasets, difficulty filtering, and long runtimes. Is there anything specific that people out there who are doing research, or who want to start a company, should explore?

Most of them, I think, are not in the reasoning space, which is what the talks have been about. I’ve been talking for a long time about character training, which I think is under-indexed, and I’ve been advising a student that’s—

Like personality training. Okay.

And how that involves different ways of changing the personality of the model, from prompting, activation, or fine-tuning through data engineering. It’s the kind of stuff that Joanne Jang does at OpenAI. How much does that matter? What are the fundamental research questions? Hopefully I can share more about what I’ve been advising a student on. I’ve been saying that for a while.

Do you like the Model Spec stuff that she’s doing?

Yeah.

Okay. That trajectory.

Yeah. I’ve been an early fan of that. That’s how she noticed me: I was the only person who covered it when they first released it. I think it was over a year ago.

I liked it.

Yeah, not many people did.

Okay. All right. All right. You were first.

I don’t know, but that’s what she said to me.

Well, we had a Model Spec talk close the whole conference, right? That was my sign of, “Pay attention to this, guys.”

But it’s real because of what it sends to developers. It has a developer benefit of showing where your model is going, and it also has a regulatory benefit. I think it’s very important to distinguish intentional behavior from a training error.

Okay.

So I think for model transparency, it’s really fantastic. I’ve said that the Model Spec is much more useful than a constitution, because a constitution is an intermediate training artifact that you give to the training algorithm in order to get the model that you want. It’s not necessarily a record of what model you got. We don’t write down our goals for the model in constitution form.

By the way, have you looked at the constitution?

Not really. They talked about it. They put in Apple’s design guidelines, but also the Universal Declaration of Human Rights.

So, at this level, I’ve seen it. I don’t know if they’ve updated it. That’s very odd. I hope they would write a Model Spec. I’m not too optimistic, but they’re the next domino to fall.

Well, my take on that, actually—I pushed for this too late, because OpenAI had already approved the talk and all that—but I was going to ask them to compare the OpenAI Model Spec to the Claude system prompt, which is their closest thing to the Model Spec.

The system prompt is incomplete because OpenAI has things in the model spec that its model doesn’t currently do, especially when they first started. When they first released it, it was like, “We want the model to be able to engage on sensitive subjects,” and maybe even NSFW was in their model spec.

Which is—

They’re just signaling what they wanted it to do. They say, “This is very hard to implement because there are all these obvious risks to doing this, but in an ideal model where we can solve every problem, this is what we do,” which I think is good for many different stakeholders.

Mostly, my thing is that there hasn’t been a good foundational research paper on that, which is a lot to do. It also runs into personalization and personality, or something similar. If open models are to win, part of it could be that everybody can have exactly the model they want.

We’re serving GPT-4.5. That’s kind of its thing. You can prompt it, but if fine-tuning is more effective than prompting, everybody can have the model they want. It’s an academic problem, or an open-ecosystem problem, where people are fighting on the turf that feels more likely to win.

Yeah.

Which is good.

Is this somewhere where, speaking as AI2 and OLMo, you want to win, or are you just advising a grad student on it?

I don’t think it’s a differentiating factor yet, but I’m very open to working on it. I think open models have a strong role-play use case, and character personalization, all that stuff, especially because people find their waifu and want to keep their waifu. That’s the derogatory term for it, but—

I would say that we’ve definitely discussed it, and I want to—

Part of it should be that it’s a base model that’s easy to take in the directions you want. We’ll have an opinion that is probably slightly conservative on personality. I’ve gone through the OpenAI Model Spec, and most of it is stuff we agree with, like being conservative on anthropomorphization.

You disagree with—

I don’t remember. I did it a couple of months ago. A lot of it is openness or transparency, which is, if we’re training an open-weight model’s personality, we’re not going to withhold anything. We also have a different hierarchy, so most of it is that type of information exchange rather than “be kind.” OpenAI’s model is pretty agreeable. If you read through it, it’s “Treat the user with respect,” and all these things.

They’re raising kids that way. Just read the spec.

Yeah, it sounds kind of stupid. But then the last thing is for people doing research: wacky model-routing things, where you figure out a bunch of different models off Hugging Face to route to. An open-model tool could use way more models more easily than any OpenAI product, because OpenAI is restricted to OpenAI’s models.

For your query.

There are people who started the other way, like Martian and Not Diamond. I don’t know who else. He would know.

There’s a bunch.

There’s a bunch. Yeah, so I don’t know if that would work. Hugging Face should work on it.

It’s a moonshot idea. You don’t know when it’ll—

Given your Hugging Face background, how does Hugging Face make money? This is a very common meme question.

I think mostly enterprise deals.

That’s what they say.

They’re doing their thing. I mean, yeah, look, they’re great. They’re big. They’re profitable. It’s just not that obvious to most people.

I like the router idea for media models. There’s a long tail of things like a background remover or a style applier, and that is actually hard to find on the tech side. I feel like you should just use the big one unless you’re under some latency or price constraint. You should just use the best model.

Even when we’re doing thumbnails, I’m like, “Okay, I’m trying to remove the background of somebody,” and I go to Replicate, and there are 55 background removers. Or I want to generate a diagram to mimic something, and it’s like, “Which model is better for diagrams?” Those are not easy to find because none of the benchmarks are—

Part of the argument is that if distillation works really well, we could just keep making the target for distillation smaller and smaller. You have models that are very narrow, right? Can you do a table reformatter from Markdown to LaTeX in a 100-million-parameter model? If you get it small enough, it’s really economically feasible because it’s effectively free, and inference is instantaneous.

My pushback is just that if you’re doing image editing, GPT-4o should do it—do all of it.

Well, yeah, but I think it does, like—

It’s just that we’re not there yet. Give it 5 years; it’ll do it right there. So why work on a router at all? You just scale up for—

I guess—yeah.

Right. Tell me where the logic is here. This is a temporary thing—

On-device.

On-device—

The local-modeling community is much smaller than people give it credit for, because most of the use for open models is still in APIs. It’s like, DeepSeek API, and if there aren’t that many models, somebody’s going to host them for cheaper than most people doing it themselves. That’s pretty realistic. But there is a small community that needs local models.

Yeah.

The best outcome is if open models can compete on not just long-tail things, but that takes the most transformation. As a side note, I resisted buying my own GPUs and building my own cluster for this reason. I was like, “The API will solve most of it. People are losing money serving me models. Why am I doing that?” Except for the fact that 4090 prices have doubled in the last year. So actually, you made money doing local models.

How does that make you money? Because your investment goes up.

Yeah, and the value of your used 4090s goes up. Interesting.

I bought a 4090. I got a 4070.

Damn it.

I was like, “What is this?” Well, then it makes me think, should I buy an RTX 5090 if it ever is widely available?

At GTC, they were doing the drops. It was crazy. You were running to the campus to buy it.

I know—

—to buy it.

Any other topics before I give a closing question? Just generally, your work on RLVR—other topics of the day—I think companies should keep considering rereleasing open models, mostly for PR and onboarding. It seems like that’s the way it’s going if OpenAI is releasing it. Are you excited about that? Do you feel like it’s a second—

The OpenAI model will be good. I expect it.

They’re pretty serious. It’ll be best in class for some size category and some subset of tasks. That’s like OpenAI: they only do things like that. You have to give them the respect.

Yeah. That’s a big win. Open wins when more people are doing it.

Yeah. Well, hopefully they’re actually open about the techniques and not just the weights.

Do we think the size of the open model tells us anything about the hardware that they’re going to build?

No.

What? No. They’re so secretive about this. That’s why they haven’t released GPT-3.5 or anything, because it’s too revealing about internal stuff or plans.

Oh, okay. No, I—you’re talking about Stargate, or what kind of hardware?

No, I think that’s a different form factor. Yeah, that’s—yeah.

I think that thing will run in the cloud. I don’t think that’ll run locally, anyway.

Well, okay, we have to talk about it. It seems like every podcast we talk about it. Apparently, the news from today, which I think you were looking at, was that it was an ear device that they got sued over or whatever.

But I think the ear form factor is pretty good. I actually did get there with Bee in terms of where this ultimately goes. You want something—you want the AI to hear what you hear, and where do you hear what you hear? In the ear. That’s pretty much it.

I don’t know if you guys have thoughts on wearables and where that goes. I try to be—I think it just knows too much. That’s really my—

But you want to give it context.

Yeah, I have false privacy hopes. I think, like a lot of people, I mean, that’s the whole thing: people don’t actually care about privacy.

It’s just really good memory.

I think the Ray-Ban Meta form factor is good. I don’t think it’s as mass market. If you get it in an AirPods-sized form factor, it’s a way bigger market for obvious reasons. But the sunglasses form factor is a thing that works, I think.

Okay.

I don’t use them for AI, but they can fit the AI to make it work.

Yeah. Empirically, yeah. It obviously works.

Yeah.

Cool. Well, the last question I was saving up was: What is Meta doing? You had a pretty interesting post back in—when was this? In April. You said, “Llama 4: Did Meta just push the panic button?” I feel like back then it didn't actually push the panic button, but now they really pushed the panic button.

That's fair. I think the panic button at the time was the whole LMSYS model not being the model that they released, along with a bunch of weirdities about the day of the week they released it. But to be a model that claims to be open and then not release the model that is your leading claim is just bad execution.

Bad execution. Yeah, which is fine.

And then the recent stuff, I think, mostly can be boiled down to: talent is cheaper than GPUs by a dramatic margin. At the end of the day, it's like, okay, if we're spending this much, they go into the room and stare in the mirror and say, “Wait, it might not actually be that ridiculous to spend this money on the top people. Might as well try it.”

They already spend it on VR.

Somebody was bound to do this eventually. It makes sense that if Apple somehow decided, “We're going to do this,” they would come in and do exactly what Meta is doing.

They need a founder-mode CEO who's like, “Screw it, we'll take the L.”

The thought that occurred to me is: Meta, instead of spending on VR, should spend on RLVR.

Everybody has a bribe number.

Right.

Just really big.

Yeah, but I think some researchers are uncomfortable with the idea that this is sort of the great-man theory of research—that you have to pay this much to get this level of talent.

The talent is definitely distributed, right?

Right. A lot of the people that they would be paying this much have the confidence to redo things, or to do some of the same things. Whether you call it “feeling the AGI” or just the drive to build things, feeling the AGI is not that different from a lot of things that have existed in Silicon Valley lore in the past.

So it's just people with the vision who are willing to execute on it and see something coming. Those people make a big difference. I think you have those people and you remove bureaucracy. Getting technically talented researchers is actually something that Meta has a lot of, or has the ability to get a lot of. So it's a lot of recycling, which is very hard on individuals and the morale of an organization, but I understand the approach.

Yeah, for sure. Cool. That's all I have. Any parting thoughts on how you're going to build the American DeepSeek? That was a nice tweet.

Yeah. Mostly, if you were asking me what my 10-year goal is, I only have a 2- to 5-year goal. As models shift more toward agents, I think scaling is slowing. There's a side of it that's a fixed cost and a fixed path to getting toward something like American DeepSeek. Mostly, I would say it doesn't have to be American if it's fully open, if you have everything and you can modify it.

There are a few things that need to happen. A lot of it is just more resources, but OLMo 2 32B is, if you squint, like the original GPT-4 level and fully open. There are a few levels that you need to go through. That's obviously a dense model; it needs to be taken to sparse, and you need to scale it. You need to have a lot more GPUs, and then you need to do large-scale reasoning.

That's the goal that I want to pursue. There's a lot of complexity in navigating how to work with AI. What does AI2 do to get there? It's very hard.

I think that—I mean, it's a nonprofit. It's hard to get the resources, and building a model is a lot of aligning a lot of different people. That's the DeepSeek story: They have great people. OpenAI has kept a lot of really good people for a long time. Anthropic has gotten a lot of good people right now. And it's a lot of incremental, hard technical problems that you need to stack up.

That's what I would like to do and make work in the next couple of years, but it's not easy to get there. So that's the pitch: AI2's best-case scenario is that AI2 is going to do other things.

You can't just run a nonprofit or a company that says, “Our goal is in 3 years to have an American DeepSeek.” No one's going to keep paying the bills on that, because you have to tell a better story.

But that's what I would like to do, and I'm sure AI2 will do many more interesting things along the way.

Like product stuff.

I don't think it's necessarily product, but what are more cutting-edge things in AI that we could make—a new architecture for certain things?

Okay.

Or what are demos of open models working better, whether you have private data or something, or just far-out ideas that could take you off the transformer trajectory? I think you still need to be doing these things to lead in AI.

Thank you for working so hard on truly open-source AI.

Yeah, it's fun. It makes it easy to align your values with what you're doing.

It'd be better for the world if more things were open, and therefore a lot of it is just willing it into existence. I think seeing what OpenAI does—or is saying they're going to do—is hopefully a win coming soon.

Yeah, DeepSeek was the most unexpected win that made some other dominoes fall, but I think that is the path forward. We'll see what it takes. Thank you so much.

Thanks for coming on.