[State of Post-Training] From GPT-4.1 to 5.1: RLVR, Agent & Token Efficiency — Josh McGrath, OpenAI

We're here with Josh from OpenAI. Welcome. How do you introduce yourself?

I work on a bunch of the thinking models at OpenAI. Recently, I've been focused on search-related work, but I'm primarily a post-training researcher at OpenAI.

You were on with us for GPT-4.1 when we were talking with Michelle, who's on maternity leave. I didn't know that. Now we're in GPT-5.1. It's been a whole generation.

Yeah, it's been wild. GPT-4.1 was a non-thinking model, and since then we sort of switched into doing—

Was that your last one?

No, we're still releasing non-thinking models. That was the one we did that was specifically for the API, so the focus has shifted a little.

How did you get into post-training?

Before OpenAI, I was doing pre-training data-curation work. Looking at the news and reading papers, I thought, “It seems like there's going to be a lot of interesting work in post-training.” I wanted to make some contributions there.

It wasn't necessarily that pre-training was dead, but it was definitely changing. Do I want to make compute-efficiency gains of 3%, or do I want to change the behavior by 40%? Honestly, it seemed more exciting to go into post-training, and many late nights later, that's definitely true. It's also a different kind of data and engineering discipline. The kind of work you need, especially in reinforcement learning and scaling it, is very strange.

Yeah, definitely. The number of moving parts in an RL run is much higher—an order of magnitude, or—

I don't know if we could say an order of magnitude, but if you think about pre-training, you're moving tokens to many machines, getting a scalar from them, and then backpropagating.

The issue with RL is that you're doing tasks, and each task could have a different grading setup. Each of those grading setups requires more infrastructure. When I'm staying up late trying to figure out what's going on with a run, there can be many more things involved than there generally are in a pre-training run.

Does it matter if you own the code for the task, or if it's an outsourced third-party person's code? My sense of it—and the external sense of it, obviously, since I don't see it up close—is that you work a lot with external partners, as well as on internal stuff. Which is better?

Honestly, I don't think I'll comment too much on how many external partners there are.

There are some, and there's some internal work.

Yeah, there are.

But what is the technical trade-off of, “I don't own this code”?

When it comes to not owning the code, when I'm babysitting a run, it doesn't really matter whether it's internal or external. The question is: Do I understand the underlying system?

You end up having to jump into a lot of code and say, “I actually don't know what this does.” I'll be watching a run and working on my pieces, while other people are working on theirs. Do I understand what their code is doing? At 12:30 in the morning, when something looks wrong and I'm looking at this code, can I get enough context quickly enough to understand it?

Throw a Codex at it.

Wrong.

I use Codex so much. It's really changed how I work. I feel like there's a degree to which I'm trapped by Codex, because if I spend 30 or 40 minutes writing something that looks like a design document, Codex can do more work than I could do in a few hours, in 15 minutes. But then what do I do during those 15 minutes afterward?

It's actually changed the flow of my day, because I now have to manage these 40-minute sessions with 15 minutes in between where I could do something, but it's not nearly as effective as this new flow of the day. I'm still getting used to that.

I think it should also be interesting for codebase understanding when you're encountering unfamiliar code.

You briefly mentioned the shopping model before we started. That's the latest, hottest thing, and we're recording this right after Black Friday and Cyber Monday. Did you find anything interesting from releasing shopping in ChatGPT right into that period?

The first thing is that I don't know why I would say in a meeting in August, “Black Friday is coming up. Maybe we could do a release by then.” In hindsight, I'm thinking, “Why would I say something like that?”

They're like, “Yes, now you own it.”

Yeah, exactly. The most interesting thing to me is the new interruptibility and the qualitative experience of using it. The same thing happens with Codex: You write a prompt, and you can press Escape and say, “I messed something up.”

We did the same thing in the shopping model. It shows you its chain of thought and the products it's looking at, and you can send it a new message saying, “I actually wanted this.”

Like, “I wanted USB-C on this,” or whatever it is.

I think that's a really new and interesting interaction paradigm that we have in a couple of different services. I'm excited to see how people use it and whether they enjoy it.

Why did it have to be its own model and not just a new tool?

Stay tuned. There's no reason we couldn't eventually do it in the same model, but if we want to try out new things, sometimes it makes sense to create a new model.

This time, it made sense to ask, “Can we do a Deep Research-style model for shopping, where it's going to look really hard across the internet for different things?” If you look at the original Deep Research model and GPT-5 Thinking on High today, I think you'll see that eventually the models all converge in their capabilities.

Would you say this is a discussion that's a little spicy, which I've kicked off in the community? Maybe 30% of the community is still using Deep Research, but a lot of people have moved over to just using GPT-5 Thinking as Deep Research.

Is that the spiritual successor, or are they direct replacements? Are there things that we lose from the original Deep Research model if we do that?

If you look at our published evaluations, they look basically on par, if not better. Personally, I use Thinking on High rather than the Deep Research model.

As we've learned over the past few months, though, people sometimes prefer the quirks of one model over another. If people like the Deep Research model, more power to them.

Was there anything special in the GPT-4o post-training? Are people really responding to personality? Is that a differentiator that people care about, and is it part of your job to care about personality?

People definitely care quite a bit about personality. Over the past few months, we've been working a lot on giving users more choice over what personality they want.

Right, through the toggles.

Yeah. Now we have those toggles.

What's your favorite toggle?

Honestly, custom instructions. I personally want my model to be a tool. I don't necessarily want warmth or anything; I just want answers, because I'm mostly using it at work.

I call this the Anton-versus-Clippy divide. Anton is the machine from HBO's Silicon Valley.

Okay.

It's a machine. It only does work. It doesn't try to be helpful or friendly. It tries to be helpful, but it doesn't try to be cheery.

So it sounds like you also come down on the side of using—

Anton. Yeah, I think a lot of developers want Anton.

Right. It just quietly does its work, and when it's done, it shuts up.

Yeah. We're doing a lot of work to provide both Antons and Clippies, and I hope people like them all.

Generally, I was thinking about what we can update people on in post-training. What do we know today, in 2025, that we didn't know in 2024?

At the time, there was still this whole PPO-versus-DPO discussion. That was a whole era.

Yeah.

Since then, we've moved on to RLVR and a lot of agent-specific RL training. Am I missing any large chunks of the post-training debates that are going on?

Yeah.

I mean, not necessarily internal debates, but my read personally, from looking at the different papers that are coming out, is that when you look at an RLVR paper or an RLHF paper, they read more like optimization papers. To me, the interesting thing that's going on is that we have this spectrum of how high-quality a signal is. At the end of the day, RLHF and RLVR are both policy-gradient methods. What's different is just the input data.

It's always interesting to me that we call RLHF nonverifiable because we've trained the model to be good at predicting human feedback. In some sense, that's verification, but obviously—

It's human preference rather than truth.

Yeah. Yeah. But if your value of truth is, “Does the user like this more?” there's something strange that I think we haven't looked at: how clean is this signal? How much do I trust it?

I totally agree that you don't necessarily trust the RLHF signal as much as, “Is this the solution to this problem?” But I think there's a whole spectrum of how high-quality the signal is and what's going to happen when you do a lot of optimization against it. That's very different from worrying about the variance of different gradients, which I think is what you end up seeing in a lot of the papers currently coming out. Rather than being very data-centric, they're pretty optimization-centric, even though I think the innovation really is where the data is coming from.

Yeah. Before I go broad, before I go deep: are there any other discussions that people are having in Europe, or sort of around this time, on post-training debates? What do you talk about with your peers at Anthropic and DeepMind?

At Anthropic and DeepMind, we're all saying, “I'm working on stuff and things.” I think it's more about talking broadly with my friends there. We're just talking about, “Man, the industry is so hard to keep up with.” We're not necessarily talking too much about methods directly, because on one level, it kind of doesn't matter.

Yeah. And I think there's something very different about academic work, where what really matters is how narrativizable it is. That's one of the reasons you see a lot of optimization papers come out: a lot of the data work has a less clear narrative around it.

I think the data and the scaling are actually more important than a specific method.

Yeah, but it doesn't necessarily have the same narrative that you get out of some of the papers that you see here. So there becomes more of a question of, given a specific vertical, how do I understand that? I wish there were actually more papers on it here, but I think it can sometimes be harder to wrap up into a clean story.

That's also something that we're having a lot of conversations about with other folks as well: what's next? Where do you go from here now that we have some kind of road map? I think what's interesting for me is that the innovations exposed by the Chinese models are maybe copies of, or discussions of, what's going on in the labs.

GRPO—you mentioned that a lot of these RL optimizations present themselves as optimizations. GRPO came out in the DeepSeek Math paper, which, when it came out, I read and thought, “Okay, this is kind of cool.” It's a little bit cheaper, but it does seem to have a broader impact on the industry as a whole than was initially appreciated. I don't feel like we've processed that enough.

Yeah, definitely. As you said, it came out in the DeepSeek Math paper, and it's an interesting optimization method. But the more interesting thing is that they have a new reward signal that we can really, really trust. When you find the answer to a math problem, it's a lot less debatable than, “Was this thing that the human preferred actually what we wanted to do?” You want to be right at math.

Yeah. Yeah.

And so I think, in some ways, that's underappreciated in what's getting published.

Yeah. Let's talk about long-horizon.

Yeah.

What do people consider very long horizon? Are we talking about 30 hours—more than a day—of autonomy? Is it just more of the same, or is there anything qualitatively different?

Okay. So first off, what I would say is that I tend to think more in terms of the actual number of tokens than time, because—

Yeah, the human in the loop can take a while.

Yeah, and it gives you a different measure to optimize against. As I was saying earlier, when I use Codex, it does something that would take me much longer. It would take me 4 hours in 10 minutes. What we can actually push on there is token efficiency.

That is a huge, huge research area.

Yeah. And so you can see that from GPT-5 to GPT-5.1, our overall eval went up some, but if you look at a 2D plot of how many tokens it takes for us to get that, it went way down.

Dude, I live by those charts. Not necessarily that one, but that shape of chart. I think that's something we think about a lot, just because it contributes so much to your experience: how long does it take to do this task?

Yeah. And I think the other thing is, as you're pushing token efficiency, it changes how many tool calls I can make and how many different things the agent can do in a reasonable number of tokens that we can actually serve.

Yeah. I think the interesting—or hard-to-understand—thing from the outside is having an explicit router in GPT-5, but then also basically having an implicit router in terms of the thinking-spending thing. That conflates things a little bit, right? At some point, you do kind of need to merge them, or else you're just going to get these weird bumps where sometimes the router at the top decides something and it's wrong, and actually, if you just handed it to GPT-5, it would have figured it out.

Yeah. And I think we'll figure out the correct abstractions over time.

Is the intention still to merge? Because that's what was said in the paper.

Yeah. I think eventually we'll have AGI and you're not going to have to worry too much about how hard to think directly. It'll just be one tool that you always go to, and it knows how long to think for and things like that. I think the abstractions and the way that we drive these things today will change. We've changed so much already—from having a non-thinking model to choosing between 2, and now we can sort of route and decide how hard you want to think. We're adding lots of knobs, and eventually it'll probably simplify.

Yeah. Another super-interesting knob that everyone is doing is context compaction, or memory compaction. What's going on there?

Nothing to share at the moment.

Let me share. Okay, clearly an important feature, clearly inspired by Codex usage as well, obviously. But from the engineer's point of view, it feels like I used to do that as part of my harness, and now the model's doing it for me. I don't know how to think about that in terms of—I guess I'm used to having more control, and now I have less.

Yeah. Is there a specific, like—

There's a specific question I'm just getting feedback on: Is this a trend that we need, where it's basically a permanent fact of life from here on out?

Oh, I see. I don't know. I worked on long context; that was what I was working on last, for GPT-4.1, where I think we 10×ed the context-window effect for GPT-4.1. And so there'll always be some dance of, well, if we want to push as much as we can do, not only should we increase the length of the context window, but we should also have strategies for keeping that context window available for as long as possible.

I'm guessing that both things will happen, just because we want to put as much power into the models as possible.

Yeah.

Yeah. I think we're still in a period where we should all be expecting changes in the interfaces that all the models give to us. That way, we can improve the models, because if we lock the interface, I think what would be sad from my perspective is if we lock the interface and discover something new about models, we might trap that improvement under an interface that needs to change.

Right? Talking about long context as well, there is some discussion about context rot, or the utilization of the context. Even if you gave us a million-token context, we probably wouldn't use all of it. What's the recommendation there? Where are things going? Are we going to have perfect context by next year? Is that an impossible dream? I don't know.

No, it's not an impossible dream. I'll give a shout-out to some of the evals that we did for GPT-4.1 called Graphwalks, where—

I love Graphwalks.

We covered this in the podcast.

Yeah, we did. I think if you look over time, all of those evals are still climbing. One of the interesting things about that is that you have to do complicated transformations across the entire context window.

But the problem is, if you only have to sample from 1 point in the context window, it's sort of easy. Whereas with those graph-walk problems, you're having to do multiple transformations across the entire context window. I think—keep watching those. They've been climbing, and they'll continue to climb. I would say that that's definitely a temporary issue that we're climbing over time.

So, is 10 million tokens realistic? Is 100 million? Where does it end? Is there a natural end, or is there no end and we just go as far as I can see?

Oh gosh, I don't know. What do you think?

I feel like there are use cases that require billions, and there are use cases that require many, many billions, maybe trillions.

Out of curiosity, what would be billions of tokens?

We just had a context-engineering discussion about a RAG codebase over support issues for a company, and it was 100,000 documents totaling about 8 billion tokens. You can't stick that in a context window for now.

That's fair. I would still say I don't know, but I've been really surprised. It reminds me of when I was doing more information retrieval work, and BM25 and these very simple n-gram indexes were just super hard to beat.

I think the agents with GPT feel really similar to me. They're just unreasonably effective. But then I will not use your 10-million-token context window, even if you gave it to me.

Maybe. But what if we're using that context window in service of some larger goal that just has a lot of sub-search calls? That's why I'm saying I just don't know, and I think that's what makes it so exciting.

Yeah. I would say that the other modalities, like video, would eat up a lot. Then, obviously, the hard sciences have proteins and all of that, where a lot of information is just encoded in physics.

I have mixed feelings about it, just because I'm like, "Well, this will never scale—not with full attention—and we probably just need to invest in systems anyway," which means we're good with what we have. I mean, get your graph walks up, but—

I don't know if we need 10–100x. Maybe we need to figure out ways to go 1,000x or 1 millionx.

Yeah.

Right. These are just different slopes. I'm definitely glad that you're happy with the current context windows. I think my dream would be to push it and see what happens anyway. But I think the engineers' incentive is always to say, "Well, the systems matter more than the models," and the researchers' incentive is to say—

"Screw your systems. We'll just scale the models."

Oh, no. So, differently. I think that's one of the most beautiful things about—

Post-training at OpenAI is that everyone—

Co-design.

Yeah, it's all co-design. I spend a lot of time just doing our systems work, and I also do lots of things where I'm making graph walks and doing a lot more on the learning side. I think it's a great culture to have a place where people just move seamlessly between the 2.

What are you guys hiring for? Presumably, you're hiring, but what are you hiring for that's hard to hire? What is the skill set where you're thinking, "We really need this. We can't find it. Please, everyone, go skill up on this"?

This is definitely my personal opinion, but I think we're still having trouble—not at OpenAI, but as a whole—producing lots of people who want to do both systems work and ML work. If you're trying to push the frontier, you don't know which place is currently bottlenecking the frontier, and it changes all the time. Even within 1 project, it might change multiple times where the current bottleneck is.

I think the education system we have right now isn't really optimized for that. I studied math, and then I was very lucky to have some great mentors after school who taught me to be a good software engineer. But it seems like, if we're going to be in this place for a while—and I think we will be—we should probably be producing more students who are great at doing both distributed systems and a lot of core engineering, as well as the statistics and other things required to be a good machine learning researcher.

If we were to throw Codex at it, obviously we can't use Codex for everything.

Let's say: which will progress faster? Which is more solvable by an LLM?

That is a spicy question.

You can't say they're both equally hard. Maybe they are. I don't know. They're differently hard. One is more hill-climbable than the other. Which is it? Because then we can go do it.

Okay. I think one thing that's slightly simpler about some of the ML research—or, you know, ML research is also distributed systems, to be clear—is that some of the things traditionally called ML research are things that you can treat a bit more as a black box. Whereas the environment to train on, building these different systems, is actually just a complicated data-engineering problem.

Theoretically, I would say that they're probably roughly equal. But I think there's some amount of effort that goes into making the environments for—

Yeah, but they require GPUs themselves as well.

Yeah. I guess they both would, but that's my guess. I don't have high confidence in it.

A lot of people are building these AI scientists that automate research. You guys have your own benchmark, PaperBench, and that's the one area that, for example, at Cognition, we've just decided not to do because it's so hard. Any other people on the post-training team you want to shout out who have done interesting work this year? They should get more attention, but they're not getting credit.

For sure, everyone on the shopping team I was just working with: Andrew Hoyle, Manuka Stratta, and John Hallman. They're all great people. Isa Fulford, obviously, is the manager for it.

And she was the original Deep Research person.

There were like 3 of them. Yeah, and so definitely that part of the team. But everyone is so great, and I think it's hard to give out a list. It's a really fun time on post-training right now. It's exciting every day. It feels like we're all enjoying our Diet Cokes together in the office late at night.

I did want to squeeze this in before we end. Nobody serious is saying that pre-training is dead. It's just a meme. There's a lot of work going on in pre-training. In fact, a lot of my researcher friends are saying that too much money is going to post-training. That's also spicy.

I don't know. One of the charts I hold in my memory from this year is the Grok 4 chart. I don't know if you've seen it, but it's basically saying, "We scaled pre-training to here, at about this level of compute, and now we're spending the same level of compute on post-training as well." That's very controversial, I guess, because we're all used to post-training taking orders of magnitude less data and compute, or whatever. Obviously, we're scaling that up now. Do we get to a point where they're equal? I don't know. But that's a topic for conversation: how much do we invest in this versus more of the different pre-training that you've done before?

Yeah. So, first off, neither one of those is dead. I think it's really interesting to be living through something when all of my other historic or technological revolutions are things that I read about in history books, and this one's live as it's happening.

Yeah. We don't know the end yet.

Yeah. There's this almost fog of war where I'm like, "Did people think that, when we got the steam engine, they would have the factories?" I don't know if you know this, but factories used to be very linear because you had to drive 1 motor across an entire room. When electricity was developed, they just tried to do the same thing, and they said, "Ah, this isn't all that useful."

It took a couple of decades before they realized what was most ergonomic. Then manufacturing was transformed by electricity. I think that really gives me no confidence in saying, "Oh, this thing is dead."

Yeah, our timelines are so short. But recently, the way good ideas get experimented with, funded, and propagated is still on a human timeline. That's not on an AI timeline.

Yeah. And so I think things will maybe be dormant, but they'll be spiky.

There will all of a sudden, you know—yeah, yeah—and then we’ll all feel different. It’s like—what’s the meme? It’s so over. We’re so back.

It’s going to be that many times. And I think—

Some emotional stabilizing to it is probably going to be good for everyone’s sanity.

Yeah. More sanity. Well, thank you so much for joining. Thanks for all the great post-training this year.

Yeah, thank you. And continue giving feedback. I love to hear what you think.

Yeah. Awesome.