No Priors 第113期｜对话 OpenAI 的 Eric Mitchell 与 Brandon McKinzie

Today I'm speaking with Brandon McKinzie and Eric Mitchell, two of the minds behind OpenAI's o3 model. o3 is the latest in the line of reasoning models from OpenAI, with the ability to figure out what tools to use and then use them across multistep tasks. We'll talk about how it was made, what's next, and how to reason about reasoning. Brandon and Eric, welcome to No Priors.

Thanks for having us.

Yeah, thanks for having us.

Do you mind walking us through o3? What's different about it? What was the breakthrough in terms of focusing on reasoning? You're adding memory and other things versus a core foundation model or LLM—what is it?

So o3 is our most recent model in this o-series line of models that are focused on thinking carefully before they respond. These models are, in some vaguely general sense, smarter than models that don't think before they respond. Similarly to humans, it's easier to be more accurate if you think before you respond.

I think the thing that's really exciting about o3 is that not only is it smarter if you make an apples-to-apples comparison to our previous o-series models, it's also better at giving you correct answers to math problems, factual questions about the world, or whatever. This is true and it's great, and we will continue to train models that are smarter.

But it's also very cool because it uses a lot of tools that enhance its ability to do things that are useful for you. You can train a model that's really smart, but if it can't browse the web and get up-to-date information, there's a limitation on how much useful stuff that model can do for you. If the model can't actually write and execute code, there's a limitation on the sorts of things an LLM can do efficiently, whereas a relatively simple Python program can solve a particular problem very easily.

Not only is the model, on its own, smarter than our previous o-series models, which is great, but it's also able to use all these tools that further enhance its abilities. Whether that's doing research on something where you want up-to-date information, having the model do some data analysis for you, or having it do the data analysis and then review the results and adjust course as it sees fit—instead of you having to be so prescriptive about each step along the way—the model can take these high-level requests: "Do some due diligence on this company, maybe run some reasonable forecasting models on this or that thing, and then write a summary for me."

The model will infer a reasonable set of actions to take on its own. So it gives you a higher-level interface to doing some of these more complicated tasks.

That makes sense. It sounds like there are a few different changes between your core GPT models. Now you have something that takes a pause to think about something, so at inference time there's more compute happening, and then it can do sequential steps because it can infer what those steps are and go act on them. How did you build or train this differently from just a core foundation model? When you did GPT-2.5 and GPT-4 and all the various models that have come over time, what is different in terms of how you actually construct one of these?

I guess the short answer is reinforcement learning is the biggest one. Rather than just having to predict the next token in some large pretraining corpus from everywhere, essentially, now we have a more focused goal: the model solving very difficult tasks and taking as long as it needs to figure out the answers to those problems.

Something that was kind of magical from a user-experience perspective for me was that, in the past, we've talked a lot about test-time scaling for our reasoning models. For a lot of problems, without tools, test-time scaling might occasionally work, but at some point the model is just ranting in its internal chain of thought. Especially for some visual-perception problems, it knows that it isn't able to see the thing that it needs, and it just loses its mind and goes insane.

I think tool use is a really important component to continuing this test-time scaling. You can feel this when you're talking to o3. At least, my impression when I first started using it was that the longer it thinks, the more I get the impression that I'm going to get a better result. You can watch it do really intuitive things, and it's a very different experience.

Being able to trust that, as you're waiting, it's worth the wait and you're going to get a better result because of it, and that the model isn't just off doing some totally irrelevant thing—that's cool.

I think in your original post about this, you also had a graph that showed how long it thought versus the accuracy of the result, and it was a really nice relationship. Clearly, thinking more deeply about something really matters.

In the long run, do you think there will be a world where we have a split or bifurcation between models that are fast, cheap, and efficient and get certain basic tasks done, and another model where you upload a legal M&A folder and it takes a day to think? It's slow and expensive, but then it produces output that would take a team of people a month to produce. How do you think about the world in terms of how all this is evolving or where it's heading?

I think for us, unification of our models is something that Sam has talked about publicly. We have this big, crazy model switcher in ChatGPT, and there are a lot of choices. We have a model that might be good at any particular thing a user might want to do, but that's not helpful if it's not easy for the user to figure out which model they should use for that task.

Making the models better able to provide a more intuitive experience is definitely valuable and something we're interested in doing. That applies to this question of whether we're going to have 2 models that people pick between, a zillion models that people pick between, or whether we put that decision inside the model.

Everyone is going to try things and figure out what works well for the problems they're interested in and the users that they have. But the question of how to make that decision as effective, accurate, and intuitive as possible is definitely top of mind.

Is there a reason from a research perspective to combine reasoning with pretraining, to try to have more control of this? If you just think about it from the product perspective of the end user dealing with ChatGPT, they don't care. They just want the right answer and the amount of intelligence required to get there in as little time as possible.

The ideal situation is that it's intuitive: how long should you have to wait? You should have to wait as long as it takes for the model to give you a correct answer.

I hope we can get to a place where our models have a more precise understanding of their own level of uncertainty. If they already know the answer, they should just tell you it. If it takes them a day to actually figure it out, they should take a day.

But you should always have a sense that it takes exactly as long as it needs to for that model's current intelligence. I feel like we're on the right path for that.

I wonder if there isn't a bifurcation, though, between an end-user product and a developer product. There are lots of companies that use the APIs for all of these different models for very specific tasks, and on some of them they might even use open-source models with really cheap inference and stuff that they control more.

It would be great if you could just tell the model, "Hey, this is an API use case, and you really can't be over there thinking for 10 minutes. We've got to get an answer to the user." It would be great if the models could get to be more steerable like that as well.

Yeah, I think it's just a general steerability question. At the end of the day, if the model is smart, you should be able to specify the context of your problem, and the model should do the right thing.

There are going to be some limitations because maybe just figuring out, given your situation, what the right thing to do is might require thinking in and of itself. It's not that you can obviously do this perfectly, but pushing all the right parts of this into the model to make things easier for the user seems like a very good goal.

Can I go back to something else you said? The first guest we ever had on the podcast was actually Noam Brown. I heard of him two-plus years ago. It would be great to get some intuition from you guys for why tool use helps test-time scaling work much better.

I can give some very concrete cases on the visual reasoning side of things. There are a lot of cases where—and this also goes back to the model being able to estimate its own uncertainty—you'll give it some kind of question about an image, and the model will very transparently tell you, “I don't know. I can't really see the thing you're talking about very well.” It almost knows that its vision isn't very good.

What's kind of magical is that when you give it access to a tool, it's like, “Okay, well, I've got to figure something out. Let's see if I can manipulate the image or crop around here,” or something like that. What that means is that it's a much more productive use of tokens as it's doing that. Your test-time scaling slope goes from something like this to something much deeper. We've seen exactly that: the test-time scaling slopes without tool use and with tool use for visual reasoning specifically are very noticeably different.

Yeah. I'd also say that for writing code, there are a lot of things an LLM could try to figure out on its own but that would require a lot of attempts and self-verification, which you could write a very simple program to do in a verifiable and much faster way. If you say, “Hey, do some research on this company and use this type of valuation model to tell me what the valuation should be,” you could have the model try to crank through that and fit those coefficients or whatever in its context, or you could literally have it write the code to do it the right way and know what the actual answer is. I think part of this is that you can allocate compute a lot more efficiently because you can defer things the model doesn't have a comparative advantage in doing to a tool that's really well suited to doing that thing.

One of the ways I've been using some form of o3 a lot is Deep Research. I think that's basically a research analyst AI that you all have built that will go out, look up things on the web, synthesize information, and chart things for you. It's pretty amazing in terms of its capability set. Did you have to do anything special in terms of any specific reinforcement learning for it to be better at that, or other things that you built against it? How did you think about the data training for it—the data that was used for training it? I'm just curious how that product, if at all, is a branch off of this and how you thought about building that specifically as part of this broader effort?

I think when we think about tool use, browsing is one of the most natural places where you think of starting. It's not always easy. The initial browsing that we included in GPT-4 a few years back was hard to make work in a way that felt reliable and useful. But in the modern era—last year; 2 years ago is ancient history—it feels like a natural place to start because it's so widely applicable to so many types of queries. Anything that requires up-to-date information should benefit from browsing.

In terms of a test bed for, “Hey, is the way we're doing RL really working? Can we really get the model to learn longer-horizon, meaningful, extended behaviors?” it feels like a natural place to start. It's also fairly likely to be useful in a relatively short amount of time. So it's like, yeah, let's try that.

I mean, in RL, at the end of the day, you're defining an objective. If you have an idea of who's going to find this most useful, you might want to tailor your objective to who you expect to be using the thing, what you expect they're going to want, and what their tolerance is. Do they want to sit through a 30-minute rollout of Deep Research? When they ask for a report, do they want 1 page, 5 pages, or a gazillion pages? You definitely want to tailor things to who you think is going to be using it.

I feel like there's a lot of almost white-collar behavioral work, or knowledge work, that you all are really capturing through this sort of tooling going forward. You mentioned software engineering as one potential area; Deep Research and analytical jobs are another, where there's all sorts of really interesting work to be done that's super helpful in augmenting what people are doing. Are there 2 or 3 other areas that you think are the most near-term interesting applications for this, whether OpenAI is doing it or others should do it? I'm just curious how you think about the big application areas for this sort of technology.

I guess my very biased one that I'm excited about is coding, and also research in general—being able to improve the velocity at which we can do research at OpenAI and others can do research when they're using our tools. I think our models are getting a lot better very quickly at being actually useful, and it seems like they're reaching some kind of inflection point where they are useful enough that you want to reach out to and use them multiple times a day—for me, at least—which wasn't the case. They were always a little bit behind what I wanted them to be, especially when it came to navigating and using our internal codebase, which is not simple. It's amazing to see more recent models actually spending a lot of time trying to understand the questions that we ask them and coming back with things that save me many hours of my own time.

People say that's the fastest potential bootstrap, right? In terms of each model subsequently helping to make the next model better, faster, cheaper, et cetera. People often argue that that's almost like an inflection point on the exponent toward superintelligence: this ability to use AI to build the next version of AI.

Yeah. There are so many different components of research, too. It's not just sitting off in the ivory tower thinking about things. There's hardware, various components of training and evaluation, and stuff like this, and each of these can be turned into some kind of task that can be optimized and iterated over. So there's plenty of room to squeeze out improvements.

We talked about browsing the web, writing code—arguably the greatest tool of all, right? Especially if you're trying to figure out how to spend your compute, you can write more efficient code—generating images, writing text. There are certainly trajectories of action that I think are not in there yet, right? Like reliably using a sequence of business software. I'm really excited about the computer-use stuff.

It kind of drives me crazy, in some sense, that our models are not already just on my computer all day, watching what I'm doing. I know that could be creepy for some people, and I think you should be able to opt out of that, or have it opted out of by default. I hate typing, too. I wish I could just be working on something on my computer, hit some issue, and just ask, “All right, what am I supposed to do with this?”

I think there's tons of space for improving how we interact with the models, and this goes back to them being able to use tools in a more intuitive way—using tools closer to how we use them. It's also surprising to me how intuitively our models use the tools we give them access to. It's weirdly humanlike, but I guess that's not too surprising given the data they've seen before.

But, yeah, I think a lot of things are weirdly humanlike. My intuition for why tool use is so impactful to test-time scaling—why is the combination so much better? Take any role: you can make a decision when you're trying to make progress against a task as to whether you get external validation or sit and think really hard. Usually, you want to do one or the other; one is more efficient than the other. It's not always just sitting in a vacuum and thinking really hard with what you know.

Yeah, absolutely. You can seek out new inputs; it doesn't have to be this closed system anymore. I do feel like the closed-system-ness of the models is still a limitation in some ways. You're not necessarily turning this—I mean, I think it'd be great if the model could control my computer, for sure—but in some sense, there's a reason we don't go hog-wild and say, “Oh, yes, here's the keys to the kingdom. Have at it.”

There are still asymmetric costs to the time you can save and the types of errors you can make. We're trying to iteratively deploy these things, try them out, and figure out where they're reliable and where they're not. If you did just let the model control your computer, it could do some cool stuff; I have no doubt. But do I trust it to respond to all of the random emails that Brandon sends me? Actually, maybe for that task it doesn't require that much intelligence. More generally, do I trust it to do everything I'm doing? Some things—and I'm sure that set of things will be bigger tomorrow than it was yesterday.

But yeah, I think part of this is that we limit the affordances and keep it a little bit in the sandbox, just out of caution, so that you don't send some crazy email to your boss, delete all your texts, delete your hard drive, or something.

Is there some sort of organizing mental model for the tasks one can do with increasing intelligence, test-time scaling, and improved tool use? I look at this and I'm like, okay, you have the complexity of the task and the time scale. Then you have the ability to come up with these RL rewards and environments, right? Maybe you have some intuition about diversity and generalization across the different things you can be doing, but it seems like a very large space. Scaling RL—new-gen RL—is not obvious to me. How do you do it, or how do you choose the path? Is there some sort of organizing framework that you guys have that you can share?

I mean, I don't know if there's one organizing framework. I think there are a few factors, at least, that I think about. In the very, very grand scheme of things, it's: How much uncertainty in the environment do I have to wrestle with in order to solve this task? For some things, it's purely factual—who was the first president of the United States? There's zero environment I need to interact with to reach the answer to that question correctly. I just need to remember the answer and say it.

If I want you to write some code that solves a problem, now I have to deal with a little bit of stuff that's not purely internal-model stuff. I need to execute the code, and that code execution environment is maybe more complicated than my model can memorize internally. So I have to write the code, execute it, make sure it does what I thought it did, test it, and then give it to the user. The amount of that sort of stuff outside the model that you have to deal with—you can't just recall the answer and give it to the user; you have to test something, run an experiment in the world, and then wait for the result of that experiment—the more you have to do that, the more uncertain the results of those experiments are. In some sense, that's one of the core attributes of what makes tasks hard.

Another factor is how simulatable they are. Stuff that's really bottlenecked by time—the physical world—is harder than stuff that we can simulate really well. It's not a coincidence that so many people are interested in coding and coding agents, and that robotics is hard and slower. I used to work on robotics, and it's frustrating in a lot of ways. How much of the external environment you have to deal with, and how much you have to wrestle with the unavoidable slowness of the real world, are 2 dimensions that I think about.

It's super interesting because, historically, one of the things that has continued to be really impressive is the degree to which these models are generalizable. When GitHub Copilot launched, it was on Codex, which was a specialized code model, and eventually that just got subsumed into more general-purpose models in terms of what a lot of people are actually using for coding-related applications.

How do you think about that in the context of robotics? There's probably a dozen different robotics foundation-model companies now. Do you think that eventually just merges into the work you're doing, in terms of there being big general-purpose models that can do all sorts of things? Or do you think there's a lot of room for these standalone, other types of models over time?

I will say that one thing that's always struck me as kind of funny about us doing RL is that we don't yet do it on the most canonical RL task of robotics. I personally don't see any reason why these couldn't be the same model.

I think there are certain challenges. Do you want your RL model to be able to generate an hour-long movie for you natively, as opposed to as a tool call? That's where it's probably tricky, to have more conflict between having everything in the same set of weights. But certainly, the things you see o3 already doing in terms of exploring a picture and things like that are early signs of an agent exploring an external environment. So I don't think it sounds too far-fetched to me.

Yeah. I think the other thing that came up earlier was the intelligence-per-cost thing. The real world is an interesting litmus test because, at the end of the day, there's a frame rate in the real world that you need to live on. It doesn't matter if you get the right answer after you think for 2 minutes. The ball is coming at you now, and you have to catch it. Gravity's not going to wait for you. That's an extra constraint that we get to at least softly ignore when we're talking about these purely disembodied things.

That's kind of interesting, though, because really small brains are very good at that. You look at a frog, start looking at different organisms, and look at relative compute. Very simple systems are very good at that—ants, for example. I think that's a fascinating question in terms of what's the baseline amount of capability that's actually needed for some of these real-world tasks that are reasonably responsive in nature.

It's really tricky with vision, too. Our models have some, I think, maybe famous edge cases where they don't do the right thing. I think Eric probably knows where I'm going with this. I don't know if you've ever asked our models to tell you what time it is on a clock. They really like the time 10:10. So that's usually what I tell people. It's like over 90% or something like that of all clocks on the internet are 10:10, and it's because it looks like, I guess, a happy face and it looks nice.

But what I'm getting at is that our visual system was developed by interacting with the external world and having to be good at navigating things and avoiding predators. Our models have learned vision in a very different way. I think we'll see a lot of really interesting things if we can get them to close the loop by reducing their uncertainty through taking actions in the real world, as opposed to just thinking about stuff.

Hey, Eric, you brought up the idea of how what in the environment can be simulated, right, as an input to how difficult it will be to improve on this. As you get to long-running tasks, let's just take software engineering. There's a lot of interaction that's not just me committing code continually. It's like, I'm going to talk to other people about the project, in which case you need to deal with the problem of whether you can reasonably simulate how other people are going to interact with you on the project in an environment. That seems really tricky.

I'm not saying that o3, or whatever set of foundation models we have now, doesn't have the intelligence to respond reasonably, but how do you think about that simulation being true to life—as true to life as the real world—as you involve human beings in an environment?

In theory, my spicy take on that is that o3 is already, in some sense, simulating what it would be like for a single person to do something with a browser or something like that. I don't know why you couldn't train 2 of them together, so you'd have 2 people interacting with each other. There's no reason you can't scale all this up so that models are trained to be really good at cooperating with each other.

There's already a lot of existing literature on multi-agent RL. If you want the model to be good at collaborating with a bunch of people, maybe a not-too-bad starting point is making it good at collaborating with other models.

Man, someone should do that. Yeah, yeah, yeah. We should really start thinking about that. Eric, I think it is a little bit spicy because, yes, the work is going on. It is interesting to hear you think that's a useful direction.

I think lots of people would still like to believe—not me—that my comment was extra good on this pull request or whatever it is, right? I can sympathize with that. Sometimes I see our models training and I'm like, "What are you doing? You're taking forever to figure this out." I actually think it would be really fun if you could train models in an interactive way. Forget just test time; I think it'd be really neat to train them to do something like that, to be able to intervene when it makes sense.

Yeah, just more me being able to tell the model, "Cut it out," in the middle of its chain of thought, and it being able to learn from that on the fly, I think would be great.

Yeah, I do think this is the intersection of these 2 things: it’s both a point of contact with the external environment, which can have very high uncertainty—humans can be very unpredictable in some cases—and it’s limited by the tick of time in the real world. If you want to deal with actual humans, they have a fixed clock cycle in their head. So if you want to do this in the literal sense, it’s hard, and scaling it up and making it work well isn’t obvious.

We are a super-expensive tool call. If you’re a model, you can either ask me, the meatbag over here, to help with something, and I’ll try to think really slowly. In the meantime, it could have used a browser and read 100 papers on the topic. So how do you model that trade-off? But the human part is important. I think in any research project, my interactions with Brandon are the hardest part of the project. Writing the code—that’s the easy part.

Well, there’s some analogy from self-driving.

Elad’s going to say that hanging out with me every week is the hardest part of doing this podcast, but it’s my favorite part. Look at how healthy their relationship is. Eric, we need to learn from this.

No, we’re honest. It’s okay. We have to work through it.

In self-driving, one of the classically hard things to do was predict the human, the child, and the dog—the agents in the environment—versus what the environment was. I think there’s some analogy to be drawn there.

Going back to how you progress the o-series of models from here, is it a reasonable assessment that some people have that the capabilities of the models are likely to advance in a spikier way because you’re relying, to some degree, more on the creativity of research teams in making these environments and deciding how to create these evals, versus scaling up on existing data sets in pretraining? Is that a fair contrast? Spiky, or what’s the plot here? What’s the x-axis and the y-axis? The domain is the x-axis, and y is capability.

Yes, because you’re choosing which domains you’re really creating this RL loop in. I think this is a very reasonable hypothesis to hold. There’s some counterevidence that should be factored into people’s intuitions. Sam tweeted an example of some creative writing from one of our models that, I’m not an expert, and I’m not going to say this is publishable or groundbreaking, but I think it probably updated some people’s intuitions about what you can train a model to do really well.

There are structural reasons why you’ll have some spikiness, just because, as an organization, you have to decide, “We’re going to prioritize X, Y, Z stuff.” As the models get better, the surface area of what you could do with them grows faster than you can potentially say, “This is the niche we’re going to carve out. We’re going to try to do this really well.” So I think there’s some reason for spikiness. But I think some people will probably go too far in saying, “These models will only be really good at math and code, and everything else—you can’t get better at it.” I don’t think that’s the right intuition to have.

Yeah, I think probably all major AI labs right now have some partitioning between, “Let’s define a bunch of data distributions we want our models to be good at and throw data at them,” and another set of people in those same companies is probably thinking about how you can lift all boats at once with some algorithmic change. I think we definitely have both types of efforts at OpenAI.

Especially on the data side, there are naturally going to be things that we have a lot more data of than others. Ideally, we have plenty of efforts that won’t be so reliant on the exact subset of data we did RL on, and they’ll generalize better.

I get pitched every week, and I bet Elad does too: a company that wants to generate data for the labs in some way, or provide access to human experts, or whatever it is. There are infinite variations on this. If you could wave a magic wand and have a perfect set of data, what would it be that would advance model quality today?

This is a dodge, but uncontaminated evals are always super valuable, and that’s data. You want good data to train on, which is of course valuable for making the model better, but it’s often neglected how important it is to have high-quality data—which has a different definition of “high quality” when it comes to an eval.

The eval side is often just as important because you need to measure stuff. As you know from trying to hire people or whatever, evaluating the capabilities of a generally capable agent is really hard to do in a rigorous way. I think evals are a little underappreciated.

Yeah, evals are especially important with some of our recent models, where we’ve kind of run out of reliable evals to track because they just solved a few of those. On the training side, I think it’s always valuable to have training data that’s at the next frontier of model capabilities.

A lot of the things that o3 and o4-mini can already do—basic tool use, for example—we probably aren’t super in need of new data like that. But it would be hard to say no to a data set that’s a bunch of multi-turn user interactions and some codebase that’s 1,000,000 lines of code, representing a 2-week research task of adding some new feature to it that requires multiple pull requests. Something that was super high quality and had a ton of supervision signals for us to learn from—that would be awesome to have. I definitely wouldn’t turn that down.

You play with the models all the time—I assume a lot more than average humans do. What do you do with reasoning models that you think other people don’t do enough of yet?

Send the same prompt many times to the model and get an intuition for the distribution of responses you can get. It drives me absolutely mad when people do these comparisons on Twitter or wherever and they’re like, “I put the same prompt into blah blah and blah blah, and this one was so much better,” because something we talked about when we were launching is that o3 can do really cool things when it chains together a lot of tool calls. Sometimes, for the same prompt, it won’t have that moment of magic, or it will just do a little less work for you.

The peak performance is really impressive, but there is a distribution of behavior. People often don’t appreciate that there’s this distribution of outcomes when you put the same prompt in. Getting an intuition about that is useful.

As an end user, I do this, and I also have a feature request for your friends in the product organization. I’ll ask Oliver or someone, but I want a button where, assuming my rate limits support it, I can run the prompt automatically 100 times every time, even if it’s really expensive. Then I want the model to rank them and just give me the top 1 and 2.

Interesting. You could also synthesize the output across them, right? You could synthesize the output and see if there’s something there, although maybe you’re then reverting to the mean in some sense relative to that distribution or something. But it seems kind of interesting.

Yeah, maybe there’s a good infrastructure reason you guys aren’t giving us that button.

Well, it’s expensive, but I think it’s a great suggestion.

Yeah, yeah, I think it’s a great suggestion.

How much would you pay for that?

A lot. But I’m a price-insensitive user of AI.

I see. Perfect. You should have a Sarah tier as one of your tiers.

Exactly. Exactly.

Yeah, I really like sending prompts to our models that are at the edge of what I expect them to be able to do, just for fun. A lot of the time, before I’m about to do a programming task, I’ll ask the model to see if it can figure it out. A lot of times, there’s no hope of it being able to do it, and sometimes it comes back and I’m disappointed, like a disappointed father. But other times it does it, and it’s amazing and saves me tons of time.

I use our model almost like a background queue of work. I’ll shoot off tasks to it, and sometimes they’ll stick and sometimes they won’t. In either case, it’s always a good outcome if something good happens.

That’s cool.

I do that just to feel better about myself when it doesn’t work. I get depressed. I’m still providing value. When it works, I feel even worse about myself, so it’s very hit or miss.

This has been great, guys. Thank you.

Thanks so much for having us.

Yeah, thanks. It was fun.