Latent Space · 2025-06-19 · 78 min

Scaling Test Time Compute to Multi-Agent Civilizations — Noam Brown, OpenAI

Noam Brown

YouTube ↗

TL;DR

Noam Brown sees test-time compute as a durable second scaling curve, progressing from o1-preview to o1 to o3 and eventually from minutes of thought to hours, days, or longer. The gain is not merely buying more inference: o3 can outperform o1-preview without a radically longer trace because models are becoming “better at thinking.” Cost and serial wall-clock time remain the eventual soft ceilings.
Deep Research is Brown’s existence proof that reasoning models can improve in domains without an easily verifiable reward. Users can distinguish a strong research report from a mediocre one even when no crisp answer key exists, which is enough to support an improvement loop. His sharper criterion: if people genuinely cannot tell outputs apart, further hill-climbing “doesn’t really matter.”
Much of today’s agent infrastructure may be temporary, while reinforcement-fine-tuning data should remain useful across model generations. Brown expects hand-built harnesses, scaffolds, and perhaps routers to be “washed away by scale” as unified models internalize their functions. Reinforcement fine-tuning is different: collected domain data can be reused on more capable future models instead of becoming stranded engineering.
Coding agents already deliver real leverage, but Brown describes them as “geniuses” experiencing their first day on the job. He sends nearly anything he would code to Codex first and often receives a pull request about five minutes later, while o3 has largely replaced Google Search for him. The missing capability is accumulated job context: an agent with six months of experience would be dramatically more useful than one repeatedly starting from zero.
Brown’s multi-agent thesis is that individual AIs are still “the cavemen of AI,” and civilization-scale cooperation and competition could produce the larger capability jump. Anatomically similar humans moved from cavemen to semiconductors, nuclear reactors, and the moon through billions of people accumulating knowledge over thousands of years. His team is also studying how to extend a model’s thinking from roughly 15 minutes to hours and days, but its multi-agent method remains undisclosed.
Naïve self-play is not a guaranteed final step to superintelligence because the AlphaZero recipe relies on unusually clean two-player, zero-sum objectives. Outside that setting, optimization can select difficult but worthless behavior—for example, a math model generating 30-digit multiplication merely because another model struggles with it. Diplomacy similarly taught Brown that collaboration requires modeling and adapting to other players, not simply converging to a defensive equilibrium.
Alongside continued scaling, Brown highlights data efficiency, evaluation quality, and iteration speed as major constraints. He calls human-like sample efficiency one of the most important unsolved questions: a poker expert can profile an opponent in roughly a dozen hands where older AIs needed about 10,000. Meanwhile, three-hour or three-week reasoning runs slow the research loop itself, making unavoidable serial time “the strongest case for long timelines.”

Digest · the substance, structured for research

1. Cicero taught Brown to play—and exposed the value of steerable agents

Building Cicero forced Brown to understand Diplomacy well enough to distinguish a bug from a strategically brilliant, non-human move. Studying tournaments, tutorials, and the bot’s unconventional behavior improved his own play; after Cicero’s late-2022 release, he kept competing and won the World Diplomacy Championship in 2025 without using the system during the tournament.
Cicero’s language model was small—about 2.7B parameters—and occasionally behaved bizarrely. It might deny something visibly present earlier in the chat, insisting, “No, you’re lying.” Players usually attributed this to tiredness, drunkenness, or trolling because they were not expecting a bot; Brown thinks active suspicion would have made Cicero much easier to identify.
Brown’s present-tense claim is stronger: “GPT-4o and o3 are passing the Turing test.” Language quality has improved enough that conversational traps may no longer reliably expose a system, though Brown’s understanding is that current bots still do not perform very well at Diplomacy itself.
The safety lesson was unexpectedly positive. Cicero conditioned language generation on concrete intended actions, making its behavior interpretable and steerable rather than “a language model running loose.” Brown says safety researchers viewed that reasoning layer as a potentially useful control mechanism; he would like to see leading bots play one another in a Diplomacy benchmark.

2. Deep Research breaks the verifiable-reward objection

Brown is surprised that established researchers still assume reasoning models can only excel in math, coding, and other easily graded domains. Deep Research directly contradicts that view: “What is the best research report that you can generate?” has no simple answer key, yet the product is popular and the models perform extremely well in this domain.
The hosts pressed on whether a broad quality spectrum makes progress hard to recognize. Brown’s answer was that people can distinguish a good report from a mediocre or bad one, even though the system is “not 100%” and will improve. That comparative signal is enough to feed the feedback loop for refining the product and model.
His boundary condition is pragmatic rather than philosophical: models can improve wherever some measure of success exists, including subjective or difficult-to-define measures. If users cannot perceive any difference between two outputs, then optimization is irrelevant; otherwise, easy mechanical verification is helpful but not necessary.

3. System 2 only works once System 1 crosses a capability threshold

Brown’s qualification to the Thinking, Fast and Slow analogy is that extra thought cannot rescue an incapable base model. Chain-of-thought brought little benefit to very small models such as GPT-2, whereas larger pretrained models showed a lift. His analogy: asking a pigeon to contemplate chess for 1,000 years still will not produce strong play.
The benefit depends on whether the task contains information that reasoning can transform. GeoGuessr and spatial problems can reward deliberation; image recognition may benefit less. Without web access, a birth-date question is largely retrieval—“you either know it or you don’t”—although reasoning might narrow the historical period and produce an educated estimate.
Tic-tac-toe captures the current division. Brown says GPT-4.5 can draw the board and make legal moves but occasionally errs, while System 2 can enable perfect play. He leaves open whether a hypothetical GPT-6 might solve it instantly through System 1 alone: larger base models can absorb capabilities that currently require explicit reasoning.
Prior knowledge still accelerates deliberation. A brilliant person given three weeks could potentially reason through a novel game against an expert, but learned intuition makes the process vastly faster. System 1 and System 2 are therefore coupled scaling paradigms, not independent substitutes.

4. The ideal harness disappears, but reality cannot be rolled back

Brown’s position on agent scaffolding is blunt: “The ideal harness is no harness.” When Pokémon became an evaluation target, he preferred letting o3 fail unaided rather than engineering a structure that manufactured a higher score. A model doing badly is useful evidence; the research goal should be broader capability that eventually lifts Pokémon performance as a side effect.
Environmental tools are still legitimate. A chess model could be offered a tool to check whether a contemplated move is legal. Brown distinguishes that advance check from committing an illegal move, receiving feedback, then declaring, “Just kidding,” and selecting another action.
That distinction becomes non-negotiable in robotics. A model may simulate a movement, discover that it breaks something, and reject it; once a physical robot has acted, however, “the thing is broken.” Brown warns against relying on a test-time-compute paradigm that freely undoes consequential real-world actions.
Routers face the same long-term pressure as harnesses. A cheap model can recognize some hard requests, but it can also be fooled or overconfident; OpenAI’s stated destination is a single unifying model, with routing eventually built into the model itself. Brown hedges that provider routing may retain value, and he concedes that today’s routers can be commercially worthwhile.

5. Domain data should outlast short-lived agent scaffolding

Brown’s warning to developers is temporal: capabilities may change radically in three months, “let alone six months.” A product team should avoid spending six months perfecting infrastructure that a stronger base model could erase upon arrival, even when temporary scaffolding is rational for shipping now.
Reinforcement fine-tuning sits on the complementary side of that divide. It specializes a model around data the provider is unlikely to bake into a general release, and the same collected examples can fine-tune future, stronger models. That makes domain datasets more durable assets than elaborate workarounds for a particular model’s limitations.
The hosts translated this into a build-versus-wait question: rush to fine-tune today, or first construct the environment and reward signal? Brown did not prescribe one universal sequence, but emphasized the durable component—collecting data that remains useful as model capability scales.

6. Reasoning emerged as a contested second scaling paradigm

Brown’s conviction came from poker, Hanabi, and Diplomacy, where thinking before acting produced “orders of magnitude” gains—roughly equivalent to making a model 1,000 to 100,000 times larger. Language models instead answered immediately, making him skeptical that pretraining alone could reach superintelligence before economic limits intervened.
At a late-2021 meal, Ilya Sutskever asked Brown for his AGI timeline. Brown argued that general language models still lacked a general reasoning paradigm; absent one, even several more pretraining orders of magnitude would fall short unless someone could spend “a quadrillion dollars.” Brown’s mistake was believing the missing paradigm would take a long time to discover.
Sutskever and others had already explored reinforcement learning, with partial successes that accumulated across iterations as models improved and experiments became faster. Brown dates his own conviction to around October or November 2023, when the team saw “very conclusive signs of life” and OpenAI recognized the result strongly enough to scale it.
That allocation was controversial because 2023 OpenAI had more constrained compute and every reasoning experiment displaced another bet. Some questioned whether the behavior would generalize or deserved heavy investment. Brown recalls a researcher who left OpenAI dismissing the “Strawberry” work, then revising their view after seeing coworkers at a competing lab react to o1’s release with “Oh, crap, this is a big deal” and pivot their research agenda.

7. Startup-style resource allocation made the scaling bets possible

Brown credits OpenAI not merely with discovering pretraining and reasoning curves, but with organizing around them. The lab operated more like a startup pursuing AGI than a university: teams could collaborate, pool compute, and make painful centralized choices instead of preserving many independently allocated small experiments.
The original reinforcement-learning motivation was not solely test-time compute. Researchers worried that data, rather than compute, would become the wall and explored RL for greater data efficiency. The eventual paradigm delivered both: better use of data and an enormous effective increase in inference-side compute.
Brown remains impressed by Sutskever’s early vision. He cites emails from 2016–17 arguing that one big experiment can be more valuable than 100 small ones, as well as internal documents from 2021, 2022, and earlier. The broader research lesson is that “the best research is obvious in retrospect”; at discovery time, deciding that one large experiment deserves resources taken from numerous plausible alternatives is neither obvious nor politically easy.

8. Codex already changes software work, but persistent experience is missing

Brown’s current stack is Windsurf plus Codex, with reasoning models preferred for difficult work. “Basically anything that I would normally try to code up I try to do it with Codex first”; a self-contained request can run and return roughly five minutes later as a pull request.
Brown says anybody can use Codex and “feel the AGI,” but the feeling fades as users get accustomed to the capability and notice what it lacks. The hosts illustrated the same effect with Sora: its initial magic gave way to noticing unnatural motion and inconsistencies.
o3 has “basically replaced Google Search” for Brown and can perform a compact version of Deep Research in about three minutes. Its latency is awkward for pair programming, where GPT-4.1 may preserve flow; asynchronous Codex changes the ergonomics by letting a longer task proceed while the user does other work.
The core limitation is organizational memory: “They’re geniuses but it’s their first day on the job.” Agents repeatedly spend ten minutes rediscovering similar context instead of behaving like employees with six months of experience. Brown frames that as a model-and-product problem OpenAI must solve, not merely a missing third-party harness.

9. Remote-work agents turn alignment into an economic advantage

Brown expects the agent surface to expand beyond software engineering into many remotely performed tasks. His advice is broad: anyone whose job can be done remotely should learn what current models can and cannot do, because the feasible task envelope will continue widening.
Virtual assistance illustrates the potential. Human delegation creates a principal-agent problem around effort, cost, and fidelity to the principal’s preferences. An AI genuinely aligned with its user might do “a way better job than a human would”—Brown carefully corrects himself from “could”—because it can follow that user’s preferences more closely.
Personal alignment and safety alignment diverge at the target. Aligning perfectly to a user fails when that user wants to create “a novel virus that’s going to wipe out half of humanity.” The unresolved hierarchy spans individual preferences, collective human goals, and the constraints between them.

10. Multi-agent civilizations could dwarf single-model intelligence

Brown says “multi-agent team” understates his group’s remit. One program asks how to scale a model from roughly 15 minutes of thought to hours, days, and beyond, allowing it to attack much harder problems; another studies both cooperation and competition among agents.
He rejects the idea that humans occupy an extremely narrow intelligence band. Cavemen and modern humans are anatomically similar, yet only the latter inhabit a civilization capable of semiconductors, nuclear reactors, and moon landings. The difference is thousands of years of accumulated competition, cooperation, and knowledge across billions of people.
The analogy yields Brown’s signature thesis: today’s systems are “the cavemen of AI.” Billions of AIs interacting over long periods could build an AI civilization whose products and answers lie far beyond those of any isolated current model.
Brown declined to reveal whether knowledge is stored through skill libraries, retraining, or another mechanism, promising more in the “not-too-distant future.” He did say the approach differs materially from historical multi-agent work, which he considers overly heuristic and insufficiently faithful to the Bitter Lesson of scalable general methods.

11. GTO, adaptation, and self-play optimize different worlds

In poker, a game-theory-optimal policy is defensively unbeatable in expectation and still profits because opponents struggle to match it. Exploitative play can earn more from a weak opponent—bluffing heavily against someone who over-folds—but creates exploitable weaknesses of its own. “There’s no way to perfectly balance the two.”
Sample efficiency determines whether adaptation works. A human expert may profile someone’s strengths and weaknesses within roughly a dozen hands; Brown’s mid-2010s systems needed about 10,000. Poker bots therefore followed precomputed GTO strategies, beating experts and weak players alike but extracting less from weak players than an adaptive human could.
Diplomacy overturned Brown’s initial assumption that GTO would transfer. In a seven-player setting mixing competition with cooperation, a defensive equilibrium behaves poorly; the agent must model others and adapt to their style. His analogy is social rather than adversarial: if everyone at the table speaks French, continuing in English is not optimal.
The size of the hidden-state space also changes the technical problem. Poker search can enumerate 1,326 possible private-card states heads-up and then account for the other players; this breaks down in Omaha and especially in Stratego, with close to 40! possible states. The poker techniques would not work out of the box for Magic: The Gathering, though Brown guesses model-free RL could likely produce a superhuman bot with enough effort.
This is also where AlphaZero-style self-play stops being automatic. Two-player zero-sum self-play converges toward a useful minimax objective; open-ended math could instead reward one model for posing pointless 30-digit multiplication that another cannot solve. Brown agrees that choosing the right objective remains the unresolved problem.

12. Cost, data, evaluation, and serial time form the next walls

Test-time compute has a pretraining-like soft ceiling: three minutes can become three hours, three days, or three weeks, but each increase costs more. Efficiency is improving in parallel—o3 may not think radically longer than o1-preview on a question, yet performs far better because it has become “better at thinking.”
Wall-clock time may be the harder bottleneck. A three-week run requires at least three weeks before researchers can inspect results and choose the next experiment; parallelism cannot remove every sequential dependency. Brown calls this “the strongest case for long timelines,” with drug discovery especially constrained by slow validation and the lack of a perfect human-biology simulator.
Brown considers human-like data efficiency one of the most important unsolved research questions. The hosts added that algorithmic improvements may help and that humans learn from more than internet text. Brown’s own point was that current models are less data-efficient than humans and that the question remains unsolved.
Evaluation can distort research toward hard questions with cheap grading, such as Humanity’s Last Exam, while neglecting fuzzy tasks that are expensive to score. Brown says academic work still matters, but promising papers must be reproduced internally and tested at scale; useful inspiration and scalable evidence are separate filters.

13. Scale may make explicit world models unnecessary

Brown thinks sufficiently capable models develop an implicit world model that improves with scale, so an explicit world-model component may not be necessary. He once believed other agents had to be modeled explicitly because they have agency and motives, but now thinks capable models can develop theory of mind implicitly.
He presents this as a current view rather than a settled result: models may need explicit treatment in some multi-agent settings, but he increasingly expects general capability and scale to absorb that behavior.

14. Robotics has a slower research loop, with no clear case for humanoids

Brown’s robotics master’s was largely nominal; after briefly playing with LEGO robots, he shifted to AI for poker. His main takeaway from roboticists was that physical hardware makes experimentation much slower and more painful than software, helping explain the faster progress in language models and virtual-coworker tasks.
He has no strong view on humanoids, but weakly favors non-humanoid robotics after hearing the Physical Intelligence CEO’s case. Drones are his concrete example: they can be highly valuable without reproducing the human form.

Speaker 1

Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO of Decibel, and I'm joined by my co-host Brooks, founder of Small Eye.

Speaker 2

Hello, hello.

Speaker 1

We're here recording on a holiday Monday with Noam Brown from OpenAI. Welcome. We're so glad to have you finally join us. A lot of people have heard you. You've been rather generous with your time on the Lex Fridman podcast, and you've done a TED Talk recently talking about the thinking paradigm. But I think perhaps your most interesting recent achievement is winning the World Diplomacy Championship.

In 2022, you built Cicero, which was in the top 10% of human players. I guess my opening question is: how has your Diplomacy playing changed since working on Cicero and then now playing it personally?

Noam Brown

When you work on these games, you have to understand the game well enough to be able to debug your bot. If the bot does something that's really radical, and humans typically wouldn't do it, you're not sure if that's a mistake, a bug in the system, or if it's actually the bot being brilliant. When we were working on Diplomacy, I did this deep dive, trying to understand the game better. I played in tournaments, and I watched a lot of tutorial videos and commentary videos on games.

Over that process, I got better. Seeing the way the bot would behave in these games also taught me about the game. Sometimes it would do things that humans typically wouldn't do. When we released Cicero in late 2022, I still found the game really fascinating, so I kept up with it and continued to play. That led to me winning the World Championship in 2025, just a couple of months ago.

Speaker 1

There's always the question of centaur systems, where humans and machines work together. Was there an equivalent of what happened in Go, where you updated your play style?

Noam Brown

If you're asking if I used Cicero when I played in the tournament, the answer is no. Seeing the way the bot played and taking inspiration from that, I think it did help me in the tournament, yeah.

Speaker 1

Do people now ask Turing questions every single time they're playing Diplomacy, to try to tell if the person they're playing with is a bot or a human?

Noam Brown

Yeah, that's the one thing we worried about when we started. It was really interesting when we were working on Cicero because we didn't have the best language models. We were really bottlenecked on the quality of the language models. Sometimes the bot would say bizarre things. Ninety-nine percent of the time it was fine, but every once in a while it would say something really bizarre. It would just hallucinate about something.

Somebody would reference something that they had said earlier in a conversation with the bot, and the bot would be like, “I have no idea what you're talking about. I never said that.” Then the person would be like, “Look, you could just scroll up in the chat, and it's literally right there.” The bot would be like, “No, you're lying.”

When it did these kinds of things, people just shrugged it off as, “Oh, that's just—the person's tired, or they're drunk, or whatever, or they're just trolling me.” But I think that's because people weren't looking for a bot. They weren't expecting a bot to be in the games. We were actually really scared because we were afraid that people would figure out that there was a bot in these games, and then they would always be on the lookout for it.

If you're looking for it, you're able to spot it. So I think now that it's been announced and people know to look for it, they would have an easier time spotting it.

Speaker 2

It's adversarial.

Noam Brown

Yeah, at this point, the truth is that GPT-4o and o3 are passing the Turing test. So I don't think they can really ask that many Turing-test questions that would actually make a difference.

Speaker 1

Cicero was very small, like 2.7B, right?

Noam Brown

It was a very small model, yeah. It was one of the things we realized over the course of the project: you really benefit a lot from just having larger language models.

Speaker 2

Right, yeah. How do you think about today's perception of AI and a lot of the safety discourse? You're going to build a bot that's really good at persuading people and helping them win a game, and I think maybe today labs want to say they don't work on that type of problem. How do you think about that dichotomy, so to speak, between the two?

Noam Brown

Honestly, after we released Cicero, a lot of the AI safety community was really happy with the research and the way it worked because it was a very controllable system. We conditioned Cicero on certain concrete actions, and that gave it a lot of steerability. It was pursuing a behavior that we could very clearly interpret and define. It wasn't just a language model running loose and doing whatever it felt like. It was pretty steerable, and there was this whole reasoning system that steered the way the language model interacted with the human.

Actually, a lot of researchers reached out to me and said, “We think this is potentially a really good way to achieve safety with these systems.”

Speaker 1

I guess the last Diplomacy-related question that we might have is: have you updated or tested the o-series models on Diplomacy? And would you expect a lot more difference?

Noam Brown

I have not. I think I said this on Twitter at one point: this would be a great benchmark. I would love to see all the leading bots play a game of Diplomacy with each other and see who does best. I think a couple of people have taken inspiration from that and are actually building out these benchmarks and evaluating the models.

My understanding is that they don't do very well right now, but I think it really is a fascinating benchmark, and I think it would be a really cool thing to try out.

Speaker 1

We're going to go a little bit into the o-series now. I think the last time you did a lot of publicity, you were just launching o1. You did your TED Talk and everything. How have the vibes changed just in general? You said you were very excited to learn from domain experts, like in chemistry, and how they review the o-series models. How have you updated since, let's say, the end of last year?

Noam Brown

I think the trajectory was pretty clear pretty early on in the development cycle, and I think everything that's unfolded since then has been pretty on track for what I expected. So I wouldn't say that my perception of where things are going has honestly changed that much.

I think that we're going to continue to see this paradigm progress rapidly. I said before that we're going to see this paradigm continue to progress rapidly, and I think that's true even today. We saw that with going from o1-preview to o1 to o3: consistent progress. We're going to continue to see that going forward, and I think we're going to see a broadening of what these models can do as well.

We're going to start seeing agentic behavior. We're already starting to see agentic behavior. Honestly, for me, o3—I've been using it a ton in my day-to-day life. I just find it so useful, especially because it can now browse the web and do meaningful research on my behalf. It's kind of like a mini Deep Research that you can just get a response in 3 minutes. So, yeah, I think it's just going to continue to become more and more useful and more powerful as time goes on—and pretty quickly.

Speaker 2

Yeah, and talking about Deep Research, you tweeted, “If you need proof that we can do this in non-verifiable domains, Deep Research is kind of a great example.” Can you maybe talk about if there's something that people are missing? I feel like I hear that repeated a lot: it's easy to do in coding and math, but not in these other domains.

Noam Brown

I frequently get this question, including from pretty established AI researchers: we're seeing these reasoning models excel in math and coding, in these easily verifiable domains, but are they ever going to succeed in domains where success is less well-defined?

I'm surprised that this is such a common perception because we've released Deep Research and people can try it out. People do use it; it's very popular. That is very clearly a domain where you don't have an easily verifiable metric for success. What is the best research report that you can generate? And yet these models are doing extremely well in this domain.

I think that's an existence proof that these models can succeed at tasks that don't have as easily verifiable rewards.

Speaker 1

Is it because there's also not necessarily a wrong answer? There's a spectrum of Deep Research quality, right? You can have a report that looks good, but the information is kind of so-so, and then you have a great report. Do you think people have a hard time understanding the difference when they get the result?

Noam Brown

My impression is that people do understand the difference when they get a result, and I think they're surprised at how good the Deep Research results are. Certainly, it's not 100%. It could be better, and we're going to make it better. But I think people can tell the difference between a good report and a bad report.

And certainly, in a good report and in a mediocre report, that's enough to kind of feed the feedback loop to build the product and improve the model performance. I think if you're in a situation where people can't tell the difference between the outputs, then it doesn't really matter if you're hill-climbing on progress. These models are going to get better in domains where there is a measure of success.

I think this idea that it has to be easily verifiable or something like that—I don't think that's true. I think that you can have these models do well even in domains where success is a very difficult thing to define. It could sometimes even be subjective.

Speaker 1

One analogy that people lean on a lot, and that you've done as well, is the Thinking, Fast and Slow analogy for thinking models. I think it's reasonably well diffused now, the idea that this is kind of the next scaling paradigm. All analogies are imperfect. What is one way in which Thinking, Fast and Slow, or System 1/System 2, kind of doesn't transfer to how we actually scale these things?

Noam Brown

One thing that I think is underappreciated is that the pretrained models need a certain level of capability in order to really benefit from this extra thinking. This is kind of why you've seen the reasoning paradigm emerge around the time that it did. I think it could have happened earlier, but if you try to do the reasoning paradigm on top of GPT-2, I don't think it would have gotten you almost anything.

Speaker 1

Is this emergence?

Noam Brown

Hard to say if it's emergence necessarily, but I haven't done the measurements to really define that clearly. I think it's pretty clear. Wei et al. tried chain-of-thought with really small GPT models, and they saw that it just didn't really do anything. Then you go to bigger models, and it starts to get a lift.

I think there's a lot of debate about the extent to which this kind of behavior is emergent, but clearly there is a difference. So it's not like there are these 2 independent paradigms. I think that they're related in the sense that you need a certain level of System 1 capability in your models in order to be able to benefit from System 2.

Speaker 1

Yeah. I've tried to play amateur neuroscientist before and compare it to the evolution of the brain, and how you have to evolve the cortex first before you evolve the other parts of the brain. Perhaps that is what we're doing here.

Noam Brown

Yeah. You could argue that this is not that different from the System 1/System 2 paradigm because, if you ask a pigeon to think really hard about playing chess, it's not going to get that far. It doesn't matter if it thinks for 1,000 years; it's not going to be able to get better at playing chess.

So maybe, with animals and humans, you also need a certain level of intellectual ability in terms of System 1 in order to benefit from System 2.

Speaker 1

Yeah. Just this side tangent: does this also apply to visual reasoning? Let's say we have the GPT-4o, natively omni model type of thing. Does that also make o3 really good at GeoGuessr? Does that apply to other modalities too?

Noam Brown

I think the evidence is yes. It depends on exactly the kinds of questions that you're asking. There are some questions that I think don't really benefit from System 2. I think GeoGuessr is certainly one where you do benefit. I think image recognition, if I had to guess, is one of those things that you probably benefit less from System 2 thinking because you do or you don't.

Speaker 1

Yeah, exactly. The thing I typically point to is information retrieval. If somebody asks you, “When was this person born?” and you don't have access to the web, then you either know it or you don't. You can sit there and think about it for a long time. Maybe you can make an educated guess and say, “This person probably lived around this time, so this is a rough date,” but you're not going to be able to get the date unless you actually just know it.

Spatial reasoning, like tic-tac-toe, might be better because you have all the information there.

Noam Brown

Yeah, and I think it's true that with tic-tac-toe, we see that GPT-4.5 falls over. It plays decently well—I shouldn't say it falls over. It does reasonably well. It can draw the board and make legal moves, but it will make mistakes sometimes. If you really need that System 2 to enable it to play perfectly, now it's possible that if you got GPT-6 and just did System 1, it would also play perfectly. I guess we'll know one day, but I think right now you would need System 2 to really do well.

Speaker 1

What do you think are the things that you need in System 1? Obviously, general understanding of game rules. Do you also need to understand some sort of metagame—usually, this is how you value pieces in different games? Even though it's a game, how do you generalize in System 1 so that, in System 2, you can get to the gameplay, so to speak?

Noam Brown

I think the more that you have in your System 1, this is the same thing with humans. Humans, when they're playing a game like chess for the first time, can apply a lot of System 2 thinking to it. If you apply a ton of System 2 thinking—if you present a really smart person with a completely novel game and tell them, “You're going to play this game against an AI or a human that's mastered this game,” and tell them to sit there and think about how to play this game for 3 weeks—my guess is they could actually do pretty well.

But it certainly helps to build up that System 1 thinking, to build up intuition about the game, because it will just make you so much faster.

Speaker 1

I think the Pokémon example is a good one. System 1 kind of has all this information about games, and then once you put it in the game, it still needs a lot of harnesses to work. I'm trying to figure out how much we can take from the harness and have it in System 1, so that System 2 is as harness-free as possible. But I guess that's the question about generalizing games and AI.

Noam Brown

Yeah, I guess I view that as a different question. I think the question about harnesses, in my view, is that the ideal harness is no harness. I think harnesses are a crutch that eventually we're going to be able to move beyond.

So, only 2 calls. You could just ask o3. Actually, this is interesting because when this playing-Pokémon thing emerged as a benchmark, I was pretty opposed to evaluating this with our OpenAI models. My feeling was, “If we're going to do this eval, let's just do it with o3. How far does o3 get without any harness? How far does it get playing Pokémon?”

The answer is, not very far. That's fine. I think it's fine to have an eval where the models do terribly. I don't think the answer to that should be, “Let's build a really good harness so that now it can do well on this eval.” I think the answer is, “Let's improve the capabilities of our models so they can do well at everything, and then they also happen to make progress on this eval.”

Speaker 1

Would you consider things like checking for a valid move a harness, or is this in the model? With chess, you can either have the model learn in System 1 what moves are valid and what it can and cannot do, versus in System 2 figuring out where it went wrong.

Noam Brown

I think a lot of this is design questions. For me, I think you should give the model the ability to check if a move is legal, if you want. That could be an option in the environment: here's an action, a tool call that you can make to see if an action is legal. If it wants to use that, it can.

Then there's a design question of, what do you do if the model makes an illegal move? I think it's totally reasonable to say, if they make an illegal move, then they lose the game. I don't know—what happens when a human makes an illegal move in a game of chess?

Speaker 1

Mhm. I actually don't know. I don't think it's allowed. You're just not allowed to. Do you just lose the game? I don't know.

Noam Brown

So if that's the case, then I think it's totally reasonable to say, “We're going to have an eval where that's also the criteria for the AI models.”

Speaker 1

Yeah, but I think maybe one way to interpret that in research terms is: are you allowed to do search? One of the famous findings from DeepSeek is that MCTS wasn't that useful to them, but I think there are a lot of engineers trying out search and spending a lot of tokens doing that, and maybe it's not worth it.

Noam Brown

I'm making a distinction here between a tool call to check whether a move is legal or illegal and actually making that move and then seeing whether it ended up being legal or illegal. If that tool call is available, I think it's totally fine to make that tool call and check whether a move is legal or illegal.

I think it's different to have the model say, “I'm making this move,” and then get feedback that, “You made an illegal move,” and so then it's like, “Just kidding, I'm going to do something else now.” That's the distinction I'm drawing. Some people have tried to classify that second type of thing as test-time compute.

Speaker 1

You would not classify that as test-time compute?

Noam Brown

There are a lot of reasons why you would not want to rely on that paradigm when you're going to the real world. Imagine you have a robot, and the robot takes some action in the world and breaks something. You can't say, “Just kidding. I didn't mean to do that. I'm going to undo that action.” The thing is broken.

If you want to simulate what would happen if I moved the robot in this way, and then in the simulation you saw that this thing broke and decided not to take that action, that's totally fine. But you can't just undo actions that you've taken in the world.

Speaker 1

There are a couple more things I wanted to cover in this broad area. I actually had a question on the Thinking, Fast and Slow side, which I'm curious what you think about. A lot of people are trying to put in model-router layers between the fast-response model and the long-thinking model. Anthropic is explicitly doing that, and I think there's a question about whether you always need a smart judge to route, or whether you need a dumb judge to route because it's fast.

So when you have a model router—let's say you're passing requests between the System 1 side and the System 2 side—does the router need to be as smart as the smart model, or should it be dumb to be fast?

Noam Brown

I think it's possible for a dumb model to recognize that a problem is really hard and that it won't be able to solve it, and then route it to a more capable model. But it's also possible for a dumb model to be fooled or to be overconfident. I don't know. I think there's a real trade-off there.

I will say that I think there are a lot of things that people are building right now that will eventually be washed away by scale. I think harnesses are a good example. Eventually, the models are going to become more capable, and I think this actually happened with the reasoning models.

Before the reasoning models emerged, there was all this work that went into engineering agentic systems that made a lot of calls to GPT-4o or these non-reasoning models to get reasoning behavior. Then it turns out, “Oh, we just created reasoning models, and you don't need this complex behavior.” In fact, in many ways, it makes things worse. You just give the reasoning model the same question without any sort of scaffolding, and it just does it.

You can still do that, and people are building scaffolding on top of the reasoning models right now. But I think, in many ways, those scaffolds will also just be replaced by the reasoning models and models in general becoming more capable.

Similarly, I think things like these routers will eventually go away. We've said pretty openly that we want to move to a world where there is a single, unifying model. In that world, you shouldn't need a router on top of the model. I think the router issue will eventually be solved as well. You're building the router into the model weights themselves.

I don't think there will be a benefit to having them. I shouldn't say that, because I could be wrong about this. There may certainly be reasons to route to different model providers or whatever, but I think routers are eventually going to go away.

I can understand why it's worth doing in the short term, because the fact is that it is beneficial right now. If you're building a product and you're getting a lift from it, then it's worth doing right now.

Speaker 1

One of the tricky things that I'd imagine a lot of developers are facing is that you kind of have to plan for where these models are going to be in 6 months and 12 months. It's very hard to do because things are progressing very quickly. You don't want to spend 6 months building something and then just have it be totally washed away by scale.

I think I would encourage developers, when they're building these kinds of things—scaffolds and routers—to keep in mind that the field is evolving very rapidly. Things are going to change in 3 months, let alone 6 months, and that might require radically changing these things around or tossing them out completely. So don't spend 6 months building something that might get tossed out in 6 months.

It's so hard, though. Everyone says this, and then no one has concrete suggestions on how to do it. What about reinforcement fine-tuning? Obviously, you just released it a month ago at OpenAI. Is this something people should spend time on right now, or should they maybe wait until the next jump of this?

Noam Brown

I think reinforcement fine-tuning is pretty cool, and I think it's worth looking into because it's really about specializing the models for the data that you have. A lot of times, we're not suddenly going to have that data baked into the raw model. So I think that's kind of a separate question.

Speaker 1

Yeah. So creating the environment and the reward model is the best thing people can do right now. I think the question that people have is: should I rush to fine-tune the model using RFT, or should I build the harness to then RFT the models as they get better?

Noam Brown

I think the difference is that, for reinforcement fine-tuning, you're collecting data that's going to be useful as the models improve as well. So if we come out with future models that are even more capable, you could still fine-tune them on your data. That's actually a good example where you're building something that's going to complement the scaling of the models and their becoming more capable, rather than necessarily getting washed away by the scale.

Speaker 1

One last question on Ilya. You mentioned on, I think, the Sara and Ilya podcast, where you had this conversation with Ilya a few years ago, about more RL and reasoning in language models. Any speculation or thoughts on why his attempt, when he tried it, didn't work, or why the timing wasn't right, and why the time is right now?

Noam Brown

I don't think I would frame it that way—that his attempt didn't work. In many ways, it did. For me, I saw that in all of these domains that I'd worked on—poker, Hanabi, and Diplomacy—having the models think before acting made a huge difference in performance. Orders-of-magnitude difference.

Speaker 1

Like 10,000 times, isn't it?

Noam Brown

Yeah. 1,000 to 100,000 times is the equivalent of a model that's 1,000 to 100,000 times bigger. In language models, you weren't really seeing that. The models would just respond instantly.

Some people in the LM field were convinced that if we just kept scaling pre-training, we were going to get to superintelligence. I was kind of skeptical of that perspective. In late 2021, I was having a meal with Ilya. He asked me what my AGI timelines were—a very standard SF question.

I told him, “Look, I think it's actually quite far away because we're going to need to figure out this reasoning paradigm in a very general way.” With things like LMs, LMs are very general, but they don't have a reasoning paradigm that's very general. Until they do, they're going to be limited in what they can do.

We're going to scale up, sure. We're going to scale these things up by a few more orders of magnitude, and they're going to become more capable, but we're not going to see superintelligence from just that. Yes, if we had a quadrillion dollars to train these models, then maybe we would, but you're going to hit the limits of what's economically feasible before you get to superintelligence unless you have a reasoning paradigm.

I was convinced, incorrectly, that the reasoning paradigm would take a long time to figure out because it was this big unanswered research question. Ilya agreed with me, and he said, “Yeah, we need this additional paradigm.” But his take was that maybe it's not that hard.

I didn't know it at the time, but he and others at OpenAI had also been thinking about this. They'd also been thinking about RL. They'd been working on it, and I think they had some success. But, as with most research, you have to iterate on things. You have to try out different ideas and try different things. As the models become more capable and faster, it becomes easier to iterate on experiments.

I think that the work that they did, even though it didn't result in a reasoning paradigm, all builds on top of previous work, right? They built a lot of things that, over time, led to this reasoning paradigm.

Speaker 1

For listeners, Noam can talk about this, but the rumor is that that thing was code-named GPT-0, if you want to search for that line of work.

Speaker 1

I think there was a time when RL went through a dark age, when everyone went all in on it, then nothing happened and they gave up. Now it's sort of the golden age again.

So that's what I'm trying to identify: why? What is it? It could just be that we have smarter base models and better data.

Noam Brown

I don't think it's just that we have smarter base models. I think it's that we did end up getting a big success with reasoning. But I think it was, in many ways, a gradual thing. To some extent, it was gradual. There were signs of life, and then we iterated and tried out some more things. We got better signs of life.

I think it was around November 2023 or October 2023 when I was convinced that we had very conclusive signs of life: “Oh, this is the paradigm, and it's going to be a big deal.” That was, in many ways, a gradual thing.

I think what OpenAI did well is that when we got those signs of life, they recognized them for what they were and invested heavily in scaling them up. And I think that's ultimately what led to reasoning models arriving when they did.

Speaker 1

Was there any disagreement internally, especially because OpenAI kind of pioneered pre-training scaling—you know, compute is all you need—and then you're saying maybe that's not how we get there? Was it clear to everybody that this was going to work, or was it controversial?

Noam Brown

There are always different opinions about this stuff. I think there were some people who felt that pre-training was all we needed, and if we scaled it up to infinity, we'd be there. I think a lot of the leadership at OpenAI actually recognized that another paradigm was needed, and that was why they were investing all of this research effort into this RL stuff.

I think that's also to the credit of OpenAI. They figured out the pre-training paradigm and were very focused on scaling it up. In fact, the vast majority of resources were focused on scaling it up, but they also recognized that something else was going to be needed, and that it was worth putting research effort into other directions to figure out what that extra paradigm was going to be.

There was a lot of debate, first of all, about what that extra paradigm was. I think a lot of the researchers looked at reasoning, and RL was not really about scaling test-time compute. It was more about data efficiency, because the feeling was that we had tons and tons of compute, but we were actually more limited by data. There was a data wall, and we were going to hit that before we hit limits on compute. So how do we make these algorithms more data-efficient?

They are more data-efficient, but I think they're also just the equivalent of scaling up compute by a ton. That was interesting. There was a lot of debate around, “Okay, what exactly are we doing here?”

Even when we got the signs of life, I think there was a lot of debate about their significance. How much should we invest in scaling up this paradigm? Especially when you're in a small company like OpenAI—in 2023, it wasn't as big as it is today—and compute was more constrained than it is today. If you're investing resources in one direction, that's coming at the expense of something else.

If you look at these signs of life on reasoning and say, “Okay, this looks promising. We're going to scale this up by a ton and invest a lot more resources into it,” where are those resources coming from? You have to make that tough call about where to draw the resources from, and that is a very controversial, very difficult call to make. It makes some people unhappy.

I think there was debate about whether we were focusing too much on this paradigm, whether it was really a big deal, and whether we would see it generalize and do various things. I remember talking to somebody who left OpenAI after we had discovered the reasoning paradigm, but before we announced o1. They ended up going to a competing lab.

I saw them afterward, after we announced o1, and they told me that at the time, they really didn't think this reasoning thing—the o-series, the Strawberry models—was that big of a deal. They thought we were making a bigger deal of it than it really deserved.

Speaker 1

Oh my God.

Noam Brown

When we announced o1 and they saw the reaction of their coworkers at this competing lab—how everybody was like, “Oh, crap, this is a big deal,” and they pivoted their whole research agenda to focus on it—they realized, “Oh, actually, this maybe is a big deal.”

A lot of this seems obvious in retrospect, but at the time, it's actually not so obvious. It can be quite difficult to recognize something for what it is.

Speaker 1

OpenAI has a great history of just making the right bet. I feel like GPT models are kind of similar, right? It started with games and RL, and then it was, “Maybe we can just scale these language models instead.” I'm just impressed by the leadership and, obviously, the research team that keeps coming up with these insights.

Noam Brown

Looking back on it today, it might seem obvious that these models get better with scale, so you should just scale them up a ton and they'll get better. But the best research really is obvious in retrospect, and at the time, it's not as obvious as it might seem today.

Speaker 1

Follow-up questions on data efficiency. This is a pet topic of mine. It seems that our current methods of learning are still so inefficient, right? Compared to the existence proof of humans, we take 5 samples and learn something. Machines might need 200, maybe, per data point. Is anyone doing anything interesting in data efficiency, or do you think there's just a fundamental inefficiency that machine learning will always have compared to humans?

Noam Brown

I think it's a good point that if you look at the amount of data these models were trained on and compare it to the amount of data that a human observes to get the same performance, these models are less data-efficient than humans. I guess pre-training makes it a little hard to make an apples-to-apples comparison, because I don't know how many tokens a baby actually absorbs while they're developing.

But I think it's a fair statement to say that these models are less data-efficient than humans, and I think that's an unsolved research question and probably one of the most important unsolved research questions—maybe more important than algorithmic improvements. You can just increase the supply of data out of the existing set of the world and humans.

Speaker 1

I guess this is great. So, a couple of thoughts on that. One is that the answer might be an algorithmic improvement. Maybe algorithmic improvements do lead to greater data efficiency.

The second thing is that humans don't learn just from reading the internet. I think it's certainly easiest to learn from data that's on the internet, but I don't think that's the limit of what data you could collect.

The last follow-up before we change topics to coding: any other anecdotes or insights from Ilya, in general? You've worked with him, and there aren't that many people we can talk to who have worked with him.

Noam Brown

I've just been very, very impressed with his vision, especially when I joined and saw the internal documents at OpenAI about what he had been thinking about back in 2021, 2022, and even earlier. I was very impressed that he had a clear vision of where this was all going and what was needed.

Some of his emails from 2016–17, when they were founding OpenAI, were published, and even then he was talking about how one big experiment is much more valuable than 100 small ones. That was a core insight that differentiated them from Google Brain, for example.

It just seems very insightful. He sees things much more clearly than others, and I wonder what his production function is like. How do you make a human like that, and how do you improve your own thinking to better model it?

Speaker 1

I mean, I think it is true that one of OpenAI's big successes was betting on the scaling paradigm. It's just kind of odd because they were not the biggest lab, and it was difficult for them to scale.

Back then, it was much more common to do a lot of small experiments in a more academic style. People were trying to figure out various algorithmic improvements, and OpenAI bet pretty early on large-scale approaches.

We had David Luan on, who I think was VP of Engineering at the time of GPT-1 and GPT-2, and he talked about how the differences between Google Brain and OpenAI were basically the cause of Google's inability to come up with a scaled model. Structurally, everyone had allocated compute, and you had to pool resources together to make bets, and you just couldn't.

Noam Brown

I think that's true, that OpenAI was structured differently, and I think that really helped them. OpenAI functions a lot like a startup, while other places tended to function more like universities or research labs as they traditionally existed.

The way OpenAI operates more like a startup, with this mission of building AGI and superintelligence, helped them organize, collaborate, pool resources together, and make hard choices about how to allocate resources. I think a lot of the other labs have now been trying to adopt paradigms and setups more like that.

Speaker 1

Let's talk about maybe the killer use case, at least in my mind, of these models, which is coding. You released Codex recently, but I would love to talk through the Noam Brown coding stack. What models do you use, and how do you interact with them? Cursor? Windsurf?

Noam Brown

Lately, I've been using Windsurf and Codex—actually, a lot of Codex. I've been having a lot of fun. You just give it a task, and it goes off and does it and comes back 5 minutes later with a pull request.

Speaker 1

Is it a core research task, or is it side stuff that you don't really care about?

Noam Brown

I wouldn't say it's side stuff. Basically, anything that I would normally try to code up, I try to do it with Codex first.

Speaker 1

For you, it's free?

Noam Brown

Yeah, for everybody it's free right now. I think it's partly because it's the most effective way for me to do it, and also because it's good for me to get experience working with this technology and seeing its shortcomings. It helps me better understand the limits of these models and what we need to push on next.

Speaker 1

Have you felt the AGI?

Noam Brown

I’ve felt the AGI multiple times, yes.

Speaker 1

How should people push Codex in ways that you’ve done? I think you just see it before others because, obviously, you were closer to it.

Noam Brown

I think anybody can use Codex and feel the AGI. It’s kind of funny how you feel the AGI and then you get used to it very quickly. You get dissatisfied with what it’s lacking.

Speaker 1

Yeah, I know. It’s magical one day. I was actually looking back at the old Sora videos when they were announced. Remember when Sora came out? It was just magical. You look at that and you’re like, “It’s really here. This is AGI.”

But if you look at it now, it’s kind of like, “Oh, the people don’t move very organically,” and there’s a lack of consistency in some ways. You see all these flaws in it now that you just didn’t really notice when it first came out. And, yeah, you get used to this technology very quickly.

I think what’s cool about it is that, because it’s developing so quickly, you get those “feel the AGI” moments every few months. Something else is going to come out, and it’s just magical to you, and then you get used to it very quickly. What are your Windsurf pro tips now that you’ve immersed yourself in it?

Noam Brown

I think one thing I’m surprised by is how few people use reasoning models. Maybe your audience is going to be more comfortable with reasoning models and use them more, but I’m surprised at how many people don’t even know that o3 exists. I’ve been using it day-to-day. It’s basically replaced Google Search for me; I just use it all the time. Also, for things like coding, I tend to just use the reasoning models.

My suggestion is, if people have not tried the reasoning models yet, it’s worth trying them. People love them; people who use them love them. Obviously, a lot more people use GPT-4o, just the default on ChatGPT and that kind of thing. I think people would be surprised at what the reasoning models can do.

I use Windsurf daily, and they still haven’t actually enabled it as a default in Windsurf. I always have to dig it up, type in o3, and then it’s like, “Oh, yeah, that exists.” It’s weird.

Speaker 1

I would say my struggle with it has been that it takes a little long to reason and actually breaks you out of your flow.

Noam Brown

I think that is true, yes. I think that’s one of the advantages of Codex: you can give it a task that’s self-contained, and it can go off and do its thing and come back 10 minutes later. If you’re using this thing more like a pair programmer, then, yeah, you want to use GPT-4.1 or something like that.

Speaker 1

What do you think are the most broken parts of the development cycle with AI? In my mind, it’s pull request review. I use Codex all the time, and then I get all these pull requests, and it’s kind of hard to go through all of them. What other thing would you like people to build to make this even more scalable?

Noam Brown

I think it’s really on us to build a lot more stuff. These models are very limited in some ways. I find it frustrating that you ask them to do something, and then they spend 10 minutes doing it, and then you ask them to do something pretty similar, and they spend 10 minutes doing it again.

I describe them as geniuses, but it’s their first day on the job. Even the smartest person on Earth, when it’s their first day on the job, isn’t going to be as useful as you would like them to be. So I think being able to get more experience and act like somebody who’s actually been on the job for 6 months instead of 1 day would make them a lot more useful. But that’s really on us to build that capability.

Speaker 1

Do you think a lot of it is GPU-constrained for you? If I think about Codex, why is it asking me to set up the environment myself when the model—if I ask o3 to create an environment setup script for a repo—I’m sure it’ll be able to do it, but today in the product I have to do it?

So I’m wondering, in your mind, could these be a lot more capable if we just put more test-time compute on them, or do you think there’s a fundamental model capability limitation today that means we still need a lot of human harnesses around it?

Noam Brown

I think we’re in an awkward state right now where progress is very fast, and there are things that we could clearly do and the models would be better. We’re going to get to it. You’re just limited by how many hours there are in the day, so progress can only proceed so quickly. We’re trying to get to everything as fast as we can, and I think o3 is not where the technology will be in 6 months.

Speaker 1

I like that question overall. There’s a software development life cycle—not just generation of the code. From issue to PR, basically, is the typical continuum of that. Then there’s the Windsurf side, which is inside your IDE. What else, right?

Pull request review is something that people don’t really think about, although there are startups built around it. It’s not something Codex does, and it could. So what else is there that is sort of rate-limiting the amount of software you could be iterating on? It’s an open question. I don’t know if there’s an answer.

Anything else on the SWE side in general? Where do you think this goes, just in terms of form factors? What will we be looking at this time next year in terms of what models are able to do that they’re not able to do today?

Noam Brown

I don’t think it’s going to be limited to SWE. I don’t think it’s going to be limited to software engineering. I think it’s going to be able to do a lot of remote-work kind of tasks.

Speaker 1

Yeah, freelancer-type Upwork work.

Noam Brown

Or even things that are not necessarily software engineering. The way I think about it is, anybody who’s doing a remote-work kind of job should become familiar with the technology and get a sense of what it can do, what it can’t do, what it’s good at, and what it’s not good at, because I think the breadth of things it’s going to be able to do is going to expand over time as well.

Speaker 1

I feel like virtual assistants might be the next thing after SWE. You know, a virtual assistant—like, hire someone in the Philippines who would just look through your email and all that. You can intercept all the inputs and outputs and train on that. Maybe OpenAI just buys a virtual assistant company.

Noam Brown

I think what I’m looking forward to is that, for things like virtual assistants, the models, if they’re aligned well, could end up being really preferable for that kind of work. There’s always this principal-agent problem: if you delegate a task to somebody, are they really aligned with doing it as you would want it to be done, and doing it as cheaply and as quickly as they can?

Speaker 1

Yeah, yeah, yeah.

Noam Brown

And so, if you have an AI model that’s actually really aligned to you and your preferences, then that can end up doing a way better job than a human could. Well, not that it’s doing a better job than a human could, but it’s doing a better job than a human would.

Speaker 1

That word “alignment,” by the way—I think there’s an interesting overarching homomorphism between safety alignment and instruction-following alignment. I wonder where they diverge.

Noam Brown

Okay, so I think where it diverges is: What do you want to align the models to? That’s a difficult question. You could say you wanted to align it to the user. Okay, well, what happens if the user wants to build a novel virus that’s going to wipe out half of humanity? You need safety alignment.

I think they’re related. The big question is: What are you aligning toward?

Speaker 1

Yeah, there are humanity’s goals, and then there are your personal goals, and everything in between. So that’s kind of, I guess, the individual agent.

You announced that you’re releasing the multi-agent team at OpenAI. I haven’t really seen many announcements—maybe I missed them—about what you’ve been working on, but what can you share about interesting research directions or anything from this?

Noam Brown

There haven’t really been announcements on this. We’re working on cool stuff, and I think we’ll get to announce some cool stuff at some point. I think the team, in many ways, is actually a misnomer because we’re working on more than just multi-agent. Multi-agent is one of the things we’re working on.

Some other things we’re working on are being able to scale up test-time compute by a ton. We get these models thinking for 15 minutes now; how do we get them to think for hours? How do we get them to think for days, or even longer, and be able to solve incredibly difficult problems? That’s one direction that we’re pursuing.

Multi-agent is another direction, and here I think there are a few different motivations. We’re interested in both the collaborative and the competitive aspects of multi-agent. I think the way I describe it is, people often say in AI circles that humans occupy this very narrow band of intelligence, and AIs are just going to quickly catch up and then surpass this band of intelligence. I actually don’t think that the band of human intelligence is that narrow.

I think it's actually quite broad because if you compare anatomically identical humans from caveman times, they didn't get that far in terms of what we would consider intelligence today, right? They're not putting a man on the moon, they're not building semiconductors or nuclear reactors, or anything like that. And then we have those today, even though we as humans are not anatomically different. So what's the difference? Well, I think the difference is that you have thousands of years, a lot of humans—billions of humans—cooperating and competing with each other, building up civilization over time.

The technology that we're seeing is the product of this civilization. And I think, similarly, the AIs that we have today are kind of like the cavemen of AI. I think that if you're able to have them cooperate and compete with billions of AIs over a long period of time and build up a civilization, essentially, the things that they would be able to produce and answer would be far beyond what is possible today with the AIs that we have today.

Speaker 1

Do you see that being similar to maybe Jim Fan's Voyager skill library idea of reusing these things, or is it just the models being retrained on this new knowledge? Because humans then have a lot of it in the brain as they grow.

Noam Brown

I think I'm going to be evasive here and say that until we have something to announce—which I think we will in the not-too-distant future—I'm going to be a bit vague about exactly what we're doing. But I will say that the way we're approaching multi-agent, in the details and the way we're actually going about it, is very different from how it's been done historically and how it's being done today by other places. I've been in the multi-agent field for a long time. I've felt that the multi-agent field has been a bit misguided in some ways, both in the approaches that the field has taken and in the way it's been approached. So I think we're trying to take a very principled approach to multi-agent.

Speaker 1

Sorry, I have to ask: you can't talk about what you're doing, but you can say what's misguided. What's misguided?

Noam Brown

I think that a lot of the approaches that have been taken have been very heuristic and haven't really been following the Bitter Lesson approach to scaling and research.

Speaker 1

Okay. I think maybe this might be a good spot. Obviously, you've done a lot of amazing work in poker, and I think as the reasoning models got better, I was talking to one of my friends who used to be a hardcore poker grinder. I told them I was going to interview you, and their question was: at the table, you can get a lot of information from a small sample size about how a person plays, but today GTO is so prevalent that sometimes people forget that you can play exploitatively. What do you think is the state, as you think about multi-agent and competition? Is it always going to be trying to find the optimal thing, or is a lot of it trying to think more in the moment, like how to exploit somebody?

Noam Brown

I'm guessing your audience is probably not super familiar with poker terminology, so I'll just explain this a bit. A lot of people think that poker is just a luck game, and that's not true. There's actually a lot of strategy in poker, so you can win consistently in poker if you're playing the right strategy.

There are different approaches to poker. One is game theory optimal. This is where you're playing an unbeatable strategy in expectation: you're unexploitable. It's kind of like in rock, paper, scissors: you can be unbeatable if you randomly choose between rock, paper, and scissors with equal probability, because no matter what the other guy does, they're not going to be able to exploit you, so you're not going to lose in expectation.

Now, a lot of people hear that and think, “Well, that also means that you're not going to win in expectation because you're just playing totally randomly.” But in poker, if you play the equilibrium strategy, it's actually really difficult for the opponents to figure out how to beat you, and they're going to end up making mistakes that will lead you to win over the long run. It might not be a massive win, but it is going to be a win. If you play enough hands for a long enough period of time, you're going to win in expectation.

Now, there's also exploitative poker, and the idea here is that you're trying to spot weaknesses in how the opponent plays. Maybe they're not bluffing enough, or maybe they fold too easily to a bluff. And so you start adapting from the game-theory-optimal, balanced strategy—where you bluff sometimes and don't bluff sometimes—to playing a very unbalanced strategy: “I'm just going to bluff a ton against this person because they always fold whenever I bluff.”

The key is that there's a trade-off here, because if you're taking this exploitative approach, then you're opening yourself up to exploitation as well. And so you have to choose this balance between playing a defensive, game-theory-optimal policy that guarantees you're not going to lose but might not make you as much money as you potentially could, versus playing an exploitative strategy that could be much more profitable but also creates weaknesses that the opponents could take advantage of and trick you.

There's no way to perfectly balance the two. It's kind of like in rock, paper, scissors: you might notice that somebody has thrown scissors 6 times, but the next time is when they throw rock, you know? You never really know, so you always have this trade-off.

The poker AIs that have been extremely successful—and my background is that I worked on AI for poker for several years during grad school, where I made the first superhuman no-limit poker AIs—used this game-theory-optimal approach. The AIs would play this unbeatable strategy, and they would play against the world's best and beat them.

That also means they beat the world's worst. They would just beat anybody. But if they were up against a weak opponent, they might not beat them as severely as a human expert might, because the human expert would know how to adapt from the game-theory-optimal policy to exploit these weak players.

And so there's this unanswered question of how do you make an exploitative poker AI? A lot of people have pursued this research direction. I dabbled in it a little during grad school, and I think fundamentally it just comes down to AIs not being as sample-efficient as humans, as we discussed earlier.

If a human's playing poker, they're able to get a really good sense of the strengths and weaknesses of a player within a dozen hands. It's honestly really impressive. Back when we were working on AI for poker in the mid-2010s, these AIs would have to play 10,000 hands of poker to get a good profile of who this player is, how they're playing, and where their weaknesses are. Now, I think with more recent technology, that number has come down, but sample efficiency is still a big challenge.

What's interesting is that after working on poker, I worked on Diplomacy. I think we talked about this earlier. Diplomacy is a 7-player negotiation game, and when we started working on it, I took a very game-theoretic approach to the problem. I felt like it was kind of like poker: you have to compute this game-theory-optimal policy and just play it. You're not going to lose in expectation; you're going to win in practice.

But that actually doesn't work in Diplomacy. Again, it's a question of how much of a rabbit hole we want to go down on this, but basically, when you're playing zero-sum games like poker, game-theory-optimal works really well. When you're playing a game like Diplomacy, where you need to collaborate and compete and there's room for collaboration, game-theory-optimal actually doesn't work that well. You have to understand the players and adapt to them much better.

This ends up being very similar to the problem in poker of how you adapt to your opponents. In poker, it's about adapting to their weaknesses and taking advantage of that. In Diplomacy, it's about adapting to their play styles. It's kind of like if you're at a table and everybody's speaking French, you don't want to just keep talking in English. You want to adapt to them and speak French as well.

That's the realization that I had with Diplomacy: we need to shift away from this game-theory-optimal paradigm toward modeling the other players, understanding who they are, and then responding accordingly. And so, in many ways, the techniques that we developed in Diplomacy are not exploitative; they're really just adapting to the opponents, to the other players at the table.

I think the same set of techniques could be used in AI for poker to make exploitative poker AIs. If I hadn't gotten AGI-pilled by the incredible progress we were seeing with language models and shifted my whole research agenda to focusing on general reasoning, probably what I would have worked on next would have been making these exploitative poker AIs. It would be a really fun research direction to go down. I think it's still there for anybody that wants to do it. And I think the key would be taking the techniques that we used in Diplomacy and applying them to things like poker.

Speaker 1

I think, to me, that core piece is when you play online, you have a HUD, which tells you all these stats about the other player—how much they participate pre-flop, blah, blah, blah.

And to me, a lot of these models, from my understanding, are not really leveraging the behavior of the other players at the table. They're just looking at the board state and working from there.

Noam Brown

That's correct. The way poker AIs work today, they're sticking to their precomputed GTO strategy, and they're not adapting to the other players at the table. You can do various hacky things to get them to adapt, but they're not very principled, and they don't work super well.

Speaker 1

Yep. Okay, any grad students listening, if you want to work on that, I think that's a very reasonable research direction that would at least get you in front of people and get some attention, at least.

The other thing that this conversation brings up for me is that one hypothesis for the next step after test-time compute is world models. Is world modeling an important or worthwhile research direction? Yann LeCun has been talking about this nonstop. Basically, no LLMs have explicitly represented world models; they have internal world models, but not an explicit world model.

Noam Brown

I think it's pretty clear that as these models get bigger, they have a world model, and that world model becomes better with scale. They're implicitly developing a world model, and I don't think it's something that you need to explicitly model. I could be wrong about that. When dealing with people or multi-agent systems, it might be because you have entities that are not the world, and you're resolving hypotheses about which of the many types of entities you could be dealing with.

There was this long debate in the multi-agent AI community for a long time—and it's still going on—about whether you need to explicitly model other agents, like other people, or if they can be implicitly modeled as part of the environment. For a long time, I took the perspective of, of course, you have to explicitly model these other agents because they're behaving differently from the environment. They take actions, they're unpredictable, and they have agency.

But I think I've actually shifted over time to thinking that if these models become smart enough, they develop things like theory of mind. They develop an understanding that there are other agents that can take actions and have motives and all this stuff, and these models just develop that implicitly with scale and more capable behavior broadly. That's the perspective I take these days.

Speaker 1

Cool. What I just said was an example of a heuristic that is not Bitter Lesson-filled, and it just goes away.

Noam Brown

Yeah, it really all comes back to the Bitter Lesson.

Speaker 1

You've got to cite them in every AI podcast. One of the interesting and most consistent findings is that it's always better for AIs to self-play and improve competitively, as opposed to humans training and guiding them. You find that with AlphaZero and R0, whatever that was. Do you think this will hold for multi-agent systems—self-play to improve better than humans?

Noam Brown

Yeah, this is a great question, and I think it's worth expanding on. A lot of people today see self-play as the next step, and maybe the last step, that we need for superintelligence. If you're following something like AlphaGo and AlphaZero, we seem to be following a very similar trend.

The 1st step in AlphaGo was large-scale pretraining. In that case, it was on human Go games; with LLMs, it's pretraining on tons of internet data. That gets you a strong model, but it doesn't get you an extremely strong model; it doesn't get you a superhuman model. The next step in the AlphaGo paradigm is large-scale test-time compute, or large-scale inference compute. In that case, it was MCTS, and now we have reasoning models that also do this large-scale inference compute. Again, that boosts the capabilities a ton.

Finally, with AlphaGo and AlphaZero, you have self-play, where the model plays against itself, learns from those games, and gets better and better and better. It goes from something that's around human performance to way beyond human capability. These Go policies are now so strong that what they're doing is incomprehensible to humans. The same thing is true with chess.

We don't have that right now with language models. It's really tempting to look at that and say, “We just need these AI models to interact with each other and learn from each other, and then they're going to get to superintelligence.” The challenge—and I mentioned this a little bit when I was talking about Diplomacy—is that Go is a 2-player zero-sum game.

2-player zero-sum games include chess, Go, and even 2-player poker. What you typically want is what's called a minimax equilibrium. This is that GTO policy, the policy that you play where you're guaranteeing that you're not going to lose to any opponent in expectation. In chess and Go, that's pretty clearly what you want.

Interestingly, when you look at poker, it's not as obvious. In a 2-player zero-sum version of poker, you could play the GTO minimax policy, and that guarantees that you won't lose to any opponent on Earth. But as I mentioned, you're not going to beat a weak player; you're not going to make as much money off of them as you could if you played an exploitative policy instead. So there's this question of what you want: do you want to make as much money as possible, or do you want to guarantee that you're not going to lose to any human alive?

What all the AI developers in these games have decided is, “We're going to choose the minimax policy.” Conveniently, that's exactly what self-play converges to. If you have these AIs play against each other and learn from their mistakes, they converge over time to this minimax policy, guaranteed.

But once you go outside 2-player zero-sum games, as in the case of Diplomacy, that's actually not a useful policy anymore. You don't want to have this very defensive policy, and you're going to end up with really weird behavior if you start doing the same kind of self-play in things like math.

For example, what does it mean to do self-play in math? You could fall into this trap of saying, “I just want 1 model to pose really difficult questions and the other model to solve those questions.” That's a 2-player zero-sum game. The problem is that you could just pose really difficult questions that aren't interesting. You could ask it to do 30-digit multiplication. It's a very difficult problem for AI models, but is that really making progress in the dimension that we want? Not really.

Self-play outside of these 2-player zero-sum games becomes a much more difficult, nuanced question. Tim said something similar in his talk: there are a lot of challenges in deciding what you're optimizing for when you start talking about self-play outside of 2-player zero-sum games. My point is that this is where the AlphaGo analogy breaks down—not necessarily completely, but it's not going to be as easy as self-play was in AlphaGo.

Speaker 1

What is the objective function then? What is the new objective function?

Noam Brown

It's a good question, and I think that's something that a lot of people are thinking about.

Speaker 1

Yeah. I'm sure you are. In 1 of the last podcasts that you did, you mentioned that you were very impressed by Sora. You don't work directly on Sora, but obviously it's part of OpenAI. I think the most recent new update in that generative-media space is autoregressive image generation. Is that interesting or surprising in any way that you want to comment on?

Noam Brown

I don't work on image generation, so my ability to comment on this is limited, but I will say that I love it. I think it's super impressive. It's one of those things where you work on these reasoning models, and you think, “Wow, we're going to be able to do all sorts of crazy stuff, like advance science, solve agentic tasks, and do software engineering.”

Then there's this other dimension of progress where you're able to make images and videos, and it's so much fun. That gets a lot more attention, to be honest, especially in the general public, and it's probably driving a lot more of the subscription plans for ChatGPT, which is great. I think it's just kind of funny that, yeah, we're also—I promise we're also working on superintelligence—but you can make everything googly.

Speaker 1

I think the delta for me was that I was actually harboring this thesis that diffusion was over because of autoregressive image generation. There were rumors about this at the end of last year, and obviously now it's come out.

Then Gemini comes out with text diffusion, and diffusion is so back. This is two directions, and it's very relevant for inference of autoregressive versus diffusion. Do we have both? Does one win? The beauty of research is that you have to pursue different directions, and it's not always going to be clear what the promising path is. I think it's great that people are looking into different directions and trying different things. I think there's a lot of value in that exploration, and we all benefit from seeing what works. Any potential in diffusion reasoning?

Noam Brown

Probably can't answer that.

Speaker 1

Okay. So you did a master's in robotics, too. We'd love to get your thoughts on OpenAI getting started with the pen-spinning trick and the robotic arm they wanted to build. Is it right to work on humanoid legs? Do you think that's kind of the wrong embodiment of AI, outside of the usual, “How long until we get robots?” Is there something that you think is fundamentally not being explored right now that people should really be doing in robotics?

Noam Brown

I did a master's in robotics years ago, and my takeaway from that experience—first of all, I didn't actually work with robots that much. I was technically in a robotics program. I played around with some LEGO robots my first week of the program, but then I pretty quickly shifted to just working on AI for poker and was nominally in the robotics master's.

My takeaway from interacting with all these roboticists and seeing their research was that I did not want to work on robots because the research cycle is so much slower and so much more painful when you're dealing with physical hardware. Software goes so much more quickly, and I think that's why we're seeing so much progress with language models and all these virtual-coworker tasks, but we haven't seen as much progress in robotics. Physical hardware is just much more painful to iterate on.

On the question of humanoids, I don't have very strong opinions here because this isn't what I'm working on, but I think there's a lot of value in non-humanoid robotics as well. I think drones are a perfect example, where there's clearly a lot of value in that. Is that a humanoid? No, but in many ways that's great. You don't want a humanoid for that kind of technology. Weakly, I think that non-humanoids provide a lot of value.

Speaker 1

I was reading Richard Hamming's The Art of Doing Science and Engineering, and he talks about how, when you have a new technological shift, people try to take the old workloads and replicate them just in the new technology, versus actually having to change the way you do it.

When I see this video of your humanoid in the house, it's like, well, the human shape has a lot of limitations that can actually be improved. But I think people want what's familiar. Would you put a robot with 10 arms and 5 legs in your house, or would it be eerie at night when you get up and see that thing walking around? Is that why we use humanoids?

To me, there's almost this local maximum of “we've got to make it look like a human,” but I think the question is, what's the best shape in a house?

Noam Brown

I'm not a product designer, so I'm not the person to ask on this. I think there is a question of whether it's better to make humanoids because they're more familiar to us, or whether it's worse to make humanoids because they're more similar to us but not quite identical. I don't know which one I would actually find creepier.

Speaker 1

Yeah. The thing that got me humanoid-pilled a little bit was just the argument that most of the world is made for humans anyway, so if you want to replace human labor, you have to make a humanoid. I don't know if that's convincing. Again, I don't have very strong opinions in this field because I don't work in it.

Noam Brown

I was weakly in favor of humanoids, and I think what really persuaded me to be weakly in favor of non-humanoids was listening to the Physical Intelligence CEO and some of his pitches about why they're pursuing non-humanoid robotics. Conveniently, their office is actually very close to here.

Speaker 1

They're speaking at the conference I'm running.

Noam Brown

Okay, perfect. I'd say listen to his pitch, and maybe he can convince you that non-humanoid is the way to go.

Speaker 1

Awesome. The other one I would refer people to is a talk Jim Fan recently did on the physical Turing test at the Sequoia conference, which was very, very good. He's such a great educator and explainer of things. It's very hard, especially in that field.

Cool. We're done asking you about things that you don't work on. These are just more rapid-fire questions to explore some of your boundaries and get some quick hits. How do you, or how do top industry labs, keep on top of research? What are your tools and practices?

Noam Brown

It's really hard. I think that a lot of people have this perception that academic research is irrelevant, and this is actually not the case. We do look at academic research. One of the challenges is that a lot of academic research shows promise in the papers, but then actually doesn't work at scale or even doesn't replicate.

If we find interesting papers, we're going to try to reproduce that in-house and see if it still holds up, and then see whether it scales well. But that is a big source of inspiration for us.

Speaker 1

Whatever hits arXiv, literally, you do the same as the rest of us? Or do you have a special process?

Noam Brown

Especially when I get recommendations. We have an internal channel where people will post interesting papers, and I think that's a good source of, “Okay, this person who is more familiar with this area thinks that this paper is interesting, so therefore I should read it.”

Similarly, I'll keep track of things that are happening in my space that I think are interesting, and if I think it's really interesting, maybe I'll share it.

Speaker 1

For me, it's WhatsApp and Signal group chats with researchers, and that's it.

Noam Brown

Yeah. I think a lot of people look at things like Twitter, and I think it's really unfortunate that we've reached this point where things need to get a lot of attention on social media for them to be paid attention to. That's what the grad students are trained to do. They're taking classes to do this.

I do recommend it. I've worked with grad students—I work with fewer now because we don't publish as much—but when I was at FAIR publishing papers, I would tell the grad students I was working with that they needed to post it on Twitter. We'd go over the Twitter thread about how to present the work and everything. There's a real art to it, and it does matter. It's kind of the sad truth.

Speaker 1

I know when you were doing the Annual Computer Poker Competition, you mentioned that people were not doing search because they were limited to 2 CPUs at inference. Do you see similar things today that are keeping interesting research from being done because maybe it's not as popular or doesn't get you into the top conferences? Are there some environmental limiters?

Noam Brown

Absolutely. I think one example is benchmarks that look at things like Humanity's Last Exam. You have these incredibly difficult problems, but they're still very easily gradable. I think that actually limits the scope of what you can evaluate these models on if you stick to that paradigm.

It's very convenient because it's very easy to score the models, but a lot of the things that we want to evaluate these models on are fuzzier tasks that are not multiple-choice questions. Making benchmarks for those kinds of things is much harder and probably also a lot more expensive to evaluate. But I think those are really valuable things to work on.

Speaker 1

That would fit the sentiment that GPT-4.5 is a high-taste model in a way. There are all these non-measurable things about a model that are really good.

Noam Brown

Well, I think there are things that are measurable, but they're just much more difficult to measure. A lot of benchmarks have stuck to this paradigm of posing really difficult problems that are really easy to measure.

Speaker 1

Let's say that the pre-training scaling paradigm took about 5 years, from the discovery of GPT to scaling it up to GPT-4. Then we give test-time compute 5 years as well. If test-time compute hits a wall by 2030, what would be the probable cause?

Noam Brown

It's very similar to pre-training, where you can push pre-training a lot further; it just becomes more expensive with each iteration. I think we're going to see something similar with test-time compute, where we're going to get them thinking for 3 minutes, then 3 hours, then 3 days, and then 3 weeks.

Speaker 1

Or you run out of human life.

Noam Brown

There are 2 concerns. One is that it becomes much more expensive to get the models to think for that long or to scale up test-time compute. As you scale up test-time compute, you're spending more on test-time compute, which means that there's a limit to how much you can spend. That's one potential ceiling.

Well, I should say that we're also becoming more efficient. These models are becoming more efficient in the way they're thinking. They're able to do more with the same amount of test-time compute.

I think that's a very underappreciated point: it's not just that we're getting these models to think for longer. In fact, if you look at o3, it's thinking for longer than o1-preview for some questions, but it's not a radical difference. It's way better. Why? Because it's just becoming better at thinking.

Anyway, these models are going to scale up test-time compute, but you can only scale it up so much. That becomes a soft barrier, in the same way that pre-training is becoming more and more expensive as you train better and better pre-trained models—or bigger pre-trained models.

The second point is that as you have these models think for longer, you get bottlenecked by wall-clock time. If you want to iterate on experiments, it's really easy to iterate on experiments when these models respond instantly. It's much harder when they take 3 hours to respond. What happens when they take 3 weeks? It takes you at least 3 weeks to do those evaluations and then iterate on them.

A lot of this can be parallelized to some extent, but a lot of it requires you to run the experiment to completion and then see the results in order to decide on the next set of experiments. I think this is actually the strongest case for long timelines: because the models have to do so much in serial time, we can only iterate so quickly.

Speaker 1

How would you overcome that wall?

Noam Brown

It's a challenge, and I think it depends on the domain. Drug discovery, I think, is one domain where this could be a real bottleneck. If you want to see if something extends human life, it's going to take you a long time to figure out if this new drug we developed actually extends human life and doesn't have terrible side effects along the way.

Speaker 1

Side note: do we not have perfect models of human chemistry and biology by now? Isn't that the number-one thing that we should all work on?

Noam Brown

This is, I think, the thing. Again, I want to be cautious here because I'm not actually a biologist or chemist. I know very little about these fields. The last time I took a biology class was in 10th grade in high school.

I don't think that there's a perfect simulator of human biology right now, and I think that's something that could potentially help address this problem.

Well, that's one of the things that we're hoping these reasoning models will help us with.

Speaker 1

How would you classify mid-training versus post-training today?

Noam Brown

All these definitions are so fuzzy, so I don't have a great answer there.

Speaker 1

It's a question people have, and OpenAI is now explicitly hiring for mid-training. Everyone is like, “What the hell is mid-training?”

Noam Brown

I think mid-training is between pre-training and post-training. It's not post-training, and it's not pre-training. It's adding more to the models after pre-training, in interesting ways.

Speaker 1

Yeah. Okay. All right. Well, I was trying to get some clarity. Is the pre-trained model now basically just an artifact that then spawns other models? Is it almost like the core pre-training model is never really exposed anymore? Is mid-training the new pre-training, and then there's post-training once you have the models branched out?

You never interact with an actual raw pre-trained model. If you're going to interact with the model, it's going to go through mid-training and post-training, so you're seeing the final product. You don't let us do it, but, you know.

Noam Brown

Yeah. We used to. I mean, I guess there are open-source models where you can interact with the raw pre-trained model. But for OpenAI models, they go through a mid-training step, and then they go through a post-training step, and then they're released. They're a lot more useful.

Frankly, if you interacted with only the pre-trained model, it would be super difficult to work with, and it would seem kind of dumb.

Speaker 1

Yeah, but it would be useful in weird ways, because there's a mode collapse when you post-train it for chat.

Noam Brown

In some ways, you want that mode collapse. You want that collapse of the distribution.

Speaker 1

I get it. We're interviewing Greg Brockman next. You've talked to him a lot. What would you ask him?

Noam Brown

What would I ask Greg? I get to ask Greg all the time.

Speaker 1

What should you ask Greg to evoke an interesting response—something he doesn't get asked enough about, but that he's passionate about, or that you just want his thoughts on?

Noam Brown

I think in general, it's worth asking where this goes. What does the world actually look like in 5 years? What does the world look like in 10 years? What does that distribution of outcomes look like? What could the world or individuals do to help steer things toward the good outcomes instead of the negative outcomes?

Speaker 1

Okay. Like an alignment question?

Noam Brown

I think people get very focused on what's going to happen in 1 or 2 years. I think it's also worth spending some time thinking about, “Well, what happens in 5 or 10 years?” What does that world look like?

Speaker 1

I mean, he doesn't have a crystal ball.

Noam Brown

But he certainly has thoughts. So, I think that's worth exploring.

Speaker 1

Okay. What are games that you recommend to people, especially socially?

Noam Brown

I've been playing a lot of this game called Blood on the Clock Tower lately.

Speaker 1

Hmm. What is it?

Noam Brown

It's kind of like Mafia or Werewolf. It's become very popular in San Francisco.

Speaker 1

Oh, that's the one we played at your house.

Noam Brown

Yeah.

Speaker 1

Okay. Got to get it. It's kind of funny because I was talking to a couple people who told me that it used to be that poker was the way VCs and tech founders and so on would socialize with each other. Actually, now it's shifting more toward Blood on the Clock Tower. That's the thing that people use to connect in the Bay Area. I was actually told that a startup held a recruiting event that was a Blood on the Clock Tower game.

Noam Brown

Wow.

Speaker 1

Yeah. So, I guess it's really catching on. It's a fun game, and I guess you lose less money playing it than you do playing poker, so it's better for people who are not very good at these things. I think it's kind of a weird recruiting event, but certainly a fun game.

What qualities make a winner here that are interesting to hire for?

Noam Brown

That's the thing. I guess you get a good ability to lie, deceive, and pick up on deception. Is that the best employee? I don't know.

Speaker 1

So, my slight final pet topic is Magic: The Gathering.

Noam Brown

Ooh.

Speaker 1

We've talked about some of these games—chess, Go—and they have perfect information. Then you have poker, which has imperfect information in a pretty limited universe. You only have a 52-card deck. And then you have these other games that have imperfect information, with a huge pool of possible options. Do you have any idea how much harder that is? How does the difficulty of this problem scale?

Noam Brown

I love that you asked that because I have this huge store of knowledge on AI for imperfect-information games. This was my area of research for so long, and I know all these things, but I don't get to talk about them very often. We've made superhuman poker AIs for no-limit Texas hold'em.

One of the interesting things about that is that the amount of hidden information is actually pretty limited because you have 2 hidden cards when you're playing Texas hold'em. The number of possible states that you could be in is 1,326 when you're playing heads-up, at least. That's multiplied by the number of other players at the table, but it's still not a massive number.

The way these AI models work is that you enumerate all the different states that you could be in. If you're playing 6-handed poker, there are 5 other players: 5 × 1,326 is the number of states that you can be in. Then you assign a probability to each one. You feed those probabilities into your neural net, and you get actions back for each of those states.

The problem is that as you scale the number of hidden possibilities—the number of possible states you could be in—that approach breaks down. There's still this very interesting, unanswered question of what you do when the number of hidden states becomes extremely large.

Speaker 1

Mm-hmm.

Noam Brown

If you go to Omaha poker, where you have 4 hidden cards, there are heuristic things you could do to reduce the number of states, but it's still a very difficult question. And then if you go to a game like Stratego, where you have 40 pieces, there are close to 40 factorial different states you could be in. All these existing approaches that we used for poker break down, and you need different approaches.

There's a lot of active research going on about how to cope with that. For something like Magic: The Gathering, the techniques that we used in poker would not work out of the box. It's still an interesting research question: what do you do?

Now, I should say that this becomes a problem when you're doing the kinds of search techniques that we used in poker. If you're just doing model-free RL, it's not a problem. My guess is that if somebody put in the effort, they could probably make a superhuman bot for Magic: The Gathering now. There are still some unanswered research questions in that space.

Speaker 1

Now, are they the most important unanswered research questions?

Noam Brown

Right.

I’m inclined to say no. I think the problem is that the techniques that we used in poker to do this kind of search stuff were pretty limited. If you expand those techniques, maybe you can get them to work on things like Stratego and Magic: The Gathering, but they’re still going to be limited. They’re not going to get you superhuman in Codeforces with language models. So I think it’s more valuable to just focus on the very general reasoning techniques. One day, as we improve those, I think we’ll have a model that just out of the box plays Magic: The Gathering at a superhuman level. And I think that’s the more important and more impressive research direction.

Speaker 1

Cool. Amazing. Yeah. Thanks so much for coming on, Noam.

Noam Brown

Yeah. Thanks for your time.

Speaker 1

Yeah. Thanks.

Noam Brown

Thanks for having me.