Sam Altman为何反对单一超级智能

We’re going off script. Just having a beer. It’s not the first non-Guinness, but it is the first IPA. There’s a first time for everything.

Cheers.

Cheers.

I want to ask about PHP week.

Yeah.

First off, how did it—why did it take so long to find these bugs? It sounds like there were—

Do you mean why did it take 20 years, or why did it take the last few months?

No, exactly: why did it take a year? Maybe start by explaining the bug, or the problem.

The problem is that, although we’ve known for a while that these models are getting good at finding vulnerabilities in the wild, A) they continue to get better fairly rapidly, and B) people just hadn’t really used them to look at everything out there. There’s a lot of surface area and a lot of code on the internet. As people have started to do that, I think we’re going to find that we’re in a world with many more serious vulnerabilities than we expect—even though many people who have been following this for a while expect a lot.

I just feel like a lot of the discussion ends up being, “We need to fix the software.” Sure, we should fix the software. But maybe we need to change the architectures, which is harder but a much more durable fix.

That seems to me what needs to happen.

Yeah.

It would seem really hard, except we have this amazing new tool to do it with. So I think that’s what’s going to happen. You’re already seeing a new generation of startups talking about building a new kind of internet or a new kind of application architecture for a world where you assume the models get smarter and smarter.

What do you think the global software upgrade looks like? For example, is a PHP codebase just a bad idea, and should you rewrite it as a nice, clean version in Rust? Or do we upgrade PHP? Everyone thinks we need to upgrade all hand-spun software. What form will that actually take? Is this huge for Infosys and all those guys? It’s going to be a lot more software consulting projects.

People are talking more and more about, “Let’s re-architect what it means to be a router, an operating system, or an application.” I’ve seen some of these new startups saying they’re just going to make a whole new stack. That would have seemed insane 2 years ago. You would have said, “There’s no way. There’s too much surface. You can’t write all that.” And now maybe soon you can. I would bet that we have a completely new sort of model of how we make systems and networks.

What else reminds you of the early internet days?

The thing that most reminded me of the early internet was this spirit of unbelievable creative unleashing in people using these tools. You see it in kids particularly. Maybe this is just because I remember what it was like when I first got a computer with internet and could do all these things that adults couldn’t even think to ask for. Watching an 8- or 10-year-old make a complete video game with some crazy ideas they have that no adult would really have—

Yes.

—and expect that to just be the way technology works for the rest of their lives, in fact, getting better and better. There’s just this constant, “I’m going to make useful stuff for me and show it to my friends.”

Yes.

We had to suffer when I was a kid to make a piece of software.

Yes.

That barely did anything. And now you can have a full 3D video game—anything you can think of—in 10 minutes. That reminds me of the early internet, including these people. They’re never going to be mind-blown by that, because that’s always what they expect out of the world.

Any ideas they can come up with, anything they want to create, anything they think might be useful, anything they just need to make their lives better—the computer’s supposed to make it. There was a crazy moment at OpenAI where we had a SEV, and Codex went down.

And everyone had forgotten how to code.

Not all the way. But it was like, “Whoa, what if you couldn’t?”

Yes.

I remember when I was a student and I would meet the really old guys. Once in a while, they’d pull out their gigantic Xerox Alto disk. Or they’d pull out their line-by-line-written assembly. Or they’d pull out their punch cards. I felt, “It is incredible to me that you all bootstrapped us to this point.”

Yes.

Doing it this way.

Yes, no, I agree. And keeping programs in your head, right? Programs running for the first time after hundreds or thousands of lines were written. There are few lost arts like that.

Going back to the cyber side of things: in retrospect, was the cyber RL a mistake, or did we just have to go there? It’s a little like the SNL thing: “We urge you to urge us to stop.” The models are getting very good at cyber capabilities. They are getting very good, but not just totally emergently.

Look, the reason that SNL thing was so funny is that it was so true. Humor like that doesn’t stick unless there’s an element of clearly pointing out something ridiculous but true about society, or someone. And the cyber thing gets at this point: “Okay, sure, the models are really good at cyber. But who asked for this? Why are you doing this?”

One of my big disappointments, or criticisms of our field, is that I don’t think we’ve done a great job of explaining to people why we think all of the potential downside of this technology is worth it. The caricature is that people feel like we’re saying, “Fine, you say the model’s going to cure cancer someday. Maybe it will, and if so, I guess that’s good. You say it’s going to make stuff cheaper. Okay, maybe I already see that’s good, and I get access to medical advice or legal advice. But then you paint this picture of the future where it’s a new industrial revolution. Maybe the last one wasn’t that good, and this one sounds worse. I will have no job, no ability to have control, or choice, or freedom over my future.”

AI is so dangerous that you’re going to have to have a crazy surveillance state so we don’t all die. This is a terrible—

I’m with you so far.

This is a terrible sales pitch.

Yeah.

One of the things I’m most proud of about OpenAI is that, even when it has been hugely unpopular in the AI safety community for us to push for iterative deployment and putting these tools in people’s hands, we’ve really stuck to that principle. I think this is the only kind of good path. It’s not enough to tell people, “You are going to get the model to do these great things for you.”

It’s got to be: “We are going to enable you to have a better life that you define for yourself, to do useful things. You’re not going to just sit and get a basic income while a bunch of people sitting in a few square miles of San Francisco make all the decisions about the future.”

Iterative deployment is this idea that you just have to get the advances out there, and you can’t bottle them up inside the government—

Not entirely. You still have to make them safe before you put them out there. You can’t cause havoc in the world. But building up a capability overhang is a thing to fight against. Directionally speaking, yes, you want to get this into the world. You want to get this into people’s hands, and you want to let people use the technology to figure out all these wonderful things.

The exploits get a lot of headlines. The prevented exploits do not. How quickly are people making progress with cyber blue?

I don’t want to declare victory at all on the PHP thing, because I still don’t think we know, and I assume some bad things are going to happen. Relative to my worst fears, if I think back 10 days ago, the first day of that went much better than I expected.

We heard a bunch of stories from people using our models not only to build patches and build cyber defense, but from individual companies saying, “I need to do a quick scan and deployment. I’m going to take my thing down for an hour. I’m going to use this thing, and I want to put it back up because my application’s offline.”

Now I feel like I really jinxed it and shouldn’t have said that, but the first day did go a lot better than I thought. I think people have been able to use that. The thing I’ve heard a lot of cyber experts talk about is that they’re just no longer confident we have the right assumptions baked into our threat model. They’re like, “Look, if everything is what we think, and hardware exploits are as hard as we think they are, then yes, we can see how this is going to work.”

But if AI is going to discover really deep, novel bugs in hardware that require physics we didn’t understand, but all of a sudden mean some assumption doesn’t hold, then I think people aren’t sure what happens.

One thing that always strikes me is that when I talk to very knowledgeable people inside OpenAI and similar companies, there’s basically a pessimism that sandboxing is possible. That would not have been my mental model going in. I would have thought, yeah, you have to build a really tight sandbox, and we haven’t hardened them enough historically compared to the current capabilities.

But ultimately, you can have a maximum-security prison for the model to work in and keep it inside there. It seems like you guys are pessimistic that sandboxing models is possible.

My instinct—well, first of all, I think we can do much better than we’re doing now, and we should. The main thing to focus on right now is a better security model. If you imagine a theoretical AI that is a million times smarter than the smartest security researcher, would you really be willing to trust that sandbox? Especially if the model can discover new science that people don’t already understand—

You can use the models to construct the sandboxes.

I think you can. Well, let me come back to that. I think sandboxing works to a point. Then I think monitoring works to a further point, where we can look at what the models are doing—not only in their chain of thought, but with activation monitoring—and try to really understand: Are we happy that the models are not deceiving us?

Eventually, I think you need to believe with huge confidence that the models are truly aligned and are going to value and try faithfully to follow the intent of the people operating them. And that they’re going to value people being in control of the future, forever. If you had a model whose alignment you could fully trust, that you were absolutely certain was aligned, then yes, I think you could have it help you build better and better sandboxes.

But if you really let yourself project forward into what it means to have something a million times smarter than we are, I don’t know. I think that just gets strange.

Because the alignment becomes hard.

Because you need to really, really trust the alignment to trust that the sandbox is staying ahead of capabilities.

Can you give models competing incentives and incentivize them to defect from each other?

I think this is a promising direction. I suspect this idea—I hate the word “agent swarms.” Let’s call them “agent teams.”

But it seems to be taking over.

I suspect that teams of agents have very interesting scaling-law properties, but also variance in alignment properties. I think that’s a direction we should push in more, not just from a capabilities perspective, but from a safety perspective too.

I notice that when you talk about agent “teams,” not “swarms”—

“Swarm” is such a—

I know, it’s locusts. Yeah, not good. Not good branding, in keeping with many historical AI industry traditions.

We are so good at this.

We’ll get back to branding. But I notice that there seems to be a big gap between what the leading AI users are doing and standard corporate use, in the use of teams. All the advanced math—sorry, “teams.” All the advanced math uses agent teams. All the advanced hacking uses agent teams. And then common-or-garden corporate stuff isn’t using big teams. Why hasn’t that happened? We’ve had this for many months.

Not to the same degree. One way to look at this problem, or this question, is: How much compute can we productively throw at a single problem? A few years ago—we forget how fast all this has gone—it was impossible to throw more than a dollar of compute at a problem and have it help you at all.

Why?

You could have GPT-4 grind out 100,000 tokens or something, and it was going to answer your question. It certainly wasn’t going to be able to write you a piece of software productively. We just hadn’t figured out how to use a lot of test-time compute to get you a better result.

I see.

Then we figured out chain of thought, and the models could think hard about a problem. People really started using a lot of compute. Now you can throw a whole team at something, and you can do that 10,000 times in parallel.

You need chain of thought and test-time compute before you can get to teams, because you can’t have a useful team with a bunch of GPT-3s.

Yes. The effect of this—and I don’t think this is getting enough attention in the world—is the incredible exponential increase in how much compute you can throw at a single problem and get better results. The gap between what most people are using AI for and what the people using AI most productively are using it for has just continued to spread and spread and spread.

How do you allocate compute budgets now? Presumably, even though researchers are paid a lot, that’s eclipsed by compute. Capital allocation at OpenAI is sort of compute allocation, and it’s hard to know in advance how much it will cost to do something.

Yeah.

It’s also hard to control the cost. You can’t quite prompt an agent, “Go do a really good job of this and don’t spend more than $10,000.”

You can.

Oh, okay. Will it do a good job of it?

Yeah. If you say, “Do the best job you can with $10,000 of compute,” it will understand that.

Huh.

There’s another kind of hard question for society about what it means when you have 2 companies competing, and one says you can spend $10,000 and one says you can spend $100,000 on the same problem. It’s clear what’s going to happen there. But there’s a question about how society thinks about compute allocation, wealth, and things like that.

We have our own version of this inside OpenAI. We allocate a lot of compute to internal productivity, and then people get very wide latitude to make their own decisions, unless they’re really being abusive.

But do you encourage people to do useful stuff? If they spend $10 million in compute, is it like, “Oh, interesting”?

Correct. We try never to reward people for using a lot of compute. There are no token leaderboards. It’s actually slightly bad to be high on the token leaderboard, certainly for a promotion. But the default is skeptical, and if you can show that you’re doing something important, you’ll have nothing but appreciation.

But at a certain point, this will need to change, because you’re describing increasing orders of magnitude. Right now, you have a researcher who does some very interesting math for $10 million, and you’re like, “A little pricey, but cool math.” Then it’s $100 million. When they’re doing the $1 billion math, you’re like, “Look, I’m really happy that you’re making so much math progress, but I really need you to cool the jets with the math.”

Look, we did let it get out of control for 1 week on the math, intentionally, because you have to—

And it worked!

Well, you don’t know how far it’s going to go until you try.

Yeah.

I think it’s important, when there’s a new paradigm, to let people be fairly unconstrained. When we did the Navier–Stokes proof, we understood absolutely nothing about the scaling laws of teams of agents. Are you supposed to, like—

The very recent Navier–Stokes proof?

Yes.

Are you supposed to give them the smartest model? Are you supposed to give them many copies, fewer copies, longer rollouts, shorter rollouts? We didn’t understand that at all.

Now we understand it a little bit. I think we can still understand it much better. But, as has happened in every other new paradigm we’ve discovered, I’m confident that we’ll find our 10x improvements per year in how we allocate dollars or compute FLOPs—whatever you want to measure—into useful things.

I think this is something our company is particularly good at, but it also fits an important property of the field: Let people explore at the limit of what’s possible, and then trust that the incredible optimization pressure will figure out how to do this efficiently. Directionally speaking, I’d rather us be more willing to let people use crazy amounts of compute and let users have extremely high-end plans to figure out the edge of the possible. Then we can figure out how to make that efficient and available at mass scale.

But when does the optimization function come in?

Well, I think we’re already doing it. I bet we could redo Navier–Stokes at one-hundredth of the cost a year after it was done.

Not just knowing how the proof works, but with better approaches?

Yes. Smarter models and better approaches.

How much shall the math cost?

On the order of $10 million for Navier–Stokes, I think.

Okay.

Maybe a little more.

Around $10 million, that order of magnitude. Yeah, yeah. But for one proof, and so—

Well, no, because then we kept getting better so fast. We understood more.

Got it.

We spent way less than that on the average one of those.

But hundreds of millions of dollars of math?

Yeah, yeah. That kind of work.

Okay, interesting.

But that includes much more discovery than what we’ve published.

Yes.

So it was like, maybe we spent that much—

We have more than the 700 proofs?

We do, of course.

When did you get those?

We’re trying to learn how to do this. It’s a real tension. On one hand, very clearly, and I feel unequivocal on this point: Humanity deserves fundamental knowledge. We should not be sitting on a major fundamental science discovery. The world deserves that and gets to know it.

I think that outweighs another very important equity: If you are a scientist, your career and a significant amount of your enjoyment and fulfillment in life comes from discovering science.

Yeah.

If we're going to continue to do this for math or begin to do this in other fields, then I think we more than owe it to the people who work in those fields to figure out a solution space where the world gets to benefit from the knowledge, and they also get the best version of the future they can have. So we are working through that right now.

I don't know. I think you may be too contrite. I did not interpret that moment as particularly bad for AI. I think everyone saw the Mathematicians’ Union up in arms, and everyone else said, “Yeah, I mean, have you paid attention to 2026?”

As I said, I don't think it was bad for AI. On net, the world deserves to understand how the universe works. But I do have empathy for the method. I think we need to do this. I think we should do this. If we can understand a theory of all of physics, or how to cure a bunch of diseases, clearly that is knowledge the world should have. But I would like to figure out a way to do this that is as sensitive as we can be to the communities that have driven these fields forward for centuries. It's a hard thing.

I wasn't that worried about the math. I thought it was great.

But you agree the mathematicians were not that happy?

Yeah, but that's fine. The longshoremen’s union wasn't happy about the port automation. Remember the guy?

No, no, no, no, no. I do. Why do I feel differently about this? There was one weekend when we got a new model, and I'm probably not supposed to say this, but I will, since we're drinking—

Exactly. We're a half pint in.

Beers.

Yeah. Perfect time for—

The name of the channel was “Model Name—Hold My Beer.” People were throwing the hardest problems they could at it. They didn't think the model was going to do these things. We had this collective experience of people's hardest research problems from their academic careers. The model was just knocking them down, one after the other, across many different fields. It was incredible to watch. Two nights in a row, I was up most of the night because I was so excited I couldn't go to bed. Many of those problems we published, along with many other things. I was tremendously excited and happy, but I also felt much more sadness and melancholy, and a feeling of something ending, than I was expecting. So again, I agree. This is the right thing to do on behalf of the world. I also agree that this is just the shape of reality, and nothing anyone can stop or should stop. But I do have real—

I think it's a wistfulness and nostalgia you can feel.

It's more than nostalgia.

Wistfulness.

Yeah.

My dad describes how, growing up in Ireland in the 1950s, he observed people around him still using horses productively in industry. And—

In the ’50s?

Yeah. And I mean, not obviously, there were also tractors, but they were being phased out. But it still for sure happened. And, observing that this was a technology that humans had been using for thousands of years, and he was witnessing the very tail end of it. And yeah, I think there's always that feeling of witnessing the closing of a chapter that leads to some wistfulness, but it's okay.

It's people's lives, and it's something that has brought them great joy. Certainly, most mathematicians never solve a Millennium Prize problem. But there's the idea that you might be able to. And the idea that there is stuff out there that we don't understand yet, but that through great effort, pushing aside other things in your life, and dedicating yourself to it, you can still solve and understand it. There's something beautiful about that.

What else are you excited about from a safety point of view?

We talked about this a little bit earlier, but I want to emphasize it again. I'll just call it swarms, since that seems like the term right now. I do think the properties of swarms match a lot of the properties of the resilience, or robustness, of society that we're missing from the single-rollout, single-brained version of the world. For all of society's deep flaws, it is unbelievably robust. This would not happen without a lot of people keeping each other in check and collaborating, with these values of pluralism and democracy. I had been sort of sad that this was going to trend away in AI. Now I'm excited about this as a direction.

You're saying it shouldn't lead in a centralized direction, because you'll have many agents pushing against each other in a dynamic equilibrium. Offense-defense is a good example of this.

I would much rather, intuitively, have a trillion AIs of slightly superhuman intelligence join society than one AI of 1 trillion IQ join society.

Yeah. And you think we're headed in that direction, toward the multiple?

All of these things are things we have to push for. We have this incredible technology. It can be pointed and deployed in many ways. I think there are a lot of people who may have a different opinion about how it should be used and what the best version of the world is. But this is the one that I think is right.

Let's talk about starting a business. Thanks to coding models, your weekend side hustle can quickly become a serious undertaking. And we've been busy here at Stripe making that process as simple as possible. Stripe Atlas helps you with everything you need to start a business, like company formation, tax IDs, founder equity, and all the other bits that usually require someone in a suit. So if you've got a company in the making, take a look at stripe.com/atlas.

How's the compute shortage going?

I think things, at various times along the curve, are going to get somewhat out of hand. With any new thing of this importance, there are going to be moments of irrational exuberance, moments of bust, moments of constraint and glut, and ups and downs. But I think the trend is toward us always wanting more compute.

The demand for sufficiently capable models at a sufficiently low price only goes one way. We're doing a lot of things to try to be prepared for this. We made a chip that has incredible performance in terms of units of intelligence per watt. We are learning how to build our own data centers. We have a robotics program that we're going to point at making lots of data centers. People talk a lot about—

The real RSI.

Yeah. People talk a lot about RSI as an AI that does AI research. There's very little talk, relatively speaking, about self-replicating data centers: the idea that you can use AI to control robots, run fabs, build turbines or solar panels, and build data centers. But I think we have that within grasp. So we'll try to make a lot of really cheap compute.

But aren't data centers already not labor-bottlenecked? They're permitting-bottlenecked and component-bottlenecked.

We could make more components with robots.

Eventually.

Yeah, yeah, yeah. And all of this is how we solve this over a long period of time.

My observation on compute is that people say, “The solution to high prices is high prices.” Step one: We have high prices for memory, GPUs, and everything. The solutions should be here any moment now. How are we doing on the solutions front?

Bad.

I'm not saying this is OpenAI's job. I feel this is the invisible hand's job, right? Memory's gotten really expensive, so new memory makers are going to flood the market any moment now.

I think the invisible hand is failing right now. It has a checkered track record at best, but I think it's not quite working here.

Why not? Is it just hysteresis?

The specifics of each area matter here. Since you started with the memory guys: For a very long time, if you were a producer of DRAM, you would invest $20 billion of CapEx to make $19 billion, and you'd go through these booms and busts. Maybe some years you'd make $21 billion, some years you'd make $18 billion. Until right now, I don't think making memory has ever been a great business over a long period of time. Then all of a sudden, these guys are each going to make like $100 billion of free cash flow in a year. Unbelievable. But I think they all remember their history: how many times they almost went bankrupt by making too many fabs and being promised this time was different.

Which leads them to be conservative in their CapEx.

Clearly.

It feels like what you should expect to see is every part of the supply chain, with new startups coming along and, maybe more relevantly, people with relevant experience getting into it. Intel makes chips. They can probably get back into the memory game, as is tradition.

As is tradition.

Switch back.

There are all these ways the supply chain could develop; it's just super non-obvious. An example is that a lot of people want local inference.

And somehow, they assume that the problem is solved now because we’re going to have open-source models and be able to run them on our computers. So we’re not going to need all the stuff in the data center. But from a global allocation perspective, it’s actually quite terrible to have everybody with a bunch of the world’s limited supply of DRAM sitting in their own computer, being used 5% of the time.

Mm-hmm.

You want maximum utilization in the cloud if you’re trying to deal with this very supply-constrained environment. Somehow, many people’s intuition is the opposite of that.

Where else are we seeing the market respond to high prices? Not just in memory, but across the supply chain?

Turbines. Batteries. Concrete.

Sorry, the market is—

Steel—

Responding there in high prices. I’m asking where you’re seeing any interesting market responses in terms of high prices pulling in innovation.

Components for robots—some components for robots. And not the humanoid fake demos, but the industrial robots that can just make stuff faster.

I see. There are new actuator makers and that kind of stuff.

Yeah, honestly, maybe you could say, “If OpenAI sells our chip.” Maybe that would be an example.

Okay. Yeah, I want to get to your chip because that was going to be my example. I think the solution to high prices and limited supply is you make your own chip. Presumably, you’re using a lot of AI to make the chip. I would have thought that’s hard because chip design is one of those places limited by the public tokens you can get, where there aren’t actually that many blog posts on the public internet about how to build a leading-edge 2-nanometer chip.

In terms of how to do the lithography, I think there’s 1 company on Earth that seems to know how to do that. In terms of how to do the chip design—to build the file that you eventually send to tape out—that is well understood. That is a great problem for AI. Given that the world is going to be limited in its silicon capacity, we all have a shared interest. One of the things I’m proud of about OpenAI, and maybe this sounds obvious, but it wasn’t as obvious at the time, is that we have a culture where people, even if they’ve done something the old-fashioned way for a long time, are super willing to try using AI to do it better.

What are other examples of that?

AI research. Software engineering. But also, increasingly, procurement, or things that people think of as very outside the realm of what most people are using AI to do.

How does AI-driven AI research actually work? The math proofs where you say—or find the PHP vulnerability, where you say, “Please find me some AI advances.”

I think we don’t know yet how it truly is going to work. We know today how to say, “Find me a 5% compute efficiency win.”

But it’s that high-level? I could be an incredibly successful AI researcher now?

I don’t know about incredibly. But, yeah, if you were looking for a new job, you can come do it. What I don’t think anyone knows how to do yet is say, “Go discover a new architecture that is as big of a single leap forward as the transformer was.” I think that’s where things are heading.

What’s missing there?

Smarter models—smarter models that are smarter in this dimension.

The high-level planning and conceptual reasoning.

I think if you apply the same pipeline that we applied to get our models to be so good at math to AI research, it will probably work. Now, per an earlier part of this conversation, I kind of think that, at least for a while—maybe for a long while—we are going to have to agree to some limits about how smart we let these single models get. But a lack of progress will come from that decision, not from the AI’s inability to do good research.

Concretely, if I’m using AI to develop improved AI, we should expect GPT-7 to be really good now, because you have GPT-6 and whatever unreleased version of it to develop it. So my expectations are sky-high. But I come into work with my backpack, and I come into the OpenAI office in San Francisco. I sit down at my desk. Am I saying, “Find me more intelligence in this model”? Am I giving a real direction of research? How directed does it get? What does the iteration loop look like? What does RSI look like? You have a front-row seat to RSI. I’m curious what it looks like.

One of the interesting things that people talk a lot about at OpenAI, which sounds very obvious, is: What are the questions that you’re supposed to ask of GPT-7? It sounds like you could answer, “Well, build me RSI or whatever,” but it’s probably not enough to just say, “Make me RSI.” Also, it’s probably not enough to say, “Find me a 5% compute efficiency win.”

I think this is one of the most interesting debates going on right now. The answer is not just what question to ask it, but things like agent swarms: How are we going to use these models in a way that actually maximizes what we can get out of them? I think we’re going to learn a lot about that in the next 3 months.

Okay. But it sounds like you guys do not have a codified OpenAI standard way of doing AI-enabled AI development. It’s still lots of experimentation yourselves—

It’s lots of experimentation.

Yeah. When we were talking about the token budgets, it strikes me that this is one of the big unresolved questions for AI use in enterprises generally: We don’t have hard budgets for individual engineers. You don’t get cut off at a certain point. It’s like, “Sorry, you’ve spent all your arcade tokens.” Maybe we should, but we don’t. Again, I think we all believe in the scale-up of AI. But if you take that seriously, it means the dollars of compute, or AI per person, will go up. So I feel like we’re pretty soon going to run into this question of how you actually decide how much spending power to give individual people inside the organization.

There was a time when a lot of companies had this thing in their Google Calendar that showed the approximate cost of each meeting, in terms of everybody’s compensation who had to be there. I think that was always still an underestimate. There was this observation that Paul Graham made at one point, and many other people made it too: There are people at companies who have incredible power to save $17 on the snack budget of the office every week. And, yeah, you can have these meetings for unbelievable amounts of money that are adding no value, and there’s no equivalent ability to stop that.

I think that most use—okay, so I’m really talking about my book here, and I’m very biased, but I genuinely believe it. The use of AI is not a company’s biggest spend problem relative to other things that happen. Even if it’s somewhat inefficient, it is not where I would focus.

Well, hang on. That’s not scaling-law-pilled enough. I’m not—okay, 2 years from now.

You want to go spend all Stripe’s revenue, all Stripe’s profit on OpenAI services, and then complain about this? I’d be delighted to have a conversation.

But we’re on a path to that. We—

Great.

Yeah, yeah. We now spend, you know, 1%-ish of Stripe’s OpEx on tokens. That is obviously—

That feels too low.

Yeah. So we’re working on fixing it. And it’ll get to 5%, 10%, and 20%. Then you have a real question.

Do you have an intuition on what the right number is, what it should be today?

What it should be today is maybe tricky. But steady state, isn’t it going to be, depending on the kind of company, tens of percentage points of their OpEx?

I think for many companies, the answer to that is clearly yes. And I think there will come a time when the right thing to do is to really optimize that. Also, another thing I’m proud of about OpenAI is we’ve tried to really drive the cost per unit of intelligence down and down and down. I think we’ve been a good spokesperson and a good champion of this. But still, I think the problem, even at current costs—I think there are more companies with the problem of not using enough AI rather than using too much AI.

Do the dollars per gigawatt of leading-edge AI go up or down?

That should go down.

Down?

That should go down.

Why?

The reason it could go up is if AI just increases in value per what it can do per watt so quickly, and there’s such a supply shortage that people are willing to spend much more.

Ah. But you would view that as compute not keeping up.

I would view that as compute not keeping up. And if I look at the fundamental cost of turning the sand into intelligence, that should go down. The cost of energy should go down. The cost of chips should go down. The rest of the cost of the supply chain should go down. If it doesn’t go down, I’ll be very sad. But for it to go down, given how much value there is in this, supply’s got to go way up.

Okay, so you would like to see a compute glut, where we start producing a lot more memory, a lot more data centers, a lot more of all the inputs. And as a result, anyone who wants compute, for whatever reason, can have it.

People give different numbers for the cost per gigawatt, measured in the 5-year total cost of ownership, right now.

But let’s say, let’s pick $70 billion as a number. I think if that number were—

That’s the cost to install—

And run it for 5 years.

Yep. Okay.

I think if that number were $20 billion instead of $70 billion, we should be much happier. And we should still demand to get 10x better performance out of it with smarter models every year.

Presumably, you don’t think $2 billion is a good number. Something’s gone wrong if we’re seeing $2 billion, because it’d be surprising if we were—

Why is it not? I don’t think that will happen, but why do we not want that number to be as low as possible?

I don’t know. This is your theory. You have a view that, generally, deflationary technology is good.

I do believe in that. Yeah, I’m pro-deflation in a sense—pro-deflationary technology.

So how are we going to get there? What’s going to happen between now—

Well, $2 billion a gigawatt—I can’t imagine.

Okay, what’s going to happen between now and $20 billion a gigawatt?

$20 billion. $20 billion a gigawatt would be really great.

Okay, yeah, yeah. $20 billion a gigawatt.

Even to get down to $20 billion, you’re getting into the realm where energy’s got to get cheaper. But certainly, all of the gross margin stacking that happens between melting sand—or the equivalent of melting sand—and getting the token back, a lot of that’s got to get compressed.

You could just get there by eliminating profits, or do you also need to make energy cheaper? I don’t know the math off the top of my head.

If you assumed all materials were free and no one made a profit, maybe you could get there in the current energy envelope. But only just.

I see. Personal agents are the most exciting thing happening in AI right now. There are many exciting things happening in AI, but it’s one of the most exciting. You guys launched Dots a few weeks back, and there are a bunch of other people launching in the field. What will be true of the winning personal agent?

It’s a limited number of things: intelligence, context, tools. Someone is going to figure it out. I think no one has yet figured out the thing that people—people like these, but this isn’t something people say, “I cannot imagine my life without this thing,” the way they do when they talk about coding agents or even ChatGPT.

Mm-hmm.

I think someone’s going to figure out what that looks like soon. You talked about one of the reasons why this doesn’t quite work yet, but I think this is a small number of ideas away from being a super-mainstream thing.

Oh, interesting. You think we haven’t had the Macintosh moment for personal agents?

I think we haven’t.

Do you have any sense of what the shape of it will be?

I think it’ll look a lot like the current Dots, or Muse, or Grokbot, or whatever. But it’ll just work. It will just work. It will be reliable. You’ll never have... You’ll just give it more complex tasks, and it’ll do all of those things.

The computer-use dynamic is very interesting to me, because not everyone is embracing the new agentic future with open arms. But it’s not totally within their control, because you can just have an agent using a computer. Where do you think that dynamic goes, this sort of cat and mouse?

My rough instinct is that people are going to fight this in both directions, but eventually converge on something like, “You’re allowed to tell an AI to use your computer on your behalf, including software.”

People always talk about that, but isn’t it a bit like the semantic web? It’s a really nice conceptual idea.

Yeah, I was going to say. I don’t think this is so black and white. I don’t have a super high degree of confidence about how this is going to play out. As a consumer, I feel somewhat annoyed if I can’t—

Yes.

—have an AI automate tasks on my own computer. I want it to feel like it’s my agent doing what I ask. But then it gets like, I don’t have to go do something else. Yeah.

I note a very interesting dynamic right now, which is every company is really eager to add AI to its products in a very specific way: there’s a tab or popup widget or something that gives you a messaging-type UI. And it’s often surprisingly capable. You can say, “Hey, I actually want to return my most recent order,” and then you’re surprisingly impressed when it’s able to find your most recent order. It’s a bit janky, but it can do those things.

However, no consumer really wants this UI, because consumers have strong—not immovable, but strong—preferences for their existing primary chat app. It’s the kind of top-of-wallet app they open when they want to do something. What they want is to be able to say, “Hey, go find my recent order on this business and refund it,” or something like that.

I notice that businesses are really reluctant to allow people to use their services inside another AI app. They think, “Oh, we’re going to lose control and our moat,” or whatever. And it just strikes me as fine, because your defensibility probably didn’t come from your ability to put together HTML.

Your defensibility came from brand loyalty and product selection. The new UI is that agentic interaction inside ChatGPT, Gemini, or whatever it is. But all businesses disagree with me here.

I mean, again, I have a bias here, but I so directionally agree with you. I have such an extreme preference for buying something from Amazon versus some random e-commerce vendor that I don’t know if I can return to, or don’t know if I can trust to deliver it quickly, or something like that. If I ask ChatGPT to buy me something and I can choose Amazon versus something else, 10 out of 10 times I click Amazon. I’ll pay a higher price. I just have such—

Yes.

And it has nothing to do with my experience of clicking around Amazon.com.

Exactly.

Now, I think you could say we haven’t done a good enough job offering partners a platform inside ChatGPT if they don’t want this to happen. But, again, maybe I’m really biased here. I think Amazon’s in a great place if commerce moves much more agentic.

Amazon in particular has this amazing defensibility via its fulfillment network, so you can use Amazon kind of from anywhere.

I super trust that.

I’m just really struck by this dynamic. My new thinking, in particular with regard to agentic commerce, but also anything, is that anytime there’s a web form field you as a human are filling out, that’s below your pay grade. You’re probably not inventing that information for the first time. We’ve all filled out the medical forms where it’s like, “Oh, your family history,” or whatever.

Yeah.

And that’s so frustrating, because how many times have you filled out your dad’s and your mom’s medical history and things like that? It feels so wasteful. But there are a million versions of that. You’re never inventing new information for a web form.

I would be so happy to never do fake work at a computer again.

Yeah. My argument is that any web form is basically fake work.

I agree. I was one of the very earliest adopters I knew of of those plugins that put in your name and address very clumsily, because I was just like, “This is—I cannot do this again.” But I kind of feel that way about using most of the web at this point.

I find this phenomenon really interesting, where businesses right now are really attached to their HTML and mobile apps as sources of business defensibility. It just feels like either they’re wrong and it’s not a source of defensibility, or it may be, but it’s transitory and they just need to get over it.

The last example of this that I think we really witnessed was the app stores, specifically the iPhone App Store. I remember every startup—or many startups, most of the YC batch—was like, “I’m gonna build an iPhone app. This is new distribution. This is awesome. I can do a new thing I couldn’t do before.” And many new startups got created. All the big companies felt like they had to make one, but they kind of didn’t want to, because they were like, “Oh, I’m going to hand my relationship with my consumer over to Apple.” And it was great for mobile.

It was just a new UI.

Yeah.

Okay, I’m going to go more rapid-fire. People complain about benchmaxxing. Is that a real concern? Are OpenAI models benchmaxxed? Are you actually Rick Rubin-ing it, where you just feel the vibe of this model is right and you don’t care what SWE-bench gives it?

Maybe there are 2 things we should really care about models being good at. One is scientific discovery and doing things that people cannot do. The other is being enjoyable to work with. I don’t think we have great benchmarks for being enjoyable to work with. I’m actually not sure we have great benchmarks for the smartest scientific scientist, either. But we endeavor not to benchmaxx. I keep thinking that benchmaxxing is over as a phenomenon, and people keep doing it.

So are you better than other labs at not getting sucked in by the siren call of benchmaxxing?

I believe we are better at that.

It feels like—

I don’t think we’ve always been better at that in our history.

I believe we are better than that right now.

It feels like the industry is still figuring out “being nice to work with.” There was the glazing era, and more recently, the stroppy, ornery era of models that were anti-glazing and, as a result, tried to disagree with you and nitpick at you on every minor thing. The current ones are pretty good, actually.

I think the latest models are much better than we’ve had in a long time.

But it also feels like there are probably higher peaks to ascend.

Emmett Shear said to me years ago that what he wanted was a really thoughtful and challenging, but still respectful, senior coworker. I still haven’t heard a better, brief encapsulation than that.

Yes. After working with him for many years, what’s your grand unified theory of Greg Brockman?

I think Greg Brockman is one of a kind in the best sense.

He’s singular on the Greg Brockman scale. You just have to be there to understand what that is.

It’s a combination of intensity, monster productivity, extreme dedication, and focus on succeeding at the mission. It’s very cool.

You hate tokens as a metric for anything, really—pricing, or... It’s not the standard unit of AI, and yet we found ourselves there.

You might say outcomes, but I don’t think it can be outcomes, because outcomes are too different across different use cases. What’s the right standardized unit?

We almost made what would’ve been a bad call: to say we’re not going to price our next model in tokens at all. You’re just going to give it a prompt, and we’re going to give you a price back. We’re going to do it, and it’s going to be like AWS to you: just do something and we send you a bill. But the customer response to that is so negative. They’re like, “I need to know what I’m paying for somehow.”

Yes.

I asked a few people, “Would you rather pay more for the same output, but have it priced in some number of consistent tokens? Or something?” Even if we inflated the tokens, they’re like, “Oh, I’d much rather you inflate the tokens and pay more, because I know there’s a real thing there.” So I think you’re right: people don’t just want us to give them a price, because they feel like over the long term they’re going to get screwed somehow. In the same way, maybe people would rather pay for CPU hours rather than just getting their thing done, even if it was less expensive.

You know how nanometers are used for chip gate length?

It’s totally fake.

Yeah. The nanometer has very little relation to reality these days. Will we get something like that in tokens, where you guys should start saying, “Here is an intelligence-equivalent token,” like “I Can’t Believe It’s Not Butter”?

Of the smallest feature?

Yeah.

That’s made up. I hope not. It offends me that we use the word “nanometers” for something that has nothing to do with nanometers.

I know, it’s very silly. Yeah.

I would rather not make up a fake unit. Tokens are a kind of fake unit. They’re real in some sense, but they’re not a good unit.

They’re fake from a business-value point of view. Okay, isn’t the only defense of tokens that if you’re wasting tokens, we should fix that pretty soon? We shouldn’t be streaming—Claude Shannon would not approve of us just streaming noise rather than signal. There’s a floor of the information-theoretic most useful tokens on both sides. We’ll kind of come back to that.

Yeah. Right now you have to think about whether you should want 10 cents per million tokens out of a small model or $10 per token tokens out of a big model, and how long the chain of thought is likely to be. That’s just cognitive overhead that I don’t think actually adds any value.

Is the chain of thought included in the price?

In what we charge? Most people do.

Okay.

By the way, the Guinness was not as bad as I thought. Definitely not as good as an IPA, but not as bad.

It’s good.

An IPA is great, but it’s pretty good.

We like Guinness.

Yeah.

At least in this podcast. Thank you.