48.对话前OpenAI科学家：GPT-5能获得奥赛金牌，但那可能具有欺骗性

That was 2020. I thought OpenAI was a good company. My perception of Sam was that he was a good leader. I thought he was pragmatic, circumspect, and a good communicator.

So do you feel sorry about the changes at OpenAI?

I think I am a little sorry. The field was just very different in the past, and I think more playful than it is now.

It has sparked criticism that GPT-5 is squeezing innovation out like toothpaste.

You know, computer systems that can actually succeed and get a gold medal at a math olympiad—that is an amazing achievement from a benchmark perspective—and that would be because it is deceptive. So I think it actually makes it a really interesting time, because if things are starting to slow a bit—

Hello，大家好，欢迎来到商业漫谈，我是诗洁。在经历了漫长的等待和难产后，八月八号GPT-5终于发布了。还是有不少国内的媒体将它称作史上最强的基础模型，但实际上外网关于GPT-5的质疑声却很汹涌。发布会的内容中出现了伪科普和演示乌龙，GPT-5的性能进展在一些人看来远未及预期。在这个节点，我对话了OpenAI的两位前科学家Kenny Stanley和Joe Lappman，他们曾亲历前ChatGPT时代的OpenAI，也是《为什么伟大不能被计划》一书的作者。二零二。二零二三年，这本书曾在中国科技圈掀起阅读的潮流，那正是 ChatGPT席卷全球，掀开大模型浪潮的一年。然而，Ken 和 Joe 却选择了离开。早在二零一五年，他们就抛出了独特观点：创新需要摒弃对目标的追求感，并追求新奇性，所有伟大的创新都是不期而遇的。当时这个论点并没有被太多人接纳。直到二零二二年底，ChatGPT出现，这一论点被强有力的验证了，因为ChatGPT本身就是一个意料之外的项目。三年过去，Kai和Joe仍在持续推动无目标创新的理念，两位AI科学家也始终保持着对AI圈的关注。这期访谈中，我们聊到了OpenAI的发展史，其本身就是创新曲线的一体两面。当ChatGPT五出现，Kai和Joe却在一片失望声中表示，这也。是，这也许恰恰意味着关于AI的研究将再次变得有趣。要知道，过去几年，尽管全世界都在瞩目和追逐大模型的进展，但在他们看来，和从前相比，学术界正在变得无聊。这是两位关于创新的布道者，这期的内容围绕AI，却又不止于AI。

1. ChatGPT Validates The Book

The Chinese version was published in 2023, when OpenAI was already world-famous, and one of its selling points was that you two authors were core scientists at OpenAI. But the fact is, you wrote the book as early as 2015, right?

So when ChatGPT emerged, did you realize that the book’s claim about how to foster innovation had been powerfully validated? We all know that ChatGPT was originally an unexpected project.

Yes, it did occur to me. I don’t think anyone realized how big ChatGPT would become, and that’s consistent with the story we tell in the book. It’s not just that the success of ChatGPT was unexpected; all of the little stepping stones that led to it at OpenAI were often unexpected as well.

The Transformer neural network architecture that ChatGPT was built upon came from research done at Google. A lot of the initial explorations that OpenAI did had nothing to do with language models. They were doing very cool and exciting research into making a mechanical hand that could play the Rubik’s Cube and play video games.

Did you two realize that our book would become known by more people because of the emergence of ChatGPT? The book was titled Why Greatness Cannot Be Planned. Is ChatGPT an example of greatness? And how do we define greatness?

I think it does qualify as greatness, at least in this case, in terms of impact. It has had a massive worldwide impact. But I’m not sure that there’s an easy, generic definition that would cover all forms of greatness.

Some great things might not be measured by impact, or at least not by the number of people impacted. Maybe you impact 2 people, but you impact them profoundly, and that’s still great. It’s a bit subjective, but certainly ChatGPT is the kind of thing the book is talking about.

So you think it is an example of greatness?

Yes.

2. Picbreeder Reveals The Power Of Novelty

The book deeply explores the composition of the soil of innovation. The story begins in 2008, and its original inspiration came from Kenneth Stanley’s unexpected discoveries in a series of experiments about robots, right?

Actually, I think it goes back to Picbreeder, which is described in the book. It was an experiment we set up to allow people to breed pictures online. Sometimes this is called evolutionary art or genetic art.

You could go onto a website and see a bunch of images. You could click on one, and that image would then have children. Just as children are slightly different from their parents, these images would be slightly different from the parent image. What you were doing was breeding the images over generations.

The important question, though, is why we would do this experiment. Why would this lead to anything scientifically important? Our motivation was that we were interested in open-ended systems—systems that continually innovate and make discoveries over long periods of time. Human civilization is an example of an open-ended system.

We wanted to understand how to build open-ended systems. Picbreeder was an experiment trying to create an artificial open-ended system to help us understand how such systems work. We involved humans because we didn’t know how to build a system like that completely autonomously. If we brought humans in, they were able to explore in an open-ended way, and we could get an actual open-ended tree of ideas. Picbreeder was a vehicle for us to learn more.

It turned out that we did learn some very deep things—things that we did not predict. The particular thing we observed that led to the book was that the best way to discover something great was not to be looking for it.

In fact, we found that if people did look for something—for example, if you came to Picbreeder and wanted a bird—you would fail, because it only works well if you are not looking for it. It was an observation, not a prediction. We observed it in hindsight, after we had already seen the results of the system.

For me, seeing that observation was completely mind-blowing. It seemed to contradict everything I had ever believed and been taught. Coming from a computer science and engineering background, I had been taught that the way to achieve something was to set it as a goal and then deliberately move toward it. That’s how we do things.

But in Picbreeder, it was the exact opposite. The best way to achieve something notable was not to try to achieve it. I became obsessed with this observation. It was so strange that I started to think it probably had larger implications, and I began to develop an understanding of why it was happening.

The reason was that the stepping stones leading to things that are interesting, such as a butterfly, don’t actually look like the thing they lead to. The pictures that led to the butterfly did not look like butterflies. If you were thinking only about butterflies, you would not breed those pictures. You had to not be thinking about butterflies in order to choose the pictures that led to butterflies. It was a paradox.

As I understood this, I started to realize that what was powerful was the collection of diversity itself. The reason the system was so successful was not that people set goals and reached them. It was that the system accumulated diverse stepping stones, which then became more places to jump off from in order to find more things.

That led me to think about novelty, because novelty is a way of getting diversity. That’s when Joel came into the picture. Joel had just started right as this was occurring to me. I remember the meeting we had. Joel came into my office, and I basically started dumping all these thoughts on him. I said, “Look, we saw all this stuff. These are crazy observations, and maybe you could make an algorithm that just searches for novelty.”

Joel took all that and actually created the novelty search algorithm. That’s where the robots started to come in, because we did some of the experiments with simulated robots using the novelty search algorithm, and it was actually working. You could get a robot through a maze more effectively if you didn’t try to tell the robot where the goal was in the maze.

I won’t go through all of it because this is already too long, but Joel and I realized that there were bigger social implications to this. At the time, it was just an algorithmic discovery relevant to the field of artificial intelligence, but it was pretty clear that there were big implications for how we run human systems.

All human systems are always directed toward objectives. Joel and I recognized that there was a message here that could be appropriate for a larger social conversation. That led us to decide to take the crazy step—which I think was kind of crazy—of writing this book.

Neither of us had ever engaged in social critique or spoken to the public at large, but we decided that this was important enough to try to do.

3. The Book Finds Its Audience

Why did you think you had to write a book?

We might both have different stories. My story is that when I spoke about this at an AI conference, a student at the Rhode Island School of Design, an art and design school, said, “Why don’t you come give a talk at the school?”

This was an audience of artists, basically. I could see why it might be interesting, but it was beyond what I was expecting. I gave them this message: “It’s not important to be able to define your objectives in order to innovate.”

What was really striking to me was the emotional reaction. They were so emotional; some of them were on the verge of tears. I swear, it sounds crazy, because I was reporting results from artificial intelligence experiments. I don’t know if anyone had ever cried about that before.

They told me, “I’ve never been able to justify what I chose to do to my parents or my teachers.” Their parents and teachers would ask, “How are you going to make money? What is your goal in life? Why are you making all this pointless stuff?”

They said, “This is the first time I feel I can articulate a justification for why I decided to do things the way I did in my life, because I’m following the path of what’s interesting.”

For me, that really crystallized the idea that this could be a great book, if we could do that.

After that, I was supercharged. If people are going to cry about this, then it’s definitely important. A lot of people probably would like to hear this message, and it’s an important social conversation.

4. OpenAI Grows Beyond Research

I think Asian children will have strong feelings about your stories, right? After completing this book, you both joined Uber AI Labs from 2016 to 2019, and then OpenAI from 2020 to 2022, almost simultaneously. Why did you choose these 2 companies?

Yeah, it was kind of serendipitous, a little bit like the book. Working at OpenAI is a very cool job, but it was just a happy coincidence, I guess you could say. When I joined OpenAI in 2020, I thought it was a good company. They had a lot of smart people and very interesting stuff going on, so it’s not surprising that there would be some innovation. I don’t think I could have predicted something as impactful as ChatGPT. That wasn’t something I could predict.

Many tech practitioners follow our show. Here’s a question for you: How do you identify an innovative startup?

Yeah, I think it’s difficult to know a priori whether a startup is going to be interesting or not. But I guess an interesting startup is usually composed of interesting people, and interesting people usually have interesting ideas.

For identifying an interesting startup, I think it depends on who you are—if you’re an investor versus if you’re going to be an employee. As an employee, I would look at whether it aligns with my interests. As an investor, I would look at it a bit differently. I should place bets on things that have high potential. If they succeed, is this innovative enough, interesting enough, or maybe novel enough to make the risk worth it?

What interesting explorations were there within OpenAI at that time?

Let’s see. In 2020 at OpenAI, I saw the early GPT models, and I also saw early image generators. I mean, now it’s old news, but at that time it was very interesting and almost shocking in some ways that technology had advanced that fast.

Now it is the future. So how was your time, Joel, when you were at OpenAI?

Yeah, it was really interesting. It was really otherworldly because you were in this environment where, as Ken said, you were a little bit ahead of everyone else. There were a lot of smart people.

Who did you respect and love the most?

That’s an interesting question. I deeply respect Ken. We were co-leading the team, and that was really fun. But it’s kind of a personal thing. I really like Jan Leike. He was the AI safety guy there, and he’s just a good person. I enjoyed talking to him.

But there are lots of interesting people at OpenAI, so it’s kind of hard to pick.

Okay, but you have to pick one. How about Ken?

That is a hard question. Of course, I would point to Joel. You have to, but it’s true, though. I mean, it is true. I respected a lot of people there, including most of the leadership, so I don’t really have a single answer that I could give to that question.

Okay, Sam has publicly expressed his admiration for your book, but with the power struggle within OpenAI, his public image has also become blurred and controversial. I’ve even heard claims positioning Ilya as the true visionary while labeling Sam as a false prophet. As former members, how do you view Sam’s leadership, and what kind of leader is he?

I was there between 2020 and 2022, and my perception of Sam—which is all from before the events you’re talking about—was that he was a good leader. I thought he was pragmatic and circumspect. He was a good communicator. He seemed to weigh questions carefully, and I didn’t see anything to criticize at the time.

How do I feel about the struggle? I was surprised, like everyone. I was just surprised. I didn’t expect it; it was a big shock to hear about that. Sam seemed firmly in charge, and I would not have expected people to try to remove him. So I am as much in the dark as everyone, because I just don’t know to this day exactly what happened.

Yeah, it just seemed like an extraordinary series of events. It’s kind of crazy: a lot of money and power, and people trying to do what’s best while having different perspectives. It’s really hard for me to make out what went on there.

With the release of GPT-3.5 at the end of 2022, once the results of OpenAI’s explorations were seen, the global artificial intelligence industry kicked off a fierce competition. At that time, did you feel a change in the atmosphere?

I guess I can say that there was a noticeable change over time. If you have a product that is so impactful, then you start to change from just pure exploration to following something with more intent. So, yeah, you could kind of feel the gears start to shift toward a little bit more of an exploit mode.

Then, over time, you get more or less some race dynamics as the interest overall is mounting, and the technology itself is, in some sense, relatively simple. Maybe training a language model on bigger and bigger datasets with more and more GPUs, and other people started to follow suit.

I would agree with Joel that there were some changes, because when you strike gold, you tend to focus. I think the most dramatic change probably happened after we were gone, because it was really ChatGPT that probably dramatically changed the company.

But during our time, you could see that there was a convergence on language models and a realization of the importance of this particular direction, so there was some shifting happening. It’s interesting to observe that from the inside: how the company responds to such a dramatic shift happening in the company and the world.

What was the phenomenon? Could you show us some stories about that specifically?

Well, the applied, commercial side of the company was growing quickly. It was originally more of a research lab, and suddenly it had this growing commercial arm, so that was a difference in the character of what was inside the company.

Of course, they continued to do research, but suddenly you had this very significant component of the company that was now very commercially oriented.

That’s also my understanding. In 2023, OpenAI stood at a crossroads. On the left was continuing to explore cutting-edge technology as a nonprofit organization, just like before, and on the right was pursuing becoming the next tech giant. Are there any misunderstandings about this company in the outside world?

2023 was after we were gone, so I can only speculate about what it was like. I would guess they didn’t view it in such stark terms. I imagine they wanted to do both: continue to be a great research lab and also exploit the commercial opportunities. So that’s what I’m guessing was the mentality.

I mean, everybody knows research is important. Sam knows research is important too, and I think that, at least, they would want to try to find a way to keep that going. But obviously, there’s an enormous commercial opportunity that is just impossible to ignore standing before them, and there’s no question that will have some implications for the culture of the company.

Do you feel sorry about the changes at OpenAI?

I don’t think I would say I feel sorry. I think the change we’re observing is a change in the entire industry. It’s not just OpenAI. I mean, AI really changed dramatically, because I’ve been in AI for a long time. It was more like a small scientific discipline that had massive social implications.

I found that it was no longer the same field that I had been in for the last 15 or 20 years, and so it’s disorienting when that happens. OpenAI was obviously the leader of the field, and it takes time to process and absorb what’s happening when something that was almost just a small, little game suddenly turns into the most important thing in the world. So that’s just a very complicated and difficult transition.

How about you?

I think I am a little sorry—not in the sense that I have nostalgic feelings for the kinds of research that were done prior to this latest revolution, when there was just more diversity of stuff going on and lots of interesting, crazy stuff. There still is, but there are just so many language-model papers now.

I kind of wish that, if I could rewind time and change things a bit, I could delay the onset of this outbreak of technology just a little bit longer. It would have been nice. I guess I have a feeling that the language-model stuff is spreading through society too quickly.

It’s super fun and exciting, and I use the tools all the time. But the field, as Ken noticed, was very different in the past, and I think it was more playful than it is now. Maybe it’s also just my inclination toward more basic research. Scaling is an important science as well, but it’s just a little less intellectually compelling to me than techniques to make something just bigger and bigger and bigger, even though it’s amazing what happens when you do that.

So personally, what do you expect OpenAI to be—a great commercial company or a great research institution?

I think its legacy is definitely as a great research institution, and at present it’s more known as a commercial giant, although it’s still kind of known as being backed by research.

Yeah, so it’s hard to say. Personally, I always expected it to be a research institution when I joined.

I don't think I expected it to be kind of a commercial giant.

That's true for me too. That was kind of a big surprise—that it turned into this massive commercial organization. Back in 2020, I think it seemed more like just a research institution, but it's kind of consistent with our book in some ways. If you looked at it in 2019, you wouldn't know what the business plan was at all. What was the plan here?

But that's basically what led to all the success: there was no really clear plan. They just followed, opportunistically, whatever was interesting, and it turned out to be massively successful.

Speaking of OpenAI, after a difficult birth, GPT-5 was finally launched several days ago. It has sparked criticism that GPT-5 is squeezing innovation out like toothpaste. What's your take?

5. GPT-5 Tests The Scaling Thesis

Let's see. I think there's a general question in the field of artificial intelligence about whether the path that it's been on for the last couple of years—which is just scaling and data—is really the path to the most exciting, revolutionary form of AI, which some people might call AGI. I think it's interesting whether we're seeing evidence that the returns are slowing, or whether this is confirming that this is the correct path. It seems like some people disagree on this question.

I think it wouldn't be surprising if the scaling path does not, on its own, lead all the way to whatever this holy grail is. It's not the only thing that matters in intelligence, and so I don't think it's necessarily a surprise if just continuing along this path doesn't lead to something revolutionary every time you come out with a new model from this kind of process. It just represents that there's room for a breakthrough, probably at this point in time.

Because I'm a researcher, I look at any limitation as an opportunity, so somebody can disrupt. I think it actually makes it a really interesting time, because if things are starting to slow a bit on the path that we've been going down, there's room for more interesting ideas to come in and do new things. From a product perspective, it's been exciting for the last couple of years, but it's gotten less exciting from a research perspective, because basically everything is just following this path that we've already set.

How about Joel? What do you think of GPT-5?

Yeah, I think it seems pretty uncontroversial to say that the jump from GPT-3 to GPT-4 seems more profound than the jump from GPT-4 to GPT-5. It certainly is an amazing piece of technology. It is a bit unclear where we are on this trajectory, whether this paradigm is plateauing or whether there's more juice to come from continuing to scale. There's a lot of uncertainty, but I guess I hope it's running out of steam again. That's more just my personal interest.

It would be more interesting if we needed different kinds of technologies. I have a suspicion that there is something strange about the way that these models are trained relative to how humans understand things, and that there are likely stepping stones to come. I don't think transformers are the end of the line in terms of where this story goes, and so I'd be excited for something different.

Does the story of OpenAI tell us that once open exploration succeeds and its achievements are recognized, it is prone to fierce competition or various temptations that undermine the innovative atmosphere? Is that hard to avoid?

6. Success Threatens Open Exploration

Yeah, I think that is a danger that's hard to avoid. Throughout history, we see cases where an innovator in any field does something revolutionary, becomes the top performer, and then is disrupted later on by some new innovator who comes in and wasn't known. That's just the story of technology and many other things.

There does seem to be a phenomenon where a big win can cause you to start converging. You just start to put all your resources into the thing that looks like it's the winner, and you maintain your dominant position for some time until someone who's more innovative has a different kind of idea and you've lost the ability to innovate as much. That seems to happen a lot, and I think it's a deep issue because it's not just that somebody lost the will to innovate. It's also that the road that you took to innovate doesn't work twice.

You may have followed certain principles and heuristics on the road to your amazing discovery, and that worked for you because you were successful. So now you think you can just do that again, but actually the next innovation is going to have a different story. The formula that worked for you is no longer the formula that will work, and I think all innovators fall into this trap.

Some innovators do reinvent themselves every 10 years or so, but it's really difficult to fight your own nature, your own heuristics, and your own curiosity.

But what do you think of Google? It's both a great commercial company and the birthplace of AI research.

Yeah, I think Google's a really interesting place. They didn't invent the Transformer architecture, the thing that really set off the language-model revolution. It's true also that, as a larger company, maybe they were a little more cautious when it came to deploying technology, and that might not be a bad thing.

I think the story goes that maybe—I can't remember if it was Google—Google had a kind of chat-like model before OpenAI did. Google was the first to discover the direction of large models, but missed out on the GPT revolution and lost a large number of AI talents.

Once the giant was awakened, I think now they're going basically toe to toe with Anthropic and OpenAI. They've maybe caught up in some respects, but it's true that being a bigger company, being more bureaucratic, and all the kinds of things that happen mean you're almost consumed by your own success, because that's a story that happens over and over again.

But I think they continue to do great research, as far as I know. What happened with OpenAI was a disruption. Obviously, it's a painful story for Google to watch that unfold with things that came from Google, but it's an eternal story of innovation. Kodak was disrupted by digital photography, and they didn't regain their footing as well.

I think Google, though, has to some extent regained its footing, like Joel said. It's pretty competitive now, and DeepMind has certainly had some considerable achievements outside language models, like with AlphaFold and things like that. But there was a moment that they missed. That's clear to their regret, I'm sure.

How can mature enterprises avoid having goal management stifle innovation? That is a big and very important question for companies like Google.

Our book does provide potential lessons that could help with that. Some of those lessons are that you sometimes do have to let go of your objectives in order to innovate, and I think that as companies mature and become dominant and extremely successful, they become more likely to be objective-oriented.

Google was one of the popularizers of OKRs and KPIs—OKRs, objectives and key results—which are used throughout the industry. So it's kind of a reinforcing cycle: objectives have led us to our success, so let's impose even more objectives.

I think it can be very hard in that environment, which also tends to become more bureaucratic, because the bigger you are, the more bureaucracy you need to keep things moving. Large companies say things like, “Let's establish a research lab, Google Brain or something like that. We really believe in exploratory research. You guys are here to do that.” Everybody should be happy.

But what happens in practice, I think, is that people understand that ultimately they're still being evaluated based on the company's OKRs. Researchers can't be fooled. They can see that their colleague got promoted because he helped with something that helped Google's bottom line, and then when you see that, you think, “They say they care about this, but the people who get rewarded are the people who actually advance the objective.”

So I'm also going to respond to that signal, because I want to get a bonus this year and I want to get a raise. This is why startups often have a disruptive advantage, because that's so hard to do in a big company.

If you create an innovation lab inside a big company, the people not in the lab will become jealous and angry. People in this lab don't have to help us advance our objectives. They can just do anything they find interesting. Then everyone else says, “Why do those people get to do that? They're getting paid all this money basically to sit around like children in a playground and do whatever they want.”

It creates a cultural risk for a big company. But the paradoxical point is that doing something that's not useful is actually the best thing to do from an innovative standpoint. I think the challenges that a big company faces are massive and require courage.

Me telling you that you should do whatever you want doesn't make me feel confident that things are going to work out. I would rather tell you that you have to achieve X this year or else I'm not going to give you a raise. Then I feel safe, because I'm protecting myself by threatening you with some kind of penalty.

That's what our book is for, I think: to empower people to have that courage. I would hope that the leadership of these companies would read the book, not because it's going to help Joel and me make money—it would be some tiny amount for us—but because it could change the culture.

This could empower them to make these very difficult decisions. When somebody creates an innovation lab, it requires this kind of courageous change to truly protect it and cause it to be what it's supposed to be.

Yeah, I'm pretty sure many Chinese entrepreneurs have already read your books.

But here's a piece of history. OpenAI was established in 2015, and it encouraged scholars to explore various cutting-edge directions in its early days, which sounds in line with your proposition: more free exploration rather than planning, right? But 2 years later, hundreds of millions of dollars were about to run out. A key question is: Is open-ended exploration really sustainable, and who should foot the bill for this?

I think there are definitely settings where open-ended exploration is sustainable. Within universities, you have basic research, and that's usually funded by the public. In the long run, the public gets more out than they put in. There are returns on that investment, which are pretty large.

It becomes trickier when you try to have open-ended exploration within a corporate setting where there's an imperative to operate on a certain timescale, and you're under pressure to optimize profits. In the worst case, if you don't make enough money, the company might stop existing. So I think it becomes the realm of really well-situated companies to have free rein to explore as broadly as possible.

But even thinking about more limited ways of exploring can still be useful. I think it's definitely sustainable as long as a company is healthy. Success should not be an impediment to open-ended exploration. Societies where people are barely able to eat are not innovating. Innovation starts happening when you're in a situation where you can afford to take risks because it doesn't mean you're going to die.

It's kind of counterintuitive that a lot of people think success will destroy the ability to do this. In principle, you should be able to continue because you're successful. I should add, though, that you can be doing open-ended innovation in a small company. That's definitely possible.

So, facing the imminent depletion of funds in 2070, OpenAI decided to focus all its resources on large language models, which gave birth to ChatGPT. Does this conflict with the idea that greatness cannot be planned? How can we strike a delicate balance between divergence and convergence?

That's interesting, though. When we arrived at OpenAI, they were doing more than just language models. So I don't think they were completely converged on language models, but I believe you that there may have been some decision made—maybe somebody saw that as a road forward. That may have happened.

I think it doesn't contradict anything in the book because our book does not argue against conviction. We're not saying you can never make a decision. If you have a strong feeling that this path is interesting, our book would endorse going down that path. It's just that we're saying you shouldn't go down that path only because you think it leads to some goal, some far-off goal.

If they made a decision in 2017 to invest in language models, they had absolutely no awareness or expectation that ChatGPT would be created 6 or 7 years later. So that wasn't the goal driving the decision in 2017. It was probably more a conviction that this was an interesting technology, which was obviously a great conviction in hindsight.

It was a really interesting conviction at that very early stage because GPT-1 was just garbage. It was not a good product for anything, but it was interesting—very interesting. Someone who had the courage to then invest more in that is a very smart person.

Yeah, there's this interesting phenomenon where you might exhibit being divergent by converging. I think OpenAI is an example of this. The constant through-line from the early days was, I think, a conviction that scaling neural networks would yield interesting results.

As a company, they were converged on that hypothesis, which no one else was really matching at the same level of conviction. It was a really high-conviction bet that would have paid off. Research would continue, more stepping stones would be built, and hopefully the system as a whole would continue to leverage them.

What do you think of the ongoing race regarding artificial intelligence? You have proposed deceptive objectives. Is this reflected in today's AI race?

I think it's definitely reflected in the race. You always see these benchmarks reported: some math test, some programming test. In my opinion, that's what our book cautions against. Doing that can be dangerous for innovation and deceptive in the sense that we may actually see scores increasing on these tests without actually increasing what we think of as intelligence. At first, it may not be true.

But something happens along the way where we become too obsessed with these particular metrics, and then we lose the big picture. Losing the big picture is the deception.

There are some interesting things happening. We see computer systems that can actually succeed and get a gold medal on a math Olympiad—an amazing achievement from a benchmark perspective. So where's all the new math, since it's PhD-level, as the leaders of all these different labs like to say? Why hasn't it invented a lot of new math?

Maybe one possibility here is that getting better and better scores on these tests is not going to lead to a lot of new math, and that would be because it's deceptive. By being obsessed with the objective, we're risking that this could be what's happening, that we're on a deceptive path.

It's interesting that this doesn't seem to be causing more concern. People are like, “Oh, it's coming.” Everything being benchmark-driven means that the entire path could be highly deceptive. Right now, every time a new model is released, it's better on the benchmarks than everything else. That starts to make it look like these benchmarks are gameable.

It shouldn't be the case that every time a new model is released, it's better on the benchmarks than everything else. That seems a bit suspicious. Maybe we're actually playing to the benchmark rather than playing to this other thing, which is intelligence.

Yeah, I completely agree with Kenneth that the benchmarks can be misleading if you just look at the history of AI. We've been through this cycle before, where neural networks, like 20 or 30 years ago, were kind of exciting. Then they were considered bad after the release of, I think, the book Perceptrons, and then they went through a winter and a summer, and so on—ups and downs.

We may not think this is the end of the story, but if AGI is around the corner, I don't know if we're ready for it. We might be barreling toward it anyway. But I do think there's a big risk of deception here.

From the perspective of the innovation atmosphere, which company is most similar to OpenAI back then among the startup companies in Silicon Valley or around the world?

When you say “back then,” do you mean when OpenAI was founded?

Maybe by 2020 or 2022, I think.

I think there's a strong temptation here for Joel and me to cite our own company right now. I mean, we're at Lila Sciences, and this is a new thing. We've started a new open-endedness team. The feeling of starting a new open-endedness team obviously reminds me of starting at OpenAI, where we started an open-endedness team.

Could you give us more of an introduction to your company?

7. Lila Sciences Pursues Scientific Superintelligence

Lila Sciences, which Joel and I have both joined recently, is trying to create what the company calls scientific superintelligence. It's taking this notion of superintelligence, which is what a lot of companies talk about, but aiming it particularly at science—the advancement of science itself.

I think there are a couple of unique aspects to their hypothesis. One is that we won't really see a great scientific revolution from internet data alone, which is what many companies are talking about. Ultimately, they say that when we get to superintelligence, the really big benefit will be new medicines, new energy, new materials, and all kinds of scientific revolutions resulting from superintelligence.

But here at Lila, what they're saying is that this won't happen just from internet data. You have to have actual interaction with the world in order to test hypotheses, get more information, and grow as a scientist and as a researcher. I think it's arguable that, because of the convergence of all this different scientific expertise in a model that's doing science and actually getting better at all of science at once, there's a plausible hypothesis that you end up with an intelligence that's different from those of other frontier labs.

I think the other part of the hypothesis is that they really believe in open-endedness because, if you think about it, science is one of the most open-ended things that could possibly exist. The enterprise of science is what open-endedness is about. It's some kind of tree of discovery that never ends.

So I think that gives it the feeling that the next big things are going to happen here. That was how OpenAI felt to some extent, too, at the time.

Joel, were any other companies from Silicon Valley coming to mind?

I'm not trying to avoid highlighting other companies. There's no doubt that innovation is coming out of Silicon Valley. The problem that we face is that the next big thing is always invisible until it's visible.

I don't know what it is. If OpenAI is a huge phenomenon, I just don't know what it is. If I did, I would tell you.

How do you view the development of AI in 2025? Do you think it's moved really fast, is exciting, or anything else?

To me, it's still moving quickly.

Interestingly, I've been surprised by coding models. They are imperfect and have rough edges, but it feels like that's one of the central practical developments. Then there's this other thing that happens where you suddenly have to debug things, and you don't understand what happened because you didn't write the code. That takes a lot more time to debug than it used to.

So you've got both this huge acceleration and this other deceleration at the same time. The belief is that the bugs are going to reduce over time as these improve, and so it's going to lead to more and more acceleration. But it leaves us at an interesting, delicate inflection point, and it's interesting to see where it goes from here.

So, did you notice DeepSeek's innovation?

I guess it's hard to miss, at least in the sense that there was a dramatic shock to the stock market. It was a super accomplishment, and it obviously represents another step in the evolving race dynamics: what teams, countries, and people are producing the best models in the world. I think China has been on a ramp to catch up with the U.S. on that front.

I also noticed that the attitude of Silicon Valley toward startup companies from China has changed because of DeepSeek's innovation.

When you say “changed,” just to clarify what you mean, do you mean that Silicon Valley is paying more attention to Chinese AI companies?

Yeah, definitely. It's got everybody's attention, for sure—a lot of attention. A lot of dynamics have changed since that moment.

The tech competition between China and the U.S. has also become tense. What's your view on the technological development of the 2 countries? Will China and the U.S. embark on different paths?

It has resulted in a tense situation, and I think it's very unpredictable what will happen. We see international developments where there seem to be events that push the countries apart, but then we see other events where, surprisingly, they may come together. So it's hard to say where this will end up.

There's also a great point in this book: the superstition of management by objectives in society may hinder breakthrough innovation. Historically, China is a country that's really good at devising plans, and we have also gained results from this planning. What do you think about the future development of technology in China? Is there anything you want to remind China of?

The answer is kind of obvious. If China really likes planning, and our book is called Why Greatness Cannot Be Planned, we can remind China that our book is perhaps an interesting counterargument to what is potentially the traditional way of thinking about achievement. I know there's a lot of planning in China.

I don't want to oversimplify, because people in China know far more than I do about how things are there. Joel and I have both acknowledged that we're not against all planning, but in certain areas, especially disruptive innovation, planning can actually be harmful. Your future is a matter of innovation. You're trying to find the right thing to do for yourself; you have to innovate for yourself.

I think that's one of the reasons the book did well in China. Maybe it provides some unique arguments that aren't usually shared widely in China, and that gives people an opportunity to think about the mechanics of innovation and how it works from different perspectives.

I've heard accounts from several friends who've talked about the challenges of education—lots and lots of tests and incredibly high-stakes tests. We have a test culture in the U.S. as well, where there is perhaps a reminder that these tests may not reflect the things we actually care about. Obviously, there's always the question of how to allocate space in really elite educational institutions, and that's always a difficult thing.

I guess the hope would be that there have to be more humane ways to help identify people who have interesting potential. The tests people take may not really reflect much about their potential, but rather their ability to focus on a test, which may not reflect the skills needed to innovate in the real world. I hope this is a message for the U.S. as much as for China: we should ease up on the ways we put children through onerous, unfun testing experiences.

You know what I saw? Ken's painting is just behind him. Do you know what it means?

It's Chinese writing. I think it means “open-endedness.”

Who sent it to you? Is it a gift?

It's a gift. Someone sent it to me as a gift. They said they commissioned a Chinese artist to make it and sent it to me in the mail. I thought it kind of represented my field, so I put it on the wall. I thought it was very nice.

It's related to our interview today. Thank you.

Yeah, it's true. That's funny.

A former Google executive said that by 2027, AI will enter about 15 years of hell mode, during which a large number of white-collar workers will lose their jobs, the middle class will completely disappear, and there will only be the top 0.1% of the wealthy and 99.9% of the poorest. What's your opinion on this warning?

It's very unpredictable. In periods of upheaval, things are complex, and I think predictions are probably not very good. The risks are obviously there: risks to jobs, the social fabric, and the economy, as well as risks to life and well-being. There are all kinds of risks.

There are also huge potential benefits. Scientific advancement can lead to all kinds of radical advances. The destruction of industries is also related to the creation of industries. I think it's ultimately very hard to understand what's about to happen or when it's going to happen, because I don't know how fast this will happen.

Philosophically, what would we want to do if it were the case that many jobs might go away? It does sound a little like science fiction: there is a kind of AGI doing all our work for us. In theory, that should be great. We should just sit back on the beach. In practice, it may be that our social systems are not set up for that.

Okay, glad to hear that. Has your life changed in any way since Why Greatness Cannot Be Planned caught the world's attention? I know that when you left OpenAI, you seemed a little frustrated and lost. Has your frustration improved in the past 3 years?

That's an interesting commentary. I mean, I did seem frustrated and lost; it was true, actually. To some extent, I was really worried about the implications of AI at that time, because I was starting to realize all of these new worldwide implications that I hadn't realized before. I've had several years to think about it, so it's true that after several years of thinking, I'm feeling at least a little more comfortable with understanding what's going on. Those things are making me feel good.

I would say the book had a lot of impact on my life. It's given me the opportunity to meet many people and talk to many people from many more walks of life than someone in my position normally would. I'm just an AI researcher, but I've had the opportunity to talk to such a diversity of people because of this book.

One time, I even had a grandmother call me at my office and ask me if I could meet her grandson to make him less objective. She had heard me on the radio, and things like that were very rewarding to me. I'm talking to you as well; this wouldn't have happened otherwise.

One thing I was reflecting on is that when you work in the world of basic research and write papers, it's really hard to trace from a paper on novelty search to any kind of concrete, positive implications for people. But there's something different about a person actually thanking you for something you wrote in a very concrete way—that they really appreciated it, that it touched them, and that it might have changed their life. It's amazing, I guess, that people care about what we wrote.

In your book, you also give advice to ordinary people, suggesting that people can better pursue their curiosity and achieve their own perfect lives. Could you share how you currently arrange your work and life?

In the pursuit of curiosity, I think I generally do, in my own work, what I believe in: because things are interesting to me, I'll do them well, and then something good will happen, although I'm not sure what. That's pretty compatible with the book. Having written the book makes me more confident in that approach.

My answer is pretty similar. Somehow, I've found that my outside pursuits have slinked their way into my research at one time or another in unexpected ways. I'm attempting to find the stepping stones in my own life toward unexpected treasures.

Looking back now, what are the stepping stones on your life journey?

When I was a kid, my dad got this magazine, 3-2-1 Contact, from the U.S. In the back of it, there were BASIC programs that you could type in, and that's what got me into computers.

There have been several of these kinds of stepping stones. I have to acknowledge that, at a certain point, when I decided to recruit Joel, that obviously led to enormous consequences for me that I would not have been able to predict at all, including sitting in this interview right now.

That's sweet. There's a quote in the book that I like the most: “When you let go of your obsession with goals, you may unexpectedly stumble upon the treasure that can change the world.” Have either of you ever stumbled upon the treasure that can change the world? Could you share a memory related to this?

Yeah, I mean, actually, I was playing with Picbreeder a long time ago.

That was when I suddenly had this epiphany that I had done something really interesting without trying to do it. Many thousands of people have now been influenced by the idea of why greatness cannot be planned, including people across many different countries, which is just crazy.

Yeah, there have been many times in my life when I discovered something that upended my entire life in a way that I never would have anticipated. I guess the easiest one is that I had no idea whether this would be important or not.

I guess there are many things in probably each of our lives that are that way, where, if not for that one moment, your life would have branched off into something else, and there would have been just a different life for me. So that's definitely a treasure to me.

Okay, thank you for your time.

Thank you. Thank you for that. It was a pleasure.

Bye.

Bye.

好了，这期节目就到这里，感谢你的收听，欢迎在评论区留下你的听后感。这期节目也已经在苹果 Podcast、网易云音乐、微博、QQ 音乐、豆瓣等平台同步上传。哦对了，欢迎关注我的公众号、B 站账号和小红书，都与播客同名，我会在这些地方不时更新文字思考、视频访谈、采访随笔等等，我们下期再见。That was the last thing on my mind. And that was the last thing on my mind.Thank you.