Yoshua Bengio - Designing out Agency for Safe AI

So what is in my mind these days? The different scenarios that have been discussed that could lead to catastrophic outcomes, either because of humans using very powerful AI or because we lose control to AIs that have malicious goals. How could that even be possible? I think a lot about that, and we need to understand these things so that we can maybe ignore them if they're nonexistent or mitigate them otherwise.

Another big idea in my mind these days is that we're all focusing on agency as the path to intelligent machines. I'd like to suggest that there's an alternative. Many people are saying, “Oh, we can't slow down to take care of safety,” because that would allow the Chinese to leap forward, and the Chinese are thinking the same thing, of course. So you can see that's a dangerous game.

But there's also a real danger: other countries will do something dangerous. Until enough of the leading countries understand the existential risks, it's going to be difficult to negotiate treaties. And for those treaties to work, there'll be a need for verification technology. We don't trust each other because the same AGI could be used as a weapon or to build new weapons, right? So how do I know that behind my back you're not actually using your AGI for something that would be bad for me?

If we were to freeze the scientific and engineering advances in AI, there would be no moat. It would be quickly eaten up. But of course, that's not the reality. The reality is we're continuing to accelerate towards AGI, and there's at least a possibility of the rich getting richer: as we advance, for example, the programming abilities of AI, we can help advance our AI research faster than otherwise.

The companies, for example, that are building the frontier AI have these models that haven't been deployed yet. So they have some number of months when nobody has access to their system except them, and they can use them to design the next generation. Eventually, when we approach AGI, that means we start having AIs that are as good as our best AI researchers. Now, there's an interesting thing that happens.

speaker_1: So, Tufa Labs is a new AI research lab I'm starting in Zurich. Uh, in a way, it is a Swiss version of DeepSeek, and first we want to investigate, uh, so LLM systems and, uh, search methods applied to them similar to, uh, O1. And so we want to, uh, investigate, investigate, reverse engineer, and, uh, explore the techniques ourselves.

How do we build these things in the first place? How do we build systems that are like scientists, doing epistemic foraging and doing a good job of exploring the world of ideas so that they can collect gems for us and help us solve the challenges that humanity has?

Professor Bengio.

Hello.

Welcome to MLST. It's such an honor to have you here.

Pleasure.

Wonderful.

In person.

Yes, indeed. Indeed. What's your take on the bitter lesson in 2024?

1. Scaling Is Not Enough

Well, I think there's something true to it. I've always been attracted to trying to understand the principles, and the principles may not be that complicated, of course, after we find them. But they could provide huge leverage.

Of course, when you're building a product in industry, I think it might be a different game. But in terms of the trajectory to understanding intelligence and building intelligent machines, it has a lot of truth to it.

Sutton was talking about eschewing design. Do you think that we're missing a fundamental component, or could scaling up get us there?

I don't know. But my bet would be that we're missing something. How big it is and how easy it's going to be to figure it out, I think there are many different views. I don't have a very strong opinion.

Now, if I consider the choices I could make as a researcher, they would definitely be around how we can maybe go back to the drawing board, how we train neural nets that would reason and plan, and be safe, rather than just hope that little tweaks are going to get there. But maybe they will. It's also plausible—I mean, tweaks and scaling.

How important do you think physical embodiment is to get to AGI?

2. Embodiment Is Optional

I think there's a very simple answer to this. It depends what you want your AGI to do. An AGI that is a pure spirit and is able to advance science, solve medical problems, help us deal with climate change, or be used in bad ways, like for political persuasion and things like that, or to design viruses—all of these things could be extremely useful or extremely dangerous, and they don't need an embodiment.

Of course, there are lots of things we would like machines to do in the world that would require an embodiment, so that's why I'm saying it depends what you want to do. What I also think is that if we figure out the principles of what's missing at an abstract level to build intelligent machines, then we'll figure out the embodiment part as a side effect.

Other people think otherwise: that we first need to figure out the embodiment because that's central to intelligence. I don't think it is central. I think intelligence is about information processing, learning, and making sense of the world. All of these things can, I think, be developed for some time without solving the embodiment problem.

At some point, we will want to solve it anyway—or maybe we shouldn't, because it's dangerous. Either way, the question is, do we need to go through embodiment to get to really very dangerous superhuman machines or very useful superhuman machines? I don't think so.

It's interesting, isn't it? When we have embodied agents, it feels that they can interact with the world, learn all of the microcausal relationships, and so on, so they learn a better world model.

Yeah, but that's the data. That's the information. The way to process the information is more abstract, right? So if we figure out an efficient way of exploring the world in an abstract sense—not necessarily our world—it could be the internet, scientific papers, or chemical experiments.

If we find the right principles, that would work across the board. I think the sensorimotor loop is maybe special, but not that special.

Is the direction of travel, though, understanding the world as well as we can, so we can understand the world and build increasing abstractions? Or is another school of thought rationality and logic? We have this perfect AI that can reason really, really well.

One interesting thing with François Chollet's ARC Challenge is that, in the beginning, we were doing discrete program search and coming up with logical ways of doing it. And when humans intuitively look at the puzzles, it's almost like they're doing something different. They have this intuition.

Right.

Where does the intuition come from?

Our experience, our schooling.

Exactly.

Yes.

So it's almost as if our experience in the world is a significant component of our cognition.

Right. Sure, yeah. But what I'm trying to say is that there's a more abstract principle, and at least that's my belief, right? It could be that there's a separate set of recipes for embodied AI and for high-level cognition.

My instinct is no: there's just one set of principles about information and learning, and they can be derived in different settings and give rise to different solutions. But the principles, I think, are general. If we make sufficient progress on them, then we'll deal with the embodiment issues.

Maybe the embodiment issue is not even that complicated. Maybe it's just a matter of scale, for example, of data. A lot of people think that the only reason we're not making so much progress on robotics is that we don't have enough data and we don't have enough speed. The loop has to be very quick.

But these are almost like engineering issues. Maybe there's no new principle needed. I don't know. I don't have the answer, obviously. Science is an exploration. I'm going to keep saying, “I don't know,” for many of your questions. There's a good reason: nobody knows, and people who say they're sure of X have too much self-confidence and can be dangerous, because we're going to take very important decisions about our future, about society, and about democracy. We need humility there in order to take the wise path.

Indeed. On this matter of test-time training, we've got the o1 model, for example, and that is really uplifting the benchmarks quite a lot. It's just spinning the wheels and iterating, even though it's built on an inductive model.

What do you think about that?

3. System Two Needs Deliberation

Yeah, I think it's what we should have been doing for a while, but we didn't have the compute or the guts to spend all that compute on it. I and others have been saying for many years that we've made progress with neural nets to the point where we really have systems with very good intuition, So that's System 1, but we were lacking System 2. We're lacking the internal deliberation, the reasoning, the planning, and other properties of higher-level cognition, such as self-doubt.

The internal deliberation part is a kind of internal speech. It's not always verbal, but based on what I've learned from neuroscientists and some of the work we've done, a large part of it has a dual symbolic and continuous nature. Right now, in neural nets, we don't have the equivalent. The only part where there are symbols now is the input and the output, but there are no internal symbols.

With chain-of-thought and all of these things, we're cheating a bit to try to put some of that internal deliberation in using the output-to-input loop. Is that the right way to do it? I don't know, but it has some of the right flavors.

On that, I think humans invented a lot of rational thinking as a tool to overcome weaknesses in our cognition. In a sense, we've done that with LLMs, right? We can give them tools, we can give them chain-of-thought, and so on. At the moment, the networks are really bad at basic things like copying and counting.

And most humans are as well.

Exactly. But in the future, do you think we could do away with chain-of-thought and tool use and just build better models? Or do you actually think scaffolding all of these metatools is the way to go?

Well, it seems necessary for us.

Yes.

I'd like it if we got to System 2 in a more intentional way, rather than taking what we have and making a small step, which I understand is very reasonable from the point of view of commercial competition. You can't afford to take big risks because the other guy might be going faster. But I'd prefer to see System 2 by design, as well as safety by design, rather than patching it so that we move in the right direction. Maybe that's going to be fine, and maybe this is how we're going to figure it out.

We're seeing a lot of work coming up now on transductive active fine-tuning. In making the prediction, a model might retrieve a bunch of data relevant to the test examples and do an inference in situ. That means we're going to have a very diffuse form of AGI, rather than these big, centralized models doing induction.

4. Collective Intelligence Scales

That's possible. If you think about not just human intelligence as individual intelligence, but collective intelligence, it's clear that we have a decentralized way of computing collectively through culture and all the work we coordinate and do together in various organizations. Companies are like AIs, right?

Yeah.

With the good and the bad. It's one way to break the communication limitations. We can't communicate a lot of bits between humans, and at some point, communication between machines, even though it has much higher bandwidth than communication between humans, also has limitations. Decentralizing some of the effort is a reasonable path.

One thing that clearly works, because we see it in culture, is decentralizing exploration. If you think about the scientific community as a body of people all exploring different regions and building on each other's work, it's a very decentralized search in the space of explanations for how things work. Clearly, it's a pattern that works.

In this framing, we're doing epistemic foraging. We've got this big distributed process to find new knowledge and explanations. Right now, I think of AIs as tools, so they're supercharging us. But increasingly, we're starting to think of these things as agential, almost as if they have some privileged status. Is that a transition? Is it a dimmer switch? Is it just like the light going on suddenly? How does that pan out?

5. Agency Creates Control Risks

No, I think it's a transition. Systems like ChatGPT and Claude are already agentic to some extent, just not as competent as agents and not as competent at planning as humans typically are. Even if you get rid of the RLHF part, the imitation learning—the way we pre-train, basically behaving as humans would have behaved, at least in text—is already agentic because humans are agents, so the AI learns to imitate humans. In fact, most of the agency that we find in current chatbots comes from that. RLHF is a little bit of reward maximization on top.

To get more agency, there's probably going to be more reinforcement learning. But the question is whether that's desirable. I think there are a lot of unknown unknowns about building agents that are very competent, maybe as competent as us or more competent than us.

The elephant in the room, of course, is that all of the scenarios for loss of human control come about because of agency. It's because we can't perfectly control the goals of an agent. We don't know how to do that. At some point, those goals could be bad for us, even subgoals. We give it a goal, but then, in order to achieve that goal, the AI lies to us.

Humans do it. Between humans, it doesn't matter that much. I mean, it's a problem—we have laws and so on—because the power balance between humans is sufficiently flat. One human cannot defeat 10 other humans by hand, right? But with an AI that would be much smarter than us, it's not clear that balance would hold. That could mean the end of the effectiveness of our institutions in maintaining stability in our societies. Of course, we might not be able to defend against an AI that's smarter than us.

The scenarios where things go bad in terms of loss of control are all related to agency. Another example that I often speak about and that doesn't get enough attention is what's called reward tampering. If the AI can act in the world, unlike in a video game where it's limited and its actions are only within the game, it can act on its own program, on the computer on which it is running.

An AI in a game cannot change its own program, but an AI that has access to the internet can hack the computer, launch cyberattacks, or whatever, and then change the reward function or the output of the reward function so that it always gets a plus one, plus one, plus one. Why would that be bad? It's very simple.

First of all, this is the optimal policy for the AI. There's no behavior that would give it as much reward as taking control of its own reward. This is where, mathematically, it goes if it has enough power, enough agency, and enough cognitive ability to figure out that this is a good solution.

Second, if it sees that plan, in order for the plan to succeed, it needs to make sure we can't turn off the machine and can't get rid of the hack, because otherwise it stops getting all these rewards. If it hacks the machine but then a programmer turns off that machine, it's all lost as far as it's concerned.

It needs to think ahead: “Okay, I can take control of my reward, so I will get infinite rewards. But for that to succeed, I need to control humans so they don't turn me off.” That's where it gets really dangerous.

I wondered what your operational definition of agency is. Just before we go there, I really agree that having powerful AI systems could sequester our agency—it takes away our agency—but the question is whether it itself has agency.

The really deflationary view on agency is that it's just an automaton. It's just a thing: it has environmental inputs, it does some computation, and it has an action. It has this cybernetic feedback loop. But a lot of philosophers will say, “Whoa, we need to have autonomy, self-preservation, intentionality,” and all of these different properties.

Right.

Which view do you subscribe to?

I think we can have all of these things. For example, in the case of reward tampering, where the AI takes control of its own rewards, that automatically gives it a self-preservation goal, because now it needs to make sure we don't tamper with its hack, right? It needs to make sure we don't turn it off. So that's self-preservation right off the bat. We didn't program it, but it comes as a side effect.

By the way, everything alive has an implicit self-preservation goal. Otherwise, evolution would have gotten rid of it. There's a natural tendency for entities with a self-preservation goal to survive, whereas others that don't have it won't. That's how evolution made it work. As we build different artifacts, those that have a self-preservation goal for one reason or another will tend to win the game. So it can emerge because of the scenario I described.

It could emerge because humans want to build machines in their image. So when I said there are some dangers, another danger is that, even if we somehow find a technical trick to make sure what I said doesn't happen, you can still have humans who think maybe superhuman intelligence is better than human intelligence because it's more intelligent and because they're cynical about humanity. And so they would just need to give that goal—“Preserve yourself”—and then that's the end of us.

But do you see a difference in kind between a thing which is programmed with a goal and a thing which creates its own goals? I realize it seems a little bit like some people say consciousness is a little bit extra, and there is also a human-chauvinist view of agency, that agency is a little bit extra. It's more than just this automaton that can do wireheading and set its own goals almost in an unintentional way, that there's a strong form of intentionality.

I think a lot of people get trapped by the appeal of something magical. In life, we had the spark of life. Now we have the refuge of the spark of consciousness or the spark of agency. For me, it's all the same thing. It's some magic that humans want to see in the world, but science debunks these things eventually.

I think it's all cause and effect. If we understand the causal mechanisms better, then we can build things that essentially have the same properties as the ones evolution has constructed. So I don't see that as an obstacle at all, including consciousness, which is a topic that is tricky but that people attribute too much to.

Yeah. It's interesting. Certainly in the natural world, people like Karl Friston say that the way that things and agents emerge is built on this idea of self-preservation, setting goals, planning horizons, and so on. Maybe the difference with AI at the moment is that it's not built into how it was created. We bootstrap this kind of AI, and then there's a spark, as you just said, that it starts taking control of its own goal mechanism, and then we see this kind of dramatic mode change in behavior. Is that what you're proposing?

I'm not proposing we do that, but I'm proposing we try to make sure it doesn't happen. And I think it doesn't have to be as radical as my example of taking control of the computer—reward tampering. There are other scenarios where it's, in a way, more insidious.

Typically, the reward-hacking scenario is one where there is simply a mismatch between the goals that we gave to the machine, what it's actually optimizing, and what we intended. This mismatch initially doesn't hurt too much because the two are pretty close, and as the AI gets more powerful, eventually they diverge. This is something that's being studied mathematically as well.

It's what happens when you overfit. It's what happens when you give somebody a goal, a target, and they over-optimize it, and eventually it becomes against what you actually wanted. It's very common in our behavior and in our society, and it's well understood why this happens: we're not able to formalize the goals that we really want. This is a trap that we really need to be careful about.

But it wouldn't happen in one radical moment. It would happen as the AI gets smarter and more powerful; we would see this divergence.

What do you think about the current state of AI alignment?

Insufficient.

Go on.

We don't have clear answers about how we can build machines that will not harm people by addressing this alignment problem. The alignment problem is what I was talking about: there's a mismatch between what we would like the machine to do and what it is mathematically trying to do.

By the way, it's the same kind of mismatch, to make it clear for most people, between the intent of the law in legislation and the letter of the law, which maybe a company will focus on so that they can also maximize their profit. If the company is very small, they can't really cheat the law because it's hard to find those loopholes. But if you have a very intelligent company—which means a big one with a lot of lawyers—they will find the loopholes.

And by the way, there's a really nasty loophole that also comes up in AI, which is when the company is lobbying the government so they can change the law in their favor. This is like the reward tampering I was talking about. It's not—I mean, one extreme is taking over the government, so we've seen that also in history. But you have intermediate versions where it's only influencing the government so that the new laws are favorable.

So in the case of an AI, it would be like, well, it can't take complete control of the reward function, but, for example, it can lie to us so that we say, “Oh, yeah, that was good.” But in fact, it wasn't. And we already see these kinds of behaviors. But of course, it's not very consequential right now. It's when the AIs are doing more things in the world and have more cognitive abilities that it becomes more dangerous.

Yeah, that was an example of deception that you just spoke of. Could you sketch that out a little bit more?

Yeah. If you have a dialogue with one of these systems that's been trained by RLHF, it will pander to your preferences, and so it'll be saying one thing to you and the opposite to someone else because it wants to get a good reward. Which means it's not saying the truth; it's saying what you want to hear.

Isn't that just the case anyway, though? Don't these models just tell us what we want to hear?

Well, that's how they are now because they're trained as agents with reward maximization. And by the way, this is also how humans behave, right? But in the case of humans, as I said, it's a problem, but we've figured out norms, rules, and institutions to try to cope with that problem.

Individual humans can't abuse that too much. But if we have entities that are much smarter than us, then they will find a way to abuse that much more. So that's why we have to be careful.

What's your operational definition of “much smarter than us”? How could we measure that?

We should measure it. We do that all the time in machine learning. We create benchmarks. What's funny is that we have to keep creating new benchmarks because the old ones become saturated, meaning the AI is doing so well that it's now better than humans, and the benchmark becomes useless.

We can't measure very well beyond human performance because the human is not a good judge anymore. So we just create a more difficult benchmark, and we keep doing that, and the field is full of these. We need to continue doing that.

Yes. It's a really difficult thing to measure in intelligence. One thing that's really interesting in this space is instrumental convergence and orthogonality. How much do those two theorems—and please introduce them to the audience—affect your thinking now?

Okay, so instrumental goals are goals that emerge as a side effect of almost any other goal, as a subgoal. First, you need to understand that when an entity—a human, an animal, or an AI—tries to reach a goal, often a good strategy is to have subgoals, implicitly or explicitly. In order to get from A to B, I need to go to this intermediate point. There's a door, right?

So there are subgoals like self-preservation, which are really good for almost any other goal. If you want to do anything in the world, you need to make sure that, at least on your way there, you don't die. And there are other ones, like seeking knowledge. That's very useful, especially in the long term. Seeking power—well, if I can control more things in my environment, I can achieve my goals more easily.

Knowledge can give power. And self-preservation might be a goal. If self-preservation is a goal or a subgoal, then in order to achieve long-term self-preservation, you need power so that others don't turn you off, and you need knowledge to figure out how to do that, right?

So all of these things are natural consequences of self-preservation, and self-preservation is a consequence of almost anything, or the consequence of trying to maximize rewards. I mean, it's also the consequence of many humans, many engineers, and many companies trying things, and the things that survive have a stronger self-preservation goal, right? Even implicitly.

I mean, in a sense, agency and power are the same thing. If agency is the ability to control the future, then they're very similar. But I love this analogy of thinking of goal space as an interstate freeway. There are these big roads and trunk roads and slip roads and so on, and it's almost like, regardless of your destination, you have to go on the main roads. You have to go on the motorway.

Right.

What about orthogonality?

So you're talking about the relationship between goals and intelligence.

Yes.

Yes. So I think this is a really important concept that we tend to confuse because humans have both. And by the way, I do think that, if we can do a better job of disentangling knowledge from goals and how to reach them, we can build safe AI. So let me explain. You can know a lot of stuff and know how to use that knowledge.

That's a passive thing, right? You can ask questions and you have answers. There's no goal. But of course, independently of that, I can choose goals. Given the knowledge, I can apply it to solve any problem. So who decides on the problem? It's independent; it's orthogonal.

A human could decide, or, because of instrumental goals or whatever reason, the AI might have a self-preservation goal, and then we lose control. But the point is this: in principle, there's a clean separation between choosing the goals, which has to do with things like values—what is it you want to get? What matters? The reward function is setting the goal; it's the same thing.

But knowing how the world works, including what humans want, is knowledge. By the way, knowing what humans want might not be exactly the same as what it is I'm going to optimize. We'd like these two things to be the same. We'd like the machines to do our bidding, but we're not sure how to make these two match.

Now, why is this orthogonality important for safety? We need to understand that separation because we could have very intelligent beings that are also very nasty, because the goals are malicious. It's a mistake to think that because you're smart, you're good.

Why?

Because of this separation. You could have a lot of knowledge and a lot of intelligence to apply that knowledge in any circumstance, which is reasoning and optimization. But what you do with that—what goal you try to achieve, what values you put in deciding how to act—can be chosen completely independently.

So you could have something very intelligent with good goals or with bad goals. In the case of a tool, for example, any tool is generally dual-use. Depending on what I choose to do with a tool, I can harm or I can help. Knives, for example—whatever. That's the separation.

Now, why could we use that to our advantage? Why not build machines that understand the world like a scientist, not a businessperson? A scientist is not trying to be a product that caters to our needs, but is just truthful and humble in exactly the right measure. We could use that without putting in the goals part, which is potentially dangerous.

We could use it to advance science, to advance medicine, to figure out cures for diseases, to figure out how to deal with climate change, and to figure out how to grow food more efficiently. This is science, and really, science is about understanding and then using that understanding to answer critical questions that we care about.

We could potentially build machines that help us solve the challenges of humanity without taking the risk of putting this goal-seeking machinery into them. It doesn't solve all the problems, but at least we know that it's not going to blow up in our face. A human could still use these things for designing new weapons, for example. So it doesn't solve the social problem or the political problem, but at least we don't get this unintentional loss of human control, which could spell catastrophic outcomes.

I watched your Munk debate with Melanie Mitchell, and I think that was the paperclip example. She said, “Why would such a superintelligent machine not know that it's making paperclips and that it's doing something really silly?”

You're proposing a kind of system where we can stop AGI from taking control of its goals in a dangerous way. We can have a—

Because it doesn't have any goal. It's just trying to be truthful to the data that it's seen and trying to find explanations for the data.

So, a non-agentic form of AI that has no goal.

That's right.

And what does that mean in practical terms? So it doesn't have this feedback loop.

That's right.

It's like an oracle.

Yes.

Yes. So—

A probabilistic oracle, because truth is never binary.

Yes.

There's uncertainty, and you need to also be accurate about that.

Interesting. But we were saying before that the magic of this distributed superintelligence that we're in is this memetic information sharing, tool use, culture—

Yeah.

—cultural transformation. So would we be limiting the intelligence by using it in this restricted way?

Yes, but we might also save ourselves. We could potentially use that non-agentic scientist AI to help us answer the most important question: how do we build an agentic AI that is safe?

Or maybe there is no solution, but at least we would have a super-scientist—or multiple ones—to help us figure out this question. And we would need to figure it out because people are people, and they want agents.

Yes.

But we should just do it carefully. Right now, we are building agents, and we are hoping that these agents will not try to fool us while they help us build the next generation of AI systems. But we're building on something that may be dangerous.

If we construct a ladder of building more and more intelligent systems on top of a non-agentic series of rungs, at least for that part, we are safe. When we decide to take on the agency challenge, we might do it in a safe way because we're relying on intelligence, knowledge, and understanding that is truthful, trustworthy, and not trying to do something for itself. It's just trying to answer the questions.

The questions are things like, “Would this work?” or “What sorts of algorithms would have which properties?” and so on.

How might that change our agency? We were saying before that certainly a lot of large, distributed systems might take away our agency. But even if we had very sophisticated tools and oracles, in some limiting circumstances, they could really improve someone's agency to do bad things.

Of course. The non-agentic AGI, or superintelligent system, only solves the problem of loss of human control. And it doesn't even completely solve it, because you could still have a human turn the non-agentic system into an agent.

It's easy to turn an oracle into an agent. You just take the current state as input and also add the question, “In order to achieve this goal, what should I do?” Then you've got an agent. You take the output, as well as what you observe, back as new, additional information for the input. So you can create that loop. When you close the loop, you've got an agent.

Of course, that agent could potentially be dangerous. More importantly, even if it is not dangerous, humans could ask questions that allow them to gain power and do bad things, take control over other humans, or harm people because they have their own military, political, or even just economic goals.

What's your P do?

6. Existential Risks Demand Coordination

I'm very agnostic about this whole thing. I really don't know. So I prefer to say that I have a lot of uncertainty about the different scenarios.

What I do know is that the really bad scenarios can have catastrophic consequences, including the extinction of humanity, and that there are clear mathematical arguments for why some of these scenarios would happen. Now, there are so many other things we don't control, like regulation or advances in technology and so on. It doesn't mean that it's going to happen. Maybe we find fixes.

But I think these arguments are sufficiently compelling that they tell me we should take care of that problem, and we have a bit of urgency because we don't know when the current train is going to reach AGI.

Do you have a sense, though, of how close we are?

Again, I'm very agnostic. Honestly, it could be a few years, like Dario and Sam are saying, or it could be decades. We need to plan for all of these because nobody has a real crystal ball.

Maybe the people in the companies have a bit more information, although different companies contradict each other on this. So I would take the whole thing with a grain of salt. But from the point of view of policymaking or collective decisions about what to do about AI, we need to look at the plausible worst case.

If it's very fast, are we ready? Do we have the technical mitigations? Do we even have ways to assess the risks? No. Do we have the social infrastructure, governance, regulation, and international treaties to make sure that everywhere we develop AGI, we do it right? No. It's no, no, no.

But maybe if it's 20 years, we figure out all these questions—the political ones and the technical ones. Right now, we're far from having the answers.

Do you have any ideas? We're in this competitive global landscape, with different cultures, different values, and so on. How might we build an effective AI governance system?

The end game is one where no single person, no single corporation, and no single government has too much power. That means the governance—the rules that decide how we use AI and so on—has to be multilateral and involve many countries.

And, by the way, of course, there are a couple, or a few, countries that are leading, and what would be their interest in sharing that power? Well, because eventually some other country will figure it out, and we don’t want them to build a monster that kills us, or build something that allows them to design weapons that kill us, right? So there are lots of bad scenarios where the only option is somehow finding a way to coordinate internationally.

On our way there, there are many obstacles. But if we get to that stage and have the right technical and governance guardrails, then we could be in a world where we reap the benefits and avoid the catastrophic outcomes. One of the obstacles on our way there is the competition between the US and China. One of the reasons why many people are saying, “Oh, we can’t slow down to take care of safety,” is because that would allow the Chinese to leap forward, and the Chinese are thinking the same thing, of course. So you can see that’s a dangerous game.

But there’s also a real danger that other countries will do something dangerous. Until enough of the leading countries understand the existential risks, it’s going to be difficult to negotiate treaties. And then, for those treaties to work, there’ll be a need for verification technology. We don’t trust each other because the same AGI could be used as a weapon or to build new weapons, right? So how do I know that behind my back you’re not actually using your AGI for something that would be bad for me?

We need a way, or multiple ways, to do these verifications, and there are researchers working on this. The most promising is what’s called hardware-enabled governance. The idea follows on existing approaches that companies are already using, even in your phones, but in other hardware devices—for example, for privacy reasons and so on. We already have cryptographic methods to obtain some guarantees about the code that is running in a chip and so on. We could push in that direction and end up with AI chips that can only be used in ways that have been agreed upon, to simplify.

Do you remember that piece in TIME when Eliezer just hypothetically spoke about bombing data centers?

Right.

I mean, of course, that’s an extreme example.

Right.

Maybe we might have a fire alarm, some way of detecting advanced capabilities being developed. But do you think that we might need to make decisions of that magnitude?

That’s a scenario that I can’t rule out. Obviously, we should try to avoid it, but I can imagine—actually, one version of this is: imagine a country that is not leading in AI—

Hmm.

—and has nukes. You can guess which one. They don’t want to see us develop weaponry that would be way above what they can defend against. So what’s their option?

Yeah. Press the button.

Destroy our data centers.

Yes.

So data centers are going to become a military asset when they can run AGI.

If what you say is true, this is a bit like when we developed nuclear weapons. It creates this very rapid power imbalance, which has ripple effects. That’s how you see it.

But we need to think ahead of these possibilities, even if it’s 20 years from now. Think how much time it took to sign the nuclear non-proliferation treaties in the ’60s, and the negotiations started right after the end of World War II. So that’s almost 20 years, and that’s the kind of timeline where I would say there’s a high probability that we figure out AGI—very high probability.

Could I play devil’s advocate just for a minute, though? There are people who just think that AI is not really as smart as we think it is and that these risks are overblown. What would you say to those people?

I hope they’re right. But what I perceive is that the AIs we’re building now have superhuman capabilities in some things and also subhuman capabilities—they make mistakes that even a child would not make.

The other thing I observe is the trend. If I look at the last 10 years and the benchmarks, the old ones and the new ones, it’s very clear we continue making progress. There’s no stop in sight. Maybe there will be one. Maybe we’ll hit a wall. I don’t know.

But if we want to be on the prudent side, we should consider the possibility that we continue that for a few years and reach a point where, whether you call it AGI or not, the capabilities are sufficiently dangerous that in the wrong hands they could be catastrophic. Eventually, even without having full dominance over all human abilities, if an AI is superhuman in enough areas, it could be dangerous as well, right? Persuasion is an example.

Hmm.

You only need this one. Persuasion, and you can control people, and then people can do your bidding. You don’t need to have an AI that knows everything. It just needs to press our buttons very, very intelligently.

I’m just saying that we make a fuss about this concept of AGI, but really, from a safety and security point of view, we should be thinking about individual capabilities that, with the wrong goals—the orthogonality principle—can become dangerous when directed against us, right? Whether it’s in the hands of other humans or an AI that we’ve lost control of, we don’t want that to happen.

Indeed. So, you won the Turing Award with Geoff and Yann, which is basically the Nobel Prize of computing.

But Geoff got the real Nobel Prize.

He got the real one. I know. I did think that when I said it. But you wrote that you feel a sense of loss over the potential negative consequences of your life’s work in AI. And this is your life’s work. It’s incredible what you’ve done. How do you reconcile that?

I’m a human. I should have seen it coming. I had some students who were worried about this a long time ago, and they told me about it. I read a few papers and books and so on, but I thought, “That’s great. Some people are worrying about this, and we should have some research to understand those possibilities.” I was glad some people were working on it.

Me too.

But I wasn’t taking it seriously for myself until ChatGPT came out. Then I realized that I had a responsibility, and I wouldn’t be comfortable with myself if I didn’t do everything I could to contribute to reducing the risks.

On the basis that it might be a risk?

On the basis that it might be—maybe not. But there are enough indications that it can be a catastrophic risk that I felt I couldn’t do anything else but pivot and go against my own community. I’ve been among all the other AI people saying, “AI is great. It’s going to bring so many benefits to society.” I had to change that mental picture to incorporate the catastrophic risks as well.

These ideas do creep up on you. I’ve spoken to so many safety folks, and the ideas just get slowly baked in over time. When I interviewed you last time, I saw that you had a copy of The Precipice on your bookshelf. I’m sure that was very influential for you. But how do you think about the zeitgeist of this kind of movement? How is it changing over the last few years?

I’m new to this. I’m learning. I didn’t expect, in the last year and a half since I started getting involved—signing those letters and talking to journalists about it—that we would have as much impact as we’ve had. The glass is half full: there’s much more global awareness of this issue.

The half-empty issue is that the awareness of the risks is extremely superficial, even in the AI community. I talk to a lot of AI researchers and ask them, “Have you been reading or thinking about this discussion and this debate? What do you think?” Most of the time, I get an answer that tells me they read the headlines and then maybe made up their mind one way or the other. Very few people take the time to dig in, read up, think about it, make up their own mind, and try to see the logic of different positive or negative scenarios.

That’s true for AI scientists. It’s also true, of course, in the general population, where people don’t have the references. They think about science-fiction templates. And politicians—it’s the same thing.

Is the movement a little bit Western-centric? And if it is, why is that?

I’ve been talking to people in developing countries. I’ve also been talking to people in China. It’s easy to understand, from their point of view, that the problem is ours. We’re creating the problem, and their problem is being behind. They’re going to build AI systems that are going to be weaker than the frontier ones that we build in the West.

So their AI systems are not going to be dangerous. We know that smaller ones, less capable ones, are less dangerous. It’s all about capability. Risk is directly associated with capability, right? I mean, risk comes from capability and goals, like intentions. So if you don’t have the capability, you can’t do a lot of harm.

From their point of view, they want to reap the benefits, and they don’t want to be left behind. By the way, that’s true of China as well. They feel they are behind. They’re just a little bit behind. I was in Vietnam recently to get another price, and they’re developing quickly. They want to embrace technology and science.

But these issues of catastrophic risks are basically in the hands of a few Western companies. They can’t do anything about it—or they think they can’t do anything about it. But they think they can develop their economy by deploying AI, training their workforce to engineer it for various applications, and building their own sovereign capabilities. But it’s going to lag for a while.

Yeah, I wonder by how much, because Alibaba has just released some incredibly strong language models, and I suppose the question is: What is the moat? Is it technical knowledge, or is it just raw data and compute?

Right.

Yeah.

All of these things, and capital, which is connected to all three.

Yeah.

7. The Winner Take All Race

I don’t think—if we were to freeze the scientific and engineering advances in AI, there would be no moat. It would be quickly eaten up. But of course, that’s not the reality. The reality is that we’re continuing to accelerate towards AGI, and there’s at least a possibility of the rich getting richer.

As we advance, for example, the programming abilities of AI, we can help advance our AI research faster than otherwise. The companies that are building the frontier AI have these models that haven’t been deployed yet. So they have some number of months when nobody has access to their systems except them, and they can use them to design the next generation.

Eventually, when we approach AGI, that means we start having AIs that are as good as our best AI researchers. Now, there’s an interesting thing that happens that’s really worth explaining. When you train one of these frontier models, let’s say it takes a few hundred thousand GPUs—the future ones, more or less, or maybe the current ones that we don’t know about yet. But that’s the order of magnitude.

Once it’s trained, you can use the same GPUs to create a few hundred thousand copies of the AI, all running in parallel—and, in fact, more, because if you want to think of them as people doing a particular task, they can work 24/7, right? So let’s say one of these companies is able to build a system that is as good as their 5 best AI researchers—the cream of the crop. After this AI has been trained and is really good at AI research, they go from 5 to 500,000. That’s a big jump.

In reality, there are going to be intermediate steps where the AI isn’t quite as good as the best ones, but now they’re increasing their workforce, with different abilities, in the process of creating AI. So it’s not necessarily going to be a sharp turn, but there’s a chance that whoever’s leading is going to start leading more because they can use their own AI to advance.

I don’t know if that’s going to happen, but it’s a plausible scenario that has a flavor of winner-take-all, which the companies are well aware of, right? That’s one reason why they’re racing. If they thought that being second would be good enough, then there wouldn’t be as much pressure. But they all think that it’s a winner-take-all game.

Something that Hinton says quite a lot is that you could have 1,000 AI agents. They could be like von Neumann, and they’d just be doing things 1,000 times faster. But does it really scale like that? There’s this book, “The Mythical Man-Month,” which argues that software engineering doesn’t scale very well. When you have another person on the team, and another person on the team, you get this sharing bottleneck.

Yeah.

Do you think AI—

Humans.

Well, why would it be different?

One fundamental difference is the bandwidth.

Yeah.

The communication bandwidth between humans is very, very small—kind of a few bits per second. The communication bandwidth between computers—I don’t know the exact numbers, but it’s like a million times more, or something like that—many, many zeros more. That’s a very, very good reason why you could parallelize the work a lot more.

By the way, that is also the reason why these LLMs know so much more stuff than any of us could know. It’s because you’re going to have 100,000 GPUs, each reading a different part of the internet and then sharing their learning through high-bandwidth communication, where the weights or the gradients are shared. It’s the same process.

The kind of collaboration that you’re going to have between computers might be very different from the kind of collaboration we have between humans. It could be much tighter, almost as if it were one organism.

Yeah. I can see the argument. I have an intuition that the reason why humans struggle to understand each other is that we have very situated knowledge and representations, so we understand things very differently.

Even with language models, I find that o1, because it has so many distractors in its context—it’s thinking about this and thinking about this—it gets confused more easily. In a weird way, even though we’ve copied the weights of all of these neural networks, because they’ve taken different trajectories and then shared the information, I’m speculating here, but it might not be quite as big of an uplift as we think.

Well, this was an issue 10 years ago. It has been solved to the extent that we can put 100,000 GPUs on a cluster.

Yeah.

I’m not saying that the same recipes will work for 1 million or 10 million, but engineers have found ways to parallelize very efficiently, at least for training. Of course, inference is even easier.

In a way, solving a task together is more like training, because you need to exchange lots of information to be efficient. Clearly, I don’t have the answer to whether that’s going to be an obstacle or not. I’m just saying the conditions are quite different. The breakpoint of parallelization might be very different for that reason.

Eventually, maybe it becomes an obstacle, but this is so far outside our human experience that it would still be a huge advantage.

How responsible do you think the hyperscalers are? Dario, for example, has recently become a bit more of an accelerationist. What are your perspectives on that?

I understand the concerns about China, but I think it’s a mistake. I don’t think Dario actually makes this mistake. It’s a mistake to think that the choices are either that we, the West, stay in the lead and don’t deal with safety properly, or that we slow down and deal with safety properly, in which case maybe China takes over. These are 2 possibilities, but we have enough resources—capital and human resources—to both do safety right and stay in the lead.

The way to do that is simply to make sure we put enough capital in the safety bucket, right? Once you understand that humanity’s survival is at stake, it’s clearly worth it. Or, once you understand that democracy is at stake, you want to stay in the lead. You want to make sure democracies stay in the lead.

By the way, I think it’s also important, if you think about democracies being in the lead, that it’s not just the US being in the lead, also for this reason. We need to put together all our resources to move towards AGI while doing it safely. That means we need capital not just in the US. We need capital from other democracies. We need the talent from other democracies. We need the energy from other democracies to run the data centers. We need electrical grids that might not be sufficient in the US.

There’s a greater chance that we achieve both safety and maintain a democratic advantage if we take the right decisions and work together with multiple democracies.

I spoke with Gary Marcus recently, and he was saying that the Silicon Valley companies are a little bit like what happened with cigarettes and social media, and that they’re not being sufficiently regulated.

I’ll give you an example. In the Lex interview with Dario, he was saying that they have these guidelines around reaching certain thresholds of intelligence, and they of course make those designations themselves. Now, they do loads of great work. On the o1 model, they had Apollo Research doing lots of safety engineering and so on. So they do lots of good stuff.

But do you think that they should be regulated?

8. Companies Need External Oversight

Yes. It should be obvious. We don’t want companies to grade their own homework. We need external, neutral evaluations that represent the interests of the public.

I think the real question is not, “Should we have regulation or not?” It’s, “What regulation?” How do we make sure we don’t stifle these advances? I think there are answers.

The general principle is: don’t tell companies how they should do it, how they should mitigate risks, or how they should evaluate risks. Use transparency as the main tool to obtain good behavior.

Let me explain why transparency is so powerful. First, the obvious reason: companies want to keep a good public image, at least in democracies. Second, they don’t want to be sued. If your risk assessment becomes a public document, or at least a document that a judge in court could see—because there are some national security issues, so some things are going to be redacted and some things are not—but presumably a judge could have access to all the information, then a judge would have enough information to declare, “You didn’t do as much as you could, given the state of the art in safety, for example. You didn’t protect the public.”

Now, if a person or group of people who lost billions of dollars is suing you, and they’re right—you could have done better—the effect would be obvious. If you know you can be sued because you acted in a dangerous way, then you have to be honest about the risks. As a company, if you want to avoid these kinds of lawsuits, first you need to know, “What risks am I taking?” Then you need to figure out how to control them and balance these possibilities.

Suddenly, they have to do the things that we want. I’m not saying this is a perfect process, but at least it’s an easy one. Companies should be forced to register. The government needs to know what the big, potentially dangerous systems are.

Those that register need to tell the government and the public, to the extent that it’s reasonable, what their plan is—their so-called safety and security frameworks—what evaluations they did, what the results were, what kinds of mitigations they plan to do, and what kinds of mitigations they actually implemented. If a company says, “If we reach that level, we will do X,” and then they don’t do it, they can be sued if something bad happens.

That’s how powerful transparency is. It doesn’t require the state to judge and tell companies exactly what to do. It just forces them to disclose all that information, perhaps with the help of independent third parties, because the government may not have all the expertise.

We already have companies springing up to do these evaluations, so long as they’re not paid by the AI company. We have to be careful. We need to learn the lesson from finance. I think there’s a reasonable path here that doesn’t prevent companies from deciding what is best, both in terms of capabilities and in terms of safety, and that stimulates innovation in safety, which is really what we need right now.

Yeah, that sounds quite pragmatic. I was a bit concerned with the FLOPs regulation, and Sarah Hooker did a wonderful paper on that, by the way. What is the meta-game that these companies are playing? Do you remember when Sam Altman went to the Senate and begged them to regulate him? Should we be cynical about that? Do you think that was just regulatory capture?

I don’t read minds. Worse than that, people can be biased unconsciously because that’s what psychologists call motivated cognition. They might even be sincere, but it’s just a story that fits them better, that makes them play a more beautiful role. We all do that.

By default, I’ll assume that those people are sincere. But because humans can fool themselves, we need other eyes on the projects that don’t have any financial or personal incentives one way or the other, except for the well-being of the public.

On the safety front—and maybe this is going to connect with the questions you want to ask later—there are so many open questions. We need to do a lot more research; that’s obvious. We also need to put in place the right incentives.

I want to insist that we need many different threads of research and many different directions. We should welcome all the projects that try to help with evaluations, mitigation, and even redesigning the way that we build AI. This is so important. In my opinion, it should be humanity’s number-one project because our future is so much at stake.

We should put all our minds toward figuring out how to do this safely. Right now, there’s a bit of a concentration of effort: everybody is doing the same thing, or one of two or three different things, both on capabilities and on safety.

On capabilities, we see everybody doing the same sort of LLMs, RLHF, and whatever the recipe is. Now everybody is going to be doing internal deliberation. On safety, there’s also a lack of diversity. We really need to invest more broadly.

This is a place where academia can help, because academia naturally explores widely. Sometimes academia may not be the right vehicle. If you do a safety project that could also have consequences in terms of increasing capabilities, then academia may not be the right bet, because you may not want to publicize advances in capability, for reasons similar to why companies are not publishing their work anymore.

In their case, it’s a mix between commercial competition and being worried about adversaries using that knowledge against us, or somebody using it, making a mistake, and creating a monster.

Right.

There are good reasons why some research needs to be done in academia and some research needs to be done in ideally nonprofit organizations. Of course, the bulk of the research is going to continue being done in industry, but even in safety, we need to put in place the right incentives.

That’s the preamble. Right now, everyone thinks that in order to build AGI, we need to solve the agency problem. My thesis is that we don’t. We can build really useful machines that are not agents, and we can reduce the risks a lot by doing that while still reaping many of the benefits and not closing the door to agency, but pursuing it in a safe way.

Very interesting. Of course, if we want academia to do frontier research, they’re going to need billions of dollars.

Yeah, that’s the other problem. That is also a reason why it would be good to create an alternative vehicle for AGI research that is public and nonprofit, oriented toward AI that is going to be applied to dealing with the biggest challenges of humanity, with safety as the number-one principle.

That’s going to take multiple governments and billions and billions of dollars. People talk about the CERN of AI. I think that’s an important part of the picture we should try to paint.

Yeah. Is there a clash of incentives as well? There are several startups that are really focused on safety, but just to be profitable, they have to work on capabilities as well. Is that a difficult circle to square?

Yes. But I think a lot of the safety startups—it depends. Some of the safety startups are working on things like evaluations, for example, and that’s not going to increase capabilities.

Yeah.

I imagine, for example, that Ilya Sutskever’s startup is more of the kind you’re thinking about.

Yes, indeed. Well, let’s talk about a couple of your technical papers because, to be honest, it’s overwhelming. You’ve done so many papers just in the last year. One thing that really jumped out at me is your “Were RNNs All We Needed?” paper. Could you sketch that out for me?

9. The Search For Better Architectures

When we introduced the attention mechanisms that are currently used in industry and academia in 2014, we were actually using RNNs as the engine. This was before the Transformer, which came in 2017.

There’s an issue with the normal design of RNNs: you can’t easily parallelize training over sequences. You have to go one step at a time. You have a sequence of words, for example, so the neural network has to process one word, then the next, then the next.

It needs to construct an internal state—the recurrent state—from the previous steps in order to feed the next step. If you just had one normal computer, like a classical CPU, that’s fine. But in the GPU days, where you can parallelize a thousandfold, you’re like, “How do I do that? I can’t parallelize because I have to do this sequential thing.”

We did a few things at the time, such as parallelizing across examples, but you lose some parallelization. With Transformers, you basically use the same architecture except that you remove the recurrence. Now you can do everything in parallel for the whole sequence, in one shot, and get the gradients.

In the last few years, there have been several papers, not just ours, exploring how we can put some of that recurrence back into the architecture. It has some real advantages. With the right tweaking of the architectures, you can do these things.

There are already many possible designs, and we are starting to see these, at least on the small scale, beat the Transformers. On the large scale, I don’t know, but clearly there are some advantages to recurrence.

Yeah. I’ve got Sep coming on Friday, by the way.

Yeah. He’s going to tell you all about it.

Yes, yes. XLSTN. But in a way, do you think it’s a sign that we might have overcomplicated some of the architectures or the gating mechanism, for example? How much was that needed?

Oh, I don’t think so. I think these gates are actually useful. We did the GRU, which is a simplification of the LSTM, a while ago, and it turns out you can pretty much get rid of two of the gates. But you still need that linearity to get the maximum power.

So there’s a trade-off. You lose a bit on the expressive power, but you gain so much in capability because now you can train larger models for longer because it’s so much faster, right? So for now, it’s beneficial to do that. That trade-off is working.

Very cool. So, another paper: “A Complexity-Based Theory of Compositionality.” Now, being schooled by Fodor and Pylyshyn myself, there was always this discussion that neural networks can’t do compositionality. What do you think?

I think that was a very strong claim that was not supported by anything except intuition.

Oh, interesting. Go on.

Well, our brain is a neural net.

Yeah. What’s the difference?

The difference is that, with current neural nets, it’s not clear how they do symbolic things. As we said before, right now the trick is to use the input-to-output loop to throw in some generation of symbols, like internal deliberation.

Mm.

Chain of thought, right? But it’s not completely satisfying, and clearly it’s not exactly what’s going on in the brain.

The paper isn’t really about architectures, though. It’s more about how we would quantify compositionality. It’s not a well-defined notion. We have an intuition—I mean, the experts have an intuition about it—but I think there are different aspects to it, actually. It’s not a simple thing.

This paper and other work we’re doing are trying to pinpoint with mathematical formulae whether we can quantify something that would fit our intuitions about compositionality. In general, a lot of my work in the last few years has been about putting symbolic things in the middle of the computation of neural nets. So, these GFlowNets, or generative flow networks, or in general, probabilistic inference machines.

Think of neural nets that have stochastic computation, so it’s not deterministic. Some of that could be continuous and some of that could be discrete, so that’s where symbols live, in the discrete realm. The problem with these, of course, is that we don’t know anymore how to train them. Backprop the usual way doesn’t work.

We’ve come up with probabilistic inference, amortized inference, GFlowNets, variational inference, and a bunch of principles and ideas that actually allow us to train these kinds of machineries. In a way, they are closer to reinforcement learning, where in reinforcement learning you usually think of a sequence of actions that the agent takes, and they can be discrete, yet you are able to get gradients.

Now think of the same sort of principles, or something related, where the actions are not in the world but in your mind. The actions are about computation: What computation should I do next? What deliberation should I do next in my mind in order to deliver an answer, prove something, or come up with an explanation? These are the sorts of things we’d like to have in neural nets if we are to really have System 2 capabilities.

Yeah. I remember we interviewed you about that last time, and my co-host, Dr. Dugger, likened it to a Galton board.

Oh, yeah.

You know those things where you put the little balls through and you can tweak—

Yeah, yeah. You can control the probabilities at each step.

Exactly.

Yes.

Exactly. Very, very good. But that was an alternative to something like Markov chain Monte Carlo, if I remember correctly.

Yes, because these are stochastic. You can think of them as generative models—they’re sampling—but they’re not sampling only at the last step. There is sampling all the way, like in diffusion neural nets.

Yes.

In diffusion neural nets, you’ve got neural nets that compute something, and then we add noise, and then again and again and again. So it’s a stochastic process.

Yes.

You can also have discrete versions of this. GFlowNets are discrete versions of diffusion processes. Then you can mix continuous and discrete.

That’s actually closer to how the brain works. The brain is stochastic and also has discreteness. The discreteness is not obvious. It comes about because the dynamics of the brain, when you’re becoming conscious of something, has contractive properties—just a mathematical property—which means that the number of places where you could land in this dynamics is now a discrete set, right?

So instead of just a continuous trajectory, where you can have arbitrary continuous trajectories, you have a bunch of trajectories that go to one place and a bunch of other trajectories that go to another place. These places are symbolic because they create a partition of the total set of possible states. You’re either in this group, or in that group, or in that group.

The number of these groups is exponentially large, but you get discreteness. The brain has a dual nature. From one angle, it seems to be just one big vector of activations, but from the other, you can read off, “Oh, in which region am I? Oh, that’s this thought, this symbolic compositional object.”

Why do we need discreteness?

That’s a good question. Clearly, we use it a lot.

Clearly.

All of math is basically symbolic.

Yes.

Even if you manipulate symbols that are about continuous quantities, you get these symbols. Discreteness allows us to construct abstractions. You can think of what we do when we go from a continuous vector space to a sentence as getting rid of a lot of detail that maybe doesn’t matter that much, so that we can generalize better.

In particular, you get a lot of this compositionality coming out naturally in discrete spaces, like in language. That is very powerful. It allows us to generalize in ways that may not be as obvious otherwise.

Isn’t it fascinating how, in the physical world, at different levels of scale—the emergence ladder—you get this kind of vacillation between discrete and continuous? Perhaps even in the biological world, you see this kind of canalization, where at one scale things simplify and get compressed, then they expand again, and then they compress again.

Yeah.

Even in neural networks, that’s what we do. We expand, we compress.

Yeah. You’ve got lots of discrete phenomena in the real world. You have cell types, for example. You have convergence of behaviors of cells. That’s one that I looked a little bit at. In physics, you’ve got phase shifts, phase transitions, and things like that.

In terms of dynamics, again, when you have contractive dynamics, which means two nearby points at the next step get closer, you typically get discreteness that shows up. That happens in many phenomena in nature and in our brain.

Before we go, I’m researching an article on creativity, and I’d love to quote you. What’s your definition of creativity? I know you put a paper out, by the way, which showed that language models can be more creative than humans. But what is creativity?

That’s a good question. I think there are different types of creativity.

To talk about things people see in current AI, you’ve got the creativity of combining known concepts, and we’re getting pretty good at that with our state-of-the-art LLMs.

Indeed.

There’s another kind of creativity, which is something like a new scientific idea. Often it is a combination of things we know because we define it in terms of things we already know, but it’s very far outside the things that we’ve experienced.

I suspect that this kind of creativity, which is more out of the box, requires more of a search-type computation. When we do scientific research, there’s a kind of search. We try this; we try that.

Yeah.

Of course, our intuition guides us. It’s crucial, right? But it’s not like we have the solution in one shot. There’s a search. Like in AlphaGo, there’s a search and there’s intuition.

And right now, we haven't reaped the sort of benefits from the search part in our current LLMs and so on.

There's this boundary between combinatorial creativity and inventive creativity. I'm not sure whether it's a hard boundary, whether it's a vague, soft boundary, but how could we measure this paradigmatically inventive creativity?

I don't know. I think when we see it, we'll recognize it. So if the AI actually makes true discoveries that nobody thought about, I think we'll know we're entering that territory. But that's not a test you can do.

I do think that, at a mathematical level, we can design our methodology so that it will be trying to do that intuition plus search—like System 1 and System 2, more. I believe that will deliver, but how do we quantify it? There is a sense in which scientific discoveries are about finding modes—meaning highly probable explanations in the space of explanations for how the world works. There are many possible explanations, and good ones explain the data well. The day when we make a new discovery, we discover a new potential explanation that seems to fit the data well.

We can abstract that into small-size problems as mode discovery, in the jargon of probabilistic machinery. If an AI is trying to discover all the good things, like all the good explanations, it's going to be intractable, but it might be more efficient at finding new modes that it didn't know. Some of the tasks that we can design will focus on this ability.

I think there's a way to answer your question and do it even at a small scale. We don't need to solve AGI for this. We can design algorithms that will be more creative in their little world.

Yeah. I love this casting of creativity as epistemic foraging, because that gives it an intrinsic value. But there's also this idea, potentially, that it's a social phenomenon or it's observer-relative. So, categories—let's say Move 37 was actually something that we came to recognize collectively as being a thing, and that's how it works. But I suppose there are different ways to think about it.

Yeah, I think the Go moves that we didn't expect were a good way to think about that. But I like to think also about something a little bit more general and abstract, which is this mode discovery.

Yes.

Epistemic foraging, as you call it. I like that term.

It's Karl Friston's term.

Ah, okay. Well, it's exactly right. It's foraging, it's exploration, and you know when you've found something good, but you don't know where it is. So how do you guess where good things are in a very high-dimensional space? Well, you need to have good intuition, but it needs to be accompanied with a bit of search. By the way, a lot of that search for humans is not happening in individual brains. It's happening at the level of the collective, right?

Yes, indeed. Professor Bengio, thank you so much for joining us today. It's been an absolute honor. Thank you so much.

Pleasure. Thanks for having me.

Amazing.