Google 研究员展示生命“从代码中涌现” [Blaise Agüera y Arcas]

The new book—it’s called What Is Intelligence? Thank you for asking. It was just published by MIT Press about 3 weeks ago, so it’s very fresh off the presses. There’s an online version of it as well that is free and very rich, full of all kinds of rich media.

Chapter 1 of that book is called “What Is Life?” “What Is Life?” is sort of the single to the album, and works as a book in its own right. That book is also for sale from MIT Press. I think I’ve explained why I think of life as being a subset of intelligence, and why the story of artificial life and abiogenesis is relevant to the story of intelligence and what it is.

The subtitle of the book is Lessons from AI About Evolution, Minds, and something something. It’s basically documenting the time since about 2020, when I got really shocked by seeing that these large sequence models seemed to be generally intelligent, and starting to think through the implications of that. What would it mean if we believe our eyes and that is what intelligence is? What does that tell us about ourselves and about the properties of intelligence more broadly? That’s the whole intellectual journey that has taken us on over the last few years.

MLST is supported by Cyber Fund. So, I’m Falm. I’m the co-founder and CEO of Prolific, and Prolific is a human data infrastructure company. So, we make it easy for people developing frontier AI models and running research to get access to trustworthy, high-quality participants for high-quality online data collection.

I am at Google. I’ve been there for about 12 years now, and I am their CTO of Technology and Society and also the founder of a new research group—well, newish; we’ve been around for a couple of years—called Paradigms of Intelligence, or PI. It’s much smaller than the previous organization that I ran at Google Research. It’s about 50 people, so enough to do some real damage.

The idea is to really focus on the fundamentals of artificial intelligence and go beyond exploiting the current models and paradigms that are working well. We believe in those, but we also think that we have to refill the bucket with new insights and new ideas as well.

I’ve just watched your talk, and you said that life and intelligence are the same thing. They are both computational. What do you mean by that?

This is a surprising claim. I know it sounds a bit odd, but what I mean by that is—well, let’s begin with life and why life is computational.

In the middle of the 20th century, John von Neumann, who was one of the founders of computer science, realized that in order for a robot paddling around on a pond to make another robot out of loose Legos that it finds floating around in the pond, just like itself, it needs to have instructions inside itself. Let’s suppose the robot is made out of Legos. Its job is to make another robot out of those loose Legos.

He imagined a tape with instructions for how to assemble a machine. The robot would also have to have a machine inside itself that would be able to walk along that tape and follow those instructions to take the loose Legos and put them together into its own form. It would also have to have a tape copier so that it could endow the offspring with that tape. The tape would have to include the instructions for the copier and the assembler—the universal constructor, as he called it.

The cool thing is that he made all of those predictions, if you like, on the basis of pure theory, before Watson and Crick and their unacknowledged collaborators had figured out the structure and function of DNA, before we knew how ribosomes worked—which are, in fact, exactly that universal constructor—and before we had discovered DNA polymerase, which is the tape copier.

All of those things have to exist in order for an organism not only to be able to reproduce itself, but to do so heritably, such that if you make a change in its genome, in what’s on the tape, then the offspring will also have that change. The kicker is that this universal constructor is a universal Turing machine. In other words, you have to have a computer inside yourself as a cell in order to make another cell. DNA is, in this very literal sense, a Turing tape.

It’s a very profound insight because, basically, he’s saying you cannot be a living organism without literally being a computer—a universal computer.

Very interesting. So, are you saying that DNA is basically a computer program?

DNA is a computer program, yes.

Very cool. Many folks in the audience would have been inspired by Conway’s Game of Life, for example, and you were talking about computational equivalence. The Game of Life, of course, is Turing-complete because it has expandable memory. Just as DNA has expandable memory, the grid size could just keep growing.

As you’re pointing to, when we see these weakly emergent behaviors, they appear very lifelike. Does that in any way downplay the biochemical and thermodynamic realities in the physical world?

Of course. A cellular automaton like the Game of Life is a lot less complex than the real world. It’s in 2 dimensions rather than 3, and there’s no thermal randomness, which turns out to be very important, actually. The fact that the computation is deterministic is also a little different from real life, where those thermal fluctuations mean that there’s always a probabilistic element to things.

You do have to extend Turing’s original ideas about computation to make a so-called stochastic Turing machine in order to really do a proper job. What von Neumann was getting at—and I’m glad you bring up cellular automata—is that they’re really a generalization of the Turing machine to the laws of physics.

Essentially, every pixel on your game board, if you like, is performing a computation. It has a state, and it’s performing some very simple computation to say what the next state is on the basis of its neighbors. The idea is that the rules determining the next state of a particular pixel are the laws of physics of that universe.

The reason von Neumann came up with this idea of cellular automata is because he wanted a system that would allow one to do computation, but in which the computation is embodied. What I mean by that is, in a Turing machine, there’s a tape and there’s a head, and then there are the symbols that are written on the tape. But the symbols are not the same stuff as the tape and the head and the instruction, or the table of rules. Those things are abstract and separate from the symbols that are written.

Whereas in a cellular automaton, the machine can literally print itself. It’s not just a laptop, if you like, but a laptop and a 3D printer in one that can print another laptop. Embodied computation is computation where the memory is written and read in atoms rather than in bits, and therefore the machine can make another of itself.

David Krakauer said to me that—we can agree that intelligence is in adaptivity, inference, and representation. Adaptivity is very important. DNA is adaptive, but it’s very slow. He was saying that the nervous system, the brain, and culture are evolution at light speed because they allow us to overcome the information-transfer bottleneck between successive generations.

Does it make sense to think of the program of intelligence at the DNA level when so much of the adaptivity seems to be happening higher up?

A lot of the adaptivity very much happens higher up. In humans, we have cultural evolution, which goes, as David says, at light speed relative to genetic evolution.

When I say that DNA is a Turing tape and that ribosomes are universal computers that construct life, that’s really only the ground level. Or maybe it’s Level 1 or Level 2. There’s physics underneath that, but there’s a Level 3, a Level 4, a Level 5, and so on. There are computers built out of computers built out of computers.

The thing about it is that once you have that ground level, then you can build as many floors above that as you like. The point is that the moment you have life—meaning that you have something that can build a copy of itself—you have a general computer, which allows you to do anything. What that means is that life, from the very beginning, is computational and can start to compute in parallel.

This gets us into symbiogenesis, which I guess we’ll cover in a bit. The fact that that ground floor is computational answers the question of why brains are computational. It’s because cells were computational from long before there were action potentials and other fast electrical processes allowing us to think.

Can you speak to this notion of recursion that you were just pointing to? Carl Friston, for example, thinks about this division in systems called a Markov blanket, like a statistical independence. We seem to observe empirically that complex, intelligent adaptive systems have nesting.

You were just speaking about having levels upon levels upon levels. What does that buy you? Is it a kind of recursion? How does that improve the sophistication of the intelligence?

Yeah.

Well, 2 things happen. One of them is things inside things inside things, and the other one is parallelism—in other words, a lot of things at the same level happening at once. They’re both important. They’re both important parts of the story.

First of all, when you have a cellular automaton like Von Neumann was imagining, that’s already massively parallel computation, because every pixel is like a little computer doing its thing. In the same way, in physical space, there can be molecules in a lot of spots, all of which are doing something. You can think of them as computational operations, perhaps, and they’re all happening at once.

In your body, you have quintillions of ribosomes, and all of those ribosomes are little, tiny universal computers working in all of your cells at once. All of your cells are working at once. But there is also nesting, because you are a person. Of course, you’re already a part of a society, which in some sense is an intelligence bigger than an individual human.

You’re made out of cells. Those cells are made out of organelles. Those organelles are made out of proteins. Those proteins are made out of molecules. That sort of nestedness is really important as well. You’re not only a lot of computers working together in parallel, but you’re also a system of computers made of computers.

One thing I find fascinating that you’ve thought a lot about, Blaise, is where the purpose comes from. Folks like Harrison, for example, have a no-nonsense physics interpretation that it’s the second law of thermodynamics at the end of the day, because there’s some notion of valence: we build these complex adaptive systems, and there needs to be something that drives them forward, something that propels them in a certain direction. Your experiments have shown that this kind of falls out of computation. What do you mean by that?

Yes. To be clear, computation and the second law very much work together here. The experiment that I did a couple of years ago that really got me started with artificial life is called BFF. It’s based on a language called Brainfuck, which is where the first B and F come from. I didn’t name it that, although I admit I enjoyed that it was called that.

This is a minimal Turing-complete language designed by a grad student. I think he was a grad student in physics, Urban Müller, in the ’90s. It’s a very minimal language. It has only 8 instructions. I only use 7 of them.

The basic setup is that I begin with a bunch of tapes of length 64, just 64-byte-long tapes, like those Turing tapes or von Neumann’s tapes. They start off filled with random bytes. There are only 7 instructions, so the great majority of those bytes—about 31 out of 32 of them—are no-ops, meaning that they don’t code for any instruction at all. They start off random and very much purposeless.

You have 1,000 of them in your soup. The procedure is really simple: it’s just plucking 2 of those tapes at random out of the soup and sticking them end to end. You make 1 tape that is 128 long, and then run it. This modification of Brainfuck is self-modifying, meaning that when you run it, it can modify values on that combined tape. Then you pull the tapes back apart and drop them back in the soup. That’s it. You just repeat that process.

If you do that a few million times, you start off with nothing much going on. Again, the huge majority of those bytes are not even instructions. There are only an average of 2 or so on each tape, so the likelihood of them doing anything is almost zero. Once in a while, you might see 1 byte somewhere change, but after a few million interactions, something apparently magical happens, which is that suddenly the entropy of the soup drops dramatically.

It goes from being incompressible because it’s all random bytes to being very highly compressible, and programs emerge on those tapes. Those programs are complex. They take some real effort to reverse-engineer, and you can see that they’re occurring in a lot of copies. That’s why it’s compressible.

The fact that they’re occurring in a lot of copies tells you what the programs are doing. They’re reproducing. They’re copying themselves. What’s so cool about this experiment is that it really shows you how life emerges from nothing. The emergence of life is, in some sense, the emergence of purpose.

What is the purpose of one of these programs? It is to reproduce. If you were to mess with one of those bytes, if you were to change it, you would in most cases break the program. When you break the program, it no longer functions to reproduce. Something that can break is something that is functional or that has purpose.

Absolutely fascinating. You said there was a phase change that was quite sudden. Would David Krakauer acknowledge that as being a form of emergence?

I think so. I’ve actually never asked David that question. We disagree on a lot of things AI-related, but I think he would acknowledge that this is a phase change and that it is an example of emergence.

I think he would, because he has a bunch of criteria, but one of them is a fundamental coarse-graining and reorganization of the microsubstrate such that the new phenomena can be described with a simple new variable. This seems to match that description.

Is it possible that there’s some kind of design bias? When we design machine-learning architectures, there’s so much information in the architecture, and in this case there’s so much information in the Brainfuck language and the terms and so on. Could that have influenced it to emerge in a certain way?

Yes. The structure of those programs does change depending on the language. We’ve tried this with other languages. We’ve tried it with Z80 assembly language, which is the assembly language of these Zilog chips that were invented sometime in the ’70s and just got discontinued last year—a very long-running microprocessor architecture.

The phenomenon is very generic. What those programs look like is shaped by the specifics of the language. But the reason those programs emerge, the reason that they develop purpose, is actually thermodynamic.

That might seem puzzling, because you would think thermodynamics is about things becoming more random, and apparently the exact opposite is happening here. You start with randomness and you get order. How could that be?

I think the answer was well characterized by a chemist—an organic chemist—Adi Pross at Ben-Gurion University of the Negev in Israel, who is now emeritus. He did a lot of work on so-called dynamic kinetic stability. The idea is that it’s an extension of the second law that says things seek their most stable state, their most stable form.

Usually, we think about those stabilities as only being fixed points, but those stabilities can be cycles too. If something dynamically makes itself, if something forms more copies of itself, that’s more stable than something that just settles.

It’s like the old joke about DNA being the stablest molecule in the universe. Obviously, DNA is fragile, but at the same time, if the DNA makes more DNA, then it will be around a long time after granite, which can only erode.

In terms of this valence question, though, does that imply to you that there is a natural drive to survive, almost? For these systems to maintain their existence, assuming that’s a primary force, they would need to have a degree of sophistication. They would need to be doing modeling; they would need to be doing sophisticated things. But is that something that just always happens? Is it a convergent property?

Yes, it is. In that sense, evolution is the second law at work. If you have a bunch of things that are not copying themselves in the BFF soup, and you have something that emerges that can copy itself, then that thing that can copy itself will write over the things that can’t copy themselves, which means it’s more fit—or more stable, if you like.

That is written into the laws of statistics in just the same way that the second law is. It’s just the kinetic, or cyclic, form of that same law rather than the steady state.

You said in your talk that merging is more important than mutation. Tell me more.

Yes. The usual thing—what we learned in school—was that Darwinian evolution consists of mutation and selection, or what Jacques Monod, the Nobel winner, called “chance and necessity.”

In other words, mutations, maybe from cosmic rays or whatever, to our DNA, sort of throw spaghetti at the wall. Whatever sticks is what remains—whatever doesn’t kill us and whatever hopefully makes us stronger.

That was my assumption as well. Starting out with these BFF experiments, I had a mutation rate where a byte could change at random with probability 1 in 10,000 or something with every interaction. Then I began playing with the mutation rate and found that this emergence of these complex programs occurred even when the mutation rate was turned down to 0.

Which is really a surprising finding. It tells you that this emergence of purpose comes about even without any random changes in the code. It’s not explainable in purely Darwinian terms.

The other things that are not explainable in purely Darwinian terms are the emergence of life in the first place. This greatly puzzled Darwin. He thought this problem of abiogenesis, or the emergence of life, was impossible to reckon with. You might as well talk about the origin of matter, is how he put it in one of his letters.

The other thing I can’t explain is the increases in complexity that occur. Why is life now more complex than bacterial life 1 billion years after it began on Earth? Why do we have human societies now? If we go back 100 million years, we had only things with much simpler brains. We had octopuses—they had pretty complex brains—but the tendency has been toward greater complexity.

There are some people who have argued against that. Famously, Stephen Jay Gould has said things like, “Everything on Earth is the same amount evolved. We’ve all been evolving for 3 billion years. It’s all equally evolved.” I think Gould was wrong when he said this. The reason being symbiogenesis: when a eukaryote is formed by a mitochondrion finding itself inside an archaeon and then becoming a eukaryote, that resulting composite organism is more complex than either of the 2 parts that made it up.

It’s the same way that a spear is more complex than a stick and a stone point. You put 2 things together, and now you have something more complex than the parts. If this idea that symbiosis, or symbiogenesis, is an essential part of evolution is correct, then you absolutely get more sophisticated things coming about later in evolution because they’re being put together from preexisting parts.

Yeah, I wanted to touch on the importance of the merge operator. We were talking about that earlier, and even Chomsky spoke about this. You could argue whether the merge operator in language evolution was the Prometheus moment, whether it was phylogenetic or ontogenetic, because you were just talking about symbiosis and merging in a physical substrate.

But it also happens in the information substrate. You get these memetic computer programs that ensconce themselves, and maybe language was that. I don’t know, but I have a theory about why merge is so important as opposed to random selection. I think creativity is about grounding. It’s about path dependence, basically.

Even the retroviruses and all of these things form a lineage. I think that if you don’t use merge, you lose the lineage. Also, something about the recursive merge operation allows you to build more complex computer programs by allowing for this kind of reuse and canalization. There’s something very natural about that.

Yeah, I completely buy everything that you’re saying. I think that’s exactly right, except that I dislike Chomsky. So, I think he’s wrong. He’s wrong about language. I’m much more of a fan of Dan Everett. I don’t know if you’re familiar with his work with the Pirahã. It’s wonderful.

He spent a long time with the Pirahã in Brazil, who are a people whose language does not obey Chomsky’s requirements for language. They don’t have recursion. They don’t have anything like center embedding. They also don’t have numbers, and they don’t have past and future tenses.

Everett wrote a great book some years ago called Don’t Sleep, There Are Snakes, which talks both about his experiences among the Pirahã and their language, and also his big fight with Chomsky over this. Chomsky’s papers are filled with theory and pseudomathematics, and have no time to give to ethnography or to actually studying any real languages. But anyway, I’m digressing.

Putting Chomsky aside, though, what you’re saying about merge—or, as I would see it, composition, functional composition—I think is absolutely fundamental. It’s how all technology is built. W. Brian Arthur has written about this and how technology evolves. Every technological invention gets invented a dozen times around the same time, as if everybody’s in telepathic communication.

The reason is that every technology has precursors. You can’t get a light bulb until you know how to blow glass, how to make a vacuum, how to draw a filament, and how to generate electric current. When all those things were there and the need for light was there, the light bulb was going to get invented. But it was invented a dozen times by different inventors with different contingent choices.

They might choose which kind of filament to use, whether it’s prongs or whether you screw in the light bulb, which way you screw it in, what the diameter is, and so on. Those decisions, as they get locked in, determine the course of everything after that which incorporates light bulbs.

So, in a way, this contingency—these choices about exactly which way those combinations go—is actually what the entire genome, or whatever it is, is made out of. In the case of BFF, the original replicators are really just single instructions that sometimes, randomly and weakly, might copy themselves. One byte moves here and there, but as those bytes get copied around, sometimes a couple of them end up together, and then they’ll copy as a group.

They’ll do better together. The contingent thing— which way they ended up getting copied, whether it was AB or BA, that they stuck together— is the information that the bigger thing is made out of.

That little extra bit, because in this case you just had single bytes, was not information to begin with. So the merger tree ends up being exactly the information that is encoded in the final genome. It’s all about the history.

Yes, absolutely fascinating. In a sense, I’m surprised you’re not a fan of Chomsky, because he was talking about automata and Turing completeness. He was the ultimate computationalist, and in a sense, what you’re describing is Chomsky’s ideas just applied lower down the stack.

That’s right. In that sense, I think he was correct, but I also think all of those ideas were already there in von Neumann in the 1950s. Even Nils Aall Barricelli, the first artificial-life researcher, worked on one of von Neumann’s machines. I think he sort of snagged time on the MANIAC to do some of his first artificial-life experiments.

They’re kind of pseudodocumented in Benjamin Labatut’s book MANIAC. It was really fun. Or no, that was in his first book, I think, When We Cease to Understand the World. But anyway, my point is that those ideas were there before Chomsky.

The thing that Chomsky really pushed, during his reign of terror over linguistics—sorry, I’m being a little bit mean—was the movement in artificial intelligence that we now call GOFAI, or good old-fashioned AI. It held that you could formalize what AI is as grammars and programs, which turned out to be wrong. That turned out to be a false start in AI, and it’s why there were so many AI winters.

There seems to be a bit of a tension, because the GOFAI folks had some very interesting ideas. I mean, I’m a big fan of Fodor and Pylyshyn, for example, and they spoke about strong compositionality. We have semantics and intentionality, and it’s possible to build these cognitive representations, but we have the issue that we can’t really design them to represent the world in a high-fidelity way, and we have semantic divergence.

Then you’re pointing to this very interesting constructive thing, and I think a constructive form of AI and compositionality solves a lot of problems because of this path-dependence problem and this canalization that we’re talking about. When you build intelligence brick by brick, you can build artifacts of incredible sophistication, but unfortunately, we can’t design the artifacts to do exactly what we want. We can gently steer them in a certain direction.

Even with Friston, I feel that even though he’s talking about the what of intelligence as prediction and adaptivity, I think the implementation matters. I think adaptivity means structure learning. I think there’s something about having a substrate which actually does this form of composition that you’re talking about that seems to be a mechanistically necessary condition for intelligence, right?

Yes. I think in many ways what we’re talking about is the tension between analog and digital ways of thinking, or bottom-up and top-down ways of thinking.

For instance, let’s talk about how you would recognize a bicycle. In the good old-fashioned AI world, you would say, “Well, you’ve got a circle detector and a line detector that will detect the lines that make up the frame of the bike,” and so on. You’ll handwrite code for all of those things.

Of course, the problem is that there are many ways of looking at a bike where you’re not going to see the wheels at once, or maybe the bike is of a weird design. There are those funny bikes that have shoes instead of wheels, and when you look at one of those in a Gestalt sort of way, you recognize a bike immediately, even if all of the rules are broken, as it were.

That’s really important, because when you’re looking as an intelligent being at the world, you have to cluster. You have to find regularities in the world whose shapes are not well defined by a set of rules. They’re not just carved up by hyperplanes; they’re blobby.

Intelligence requires methods that are very neural-net-like, that look more like continuous function approximators. That’s why gradient descent is a good idea, for instance, and learning these things via smooth functions is a good idea. Trying to encode them with rules never worked out well.

On the other hand, DNA is discrete, right? There are 4 symbols, and you order them in a certain way, and that’s it. It doesn’t mean that there’s no randomness in the way proteins are folded and so on, but composition at the level of DNA really does have to do with chopping up programs essentially made of discrete symbols, inserting bits of code, and so on.

When you’re looking from the bottom up, it’s a very, very quantized world. But when you start to look at giant, complex things like us from a high level, you have to begin from a more continuous perspective.

I think you’ve hinted that there are natural convergent patterns in computation. Can we sort of get a convex hull of your philosophy?

We could try. I hesitate to say I’m an anything-ist, but functionalist comes closest.

Functionalist.

Yeah. The reason for that is that in the old days, in the 19th century, we used to think that to be alive meant that there was some vital spirit or vital force that living things have and dead things don’t. As we started to figure out that the laws of chemistry were the same for living things and dead things, and that urea can be synthesized in a test tube and so on, those ideas really went out of fashion, and we moved into a very strict materialist kind of perspective.

Right, or everything is just physics. I mean, I was trained as a physicist. I believe in physics fully, but I also think that there is more to life, in the sense that if everything is just physics, then you have no way of saying what it means for you or me to be alive. To understand what it means to be alive, I think you have to come to grips with the idea of purpose. You have to bring teleology back into the equation.

What I mean by that is that a kidney is not just a collection of atoms. It’s an organ that performs a function, right? The function is to filter urea. If you implant an artificial kidney that works on totally different principles but also filters urea, it’s an artificial kidney. It’s still meaningful to say that.

That means there is something about the word “kidney” that means something which goes beyond the matter that the kidney is made out of. Conversely, if I come back from the future and show you an object, and you’re like, “What is that?” and I tell you it’s an artificial kidney, there’s nothing about this set of weird carbon nanotubes and so on inside that would say to you, “Kidney.” It’s just that if you happen to implant it in a body and sew it in the right way, then all of those relationships would show up in the right way for your body to persist.

This idea of things serving functions for other things, and functions only having meaning in the context of yet other functions, is ecological. I think this is really central. A rock on an inanimate planet has no function. If I break it in half, I now have 2 rocks. But a living thing has a function.

The hallmark of function is multiple realizability, just like Turing talked about for Turing machines. If you have a need to make ATP for energy inside your cells, you’re going to have multiple pathways for doing it, because sometimes the aerobic way works and sometimes you need the anaerobic way. Whenever you start to have multiple pathways—wings in insects, wings in bats—there is a function in play.

The alternative position would be essentialism. Folks like Anil Seth and John Searle think that certain types of material have a certain type of causal graph. For example, brains might give rise to consciousness, and if we simulated a brain, it wouldn’t have the same causal graph; therefore, it would be different.

But I would like to—we’ll just park that for the moment. It seems a little bit like you’re talking about this as a computer software architecture diagram. It’s like that Ship of Theseus type of thing, where we can swap things out and ask whether it’s still the same thing.

But I think path dependence is very important. The kidney evolved; it has this rich phylogeny of evolution. When you replace it with something that came from a different substrate, which has a different provenance, then it’s almost like it is a kidney now and it works now, but it breaks the ecology.

Like, imagine in an ecology if I swapped a plant out with an artificial plant and kept doing that. It might work now, but doesn't that affect its future trajectory?

Yes, it does. But that's exactly what symbiogenesis is all about.

Often, you will have a repurposing of something that was designed, if you like, by nature. One of the cool things about the BFF experiment is that it shows you how intelligent design can happen without any intelligent designer. Something that was designed for one purpose, or to serve one function, can come back around and serve another function. That brings a whole different contingent history with it.

The RSV example that I gave—the ability to fuse cell membranes together—came from a virus whose original purpose had nothing to do with building placentas, but it gets incorporated and repurposed. This is the kind of bricolage that life is made out of.

I think that kind of replacement, parallel pathing, and so on doesn't just happen when we make artificial kidneys. It's happening all the time in nature, and is the very hallmark of life. So, yes, I disagree strongly with Anil Seth and with John Searle on this point.

The brain-prosthesis experiments that you've alluded to—the idea that if you took an emulator or a simulator of a neuron and plugged it into your brain so that its inputs and outputs were connected to the other neurons, then the other neurons wouldn't know the difference. What if you did that for half of your neurons, or for all of them? Would your consciousness get dialed down even if you behaved the same way? Of course not.

For me, your consciousness is obviously a function of the relationships of all of those things with each other. It doesn't mean that it's as simple as a computer program where you can just substitute one subroutine for another. We've made computers very abstract in that way, but biology is wet and messy. The interfaces are complex and hard.

This same idea of multiple realizability and repurposability is the very stuff of life.

What is your position on consciousness? What is it? What's its purpose? Is it epiphenomenal? Can it be measured?

Yes, great question. I think that the idea of philosophical zombies, which David Chalmers has talked about—the idea that maybe something could behave just like you or me but be dead on the inside, not have any experiences, not feel anything—is actually a lot less coherent than it sounds.

I'm a functionalist about consciousness, too, and what I mean by that is twofold. One is that I don't think consciousness is some kind of epiphenomenon that we just happen to have for reasons that have nothing to do with our behavior. Nor do I think that it is somehow tied to anything about the way we're physically made. I think it is functional.

Why do we have it? In my team, Paradigms of Intelligence, we've been doing a lot of work over the last year on multi-agent reinforcement learning. The reason is that we're very interested in the precondition for symbiogenesis, which is symbiosis: cooperation.

When 2 things, or 700 things, or whatever, start to cooperate closely, that's the beginning of them really fusing together and becoming 1 thing. In order for 2 intelligent agents to cooperate, it turns out they have to have a theory of mind. They have to model each other and be able to put themselves in the place of the other.

We have a whole long theory called MUPAI about how that all works, but the CliffsNotes version of it is that it requires you to do induction over a universe that includes not only the game that we're playing, but also what is happening in your head and what is happening in my head.

In other words, you have to have a universe that includes yourself and the other, and that allows you to generalize over the class of you and me. I know that my internal state is happy when I smile, and when I see you smile, I know that you're happy on the inside as well. I can make that inference in the same way that if I see a bunch of peaches, I know that they're all the same object and I know what the backside of one will look like, and so on.

This ability to do psychological induction is really important for cooperation, and that's why we have it. One of the consequences is that we model ourselves, and we model our own models of others' models of our models, and so on. There's a kind of strange loop, as Douglas Hofstadter would have called it.

Yes, I love Douglas Hofstadter. So there's this kind of self-modeling, and then second-order self-modeling and third-order self-modeling, which could be applied to other agents. Of course, in the real world, we are computationally bounded. We can't make sense of all of the complexity. So when we do this modeling of other agents, our modeling is quite cartoonish and quite structured.

And it only goes up to sixth order as well, at most.

Oh, interesting. How does this affect—we haven't really spoken about agency yet—your ideas of purposeful behavior? Presumably, you could have a strong agent that's just doing something quite trivial. But when we have this collective intelligence and this information synchrony between agents, how does that affect your ideas of purposeful behavior?

I sometimes use the example of rowing to describe what's happening when purposes merge into a single purpose and consciousnesses merge into a single consciousness. There's this term that I learned from Daniel James Brown's book The Boys in the Boat: “swing.” That's when the 6 oarsmen—or 8 oarsmen, sorry—all achieve this kind of state where they're in perfect sync with each other.

You know it when you experience it. The boat acquires a soul, as it were. You all feel like you're pulling as 1. Boats with that property go a lot faster than boats where people haven't quite achieved that sync.

That, I think, is kind of what happens when we think of ourselves as being a self, despite the fact that our brain actually consists of a lot of parts. In the same way that the oarsmen, in some sense, began with their own purposes, their own self-models, and their own models of the other parts of the brain, through this process of subjective symbiogenesis, I guess you could call it, all of those wills become 1 and all of those selves become 1 self.

In hiring, for example, you want folks with high agency, but you also want alignment, which is the potential for this kind of synchrony. We often do a thought experiment on MLST where you can look at a boat or a flotilla of boats, and you're trying to draw a boundary. The boundary for the agent should be the minimal description. It should be where most of the agency is, where most of the planning and future modeling is happening.

Usually, it's the pilot; it's the driver of the boat. But you're talking about the situation where there is such synchrony and alignment between the agencies that almost the best intentional stance, if you like, is to draw a boundary around all of them.

I also think that there's not necessarily a single right answer. In my book, What Is Intelligence?, I talk about a few interesting cases. One of them is, for instance, the conjoined twins Abby and Brittany Hensel. I don't know if you've seen them on YouTube or on TV shows. Fascinating case.

These are 2 people who share 1 pair of arms and 1 pair of legs. Each of them controls 1 arm and 1 leg, so they're in a 2-legged race. They often speak in synchrony. They play volleyball and sports, they drive a car, and they can write emails with no problem.

They also sometimes have differences of opinion. They'll come together and apart in a remarkably fluid way, and all of that is done purely with behavioral cross-cueing, as Mike Gazzaniga would call it. Their nervous systems are separate: separate brains and separate spinal cords.

In that case, they're able to model each other extremely well because their entire lives they've been right next to each other. Another interesting case would be split-brain patients of the kind that Gazzaniga spent a lot of his career studying. Those are cases where, in adulthood, the brain is essentially cut in half.

Each hemisphere can only see the left or the right visual field, and controls 1 arm and 1 leg. The most fascinating thing about these split-brain experiments is that, from the outside point of view, it is obvious that there are 2 consciousnesses in there.

Each hemisphere is conscious of different things. You can make disjunctions between what shows up in the left and right hemispheres, and the left and right hands can be drawing different things, and so on. But if you talk to somebody who's a split-brain patient, they're always like, “Yeah, I'm still one person.” They will never admit that there are 2 people in there.

So is there somebody who is right and somebody who is wrong? No. This is entirely relational. It's a relational description. And the fact that, for them, they're the same person they always were, just occasionally something takes a little more work. Occasionally one hand will be buttoning the shirt while the other hand is unbuttoning it. It's just an inconvenience.

There are split-brain experiments as well, even just with a normal brain. And I can believe that we are sort of separately conscious in different parts of our brain. You get out of bed in the morning, and you must be a slightly different person. But we kind of gloss over that, don't we?

Absolutely. We make a narrative. The best, coolest experiments about this, I think, are the ones from Petter Johansson at the University of Lund. He's done a bunch; he was the one who discovered choice blindness. In these experiments, a subject is—I think the very first one was face choice blindness—shown 2 faces on cards and asked which one is more attractive, and you pick. Every so often, the one you're handed to explain why you thought that face was more attractive is the one you didn't pick.

So there's a kind of sleight-of-hand trick. The cool thing is that very few people notice that they're being handed the wrong face. There is no difference in the fluency or the latency of the description. You have an inner lawyer ready to spring up and justify whatever choice you made, even if it's not the choice you made. That narrative that you invent then influences your future choices.

It's as if we all make up a story about ourselves. And, of course, the reason is that we're all split-brain patients in a way. The left-hemisphere interpreter that generates the speech is likely not the same part of the brain that actually did the choosing, if you know what I mean, and yet all of those parts of your brain are invested in the idea that they're all in the same boat, that it's all one me. So they're all covering for each other.

In the same way, in a split-brain patient, if you show the non-left-brain-interpreter hemisphere “Stand up,” the person stands up, and you ask them, “Why did you stand up?” They'll say, “Oh, I was thirsty. I'm going to the kitchen for a drink of water.”

It's the same thing with artificial intelligence. It's becoming more sophisticated, and there's the social question. I suppose, actually, you can think of it as a Ship of Theseus for society. We're going to have agents embedded in society, and we're going to form a large collective intelligence. Do you worry about that future? I mean, what do you predict is going to happen?

Well, there are certainly things that I worry about. I don't want to come across as a Pollyanna. I'm worried about polarization. I'm worried about disinformation. I'm worried about our political and economic systems not necessarily being fit for purpose in the world that we'll all be living in in 20 years. But I'm certainly not concerned about a lot of the kinds of things that I hear Eliezer Yudkowsky talking about, for instance.

One of the reasons that I feel very differently is because I feel like human intelligence, in the usual sense that we think of it, is already a collective phenomenon. We're not that smart individually. We're not that much better individually than our primate cousins. It's only because we get together in large societies of millions and billions of people that we can do these amazing things, that we can transplant organs and go to space, and so on. Individually, we're just not all that.

So for me, AI is actually a part of human intelligence. It's literally already the same thing. I find it very interesting that we only achieved general AI when we began to literally train the models on reams and reams and reams of human language. So AI was human intelligence from the start.

I suppose the thesis of Eliezer is that it's possible to have artifacts which are dramatically more intelligent than we are. Maybe you think there's some kind of a limit, but do you think, in principle, that we could build artifacts which are significantly more intelligent?

Well, I think that collective humanity is already vastly more intelligent than individual humans. In that sense, and in many cases, it operates at very different time scales. For instance, I think these things are already true.

In a sense, our biggest difference is about thinking of it as an other versus already as a part of ourselves. What do we even mean by human? There was a wonderful paper from 2006 called “The Science of Cycology.” I'm not remembering her name, but she is a psychologist, and “The Science of Cycology” is spelled C-Y-C-L-O-G-Y. She asks people to draw bicycles.

First, they say, “Do you know how a bicycle works?” Everybody says, “Yeah, of course I know how a bicycle works.” “Okay, draw one.” Nobody can draw it, even if it's just looking at a sketch of a bicycle and saying, “Okay, where does the chain go?” or “Where are the pedals?” Most people don't know. They make some very fundamental error in this. It's a very funny paper, but the point is that we all have these illusions about what our own knowledge is, what our own capabilities are, and what our own intelligence is.

We already have this thing, in the sense that we identify what we think of as our intelligence with something that is actually in a bunch of other people and a bunch of other stuff around us. We do that kind of unconsciously. So for me, there's not really a discontinuity between what's already going on and AI. It's really just more of that.

Interesting. I think they would make the argument that you could build a single artifact which is more intelligent than the totality of humans. But just parking that to one side, I spoke with Judith Fan. She's a wonderful professor at Stanford, and she's done studies on drawing, comparing how humans draw to computers using CLIP models and stuff like that.

She found something fascinating. Because we have quite an abstract understanding, when we make sketches, she was grading it on the progression—progression 1, progression 2, progression 3—and we start very coarse and very abstract, while AI systems start with the edges and the details. And that, to me, indicates that AI models today don't really understand things at a very deep, abstract level like we do, perhaps because we have this compositional synthesis of knowledge that we were alluding to earlier. Do you see that as a gap?

There are a few questions, I guess, hidden in there. One of them is: Do I think of LLMs, for instance—of today's frontier models—as being less than or different from, in some basic way, our brains? What are those gaps?

First of all, they're obviously very different. Their architectures are different. They're trained in a very different way. The remarkable thing for me is actually how convergent a lot of their properties are with those of brains, despite all of that.

The fact that you find internal representations in many of them that surprisingly resemble ones you can measure in human brains—these Brain-Score-type measures from Martin Schrimpf and colleagues—or that sensory modalities in humans can be reproduced remarkably well even by models trained on pure language is really remarkable. It speaks to how much is encoded in language, how much of what is encoded in language is a reflection of the architectural properties of our brains and umwelts, and how much of that is then reconstructed, essentially, by those models.

Now, the question of what we draw first when we draw a picture and how that all works—I mean, remember that image-synthesis models like CLIP or what have you are working in pixel space to begin with. Diffusion models, by the way, work very differently from various other kinds of models. We now know that you can drive a robot with a transformer.

If you give one of those robots a paintbrush—or a pen—and you say, “Now draw,” what it will draw is going to be very different from what you get from a diffusion model that starts filling in pixels. And, for that matter, all of that is different from what happens in your own head when you're visualizing something. So I think a lot of this is not so straightforward to analyze because of all the differences in the way that AI and representation space work.

I do think that today’s models are highly compositional. Even with many of those original image-synthesis models, the fact that you could say, “A teddy bear at the bottom of the sea playing with the Speak & Spell,” or whatever, and it’ll do it tells you that they can compose. Again, are there capabilities like ours? No. There are definitely places where they’re better, places where they’re worse, and places where they have surprising gaps. So it’s different, but I wouldn’t say that there’s a fundamental lack of composition there at all.

I think, if anything, the biggest gap between transformer-based models and what we do is actually narrative memory, or being able to form long-term memories and, in that way, have a kind of persistence of a self over long periods of time. They don’t have that yet.

I’m conflicted. You are pointing to this universal representation hypothesis. I think Chris Olah popularized it with some of his visualization experiments, and it’s true: the representations are very convergent. Other things lead me to believe that the models produce these superficial impostors—that they give you exactly the right answer but for the wrong reasons. One of the hints of that is when you do variations on the input: it’s not robust.

There’s the Turing machine argument as well. These LLMs are finite-state automata, but they can access tools that are Turing-complete. So perhaps we could say the system is Turing-complete, but I don’t believe that ChatGPT is effectively searching the space of Turing machine algorithms. It hasn’t been trained to do that, but it is surprisingly robust with the ARC challenge. It can do really well, especially if you do some evolution, some refinement, and so on. So it feels like we’re knocking on the door, but there’s something missing.

I think that in many of those cases, we’re not doing a fair human comparison. This is a little bit similar to our illusions about knowing how bicycles work and so on. I hear a lot of people say things like, “Look at this case where we just flip the logic: we change it from do to don’t, and then it gets it wrong 30% more often,” and so on. My first question is always, “Have we done the human baseline?” It turns out that, surprisingly often, the human baseline shows the same property.

This doesn’t mean that humans are incapable of doing the fully robust, fully general version of these things. If you’re a logician, or if you think about it carefully, you can really write down your premises and be super robust to flipping the “nots” in the way something is formulated. But most of us don’t operate that way most of the time. We’re highly susceptible to logical illusions, cognitive illusions, et cetera, which turn out, in many cases, to be surprisingly similar to the machine case. So I’m kind of unmoved by a lot of those. I think often we’re being a little sloppy about how we do it.

It’s certainly the case that transformers aren’t searching systematically over all possible Turing machines. We don’t know how to do that. You have to take shortcuts of various kinds in order to make that whole problem of induction over programs computationally tractable, whether you’re a brain or a transformer.

Blaise, thank you so much for joining us today. It’s been an honor.

Thank you. Thank you for the really thoughtful questions.