大语言模型与涌现：复杂系统视角（David C. Krakauer 教授）

The brain is an organ, like a muscle. If I outsource all of my thinking to something or someone else, it will atrophy just as your muscles do. There’s nothing confusing about that. It’s just a fact of physiology.

So I’m David Krakauer, and I work on the evolution of intelligence and stupidity on planet Earth. Science is a humanistic endeavor. The purpose of science in the universe is to make the universe intelligible to us, not to control it, not to predict it, and not to exploit it. Science is no different from poetry: we’re trying to make sense of the world, trying to give it meaning in relation to our own existence. Superintelligence is only interesting to the extent that it makes me more intelligent, not to the extent that it makes me more stupid, more servile, or more dependent. Oh, good.

Tim Scarfe

Oh, has the camera just started?

That was actually-

Tim Scarfe

We’re saved.

That was sympathy. That was sympathy.

Tim Scarfe

Amazing.

1. Intelligence Does More With Less

Well, I think the first paper of yours I read was a couple of years ago with Melanie. We interviewed Melanie about the debate over understanding in language models, I think it was called. At the time, there was this fervor, this hype around the “Sparks of Artificial General Intelligence” paper, for example. They had early access to GPT-4 without RLHF, and they were saying, “Isn’t it amazing that we have these emergent capabilities?”

Yes. Yes. No comment.

There’s just this whole question of what emergence is and what intelligence is, and we can talk about it all. My particular interest is the evolution of intelligence, which I consider to be extraordinarily varied. I consider bacteria intelligent, as you probably know. One way of framing a lot of this is that, for me, intelligence manifests most clearly when you can do a lot with very little in terms of input.

I’m less and less impressed when you manifest so-called intelligent behavior when you have more and more information at your disposal. Unfortunately, the way that AI has evolved is in the direction of confusing being very knowledgeable with being very intelligent, and I think that, in some sense, encapsulates my critique.

You said in your talk yesterday that all of the reasonable definitions of intelligence, if anything, try to marginalize away the contribution of knowledge. It’s about adapting to novelty—adaptivity in general. Your evolutionary perspective is very interesting. You were pointing to this yesterday when you were talking about preserving and accumulating information, and how preserving that history is very important, not only phylogenetically but also ontogenetically.

Yeah.

Can you explain what you mean by that?

Yeah. It’s a really interesting question. One of the big confusions in the intelligence world is whether we’re allowed to call ingenious adaptations intelligent. That is, are capabilities and adaptations intelligent, or is that just what evolution gives you? There are people out there who would say, “No, that’s not intelligent,” because intelligence isn’t a capability; it’s the ability to acquire capability. It’s the capacity to acquire capacity, as Woodrow puts it, and François Chollet and others like that have adopted that perspective. So the question is, where does that come from?

You can actually derive this mathematically. In the 1970s, a very prominent theoretical chemist who won the Nobel Prize for work on fast chemical reactions, Manfred Eigen, started developing a series of theories that were finally formalized in the 1980s, called quasispecies theory. To cut a very long story short, what that theory gives us is a fundamental bound on the rate at which information can be acquired in any evolutionary process.

It turns out that the fundamental speed limit is established by the generation time. It’s essentially 1 bit per selective death. That even has a name: it’s called Muller principle. The way to think about it is that you have lots of variants, each of which has a different hypothesis about the world. Selection kills all the ones that have the wrong hypothesis and keeps the one that has the right one.

It turns out that you can only maintain 1 bit, because you don’t know which bit would be responsible if it were many. So it’s 1 bit per genome per generation. Now, if you’re a large multicellular organism, all of the adaptive information exists at frequencies that are higher than the generational frequency. What do you do? You have to build a system that’s extragenomic.

Yes.

We call them epigenomes, or we call them brains, and these are systems that can acquire high-frequency information that goes beyond the selective dynamic. That’s the basic idea. You can actually make a qualitative distinction, and that boundary is called the error threshold, where you have to acquire an inferential organ mechanism to extract high-frequency information from the world. For some people, that’s intelligent. I don’t share that view because it’s somewhat arbitrary, but it’s principled.

2. Culture Breaks Evolutionary Limits

You said yesterday that, of course, the nervous system is an example of one of these things, but you said culture is evolution at light speed.

Yeah.

What do you mean by that?

Well, I don’t know how technical we can get here.

It’s very technical.

Okay, we can get technical.

Yeah.

One way to think about this is to imagine a genome as being a point in a configuration space. It’s called sequence space. As you evolve, you move around in this space. You can think of it as a graph where adjacent nodes are connected through mutation, which tends to be local, though not always.

The Eigen theory tells us how much information can be preserved at a certain rate of mutation, or what the fastest speed is at which you can move adaptively in sequence space without losing everything you’ve acquired about the past. This is the error threshold. If you’re in a position to take the information that you’ve acquired in the past and store it—refrigerate it, quite literally, put it in a library, put it in a book—

Mm.

Now you can move on that graph much more quickly because you’re not corrupting the information that you’ve accumulated up to that point. Culture actually has no upper bound because, as long as I store—save to my hard drive, save to my library—the information that I’ve acquired up to now, I can produce variants at any rate. As long as one of those is good, I add it to my library.

So culture actually breaks evolutionary light speed and is a qualitatively different process of evolution from organic evolution because of that.

David, you’re at pains to distinguish capabilities from intelligence, so why is culture or a library a form of storing—I don’t know whether you would call it intelligence?

Well, again, I don’t think so. I wouldn’t. I make a big distinction between knowledge and intelligence, and we talked about this: smart people don’t need a lot of knowledge to solve a problem. Not-so-smart people do, and that’s our experience in life.

You go to a friend who’s swotted up on all of the solutions to the problems, and you ask them, and they answer. You’re not as impressed as you are by that friend of yours who spent the last week at a pub or on a hike, not in the library. Those are the people who impress us more, and I think we’ve lost track of that, actually. When it comes to intelligence, less is more, not more is more.

Yes, indeed. Yesterday, you said that intelligence is doing more with less, and you actually said stupidity is doing less with more, which was very interesting.

Well, no, again. Let me give you the background to that. At SFI, we’re very interested in this idea of emergence, and Phil Anderson, in 1972, wrote a very famous paper called “More Is Different.” Phil was reacting to high-energy physics.

High-energy physics deals with very small things, where the symmetric laws of physics produce a corresponding symmetry in the configuration of states in the physical system. But as you make things larger—and he gives the example of NH₃ and PH₃, ammonia and phosphine—

The underlying laws of physics and protein folding are symmetric, but when you have a very large structure, it gets stuck in a potential well, and so you break the symmetry. It doesn't matter if the law is symmetric. It's like saying I have a ball: it rolls down a hill but rolls into a very deep valley. It gets stuck, and there's an energy funnel.

It doesn't matter that the law is symmetric, because now the only thing that's going to tell you what state you observe is the initial condition—where you started in the landscape. His point—this is called a broken symmetry—and Phil's point was, as you get larger and larger, in this particular instance, in terms of the atomic mass of the molecule, you're more and more likely to have to use an additional parameter to tell you where you are.

The consequence of that is that if you're going to have a theory now, which isn't just a description—a list of initial conditions—in other words, you have to take averages, you have to do coarse-graining.

Yes.

And that's the foundational principle of complex systems: in order to go from a very nonparsimonious microscopic description to a parsimonious macroscopic one, you have to take averages in a clever way. And that's the essence, by the way, of emergence.

3. Scaling Laws Do Not Prove Emergence

Yes, indeed. And we should talk about your paper, recently released, “Large Language Models and Emergence: A Complex Systems Perspective.” As you were just alluding to, emergence is “More Is Different.” In a minute, we'll talk about scaling, criticality, compression, novel bases, and generalization as being the principles of emergence.

Yes.

But just rewinding a tiny bit, many folks in the LLM literature, famously Jason Wei—I was speaking with Daniel Hendricks last night, and he told me that he said it first, even before Jason. They have this rather cartoonish version of emergence, which is something along the lines of: when there is a sharp discontinuity in capabilities, we can say that... And they gave examples, didn't they, of three-digit multiplication and so on.

Because you have these scaling laws, and then you see these divergent appearances of capabilities. You're having none of that, David.

None of that. Well, the joke there—there are a few things to say. Exactly. You just put it very well, and you don't need to repeat it. But this three-digit addition goes from being—I can't remember the percentages, right? But let's say they're under 50% when you've got 100 billion parameters, and they go to 80% when you've got 175 billion or something like that.

And you think, “Well, okay, I can do three-digit addition very effectively on an HP-35 calculator with a 1K ROM.” But that's an order of a billion times smaller in memory footprint. So you think, “Okay.” You can engineer a solution into a tiny memory footprint very efficiently, using approximations that we're familiar with.

Or you have to train it with all the data in the world at considerable expense. It does lots of other things in addition to three-digit addition, right? That's the interesting point about three-digit addition, and you call it emergent. I would simply call that really shit programming.

In other words, I think that is the right way to talk about it. If you're doing really shit programming with natural language, you need loads of it to achieve the goal of interest. The fact that it's discontinuous is neither here nor there. I think that's almost an irrelevance, and it's never really had much to do with the emergence debate, except as an analogy to what are called first-order phase transitions.

Those are essentially where the first derivative of the free energy of the system in relation to a control parameter—let's say temperature—is infinite. There are these technical definitions in relation to the theory of phase transitions, but this is nothing like that. We can talk about that, because a phase transition is characterized by a demonstrable change in the internal organization of a system.

Mm.

That's really what it's about. It's not about the discontinuity. That's superficial.

Yes. So, broadly speaking, you think it's about this coarse-graining, where there is an entirely novel description using novel bases. You gave the example in the paper of how a microscopic description could be of how molecules interact with each other. A macroscopic version would be something like Navier–Stokes, so fluid dynamics.

Exactly.

And this is an entirely aggregate description of the system, which only works at a certain scale.

Right. And notice that's completely nondiscontinuous. When does something cease being the atomic theory of gases or H₂O molecules? And when are you really at the hydrodynamic limit, where you can write down Newton's laws of compressible and incompressible fluids, the Navier–Stokes equations?

Well, there isn't this magic discontinuity. It's just a limiting process. And yet, when it comes to this idea of intelligence, which is about doing more with less, all of a sudden you don't have to track all those molecules anymore. You can just look at the average densities.

Yes.

The key point here of emergence, for many of us, is that there's a sufficient change in the internal organization of the system that you can get this more parsimonious description of its behavior, which screens off—that's a sort of term of art—the contributions of the molecular degrees of freedom. You don't need to track them. They won't help you. They're surplus to the prediction that you're trying to make.

Yes, indeed. So for you, David, when we see a break with scaling laws—right now, we are memorizing everything. Capabilities and scaling laws are commensurate. There have been studies that Daniel Hendricks mentioned last night suggesting that it's something like 96% correlated. You would expect to see a deviation.

So when we can have intelligence without scaling, that would actually imply that some kind of coarse-graining has been established.

Yeah, that's interesting. I think... Sometimes it's called the breaking of scaling. Scaling laws are not evidence of emergence.

Yes.

The most famous example that we work on at SFI—my colleagues Jeff West, Chris Kempes, and others—is allometric scaling. It's good that we're in St. Andrews, right? D'Arcy Thompson, the great Scottish mathematical biologist, essentially invented the field in his beautiful book On Growth and Form, which he published in 1917. I went to see his collection this morning here, so that's an aside.

This theory tells you that at all scales you'll have a fixed relationship between mass and metabolic rate. It comes out of an optimization principle that applies at all scales, which is very interesting. There's nothing emergent about it. Emergence means there's a change in the organization that requires a new scaling law.

Yes.

Again, there's quite a lot to say about this, but minimally, to start investigating a potential case of emergence, you would want to see that the scaling law is broken. Then, secondarily, you'd have to go inside the system and ask why.

This is the big objection, I think, that Melanie, John, and I have to emergence claims: they're based only on the external manifestation of a task and not on the corresponding internal microscopic dynamics, which you want to somehow map onto the macroscopic observable. That's the essence of emergence, the micro-to-macro map. This is just macro.

Yes, indeed. And I'm a massive fan of Melanie, probably her biggest fan, actually.

Mm-hmm.

She did this Copycat work for her PhD thesis.

Yeah.

I suppose the question is, what does coarse-graining look like for language models? What would it look like? Analogy is a wonderful candidate.

More broadly, I think what you're pointing to is that these models learn fractured, entangled representations. They're microscopic representations, and what we would expect to happen if they were emergent would be that they would learn these factored, unified representations, which correlate with the world and carve the world up by the joints.

Perhaps you would extend that and say they should actually respect the phylogeny and things like symmetry. They would respect the way that the world works, which means that they would be able to take creative, intuitive steps because they know how they got there.

Yeah. I think that's a very good point. I think that's exactly right. I mean, it's a deep question, right? But I think there's a sense in which our nervous system and our bodies respect certain fundamental constraints in the physical world, which is what you're alluding to. And this has been borrowed, of course, by neural networks in things like convolutional neural nets.

Yeah.

Because then you're saying there is a structure in the world in which there tends to be some covariance in visual scenes, and convolutional networks capture that covariance very naturally. So that's an example of using a stronger prior that respects something you know about reality.

There is, as you know very well, a bit of an allergy in the neural-net community against building priors into models because you want to do it all—

Kind of inductively, which is a little bit of a confusion, it seems to me, given that the entire structure derives from the 1943 McCulloch–Pitts model, which was a nervous-system-inspired concept. So why not use a bit of world inspiration too?

Yes, indeed. You mentioned yesterday The Road to Reality by Penrose, and, funnily enough, I interviewed Michael Bronstein, who is one of the founders of this geometric deep learning idea. It is basically Platonism, right? Let’s imbue all of these symmetries because we think that the generating function of the world is constrained by these symmetries, and it’s an abstraction which doesn’t leave out any detail. Are you amenable to that idea?

No, I’m not, because it’s so anti-complexity. To be honest, I’m not familiar with it. Sometimes the way I like to put this is, in 1918, the great mathematician and physicist Emmy Noether—

Yes.

Right, one of those hidden figures by virtue of sexism in society, publishes a very important piece of work where she shows a relationship between the symmetries of the action functional—the symmetries of the Lagrangian, the principle of least action—and showed that shifting the time or space coordinate corresponded to the conservation of a particular physical observable. Change time, and you conserve energy. Change space, and you conserve momentum.

Darwin is, in some sense, anti-Noether. The Origin of Species says, “Change time, everything’s different. Change space, everything’s different.” And this gets to Anderson’s broken-symmetry idea, that it’s unlike physics because it’s not symmetry-dominated, and consequently, various observables are not conserved.

So I actually think the power of symmetry simply ceased to be the important organizing principle once life came into existence, and certainly intelligence. So I’m not sure what he’s claiming. That’s different, by the way, from saying that there are symmetries in the world that perception exploits.

Mm.

That’s clearly true.

Mm.

And I don’t, again, know if that’s the—

4. Evolution Depends On Scale

How convergent do you think evolution is? If we ran it in a parallel universe 1,000 times, do you think there’s this kind of real ontological coupling, or is it quite divergent?

Ooh, that’s a good question. It depends who you ask.

Right. What would you—

If it were Simon Conway Morris, he’d say, “Very.” If you ask Gould, he’d say, “Not very.” I think it depends on the resolution of your measuring device, because at a very coarse-grained level, allometric scaling theory says everything scales in the same way.

It doesn’t matter how many times you run it. The scaling of metabolic energy to mass is dependent entirely on the dimensions of space. The three-quarter scaling law is 3 dimensions of space over 4 dimensions of space, where the extra dimension is fractal. That has nothing to do with life. I mean, it has something to do with life, of course, but it doesn’t depend on a contingent history of life.

Yeah.

So at that level of granularity, if that’s all you could see, all you could measure—the average mass of an organism—there would be very little variation. But if you zoom in, then of course you see more and more variation, which is a kind of fossil record of its unique history.

Hmm.

Biologists do tend to zoom in. Psychologists do tend to zoom in, and I think that’s almost dispositional. Physicists like the very non-granular, averaged view. Biologists like the very natural-historical view. It’s not surprising that those two traditions come to different answers.

Yes, indeed. Could you explain what you mean by this distinction of knowledge out versus knowledge in with emergence?

Yes. Okay. So first of all, let’s just establish that with emergence, we mean that there is a description of a system that’s more parsimonious, where a given level of observation does not benefit from more microscopic detail. We gave the example of fluids, right?

Yeah.

There are many examples. So how does it come about? How does emergence come about? There’s an infinite number of examples. But physicists tend to be interested in how many identical particles, all subject to the same context or environmental inputs, experience emergence. I gave that example: add molecules, change the temperature. That’s just one dimension, the average energy.

When it comes to evolution or learning, there isn’t one input that every molecule is experiencing. You’re not just changing the temperature uniformly. Everyone is experiencing a different, unique parameterization. So the theory of emergence was developed mainly in the physical domain, where you had large numbers of identical things with a global signal.

Hmm.

And now emergence claims are being made for large systems of non-identical entities, all experiencing a unique signal. So there’s already this question of whether or not we can even talk about emergence. We don’t typically use that term, by the way, because we talk about evolution, learning, development, and engineering.

No one would say an iPhone is emergent, because you say somewhere there’s a plan that tells you exactly what to do with every component. Just as there’s a plan that tells you what to do with every cell in development. Now, it’s a complex, distributed plan, part genome, part environment, but nevertheless, there’s an abstraction in which you could talk about it in that way.

So knowledge-in systems are systems where, essentially, to get the structure of interest, the pattern of interest, you have to parameterize each component individually, more or less. Knowledge out is the example of physics, where you say, “God, all I did was change the temperature and I got solid from a fluid. I got a plasma. What happened? This is a completely different state of matter, and all I did was change one thing.”

That’s the big distinction. The challenge, I think, for biology and machine learning is how to talk about emergence when you’ve violated the prime directive, which is that you’ve been allowed to have local modification of each component.

Yes.

And I think you can, but that’s the essential distinction between knowledge in and knowledge out.

One thing that fascinates me about emergence is—we’ll get onto agency—I think of that as a form of causal shielding or causal disconnectedness, certainly from the perspective of an observer. But there are philosophers like George Ellis, I believe, who said that causation is shielded between the levels and causation happens between the levels. To you, what is the relationship between causality and emergence?

Ooh, that’s interesting. I think one way to say it is that, trivially, you do have a new coarse-grained set of effective mechanisms that you can talk about as being genuinely causal in the interventional sense of Pearl, not in the fundamental Newtonian sense.

Mm-hmm.

And these are complementary conceptions of causality. But I think I would say it would be completely reasonable to restate the emergence claim as a more parsimonious causal mechanism.

Yes.

There is an aggregate observable that conforms to your preferred neo-Pearlian framework, which works with the do-operator and the appropriate treatment of conditional distributions. So I think it is appropriate to think of it as a new form of causality.

Fascinating. On this notion of agency, how would you define it in relation to things like intelligence? Just very personally, there’s the autonomy and the causality thing, but there’s also this intentionality and directedness thing. So it’s not just about autonomy in one direction; it’s the ability to set your own direction. What does that mean to you?

Yeah. So I’ve actually been developing a framework for thinking about this. The way I think about it is in 3 different, more sophisticated conceptions. The first is the conception of action from physics.

Yes.

Balls roll downhill.

Yes.

You wouldn’t call that agentic. Some people probably do, unfortunately, but okay. That’s the simplest. We have a perfectly good Newtonian, Lagrangian, Hamiltonian theory. We don’t need to use the word “agentic” for that.

Then we have this Darwinian concept: adaptation. Well, adaptation’s a bit more than a ball rolling down a hill because it says that the ball gets better at rolling down the hill because it has an internal record of its past efforts. The general term that Kant gave to that, and John Holland gave to that, and Marrigo Mann gave to that, is a schema.

You can think of it as a simple lookup table that says, “If I find myself in this situation, I should do that. If I find myself in that situation, I should do that.” A bit different from the moon, a bit different from a ball. Then the most sophisticated, to me, would be the agentic, and that adds something to the adaptive, which is a policy.

It says, “This is what I want to do.” Notice adaptation could just be reactive. You could say, “If I find myself here, then do that.” Policy actually says, “I would like to do this into the future.”

So I think there’s a very nice spectrum that takes us from a fundamental conception in theoretical physics through to a conception in evolutionary theory, through to a conception that seems to be appropriate for a psychological or cognitive system. And they’re adding, if you like, complexity to the internal schema to accomplish that particular objective.

And does agency entail this kind of emerging coarse-graining that we were speaking about before? I mean, it certainly seems to. We look in the natural world and we see planets, we see things, but then there are these agents that have a level of sophistication which is much greater than that. And would that be an example of this kind of coarse-graining you’re talking about?

I think for us to understand them, the answer is definitely yes. There’s a secondary question of what we sometimes call endogenous coarse-graining, and that’s where it gets more complicated philosophically, because when we reason—let’s say I think I’m reasoning with language—but I’m still recruiting millions of neurons to do it. And so, at that level, if I was a solitary agent, it would be very difficult to know whether I’m really doing endogenous coarse-graining.

The data for endogenous coarse-graining comes from communication. So if I tell you, “Tim, this is what it means to integrate by parts,” or something, “This is what a Fourier transform does,” and you look at it and you say, “Oh, thank you, I know how to do it now,” I’ve actually communicated to you a very low-dimensional symbolic scheme which you can then use to program your neurons.

At that level, I think there’s no doubt, because we’ve actually done coarse-graining quite demonstrably, and it’s actionable in terms of what your neurons are doing. So I think this is Wittgenstein’s private-language problem. If all you get to do is observe a system in isolation, I would never know. But I think in this social collective-intelligence setting, it goes without any shadow of doubt that there’s evidence of emergence.

Can we go over to exbodiment?

Yeah.

And by the way, I’m a big externalist myself, but I hadn’t come across the term exbodiment.

Yeah.

Which has a very specific meaning.

Yeah. So I was very interested in this idea of embodiment, which we’re all familiar with, which is that we can outsource computation to constraint. We can use the reduced degrees of freedom of a limb to reduce the complexity of our policy. That’s—everyone knows that.

But what about an object in the external world? What about a pencil? What about a fork? What about an astrolabe? What about a Rubik’s Cube? And there, it’s not embodiment. It’s something else. And the something else there requires contributions from culture, right? In other words, no one person built the chessboard. Lots of people did together, and they collectively discovered a very effective representation of a certain kind. That’s one part of it.

So let’s make a distinction between the exbodied, which is collectively arrived at, and embodiment, which is your characteristic. The second point about exbodiment that interests me is how we have to go via an external material vehicle to get back into the brain.

And the example I like—and there are many examples—let’s give some simple ones: a map. So we collectively make a map of this city of St Andrews. But you can give that to me, and I can memorize it, and I can burn that map. And as we discovered this morning, I’m not very good at navigating, so I’ll be able to navigate freely without a physical object.

And this feedback between the collective construction of an exbodied artifact and its internalization into the individual mind, I call the embodiment helix. And it just gets better and better and better, right? Because now I have a map, I can explore more of the space, I can contribute to that collective script, which you can then internalize into your mind.

This is a very undertheorized process, and we’ve been working on it very carefully in the last several years in relation to problem-solving artifacts, like the abacus, like the Soma Cube, like the Rubik’s Cube. How does that dynamic work? What does the physical object do that your brain cannot? And we can talk about that, but we can quantify that quite exactly.

David, your institute has done some fascinating research on collective intelligence in general, and I know it sounds like a contrived question to ask, but what is the relationship between—or maybe what is the boundary between—the individual mind and the collective?

I think that’s a really hard question to answer. So I’ve worked on these mathematical formalisms for finding boundaries. Many people are interested in them. I call it the informational theory of the individual. There are theories of Markov blankets that you’re familiar with.

So there are many people interested in this question of where you draw the boundary, and I think the reality is you can draw many. So let me just give an example from biology and then come back. What’s an individual ant? Well, all the workers are clones. So you say maybe that’s not really the individual; that’s a constituent of the individual. That’s like a cell in you.

But there is a level of observable resolution where you could say a cell is individual. It is dividing semiautonomously. So a lot of it has to do, I think, again, back to that really critical conception of scale of observation.

The formalism that we started developing was one that was scale-dependent. It says, “At a given scale of observation, the key characteristic of the bounded object is that it contains sufficient information to propagate itself into the future.” You don’t have to look elsewhere. And clearly that’s true for a cell over a given number of divisions at the resolution of the cell.

Yeah.

So it’s individual. It’s not entirely true for the body, because of course that doesn’t really propagate into the future. It’s true for your genome, which does partially propagate into the future. It’s true for ideas, which are partially carried by many minds.

So if you ask me to propagate the work of Einstein into the future, you’d get a fairly incomplete jigsaw puzzle. But if you took a group of people, it would probably be quite complete. So it really depends, again, on the complexity, the size of the object, the artifact, the concept, and how many informational contributions you require.

And the theory tries to calculate that over a particular timescale and a particular spatial scale. Mind is so fascinating because it would depend what you said. If it was my preference for a certain flavor of ice cream, you don’t need much collective intelligence for that.

But when it comes to things that we actually care about, like affine or projective geometry, unless you have a Coxeter who can do it on his own, you need a bunch of us to do it together. So even that’s interesting, right? Because there’s heterogeneity in this individuality with respect to what can reliably propagate information into the future. So it’s very difficult.

Yes. You reminded me of my friend Ken Stanley. I’m sure you’re familiar with his work.

Oh, he visited us. He was on sabbatical at SFR.

He’s my hero. We’ve just had him on the show, and he was talking about the need for evolvability. So certainly when we have representations, it’s not about where you are; it’s how you got there, but also where you can go from there.

Yeah.

And agency has a little bit of a shared principle in the sense of it’s about controlling or modifying the environment to meet your goals and so on. And here you’re talking about this general principle of information propagation into the future.

Yeah, no doubt. And that’s a really interesting point, because you’re not only propagating the variation—the information in the Shannon sense—but the variational operator—

Yes.

Right? Which could be mutation; it could be different mechanisms of imagination or novelty generation.

Yes.

And so that’s actually a very good point: even within genetics, right, you don’t only propagate the genes, but the mechanism of genetic segregation.

5. Technology Outsources Human Thought

So, David, you’ve looked at various forms of representation and computation—organic, inorganic, cultural, et cetera.

Can you tell me about that?

Yeah. One way to think about this, and it gets to our opening conversation about the limits of evolution and why brains and minds had to evolve to capture information that couldn't be captured by natural selection, is that we've transitioned from a more mortal style of information processing—as in bacteria and so on—which are very dependent on life and death for information to be captured and stored—to us, where, in some sense, much of the information is stored in culture itself, so it has more of an immortal flavor.

If you think of us as analogous to cells in a body that turn over quickly, we turn over relatively quickly relative to the edifices of knowledge that we construct. I actually think you could argue that evolution itself, as a process, has moved from more mortal styles of computing—I mean, information processing in the organic setting—to more immortal-like things that we're familiar with, such as software and hardware.

And you might ask—and I have argued this, actually—why technologies exist. The way to think about this is that we have technologies for everything that we're bad at. We don't play tennis well with our hands. People do, but it's not quite as exciting as watching Roger Federer or what have you.

We don't calculate well, hence calculators and the abacus. We walk well, and we sing well, so these are things that we don't really have technologies for. They're also harder to build. There's a sense in which computers and software and archives more generally—bodies of knowledge—represent an inevitable contribution that complements all those domains in which human reasoning is deficient.

That's one way of thinking about it. My biggest fear is not that AI and ChatGPT and all of this will degrade our thinking and creativity; it's that it already has.

Well, that's another point. Okay, so that's again an interest of mine. I just want you to make the point that these technologies exist for a reason—

Yes.

—and they're there to, in some sense, complement deficits. I'm not saying they're more intelligent than us at all. That's quite the opposite. I think they're intelligent the way a calculator is.

Yes. Yes.

There's this other question which is really important, and you can actually generalize Eigen theory. This is a little bit technical again, but you can show that, in the same way that we outsource to the body and to artifacts as a means of increasing fidelity and reducing energy requirements, we will outsource everything if we can. We see evidence of this all the time.

The most obvious example is the difference between outsourcing to maps and outsourcing to GPS. If you have a system guaranteed to be more effective at navigating than you are, there is no reason for you to acquire that skill. I think we're already seeing evidence—and there are lots of studies, some of which have recently come out—suggesting that human cognition is attenuating by virtue of outsourcing it to calculators.

It's one of the reasons why I'm not very optimistic about technology, because up until now, you could argue it was net positive. No one's going to argue with a sextant or a slide rule or something. You think, "Yeah, that's great. I couldn't do it otherwise."

But I think you now can argue with the idea, or be a proponent of the idea, that the long-term implication of technology is the significant diminution and dilution of what it means to be human. I now receive, on a daily basis, tens of emails written as collaborations between humans and LLMs, and it's 100% rubbish. I think the ratio of the human to the LLM is just going to keep going down, and eventually everyone will sound the same.

It's not to suggest that the tool can't be useful, but this evolutionary drive to increase fidelity and reduce energy is so strong in us that we will eventually outsource ourselves. I don't know why people are excited about the technology if that's the likely omega point of the evolution of the species.

Yes. Could we expand more on this diminution of agency, creativity, and thinking? In universities, will we need to create separate areas for students to actually think, read books, and do work without AI?

Another thing is that we don't believe in this superintelligence idea, but if superintelligence were invented, the folks in Silicon Valley are saying, "Don't worry about labor anymore—the old way of doing useful economic tasks. Now we just need to buy lots of GPUs," and we need to defend the GPUs because people might want to bomb the GPUs, because they're the source of all power. That's very much their view. Would that change if we did have superintelligence?

There is a world where I would delegate the gym to you. I'm not going. There's no point in me going because you can go for me. I'm not going to get up and walk around because you can get up and walk around for me.

The consequence of that would be significant trauma to our physiological and mental well-being. Everyone understands that. The brain is an organ, like a muscle. If I outsource all of my thinking to something or someone else, it will atrophy just as your muscles do. There's nothing confusing about that. It's just a fact of physiology.

Now you might say, "Just as long as I have people who can carry me everywhere whenever I need to be carried, then it doesn't matter that I lose my legs." Okay. So there is a sense in which we become the male deep-sea anglerfish that's just a testicle, reduced to its absolute minimum for retaining individuality in the sense of propagating information forward in time.

It's not a future that I consider desirable. Superintelligence is only interesting to the extent that it makes me more intelligent, not to the extent that it makes me more stupid, more servile, or more dependent. It just seems to me such a glaringly obvious remark that I don't understand why people are falling for the moonshine.

David, it's been such an honor having you on the show. Thank you so much for joining us.

Thank you. It's been fun. So I'm David Krakauer. I'm a faculty member at the Santa Fe Institute. I'm also the president of the Santa Fe Institute, and I work on the evolution of intelligence and stupidity on planet Earth.

Incredible. David, this has been great. Thank you so much.

All right. And yeah.

One of my favorite ever interviews.

What the fuck's he talking about?

No. You'll have a lot of time.