碎片化纠缠表征假说（Kenneth Stanley、Akarsh Kumar）

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

So it goes back to these observations from Picbreeder, which is this picture-breeding website where neural networks—a special kind of neural network called compositional pattern-producing networks—were bred or evolved by people to produce images. This is a really unusual and exotic form of training that's very different from modern deep learning. It's completely different from it, which is one of the important points about it. When people were able to actually breed interesting-looking pictures, we learned a lot of lessons from that. The novelty search algorithm and things like that came from those lessons.

1. The Hidden Picbreeder Lesson

But there was this one dangling lesson that just hasn't gotten out, and that's what this paper does. That lesson was that the representations, the underlying representations of these images, which are basically encoded by these neural networks, are absolutely incredible and amazing. There was no good explanation for how they could be as good as they are. We noticed this very early on: they have unbelievable modular decomposition, which means that it's almost like they were engineered by a person.

There's a network that generates the image of a skull, and the network has decomposed it such that there's a component responsible for the mouth that can do things with the mouth, like open and close it. There's another dimension that can make the mouth smile. Things like that are not data-driven. What's really amazing is that we're talking about dozens of iterations in terms of the amount of search involved to get a representation like this.

I've always had many discussions over the years about why the representation is so amazing in these Picbreeder images, why it has this unbelievable modular decomposition, and why it has an extremely efficient and compact representation. At one point, Joel Lehman and I were playing around to see what would be different internally if we tried to get SGD to produce the same images—how it would represent these images. It was just dramatically different. But then we did nothing with that. Joel and I both knew this, but we didn't really do anything with that information.

I've been mulling it over for years: with this weird kind of open-ended search process, you get these incredible representations. With conventional, objective-driven SGD, which is the backbone of all of machine learning right now, you get a completely different kind of garbage representation—just total spaghetti. We came up with some terminology that we put in the paper to articulate these differences more clearly. Basically, you're talking about amazing versus garbage.

The question that the paper addresses is, what does this mean? I think it has endless repercussions and potential implications. We're basing the entire field on something that produces this complete garbage under the hood. Does this mean anything?

The fact that it's images makes it easier to see that the representation is garbage. The fact that it's a network that produces one image means that we can visualize every single neuron and what subcomponent it's actually computing. It's very easy because it's just a single image, so we can see all at once how it actually computes the image and observe the internal representations.

If I just showed you that for SGD—if I said, “Here's a skull. I made this network reproduce this skull, and the internal representation is crap”—I don't think it would be hugely resonant in the field. It would be, “Oh, it's not super surprising that it's hard to understand this really complicated, entangled representation. It's just emergent from how SGD climbs these gradients.”

But the thing that makes this really intriguing, and the reason the paper goes beyond just that, is that it gives you something that otherwise could never exist: a counterexample. There actually do exist networks that don't have that issue. You would think that this is just intrinsic to neural representation—that somehow they just look like entangled messes, and that's the way life is. But clearly, it is not how life has to be, because we now have concrete examples from Picbreeder that show beautiful internal representations.

I think the paper is trying to get at what that means and how we should interpret it. Should we perhaps aim for algorithms that actually achieve these kinds of really amazing modular decompositions? Or should we just say, “No, that's not important. Let's forget it and dismiss the fact that we know they exist”? I think that would be a pretty premature move.

There's a deep lesson here for us to learn in the field about internal representation, and it's totally off the radar—way off the radar—of anything that's being discussed. Mechanistic interpretability is not on the same page as what we're showing here. Of course, it can play a role in further interpreting this result, but this is a new thing to look at and contemplate as we try to understand its implications.

2. Humans Shape The Search

Don't we just need to have humans in the loop? I mean, wouldn't it be great if we could just have these evolving topologies without humans?

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Uh-huh.

The systems could themselves learn the categories, the abstractions. Is it just a matter of missing data, or is there a fundamental gap in capability?

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Yeah, so it's clear that part of the explanation for these amazing representations in Picbreeder is that humans were in the loop. So that's part of the explanation. But the question is, what's the deeper lesson? What did the humans do that caused this to happen?

It clearly wasn't intentional. The humans weren't thinking about the underlying representation at all. They were just choosing things they liked. So why does that lead to this virtuous type of representation? I think one really interesting lesson this shows is that it matters not just where you get, but how you got there. That's something missing right now, because we tend to care only about where you get.

We look at the benchmark score in the field, and that's the result. It's doing really well. It just passed the Math Olympiad. But what if it matters how you got there? You could say, “Why does it matter how you got there? If it performs well, it's the same either way.”

That's what the paper is all about, basically. You can have these 2 networks, and they both output a perfect-looking skull—the exact same skull. One was found in this completely different, open-ended way, with people guiding the search. Another was found objectively through SGD. Very different trajectories through the search space were taken, which means that under the hood, the representation is radically different.

You say, “What does it matter, then? What's the underlying representation?” Well, it can matter a lot how you represent the world. So the question with humans is, are they the essential ingredient? I think they can't be. It can't be that the only way to get a good representation is to have a human guiding the search.

There's a deeper explanation for why the humans were successful at getting to these amazing representations, and it has something to do with the underlying, open-ended nature of what they were doing. In other words, on the road to getting an image of a skull, they were not thinking about skulls. When they discovered a symmetric object—an ancestor to the skull—they chose it even though it didn't look like a skull, but that caused symmetry to be locked into the representation.

From then on, symmetry was a convention that was respected as they searched through the space of symmetric objects. Somehow, this hierarchical locking-in over time creates an unbelievably elegant hierarchy of representation. It's plausible, at least to me, that there are algorithms without humans that could take similar trajectories through search space. Maybe not quite as perfect as the human trajectories. I wouldn't be surprised if we can't actually hit that ideal, but there's probably a continuum where we could come closer.

What are the implications? If the order in which you encounter the principles that lead to your final understanding of the world matters for how you represent the world, and therefore for your ability to be creative in the future, then does it matter what order we allow these large models to encounter the different principles on the road to total understanding of everything in the universe?

I would guess this implies that it probably matters. That opens up a huge range of possible creative opportunities for alternative ways of thinking about training that would lead to better representations.

You brought in the open-endedness aspect, which is fascinating, because you’re saying they weren’t looking for the skull, right?

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Yeah.

So what were they looking for?

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

They were composing these primitive basis functions that they have in their mind, so they know that symmetry is good. Where did they get the symmetry idea from? It must be somehow gleaned.

Actually, our function space is restricted in some very important way. We know that we have certain things we can compose, and we know that we can compose them in certain topologies. We know that invariably, if we follow that trajectory, we will land on interesting things, even though we don’t necessarily know exactly what we will land on.

Mm-hmm. Yeah. Yeah.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

That’s how this happened. People in general have intuitions about potential, not just about where we are right now, but that this might lead to something.

In fact, it’s very complex because the more people play with Picbreeder, the more they get intuitions about Picbreeder itself. They start to understand what might lead to what and what’s promising, which is different from what’s aesthetically pleasing in the moment. They’re both enjoying the image and predicting, “Well, where might this go?”

It’s not that they’re predicting they’re going to get a skull, but they’re predicting that symmetric things are really interesting and beautiful, and asking what other kinds of symmetries might come out of this. They’re thinking that, and that helps to get this kind of virtuous ordering, which causes a sequence of lock-ins of different conventions of increasing complexity. That then creates this amazing representation underneath the hood.

3. World Models From Nothing

I want to point out, though, that this view of the world is so radically different from the data-driven view that we live in right now. What’s really fascinating to me about it is that it’s not data-driven. We think that eventually, if you experience enough of the world, your representations might start to mirror the way that the world is, and you get this kind of isomorphism between the organization of your brain and the organization of outside reality.

Well, that’s a data-driven view. Almost everything we think about is data-driven. The bitter lesson is sort of a data-driven philosophy, but this is completely contrary to the bitter lesson. What you’re saying here is that we see almost nothing of the world. PicBreeder knows nothing of the world. There’s no pretraining at all. You start with some blobs.

Over a few dozen iterations—I mean, a dozen is crazy. It’s peanuts. We’re used to millions and billions. We don’t do dozens. That’s not what we do in our field. But here we have dozens of iterations, not enough to be exposed to almost anything. It somehow finds within this newly constrained space of dimensions discovered by humans things like the difference between opening and closing a mouth, or smiling and not smiling.

Those dimensions exist now in this space of the network, but not because of data. They’re whole-cloth, de novo discoveries that are not data-driven. So there’s no bitter lesson. It’s just out of nothing.

There are even crazier ones, like the apple one, which is in the paper—in the appendix. This apple has an unbelievable weight, a single weight in the apple representation, which is a single continuum. If you move along that continuum, you swing the stem of the apple back and forth, from left to right.

Maybe someday we’ll put an animation over this so we can show it. I could give you the animation. But you can see that the swinging stem is one dimension, and it’s three-dimensional. It’s not just a two-dimensional thing. It moves as you would expect in a rotation in three-dimensional space. It has a shadow underneath it, and it’s like a green leaf.

The underlying apple, which is a symmetric object, is not disturbed at all. It’s totally independent. It’s been decomposed. Then there’s this one thing, which is the stem swinging.

What I’m saying is that it’s absolutely incredible. It’s mind-blowing that, if you think about that as a world model, it’s an actual, true hypothesis about the world. This is the way that stems look when they swing. But this model has not been trained on anything in the world. It’s never seen swinging stems, let alone apples, at all.

I can almost guarantee you that in the training trajectory itself, there was no swinging of the stem. After all, if the stem started swinging, that would mean it already had that ability. So it’s circular to argue that.

This is something that arose out of the fact that the representation is so elegant that it somehow has an internal hypothesis—I think of it as a hypothesis about the world—which is correct. You have to ask yourself how many of our hypotheses are like that instead of data-driven hypotheses.

We do sometimes have these unbelievably elegant underlying representations of the world that are often unique—unique to an individual, not necessarily universal across human beings. Everyone’s representation is unique. So this is a totally different way of thinking about representation and knowledge and how it comes to be.

Yeah. The thing I’m trying to understand is that we can agree that humans have this incredible abstract model of the world. We have a language of thought, and the way we understand things is through these compositions, these topological compositions of little basis functions, right?

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Yeah, I think it’s true that PicBreeder is a kind of psychology experiment. That is a dimension of what we’re seeing.

4. Compression Is Not Enough

It’s true that intelligence and compression are often equated, and they clearly have a relationship. I think in the paper we talk a bit about this and even speculate that there may be something more to say than just compression, in the sense that the factored aspect matters. How you factor matters. It’s not just that it is compressed.

If I know that a face is composed of eyes, a nose, and a mouth, and actually factor those out, even if you had a greater compressed version of the face that didn’t factor those out, I would prefer the factored version. That’s still, in some sense, better.

You could say that I can generate new faces in a principled way. So I’m not sure that the maximally compressed version is always the most so-called intelligent, depending on what we mean by that. There are multiple factors to consider, but obviously compression is a virtue.

These extremely inefficient representations that you see in regular SGD are obviously part of the problem.

Speaking about the human aspect of it, just to go back to that for a second, one thing that I think is important to consider is that you can extrapolate outside of PicBreeder this principle of searching through regularities, or finding good isomorphisms with the world in some kind of sequence. That’s just a general way that people make discoveries.

It’s not just in this very almost psychological-test-like environment that this kind of thing happens. It’s a general aspect of human exploration—intellectual exploration.

You can imagine the difference between somebody who learns calculus from a textbook and someone who invents it for themself because they were curious. They both end up knowing the same thing. They both take the test, and they both get a good score on the test.

But it’s pretty obvious that the person who found it through their own independent exploration is probably going to be doing much more interesting math after that test. Why? It’s because they went through a PicBreeder-like exploration process.

Of course, this is easier said than done. Most people won’t do that. But the way that you got there matters a lot, all throughout life.

5. Creativity Needs Better Abstractions

If we did have a large language model that could understand abstractions at the level we do, why couldn’t we just run it autonomously? Why couldn’t it step by step know what’s interesting—

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Mm-hmm.

—and just traverse this phylogeny and find interesting things and self-reflect? It feels to me that the missing link is having the correct level of abstraction and being able to do this iterative, open-ended search. They can’t do that because they simply don’t have the abstractions. We have the abstractions.

Could we actually build an autonomous, open-ended system that could find the abstractions the way we do, or do you still need the humans?

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Is there still something missing in this respect from large language models—their ability to leverage abstractions, to really think creatively out of the box, and so forth? There are different ways you could say this, but that’s basically the big question, I think.

I basically agree that I don’t think what we have now can really match the very best of human creativity. I think that’s safe to say.

I think a lot of people would probably agree with that. It can do some level of creativity—what I would call derivative creativity—which is sort of like the bedtime-story version of creativity. You ask for a bedtime story, you get a new one, and it's actually new. No one's ever told that story before.

But it's not particularly notable. It's not going to win a literary prize. It's not inventing a new genre of literature. There's basically nothing new really going on other than that there's a new story. So that's derivative. In my nomenclature, I call it derivative.

And so that's pretty much, I think, where we are. When we do things like try to leverage these models to explore really important creative spaces, I think that's an obstacle. It doesn't totally stop us, because there are ways of getting out of distribution even with that limitation—for example, wrapping evolution around it, like we did with Evolution through Large Models, as DeepMind recently does with AlphaEvolve.

It can actually get out of its own distribution, but I think it's an inefficient way compared to the human mind, which leaps through levels of abstraction that are encoded in its representation. That points back to, I think, the lesson in this paper: it could be that part of why we're trapped in this box of derivative creativity is because we don't have this really nice style of representation.

We call them unified, factored representations. In other words, concepts are unified, and they're well-factored into the different components that actually correspond to what's interesting within the domain. Maybe they lack that. In other words, they have what we use as the other term for the bad kind of representation, which is a fractured, entangled representation.

Concepts are fractured into pieces and entangled with each other in ways that are inappropriate. Maybe it's because SGD naturally does produce these fractured, entangled representations—at least conventional SGD, the way it's being applied here. I'm not necessarily saying that no version of SGD can actually create the more elegant form, but the way that we apply it conventionally, it tends to produce this fractured, entangled representation.

Maybe that's part of why it doesn't have access to the abstractions, as you put it, that are necessary to do truly transformative creative ideation. That would require a very well-factored version of the world, which is just lacking. There's lots of evidence of the problem with having entanglement and fracture in representation.

We go through some of it. In one section, we give a lot of examples from more recent large models. We don't just look at Picbreeder and images. We give hints of it because it's hard to get direct evidence: you'd have to look under the hood at the actual representations, which is quite complex to do, of course, for things that are not just images.

Concepts and knowledge, of course, are very complex, and the whole field of mechanistic interpretability is trying to grapple with this. But we give some hints that there is evidence of this kind of entanglement and fracture. Maybe it's a result of the fact that we're doing this conventional type of objective-driven search that gives you the lowest-common-denominator version of representation, which is then going to be a creative straitjacket when you're trying to get outside of the box.

That isn't the case for Picbreeder networks, which can imagine things that are almost unbelievable, like the swinging stem of the apple, even though it's not part of a world that it has ever experienced. I think there's a lot to learn in representation that can help address what we intuitively perceive as a limited ability to be creative and abstract.

6. Building Networks From Seeds

Basically, there's a spectrum of representations. The problem with neural networks is that they can't distinguish the good ones from the bad ones. What Kenneth is basically saying is that we don't have to do it the Lottery Ticket Hypothesis way, which is where you start with an infinite number of degrees of freedom and then, like a block of clay, carve it down afterward.

The really smart thing to do is to build it up from first principles at the beginning. So you actually build these neural networks—

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Mm-hmm.

—piece by piece. He's advocating for a non-stochastic gradient descent, possibly an evolutionary method of doing this. But honestly, Keith, the thought that I was left with is that he thinks there is an algorithm that can do this.

The reason why it works in the real world is because we have this epic panoply of computation. The universe is executing all of this computation at the minuscule level, right?

Yeah, yeah.

And that is the physical process which gives rise to evolution, intelligence, and all of the interesting phenomena we want to capture. Even though Kenneth is absolutely on the mark when he points out the reason why neural networks aren't creative, and indeed what creativity is, no one has better intuition of evolution than Kenneth.

But he still seems to be slightly off the mark when he believes that we can construct an algorithm completely disconnected from how the physical world works, or largely disconnected, and still capture some of the phenomena.

Right. Well, in the spirit of his title, questioning representation optimism, I think Kenneth has engineering optimism. To a degree, I share that optimism. You and I have both spent enormous amounts of time in our Discord server arguing about things like this, so I'm intimately aware, at a gut level, of the absolutely cosmically insane amount of computation that goes into powering the algorithm of evolution, with a capital E.

It's happening on Earth in many forms—not just life, but memetics, information, and whatever else, right? Higher-order phenomena. I'm just saying there are probably very clever hacks—at least I hope, and maybe Kenneth hopes—that will get us somewhat toward a better method.

Akarsh had a very good take on some of these evolutionary aspects, in particular, it not being an optimization but a constraint. It's good to see new researchers taking up that mantle. I think a big part of the point of this paper, and their point, is that we need to think more about these processes.

You brought up a lot of interesting takes just now, and intuitions about how POET and these algorithms might work. I think the building-up is essential. I imagine future training of neural networks will look like this.

You'll start with a neural network that has 100 parameters, and you'll train it on some subset of the data—a simple subset. Then, in the next phase, you'll train one that has 1,000 parameters, and in some way it's expanded from that lower-level network.

We'll have algorithms that identify subnetworks in there that do something in particular—say, these 12 neurons. What we'll do is make those 120 neurons and then train them, keeping them isolated as a group, right? That way, they grow that abstraction further.

Then it'll be 10,000, 100,000. You'll keep scaling it up, right? What you'll end up with is almost like a neural network that has a very specific kind of—not sparsity, but modularity. A very specific kind of modularity, because it was grown from seeds rather than starting at the massive scale and allowing it to do all this really obnoxious and counterproductive entanglement.

Yeah. I think we're getting to putting the pin in the center of the dartboard here. He said the problem is that neural networks have too many degrees of freedom. They're like a pile of sand, right? We're different from that because the very basis of how we think is correlated to how the world works.

There's something about building up. You said building up, not tearing down. He said it's not about what you know; it's about how you got there and how—

Yeah, exactly.

It's about how evolvable the knowledge is and where you can go with that knowledge. It's not necessarily that stochastic gradient descent is a bad thing in and of itself; it's simply that it's statistically intractable to use anything other than stochastic gradient descent. Otherwise, it simply doesn't work.

So, yeah, I don't think he's necessarily saying that. I mean, maybe he is, but he's saying that these evolutionary algorithms are really, really good when you're building up, when you have a very, very sparse search space.

The other thing, which is very important and which we didn't get to, is this whole building-up thing, like the NEAT algorithm, which you can bring in. This is the idea that we need to have a form of training or learning about the world that monotonically increases information and complexity.

We're adding this thing and we're adding this thing, and every time we compose things together, we're looking at the evolvability.

So we're not just saying, “Can you do the thing now?” We're saying, “Is it actually likely to be able to deal with future things that I might encounter?” When children are born, they still have 2 legs, basically. So you shouldn't just think of it as a random crossover operation. It's an operation that respects the provenance, the topology, and the structure, right?

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Mm-hmm.

So there are, in practice, crossovers and mutations within certain topological frames, but some things are held constant because they need to be held constant. On your comment about degrees of freedom, I think Kenneth was saying that we actually want to reduce degrees of freedom. He wasn't saying, “We just need degrees of freedom for stochastic gradient descent, but degrees of freedom are okay.” I think he was saying that there's actually a magical Goldilocks zone when it comes to degrees of freedom.

We want the degrees of freedom to be at least a representation of how the world works, but necessarily there should actually be more flexibility than how the world works. If there's too much flexibility, you get a weird kind of mode collapse, and you don't see the emergence of evolution.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Degrees of freedom, the way I think about it, is like a memory with an algorithm. You want to use as much as necessary and no more. It's really hard to figure that out.

Yeah, I mean, just imagine Einstein. Of course, there's an element of serendipity, right? You have to be in the right place at the right time. But there are still just mental degrees of freedom, and simply, if there are too many degrees of freedom, then relativity wouldn't be conceived of, right? You need to have that spark.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

What's funny is that he added one additional parameter, the cosmological constant, right? He was kicking himself for lots of his life that that was his greatest blunder, right? He considered his greatest blunder adding an additional parameter. Think about that. Simplicity is such a driver of successful models.

It turns out, though, in the spirit of just as many parameters as you need and no more, now we need the cosmological constant for very valid scientific reasons, right?

On your comment about degrees of freedom, separating them from training and how things are enacted in the real world, I like thinking of them more in the same frame. This is actually related to what we've been talking about when we discuss agency and autonomy. Kenneth is using the frame of creativity, but it's all the same idea, right?

When we have agents that are acting in the real world, we need them to make intuitive decisions that make sense. That is precisely the reason why current AI doesn't do intelligent things when it's autonomous. That is the reason why we have to supervise current AI every single step of the way, because it would do stupid things. So I think if you did reduce the degrees of freedom to actions that actually made sense, the probability of it working autonomously would be much higher.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Yeah, that's a bit unknown to me. I'm not sure, because I think there are trade-offs there. Again, it's this Goldilocks thing. If you reduce it too much, then it ends up not being able to do the things that you need it to do. If there are too many parameters, it's able to do the things you need it to do, plus it does a bunch of stuff that you definitely don't want it doing, you know? So there's this weird balance in there.

I think that's a very difficult question and a very hard engineering problem. But this paper is pointing in directions that we need to go. One really great thing about this paper, which you mentioned earlier, is how visual it is.

If you look, for example, at the intermediate layers for the evolved compositional pattern-producing network versus the SGD-trained one—come on, folks. I don't care how much of an SGD fanboy you are out there. Just look at the images and tell me with a straight face that there's any merit whatsoever to the entangled, convolved, contorted, convoluted nonsense that you end up with under SGD. Seriously, I don't even know if we can have a conversation if you think there's value in there.

If you look at the evolved ones, they're like a stencil that you could imagine an artist would have: “Here's an oval thing and some gradient lines,” and this sort of thing. The point is, it's not just about appeal to humans, okay? It results in extremely powerful transformation capabilities, like these parameter sweeps where there's a parameter that controls the mouth opening and closing, the mouth being wider or winking, or the apple's leaf literally rotating as if it was on a 3D axis, okay?

These are neural networks that were trained solely on 2D images, and they have evolved a concept of 3D rotation. If you're not impressed by that and blown away by how cool that is, and if you can't appreciate that that's relevant to the real world and agents operating in the real world with an effective world model, I don't know. I can't really pull you out of that hole.

Yeah. I wrote some software to visualize that, so we can show that on the screen. But this actually—

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Yeah, it's crazy.

—comes back to the famous episode 61 of MLST, when we had Randall Balestriero and Yann LeCun. It was called “Interpolation, Extrapolation, Something Else.” I can't remember now.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Yeah.

We showed this neural network visualizer. It was the TensorFlow Playground, I think it was called. It would show you the evolution of training for all of these different types of neural networks, with different activation functions and learning rates.

One of the datasets was a spiral manifold. It would show you a spiral, and you could kind of see it trying to fit the thing over time. One of the reasons why it can only interpolate when inside the training support—which basically means, roughly speaking, inside this high-dimensional space of data—is because the activation functions are ReLUs, right? They're piecewise-linear functions, which means they just chop up the space.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Yeah.

What's interesting about the evolved CPPN—the compositional pattern-producing networks—is that the functions can actually be trigonometric functions. These are continuous functions, because most of the time in a neural network, if you give it a test sample that is outside of the training support, you're in no man's land. It gives you nothing. Whereas with these CPPNs, if I understand correctly, they actually do extrapolate outside of the training support because these trigonometric functions just keep going, right?

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

The problem isn't just the activation functions. It's the training methodology and what would happen. For example, let's say we just keep ReLUs. I think if you still found a way to do this kind of evolutionary building up from simpler mappings, what you would end up doing is having a simple higher-order layer that, say, took the spiral and chopped it up into 4 quadrants—nice quadrants—and then one that did some 45-degree-angle chopping, or whatever.

So it would still end up being a piecewise-linear division of the space. But rather than, if you go and do this over at the TensorFlow site, having all these messed-up, weird lines at funky angles and whatever, you would end up with the type of chopping up that you and I might do, right? Quads and then hexagons, and whatever else. The point is, whatever activation functions you have, there's a better way to train them.

Yeah. I mean, there are a few points here, because I guess where I was going with this before is that you take Y = X², and the reason why we think of it as robust is that for any value of Y, it kind of does something—it does something reasonable, right? It's not just memorizing little regions in the training support.

We often speak about the limitations of neural networks in a variety of different ways. There's a complexity limitation, a computational limitation, and a training limitation. But it's quite interesting to play with this and pull it apart a little bit.

As you say, the SGD thing is actually a serious limitation, because there are neural networks that can do a hell of a lot more if they're not trained with SGD. Schmidhuber gave an example of an RNN which is Turing-complete under certain conditions, but it's not trainable with SGD. There's also the other matter that when you start using these weird and wonderful activation functions, you can do things outside of the training support.

And then there's the other matter that when you start doing non-SGD training, you can actually build hybrid systems as part of the training process. The way we recognize faces, we do some kind of composition, splitting, iteration, and recursion. Imagine if you could just train that entire system with another form of training process. So it rather opens up the whole game.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Right. Yeah, 100%. You can handcraft neural networks. For example, there was that DeepMind paper—the one where they handcrafted sort of a universal Turing machine in an LLM form—but you also have to add external memory. So they had to do extended generalized autoregression rather than just autoregression.

You can construct them, but they're not trainable. The core thing we need to improve is the training methods. I think nobody really knows what they look like at the moment. There are some hints, right? POET, maybe what you're talking about—these kinds of hybrid systems where things are trained in different ways, curriculum learning, or some type of building up from seeds that are somehow kept slightly isolated.

I don't know. We don't know because everybody's just stuck on SGD, scale. Scale is enough. SGD attention's enough. Until the industry fails enough to achieve anything real with those methods, I don't think sufficient resources are going to go into the alternatives.

Yeah.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

But they're getting close.

And the final—

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

They're getting close because even the do—

Mm.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Even the doomers are having conversations about, “Why aren't we all dead yet?” Once GPT-4.5 came out—or 4.whatever—I thought we were doomed then. What's going wrong? So they're reevaluating their apex of rationality over at LessWrong to figure out what went wrong. Why aren't we dead?

I know, but even this, I think, is the perfect intuition pump for people to understand why we are not doomers, right? Look, we're not horrible people. If we actually believed, as the doomers do, that this technology is autonomous—that it has agency and could just go and do things on its own—we would be doomers as well. We would be regular guests on Liron's YouTube channel. But it just doesn't work like that, right?

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

No.

It's not even close to working like that.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Well, let me just caveat something there. I'm on the record as having—again, it depends on how you define doom—a sort of, let's say, all-cause doom of 1/3 or whatever it is. I'm also extremely concerned about AI harm. AI is causing harm today. Massive harm. It's causing massive harm. It will continue to cause massive harm. People are using AI to do heinous things and cause lots of harm, and I absolutely think we need to protect ourselves from that.

I'm just not an AI doomer who thinks AI is going to become superintelligent and kill us all, because we don't yet have the right architecture for that. We haven't discovered any architecture that, in my opinion, is machine-trainable and will lead to AGI.

Yeah. I agree with all of that. The way I think about it is that there are 2 types of AI. There is the agentic, monomaniacal, separately intelligent form of AI that Liron is talking about. I don't think AI is intelligent, because to be intelligent requires creativity. This is what Chollet says. It's about being able to invent new knowledge given a novel situation. AI doesn't do that.

But the AI that we have now requires humans, right? It's a tool that humans use, and it makes us more creative and smarter, and it has all sorts of ramifications. I agree that there's a lot of harm to be discussed there, but I do make that clear separation between the 2.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Yeah, and I make a similar separation. I just use different words. So I talk about the AIs we have now as being narrow intelligences, not general intelligences, and I consider those 2 things categorically different, not just different on a spectrum. I know we have some differences on that.

We discovered this incredible book, “Why Greatness Cannot Be Planned.” It's got a butterfly on the front cover, and the reason Kenneth put the butterfly on the front cover was that there was this phylogeny created by Picbreeder. It was basically a cross between Tinder and Flickr, where you log in and select 2 images that you like and breed them together. Behind the scenes, it's using the NEAT algorithm, which is a way of evolving neural network topologies.

Essentially, the topologies of the 2 CPPN neural networks are mixed together, and you get a new image. A CPPN is simply a neural network that has a broader array of activation functions, including trigonometric functions like sine and cosine and whatnot. It also takes an input of an X and a Y and a couple of other things, and it has an output of a hue, a saturation, and a luminance.

If you want to generate an image with it, you basically enumerate a bunch of pixel values, and it's resolution-independent, which is quite cool. You can make very high-resolution images or low-resolution images, and it would just generate an image of something. It might be a skull or an apple or whatever.

Essentially, humans could supervise this breeding process. When you looked at the phylogeny, even though the humans weren't looking for anything in particular, you got these weird intermediate steps that did not resemble the amazing thing that was discovered. What's interesting is that amazing things were discovered in surprisingly few steps.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

It's really fascinating on so many levels. I highly encourage people to check out the NEAT paper and Picbreeder and a lot of related things. As you pointed out earlier, I think the crux of the difficulty is: We did Picbreeder, we got a skull and a butterfly and lots of other things. But I don't need a skull and a butterfly. I actually need something that generates knights' helmets or dragonflies—not butterflies, dragonflies.

And therein is the crux of the problem, because we did this open-ended, cool thing. Part of the downside of an open-ended search is that you don't know where you're going to end up. I think in the interview Kenneth referred to this as the “cone of inevitability,” right? As you progress further and further along the time dimension or the step dimension of evolution, where you end up becomes less and less certain.

The crux of the problem is: How can we do open-ended evolution in such a way that we end up with the things that we need? We need something that detects school buses, something that finds pedestrians in the crosswalk, and something that can generate movies about samurai. There are things we need, and so there's this conflict, right? We do have goals. We have certain goals. We need to protect pedestrians and monitor school buses and whatever else. Yet we want to try and achieve them with open-ended evolution.

I know. It's such a paradox that you find what you need when you're not looking for it. But there are so many forms of deception in open-ended search that this deception word is very interesting, and it's all about the fact that what you think you want isn't actually what you want, right? Even you just said now, we want school buses to be safer, and there are many discoveries in the search space that would make school buses obsolete.

In fact, YouTube is now... It's completely taken over Hollywood. If you're trying to be a traditional filmmaker, you don't make any money anymore because everyone—

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Right.

—is watching it here on YouTube. But my God, YouTube could be out of business in a few years. We could easily discover something that wipes YouTube away. So it's about this epistemic gap between what we really want and what we think we want.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Yeah, no, that's a good point. Why do we need all the kids to go to a physical location every day of the week? Maybe there's some other pure virtual or more local system. You're totally right. That's really the crazy thing about life and the universe, and we need more variety. We need not put all our eggs in one basket. We need to explore alternate methods, folks.

GPUs were invented for video games. They also happen to work really well for a certain kind of machine learning, so let's just do that.

Yeah. And that, by the way, in the hardware lottery, is also a manifestation of Kenneth Stanley's “Why Greatness Cannot Be Planned.”

Because nobody thought, when they were developing the GPUs, that they would have anything whatsoever to do with AI. YouTube started as a video dating website. It had nothing whatsoever to do with a new type of media. Wonderful. Well, Dr. Dugger, thank you very much.

Yeah, it was a pleasure. It was so great to be with you guys again, and I think it was about 4 years ago that we interviewed Kenneth the first time.

Mm.

It was eye-opening then, and it has opened a lot of great open-ended exploration for me personally.

I think he won you over in the end, didn't he?

Sure. I'll concede to that.

Today we have Kenneth Stanley, and Kenneth is my hero, as folks on the show will very much know. We also have Akarsh Kumar from MIT. Akarsh, since you haven't been on the show before, can you give us a bit of an introduction?

Yeah, definitely. My name is Akarsh. I'm a third-year PhD student at MIT. I'm working with Phillip Isola, and I'm interested in researching emergence, open-endedness, and all the cool stuff around artificial life and meta-RL.

Anything that's really an emergent process that creates intelligence or creates some complexity, I'm very interested in, and I'm super excited to be here.

And you've had the great fortune to be writing a paper with Kenneth Stanley. I think not many people would have that accolade on their résumé.

Yeah, I'm super privileged. I'm super happy to be here with them.

7. Questioning Representation Optimism

As I understand it, you guys have just written this paper called “Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis.” The basic idea is that the types of representations that neural networks learn are chaotic. They're just like spaghetti, right?

What we want to have in artificial intelligence is a way of carving up the world by the joints, right? A way of actually understanding how the world works at an abstract level. It seems to be the case that we humans, by some kind of osmosis or magic or God knows how we do it, have a very powerful abstract model of how the world works, and neural networks don't do that.

Kenneth Stanley

Yeah. It does seem like we do this in a very sophisticated way. But I think the paper isn't so focused on saying, “Hey, look how good humans are,” and then, “Look how bad LLMs are,” for example. It's not really that story.

It's more that there's an observation of the underlying representation that you get from SGD, and just how terrible it looks, which is really visceral. You don't need quantification to see that there's something wrong there.

I think that observation on its own would make only half a paper, because a lot of people would think, including maybe I would think, that's just the way it is. It looks the way it looks, but it works, so it's fine. Of course, people quibble about the differences between human-level cognition and LLMs, and I don't doubt that humans do something beyond what LLMs today can do.

But you can't really look at that by itself and, when you observe the underlying representation, just say, “Oh, clearly it's messed up,” because we don't know what it should look like. There's no baseline to compare it to.

What's really unique here in this paper is that we do have a comparison point from a very unusual set of circumstances that led to a very different-looking representation. So actually, you can compare it to something, and indeed, the thing that we compare it to looks like it should look intuitively.

It's not necessarily clear that this is how it looks in the human mind or in the human brain. We don't know. There's no way to go in and check. But it definitely would be nice if it did. It looks the way you would think it should look.

And so it really raises questions. You could still take the position that, “Oh, well, I'll just dismiss that, and I still think it's fine the way it is with all this horrible-looking entanglement.” But it definitely raises questions: Should it look like that, now that we know there are representations that don't look that way?

I think what's really valuable about this is not necessarily that it shows there's a fundamental flaw—it may or may not—but that it raises really deep questions that we can pursue now that we wouldn't otherwise have pursued. There are lots of questions we can now go after to try to understand the nature of the representation, and maybe whether it is potentially flawed in a way that could be rectified.

So, Akarsh, Kenneth was just saying that this shortcut rule, or Goodhart's law—you know, we refer to it via many different terms—the reward hacking and things like that. There's this problem in machine learning that you put an objective into the system, and generally the objective is created by us. It's a proxy because we don't understand reality, so we create a proxy.

Initially, it works very well, and then it actually gets much, much worse if you continue to train on it. That is a really fundamental problem throughout all of machine learning. I just want to break it down. You mentioned SGD a minute ago. Why is it SGD that's the problem, as opposed to something else?

In our paper, we don't term it SGD; we term it conventional SGD, and we do that on purpose because we're not 100% sure whether the problem is SGD, whether we just need to get rid of SGD, or what the exact problem is.

We're just saying the current paradigm, which encapsulates a fixed architecture, a fixed objective, and SGD as the target, as the thing that's doing the chasing of the target—that entire paradigm has something wrong with it because it's susceptible to shortcut learning.

People have known this for a long time. Melanie Mitchell talks about this a lot, which is basically just Goodhart's law for representation learning, right? Basically, if you try to solve a task, you're going to perfectly solve that task, but what you really wanted was a good representation of that task.

As our paper shows, there are many different ways to solve the task. You can solve it with a bunch of heuristics and a bunch of if statements, or you can solve it with the right abstractions, and those are the 2 ways to solve it, right?

Which one you end up employing really matters for what you really want, which is adaptivity, generalization, creativity, out-of-distribution generalization, and especially continual learning. We're going to talk about that a lot more later.

I guess the point is we don't just care about the training loss. That's not what we care about. We train it on the training loss, but that's not what we care about. What we care about is all the downstream stuff that we're going to use it for later, which is much harder to quantify, much harder to formalize than just the training loss.

Kenneth Stanley

Let me add one thing. I think one of the really interesting things about the observation in this paper is that it pokes a hole in a very deep assumption that we have: if the results are good, then what's underneath the hood is also good. It's a kind of assumption that's unstated most of the time.

But it's clear here, when you look at the underlying representations in this paper, that it's not necessarily the case. You can't just look at benchmark performance, which is most of what we look at when we talk about whether a model is good, and know that things are actually okay under the hood.

As Akarsh said, that does have implications, because you may think, “Well, who cares? If it does well on the benchmark, then it's good. What else matters? Why should I care?” But as Akarsh says, there are all kinds of downstream implications if your underlying representation is terrible. I'm sure we'll get into those.

Just this idea that you can't be confident just because things look good on the surface is thought-provoking, I think. It leads to a lot of questions about what's really going on.

One thing I really want to add quickly to Ken's statement is that I think at the end of the paper, we have a quote from—or something like a quote or a paraphrase of—Ken's dad. I think he says something along the lines of: 2 mathematicians can both ace a math exam. One can go on to become a great mathematician who discovers a lot of things in the field, and the other can go on to discover nothing.

So the test just doesn't give you a picture of what we really care about, which is downstream: how they influence the field and how their research progresses.

Yeah. I mean, this might be a good time, Kenneth, to talk about the story of your book a little bit. On the front cover of the book, there was this butterfly, and I'd love for you to tell the story of the butterfly.

But in the book, you went to great lengths to talk about this concept of deception, which is simply that when you have a complex objective, there's this epistemological gap, right? Again, the reason we have proxies with Goodhart's law is that we don't actually understand the thing that we're modeling, so we have a proxy.

Kenneth Stanley

A lot of people know my work for this point related to the butterfly, which comes from a system called Picbreeder that is also relevant to this paper. But, just for context, we’re actually going to draw a new lesson in this paper from the usual one that people have heard before from me, if they’re familiar with my work. The original lesson had to do with the fact that there are some things you can only find by not looking for them. Setting an objective and saying, “I want a butterfly,” for example, can actually cause you to fail and not be able to get to the thing that you want.

This was true in the Picbreeder system, which was basically a big crowdsourced experiment in people exploring image space generated by little neural networks called CPPNs, or compositional pattern-producing networks. The Picbreeder system allowed people to effectively breed pictures. We found inside the system that people who decided they wanted a certain image and tried to evolve that image would fail, while people who weren’t looking for anything in particular would discover all these amazing things. The butterfly was the symbol we used, because we put it on the front cover of the book, for one of these serendipitous kinds of discoveries.

This led to the idea that there are many things in the world that we’re not going to be able to find if we directly search for them because of deception, and that’s the underlying reason. Deception means that the stepping stones that lead to these interesting artifacts you might want to find don’t resemble them. If you have an algorithm that’s trying to follow a gradient by matching closer and closer to the objective, getting a higher and higher score or a lower and lower loss, that would mean you’re going to get stuck in a dead end because of deception. The things that lead to the thing you want actually don’t look like the thing you want.

This is true in the lineage of many of these images in Picbreeder. It led to the novelty search algorithm, which was the idea that, if that’s the way Picbreeder users succeed, maybe there could be an algorithm that isn’t driven by where it’s trying to go and instead just tries to do something new and interesting all the time. That then led to the field of quality diversity. You see quality diversity inside major systems now. For example, very recently, AlphaEvolve has MAP-Elites under the hood, which is a quality diversity algorithm. These are systems that try to get out of deception and illuminate spaces of many interesting things.

That insight led to a lot of things, and the butterfly is symbolic of it. But there was one further insight missing from all of this, which is another deep insight. It shows that this crazy Picbreeder experiment had all of these really unexpected discoveries underneath the hood. What happens if you do get to something as an objective? What was interesting with Picbreeder was that we did experiments later to confirm the hypothesis that you couldn’t get to these things by trying with the same algorithm.

The underlying algorithm in Picbreeder was the NEAT algorithm. Someone found a skull, for example, and the skull is used all over this new paper. We then said, “Okay, we have this hypothesis that you wouldn’t be able to find the skull by actually looking for it.” We tried to automate the NEAT algorithm to start from a random position, a random configuration of the network, and random weights, and then just try to get closer and closer to the skull. This generally would not work. It was too hard for the NEAT algorithm to do that because it is deceptive. The things that lead to the skull don’t look like skulls, so image matching is not a good heuristic.

There was one caveat to that, which was interesting. With relatively simple images, like some of the easy images from Picbreeder—a crescent, for example—we could evolve to it as a target. But we noticed something weird about it when we succeeded: it was always triple the complexity of the network. It’s important to note that, with the NEAT algorithm, the networks grow in size as they optimize, so we could see the difference in the size of the networks. If there was a success, it would be much worse because it was obviously inefficient compared to the original discovery.

This was just a side observation. We didn’t really push on it too much, but it’s really interesting that the open-ended process—the process where someone isn’t looking for something—seemed to lead to a more efficient representation. Later, when deep learning became dominant, Joel Lehman and I started to think, “Let’s just see what SGD would do if we tried to train a network to output a single Picbreeder image.” You don’t get a bigger network in effect because you just get whatever size network you start with; it doesn’t grow.

But we found we couldn’t start with the same small network that was found through Picbreeder. It was too small and had too few degrees of freedom for SGD because it’s deceptive in that space. You need a lot more degrees of freedom. This is one reason that you can train these huge networks and do amazing things. If we started with a big enough network with SGD, in this case, we could reproduce Picbreeder images. No matter how complex they are, we could do it.

But when we looked under the hood, the representation was terrible compared to the underlying representation we see in the original discoveries from Picbreeder. This is a really weird principle if it actually generalizes, because what it means is that the story doesn’t end with the question of whether you solved the problem. If you solve the problem in one way, it’s worse than if you solve it in another way. We never talk about this in the field. If you solve the problem by being able to output a picture of the skull in a serendipitous way, it has a beautiful underlying representation, which is what the paper shares.

But if you solve the problem through brute force—“I’m just going to move deliberately toward that image through stochastic gradient descent”—it’s terrible in comparison. You may say, “How do I know how terrible? Why do I say it’s great versus terrible? Why would I know this so well?” The really cool thing about 2D images, like Picbreeder images, is that they make it easy to see directly and explicitly what is represented at every single intermediate node. We can query every node in the network, not just the outputs, but every single neuron. We can ask, for every X-Y position, “What are you computing?”

We can draw a picture for every single node in the network and literally see what the representation is and how it builds the final image up piece by piece. It was just stark that what you see from SGD is horrendous. For example, it has no recognition of the symmetry of the images, like the butterfly or the skull. It has no idea that they’re symmetric. It computes these horribly entangled and fractured representations, where pieces of the left and right sides have no coherent organization and have to be re-represented multiple times when they should only be represented once because the image is effectively symmetric.

You can see that explicitly, and you can see that the version from Picbreeder beautifully captures the underlying representation. It doesn’t just capture symmetry; it has an unbelievable modular decomposition. It knows what a mouth is. There are separate controls for the mouth. The mouth can open and close, which is absolutely incredible and mind-blowing to me because it’s like there’s a world model of what a mouth is there without being data-driven. How is that even possible? It’s like, contrary to the bitter lesson, there’s not a lot of data here, but we’re getting world models out of this thing.

I think this observation—that it matters how you got to the solution and how it’s represented under the hood—just hasn’t gotten the light of day until now. It’s a companion to the old insight from the book, which is that sometimes the only way to find something is by not looking for it. But now there’s this caveat: even if you do find it by looking for it, you may pay a steep, steep price in terms of the underlying representation.

Yeah. To play back a couple of things there, there was a paper called “The Lottery Ticket Hypothesis” by Jonathan Frankle, and he said that SGD only works when you have many, many degrees of freedom. When you start with this kind of dense monstrosity, you train it with SGD because it can use those degrees of freedom, and then you sparsify afterward.

We should be clear to say that this isn't a limit of neural networks as a representation. There exists a neural network representation that does this. A CPPN is a neural network.

But the question is: how do we start sparse and build up rather than starting dense, using SGD, and coming down? And then I want to pull apart a couple of things. You're framing this just as an optimization problem, right? I think there are other interpretations for what's going on here, because it might not be contrary to the bitter lesson.

It might be the fact that we humans have had the benefit of billions of years of evolution, so we've had all of the data. We have learned all of these abstract motifs about how the world works, and through a form of interactive imitation learning—I certainly think this is what happened in Picbreeder—we're unwittingly imbuing the model with the representations that we know about the world.

And that explains why, in so few intuitive steps, if you look at the phylogeny for Picbreeder, you can get to the butterfly in something like—was it 5 or 6 steps? So is the optimization process just cleverly telegraphed from the humans, or is it actually learning it in some fundamental way?

Kenneth Stanley

I've had this discussion, or you might even call it a debate, with a number of people. I don't want to claim to know the answer, because I think there's a deep, almost philosophical question at the heart of this where the answer is actually not known. But I think at least it's important to convey that there is another side to that argument that's valid, or at least reasonable to consider: no, the human has not supplied enough information for that to be a satisfactory explanation.

Humans generally—we're talking about dozens of steps to hundreds in the entire run. Think in terms of deep learning: we think about millions or billions of steps. This is absolute peanuts. In the dozens or hundreds of steps—it was dozens to get to the skull, for example—there just isn't enough time to show it things like a mouth opening and closing, whatever that would even mean. That would be strange.

It's also a strange trajectory to take. The human isn't thinking that they're training; that's not the thing in their mind. From a training perspective, you would show examples of closed mouth, open mouth, closed mouth, open mouth—show the space so you can understand that dimension of freedom. People would not do that. That's not how people think; people are choosing things because they look good.

So we don't know for sure. Maybe, inadvertently, somehow, miraculously, they just trawled through the space of all of these dimensions of freedom in just the right way. But I think that's probably implausible, and there needs to be another explanation.

I think it's actually that the open-ended process causes the human to move instead through a hierarchy of regularities, starting with things like the discovery of symmetry, just because humans like symmetry. They see a symmetric object, they like the symmetric object. That locks symmetry into the system. Then from there, they continue to search and find symmetric things that now vary on the symmetric convention that's been established, and add more to that.

And it's a side effect of that process because it develops this hierarchy of representation, which is very virtuous in its structure. A side effect of that is you get some kinds of degrees of freedom, or dimensions, that just happen to align with things in the real world, almost miraculously. We see multiple examples of that in Picbreeder's underlying representations.

My favorite is the apple. The apple, which is in the paper's appendix, has a stem that has a single weight—one weight in this network of, I think in this case, hundreds of connections. One of the weights can actually control the swing of the stem. It looks like a 3D projection of a stem; it moves back and forth like this. There's a shadow that moves underneath it, and this is a single dimension that you can sweep to cause this stem to move back and forth.

Again, I think of it as a hypothesis about the world, or a world model, which has absolutely no data behind it. So it's insane. It's like the exact opposite of Move 37. It's like, play millions of games and then maybe it's not a surprise that something might happen that humans have never seen before that's actually quite valuable. It's an origin of a creative insight.

But this kind of insight that we're seeing here is not from millions of trials, or seeing all of the possible ways the world could work. It's from a very, very limited amount of interaction, and just somehow it's a coincidental alignment with the way reality actually works. But it's happening over and over and over again.

So I think it makes me wonder: when humans have really deep insights, something like the theory of relativity, something like this, which one is it more analogous to? Is it more like Move 37, like Einstein did 10 million simulations in his head of what might happen when certain physical interactions take place? Or is it just that something about the order of the hierarchy of the things that he integrated into his model of the world over his lifetime just happened to precipitate this in the right place at the right time, which actually does align with the world?

I mean, many times it wouldn't, so he's kind of lucky, and we all are when we have a good idea. But nevertheless, because of this virtuous ordering and the way that representations integrate, it can lead to these amazing serendipitous alignments once in a while, which just come out of thin air.

And that might be the kind of thing that we're observing here, which is a different kind of thing. It's not the bitter lesson style of thing. It's a wholly other type of thing, which I think we need to dig into more now that we've exposed it, to understand better what it actually means, because I don't think we have a good understanding yet.

Yeah, let me jump in. It's great to see you again, Kenneth. And Akarsh, nice to meet you.

The reason I want to jump in here is, first of all, this paper is brilliant, insightful, and important. It's visually one of the most beautiful papers I've seen in a long time. I think even people just looking at this will get the insight that you're talking about right now, like with Einstein.

Sure, we're not Einstein, but I think anybody who introspects the way they think, the way they think about solving problems, the way they think about a skull, the way they think about an apple, they're going to find that the images in this paper completely reflect the way we think about the world, the way we model the world, right?

And so, by this kind of open-ended search and interaction with humans, like you said, the neural network is able to grow in such a way that it captures these fundamental world-model aspects. It's really cool. It's really cool.

Kenneth Stanley

Thank you. Likewise, it's great to see you, Keith. This brings back good memories of that old interview that was really fun. I appreciate your comments.

I'm obviously not going to agree with the part about the beauty of the paper, because that's for others to judge. But obviously I agree that this is something we need to think about, what this means. That's why we put it out there.

I wanted to say something about Picbreeder more concretely, really quickly, just to drive the point home a little bit. These people who are on Picbreeder were selecting only images via—they only saw about 15 mutations, and they just clicked one. That means they're injecting a few bits of information into the system, and you end up with a butterfly after a few dozen generations.

From an information-theory point of view, you're not able to put enough information to describe the regularities of the butterfly, like symmetry and the skull. And I think the thing that really makes Picbreeder work—and it goes back to your point about the creative space of all human outputs—is the evolution of evolvability.

People were only selecting for what they wanted, what looked good. In the general sense, in an open-ended sense, they were selecting what they thought would be cool to follow or pursue. But implicitly, there's also selection pressure for evolvable things.

If there are 2 versions of the skull, one that's like spaghetti and one that's very modular and composable, after a few generations of evolution, the one that's more evolvable will be the one that wins out, right? Just like in natural evolution: the evolution of evolvability.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

And this evolvability combined with the serendipity is what I think gives you these nice representations. If you just try to get a skull which moves its mouth up and down, you're just wanting that one type of regularity. One, you're not going to be able to find it, and two, even if you do, the representation is going to be overfit to that thing, right?

What you really want is evolvable in any direction of interestingness, and that's where the serendipity really comes in: you have to embrace whatever the system gives you. If it gives you a regularity of symmetry, or it gives you the moving of the mouth, or it gives you the eyebrows winking, you don't know which one it's going to give you. You just have to see what it gives you and go from there. And I think the combination of evolvability, and in general adaptability, combined with serendipity is what's giving you these nice representations.

I think that has a lot of analogies to natural evolution and scientific discovery as well. And building on top of that, also, really quickly, is parsimony. I guess in deep learning, people are obsessed with the idea of compression, right? They think compression is intelligence, compression is everything.

In some ways, it kind of seems like it has to be true, but in other ways, I'm not sure that compression captures this idea of adaptability fully. You can compress the hell out of the skull and get a very nice compressed representation. That doesn't mean it's going to be adaptable or evolvable in the way that you want, right? In the same way, you can probably compress all the knowledge about the world and your mental model of the world in a very small zip format, or a very compressed thing. But that doesn't mean it's adaptable in the sense that it's going to give you creative leaps and the way to connect the dots, if that makes sense.

Yeah, sure.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Yeah. Yeah.

So we'll be looking forward to your next paper. “Compression is not enough,” right?

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

That sounds good.

Yeah. Copyright Keith Dugger.

But some people say polysemanticity, or grokking, or scale, and it just appears like the neural network isn't grokking it, but actually it is, and maybe if you just trained it a little bit differently.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Part of what's important is just to point out that we've identified a really interesting phenomenon, but we don't have answers to all these questions yet. And part of what we're doing is saying, “Well, let's go find out the answers to these things.” It's possible that some of them help. It's possible some of them don't help.

I think one thing to think about, just to observe up front, is that the difference that you see visually is so dramatic and stark. Anybody who looks at the figure of the 2 different versions of the skull, it's hard to believe, although we don't have the evidence yet, that there's something that's going to fix this that's just real simple, like a grok. I imagine grokking does something good, but it's hard to believe it ends up making something that looks like the CPPN version of the skull from PicBreeder. That's hard to believe.

But nevertheless, it's possible that things like grokking and all kinds of other things that we do, like mixture of experts, maybe convolution helps in some ways, although that's not in the LLMs, but it could be in other networks helping, are all just questions that are not answered. But I think one thing to think about with respect to things like grokking is this: even if you're trying to rationalize and save your faith in how these things work right now and think, “Oh, well, that's okay. Grokking will fix this,” just think about this: wouldn't it be nice if you didn't have to do grokking?

What grokking basically means is that you get this absolutely horrible entangled mess, and you clean it up later, once you start figuring things out. You start deleting all the excess and redundancy; all the fracture gets fixed. Wouldn't it be nice if you didn't have to do that and it was just good in the first place? I mean, what we see because of the PicBreeder CPPN is that that actually is possible.

Right.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Without this weird example from PicBreeder, almost no one would believe or buy that that's even possible. It's like, well, yeah, of course, I'd love miracles to happen every day, but it's just not going to happen. But the thing is that it did happen, and it happens consistently. So it raises this question: even if grokking is helping, is there a better way or maybe a complementary way? Because maybe both factor into why human brains get to where they get to.

These are unanswered questions. This is not to say that grokking doesn't help. We need to check and find out the degree to which it helps. But one other kind of interesting thing to think about is the efficiency of training—how expensive it is. We're building and spending billions, hundreds of billions of dollars creating these giant data centers to do giant kinds of training runs.

It could be that, yes, SGD with brute force will do basically almost anything, at least in terms of being able to get the training data. Obviously, there are implications for generalization from what we're observing here. But at least, if you cover almost all of human knowledge, you could do a lot of useful stuff. And so you just brute-force it into the system and you get this FER representation, which sucks, but it still works.

But the thing that this raises is the question: it's really, really hard, if you're in a point in search space where all the regularities are broken, to get to the next point in terms of higher-level intelligence. You might be able to do it, but the representation is not doing you any favors in doing it efficiently.

Right.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Because all the degrees of freedom that you have are the wrong ones. And so you're constantly overcoming and making up for, and overcoming and making up for. How much cheaper would this whole thing be if we just did these things in a way that actually worked the way we're seeing in these CPPNs?

If it was actually always getting the right dimensions, or close to the right dimensions, that actually align with the modular decomposition of the way the world actually works, then this could be multiple 10X, 100X more efficient in many ways. So even if you still believe we're going to get there with just SGD, this could at least have implications from an efficiency point of view.

I suspect it has more implications than just that, because when we go beyond efficiency to things like creativity, what you're seeing is that the dimensions that have been discovered in the skull, for example, align with new skulls, imagining new things in the world. You wouldn't be able to imagine these new things if you didn't have those dimensions already grokked, so to speak. And so your ability to imagine something new is highly compromised.

And of course, the whole thing—the conversation, the public conversation right now—is turning to exactly this issue of creativity. People are saying, “Well, where is all the new knowledge? Isn't that next on the menu? We're acing all these tests, and it's amazing: Math Olympiad, beating the PhD-level people.”

But people don't get PhDs because we want them to score well on tests. They get PhDs because we expect them to invent something. They're supposed to come up with new knowledge. Where is this going to happen?

And so this suggests that there could be serious implications here, because the ability to invent something new requires, as a prerequisite, that you understand the underlying dimensions of reality. Even if you're going to break those dimensions, as some really innovative people do intentionally—break a rule—it's because they know the rules before they break them that it's interesting. If they just don't know anything and don't understand how the world works at all, they're not going to be brilliant and have these amazing insights.

I think there are really big implications here in terms of both efficiency, creativity, generalization, and continual learning. What I said about wanting to keep learning depends on how you represent the world and how efficient that's going to be. I don't think we can just say, “Oh, grokking might fix it, and everything's going to be fine.”

Let me just jump in on 2 points. So 1 is—and I want to know if you think this analogy is fair—the issue of the problem with fractured representations. I think of it as: if you set out to build a nice UI for an application or something, and instead of starting with a stencil that has triangles and squares and ellipses and whatever, you start with a jigsaw puzzle, you know, it's going to be a lot harder, right, to try and build a nice UI out of fractured, weirdly shaped components. That's a fair analogy, right?

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Yeah.

Okay. And I guess I wanted to ask you: I really see connections between this kind of path dependence that you talk about.

It matters a lot how you got to your state of knowledge and how you got to your representations. I think there’s a direct or deep connection with POET, your earlier paper on this kind of increasingly complex curriculum and environment, where you start off training in simple cases and make them more and more complex. There is a connection there, right? Maybe that’s a simple tool that can be utilized.

For sure. I mean, that’s one of the other factors we talked about that might help mitigate the issue: the kind of open-ended search where it naturally guides the process through a set of tasks of increasing complexity, but in a way that’s divergent. I mean, that’s what’s happening in Picbreeder. So we could extrapolate that that’s actually partly, or maybe largely, responsible for what we see in these representations.

Mm-hmm.

It raises questions about whether there are very different training paradigms from just dumping in all the data in the world in a kind of batch process—

Exactly.

—ones that are more intentionally focused on the chronology being intuitive, or at least aligned in some way with building good representations. It evokes ideas about curricula and things like that. But there’s also a concept of a natural curriculum where the human being themselves, by their nature, tends to learn things in an order that’s actually useful for building good representations.

For example, you take little kids learning arithmetic, and if you start trying to teach them calculus, they’re just going to ignore it. They’re not going to start absorbing it. That’s not the same as LLMs. They’ll take anything you feed them and start to make the connections, which arguably is really unhealthy, because if you start to learn calculus before you’ve learned arithmetic, you’re actually going to create some kind of heuristic version of arithmetic. You can’t avoid it at the same time as you’re learning arithmetic somewhere else in your brain. And this is what causes this kind of redundancy or fracture. You get multiple representations of the same thing, some of them diminished in their capacity.

Well, that’s shown in the mechanistic interpretability paper. For example, the biology paper about where you find arithmetic that’s done in some bizarre way. Like, “I’m going to add 39 and multiply—”

Yeah.

“—or subtract 53—”

Yeah.

“—and add 2.” It’s exactly what you’re saying: a fractured representation of—

Yeah.

I mean, one other really important point I want to add to this is that this may also apply to humans. I don’t want to seem like I’m saying that all humans have unbelievably beautiful, unfractured representations.

No, of course.

We also, I think, are victims of going through things in a bad order sometimes.

Absolutely.

Yeah, so it’s interesting.

Well, I can give you a personal example of that. In high school, when I went to sign up for Physics 1, they put me in the class for people who had not had calculus, and I had had calculus. So I’m in this class and I’m thinking, “What the heck are we doing? We’re just memorizing all these long lists of equations for a cannonball in this situation.” And they said, “Actually, we put you in the wrong class. You have calculus. You’re supposed to be over here.”

So after a week, I switched to the other one. It was so much easier because I knew calculus.

Yeah.

I didn’t need a formula for this specific cannonball situation. I could derive it or just calculate it directly. It was a radically different learning mode.

Oh, yeah, I was going to comment on the biology paper that you mentioned. That’s kind of how it’s doing arithmetic: it’s memorizing all these random heuristics. If you look at one of the circuits they found, it’s as if they’re trying to add 23 and 57: “Oh, yeah, it’s around 55 plus around 25, so it’s going to be around this.”

At the end, it all magically works out, but it’s just a bunch of if statements and paths—

Combinatorially weighted.

—where that gets you there. So if you learn arithmetic like that, how can you adapt that to new situations?

And even with that, we’re not going to get into the computational argument because Keith will go off on one about Turing machines. But there are actually limitations to what you can do. The neural network has to learn some fractured version of multiplication, or whatever it is, because it has a finite amount of computation.

We were just talking about grokking, and the way I understand that is that as the training process progresses, they start off learning quite simplistic, low-frequency representations. Then you train and train and train to grokking, and eventually you learn very high-frequency representations. It just so happens that many of those high-frequency representations are more aligned with the natural factorization of the world. But that’s basically coincidental. There is no principled way to distinguish good representations from bad representations.

I tell them never to use GPT to generate anything because it’s always obvious. But what you can do is write something and discriminate with a GPT model, right? So it’s good at discriminating, not good at generating. Would it be possible to have our cake and eat it? Could we build some kind of bottom-up algorithm that does something a little bit similar to Picbreeder, and every step of the way it’s asking a language model that’s been trained on everything in the world, “Does this look good? Does this look good?” Would something like that work?

Yeah, like a Picbreeder for AI and intelligence, right? I mean, creating an open-ended algorithm that solves this issue, I think Jeff Clune, in relation to our paper, calls it the trillion-dollar question or the trillion-dollar algorithm, because that’s basically like creating a human representation.

If you think that UFRs, or unified factor representations, are akin to a human, then that’s basically like creating a human representation, right? So, yeah. I mean, I guess the answer from my side is that some algorithm should exist, and we’re going to try to find it, but right now we’re still in the preliminary stages of this kind of thing.

Yeah, there doesn’t seem to be any principle that says you can’t do this algorithmically. Presumably, it’s all algorithmic. So hopefully this is just a nudge to have us start looking into it.

You can go on your merry way down the path that we’re going down, just making things bigger and having more and more data, and ignore this. But then you’re at risk of disruption, because if somebody does actually take this seriously and it works, obviously it would have profound implications. I mean, you just look at the 2 pictures of the underlying representations in the skull. If that actually were translated into the world of giant LLMs, there are going to be implications for that.

Yes.

So I don’t think we can just ignore this question. The question of these magical types of algorithms is on the table now.

Yeah, so how can I get in with you guys on the ground floor of answering this trillion-dollar question?

Andrej Karpathy posted on Twitter and said VO3 has just been released, and loads of people had been releasing videos of hairy gorillas in the tundra doing ASMR videos and all this crazy stuff. There was this ideation and curation. People came up with these incredible ideas, and then they generated images with VO3.

Now Andrej Karpathy has said, “The obvious next step is that we kind of do gradient search over the creative space.” So what if Zuck built Instagram version 2, and now the ideation and curation were basically optimized using stochastic gradient descent? The images would still be in a mix of collaborative filtering, so users would be saying, “I like that. I don’t like that.”

My intuition is that would mode-collapse, and maybe it would work better if it were mixed in with human creations. What do you say? That’s a very complicated question. I think you have to disentangle the degree to which the human is part of what you’re calling creative and the degree to which the model is. With VO3, the human came up with the idea of the ape doing the ASMR or something like that. So who’s getting the credit for being creative? It’s not that it’s not impressive.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

I mean, certainly the model did something impressive, but where’s the creativity? And so I think, though, in the larger picture, if you want to ask, can the model totally autonomously, on its own, be genuinely creative—whatever that means, because people are going to disagree about what that means—then is it just a matter of some kind of gradient-following algorithm that’s going to uncover some really interesting, insightful types of creative scenarios that would be really valuable for all of us?

And, yeah, I think it depends on the underlying representation. I mean, that’s the answer: how is the world represented inside the model? Because that’s going to be what determines what the adjacent points are where the gradients can actually get you from where you are. And if the model doesn’t represent the world in a coherent, parsimonious way, then you’re going to find your options more limited.

Not so limited that you won’t be impressed at all. You’ll probably still be impressed, but more limited than a really creative auteur or someone who’s coming up with amazing new genres and a new way of thinking about film. That’s probably not going to fall out of this. And so it just depends on the degree of this creativity that you want. And I do think that I distinguish between—I call it derivative creativity and transformative creativity.

You’re going to get a lot of the derivative style, but a lot less of the transformative style if you have a bad underlying representation. And so that’s the trade-off that we’re working with. It’s still going to impress a lot of people, but the mode-collapse problem, I think, is a real concern here, because as we freeze pop culture in the year 2025 and just live inside of that bubble for the rest of eternity, it’s going to get more and more tiresome.

And so the ability to have pathways out of that, again, will depend on how we represent the world, and there’s a danger with this road that we’re going down that we’re going to be trapped. And we’re already seeing this. I mean, it looks like what’s on the radio doesn’t sound as different from 20 years ago as it did 20 years before that. And this kind of convergence could be accelerated by the kind of stuff we’re talking about here. And ultimately, I think it all boils down to representation.

It doesn’t work autonomously. It’s not creative because it’s not built on the foundation of a representation that describes the world well. So as soon as you leave the thing on its own doing anything with any flexibility, it’ll just go crazy. But I guess the question is, does it matter? We have AI, and we use it in tandem, and we can do great things with it. Is that a problem?

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

I totally agree that, in terms of creativity and the amplification of what humans already can do, AI is hugely valuable in its present form. It can amplify things in really interesting ways. But ultimately, the germ of the ideas is coming out of the human mind.

And so you could be satisfied with that or not. You could say, “This is good enough. I’m happy.” But I think, in terms of what we can’t do because of that, and what we might someday in the future be able to do, the things we can’t do are the things where the human mind isn’t having the ideas.

We’re worried about things from a scientific perspective, for example, like the kinds of things that the new place that I work now, Lila Sciences, is thinking about: automating the wheel of science, where we would like these ideas to come faster to solve the problems of the world. But it also applies to art as well. When are we going to get the next big idea in music that’s going to really shake things up? It’s taking a long time here.

Of course, we can just wait for humans to come up with these things, and if we wait long enough, it might happen. But the question is, can AI accelerate that process, both in the sciences and the humanities? There’s a separate question of whether that’s a good thing, but it’s just an interesting question: can it be done?

And if it can be done, then that’s not what’s happening with current models yet. I think when we talk about representation, it starts pointing us in the direction of how that will ultimately be done: by taking seriously the underlying representations and understanding that, to actually see something novel in the world in a genuinely interesting and deep way, requires you to decompose the world in a special way that’s highly organized and unique.

That’s just not what this fractured, entangled representation seems likely to do, at least as far as we’ve understood it right now.

Actually, this question came up, Ken, in your Doom debate: is survival an objective or a constraint? And, Ken, I think—I want to say that was a brilliant interview. I thought your performance was excellent. I really enjoyed watching that debate. I saw it from start to finish.

I think your communication of open-ended search and this take on it has been sharpened significantly since about 4 years ago, when we talked. I found it really compelling, so I thought that was a great conversation. And in there you say, “Survival of the fittest is not an objective. It’s a constraint.”

It’s a constraint on the system. You have to survive if you’re going to propagate, but subject to that binary—survive or not survive—it’s not part of any objective, I think, is the way you communicated that.

8. Evolution Is Not Optimization

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Yeah, yeah, I think that’s an important point here. I don’t think that the right mental conception of biological evolution is as an optimizer. And this causes endless confusion because, in the field of AI, it causes a lot of confusion. Unfortunately, early genetic algorithms essentially were using selection for explicit optimization.

The metaphor was broken, in my view. Basically, we had the metaphor for selection, but the metaphor is still wrong because that’s not overall what evolution is doing. It’s not selecting just to get to a single target—a single target in the search space that we’re trying to get to, like the solution to one problem.

But the problem is that, in the field of AI, people think of genetic algorithms as effectively an appropriate metaphor for what evolution is. It’s sort of a bad optimization algorithm. People think of it as the poor person’s optimization algorithm, and so they get really dismissive of evolution as a useful metaphor. It’s like, “We played with that in the ’80s,” but nothing really came down that path.

But I think that’s why it’s been damaging: it actually is a very deep metaphor if you think of it, as you said, as a constraint. Think of survival as a constraint, not as an objective. There are many, many things that we could have thought of as objectives, like flight or photosynthesis. These could have been objectives for machines. They would have been single runs where, if they succeeded, we would have celebrated and said, “This is an amazing achievement. We discovered the ability to fly, the ability to process sunlight.”

These would have been achievements that were objectives, but they’re not the objective of the system in the usual sense. And the reason that the system is discovering them is not because survival obviously leads to these things. It doesn’t follow directly at all. You can’t say, “Well, we have an objective and we’re optimizing it. It’s called survival, so of course we’re going to get photosynthesis.” Obviously—I mean, no one could predict this at the beginning.

Right.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

It makes absolutely no sense at all. It’s an orthogonal issue.

Sure.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

It’s sort of like a side effect. The point is that the things that we care about are actually the side effects of the constraint. It’s not directly a consequence of the constraint, but we have to understand it as, like, the side effect is actually the main event. We care about the side effect.

And so there’s a really important point. When we look at it in that light, if we change the metaphor and think of evolution as actually an open-ended, divergent process without a final objective, then we can see that there are huge consequences for, again, things like representation. And I think this is a very important point.

It gives us hope for this idea that there might be an algorithm which isn’t an evolutionary algorithm, which is more like what deep neural networks do, but some kind of learning algorithm in large neural networks that’s more analogous to evolution and would then get these unified, factored representations. Because evolution has done that.

So I claim that the underlying representations in DNA are incredible in a similar way to the skull in PicBreeder. And the reason is that what you see when you have offspring is fundamentally that you get changes. Clearly, your child is not a clone of you.

So you get changes, but the dimensions of variation that are searched almost always preserve the underlying most important regularities. In other words, humans always have bilaterally symmetric children. The problem is that selection is captured by a genetic algorithm: it does the selection part. But what it doesn't capture is this divergent aspect, because genetic algorithms are traditionally optimization algorithms, which are trying to use selection to converge to a point, which is what optimization algorithms do.

But it's a huge digression from what natural evolution actually does, because it doesn't converge to a point, and it's not trying to. It's just diverging subject to the constraints of survival. So we can think of it as a constraint. And the things it actually finds, which would have been objectives if this were a genetic algorithm—flight or photosynthesis, amazing achievements—are just side effects. They're not actually the goal; they just happen as a side effect of having this constraint of survival.

It's a very different algorithm from an optimization algorithm in that sense. The interesting thing about this is that it then expresses an algorithm which is much more like PicBreeder in the way that PicBreeder users are actually searching the space of images. So PicBreeder, in effect, is a much better metaphor for evolution in nature than a genetic algorithm. That's important because we also notice, I believe, that representation in nature is similarly unified and factored, the way that we see it in PicBreeder—in other words, amazing.

The evidence in nature is more indirect, because I can't just look under the hood and see the underlying representations. But there's evidence of how amazing the representation is from the kinds of variations that you see from one generation to the next. For example, the fact that a human being always has a bilaterally symmetric child. If you think about it, this is really remarkable. It seems like such an obvious thing, like it's not even that interesting, but it's actually really remarkable from a genetic algorithm perspective, because preservation of regularities like that is not at all a foregone conclusion.

In a regular genetic algorithm, you're messing things up on every single mutation. You don't get this really high probability of preserving some deep underlying regularity. That would be incredible if you were having such a fortuitous set of offspring that they all preserve some really important aspect. But what's happened in the representation in nature—the way DNA has become organized, the way that the genetic regulatory network works as a hierarchy—is that it's been canalized. Biologists use the word “canalized,” or “canalization.”

It's like it dug a trench into a mountainside and created a canal, so that if there's a change, like a mutation or an earthquake in the analogy, you still get the water to run down through the same canal. And that's the developmental pathway, with bilateral symmetry and the other regularities: you have arms, you have legs. It's true, obviously, there's a miscarriage rate, but it's incredibly low compared to the general destruction of the phenotype that you see in a traditional genetic algorithm.

And there are things you never see, like trilaterally symmetric offspring and things that could happen. If you were using an indirect encoding, or something like an L-system, in a traditional genetic algorithm—which can express body plans—you could get changes like that. It's very easy to go from bilateral to trilateral, or from 5 fingers to 10 fingers, in a single jump; it never happens in biology.

So biology has learned underlying regularities, just like the underlying Picbreeder genome for the skull. There's a strong analogy. Why does that matter? It matters because it shows, as a proof of concept, that there are forces in nature that are not guided by humans which, because of the divergent, serendipitous, open-ended aspect of the search, similarly end up with representations that arguably are approaching what we're calling a unified factored representation in the paper.

And so it gives hope that there may be algorithmic interventions possible independent of human users making selections themselves that could do something similar with representation, because we also see something similar in nature, in a sense, in an automated environment, but not an objectively driven one in the traditional sense.

A lot of people in ML think that we perfectly understand evolution and that it's just like a genetic algorithm. I think Ken would agree with the statement that if we really understood it, then there should exist an algorithm that we can scale up right now that can recreate all the grandeur of evolution, right? All the flight, photosynthesis. There's not a single algorithm that we can scale up and definitively say will do what evolution did, given the current state of genetic algorithms in ML. Would you agree with that, Ken?

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Yeah.

So if that's true, that means that we don't really fully understand it. We understand aspects of it, like Ken's amazing work on open-endedness and serendipity, but we don't have a full picture of evolution. That's why genetic algorithms are nowhere close to a full picture, and we don't really understand everything that evolution is doing.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Agreed. Yeah.

Yeah.

Just for the ML audience out there who claims that we understand evolution fully.

And I wanted to jump in along this line of not understanding evolution completely. I want to put in another dimension, which is very well known in almost anything in the natural world, but certainly with life: energy constraints. There are these very heavy constraints on resources, energy, material, et cetera, that life has to endure in order to survive.

So again, I'm going to beat on my parsimony/simplicity sort of drum here and just get your thoughts on that. It isn't just open-endedness; it's not just divergence. I think there are other aspects to natural evolution. One of these is the constraint to use fewer resources, less energy, and simplicity of solution. Do you think that's fair? Is that another important dimension?

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

I think it's part of the survival constraint too. I think you could think of it as part of the function that computes survival, whatever survival is. To the extent that efficiency is important for survival, then you're going to stay in regions that are relatively efficient. So that is having some effect on representation.

But similarly, in Picbreeder, there's perhaps less concern for efficiency, at least it's not as explicit. But it could be implicit in the sense that users perceive evolvability and like it. Users get a sense that, “Oh, this thing is really malleable, so I'm going to keep playing and exploring in this area.” Maybe that's a side effect of being efficient in some sense, that it's so evolvable. But it's an indirect correlation. It's not exactly explicit that necessarily efficiency equals evolvability. It could be something implicit and indirect.

And so I think overall, as Akarsh said, there's still some uncertainty here about what actually matters and what doesn't matter, especially with respect to the representation. Because we're having 2 conversations at once. One is, what is the best metaphor for biological evolution? But that's actually not the key question. The key question is just what leads to these really nice representations. It might not actually have to be a perfect metaphor.

Right.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

So it's a little bit confusing to disentangle these things.

Well, that's why I was suggesting that, because there are other lines of evidence and a lot of discussion on, let's say, evolution—the literature on evolution itself, on evolution as a general meta-algorithm, not just biological evolution. In order to achieve the degree of variety and diversity that you actually get in life, there has to be an element of shock and challenge.

It can't just be lots of resources around, because then the Earth would have just been—if the Earth had started off as a massive ball of consumable cheese or something like that, it would have just been covered by gray goo that never had any incentive to actually diversify, crystallize into different solutions and various forms.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Well, I'm not as sure, because you don't want too much pressure.

Well, yeah, it's a Goldilocks zone.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Yeah.

It's a Goldilocks zone.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

When you have too much constraint, then you can't do divergence. You have to be safe. You have to play it safe.

Right.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

So you want to have some opportunity to be opportunistic. But if everything survives, then you go to the other extreme, and uninteresting things start surviving.

We've said in some of our publications about open-ended evolution that we think—we call it the minimal criterion—the minimal thing you need to do to be able to pass on to the next generation is that it has to be nontrivial. We think that is important. Once it's trivial—if it's just that you hit a minimal mass, then you get to have an offspring—this is just a thought experiment; it's impossible.

Right.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

But if somehow God intervened and gave you a child just because you hit some mass, it might just create a situation where the world would fill up with inert blobs, because all they need to do is get bigger. And so you do want something nontrivial in these constraints; they enter into the nontriviality of the minimal criterion. So it's a complex question about how to set something up like this, and now we're talking about transferring these insights into the way you train a neural network.

There's another leap of cognitive complexity for us to think through. What is the analogous thing in just a training session where you're being exposed to data?

I like that Goldilocks analogy a lot for evolution because I think Joel and Risto had a paper where you need some sort of catastrophic events to happen in order to get adaptable solutions. Extinction events, like near-extinction events. So that's one extreme. But on the other way, as Ken said, you can't have global competition just everywhere, or else you just end up doing local hill climbing—

Right.

You need some sort of local competition and not global competition to find the balance.

I have to thank you, actually, Kenneth, because when we talked 4 years ago, up until that point, ironically, I was trained a lot in biology and biomedical engineering, but up until that point, I really hadn't thought that much about evolution. Since the conversation with you, I've been thinking about it more and more and more and reading more and more, and just becoming increasingly fascinated at how powerful and amazing this algorithm of evolution is.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Yeah, I appreciate that. That's great to hear. Evolution is just wildly powerful and amazing.

Yeah.

The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

To think of it as a GA is to underrate it and miss what it is. It's a real shame in our field, especially in AI, that you get people making this analogy a lot. I see it on social media, and GAs have done a disservice to our field's understanding of how profound this process is. I mean, it created all of living nature, and it begs an explanation. It's biblical, literally.

Yep. Yeah, and I think there's a vast underappreciation of the fact that evolution occurs at many scales and in many systems. It occurs in psychology, it occurs in memes, it occurs in the information space. It's a very general, almost meta-algorithm, right?

I would encourage people to look into whatever domain they're in, whether there's any kind of evolutionary perspective on it, because it can be very fascinating. A lot of human behaviors that people dismiss as irrational are actually not irrational if you consider them from the perspective of an algorithm operating on an ecosystem of species or individuals or things like that. So it's just fascinating territory.

Kenneth Stanley

PicBreeder is like that. I mean, PicBreeder has a lot of evolutionary properties, and here it is creating a neural representation that has these fascinating properties internally, and that can't be dismissed. The evolutionary properties of what human beings are doing in PicBreeder are obviously part of the explanation.

But it's not that we want to depend on humans guiding the search process. It's whatever they're doing that's somehow analogous to what evolution did, which allows it then to discover these incredible internal representations. We want to abstract that. We want to find the correct level of abstraction to allow this to happen in autonomous fashion so that the training regimens of these large models lead to a similarly amazing underlying representation.

I think the question that looms is how far the representation currently is from this ideal, because it's not totally known. We show in the paper a number of pieces of evidence—indirect evidence—that there is fracture and entanglement inside today's large models. We go through the literature, and we also show new examples.

But the reason I use the word indirect is because, unlike these PicBreeder images, we can't just go in and look at a neuron and know explicitly what it does, because that's what's so nice about PicBreeder images: they're 2D. You can just look and see exactly what's represented. It's more implicit in a real giant, multibillion-parameter network. So the easiest thing to do is just look at indirect evidence of fracture.

There's this overriding lesson here that even when you're getting answers right, you could be messed up under the hood, which further confounds our ability to analyze what these models really know. Because even if they got everything right, they could be just like the network that outputs the skull, which is basically almost perfect, or it is perfect.

It could be perfect, and every question we ask could be perfect, and it could still be an absolute mess underneath the hood. This means that the indirect evidence could diminish over time as the models get so-called better. But it might not mean that they're fixing the underlying representation to be less fractured and entangled.

I think it's super important that you name this concept imposter intelligence in the paper. So I think it's important that we point that out. The idea is that if—sure, if you have enough fractured, entangled representations—you can perfectly reproduce an output that looks identical to the one you get from a unified—

Kenneth Stanley

Factored.

Unified, factored representations, right? The output can look identical, and that's imposter intelligence. Correct?

Kenneth Stanley

Yeah. Another good metaphor is to think of it as an imposter. The representation of the skull is somehow a farce. It's not what you expect. If you just look at the output, it's great. It looks exactly like a skull. But underneath the hood, it's not capturing any of the underlying components or regularities.

In some sense, it's not really a skull. It's an imposter underneath the hood. It's very misleading. And so, if you think of the skull again as a metaphor for all of human knowledge—that's what an LLM is trying to capture—it's not just a single image. It's like an image of all of human knowledge.

For any input, it should output something that's convincingly human. Then it could similarly be an imposter. Underneath the hood, everything could be organized wrong, not the way you expect. It's like a giant charade.

Again, this is very confusing and counterintuitive for people, because people are naturally, including me—I would react like, “But should I really care? When you say it's an imposter, but it's getting everything right and it's human-level, what are you objecting to?”

But the point is that it can still be an imposter, because what we care about here is not just that it's going to get answers right, get good test scores, and seem plausibly human when you talk about things that are in distribution. We want it to be able to go outside—to do things that are creative, to be able to continue to learn, to get to the next level, including learning on its own and getting to the next level.

These are the next frontiers for the field. If it's an imposter underneath the hood, then these kinds of things are going to hit a wall or become insanely expensive. It could be that you can always push through that wall, but the expenses just go up and up like crazy, exponentially worse. I don't know what it means, but it could be something terrible.

We might already be seeing that. The amount of money we're spending here raises questions: Is it necessary? Does it have to cost this much in energy and money? Being an imposter is not a good thing, even though the results look good. That's what it means to be an imposter. We would like to know if that's the case.

We don't have direct hard evidence yet. What we have is some evidence from these PicBreeder images that suggests there's a concern here, but we don't know how bad it is.

Yeah.

Kenneth Stanley

Speaking on the evidence part, this goes into mechanistic interpretability research.

People have been trying to dissect how these models think and operate for the longest time now, right? It’s notoriously hard because they look into these models, and you have things like polysemanticity, where one neuron is activating via ANDing circuits and also black holes or something random. That’s why it’s so notorious. I think fractured and entangled representations are one of the reasons why mechanistic interpretability is so hard, because how are you supposed to interpret something that messy, right? It’s just a hard problem.

I guess this begs the question: neuroscience isn’t easy either, right? If you try to look inside a human brain, it’s not exactly smooth sailing there, either. But is there a better organization for a neural network that is also interpretable? I think we say in the paper that it’s an aspirational ideal. Maybe it’s not possible, but it’s a good direction to move toward, right? At least toward something more unified and factored.

One thing that is lingering in my mind a little bit, and we talk about this a lot in many different ways on MLST, is that in the real physical world, we seem to have this emergent phenomenon of intelligence and even evolution, actually. We could all agree as materialists that you have these physical rules, particle interactions, and dynamics that emerge, and that’s evolution, and it’s beyond our cognitive horizon. We simply can’t conceive of it. So what we do is create these abstractions and think about it with algorithms, and these are idealizations or abstractions. It’s just unimaginably complicated.

But there’s always this question: does it even make sense to think that we could, in a disconnected electronic way, recreate something that had many of the properties of the thing in the real world? Even with PicBreeder, for example, I think we’ve been couching it as an algorithm, but actually it is also a constraint. The magic comes from humans, and humans are still embedded in the physical world to a high enough fidelity that we see the phenomena that we’re interested in.

But even with PicBreeder, does it really have the evolvability that you’re talking about? If humans stopped using it, wouldn’t it just mode-collapse? So there’s always this question: don’t we just need the world, right, to actually give us the type of intelligence that we want?

Yeah.

It could be that a process that yields representations as good as, presumably, what humans have at their best would require some interaction with the real world. But I don’t know if that’s really that prohibitive. We’re getting to the point where computers are effectively directly interacting with the world.

I think it’s interesting that in the field of AI, if you go back 10 years or so, most of the interactions of AIs with dynamic training environments would be in simulations. It’s not the real world directly; the environment is an abstraction of the real world, too. But we’re now getting to a point where these models are actually directly interacting with real-world data.

I think it’s conceivable that they could, like PicBreeder users, simply explore the world itself. Of course, unless they’re robots, they’re not literally out there exploring, but the internet is a proxy for the world. They could explore the internet and gain knowledge in some more natural way, which is more like a PicBreeder user.

The important part is that it allows them to discover an order or a chronology on their own, the way we do to a large extent. We develop curricula for students; obviously, that’s what school is. But it doesn’t account for the first 3 or 4 years of life, which are obviously very formatively important. There’s lots of life outside school, too.

If you take a person who invented mathematics independently of going to school, presumably that person is going to be a better mathematician than somebody who just read the textbooks and memorized them and gets the same exact score on the test. What led you to knowing this is going to affect what you’re going to do in the future. These are things that involve real-world interaction.

I think it’s conceivable that now, with internet access, you could imagine a world where there’s more control on the model side in terms of how it goes off and experiences the world. It’s not easy. It’s not as easy as, “Oh, well, we’ll just dump in all the world’s data and just train.” Obviously, that’s what we do because it’s easier to think of. It’s harder to conceive of what I’m saying, but I don’t see it as impossible, and it could end up a lot cheaper.

One of the offshoots of that could be that it’s multiple orders of magnitude more efficient to happen in a natural way. While we might think it sounds like a tough haul, it might actually be easier in some sense because it is so much more efficient.

To that point, then, do you think that the structure and organization of the internet—or even language, which is a great example—is incredibly diverse? Different cultures have different words for different concepts and so on. We spoke about this before, Kenneth, but one school of thought is that there are natural categories. If you could press play on evolution thousands of times over, you would see this kind of—just as we have morphological convergence in evolution—there would be a kind of informational convergence. Do you subscribe to that?

Kenneth Stanley

Do you mean that if more than one person explored through the world, they would converge to similar categorizations?

Well, I suppose if we had 1,000 parallel universes and we just watched the evolution of language and the internet in all of those different universes, would we see a kind of convergence toward natural categories that are representative of the world?

Kenneth Stanley

Mm. Mm-hmm.

Or the other school of thought is that it’s very constructive and remarkably disconnected from the physical world. I feel that even though there are more degrees of freedom in the virtual world, it’s very difficult to imagine how this works because of the complexity of emergence and whatnot.

Kenneth Stanley

Well—

But there seems to be some kind of grounding.

Kenneth Stanley

Yeah.

So I think that the closer you are to the origin of an open-ended process, the easier it is to predict what might happen. From the very beginning of evolution, whatever that means—the first cell or something like that—you might be able to say, “If we reran this film again, there’ll be something like photosynthesis.” Some early things might be likely, such as algae or something like that.

But the farther you go, I think the less likely it is that you’re going to get the same thing again. Almost certainly not humans again. You might get something as intelligent, but not what we are. It would be something different in some different part of the space.

In my old research group, we had a term for this. It’s not a known term; it was just an internal term, but we called it the “cone of inevitability.” The closer you are to the origin, the more inevitable things are in an open-ended process. As you go outward, the cone expands and there are more possibilities, and it becomes less and less inevitable as you go out.

This would be true starting at any point in an open-ended process. Consider the invention of the internet: what do you think will happen in the next couple of years? It’s probably somewhat predictable and similar to what did happen. But then you say, “What would happen in 10 years, 20 years, or 100 years?” I think it becomes less and less predictable over time, and it will diverge in different ways. That’s a general principle, I think, for open-ended processes.

Really quickly, we know that eyes evolved multiple times in animal species, right? There’s some stuff that’s just so fundamental to how the world works that it’s kind of inevitable how it comes about, like eyes. But I would still argue that there’s so much other stuff that seems unpredictable.

I can’t say for sure, but good luck predicting that peacocks would exist, right? Their beautiful feathers seem like a one-off in evolution.

Kenneth Stanley

There’s also one thing we said in our book: you can think of searching for novelty as a form of information accumulation. This applies to any kind of divergent search, something that continually finds new stuff.

Kenneth Stanley

And so, to find something new, you have to exploit some property of the universe that hasn't yet been exploited. That seems to be a principle. Humans keep doing new stuff, but eventually we have to get off the planet to actually do something new. We exploit what's here, so we have to go somewhere new.

As you wait a long period of time in something like evolution, you're getting a kind of encyclopedia of everything that's possible given the physical structure of the universe. You have eyes because there are photons, and you have ears because there are sound waves. It's almost like our structure is documentation of all the degrees of freedom that exist in the universe as it's constructed. It's almost like an encyclopedia accumulating information.

With that philosophical viewpoint, you might say that there is some kind of predestination to it. We're going to see, over time, as long as it's a divergent process, some exposure of everything that's possible—some kind of... But it's not necessarily the same configuration. There are probably many configurations that can express these different degrees of freedom in functional ways. To some extent, those are the fundamental things that exist and can exist, and so we're going to expose them if you wait long enough. There's some degree of predestination there.

And even if you do—this reminds me of Picbreeder a lot. Another thing I want to say is that everyone in physics always talks about emergence, right? You have simple laws; underneath the hood, it's just super-simple programs and stuff. That doesn't mean that the compositions and the higher-level emergent phenomena are going to be predictable, even if the simple rules are there.

In physics, the Standard Model of physics is relatively simple, right? I don't understand it, but it should be theoretically super-simple. The fact that biology emerges from that, and that chemistry, biology, and sociology all emerge from that, is, I think, unpredictable. The fact that it's not reducible in an intuitive sense means that just because there is an underlying simplicity to it doesn't mean that it's always exploitable.

Maybe if we think about the current state of machine learning and attempts to develop AI, I would like each of you to give a brief recommendation for how you'd like to see the research paths change. What would you change about current research paths if you wanted to push us more rapidly toward AGI, assuming we want to get to AGI? Let's assume maybe that's a goal. Kenneth, starting with you, what's your recommendation to the field?

Kenneth Stanley

9. A New Research Direction

I think, consistent with this paper, we need to understand underlying representations better and figure out the degree to which they actually are imposter representations that don't really capture the things that we expect them to. Then we need to see what the mitigations are. There are a lot of possible mitigations, and we don't really know what their implications are, so we need to do a lot of exploring.

Even beyond the paper, I think it's really important to look at the question of creativity and open-endedness and to take it really seriously. What I find really interesting in the field is that the current focus, or at least up to very recently, has been on chains of reasoning, reinforcement learning, and improving the ability to reason so that you can solve specific problems.

I just want to point out that doing that is useful. There's no doubt that it's useful and impressive. But it's antithetical to creativity. Once I say that what you need to be good at is, if I define where I want you to go, then you can get there, I'm basically training you not to be able to be smart if you don't know where you're going. But that's what creativity is. It's about being able to get somewhere and be intelligent even though you don't know where your destination is.

People struggle with this notion. I think that's why it's not on the radar very much. People often think, if I say that, "If you don't know where you're going, then you can't be intelligent. You have to be random then. What kind of intelligence is that?"

But I totally disagree. That's what the most interesting people are doing. They're saying, "Actually, based on my life experience, if I take this path, I have a gut instinct, and I can justify that gut instinct. It's not completely implicit. Some of it I can explicitly explain. But what I have a gut instinct about is that it will open up new possibilities. It's not that I know what they are yet, but it opens up a new playground, and I understand this."

Mm-hmm.

Kenneth Stanley

By training things to just solve a problem directly, we're training them not to be good at that, which is really funny, because that's the most interesting aspect of human intelligence. It's really underrated that that's what we're good at. We're good at saying something is interesting, to the point where people think it's almost preposterous that you could even think in a situation like that.

If you put a kid in a playground, that's what they're doing. They don't have a goal for the day. We believe that this is not part of human intelligence. So how are we going to train it to have a chain of thought when it doesn't know where it's going that's still intelligent? It's a completely different thing.

Right. Well, that's the great irony of your work and the title of your book, Greatness Cannot Be Planned. I think that's the core irony.

Yeah.

So, Akarsh, what would you recommend to the field? What should we all be doing differently to progress more?

I think one of the high-level things we should be doing is not putting all our eggs in one basket, right?

Okay.

That's the main point of the open-endedness lesson. Obviously, there should be people scaling up these LLMs to see how far the current paradigm can get us, because I'm also curious to see how far it takes us.

So you're applying your principles to the field itself.

Yeah. I asked Risto Miikkulainen this when I first got into AI. I said, "What's your path to AGI? Is it evolution or is it the current deep learning?" And he was like, "I don't know, and I don't want to give you a recommendation," because the point is that none of us are supposed to all follow one thing. We're supposed to all follow what we think is the coolest thing and not just decide, "Oh, we need to put 100% of our resources into this one thing."

In that light, I think there needs to be more people looking into artificial life, Picbreeder, and the ideas from our paper, because I think it's a very promising direction. One thing I'm really curious about is that, even if the current paradigm is very cool and you really like it, it's very different from how humans operate and how evolution operates. It's very different. No one will deny that, right?

We don't just train on IID batches of data for 1 million steps. No one will deny that. I think academia really should try to understand evolution and artificial life a lot more.

Awesome.

I think a lot more investment should go there.

Great.

Awesome. Well, gentlemen, it's been an absolute honor having you on the show. Thank you so much for joining us today.

Thank you. And awesome paper.

Thank you.

Awesome paper.

Kenneth Stanley

Thank you. Thanks to both of you.