The Fractured Entangled Representation Hypothesis (Intro)

2025 is fast becoming the dawn of a new age of artificial intelligence, an age of miracles. But what if I told you that the AI we know today might not be as good as it appears, that what lies underneath the glorious facade is not really intelligent? It's an imposter. AI can create breathtaking art from a single sentence, write complex code in seconds, and converse with us like an old friend. This explosion of capability has led to a powerful and optimistic assumption: as we scale these models, their underlying understanding of the world will get better and better. And yet—

It's not just that they're beyond human understanding; it's that they're trash.

With conventional SGD, which is the backbone of all machine learning right now, you get a completely different kind of garbage representation—just total spaghetti.

Total spaghetti. So if the internal wiring is a complete mess, how could it possibly produce such brilliant results on the outside? The surprising reason is that it's learned to fake it.

Another good metaphor is to think of it as an imposter. The representation of the skull is somehow just a farce. If you look at the output, it's great. It looks exactly like a skull, but underneath the hood, it's not capturing any of the underlying components or regularities. So, in some sense, it's not really a skull. It's an imposter underneath the hood.

1. SGD Creates Fractured Representations

To get familiar with the imposter, we have to look at the engine driving almost all of modern AI. The dominant method for training AI today is called stochastic gradient descent, or SGD. It's basically a brute-force search, painstakingly adjusting every single grain of sand over and over until its output matches the correct answer, until the thing looks like a sandcastle, basically. It works, but a groundbreaking paper from Kenneth Stanley and his team reveals a big difference between the AI we build today and a different path we could have taken.

By the way, Kenneth Stanley is a hero of mine. He wrote Why Greatness Cannot Be Planned. The special edition show we did with him 4 years ago was peak MLST. But anyway, when you look at these internal representations created by SGD, to put it politely, they're a mess.

Garbage representation, just total spaghetti. We came up with some terminology that we put in the paper to more clearly articulate what these differences are, but basically, you're talking about just amazing versus garbage. The question that the paper addresses is, what does this mean? I think this has endless repercussions and potential implications—the fact that we're basing the entire field on something that produces this complete garbage under the hood.

The paper gives this garbage representation a formal name: a fractured, entangled representation. It argues that concepts which should be unified are fractured and discombobulated into overlapping pieces, and behaviors which should be independent become entangled. It is, in essence, the difference between deeper understanding and elaborate memorization.

I can give you a personal example of that.

Yeah.

In high school, when I went to sign up for physics, for Physics 1, they put me in the one that was for people who had not had calculus, and I had had calculus, right? So I'm in this class and I'm like, “What the heck are we doing? We're just memorizing all these long lists of equations for a cannonball in this situation.” And they're like, “Actually, we put you in the wrong class. You have calculus. You're supposed to be over here.” So after a week, I switched to the other one. It was so much easier because I knew calculus, so I didn't need to have a formula for this specific cannonball situation. I could derive it or calculate it directly.

Very interesting.

You know, it was a radically different learning mode.

2. Replication Versus Discovery

This distinction is really important: basically, whether intelligence can only replicate what it's seen versus one which can go on to create something new.

Two mathematicians can both ace a math exam, and one can go on to become a great mathematician who discovers a lot of things in the field, and the other one can go on to discover nothing. It doesn't give you a picture of what we really care about, which is downstream: how they influence the field and how their research progress plays out.

Today's large language models are the second mathematician. They ace the benchmark test, but they are imposters lacking the deep, structured understanding required for inventive creativity, which is to say, taking the next step forward out of the box. But what if there's another way? The paper discusses another leading paradigm founded in an old online experiment Kenneth did many years ago called Picbreeder.

The Picbreeder system allowed people to effectively breed pictures. We found inside the system that the people who would decide they wanted a certain image and try to evolve that image would fail. Then people who were not looking for anything in particular would discover all these amazing things. The butterfly was the symbol that we used because we put it on the front of the book; it was one of these serendipitous kinds of discoveries. And this led to the idea that there are many things in the world that we're not going to be able to find if we directly search for them because of deception, and that's the underlying reason.

3. Unified Factored Representations

We'll come back to this concept of deception and what it means later, but first, let's talk about this new architecture which improves on SGD. The way these new networks learn is completely different. The representations they create are beautiful. They actually represent the objects at a deep, abstract level. Kenneth and his co-authors called this a unified factored representation. Instead of a tangled mess, the system builds clean, modular, and shockingly intuitive models of the world.

The underlying representations of these images, which are encoded by these neural networks, are absolutely incredible, amazing. And there was no good explanation for how they could be as good as they are. They have unbelievable modular decomposition, which means that it's almost like it was engineered by a person. There's a network that generates the image of a skull, and the network has decomposed it such that there's a component of the network responsible for the mouth that can do things with the mouth, like open and close it, or there's another dimension that can make the mouth smile.

Would you believe me if I told you that this deep understanding materialized bottom-up? It was built brick by brick, as it were, without being trained on a massive data set with billions of free parameters.

Absolutely incredible, mind-blowing to me, because there's a world model of what a mouth is there without being data-driven. How is that even possible? There's not a lot of data here, but we're getting world models out of this thing. This observation that it matters how you got to the solution, how it's represented under the hood, just hasn't seen the light of day until now.

It's a companion to the old insight from the book, which is that sometimes the only way to find something is by not looking for it. But now there's this caveat: even if you do find it by looking for it, you may pay a steep price in terms of the underlying representation.

The most intuitive evidence comes from sweeping the parameters, or the factored representations, as Kenneth would call them. By changing a single connection in the network, you can actually see which factor of variation it represents. In this new type of network, sweeping these values results in a commensurate semantic change. It might be opening the mouth on a skull, winking the eye on a face, or swinging the stem of an apple. It's like the network understands what these objects are at a deep level.

In conventional networks, the same action just produces meaningless, chaotic distortions. This is what we mean by the imposter. And in case you didn't get the memo, this is basically how ChatGPT works now. We needed to have a huge number of free parameters in the network to make it trainable, right? To make it statistically tractable. But it's precisely that reason that we end up with a sandcastle. It looks like a castle, but it doesn't have any structural joints. It doesn't look anything like we know a castle to be.

4. Deception Guides Discovery

Why is one network a sandcastle and the other one the real deal? The secret lies in abandoning the fixed objective during training and building bottom-up, not chipping away top-down the way SGD does. We also need to embrace a counterintuitive notion called deception.

Deception means the stepping stones that lead to these interesting artifacts that you might want to find don't resemble them. If you have an algorithm that's trying to follow a gradient by matching closer and closer to the objective, getting a higher and higher score, you're going to get stuck in a dead end because of deception, because the things that lead to the thing you want actually don't look like the thing you want. And this is true in the lineage of many of these images in Picbreeder.

The paper showed the path to the skull in Picbreeder. The key idea is that sometimes the stepping stones which lead to something important don't even resemble the thing you end up discovering. It might seem like total serendipity, total randomness, but humans have a nose for what's interesting, which has a lot to do with the foundational cognitive prize which nature has bestowed to us through constraints in our evolution and physical environment, and of course, our life experiences on top of that.

On the road to getting an image of a skull, they were not thinking about skulls. When they discovered a symmetric object, like an ancestor to the skull, they chose it even though it didn't look like a skull. That caused symmetry to be locked into the representation. From then on, symmetry was a convention that was respected as they searched through the space of symmetric objects. Somehow, this hierarchical locking-in over time creates an unbelievably elegant hierarchy of representation.

This hierarchical locking tells us something really important about how representations emerge. We think it's about finding the right building blocks now, but weirdly, it's about making future discoveries more likely. It's a bit like how good code now reduces technical debt in the future. Bad code is the sandcastle. And there's an evolutionary principle which makes sure that these superior foundations win out over time.

PicBreeder, I think what was especially at play was the evolution of evolvability. When people are only selecting for what they wanted—what looked good—there is also implicit selection pressure for evolvable things. If there are 2 versions of the skull, one that's spaghetti and one that's very modular and composable, after a few generations of evolution, the more evolvable one will be the one that wins out, right? Just like in natural evolution: the evolution of evolvability. This evolvability combined with serendipity is what I think gives you these nice representations.

The thing that I think makes this really intriguing is that it gives you something that otherwise could never exist: a counterexample. There actually do exist networks that don't have that issue. You would think that that's just intrinsic to neural representation, that somehow they just look like entangled messes, and that's just the way life is. But clearly, it is not how life has to be.

5. Open Endedness Shapes Intelligence

This leaves us with a choice: the path of a singular, goal-oriented kind of optimization, which creates brittle, fractured imposters, or the path of open-ended exploration, which ostensibly creates robust, unified models. They argue that this choice fundamentally impacts 3 important things that we want from AI: generalization, creativity, and continual learning.

If you think of the skull, again, as a metaphor for all of human knowledge—that's what an LLM is trying to capture—it's not just a single image. It's like an image of all of human knowledge. For any input, it should output something that's convincingly human, but it could still be an imposter. It could be the same underneath the hood: everything could be organized wrong, not the way you expect. It's like a giant charade.

This is very confusing and counterintuitive for people because people are naturally inclined to react this way. Even I would react: “But should I really care? When you say it's an imposter, but it's getting everything right at a human level, what are you objecting to?” But the point is that it can still be an imposter because what we care about here is not just that it's going to get answers right, get good test scores, or seem plausibly human when you talk about things that are in distribution.

We want it to be able to go outside, to do things that are creative, to be able to continue to learn, to get to the next level, including learning on its own and getting to the next level. These are the next frontiers for the field. If it's an imposter underneath the hood, then these kinds of things are going to hit a wall or become insanely expensive. It could be that you can always push through that wall, but the expenses just go up and up, exponentially worse. I don't know what it means, but it could be something terrible. We might already be seeing that. The amount of money that we're spending here raises questions: Is it necessary? Does it have to cost this much in energy and money?

6. Creativity Defines Intelligence

So this flips everything on its head, right? The very thing we're trying to control—the objective—is a bottleneck for the thing we actually seek: creativity. True creativity might even be intelligence. It's certainly the nearest quantity I can think of to describe what intelligence is.

Once I say that what you need to be good at is getting somewhere that I define for you, then I'm basically training you not to be smart if you don't know where you're going. But that's what creativity is. It's about being able to get somewhere and be intelligent even though you don't know where your destination is.

The biggest risk may not be that our machines become too intelligent, but that we become too narrow in how we define intelligence. The blind pursuit of benchmarks and performance metrics might actually block us from discovering the real thing.

I think one of the high-level things we should be doing is not putting all our eggs in 1 basket, right?

Okay.

That's the main point of the open-endedness lesson. Obviously, there should be people scaling up these LLMs to see how far the current paradigm can get us. More people should look into artificial life, PicBreeder, and the ideas from our paper, because I think it's a very promising direction.

We need to build an AI which doesn't regurgitate patterns from its training data, but actually understands the deep structure of the world. An AI that can look at new scientific challenges, that can discover entirely new principles. The path to artificial intelligence is not a straight line towards a known destination. It's a divergent, unpredictable, open-ended search into the unknown. It's possible that the most important discoveries that we will eventually make will be the ones we aren't even looking for now. And by the way, folks, if you want to watch the entire roughly two and a half hours worth of goodness with Kenneth and Arkash, his, uh, co-author at MIT, um, yeah, we will be releasing that pretty much on our next episode. So hopefully this has whetted your appetite for that. Cheers.