关于智能的所有定义都错了。原因何在——Michael Bennett

Tim Scarfe

How close are we to AGI?

It's interesting how much it has stuck around. We have just replaced the pineal gland with a Turing machine.

Tim Scarfe

You're a big fan of what I would call biologically inspired intelligence.

A biological system with a tiny fraction of the energy and training data could do so much more.

Tim Scarfe

Is that fair?

Because whatever that software does has to pass through an interpreter, and the interpreter decides what it does. Consciousness is basically an illusion. One of my supervisors accused me of writing libertarian biology because one of the results of my thesis is called The Law of the Stack, which I like because I enjoy dramatic names.

MLST is proud to be sponsored by Prolific. This is Enzo Blindout.

Yeah, that's that's kind of the the the the goal that we're working towards. So we're trying to make uh human data or human feedback uh or actually any kind of feedback at that uh um we treat it as an infrastructure problem right we try to make it accessible we make it cheaper you see this pattern in almost any company and and even in academic research as well every academic researchers cares about the quality of their data let's abstract it away let's put a nice API around it to make it just like the same way you also do CI/CD or you do model training pipelines we effectively democratize access to this data.

Yeah. Okay. I'm trying to think of my beliefs. Give me one second.

What were they again?

Entirely cognizant of it.

Uh should I look at the camera? Should I look at you?

Look at me.

Okay.

Yeah.

My name is Michael Timothy Bennett. I am a computer scientist who has to use his middle name because there are too many Michael Bennetts in the world. I am interested in understanding AI, intelligence, life, the universe, and the nature of existence, and I spend all my time doing that. I have a side hobby trying to build AI.

Tim Scarfe

I got in touch with you, actually, quite a few months ago. It was when your paper called What the F*ck Is Artificial Intelligence? Did I get the name right?

Yeah, just a couple of extra letters, but yeah.

Tim Scarfe

Okay. That was doing the rounds, and I flicked through it at the time. I've now just spent the last couple of hours reading it word for word, and it's actually brilliant. I do recommend that folks at home read it, especially folks in the MLST audience, because we're a little bit eclectic in our taste.

We are idea collectors, and we like different approaches to AGI, as well as hybrid approaches and a little bit of philosophy, consciousness, and whatnot. So, in that respect, you might be the perfect guest.

Thank you.

Tim Scarfe

This is all very good. In that paper, you were talking about what intelligence is, various approaches to AGI, and also approaches to categorizing them. Tell us about that.

Intelligence is a hotly debated topic. It has been for a long time. I started off with the Legg–Hutter definition: the ability to satisfy goals in a wide range of environments. But as I delved more into biological intelligence and other things, I arrived at a definition of intelligence as the efficiency of adaptation—how sample- and energy-efficient you are.

Later, I found Pei Wang's definition, which preceded mine by several years: adaptation with limited resources. I think that's really succinct and clear. Of course, there are myriad other definitions, but my favorite is Pei Wang's.

Tim Scarfe

Me too. Pei, if you're watching this, I'm sorry we haven't published your interview yet. Basically, the reason is that I loved his On Defining Artificial Intelligence paper so much that we've been meaning to make a special edition on it. I held back his material for quite a while, and we still haven't done it.

Even though it's my favorite definition as well, and of course it informs his NARS framework—his Non-Axiomatic Reasoning System framework—some might say it's almost a bit tautological just to say that intelligence is about adaptation with insufficient resources. In his paper, he went to great lengths to say that we need simplicity and fruitfulness in the definition. It's actually a really difficult thing to get a handle on.

Yeah. I think a lot of people who look at what intelligence is end up writing very long and complicated definitions. Then you spend more time trying to figure out what the definition means than actually thinking about intelligence. That's another reason I like Pei's, I guess.

Tim Scarfe

Yes. Clear. So, as you know, I'm a big fan of François Chollet. For much of MLST history, there was a rule that I was only allowed to mention François's name once per show. We've relaxed that a little bit recently, but maybe that's a good place to start.

How does Chollet define intelligence? You also said that Chollet was inspired a little bit by Legg and Hutter, certainly in terms of the use of Kolmogorov complexity.

His work is very much descended from that way of thinking. He defines intelligence in terms of the ability to acquire skills, which is, I suppose, perhaps a more benchmark-focused version of Pei's definition. That makes sense because Chollet was proposing a benchmark in that paper. He was looking at each test question as a skill, and the ability to acquire it as what it was testing. That makes a lot of sense.

But his formalism, which almost seems like an afterthought in that paper, is where you can see much more of Legg and Hutter's influence. It once again uses Kolmogorov complexity and frames things in the same terms, even though in that paper Chollet goes on to say that he doesn't think compression is sufficient for intelligence. He was very clear about that in the paper, and then he uses it.

Tim Scarfe

Oh, I think that might be because he describes a meta-learning process, which is the intelligence, and that produces skill programs. So the programs are a compression, but the process which created them is doing something more, maybe.

Yeah. I felt like that part was—I chose to focus on the test part at that point. I thought, “Oh, this isn't the bit that he was really focusing on.” I don't know. I'd have to ask him, I guess.

Tim Scarfe

Yes, but you did say something interesting. Maybe a distinction from AIXI is that he thinks of LLMs as being a kind of interpretive collection of skilled programs. He thinks programs are the output of an intelligence system, not the intelligence itself.

With AIXI—and we should talk about what that is—I don't think the concept of a program was an explicit output artifact. It's more a definition of an agent which can succeed in an environment. Is that fair?

Yeah. I suppose a lot of this is semantics, a little bit. The model is a general reinforcement-learning agent. It takes the standard reinforcement-learning framework and tries to make it what you might call an upper bound or a superintelligence based on Solomonoff induction.

Solomonoff induction is a formalization of Occam's razor: if I have 2 explanations, I pick the simpler one. The idea is that AIXI achieves this upper-bound intelligence because, if you accept that Occam's razor is some sort of optimal heuristic that you can use, it does this using complexity, which is the optimally compressed version of a model. If I can compress something more, then it's simpler, and if I just take the most compressible models, I can get the simplest ones.

This is useful for thinking about what a superintelligence might do. Because it's a general reinforcement-learning agent, we can model it out. We can build approximations of it. I disagree with some of the theoretical foundations, and a lot of my publications are about what we could do better, but I find the overall idea very compelling, and it has informed a lot of my work.

Tim Scarfe

Yes, because, from reading your work, I guess you said that if we wanted to create AGI, it would be something that looked like a scientist. If we frame it at the right level, a scientist can generate hypotheses, and they're an agent: they can act in the world, and they're embedded in an environment.

You're framing it at a sufficient level of embedding that you can actually capture the dynamics of the system. In that respect, it's an agent, and it has this principle of compression. Maybe you can contrast it to active inference, because that's quite similar. It's about an agent that balances energy and entropy, sort of like predictive control and simplicity in some natural way. How is that different?

I would love to see someone try to do active inference. I guess active inference has a simplicity bias built into it. There's a regularizer, but the focus is more on explaining something else.

There's a whole bunch of ideas there: you've got the free-energy principle, active inference, all this stuff about Markov blankets, maintaining the border of an organism, and having an internal and external world. The target's kind of different.

Tim Scarfe

But I think that's fair.

We have an agent, and the agent is doing prediction in the environment, and it can act and so on. I see active inference trying to produce a simple model, so there’s this assumption, or principle if you like, that simplistic models, if they predict well, must be good.

Tim Scarfe

Yeah, yeah. This is a very popular, almost orthodox assumption to make, because Occam’s razor does kind of work. But even as far back as 10 years ago, there were people pointing out that this assumption is based on something like Solomonoff induction. It performs reliably within bounds based on the original assumptions, but once you put it in an interactive setting like with AIXI, now you’ve got the subjective notion of complexity that the agent has, which is used for its version of simplicity, because it’s perceiving through an interpreter.

In the case of a universal Turing machine, you can think of it as just an instruction-like language. When I say something in a language, how long it takes me to say it depends on the language I use. If I have some memetic, single-syllable word to describe a complicated concept, then the length of that concept is 1 in my language. In my subjective world, that’s fine.

But if I have an external world that assesses complexity—in the case of Legg-Hutter intelligence, which is a measure of intelligence based on the complexity of the model it comes up with—it’s got a different concept, a different sort of interpretation. You can make it perform arbitrarily well or arbitrarily poorly by shifting the goalposts of interpretation. You can make simplicity completely disconnected from performance, if you like.

Not that that actually happens in reality. It’s not so cut and dried, but it’s certainly not optimal, which is, I think, what a lot of people were hoping for with the original formalization.

Tim Scarfe

Yes, formalization.

And of course, it’s so interesting that, when you dig into this deeply, it just becomes apparent how difficult this problem actually is. Many people might just think, “Oh, yeah, defining intelligence—we’ve got that now, ages ago.” I think there was a distinction as well: certainly, Legg and Hutter were very focused on the simplification of the model, and Occam’s razor. I think Chollet did overcome one hurdle, which is task generality, as well as this difficulty-aware generalization.

So Chollet’s definition is not a general definition of intelligence. It’s very much a specialized definition. It’s intelligence relative to a scope of tasks.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Oh, yeah. The generalization difficulty is the relative entropy from that scope of tasks to the wider scope of tasks. If I understand correctly—and I think it’s been years since I read that 2007 paper by Legg and Hutter—it was about an agent minimizing Kolmogorov complexity that can do well on the expected performance across a wide range of environments.

The definition of “task” here is important. I agree with Chollet that the tasks are what’s important. I just disagree about what constitutes a task. Something like Legg and Hutter’s definition, with the environments and the goals, is the reinforcement learning framework. You’ve got things like actions.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

These are all high-level abstractions that we humans use to simplify the world. As the problem of relative complexity in an interactive setting kind of illustrates, if I use a different set of abstractions to achieve the same ends, I can make something very difficult or very easy.

If I’m trying to talk about tasks, then I need to talk about embodiment as well. I can’t just rely on this idea of a software mind, because whatever the software mind does depends on the interpreter or hardware that you have. You have to look at the system as a whole.

In cognitive science, they’ve got this idea of enactive cognition, which is not just embodied but situated in the environment as well. If you change the environment, then the same goals change difficulty. That’s an idea you can see in Legg and Hutter’s definition, but it needs to be formalized as part of the process of intelligence. If you assume you have a set of actions or assume an environment, you’re bypassing a lot of what intelligence needs to do to solve a task. And so—

It doesn’t make sense to think of a computer program in an absolute sense. Programs have purpose; they are situated in a context, in a world, in an environment. More broadly, you’re a big fan of what I would call biologically inspired intelligence: that we should create intelligence which has properties like self-organization, delegation, causal learning, and all of this kind of stuff, because that’s much more like how it works in the real world.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Yeah. So, to solve the subjective complexity and subjective performance problem, I had Legg and Hutter very briefly as a supervisor during my master’s, and then continued working on that sort of thing as I progressed through my PhD. I wanted to address this subjective complexity, subjective performance thing, and that turned out to be about defining this process of coming up with an abstraction layer.

If you think of the Turing machine with respect to which AIXI or Legg-Hutter intelligence is computed as an abstraction layer, then you’ve got a software mind on a hardware abstraction layer, and then that’s interpreted by physics. In a conventional computer, you’ve got Python interpreted by a C program, interpreted by something else, and it just goes all the way down to hardware.

But it doesn’t really stop at hardware, because hardware is a state of a physical world, and it’s interpreted by whatever physical laws that world runs according to. You could then say, “Well, knowledge of physics is kind of incomplete.” So where does the abstraction end?

If you really want to make an objective claim, or a claim about objective behavior, to be more exact, you need to formalize what must be true of all abstraction layers, not just a fixed subset assuming some basic layer that you can identify. We’re all interacting with the world through our own abstraction layers anyway, so if we want to make claims that generalize to other abstraction layers, it helps to have this framework.

Wait, where were we at the start of this?

No, no, that’s great. That makes sense. Let’s bring in the causality component. In your paper, you were describing almost like single-direction arrows of causality. We have the hardware, the C compiler, the interpreter, and all of this kind of stuff.

Part of what we were saying is that, to build a living, breathing, lifelike system, you need to respect causality. You can’t just take something out on its own. But what I was more broadly interested in is whether it’s always the case that causality goes in one direction, or whether it’s actually quite multiscale and bidirectional.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

It is definitely multiscale and bidirectional. In the same way that cells can network, each cell has its own goal-directed behavior in biological systems, and they can network with other cells in their perceptual field, if you will. They are then constrained by the collective of cells of which they’re part, in the same way that a human within a legal system is constrained by the behavior of the other humans around them. They’re not going to suddenly run down the street naked.

So, yes, there is definitely top-down causation, and we can see it in our own multiscale architecture that we’re a part of as a species.

One really cool thing you did in your paper was draw a plot and describe what it is. You had abstraction on the y-axis and delegated control on the x-axis, and you gave an example of how a centralized form of governance would be in the top left and a free market would be in the top right. Tell me about that.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

I actually have a much better version of that graph that’s coming out in the final version, because it’s provisionally accepted. That’ll be much clearer.

The new version of the graph has food stamps versus UBI to illustrate different levels of delegation of control. These things both distribute resources to members of an entire population, but food stamps centralize control; they don’t delegate control. They only delegate some of the resources, and people are very restricted in what they can do with them. That illustrates the difference between decentralization and delegation of control.

You can think of every system as a stack of abstraction layers. A computer is typically arranged into hardware, machine code, assembly, C, and all this—the stack—but so are human organizations. In a military organization, we’ve got soldiers, then a squad, then a platoon. You’ve got these different levels of abstraction at which you can look at the system, and each layer is the behavior of the parts of the layer below.

In biological systems, the behavior of the parts can be an organ, the behavior of organs can be an organism, and the behavior of a set of organisms like humans can be language.

So you can just move, and you can use this framework of abstraction layers to understand some of the relative advantages that biological systems have. Should I keep going, or—

Tim Scarfe

Well, does that imply that the abstractions are real?

Right. So, you know, I see you as an agent, and I see you as factorized quite neatly into organs, brains, eyes, and whatnot. But if we want to build an artificial intelligence, one approach is to handcraft the abstraction hierarchy; another is to adaptively learn it.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Yeah. Adaptively learning it is definitely the way to go, because we have learned the abstractions we have because these things are useful to us. A chair, for example, is useful to me. It is something that is a cause of valence to me, or is a step on the way to causing some positive or negative valence. A table is the same thing.

I don't have the concept of half a chair that I think about. It's just not useful to me to think about it; I have to combine this other concept of “half” to even describe it. The world is, in every aspect, divided into these simplifications, these classifiers.

If we want to go back to talking about an intelligence or something that builds programs, I'm building all these classifier programs for a television, a light, a chair, and a table. This is all stuff that matters to me. I don't build classifiers for things that don't matter.

You can even tie this in with things like the Fermi paradox: why we don't even notice something that might be classified as intelligent. Its behavior is just not relevant to anything that causes us valence. That would tie in with Mike Levin's work on mind blindness.

Very good. You also spoke about the need for actually learning causal relationships between things. Tell me about that.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Right. I started with a bit of Pearl's work because I was reading The Book of Why and looking at optimal agents. Of course, I was thinking, if it's going to be optimal, it has to learn some sort of representation of its own interventions in the world.

If I want to be able to get food and navigate my environment, I need to be able to tell the difference between, if there's a fly on my shoulder, for example, and the world moves around the fly. There are 2 reasons that could have happened: either my shoulder moved or the fly moved. The fly needs to know which, or it's going to get squished. Same with humans, and in insects all the way up.

Others have already suggested that this is important to the notion of subjective experience, because you need a subject to have experience, right? You need to have an “I” to know that I did something. I started looking at how you would arrive at, how you would construct that as a self-organizing system.

This came into an alternative to simplicity that I was working on for optimal learning. I call it weak policy optimization, or weak constraints, or even—after feeling particularly cocky—called it Bennett's razor. That one hasn't caught on, but I'm working on it.

Well, you never know.

Yeah, Bennett's razor, everyone.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

So we can get this—the point was, if we can define an optimal agent that learns optimally, it must construct this sort of representation of itself. I call it a causal identity for self. It's like, I have an identity that I associate with the causal effects of my actions.

But if I don't start off with a world divided into objects, then I would also do that for other things, like a chair or a television. It wouldn't just be a passive classifier, in the sense that we tend to think of things as value-neutral because it's simple. But these things are not value-neutral if you have a system that is impelled by attraction and repulsion from the ground up, right?

It's not like it's going through an interpreter and having valence attached after the fact. In the case of a biological organism, it's a network of cells, each of which is being attracted and repelled. Each collective of cells within that is being pushed and pulled by attractive and repulsive forces, and the organism as a whole is being attracted and repelled. You can think of this as a sort of tapestry of valence.

It's developing representations and classifiers of the world, not just of itself but of other objects. All of these objects would be the causes of that valence, or in some way causally relevant to what causes valence.

There was scale-maxing, which is the San Francisco thing. And then there's simp-maxing—let's make it simple. So, like, AIXI would be an example of that. And then there was the W-maxing, and W, I think, means world.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Well, it meant weakness, but I just thought it was funny because it was like wind-maxing.

Oh, interesting. What was the interest? You were talking about things like enactive cognition, where we consider the whole environment and everything. Is that roughly it?

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Yeah. So it's like, if I use a sort of dualist approach—when I say dualist, I'm referring to, for anyone who's unfamiliar, Cartesian dualism. It's the idea that you have mental substance and physical substance.

Descartes was trying, in the 16th century, to come up with an explanation of the mind that conformed to church doctrine. He said that you had mental substance, which is all our thoughts and things, interacting with the physical substance of the world through the pineal gland and the animal spirits around the pineal gland. It sort of bumps it, and then it bumps us, and then we act.

This, even in the 16th century, came under some criticism, but it sort of stuck around. It's interesting how much it has stuck around, because we've kind of done the same thing with AI. We have just replaced the pineal gland with a Turing machine.

Enactive cognition is the idea that your cognition is in the world, right? It's not a mental substance. It's not just embodied in the sense of, like, I'm not just a body, but I am part of the world around me. My memory extends into the world. I can write things on a piece of paper, and I enact my cognition by interacting with the people around me and the objects around me.

Yes. I'm so glad you brought this up, because the other day when we were chatting, you were talking about the pineal gland.

I thought, “Oh my God, what the fuck is he talking about?” And yes, you're saying that in the 1600s we had this Cartesian dualism—you know, that the mind and the body are 2 different ontological substances—and people at the time thought the pineal gland was almost like the mediating thing in the brain between them.

Obviously, you're not saying that the pineal gland is the mediating thing. Maybe you are; I don't know.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

I'm definitely not saying that.

Okay, good. Just get that right. But the very interesting analogy is that you're saying there is this thing called computational dualism, and I want to press on this a little bit. Are you saying that any form of functionalism or computationalism is computational dualism, or are you saying this use of a Turing machine in some formalisms is computational dualism?

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

I'm using “computational dualism” more to poke fun at the idea of defining just a software intelligence. If we just have software by itself and don't say anything about the hardware, then it can't really be intelligence if intelligence is measured in terms of performance in the environment.

Whatever that software does has to pass through an interpreter, and the interpreter decides what it does. We can just make it arbitrarily stupid if we want.

Yeah.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

I used that term “computational dualism” as almost like a rolled-up newspaper to whack people on the nose, because I was getting frustrated with repeating myself.

Tim Scarfe

Oh, I like it. I like it. How does this idea relate to—I mean, I spoke with a few folks about mortal computation, right? Turing machines and programs are great because they allow us to think about computation in the abstract and think about programs in the abstract. These are things that can potentially run on any computer, on different substrates, and whatnot. The world isn't really like that.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

No. There are finitely many copies of every piece of software we make. I love the branding—mortal and immortal computation—but there's no such thing as immortal computation. There are just finitely many copies of the software we make.

I get that people might quibble about it: “Yeah, but you can copy the thing.” But in the context of trying to define intelligence, it is a terrible concept. There's just mortal computation. Let's not complicate the matter by adding in an extra concept that doesn't apply.

Yes, yeah. And then, just to help folks understand, mortal computation is that the stuff is the computation. There's no meaningful disconnection between the program and the stuff that does it.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

I've seen a few people give different interpretations of it. The best is probably that of Alex Ororbia and Karl Friston, where they talk about—

Oh yes, I interviewed him, by the way—Alex.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Oh, right. Cool.

Yes. Cool guy.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Yeah. So they've got quite a rigorous definition that extends over several pages. The most common definition I've seen is the internet version, which is just people going, “It's immortal.” But I think the term was first used as an afterthought by Hinton at the end of a paper that was mostly about feed-forward neural networks.

Oh, yeah. Was it his newly proposed architecture? Was it the Forward-Forward algorithm from 2022 in NeurIPS?

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

I think so. I can't remember the name of the paper, but, yeah.

I remember I saw that, and I'd already been writing about embodiment and how you've got to take into account the abstraction layer if you want to have any claims that hold up about performance.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

And so my response to that was to come up with the term computational dualism and write an irritated paper.

Grumpy papers are my favorite papers, which is amazing. Okay. Very good. Very good. And so, in this kind of categorization of different approaches, we haven't spoken about the Silicon Valley scale-maxing and so on. Maybe what drives you? How close are we to AGI? The folks over there think that we've already done it, right, just by scaling.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Well, if you want something that can do jobs and automate a lot of the economy, then sure, we've got some form of AGI. If you want something that's actually intelligent, like a human, though, we do not have that.

A lot of that is because, even just interacting with—and I use things like Grok and ChatGPT—it's not like I'm not interacting with this stuff, but if it were anywhere near as intelligent as a human, I wouldn't have to do all the work that I do. I would be able to offload a lot of it. It's not sample-efficient. I know there are claims about the ARC test, but just interacting with these agents, you can see that it's not really sample-efficient.

I don't know what they did there to get those results on ARC-AGI-1, and I don't know what the claims are about ARC-AGI-2. It's very hard to tell what is true when people don't release the code, the results, and everything that you can puzzle through and work out.

So I think we're probably a good ways off something that really resembles human intelligence, and we need to look at something that isn't just a software innovation, but a hardware innovation. The reason I think that is because we're using these abstraction layers in the form of the hardware we have, purpose-built for very useful, standardized software applications that we can roll out and copy and run on many different computers.

If we want something that's as adaptive as a biological system, we need something that is modular and cellular and efficient like a biological system. We can't be expecting some—I mean, biological systems with a tiny fraction of the energy and learning data can do so much more.

Yes.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Tim Scarfe

Which is great because I like having a job.

Tim Scarfe

Absolutely. By the way, hot off the press: did you hear that Elon released Grok 4? I don't know who released it, but Greg from ARC Prize posted this morning. Greg's a good guy, and apparently it scored about 16%. Muhammad and Jack, I think, are around 15.5% at the moment, so Grok 4 is now in the lead. How do you interpret that? Do you think it's just sort of a bit of memorization?

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

I don't know. I'll have to interact with it, right? Maybe it'll be really impressive, maybe it won't. I suppose the proof is in the pudding. If this starts to do a bunch of really useful jobs across the economy, then we can say with certainty that we're closer to it.

These benchmarks—even the ARC-AGI benchmark—is a great benchmark, right? I've been looking at that my whole PhD. It's great, but it's not perfect. François Chollet would not claim it to be perfect.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Sixteen percent is a great result, but I want to see if it can add long numbers.

That would be a good start.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

I don't know. That's my usual thing. I see a new toy and think, “Let's see if it can add long numbers.” It almost always can't.

Well, devil's advocate: no one's going to argue with you if you say vanilla LLMs are basically databases. No one's going to argue with you, right? But you can add tools to them.

You were starting to talk in your paper about some very interesting hybrid approaches. Obviously, on the LLM side, you add tools, and there's an interesting discussion to be had there about whether you train them with stochastic gradient descent, how they use tools, and what they can do.

But you're also talking about some other interesting approaches. There's the Non-Axiomatic Reasoning System from Pei Wang, and there's the Hyperon system from Ben Goertzel. I don't know much about those systems, but honestly, when you were describing them, it seemed a little bit like they were everything but the kitchen sink. They can do a bit of Bayesian inference over here and some neural networks over there. That could work, but what's your assessment?

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Yeah. Should I go over that too?

Oh, yeah, yeah. Just weave a path.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

I took inspiration from Sutton's Bitter Lesson, where he talked about search and learning, and divided it into two basic tools. We've got approximation, which is what the LLMs are. By definition, they're inexact, and that's really great for trawling through large amounts of data and coping with noisy data, because it's an approximation. You can do a lot with that, and with the computational resources we have, approximation works beautifully.

Then you've got search, which is like iterating through a flow diagram or something like that. That's great for precision, or things like navigation on your phone. When you combine these things, you get something like AlphaGo or AlphaStar. You can use the approximation part as a heuristic to guide the search. You can combine these in many different ways, and these hybrids allow us to create much more effective intelligent systems of some form or another.

Well-known examples of this are things like AlphaGo or AlphaStar, but there are also more comprehensive architectures that are meant to emulate the versatility of a human mind. NARS is a system where you can integrate many different components. I've seen some of the experiments involving NARS and LLMs that were presented at the 2023 AGI conference.

Hyperon is an inherently modular system that's meant to allow you to plug and play lots of different modules. It's meant to be decentralized and adaptable, so you can plug all these things in as they develop. It can include the kitchen sink if you want to plug that in.

I think it was—yeah, sorry. It was more Hyperon I was thinking about. The kitchen-sink one. Maybe you'd do a better job than me at this, but roughly speaking, it's about building up a whole bunch of reasoning about something, then adapting and modifying it over time until it can make deductions and inferences about things.

I remember there was some stuff with time constraints in there, so if it got stuck on something, it would move on, and it would rank things by—it would take into account the resources it had. In practice, it would actually be quite useful. You can put it on a little robot, have the little robot run around the room and do stuff.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Yes.

Tim Scarfe

Um.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Yes.

Tim Scarfe

Which is cool. But they seem to come up with better and better benchmark results every year, and it doesn't seem to get much attention in the mainstream machine-learning space.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Yeah. What do you think about benchmarks, by the way? Grok—I actually interviewed the guy who created Humanity's Last Exam the other day, Dan Hendrycks, and I think it was at 26%. Today, with Grok 4, it's about 46%. What's the point of benchmarks if they're so easily saturated?

Great marketing. I mean, we love measuring sticks. It's nice to have measuring sticks. It lets us know that we're progressing in a direction—whatever direction we point the stick in, I guess. But humans are adaptive. If we set up a measuring stick and it's less than perfect, we will find a way to exploit that.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

I think this isn't necessarily a bad thing. It's just that people should interpret benchmarks as what they are, which is—

Measuring sticks.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Yes. Talk to me about consciousness.

Okay. Well, that whole spiel about abstraction layers kind of led me down a very long and winding rabbit hole with the abstraction-layers thing.

After doing that, I mentioned before the idea of coming up with a causal representation of the self and this idea of tapestries of valence. If you keep scaling up the ability to learn the causal causes of valence, then you don't just get a sort of do-operator for the self or a representation of the self.

Actually, I should start there. That self thing—when I was looking at that in the original paper that I put it in, I thought, well, if you've got this self and it is inherently valenced, and it's made up of sensorimotor activity, wouldn't that be the basis for subjective experience and explain something about consciousness?

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

So I put that in the paper, and people liked it, and I thought, “This consciousness thing is great. I’m going to keep doing this.” I like this, and people aren’t laughing at me, so I kept going. Then one of my supervisors said, “Hey, that’s like my theory. I did a causal self thing,” and pointed me at his paper. His paper was about something called reafference in the insect central complex, and he was trying to show that flies have subjective experience.

This tied back to some work from about 20 years ago where someone was saying that this sort of representation of the self is where human subjective experience comes from. We have something called reafference in the mammalian midbrain, and so many animals have this. It’s what enables us to tell when I am pressing down on the chair versus the chair pressing up on me. This is very useful for causal relations.

Then I started thinking about consciousness more generally and how we could make up our subjective experience with these causes of valence. I was talking about things like the television, the chair, whatever. Someone started beating me over the head with a copy of David Chalmers’s work on the hard problem of consciousness.

Oh yes.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

After a lengthy argument over that, I decided, “Well, now I have to write about it.” So I started writing a paper that ended up as a 70-page paper, much to the dismay of my supervisors at the time, who would say, “Please tell him to stop. Just graduate. Just finish the thesis.”

But I kept writing this, and then it got cut down to a much shorter paper. I ended up bringing on one of my supervisors as a collaborator. I found another supervisor who was an expert in consciousness to come on and help me finish that, and I talked about the hard problem of consciousness.

I proposed to solve it by showing that what’s called a philosophical zombie is impossible in every conceivable world. This is because if you go down all the abstraction layers, you can say, well, every conceivable world must include just change or difference; otherwise, there’s just a sort of universal oneness. As I put it in my thesis, becoming one with the universe is beyond the scope of my work.

So you’ve got a set of states, and you can build up a formalism from that. I argue that this formalism describes all conceivable worlds. If you can show us a philosophical zombie—that is, just to explain what that is—it’s something that’s like you or me but not conscious. It’s in every way identical to you or me: it’s as energetically efficient, it’s as smart, it does everything we do, just missing consciousness.

I got really into philosophical zombies when I read the work of Peter Watts, who is a science-fiction author that writes cosmic horror about philosophical zombies.

Do you want to continue? I’ve got a couple of questions, but—

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

No, no, let’s do the questions, because I can go on ranting for ages.

Well, no, no. I mean, this is brilliant. Yes. So, philosophical zombie. I mean, I love Chalmers. We’ve had them with all of the functional dynamics and behavior, without the little bit extra about the phenomenal component.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

And so, I might be a phenomenal zombie.

So I think what you’re saying is that there’s this kind of lowercase-s and capital-S subjectivity. If I understand correctly, you’re saying that the capital-S subjectivity can be discounted: there are no phenomenal zombies. I don’t know whether you’re saying it’s because there is no hard problem of consciousness, or that consciousness is basically an illusion.

So we should think about subjectivity in terms of purposeful, perspectival representations, right? But we shouldn’t assume that there’s some magical extra realm on top which is called consciousness.

I suppose there’s a lot to interpret there. I mean, if we’re asking whether there’s some mental, nonphysical thing—that is, if we take physical to mean something that interacts directly with the physical world—the advantage of doing that all-conceivable-worlds thing is that I don’t need to talk about physical and nonphysical. I like that, because then I can entertain ideas of magical worlds or whatever, but that’s beside the point.

I’m not saying that consciousness isn’t a thing or that it’s an illusion.

Right?

I’m saying that if we can enumerate a bunch of things and explain a whole lot of the features of consciousness as a necessary consequence of just the state of the environment changing from one to another in all possible worlds, all conceivable worlds, and if one accepts that my description of a conscious organism—without having said that anything is just physical or whatever—is compelling, and I think it is, then there is no conceivable world where you can have something that is as intelligent and acts the way that I do without being conscious.

Consciousness is a necessary adaptation, and the idea of information processing without consciousness is implausible.

Tim Scarfe

Yes. Very good. Very good. So you are saying, basically, that it is just something that happens when you have configurations along the lines that you’re describing. I guess, in a way, I can’t challenge you because you’re already saying they need to be biological, physical, and real.

My obvious retort to that would be, well, a computer simulation of those things obviously wouldn’t be conscious, but you’re not really saying that, are you?

No. There’s one question that, at the end of my thesis, I sort of ruminate about, because it’s—I like this question. I like that I couldn’t figure this out, because it’s fun to think about.

That tapestry of valence I mentioned, with all the cells getting pushed this way and that, means you can say that a conscious state is a tapestry of valence. Now, if I simulate a bunch of cells and give it a tapestry of valence and all the necessary ingredients, like the 1st-, 2nd-, and 3rd-order selves—I only talked about the 1st-order self just before, but you can keep scaling the system up and getting predictions of predictions of predictions of stuff—and if you get all the necessary ingredients for consciousness and you just simulate it in a computer, there are 2 possibilities.

Either a conscious state has to be realized at a point in time by one state of the environment, or it doesn’t. If it does have to be realized by a state of the environment—that is to say, all of the parts of the conscious state are there at the exact same moment—then, in the formalism, that has a much clearer statement. Well, then it’s just humans and organisms and highly distributed systems. You could make a nanobot swarm that’s conscious, but you can’t make a single-thread CPU conscious, because it’s not really doing all this at once.

It’s kind of like looking at a thing: a program counter loads something into a register, does some stuff, and shuffles some stuff around. It’s spread, smeared over time, and so the idea is that smearing consciousness over time would kind of kill it.

The other possibility is that—how would we know? Am I allowed to swear on this?

Tim Scarfe

Yeah, of course.

Yeah. How the fuck do we know if we’re being simulated? We don’t; we have no way of knowing. I think I’m existing at a point in time. I could be running—this whole thing could be running on a single-thread CPU in some kid’s basement.

I’m not endorsing the simulation hypothesis. I’m just saying, I like the idea that there’s an infinite stack of abstraction layers. Maybe those abstraction layers end in some kid’s basement. I don’t think they do. It doesn’t matter. I’m making fun of the simulation thing.

But if it’s smeared across time, if I can have a single-thread CPU load stuff into a register and simulate all the stuff that has to do with consciousness, then I could make populations of humans, or what are called liquid brains of ants.

A solid brain is something like a human brain with a persistent structure that supports a bioelectric abstraction layer that can do information processing—very useful—but it requires a certain stability. It has to maintain its form.

A liquid brain is something like a population of humans that doesn’t have to maintain its form. Its computation is not in the form of electrical signals, but people moving around, doing actions, and interacting with the world. Maybe we’ll network ourselves up; I don’t know. But that’s a liquid brain, and a colony is another liquid brain according to my theory.

If consciousness has to be at a point in time, then a liquid brain is not conscious. If you can smear it across time, then you can have a conscious liquid brain, which is cool but has some weird implications.

Tim Scarfe

There is just one potential objection, which is that a lot of theories of consciousness kind of brush it to one side and treat it as something that is epiphenomenal, which means it’s not causally embedded in the system. We did ask Friston about this, and he had a bit of an interesting response.

He said that phenomenal states are just parts of the generative model. It’s all in there; it’s all kind of part of the causal nexus or whatever. What do you think about that?

I very explicitly argue that a phenomenal state is a tapestry of valence, and that I am not just doing a representation that’s value-neutral. These classifiers of the world, like the television and so on, are me being attracted or repelled from a physical state at different levels of abstraction, at different scales, all happening at once.

Tim Scarfe

So there are a lot of sensations going on there, which is why it’s not just loading a file. I feel something as I try to process this information because I’m being impelled by it. But isn’t consciousness something immaterial, something unobservable? For me, it makes sense to say that it’s a correlate of something physical in your modeling, but it has to be somewhere else, outside of the system.

Yeah. And I guess that’s the idea of a first-person ontology. There’s something that I know by being conscious that you can’t know by watching me. But that’s the point of the abstraction-layer thing, I guess: if I do this formalism of all conceivable worlds and have the luxury of saying that I have this God’s-eye view into everyone’s head, then in the formalism it’s nice and neat from a sort of explanation point of view.

Here, yes, I am stuck in my first-person ontology, and I can only say what I’m interacting with through my abstraction layer. But it’s also kind of like saying that if we buy into the idea that we can’t say anything about things outside of our abstraction layer, we’re basically giving up on science altogether. Why would we draw a different line for consciousness than for everything else when we could clearly draw the same line for everything else?

That’s another thing. I’m not saying that it’s not a line we could draw. We could say, “I don’t know—the television’s on. It just might be on. I don’t know. Prove it to me.” According to my first-person ontology, I can continue to hold whatever belief I like, really.

Tim Scarfe

Very good. Very good. I’ve enjoyed this.

I’ve enjoyed it too.

Tim Scarfe

Yeah. So, D(I)SI, the Diverse Intelligences Summer Institute, was something I found out about 3 days before the application deadline. I thought, “Oh, I should do that.” Cold weather and talking about intelligence. So I have come here and met a lot of people who do everything from machine learning to biology to philosophy, and had very intense discussions about these things for 3 days.

I’m not sure my brain still functions, but I know it’s ticking away there with something. We’re supposed to come up with a project. I have a weird problem: there are too many projects. There are too many good projects.

Tim Scarfe

Yes. But it’s interesting because it sort of ties in with all this. There are a lot of people who are also big fans of Mike Levin’s work.

I’ve been writing more recently. I’ve been writing a lot about the tapestry of valence stuff and talking about abstraction layers. Everything I’m doing has become about that because it’s just what I’m enjoying at the moment.

I’ve been looking at AI safety, for one thing, as: well, it’s not about the AI in isolation; it’s another swarm architecture in which we are a liquid brain into which the AI plugs in. It’s just a part of the organism. Rather than worrying about aligning a policy or whatever, it’s really about designing the system as a whole to accommodate these different components and make use of them, in the same way that a biological system makes use of the resources available to it and then networks cells together to form stuff.

Tim Scarfe

That’s actually very interesting. Yeah. The way biology works is that it is very decentralized, and there’s this delegation and canalization. Weirdly, it seems quite orchestrated even though it is so decentralized. Humans mature in similar ways. We have similar behaviors and so on, but we also have a degree of agency and freedom on top of that.

I mean, if you were to design an artificial or augmented AI system, how could you have your cake and eat it, and have something decentralized and steerable?

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

One of the results of my thesis—one of my supervisors accused me of writing libertarian biology—is called the law of the stack. I like dramatic names, but it’s basically that if you’ve got a high level of abstraction, say some software running in a computer, and it’s learning, its ability to learn and adapt hinges on the ability to adapt at lower levels of abstraction.

There’s a paper coming out soon called “Are Biological Systems More Intelligent Than Artificial Intelligence?” It basically says that, in addition to the waxing thing that biology seems to do better, biology seems to delegate adaptation down the stack. Whereas computers are like an inflexible bureaucracy that makes decisions only at the top, if we want to make something as adaptive and efficient as a human body or whatever, we need to emulate this sort of delegation.

I watched far too many Mike Levin interviews, and he was talking about cancer and how cancer can be seen as a cell that becomes isolated from the informational structure of the collective of which it’s part. It reverts to primitive transcriptional behavior, which basically means it just starts reproducing, eating, and expanding, like cancer.

You could think of it as: under what circumstances does part of a collective system become isolated from the informational structure? I formalized that in this stack thing and said that each cell is like a little task with a little policy. If there aren’t any correct policies available for the overall collective, then the only way for the higher-level thing to continue is to break off some of the parts until there are some correct policies that exist.

There are 2 ways this could happen. One, you impose lots of nasty stuff from the outside. Make the thing hard—make it too hard to have a correct policy. It’s just too difficult, right? So, in the case of a biological organism, like lighting it on fire. Another way to do this would be to impose too much top-down control and cut yourself off at the ankles, eliminating otherwise good policies by overconstraining the members of your collective.

If you think of this in terms of humans, this would be an overly constrained system. This is why my supervisor was making jokes about libertarian biology, because I was saying that if you overconstrain the members of the collective, then they’re going to break off and do crazy shit. It’s like having too much of a totalitarian state or something like that: too many laws, too many restrictions.

I used to live in Italy, and I could see this in the way people used lines. They didn’t line up. They just kind of—

Yeah, yeah. They went towards the front of the shop. There were rules for everything in Italy, so nobody obeyed any of them.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Yeah. Other than that, it’s a great place.

Yeah. I loved living there. As soon as I got over the whole “nobody standing in line” thing, it was fine.

Tim Scarfe

Yeah. And don’t drive.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Yeah. Well, I did a lot of that. It was terrifying.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

I had to stop at the side of the road, and my boss, who was Italian, was yelling at me. He was like, “Why aren’t you driving?” I’m like, “I had to stop. Everyone’s crazy. They’re all driving like maniacs.” I know, I know. It’s insane. Yeah, go on.

The AI—if you put too many constraints on it, you’re more likely to end up with that. You want to constrain it only in the areas where you actually need it to be constrained if you’re trying to avoid some sort of dangerous behavior. Don’t constrain something to a set of impossible circumstances. It’s just going to break.

I’m interested in this concept of what it means to be alive, because we’re kind of dancing around this a little bit. We want to create artificial systems that have the vibrancy, and we could have different perspectives on what that means. Maybe it’s certain patterns of information processing, diffusion, and whatnot.

There are some existence proofs. You look at Conway’s Game of Life, artificial life, and you look at these things and think, “Wow, that seems very lifelike.” I can’t really put into words why it is, but it seems like it is. On the other side of things, I love building distributed, agent-based systems using the actor pattern, even with computers. I love separating things out into autonomous little units of computation that can run distributed, but they’re very much not alive.

They have some cool computational properties, but they’re brittle. Maybe I could make them do metaprogramming, have a little LLM, and it could kind of heal itself and update itself, but it’s still not really alive. It’s knocking on the door of it, but it’s not really where we want to go. There are some interesting approaches, like neural cellular automata, where you do this emergentist sort of optimization. It’s self-organizing and can heal itself from the bottom up, but that’s quite domain-specific. So we’re dancing around this idea of creating systems that are alive.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

There are 2 things I want to talk about there. One is, you mentioned healing. As a simple finite-universe thought experiment, there’s only so much stuff you can cram into a bounded system or finite space. If I’ve got, say, a biological system made up of abstraction layers to process information, then as it delegates control in order to make efficient use of space, it’s going to make weaker constraints take simple forms.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

And so if you delegate as much as you can, you're generally going to get simping with waxing as you scale things up and do this. This means that something that maintains homeostasis, like a self-replicating organism, is going to need to delegate control a lot in order to do this. Being alive almost requires this kind of combination of simping and waxing. I was thinking about this because I wanted to know why things are alive, and I figured this would be a fun side gig with my thesis. I'm waiting for examiner feedback on this.

I thought, well, a rock is something that simps and just persists through simping. By just simp-maxing, as the universe transitions from one state to another, it's going to destroy some objects and preserve others. If something is simple, why would it persist? It would persist because something that's simple is more likely to stumble into a suitable weak constraint, because there is this basic correlation thing we've got going on. Rocks and lots of objects like that in the universe persist by being simple.

Yeah.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

But then if you take something that self-repairs, it's doing the opposite. It is becoming more complex than the same thing would be if it didn't self-repair. It has increased complexity and massively increased its ability to embody weak constraints. I would argue that life is that which waxes at the expense of simp; things which are not alive can just simp.

Tim Scarfe

And we were just talking about healing and self-organization and so on. It doesn't just come from within; it also comes from the outside: interactions with the surrounding environment.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Exactly. I think a lot of what agency and intelligence is about is storing a history of information. It's almost like you've got a hard drive of human knowledge, which is our culture, and all of this is being stored and survives. It's kind of like it's ontogenetically imbued into all of us during our lifetimes. This is why David Krakauer has said that culture is evolution at light speed, right? Language and culture help us to transgress the physical limitations of DNA and physical evolution. So there are just many different ways of thinking about how this machine works.

I sent an abstract of my thesis to David Krakauer last night, and I don't know what he thinks of it yet. I will wait and see. He might think it's terrible.

I think he would be honest with you if it was.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Yeah. But I think these ideas are compatible. I don't know. With the start of my PhD, I sort of saw the Free Energy Principle and thought, “What is this? I don't know about this.” Then I read more and more, and I'm like, “Oh, okay, I can come around to it now. I like it.” So maybe there's something I've missed, but I feel pretty confident in this.

Just to be clear, the “life is waxing without simp-maxing” thing is not something that has gone through peer review yet. I have submitted it for peer review, and we'll wait and see what sort of hate mail I get in return. It's definitely something I would want to run past Friston if I get the opportunity in the future and hear what he thinks of it. Yeah.

Yeah. Very cool. Very cool.

Michael, this has been amazing. Thank you so much.

Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Thanks so much for having me on.

Awesome.