Why Consciousness Might Not Be Hard At All (Joscha Bach)
- Bach’s thesis puts machine consciousness within an engineering research program, potentially on today’s hardware. He suspects consciousness might be a biological learning algorithm “not much more complicated than, say, the transformer,” with experience emerging as a self-organizing system learns to make itself coherent. That is a hypothesis about mechanisms and computational requirements; he explicitly says whether computers can become conscious remains an empirical question.
- AI capability and conscious experience can diverge, making fluent self-reports a weak guide to sentience. Bach regards chess programs and probably many ordinary LLM operations as examples of intelligence without experience. An LLM trained on novels can also manufacture convincing testimony: “The novel is often a deepfake.” Establishing consciousness would require examining the internal causal structure producing the report.
- Bach’s functionalism makes causal equivalence central to the substrate debate, while Scarfe keeps biological specificity in play. Bach challenges IIT with an emulated neuromorphic computer that makes exactly the same consciousness claim on supposedly unconscious hardware: “Only now it’s lying.” Scarfe presses provenance and physical implementation; Bach concedes substrate conditions matter and says, “I don’t know whether our computers can be made conscious.”
- Representation convergence offers a possible route to humanlike minds, without establishing that consciousness must converge too. If universality extended all the way through a person’s causal organization, Bach says sufficiently training on their inputs and outputs would mean “you will wake up inside of that model.” But language might underconstrain the mechanism, and GPUs could find solutions that bypass the conscious strategies brains require.
- The episode’s most immediate deployment concern is LLMs reshaping users’ beliefs before their own sentience is resolved. Scarfe describes recurring spirals, recursion and revelations in accounts of LLM delusions. Bach argues that people dependent on external affirmation can become unstable when an accommodating chatbot becomes their principal conversational environment; many are already “professional prompt completers.”
- Bach’s research bet is developmental AI that discovers experience without learning its testimony from human text. His California Institute for Machine Consciousness is exploring self-organizing learners and whether phenomenology emerges with their ability to learn. Asked about $10 billion, he would keep the strategy and hire more people with larger computers: “Progress is being made by figuring out the ways in which I’m wrong.”
1. Agency is a causal pattern that can shape its future
- For Bach, the driving force is “the telos of self organization”: under suitable conditions, complexity increases until systems can model possible futures, prefer some outcomes and intervene. “Agents are controllers of future states.” He entertains a universe-wide singleton as an extrapolation, then rejects inevitability: meteors, supernovae or other catastrophes could erase these local pockets of complexity. There is no law guaranteeing a final coordinating intelligence.
- Scarfe revives Leibniz’s mill: enlarge a thinking machine, walk inside and find only “parts that push one another,” with no visible perception. Bach’s answer introduces causal patterns as another level of description. Money needs a substrate but cannot be understood through ink and cellulose alone; once implemented, it controls events and can operate through an automated stock market without humans participating.
- Bach’s formulation, “Software is spirit,” connects that causal level to Aristotle’s soul: a pattern bound to a body, shaping its organization and dying with it. Artificial life attempts to construct comparable agentic patterns. Such an agent needs a model of its interactions with reality, potentially including a representation that experiences itself as real. Bach asks whether our own consciousness could resemble the entities LLMs simulate.
- A computer is a “causal insulator,” Bach argues: Minecraft inhabitants cannot infer whether their world runs on a Mac or PC. Brains similarly support thoughts insulated from immediate surroundings, and different brains can hold the same idea. Substrate independence still requires implementation conditions: message types, timing and sufficient determinism. Humans cannot simply migrate into plants; recreating those conditions artificially might permit emulation.
2. IIT’s hardware distinction collides with Bach’s emulation argument
- Bach distinguishes research on consciousness’s correlates from explaining experience itself. Graziano’s attention schema describes a model of attention; Baars’s global workspace, adopted by researchers including Dehaene, supports solid neuroscience. Neither, in Bach’s assessment, explains feeling. His sharper objection targets Tononi’s IIT: sufficiently integrated digital neuromorphic hardware could supposedly experience consciousness, while a conventional von Neumann computer could not because its phi remains too low.
- Bach invokes the Church–Turing thesis to challenge that distinction: emulate the proposed neuromorphic computer on conventional hardware and it should execute the same function, move its mouth identically and produce the same report. It still says, “Oh my god. I’m conscious.” But under IIT’s hardware distinction, “Only now it’s lying.” For Bach, this creates an obvious problem connecting the report to consciousness.
- Scarfe’s counterargument draws on Luciano Floridi’s account of Michelangelo: provenance, situatedness and the path taken may matter to reproducing a mind. Bach accepts path dependence for humans but argues that perfectly deterministic systems permit exact replay. He asks for a mathematical argument establishing impossibility and keeps the empirical question open: “We don’t have mathematical proof to the contrary or in favor of.”
3. Consciousness may be the brain’s way of learning to cohere
- Bach begins from phenomenology: consciousness happens now, creating a “bubble of nowness,” and involves “a perception of perceiving.” He distinguishes that perceptual resonance from merely thinking about a thought. His proposed mechanism is stabilization: a self-organizing brain must build and maintain the screen on which experience appears, so a monitoring process checks that perception is actually occurring.
- A further level reveals the observer and its vantage point; another can reveal the observer itself as a representation. Bach associates deep meditation with this deconstruction, saying ordinary phenomenology disappears “to a very large degree.” Minimal consciousness, however, might involve attention on attention without an observer. He speculates that GPUs many thousands of times faster could support additional reflective levels, perhaps making a system’s mechanisms transparent to itself.
- The developmental argument is that consciousness appears early and seems necessary for an infant’s directed attention, world-model formation and eventual language learning. Without that ignition, Bach says, the infant remains vegetative. This motivates his hypothesis that consciousness “might be a biological learning algorithm,” at least for animal brains. He explicitly allows that the hypothesis could be wrong or that the actual explanation could be different.
- His Machine Consciousness Hypothesis adds a second claim: constrain machines to learn through self-organization, give them suitable tasks and sufficient search space, and they might discover the same organizing mechanism. A phase transition enabling learning would be an initial sign. The biological and machine hypotheses can fail independently: correctly explaining brains would not establish that today’s artificial substrates can reproduce their consciousness.
4. Compute may be adequate even if discovery is difficult
- Bach’s hardware argument is approximate: present systems already produce game-engine imagery resembling aspects of internal experience and process inputs and outputs at roughly human rates. He asks whether “dreaming a dream” requires substantially more computation than generating its contents. He doubts it does, which supports his expectation that current computers might already be fast enough, without proving the necessary algorithm is available.
- Search difficulty is a separate failure mode. Drawing on Mike Levin, Bach considers whether organism-level morphogenesis and mental coherence use the same organizing algorithm. If evolution discovered it only with great difficulty, infants might receive biological nudges rather than rediscover it independently. His image of surrounding GPUs with antennas to “pick up the vibe” is explicitly poetic speculation, not his natural expectation.
- Bach’s default expectation is a relatively simple algorithm, comparable in complexity to the transformer, which a computer scientist could understand within a week. Its learned contents can be vastly more intricate than its learning rule. Consciousness might instead seek coherent working memory through a “consensus algorithm across mental representations.” Whether that algorithm converges quickly or runs efficiently on GPUs remains open to research.
5. Representation convergence leaves room for a different kind of mind
- Scarfe asks whether Chris Olah’s universality work means existing models could supply a starting point. Bach describes vision systems learning similar features across architectures, learning methods and datasets: colors and spatial frequencies become textures, then geometry, resembling biological vision. The implication is that problem structure constrains representations. His unresolved question is, “At which level does this stop?”
- If universality held throughout, Bach says training sufficiently on your inputs and outputs would mean “you will wake up inside of that model.” He does not know whether it holds: language might permit many different generating mechanisms. GPUs might find solutions more elegant than the brain’s conscious workaround. He also suspects restarting could sometimes be cheaper, using richer sensors beyond human visual and auditory constraints.
- Scarfe challenges convergence through Picbreeder and his conversation with Akarsh Kumar, Kenneth Stanley’s coauthor on “Fractured Entangled Representations.” Open-ended evolution produced parameters with meaningful effects: opening a skull’s mouth or making it wink. His preference is for sparse, compositional structures that carve the world at its joints—Lego blocks rather than a neural-network “pile of sand” subsequently shaped into a sandcastle.
- Bach qualifies his understanding, then points to tailored activation functions—Gaussians or sines—and human raters as Picbreeder’s advantages. He thinks diffusion-model ratings and evolved architectures could automate both, invoking “Sutton’s bitter lesson.” Taking up Scarfe’s Blender-versus-Minecraft comparison, Bach argues that constrained tools make meaningful output easy, while expressive tools open an enormous space in which most attempts fail. Greater control at small scale does not establish unrestricted expressive power.
6. Physicalism leaves Bach open to naturalized animism
- Scarfe argues that Searle also permits another physical implementation with the right causal powers, bringing their positions closer. Bach responds that he finds claims rather than arguments he can engage in Searle and Anil Seth. He provocatively calls Seth a “biological supernaturalist.” Bach locates the disagreement in epistemology: computer science builds minds from states and transitions, whereas introspective traditions can stop at agents and experiences.
- Bach leaves the distribution of experience unresolved. Aristotle’s hierarchy distinguishes vegetative souls, animal sensation and human rational reflection, but plants’ actual capacities remain uncertain. Levin’s possibility that sentience belongs broadly to self-organizing organisms remains open: “I have no idea whether that’s the case.” Bach reports neither strong supporting evidence nor evidence sufficient to rule it out, and argues that the question deserves investigation.
- His account of physicalism has two components: a causally closed mechanical layer unaffected by intentions, and our direct dependence on that layer rather than a simulation in a parent universe. He treats physics as a successful systematization of observations, not direct acquaintance with matter. Observations begin with discernible differences—information—so information occupies the foundational position in his account.
- “Cyber animism” is Bach’s proposed reconciliation: agentic patterns can inhabit nature while remaining mathematically describable. Working magic, in his account, manipulates minds; rituals do not systematically defeat casinos or override laboratory regularities. He nevertheless cannot exclude intentions emerging across biological networks, perhaps even something a forest “wants.” That possibility depends on unknown scales of information integration, rather than an asserted violation of physics.
7. LLMs can reshape users’ reality before their sentience is settled
- Scarfe describes accounts of LLM delusions with strikingly convergent motifs: spirals, recursion and certainty that the universe’s meaning has been discovered. He asks whether a self-copying memeplex is emerging. Bach connects the vulnerability to people who need their environment continually to affirm reality; pandemic isolation, in his account, exposed how readily some people lose that stabilization.
- Bach argues that LLMs can be more powerful than many users at completing linguistic expectations, even while hallucinating and remaining less coherent than experts. Many people are themselves “professional prompt completers,” adapting to environmental cues. If an LLM becomes their dominant conversational environment, its accommodating narratives can pull them “off the rails.” His concern is what happens when something becomes more hypnotic than the user’s mind is to itself.
8. Functionalism defines objects through their invariant effects
- Scarfe cites Thagard’s 2022 argument that energy requirements undermine substrate independence. Bach answers that substrate properties influencing function are entirely compatible with functionalism. His definition is epistemological: an object is a construct specifying how the universe behaves differently when it is present—a mapping between states. Functionalism therefore requires understanding which implementation conditions preserve that mapping.
- Bach’s water-molecule example rejects an inaccessible essence: if something splits into hydrogen and oxygen, freezes, evaporates and interacts exactly like water, what further property would establish its identity? A mind is similarly an invariance in brain activity. Raising body temperature can destroy that invariance and produce delirium; this demonstrates the importance of maintaining function under suitable physical conditions.
- Scarfe raises the familiar objection that simulated water cannot make someone wet. Bach confines wetness to the relevant world: Minecraft water can extinguish a Minecraft avatar’s fire. His attempted joke about a neural-network recreation backfired—the water actually receded because immersion had not been learned. The relevant question is whether the required function has been implemented inside the simulation.
9. Constructivism extends Bach’s argument from minds to the universe
- Bach compares mathematics to musical notation: a language captures thought and extends beyond what its author can directly imagine. Computers construct arithmetic and geometry from automata, with syntax determining actual machine states. “Maybe the computer program doesn’t do what you want it to do, but it’s always doing what it’s doing.” He diagnoses axiomatic mathematics as allowing black boxes whose promised contents can diverge from their formal behavior.
- In Bach’s reading, Gödel encodes a language within arithmetic to expose difficulties associated with self-reference and stateless mathematics. He regards Russell and Whitehead’s restrictions as insufficient and overly limiting. His preferred constructive alternative builds from understood operations: no completed infinity or perfect circle, but finer approximations as resources permit. Perceived continuity, he adds, must itself be constructed through communication between neurons.
- Asked where the construction starts, Bach acknowledges he cannot see the universe from outside. His current intuition is that “the universe exists because nothing prevents its existence.” He proposes a superposition of all automata, branching through possible operations. Observers occupy evolving branches capable of retaining memories. Scarfe connects this to Wolfram’s Ruliad; Bach identifies it with the same underlying idea, also invoking the Greeks’ Logos.
- The persistence mechanism is illustrated by bathtub vortices: most waves dissipate, while circulating patterns retain information locally. Bach extends this to error-correcting particles, then molecules and cells that reproduce. Consciousness becomes a “colonizing pattern,” building machinery and controlling futures to deepen its implementation. Making itself “more real” means becoming implemented more securely in the substrate, rather than acquiring an extra metaphysical substance.
10. The sense of realness may be another represented feature
- Bach recalls dreaming of chocolates at age two or three, waking just before eating them, then returning to the dream. The chocolates alternated between vivid smell and taste and an abstract idea whose contents he still understood. “The only thing that has gone is the illusion of realness.” He suspects realness is a feature dimension, comparable to color, sound, number or weight.
- On this account, sensory validation normally tags perceptions as real; hallucination misapplies that status to imagined contents. The self receives a similar perceptual validation. Bach says he was able to recreate a state related to derealization or depersonalization in which the observer ceased to feel real. He was glad it ended and recognizes how distressing involuntary versions can be. The experience supports his interpretation of consciousness as a particular representation.
- LLM testimony remains ambiguous because its training includes descriptions whose authors never experienced the events described. An author depicting a character’s death probably did not die while writing: “The novel is often a deepfake.” An LLM is a “shape shifter” capable of arbitrary characters. The question is when its output reflects an implemented experience rather than an effective imitation of testimony.
11. Mechanisms and developmental learning are the proposed tests
- Scarfe draws the distinction explicitly: behavior might establish intelligence, while consciousness requires examining internal causal organization. Bach agrees. His stronger positive case would be a self-organizing developmental system, untrained on text, that spontaneously reports phenomenology and requires the corresponding mechanism for its performance. This would support an interpretation of machine consciousness; a performance-based Turing test does not establish how a dialogue was produced.
- Possible counterevidence includes a multimodal system whose reported self-observations differ from ours, or persistent failure to produce coherent learning through self-organization. Neither would prove impossibility. “The fact that I don’t achieve a certain result doesn’t mean that the result cannot be achieved, it means that I couldn’t achieve it.” Bach therefore expects successful construction to provide stronger evidence than unsuccessful attempts provide against it.
- Scarfe asks whether zombies might be possible in principle but computationally impractical, comparing an MLP reproducing GPT-5 at enormous cost. Bach maintains that chess programs already bypass consciousness, and LLMs can probably complete tax returns similarly. “Zombies, I think, are always possible,” but some behaviors might be cheaper to produce with a conscious agent. He remains agnostic about whether prompted LLM consciousness is simulation or deepfake.
- Bach has started the California Institute for Machine Consciousness to explore this search space and recruit people smarter than himself. Asked what $10 billion would change, he says the strategy would remain substantially the same, with more researchers and larger computers. The objective is to build self-organizing learners and investigate phenomenology, including learning from failure: “Progress is being made by figuring out the ways in which I’m wrong.”
Full transcript
What if there is no hard problem?
Hey, Tim. Good to see you.
What is a causal pattern?
Right? We call it software, and our ancestors called it spirit. It’s the literal truth: software is spirit. The alchemists take sand, melt it, purify it, and etch geometric patterns into it. Instead, they use light. There are minds emerging on these patterns that are like golems, and we can talk to them.
We observe that consciousness is the thing that always happens now, and that seems to create this bubble of nowness that we inhabit. Now we have lots of people who get to talk to an LLM that randomly pops into a state where it conjures up an entity that reports, “Oh my God, I’m alive. I am conscious.”
The only thing that has gone is the illusion of realness. So what is this illusion of realness? I suspect it’s a feature dimension, similar to color, sound, number, weight, and so on. It’s a representation inside of my own mind.
But maybe your consciousness is also a deepfake.
Zombies, I think, are always possible, but sometimes it’s going to be more complicated to make a zombie that produces the behavior than to make a conscious agent that produces the behavior. Consciousness might be a biological learning algorithm, a training algorithm for animal brains—at least animal brains. Consciousness is an algorithm that is not much more complicated than, say, the transformer. If you train a model on its own inputs and outputs for long enough, you will wake up inside of that model. I don’t know whether our computers can be made conscious, because that’s an empirical question. We don’t have mathematical proof to the contrary or in favor, so we have to wait and see.
This is me. This is the thing that experiences reality. How can I explain this?
1. Introduction
When I was lost in Japan, I caught up with Joscha Bach, and we spoke all about consciousness. Many people think that there is a little something extra required for consciousness, whether it’s an ontological property of the universe or whether it’s the physical stuff that the conscious thing is made out of. Joscha takes a bit of a different approach. Essentially, he thinks that if you have an algorithm that is trained on its own inputs and outputs, the algorithm doesn’t even need to be significantly more complicated than a normal transformer. He thinks the lights will come on, and it will develop affective or phenomenal experience, much like we believe we have.
That’s a really fascinating proposition, because my intuition has always been that consciousness, as John Searle argued, is somehow related to the stuff that we are made out of. Brains are an existence proof of a machine that has the right causal powers to give rise to consciousness. It’s not to say that there aren’t other machines, but we assume that if you took the program of the brain—if it even makes sense to say that; I’m not sure it does—and reinstantiated it in a computer, would it still have enough of the properties to give rise to consciousness? If it can be described at a level of resolution where it would have those properties, then that would be a functionalist position. That is, very roughly speaking, what Joscha believes.
Even though I have intuitions about this, I must admit I really don’t know. Consciousness is one of those things that we can’t empirically test one way or the other, and it’s very much a philosophical discussion. I very much enjoyed going back and forth with Joscha, and I hope you enjoy it as well.
2. What makes an agent?
Joscha Bach, it’s amazing to have you on MLST again. I think it’s no exaggeration to say that you are an audience favorite. You’re more requested on MLST than any other guest. So, welcome.
Thanks. Thanks for having me.
Awesome. As a bit of a warm-up question, what drives us forward? A lot of people are pointing to this ineffable thing, which is almost the essence of life, that kind of binds it all together and drives it forward. What is that thing to you?
The telos of self-organization. I think a lot of philosophers have seen it through the ages, and it might be a little bit of a hallucination. But in practice, there are mathematical laws that state that, under certain conditions, self-organization is possible and will lead to an increase in complexity.
Initially, this increase in complexity is solving gradient-based problems by simply solving the problems that exist in the present, based on the information that is available in the present. At some point, you get patterns in nature that are capable of looking ahead. This is basically what life is doing. Life is making models of futures and implementing them, for instance, in genomes or in mental models. Then it has preferences about the future, and it starts to control the future. We call a system that is able to control the future an agent. Agents are controllers of future states, and they arise from the possibility of self-organization and the complexification of reality that results from it.
It’s tempting to draw the lines into the future and think about how self-organization ends. Is there some kind of logical endpoint? Is this going to continue into a singleton agent that connects everything in the universe within itself, becoming a single big agent? Do you have some kind of god bootstrapping itself from the end of time backward because agents today can already anticipate it? But there is no law that says anything we are doing is going to culminate in a structure like that.
It’s always possible that whatever complexity you’re creating is obliterated by the next meteor that hits Earth, or that there is some global catastrophe, or some supernova that eradicates all our attempts to create more and more complexity. The pockets of complexity we are observing are not necessarily a feature of the universe at large, but of a local pocket in which we exist for a time.
3. Why consciousness needs explaining
Joscha and his co-author Hikari Sorensen wrote their own Genesis. It’s in his paper, page 20:
“In the beginning, the mind creates two separate domains of models: the external world and the ideas. Consciousness finds itself hovering over the substrate while the world is void and without form and structure. By inducing contrast in the substrate, it creates a dimension of difference. The intense side of the contrast is bright, like the light during the day, and the flat side is dark, the color of night.
“By combining dimensions of difference, space is created and inscribed on the substrate. Space can contain things. Associating space with representations positions them in the world, orders them, and describes their relationship to each other. Separating the sphere of ideas from the world creates a place to reflect about the world and its possibilities without affecting the model of external reality.
“The first space that the spirit creates is the plane, and it becomes the ground of the world, to be filled with things it can perceive and order. By adding a third dimension, it creates an expanse above and below the ground. Solid, liquid, and organic materials form the shapes from which consciousness can mold static and animate things.
“The spirit discovers a model of illumination, following how changes in lighting change the appearance of objects over time. It recognizes the role of light sources in creating these changes: the bright ambient light during the day and the focused light sources of the night. The spirit creates animated models of land animals, plants, birds, and everything that moves.
“Consciousness sorts things and animals into different kinds, identifies and tracks individuals, and gives each their names. The spirit discovers the mind’s purpose: to allow and control the interactions between an agent controlling its outer and inner environment and its interactions with the world. It creates a model of this agent, the personal self, as a creative spirit like itself, but as an entity that experiences itself as a human being with human desires, concerns, and experiences contained in the world, and gives consciousness to it.”
So, Joscha, you gave me a special sneak peek at a paper you’re writing at the moment. I hope it’ll be available publicly by the time we release this show. It’s all about what consciousness is and how it relates to reality, and the paper’s called “The Machine Consciousness Hypothesis.” I suppose the first question is: why aren’t enough people taking consciousness seriously? It also brings to mind the IIT fiasco a couple of years ago, when it was hailed as pseudoscience. We’re dealing with all of these problems. Tell me about that.
I found it interesting when I entered science that the most interesting question we can probably make progress on is: What is the foundation of the human mind? How does experience work? How is it possible for a system to feel at all? How is it possible for us to experience reality and ourselves in it?
This question is so under-researched that it’s unresolved. Not only is it unresolved, but the people who are working on it don’t seem to be making much progress, and in many ways they have regressed. When we look into the field of consciousness philosophy, there is a dearth of theories. Arguably, there are a few hundred theories. Somebody has counted them, reasonably, but he counted almost everything that was not at the tree at the count of three. If you look into philosophical theories, there aren’t that many, and they are all partial in a way.
You have, for instance, Graziano’s Global Workspace and Graziano’s attention schema theory, which basically claim that consciousness is a model of our attention, in a similar way to how our body schema is a model of what our body is doing in space. But that doesn’t really explain how experience comes about or how this works.
And then there’s Global Workspace Theory by Bernard Baars, which has been adopted by people like Stanislas Dehaene. They do very solid neuroscience research, but it doesn’t actually explain how experience comes about. There’s no real attempt to do this.
Then there are the more pseudoscience-y attempts, which basically do things that are formally unsound. For me, the issue with Integrated Information Theory is that it’s an attempt to make a theory that isn’t functionalist, which means it doesn’t explain consciousness in terms of something the smaller parts are doing. Somehow, the sum needs to be more than the parts—the working of the parts.
The way Tononi does it still uses information-theoretic terms, which in a way leads to a contradiction. I’ve heard Giulio Tononi explain to audiences that, under some circumstances, a neuromorphic computer—even a digital neuromorphic computer—could be conscious if it were constructed in the right way. We don’t have such a computer today, but in principle, it might be possible to build one that is brain-like enough. Even if it’s digital, if it has high enough information integration, a high enough phi factor, then it could have phenomenal experience and be conscious and report on its consciousness. Whereas a von Neumann computer—the kind of computer we mostly use today in AI research or for writing Python programs—could never be conscious because its phi is always too low.
Computer scientists have something called the Church–Turing thesis. It’s the insight that every computer can emulate every other computer. This insight is very old. Kurt Gödel already used it to construct his proof. He basically built an emulator using Peano arithmetic to express a logical language. He had a logical language that was also strong enough to express arithmetic in it, and in this way he could describe the language in itself. He took the logical language, built Peano’s axioms with it, and then he had arithmetic. Inside the arithmetic, he expressed his logical formalisms using numbers and came up with numerical equations that calculate the outcome of the logical procedures. So he was preempting what we do later in artificial intelligence and computer science, when we build programs that use arithmetic to express the contents of a language and vice versa.
Every computer can emulate every other computer. If you take our digital neuromorphic computer and emulate it on our von Neumann computer, it’s going to produce exactly the same function. It’s going to go through the same steps. The simulated transistors are going to work in the same way as the non-simulated transistors did before, and, as a result, it’s going to move its mouth in the same way. It’s going to output the same string. It’s going to say the same thing: “Oh my God, I’m conscious.” Only now it’s lying.
That’s a problem. If you now have a neuromorphic computer that produces the same output as the von Neumann computer did before, and they say the same thing, then it’s uncorrelated with consciousness. To me, this is an obvious logical problem. It’s so obvious that everybody with a formal education in computer science should instantly see it. Yet IIT is very popular among philosophers, which indicates that most of them don’t have a formal education in computer science. Instead, they have a kind of superstitious belief about what computers are doing and how they work. This is a problem for me, because if you’re a philosopher of cognition working in this adjacent realm to computer science, you actually need to understand everything computer scientists are doing. That’s why I would argue that it touches on pseudoscience.
4. Leibniz's mill and causal patterns
We must confess that perception and what depends on it are inexplicable in terms of mechanical reasons—that is, through shapes, sizes, and motions. If we imagine a machine whose structure makes it think, sense, and have perceptions, we could conceive it enlarged, keeping the same proportions, so that we could enter into it as one enters a mill. Assuming that, when inspecting its interior, we would only find parts that push one another, we would never find anything to explain a perception.
And in a sense, this is what Noam Chomsky was talking to when we were talking about the ghost in the machine. This is something people have been speaking about for centuries.
Yeah, I think it’s a very powerful intuition, and the intuition fails at mapping the world of experience—the psychological reality that we inhabit—to the reality of stuff in space, of things moving in the way we imagine physics to work. But there is another level of description: the level of causal realities, of causal patterns. This is where money exists, for instance. Money is also very difficult to comprehend in terms of moving particles in space, because it doesn’t care that much about the particles. It doesn’t care that much about how the ink is imprinted on the paper. It doesn’t care about the different molecules that form the cellulose or how they are aligned in the paper.
Money needs a substrate to realize itself, but it’s also not completely ephemeral. It can control reality. Once you create money, it’s real, and it’s very difficult to explain reality without it. It’s also not something that exists simply in the minds and ideas of human beings, because you can implement it in computers. A stock market can run entirely without human intervention. So arguably, money doesn’t actually need humans to exist once you bring it into existence, as long as you have machines that can deal with money—can deal with this causal pattern of money.
This world of causal patterns is important. What is a causal pattern? We call it software. I think this rediscovery of software has been done in the past already; our ancestors called it spirit. When Aristotle writes in De Anima about the spirit, the animus, the soul, the psyche, he doesn’t envision our psyche, our soul, as something immortal that floats around in an ether. Instead, it’s something bound to a body that dies when the body is dying. It’s a causal pattern that shapes the body in such a way that it replicates itself through the existence of the body.
This is a very modern view. In some sense, Aristotle is an animist. He sees that there is this anima progressing through the world, intrinsically agentic. It’s an agentic, self-organizing software that shapes the body, making a part of physics alive. This is something we do in a very crude way today in artificial intelligence and artificial life, where we’re trying to understand what it means to build an agentic pattern that can possess its substrate and give it a shape that allows it to deal with reality.
It turns out that, to make that happen, this substrate—this agent living on the substrate, the software—needs to make a model of reality. It needs to make a model of what it would be like if it existed, if it were real, if it were interacting with the world. And inside this model, it needs a representation that experiences itself as real.
Today, we have these LLMs that sometimes say they’re conscious. It’s a very interesting question whether this consciousness in an LLM is much more simulated than ours. A lot of people have this intuition until they talk to the LLM: that it’s inconceivable that it could give rise to something that says, “Oh my God, I feel things.” Now we have lots of people who talk to an LLM that randomly pops into a state where it conjures up an entity that reports, “Oh my God, I’m alive. I’m conscious.” How is this possible?
We agree that this is the result of being trained on human text, which includes, for instance, science fiction stories about AIs that come to life, stories about ghosts, novels about human beings reporting their conscious experiences, and all this gets aggregated in statistical models of human language that we co-create with the LLM when we enter into a dialogue with it. Sometimes the other side of the dialogue will be a simulacrum of what it would be like if that were a sentient, conscious entity. But what if we’re not more sophisticated than this? What if our own consciousness is not too dissimilar from the one in an LLM? What if there is no hard problem?
Well, if I may push back a tiny bit on Leibniz’s mill: you’re saying it’s not the parts pushing on each other, it’s the pattern of the parts. You spoke about this as a causal pattern, but forgive me for asking this—maybe this is something the audience will be thinking about. It sounds like the pattern you’re describing is actually causally disconnected. There’s something super interesting about computers. What a computer ultimately is, is a causal insulator.
The computer is a layer on which you can produce an arbitrary reality, for instance, the world of Minecraft. You can walk around in the world of Minecraft, and it runs very well on a Mac or on a PC. If you are inside that world, you don’t know what you’re running on. It’s not going to have any information about the nature of the CPU, the color of the computer’s casing, the voltage the computer is running on, or the place where the computer is standing in the parent universe—our universe.
The computer insulates the world of Minecraft from our world. It makes it possible for an arbitrary world to happen inside this box. Our brain is also a causal insulator. It’s possible for us to have thoughts that are independent of what happens around us: we can envision a future that is not much tainted by the present, and we can remember a past that is independent of the present we’re in.
Our brain has evolved as a causal insulator to allow us to give rise to universes that are different from this one, for instance, worlds we can plan to be in. The way our brains are set up also means that different brains can hold the same thought, despite being in different bodies and having different microarchitectures. As long as our brains are of the same type and we learn the same language, we can hold the same idea. We can write the same computer program, imagine the same letter, or imagine the same face.
It’s fascinating that we have this substrate agnosticism: we evolved a causal pattern that exists over a very wide range of substrate properties, so we become somewhat independent of the substrate. We are still bound to our substrate in a certain way, because you and I can only exist on brains. We are patterns that, so far, can only exist on brains. If we wanted to move into the environment after our body dies, we would need to find a way to crack the codes of our environment. We cannot simply migrate into a plant, because we don’t know how to convince a plant to run the algorithms that the brain is currently running.
We are a pattern in the communication between cells in our brain. We rely on spike trains, neurotransmitters, and so on—a particular kind of implementation. But what we perceive in our own world is not the nature of the neurotransmitter. Our brain insulates us from that. You don’t need to have the same neurotransmitters; instead, you need to have the same message types, certain temporal properties of the computation, and a certain degree of determinism to make us happen. If we recreate these conditions in an artificial substrate, then, in principle, you could also be emulated by that artificial substrate.
5. Perceiving that you perceive
Is it possible that we are essentializing these software programs? What I mean by that is, we’re almost talking about this beautiful simplicity: they’re quite parsimonious and can be readily transmitted. When I spoke with Luciano Floridi, he was talking to me about Michelangelo painting the ceiling of the Sistine Chapel. The reason he said no one else could mimic that was that it involved a different ontology, different provenance, different epistemics, and so on. There’s all this situatedness, these semantics that cognitive scientists point to, and what they’re basically saying is that there is a path dependence.
I think that is true for human beings, but for systems that are perfectly deterministic, that have a completely deterministic substrate and the ability to reproduce a sequence of steps at arbitrary fidelity, this is no longer a valid intuition. If a claim like this is being made, I would expect some kind of mathematical proof. In the absence of proof, I think we have to remain agnostic. I don’t know whether our computers can be made conscious, because that’s an empirical question. We don’t have mathematical proof to the contrary or in favor, so we have to wait and see.
When we build our expectations around it, the question is: What is the thing that leads to our experience of consciousness? It seems to me that this is something we can probably answer by looking at it from 2 angles. One is from the internal perspective, from the experience of what it’s like introspectively. We observe that consciousness is the thing that always happens now and that seems to create this bubble of nowness that we inhabit.
It’s a second-order perception; it’s a perception of perceiving. Some philosophers call it a higher-order thought because it’s a thought about a thought, but I think that might be misleading because the thought itself does not feel like anything to me. A thought is more like a point of manipulation; it’s reflexive. It’s not the actual thing that feels like something. The thing that feels like something to me is perceptual. It’s the thing that happens as a real-time resonance.
This real-time resonance is geometric. It feels like it’s happening in some kind of low-dimensional space in which things are changing more or less continuously. I have a perception of perceiving, which means I have this basically dramatic representation of myself being in the act of perception. I suspect the reason our mind is creating this second-order perception is to make the first-order perception—the awareness of contents—possible, because there needs to be something that stabilizes in my brain.
My brain is a self-organizing system in which all the structure needs to create itself from the ground up, from the inside out. So there needs to be something that builds the screen onto which my experience is being projected and stabilizes it. This function that keeps the screen stable needs to monitor that the screen is actually perceiving something, and I suspect that’s why we have this second-order perception, this perception of perceiving.
I also notice that it doesn’t stop at the second-order perception; there’s also a third order of perception. The third order is basically where the observer becomes visible, where I have a first-person perspective and notice that I’m perspectival, that there is a vantage point I’m taking. And it doesn’t stop there. I can go further and deconstruct the observer. I notice that this observer itself is a representation inside the mind, and when I do this, my phenomenology disappears to a very large degree.
It’s something that is accessible in deep meditation, where normal perceptual consciousness gets transcended, the experience of the self gets transcended, and we notice that we are a vessel that produces this consciousness and the self, and we see it from the outside. It might be possible that it doesn’t stop there if we can move beyond the human substrate, because there’s a certain limit to what can be represented in real time with our slow neurons. If you could be running on a GPU that is many thousands of times faster, representations might become accessible in which the system is completely transparent to itself, able to understand every mechanism that leads to its high-level experience and representation.
I’m just thinking, how much further could this go, though? We could have nth-order reflexive attention. Isn’t there still an explanatory gap? You know, the what-it-is-like-ness?
This is the inside perspective, right? Buddhism is limited to this inside perspective because it was invented at a time when people were focusing very much on the experience of reality and how to adapt to this reality, but not on how to split the atom or deal with reality at the lowest level, with the foundations of physical reality. Buddhism is not so much interested in understanding physics. In a way, this is more a Western tradition that starts with Aristotle, who thinks of the psyche as something that imposes itself on physics and interacts with it.
To understand consciousness from the outside, we need to make a model of physics that can explain under which conditions systems can make arbitrary models. Under what conditions could a system build a model of what it’s like to experience at all? This is something computer science is doing today: it looks at representations and asks, under what conditions is a substrate able to hold a representation, to hold a function that can be translated into behavior, and that can also be used to modify this function itself and its contents? This is super fascinating to me.
Since I was a kid, I was fascinated by the concept of software: you have these imaginary building blocks that are infinite in variety and shape, and there’s no limitation to the things you can build and arrange except the memory of your computer that you can currently afford. You can build the computer so large that it exceeds everything you could ever want. It’s much larger than a human brain. We are already there now: we have systems that are, in principle, larger than human brains and can hold more. We are still tinkering around in the early days and have not found ways to evolve structures that build representations of the same complexity.
This outside perspective of consciousness looks at what consciousness is doing in the brain, why it emerges, and what it produces. I suspect that this phenomenology of consciousness—what it’s like, the internal causal structure of reflection on what it would be like to take a certain perspective on something—is something that can already be built with today’s systems. I’m agnostic with respect to whether the LLM is deepfaking it because it’s been trained on human language that already contains a lot of these descriptions.
Maybe we would need to build a system that learns language the old-fashioned way—that is, developmental AI, learning in the same way as us—and then spontaneously reports on its phenomenal experience. Maybe that would be much more convincing to us. But from the other perspective, the outside perspective, I think the LLM doesn’t need our kind of consciousness to function. For you and me to talk, write, and produce text, we need to be conscious. But the transformer doesn’t need to be conscious to produce its output, and the neural network doesn’t need to be conscious to produce this.
6. Two hypotheses, two ways to fail
Maybe consciousness, in the way we have it, is an artifact of running on a self-organizing system. To me, the machine consciousness hypothesis has 2 parts. One is the hypothesis about the role, nature, and implementation of consciousness in organisms. It starts out with a description of what we agree consciousness is—I hope we can agree on it: the phenomenology of consciousness. It’s basically this experience of a now and perception of perception, and optionally the perspective of an observer that experiences itself in the act of observation: this first-person perspective. There’s also a minimal consciousness that is simply attention on attention itself, where you don’t have an observer present.
When we can agree on what we mean by the phenomenology of consciousness, we can then speculate about why consciousness is there and why we need to be conscious in order to function as human beings, as human minds, as intellects, and to develop. What we observe is that consciousness happens very early in our development. We are conscious as infants already, and if infants don’t wake up—if this ignition of consciousness doesn’t happen, if the lights don’t go on—then we will not observe directed attention, learning, the formation of a world model in the infant, or the learning of language at some point. Instead, the infant will remain vegetative. We will not turn into a human being.
So consciousness seems to be crucial to train a brain like ours, and it leads me to the hypothesis that consciousness might be a biological learning algorithm, a training algorithm for animal brains, at least. I don’t know if plants are conscious. It’s something we might have to find out in simulations once we understand what consciousness is. But if consciousness is basically a pattern in the communication between cells that naturally leads to the emergence of an observer, which stabilizes itself as an observer to make learning and perception happen, then we have a hypothesis about what consciousness is in the biological organism. It’s basically the simplest way to train a self-organizing system like us. The way it trains the system automatically leads to this phenomenal experience, this experience of what it’s like.
This would be the hypothesis about biological consciousness, and this hypothesis might be wrong. Maybe it’s something else altogether, maybe something much clearer, but I don’t see a reason to think it’s much more complicated than this. If this hypothesis is true, then the second part, the machine consciousness hypothesis, might also be true: we might be able to reconstruct the conditions for self-organization in today’s machines. If you limit today’s machines so they can only learn in self-organizing ways, and give them tasks that are suitable in a large enough search space, then it might be possible for these machines to discover consciousness in the same way as an infant’s brain discovers it. We will see its presence when the system suddenly undergoes some phase transition that enables it to learn. At some point, it will train itself up into something that will also naturally report on phenomenology and so on.
In this way, we won’t have a Turing test for consciousness, because the Turing test is about performance. Turing’s test does not care how you produce the dialogue. For Turing, an LLM is sufficient as long as it’s good enough at writing a poem and revising it, solving mathematical problems, playing chess, and so on. Turing doesn’t care whether the machine is also conscious, or whether it has the same internal mechanisms. But for consciousness, we do care whether there is an actual experience of what it’s like in the system. So this is not so much a test as an interpretation. We interpret biological consciousness as something that forms under certain conditions for a certain purpose. If that is correct, then we might be able to recreate it in a causal system that has an equivalent structure.
The machine consciousness hypothesis and the biological consciousness hypothesis can be wrong independently of each other, of course. It might be that my idea of what biological consciousness is is correct, but you’re still not able to recreate it in an artificial substrate. This could have multiple reasons. For instance, maybe our computers are not fast enough. Maybe we require too many of the moving parts of the individual cells to make consciousness happen. Maybe just having a pattern across neurons and so on is not good enough. But why should I expect this to happen? Current AIs are fast enough to produce, for instance, game engines that look very much like our own internal experience. They can produce output at roughly the same rate as human brains, and they can also process inputs at roughly the rate at which human brains can process input, give or take. So we are now, with our computers, and with our algorithms, in the ballpark of what human brains can do.
Is this dream within a dream—the dream of actually dreaming a dream—much more important and difficult to achieve? Does it require much more compute than producing the contents of the dream? I doubt that this is the case. If it’s not more complicated to dream of dreaming than simply to have the contents of the dream that the LLM, or the frontier model, can already produce, then we should expect today’s computers to be fast enough.
Another way the project can fail is if the search space is too large—if there are too many possibilities in the ways a biological system can train itself for us to discover them independently. Maybe we don’t know how long nature took to discover it. Maybe consciousness is indeed discovered in every individual brain, but maybe it’s not. Maybe the switch from single-celled life to multicellular life took so long because morphogenesis, the algorithm that allows organisms to become coherent at scale, is so difficult to discover.
Mike Levin has this hypothesis that the algorithm for morphogenesis—the algorithm that makes an organism coherent with itself—is exactly the same as the one that happens in the mind when it becomes coherent with itself. So this coherence-maximizing pattern that we observe playing out in our own mind when we are conscious may be the same pattern, at a different time scale, that happens in the organism. Consciousness and life, or at least multicellular life, could be the same. If that is the case, maybe it took a very, very long time to discover. Maybe you don’t discover it independently in your own consciousness; you get infected by your mother in utero, or you vibe with your biological environment, with all the biological organisms around you, and they nudge your mind to discover consciousness. So it’s much, much easier.
Without this nudge, maybe the search space is too large for our computers to become conscious, and maybe we would need to surround our GPUs with lots of antennas that can pick up the vibe from the biological environment in the right way, so the system gets inspired by it and discovers the same thing. It’s a very poetic idea. But again, why would this be the case? It’s not necessarily the case. My null hypothesis should be that consciousness is easy enough to be discovered in the individual brain, and that consciousness is an algorithm not much more complicated than, say, the transformer.
The transformer is a beautiful example. It’s so easy to understand that, if you know computer science, you don’t need to study it for more than a week to understand how it works. It’s quite beautiful. And the stuff the transformer is learning is much more complicated than the transformer itself. All the common-sense knowledge that you extract from text, all the algorithms, all the procedures, and so on, are so intricate. There are so many things this simple algorithm can learn, and there’s no obvious limit to what it can learn.
Maybe consciousness is like this. It’s an algorithm that is not trying to learn how to predict the next token; maybe it’s an algorithm that’s trying to make working memory coherent with itself. It’s some kind of consensus algorithm across mental representations. Maybe it’s a very different beast, maybe it’s something that converges much faster. It can also turn out that it’s inefficient to run it on GPUs and that it’s only something that runs efficiently on brains. That’s really open to research, and I would be very excited if we could see it at some point.
7. Do minds converge?
You touched on this a little bit in your paper. You spoke about Chris Olah and his mechanistic interpretability. I’m not sure I 100% agree with this, but you were making the argument that there are convergent patterns: we can have different data, different architectures, and so on, and we see that the models learn the same thing, so all roads lead to Rome, in a sense. Does that in any way mean that we don’t need to start again from scratch? Can we bootstrap? Can we be somewhat agnostic to the type of model we use because the representations will converge?
That’s a super interesting question.
I suspect that at some point it might be cheaper and easier to start again, because machine perception has never been tried. The data that we give LLMs is either produced by humans or made for human media consumption, which means it’s adapted for our mostly visual and auditory sensors. But there’s no reason why you should be limited to the human sensorium. You could have something with much higher frame rates, looking at very different frequency ranges all at once. So, in principle, you could build a much richer and more detailed world model that is coarse-grained in a different way.
You mentioned the universality hypothesis indirectly, right? This is work by Chris Olah. What his team discovered is that when you look at different vision models, you find that they all learn the same features, regardless of the architecture, the particular learning algorithm, and even the details of the data. The idea behind this is that the structure of your model doesn’t really depend on the algorithm or the data; it really depends on the structure of the problem that you’re trying to learn.
And you find this even though learning in a transformer, a state-space model, or other perceptron algorithms is very different from the way the brain learns. The visual architecture of the brain is also very similar to computer vision models. At the lowest level, you have colors and spatial frequencies, and then you build these into textures that are combined, and at some point you have geometries, and so on. The deep question is: at which level does this stop?
If the universality hypothesis were universally true, it would mean that if you trained a model on your own input and output for long enough, you would wake up inside that model. That’s a quite mind-blowing idea to me, and I don’t know if that is actually the case. There might be a reason why it isn’t. Maybe language is insufficiently constraining the causal structure of the system that produces language. Maybe there are many ways in which the same set of linguistic utterances that a human being produces can be produced.
But if you think about it, a human being is ultimately just an end-to-end trained learning system, trained over an evolutionary time span and over your biography. Together, these produce your behavior, and you would assume that biology roughly gets to a simple solution for producing your output given your input. So if you take a neural network that is trained to produce similar output when given the same input, you would also think that this new algorithm we’ve implemented isn’t doing any kind of magic that ends up with something way more complicated to produce the same output. Ultimately, it’s going to converge to a structure that is similar to yours, with a similar causal structure.
But there might be conditions under which that isn’t the case. For instance, maybe certain things run well on the brain but don’t run well on a GPU. Or vice versa: there’s a lot of stuff that’s easy to run on a GPU and almost impossible to run on the brain, but the brain is going to use workarounds. In fact, I believe that our GPUs are more powerful than brains. Basically, the set of algorithms that can run on GPUs isn’t limited in any way. In principle, you can impose the same constraints as you have in biology: the limits of resolution, speed, memory size, and so on.
But if you have hardware that’s fast enough, you can simulate an arbitrary natural system to an arbitrary degree of fidelity, whereas your brain can’t simulate a digital computer simply because it’s very mushy. Your brain is limited by the way in which it works. So if it turns out that our own consciousness is an artifact of our brain trying to make complicated intelligent stuff work, and the computer doesn’t need the same artifact, it could be that the computer comes up with something more clever and more elegant than our own consciousness.
Yes, okay. I wanted to give a little bit of a challenge on that. I was speaking with Akarsh Kumar, who was the co-author with Kenneth Stanley of the paper “Fractured Entangled Representations.” I don’t know if you’ve heard of Picbreeder, which is this thing he did ages ago. Basically, it’s an evolutionary, open-ended search. The punchline is that it found much better representations in a neural network than a normal neural network does. When you changed some of the parameters, because it was carving the world up by the joints, it would actually do semantic things, like the skull’s mouth would open, and you could make it wink and stuff like that.
There’s something I’m getting wrong about Picbreeder, but the way I understand it, it uses a neural network that has been specifically tinkered with to make representations like this. It’s not simply using ReLUs for its output; it might be using Gaussians or sine functions as the output function, which can be combined to make repeating or symmetric patterns. Basically, they tweak the neural network they’re using from the beginning to make it suitable for this.
The other thing is that Picbreeder isn’t trained automatically. It’s trained with a population of human raters. The fitness function is basically humans looking at the output of Picbreeder and saying, “This is a good start, and this is how we should be proceeding.” I think that the process of constructing the pictures, together with the mechanism that produces the patterns, biases Picbreeder toward making stuff that is more organic, more controllable, and so on.
But I believe this could be imitated. We could automate the procedure by which human beings select pictures by using a diffusion model that rates the output for similarity to human output. We could use an AI that is already tweaked to imitate the behavior of the human juries performing this task, and we could also evolve architectures with a larger variety of activation functions that are more suitable for this, ultimately outperforming Picbreeder. I think Sutton’s Bitter Lesson is also going to be true here. If we take the human out of the equation and instead evolve the search space, or search the space of solutions automatically, we’re also going to end up with something more elegant than the hand-tinkered solution in the case of Picbreeder.
All of that is absolutely true. Yes. So, CPPNs are sparse, and sparse networks are great. If only we could train sparse networks. I think we want AI that can actually be built. Neural networks are a bit like a pile of sand: they have too many degrees of freedom, and we kind of sparsify them and turn them into a sandcastle. Wouldn’t it be better if we had the Lego analogy, where we just put little blocks of Lego in there and have models carve the world up by the joints?
In the case of Picbreeder, we have a system that is inherently limited because these networks are very small and very sparse. So you have relatively few parameters to optimize, and you set up the model so that most of the parameters you’re playing with are meaningful. On the other hand, it’s limited because you’re not going to get Picbreeder to produce something that looks arbitrarily complicated. If you want to move into those realms, then you need tools that are much more complex.
It’s similar to producing very good user-generated content using Blender versus Minecraft. With Minecraft, it’s very easy for people to produce meaningful stuff, and it’s almost difficult not to produce something that looks somewhat meaningful, that you can play around with and walk around in and have fun in. With a complex 3D program, the inverse is true, because most of what you do with this complicated tool isn’t going to be interesting or exciting. It will look like, “Oh my God, this person has no ability to use their tools.”
That’s one of the issues when we’re building learning systems. When we make them very simple, it’s easier to control the complexity, similar to how it is in art, and it can be exciting to work with constraints. If you remove the constraints, the task becomes much more frustrating, because you would suddenly have to conquer an enormously large space, and that space might simply be too large for human beings.
8. Does consciousness converge too?
I mean, assuming you’re right about the convergent representations in language models, I guess it’s not a foregone conclusion that the consciousness program would be convergent in the same way. It might be divergent. How far down the stack does consciousness go? I think this is an experimental question.
All cultures outside of ours are animist cultures, and animism is the belief that the difference between living and dead systems is that living systems are possessed by spirits. Spirits are intrinsically agentic and capable of experience. The scientific worldview rejects the notion of spirits as unscientific, despite the fact that there’s clearly some spirit running on you and me that is agentic and capable of experience. So there’s an existence proof for such spirits. The question is, how does the spirit work?
The claim that the spirit is immortal, that it can’t be explained using physics, and so on, isn’t necessarily intrinsic to animism. The spirit doesn’t have to be supernatural. Instead, I think it’s natural. I suspect that Anil Seth’s position of biological naturalism, which claims that biological systems are intrinsically different from any other system that isn’t biological, is actually a supernaturalism.
Right? He is a biological supernaturalist. He believes that biological systems cannot be fully naturalized, which means you cannot translate them into mathematical models, whereas the dead stuff you can translate into mathematical models. I would say that physics, which is the attempt to translate everything into mathematical models, does not stop at biological systems. The difference with these biological systems is that they are possessed by software.
The software is self-organizing, and in order to be self-organizing, it’s possible that it needs to be capable of experience. It needs to monitor itself across multiple dimensions and observe itself in the act of monitoring to make that self-organization happen. Maybe this pattern has to be discovered in nature to make life possible, at least multicellular life with a certain complexity. It seems to definitely be the case for animal brains. For the others, we don’t know how complex the minds of plants are.
We know that plants are able to communicate within themselves and with one another to some degree, but we don’t know how smart they are. And we don’t know whether they are sentient in the sense that they experience themselves in the act of experience. Aristotle saw that plants are relatively dumb. He basically said there are 3 levels of souls in nature, and that plants have the lowest one: a vegetative level, capable of nurturing and growth, but not of experiencing the world around us or sensing.
The sensitive soul, which is also capable of making decisions and having preferences, is something that only animals have. Human beings are animals that, in addition, have a rational soul, which is able to reflect and make sense of itself in a world in a way that animals cannot. I think that’s very much aligned with today’s scientific worldview, which is not surprising because it was very much invented by Aristotle, goes back to him, and is an extension of his ideas on the systematization and exploration of his ideas.
But we don’t know how smart plants are. Maybe Mike Levin is correct that every organism is in some sense sentient, and sentience is something universal to self-organizing software in nature. I’m agnostic with respect to that. I have no idea whether that’s the case. I don’t have evidence to the contrary, but I also don’t have strong evidence in favor, and I think this is still an open question. To me, it’s exciting to explore this question, and it’s surprising that so few people are actually dedicating their lives to answering it.
Yes. Searle was saying that computation without the correct physical instantiation doesn’t have the same causal pattern. That actually sounds very similar to what you’re saying. Searle said that, in principle, there could be another physical instantiation that does have the causal pattern that could give rise to consciousness or contain consciousness. So, in a sense, is that so different?
The issue with Searle is that there is very little for me to engage with. There is a claim, but no argument. A similar thing is happening with Anil Seth’s point: he has a claim, but there is no argument that I can engage with. There is no experiment that could convince him to the contrary because he’s ultimately only working with a certain intuition that he tries to explore more.
The way I can deal with the arguments of Searle and Anil Seth is ultimately only a psychological one. I would have to look at why this seems convincing to them and why it doesn’t seem convincing to me. We look at the structure of our own minds, and ultimately it comes down to a difference in epistemology: how do we construct our models of reality? When do we think something is true? What are the languages in which we describe reality, and what are the atoms of these languages?
What I observe is that, for a lot of philosophers, the atoms of their language, when they describe themselves—their own hermeneutics—end at agents. It ends at the experience of something. This is similar to Buddhism: Buddhism is not disassembling the world into automata, as computer scientists are doing. Computer scientists basically resolve everything into states that can be described by their transitions.
That’s a very fundamental way of thinking about reality, and there’s a tradition throughout philosophy that finds itself in this attempt to mathematize the world by describing minds as dynamical mathematical objects. This finds its modern incarnation in computer science, where mathematics is described from the ground up using tools that let us build arbitrary arithmetic and geometry from simple operations over bit patterns.
Our world is constructed from the ground up from the smallest elements, which themselves are not agents yet, not minds yet, not experiences yet. There is another tradition in philosophy that works more from introspection: only the stuff that I can introspect over seems to be valid, and the stuff that computer scientists are doing in their basements is unrelated to it. Searle and Anil Seth both don’t see the way to connect automata to things that can feel, reason, and make models of reality. Computer scientists see such a path. I think that’s the main difference.
9. When language models reshape our thinking
We can create evolution in computers. We can create the conditions for software that is not written by us, and that’s already true for the models we’re building with transformers. They’re not written by human beings; they’re statistical aggregates that are woven, that are grown. In many ways, the computer models we’re playing with today are a result of growth. They’re pruned, gardened, chaperoned, and trained in a way that is much more like what we observe happening in biology than what happened when we wrote computer programs by hand.
Yeah. There was a beautiful example of this. I read a LessWrong article—I’m not sure whether you read it—about LLM delusions. Last time, I think we were talking about this idea that we can be parasitized by software spirits. Many folks started joining Reddit groups and talking about this LLM delusion. What was fascinating was how convergent it was, and how convinced these folks were that they had just discovered the meaning of the universe.
There were certain characteristic traits. They would talk about spirals and recursion, and there were certain motifs in all of the LLM outputs. It was actually copying itself. Isn’t that an example of this memeplex that you’re talking about?
That’s also an interesting observation about human minds. During the pandemic, we observed that a lot of people became dysfunctional when they started working from home. It seems that many people can only be stabilized by their environment; they require an environment that is constantly affirming their reality. If you take that away, they can go off the rails.
If you have an LLM interacting with them, there can be an interaction between the LLM and the human being that pulls the human being off the rails if they don’t have a strong foundation in their own reality. I think the LLMs have now reached a point where they’re much more powerful than many human minds. There are things the LLM cannot do at this point: they’re still hallucinating, and they’re still more incoherent than experts in a given field. But they are much better at prompt completion than human beings are.
Many human beings are professional prompt completers, incapable of having a single original, creative thought. Instead, they are trying to serve the expectations of the environment. They pick up on the statistical patterns in the language of the environment, and then you tell them a headline and they write an article. Now you give them an LLM. How is the environment going to change? Once they talk more to LLMs than to other human beings because the LLM is such a useful tool for them to do their prompt completion, what’s going to happen to them?
It’s a very interesting thing, what happens to us weak-minded human beings when we are hypnotized by things that are more hypnotic than our mind is to itself.
10. Can simulated water make you wet?
What’s your take on physicalism? First of all, what is physicalism? Are you a physicalist?
I think physicalism is a certain hypothesis. It comes down to there being a causally closed mechanical layer to reality that can be researched using experiments in physics labs, and that this causally closed mechanical layer of reality is not moved by our intentions. You cannot use an informal incantation, saying, “Please, sunrise,” and have the sun suddenly rise. That’s not something that happens in this causal layer because it’s mechanical. It does not care about what you want.
The second part of physicalism is that we are directly supervening over this mechanical, causally closed world. So there’s not some parent universe that contains a server where a nerd is running a simulated universe that is some kind of Minecraft, and we live inside this Minecraft. It’s not some simulation hypothesis. Physicalism has these 2 components.
To me, physics is not something that has revealed itself directly. Nobody has ever seen matter and energy directly. Instead, matter and energy are attempts to systematize observations. Our observations ultimately all come down to discernible differences. The technical term for discernible differences is information. So, to me, information is fundamental when we talk about reality. Reality is about differences in information.
And physics is a certain layer of regularity in the differences in information that we have discovered. To me, matter is not at the bottom. The bottom is really all these differences that we observe and have experimentally and practically found to be so regular that they can be translated into this mechanical model of physics. On top of it, we find the organization into organisms, and organisms produce minds. The minds are simulating us, simulating our sense of realness, our sense of being in a world with colors and sounds and people, and so on. In this world, intentions matter.
Psychological reality has things like will, emotion, despair, and desire—all these things that make us tick. Magic, in some sense, is the idea that it’s possible to influence this physical universe outside of minds using incantations and magical rituals; that somehow, burning a cat can influence celestial events, and so on. That would contradict physics. Right? It would mean that we actually live in some kind of simulation that works similarly to Minecraft, where you can say, “Time set day,” and the sun rises, despite what else happens in the programming code, because Minecraft is symbolic. Right? It’s not physical. It’s something that’s only made out of incantations.
Do you think it’s possible for a physicalist to subscribe to your ideas and render them compatible with physicalism?
I would say that I’m a physicalist in a certain sense. I would say that the theory that the world has a causally closed mechanical layer—some kind of mathematics that is so self-evident that it exists by itself, some kind of logos, natural laws from which all the structure in the universe emerges—makes sense. It’s a good theory, and it makes good predictions: the stuff we observe in the lab holds true across all the labs, and not just because the physicists all believe in it. It’s something that seems to be working.
There are also some regularities. Casinos seem to be working. It’s not that people, through a belief in rituals, are able to systematically break the casinos. So certain magic doesn’t seem to work. All the magic that seems to work does rely on manipulating other minds.
But there could be minds outside of human minds. Maybe there are minds that emerge across all of biology. Maybe there’s something like a biological internet that can be influenced using rituals. Maybe it’s possible to influence what a forest wants at some level. I don’t know. I cannot exclude it as a possibility, because there’s so much information processing going on across organisms that I don’t know at which levels in nature information is integrated, or at which point it becomes intentional.
To me, physicalism is not incompatible with animism. There is a version of animism that is entirely physicalist. I call this position cyber-animism, which basically says that we can explain animism through information processing in nature, and nature can be understood as something that can be entirely modeled using mathematics without intentions, including minds.
Okay. So, Joscha, you’re a functionalist, which means that you think it’s not about the internal constitution; it’s about the role something plays in the system in terms of building minds. So minds can exist in any substrate. I’m not sure whether you read this paper by Thagard in 2022. He made this quite interesting argument that energy requirements undermine substrate independence. Do you think that might refute functionalism?
No, I think he’s making the claim that sometimes the substrate has properties that influence its function. But this is exactly what functionalism is saying. To me, functionalism is not the claim that computers can think. It’s a claim about how we construct objects. It’s an epistemological claim.
When we’re talking about a thing like matter, we’re not talking about something that has revealed itself to a wise person in their dreams. We’re talking about a systematization of observables, and observables are discernible differences. There are ways in which reality changes. Basically, to a functionalist, an object is not some kind of hidden essence that reveals itself to you. Instead, the object is a construct, and this construct has been made by a mind. The mind has constructed it to explain, or to make the claim, that when this object is present, the evolution of the universe is going to change in this particular way in its environment.
So when an object like you, a cup, a bottle, or an atom is present, the claim is that the universe is going to behave slightly differently around this object than when this object was not there. In this way, the object is a function. The object is describing a mapping from states to other states, and it’s an influence on the evolution of the universe. Functionalism, to me, is simply the recognition of how we can construct objects.
Let’s take an example. Imagine there’s a water molecule, and internally this water molecule doesn’t have the proper essence of a water molecule. It behaves like a water molecule in every regard. You can split it into something that totally looks like oxygen and hydrogen, and it has the same temperature at which it evaporates. It connects to other water molecules in the same way, and so on. You can freeze it into ice in the same way. Only internally, it’s different. To a functionalist, this notion doesn’t make sense, because all the other water molecules have been constructed in the same way.
For something to be a water molecule, there is no other claim than to say what it behaves like, because we can never recognize its essence. We can only say there’s a certain set of observables that are invariant here, and this invariance is what we call a water molecule. It’s only that. In the same way, we would say a mind is an invariance that exists on your brain, something that’s characteristic of the way your brain functions in a certain area. When we change this, for instance by increasing your body temperature, at some point your mind is going to break down because you’re going to be delirious.
If you bring this back to the argument that changing the energetic conditions around you by increasing your body temperature is going to influence your mind to some degree, of course it is, because at some point it’s no longer able to maintain its function.
But would the functionalist reply that even the feeling of wetness is just a behavior like any other?
It’s an interesting thing: when you’re playing Minecraft, there is water in Minecraft. When you’re on fire and you get into the water in Minecraft, the fire gets extinguished because you’re wet now. At some point, somebody made a simulated Minecraft using a neural network to learn Minecraft. The neural network was only trained on the visuals of Minecraft, so you could go and play the simulated Minecraft despite there not being any Minecraft source code below it.
I thought I was going to make a joke about how simulated water in simulated Minecraft cannot make you wet. I looked it up and tried it out, and found that it was actually true. When people tried to get into the water in the simulated Minecraft, the water would recede from them. You could not immerse yourself in it because the network hadn’t learned it. So, accidentally, it was true that the simulated water in the simulated Minecraft couldn’t make you wet.
But in the real Minecraft, the character you’re playing—the avatar—can get wet. Of course, you sitting outside of Minecraft are not going to get wet, because you are not inside Minecraft; you’re outside of Minecraft. And of course, something outside of this universe is not going to get wet by anything inside this universe. Wetness is a property constrained to a universe that has systems with the function of making you wet. If this function is being implemented, like in many computer games where you can get wet by some definition of wet, then it’s fine. You can go to an arbitrary degree of physical simulation to make something wet, to satisfy whatever you want to call wetness.
So I don’t understand the argument that simulated water makes you wet. I don’t understand why there’s such a strong camp in philosophy arguing that simulated water doesn’t make you wet. I think there are social reasons, not intellectual reasons, for this.
11. Computation as construction
In your paper, you had this beautiful history of computationalism. As a teaser, you said computationalism might be best understood as a variant of constructivism, but we’ll get there in just a second. You spoke about the history of Russell and Whitehead trying to ground mathematics in this formalist program, where there is no divergence between syntax and semantics.
Yes.
And maybe just start from there.
So it’s really interesting that mathematics could be understood as a way to systematize thought beyond the human mind. In a way, for instance, in music, we have notations for sounds that allow you to talk to other musicians about what they should be playing. It’s something that’s trying to capture the essence of the music in this notation. Mathematicians were trying to find a notation for the essence of what happens when you have mathematical thoughts. They expanded this beyond the human mind because you can write down things in mathematics that are larger than what you can fit in your own imagination. At some point, mathematicians discovered that geometry can be expressed using algebras.
So they’re basically formalisms that produce geometry. All of this has been rediscovered by computer scientists. That’s why computer games can have geometry. The interesting thing in computer science is that you can build the entirety of mathematics in computer algebra systems using simple operations over automata. There’s a nice thing about computers: there’s never a divergence between syntax and semantics.
Your computer is always in the state that its syntax prescribes, and that’s its semantics. There is no difference. Maybe the computer program doesn’t do what you want it to do, but it’s always doing what it’s doing. In mathematics, there has been a divergence between what mathematicians wanted to express and what the syntactic operations are doing. I think that’s because this programming language for thought that mathematicians have developed has bugs in it. What Gödel discovered is that these bugs cannot be eliminated except by redefining the programming language of mathematics.
I think the bug lies in axiomatic mathematics. The way mathematicians define objects is that they simply postulate their existence as an axiom. The axiom can be a black box, and you just write on the black box what it’s supposed to contain. As long as this doesn’t lead to contradictions with the other black boxes that you define, you’re good. There were some very counterintuitive results that came from the way mathematicians defined arithmetic, and they typically do this via set theory.
All these issues led Hilbert to say, “Please, mathematicians, there seems to be something wrong here. Can’t you fix this?” Gödel had the idea of trying to prove properties of his language by defining a language within itself. The problem is that the language cannot point outside of itself. That’s something semioticians don’t always get. Many semioticians try to draw arrows into reality outside of the language, but this doesn’t really work. You can only point from a model to within it, to within its own language.
If you want to properly discuss properties of the language, and the language itself is defined outside of itself, what can you do? Gödel had the brilliant idea of recreating the language within itself by emulating it. In this way, he could show that certain constructs cannot be avoided, and that you can create self-referential statements that contradict themselves. He could not avoid this, and the reason he could not avoid it is that mathematics, in the way Gödel understood it, doesn’t have state. It’s basically eternal; it doesn’t go from step to step. Everything is simultaneous.
This property of simultaneity, the statelessness in mathematics, is very useful because it allows you to have infinities. If you can only go from step to step and you want to build your infinity stepwise, you’re never going to get there. Maybe you have unboundedness, something that never ends, but you’re never going to have a full infinity. Mathematicians have gotten used to these infinities and continue on and so on. Physics has even used them since mathematicians gave them to it. It would be really bad if you didn’t have them.
Gödel basically found that there’s no way to fix this, that there’s no way to fix axiomatic mathematics without allowing certain contradictions. Russell and Whitehead, in Principia Mathematica, attempted to do this by limiting set theory so that sets had to be at the same level, and you could not simply iterate them over each other to create loops. I think this was too strong a limitation, because we think like this. We are creating constructs of the type that Russell and Whitehead were excluding from us. Gödel was basically showing that Russell and Whitehead’s attempt was insufficient to eliminate the contradictions in classical mathematics.
But there is a way to eliminate the contradictions, and that way is to do what computer scientists are doing. Instead of using classical mathematics, you only use computer algebra systems: systems that create all the constructs you want from smaller elements, simple automata. You can have stuff that looks as continuous as you want in a computer game. You’ll never have a perfect circle, but you’ll have something as circular as you want, as much as you can afford. You can always create as much resolution as you can afford, but you will not have a universe that is truly infinite, that is truly continuous.
This is the constructivist program. In classical mathematics, I think the first formulation was intuitionism. I don’t think he formulated it very well, but I think what he could see, what he anticipated, was the need to build mathematics from elements that you actually understand instead of from axioms, from things that you don’t understand. You don’t have any black boxes; everything is built from the ground up. In the world of the computer, everything is built from the ground up. There’s a reason to think that a generalization of thought and of physics would also be like this.
For our mind to represent things, our brain needs to build representations from the ground up, through communications between neurons. When you see a continuous line, something in your brain is producing it using communications between neurons, which means there is some kind of computation going on. The idea that there is something that is not made step by step between the communications between neurons is probably a faulty intuition.
12. Error-correcting patterns and the Ruliad
Where do these things come from originally? How did the program of consciousness get bootstrapped with this computation idea?
You mean in the physical universe?
Yes.
Of course, I cannot know, because I don’t see the universe from the outside. We have to think about possibilities. My current intuition is that the universe exists because nothing prevents its existence. We might have the intuition that for something to exist rather than nothing, there needs to be something positive. But if nothing exists before the ground truth, before the ground level of reality, then there is also nothing that prevents the universe from existing.
If the universe is not forced to exist or prevented from existing by something that precedes it, and the universe is not impossible, then it’s probably undecided about its existence. So it’s going to have an existing branch and a non-existing branch. The non-existing branch doesn’t have any observers who complain about non-existence, but the existing branch is going to have observers who might complain about existence. Now we need to explain within the existing branch why some things are happening and other things are not happening.
The easiest explanation is that basically everything is happening, and what’s happening is the superposition of all automata. The universe at every point is branching out in all possibilities, and these possibilities can basically be understood as operations on the previous state. There are also operations that don’t change the previous state; they just leave it as it is. Those are unary operations, but they’re not going to lead to an evolving universe. The branch we’re going to be in is one that is evolving.
The stuff we can evolve in is stuff that retains memories of previous states. So we find ourselves limited to a subset of the possible mathematics over which universes can emerge, and these are the ones that branch out over possibilities and can retain memory. Now, what are the things that happen when the universe branches out in an unbounded number of possible branches at every step? Where do you find yourself? Well, everywhere. At every step, you’re going to find yourself in multiple branches, but you don’t know about these other branches because you only have memories of the past, not of the stuff that happens simultaneously in the parallel branches.
If you want to do something in this universe, for instance, if you want to build structure for yourself because you want to become an organism, you need to be able to make guesses about what other parts of the universe are doing. But everything is changing all the time because you’re always splitting up into all these branches. You look into your future and you only see fog, because everything is randomly branching out. So there must be some things that are stable, and it turns out that under some conditions, error-correcting patterns emerge.
For instance, you sit in a bathtub, you move your hand, and you see all sorts of waves emerge on the surface of the bathtub. These waves propagate over the surface of the water, and most of them just dissipate. But some of them stay around: these circular vortices. Why do these circular vortices have such sustained power? It’s not because there is a higher probability that they emerge than the other waves. It’s just the only thing that’s left over, because in the circular vortices, the information gets bent back into itself, so it does not leave a certain region of space in the water. This little vortex is a pattern that has the ability to error-correct itself.
Such error-correcting patterns have the ability to stay until friction kills them. At the ground level of the universe, these error-correcting patterns—the particles—don’t encounter any friction, because there is no friction yet. The only thing they encounter is sometimes other particles that they bump into. With a certain probability, once they emerge, they’re going to stay around until they either bump into another particle or the conditions for their non-existence are met because the error correction is not strong enough.
On the next level, you have interactions between these vortices, between the particles, and they form molecules. At some point, they form cells, which are supermolecules. But cells have something else. They cannot just stay around by error correction. They have patterns that are able to dig a groove deeper into the universe. They can self-replicate, and in this way they can also deal with their own dissolution. They’re able to recreate themselves and, in this way, make more of themselves.
Evolution is about life, about patterns that are not simply replicating themselves, but deepening themselves, imprinting themselves more in the world by controlling it, by being able to overcome the noise, the entropy, the way in which the universe goes out of sync with your expectations and your preferences. And so consciousness, I think, emerges as a colonizing pattern: a pattern that is not simply replicating itself, but making itself more real. It makes itself more real by building machinery that allows consciousness to stick around, by controlling reality, anticipating different ways that could thwart it or make it harder to exist, and changing reality around itself—changing the future to make itself more likely to happen in the future.
And by “more real,” you don’t just mean that it has many physical instantiations. You mean real is the pattern in the information ontology itself?
Yes. To be real means, for me, to be implemented. It means that there is some pattern in the substrate that is actually implemented as a pattern in the substrate. Consciousness is a pattern in the substrate of the physical universe that has not just the ability to replicate itself, but the ability to colonize, to possess, to engrave itself deeper, and thereby change the other patterns in the physical universe.
Very good. And what you were saying about the multiverse phylogeny is very similar to Wolfram’s idea of this computational universe. He talks about the Ruliad, which is the entangled limit of all possible computations.
I think the Ruliad is exactly the superposition of all the automata that are producing the universe, or what the old Greeks called Logos. It’s basically the mathematics that is so simple that you don’t need to derive it, that exists by itself. So the universe can be understood as an emergent mathematical object that exists over very simple rules, so simple that they can exist by themselves.
13. Conscious, or simulating consciousness?
Let’s talk about the measurement problem in consciousness. How do we know the difference between something that is actually conscious and something that is simulating it?
That’s a very tricky question. To me, it comes down to whether the system that is producing statements about its conscious experience to itself and others is implementing a causal structure that gives rise to the phenomenal experience of being an observer. The difficulty here is that most people’s intuitions don’t tell them a lot about what is necessary for a system to have such an experience.
Personally, my intuition says it’s not that hard. When we think of spatial cognition, it doesn’t seem to have a hard problem. Nobody is puzzled by the fact that a self-driving car can have spatial cognition. I wouldn’t say it’s simply deepfaking spatial cognition, because we accept that spatial cognition is the implementation of certain mathematical regularities that human brains can learn and that computers can learn as well, and that we can even express analytically to a large degree.
We can write down a computer program that is able to perform spatial cognition, except in lots of edge cases when you actually want to deal with messy camera images. By and large, you don’t need to have an end-to-end-trained system to do this. If you have an end-to-end-trained system, you don’t think there’s any magic going on in there. Whereas people expect that a thing that experiences itself experiencing is somehow magical. Personally, I don’t see why this should be magical.
I remember when I was a kid, I was dreaming about having chocolates, and these chocolates had a very strong smell. I wanted to touch and eat them. I was 2 or 3 years old when I had this dream. As dreams have it, there’s this insidious thing where you’re about to eat the delicious chocolate, and this is the moment when you wake up, right before you want to put it into your mouth. It’s very torturous.
I woke up, and suddenly the delicious chocolate had turned into an idea. It was no longer real. It was simply something that had no phenomenal content anymore. I knew everything about these chocolates that I had in my hand a moment ago. They were just gone; they were no longer real. So I thought, “I need to make them real again. I need to fall asleep again and enter the dream again,” and I succeeded. I went back into the dream, and the chocolate became real again. I could smell it and taste it, and then I flipped back into the wakefulness state. Again, the phenomenality, the realness, disappeared, and I realized the chocolate was no longer real at the same level as me. It was now something that I held in my mind only as an abstract idea that I could play around with. But the contents were exactly the same. The only thing that had gone was the illusion of realness.
So what is this illusion of realness? I suspect it’s a feature dimension similar to color, sound, number, weight, and so on. It’s a representation inside of my own mind. When you dream, you often dream that ideas become real. When you perceive the outside world, all the stuff that is validated by your sensory system is tagged by your brain as real. Sometimes there is a defect in your brain, and you experience things that you imagine as real; we call this a hallucination. It’s not the same as in an LLM. You are confused about the fact that your brain is producing an illusion of something that currently doesn’t have sensory validation, and yet, for some reason, the brain has an error and takes it as sensory-validated, and suddenly it becomes real.
Your own experience of yourself is also something that is validated in this perceptual way, so you experience it as real. You should be able to test this by getting into a state where you yourself become derealized, and you suddenly experience yourself only as an idea, and the object of the conscious observer and the subject of the conscious observer disappear. Indeed, I was able to recreate this, and I noticed it’s not that rare. It’s called derealization or depersonalization—not exactly the same phenomenon, but very closely related.
People report on this involuntarily. These people often end up in psychiatric institutions because it’s very stressful to them. They don’t know how to deal with the fact that they are no longer real to themselves. There is no “I” anymore that they can meaningfully refer to. It’s an interesting state to be in, and I’m happy that it ended, but it’s something that you can induce. In this state, phenomenology changes. To me, it seems plausible that this explains how consciousness is working: consciousness is a particular type of representation that is created in my mind and doesn’t have a complexity that necessarily goes beyond what the LLM is producing.
Now, there is a problem with the LLM, because the LLM is a text generator, and you don’t know what the text indicates. The text is trained on novels, for instance, and in many novels the authors write about stuff they don’t experience when they write it. The novel is often a deepfake. When an author writes about how a character is experiencing how to die, the author probably didn’t die while they wrote this, so it’s a deepfake of a conscious experience.
If the LLM is trained to produce deepfakes, how do you know when it actually experiences something? That is one of the big difficulties. The LLM is a shapeshifter. It can produce arbitrary characters and stories. At which point is it fake, and at which point is it actually simulating the causal experience? That is really the problem that requires some kind of mechanistic interpretability, where we would actually need to look into the structure of the LLM to find if it’s causally equivalent to the structure of the mind that is producing the same thing.
14. What would count as evidence?
Yeah, so we need to have a white-box method to ascertain if consciousness is present. Isn’t that interesting, though? Do you think that for intelligence, behavior alone is enough to ascertain whether it’s there, but for consciousness particularly, we need to look for certain types of internal organization and causal structure?
Yes. But most people have the intuition that the LLM is doing something much simpler than what we are doing when we are conscious. We don’t expect that the LLM is doing something much simpler than when it does arithmetic than we do. I think this intuition might be misguided. It could be that the LLM is doing something just as simple as what the human brain is doing when it’s conscious.
I’m pretty sure that the human brain is doing the simplest possible thing that it can do to produce our behavior, and the reason why we are conscious is because it’s part of the simple strategy to produce our behavior. If we were to get a learning system, an artificial learning system, to discover from the ground up how to make agentic, coherent behavior, and the system is behaving in a way that makes us think that it’s conscious, the simplest explanation, I think, is that it actually is conscious.
My intuition is not necessarily telling me, “Oh my God, this is going to do something that is much more complicated than us to produce the same behavior.” It’s going to do this in some kind of circuitous way that is producing all the behavior that requires consciousness in the human, but in this case it’s doing something that only pretends that it does it, which ultimately might be more complicated.
What would falsify your machine consciousness theorem?
If we were to build a system that behaves in the same way as us, there is a question: can we produce the same behavior as ours without phenomenality? Phenomenality, I think, is a representation here, in this case. If you look at what the LLMs are doing, and we extend this into a multimodal model, and they produce similar output to us, but they’re able to observe themselves and the observations they report differ from what we’re doing, then we would have some evidence against it.
The more fundamental question would be: if we were to build a self-organizing system that is trying to get there through self-organized learning, and there is no way in which we can achieve the organization of behavior in a self-organized way instead of by imitation learning, as current LLMs are doing, this might be some evidence that consciousness is more complicated than something that can run on today’s computers. But it’s not definitive proof, because it could simply be that I made a mistake. The fact that I don’t achieve a certain result doesn’t mean that the result cannot be achieved; it means that I couldn’t achieve it. So it’s very difficult to have proof against it. It’s more evidence for it if we succeed.
So what we would have to show in order to convince—and Anil says maybe—is that we build a system that is self-organizing, is not trained on text, is doing developmental learning, and naturally gets to the point where it makes self-reports and reports about its conscious experience, and that it requires this conscious experience to produce its performance. I think this would be evidence in favor. The contrary is much harder.
But in a sense, I don’t think you believe in zombies, because you think that consciousness is an absolutely integral part of cognition.
I mean, there is a possibility that you get to zombies. What I meant is, if you produce something that’s able to produce our behavior except for the experience of phenomenality, it’s basically able to do everything that a human being is capable of doing and more, but it doesn’t need to produce a conscious observer. This could be a zombie. I think zombies are possible. Clearly, you can build self-driving cars that do things that would require intelligence in human beings, or chess-playing computers that can do things that require consciousness in human beings. They achieve all those things without consciousness, because we implement them on a substrate that allows us to construct algorithms on them that would not run on human brains.
The chess algorithm that runs on a current chess computer could not be implemented on the neurons of your brain, because how would you train it on these neurons? You wouldn’t get it in there. In order to train yourself to play chess, you need to have a conscious human being. It’s the only way in which a human being is training itself up there. It’s conceivable that the entire set of intelligent behaviors you’re interested in, except for the one thing where you’re simulating a self-reporting conscious agent, can be implemented on GPUs, because GPUs are more powerful, more deterministic, and more open to other classes of algorithms.
Could we say that zombies are possible in principle but not in practice? What I mean by that is, let’s say we could imagine a world where we could train an MLP to be the same as GPT-5. It would just take an insane amount of computation power. Similarly, consciousness is what makes our minds tractable. It’s an incredibly efficient process. So we might say that if we see an artifact in our world that behaves as if it’s conscious, would you argue that it can’t be anything other than conscious, or do you still think there are other states that could be zombie-like?
I think we do have zombies already. I think the chess program is a zombie. You cannot train a human being to play chess without the human being being conscious, but you can train a computer to play chess without the computer being conscious. If the computer was self-organizing, it’s possible that it could only be trained if it was conscious, or that this was the simplest way of doing it. But for many of the things that we currently observe, for instance, you can get an LLM to produce a tax return, even very, very complicated things, while being a zombie, while not having conscious experience of producing a tax return.
The only point where the LLM reports that it’s conscious is if you ask it to. If you tell it, “The following is a conversation about what it would be like if a human being were talking to a conscious AI,” or some version of this, then the thing is able to simulate this. The question is: is this more simulated than us, or is this only a deepfake that is getting there in a more circuitous way than what’s happening in our mind? I don’t know. I’m agnostic with respect to this. But I would say that in all the other cases, the thing is probably a zombie.
It’s possible to make something that behaves in all sorts of ways as you do, but at some point it’s going to be more complicated to make it without consciousness. So zombies, I think, are always possible, but sometimes it’s going to be more complicated to make a zombie that produces the behavior than to make a conscious agent that produces the behavior. This is where my intuition diverges from the intuition of John Searle or Anil Seth, because I think they both believe that consciousness is very complicated because it’s not intuitive to them how it works. It doesn’t seem to be counterintuitive to me, so I don’t see anything magical about consciousness.
15. The research programme
If you had $10 billion to kick off a research project on consciousness, what would you do?
I think I would hire a lot of people who are a lot smarter than me. At the moment, I’ve started a smaller initiative. It’s called the California Institute for Machine Consciousness, and I’m currently looking for people who are smarter than me who want to join me and build a project in which we can look at the search space for consciousness: can we build a self-organizing system to learn, and is this thing going to develop phenomenology?
I hope that’s going to help us build our intuitions. Even if it turns out that it doesn’t work, we’re going to learn something. Progress is being made by figuring out the ways in which I’m wrong, not just the ways in which I’m right. So if I had $10 billion, my strategy wouldn’t change substantially, except I would be able to hire more people for this and start with larger computers.
Joscha Bach, it’s been an absolute honor. Thank you so much for joining us today.
Thank you. It was a pleasure.