大脑不过是彼此对话的专业化智能体——Dr. Jeff Beck

Geometric deep learning is a big part of the stack, if for no other reason than when we talk about modeling the physical world, that means incorporating the symmetries that exist in the physical world. We're highly motivated to employ a lot of those methods and techniques.

But is the world written in code, or do you mean exploiting the regularities?

Exploiting the regularities? No, look, the world is translation-invariant. The world is rotationally invariant—well, not really, because there's gravity, but in principle, you know, there is a principal axis. It's certainly rotationally invariant in the xy plane.

Yeah.

If you want to have a good model of the world as it actually is, it should incorporate those features. Of course, you can discover them in a brute-force way, but the mathematician in me really wants to build the symmetries in. Fortunately, we've got a lot of great tools that were developed over the last several years that can do that.

What's your view on agency?

If I'm being an FEP purist, I have to say, well, there's no difference between an agent and an object in a very real way, or at least there's nothing structurally distinct between how we model an agent and how we model an object. It's really just a question of degrees, right? An agent is a really sophisticated object, right? It has internal states that represent things over very long timescales.

It has sophisticated policies that are context-dependent, which is basically saying really long timescales again, and things like that.

There's the kind of philosophical, highbrow notion of agency, where we introduce notions of intentionality and self-causation and things like that. The really no-nonsense version of agency is just a thing which acts and performs some kind of computation. I guess you could almost model anything as an agent.

Yeah. Well, if your definition of an agent is something that executes a policy, then anything is an agent, right? A rock is an agent, right? Everything has an input. A policy is an input-output relationship.

When many people talk about agents, they're adding a few additional elements that I think have a lot to do with how the policy is computed, right? So, for example, when we think of the difference between us and amoebas, we often cite things like planning, counterfactual reasoning, and goal-oriented behavior. We're specifying things that are specific to how we compute our policies, right? They're latent variables that represent policies that are compatible with reinforcement learning, and that's the defining characteristic of an agent.

But you could very easily say, from an outside perspective, if you can't look at how someone or something is doing the computations, and the only thing you observe is the policy, does that mean that you can never conclude that something's an agent? I would say no. You'd still like to be able to conclude that this is an agent, even though the only thing I ever get to measure is its policy.

But do you think we should have some notion of the strength of an agent?

The strength of an agent? Or how is this a measure of agency? Is that what you're getting at? Yeah. I think you could use notions of transfer entropy and things like that in order to estimate the timescale over which something is incorporating information, or the degree to which it's exhibiting context-dependent behavior. That would be a pretty good measure.

Now, is it normative? No, it's not. It's a measure, and you could use things like that. But at that point, you're really just talking again about policy sophistication, right? Not whether it has a reward function, or whether it's actually executing planning.

Yeah. Certainly, intuitively, agents to me seem to be kind of causally disconnected because they're planning into the future. They're not impulse-response machines. They're not just part of the mass of things going on around them. They're just obviously disconnected from the locality.

So here's the trick. I've got this agent and I know exactly what it does, right? It takes into account information. Internally, it rolls out a whole bunch of future consequences of various different actions or plans that it could take, selects the best one, and then executes it.

All of those variables that occurred inside, from the outside perspective, just look like a function transformation. Unless I'm somehow going in and recording and demonstrating that the manner in which it is calculating its policy involved doing those rollouts, I wouldn't be able to show that it's actually doing those rollouts. I would just be able to conclude that it has a really sophisticated policy.

So, can you conclude that something is actually doing planning? I think that's a really hard question, as opposed to having an incredibly sophisticated policy.

I think my intuition is that a simple input-output mapping can't be an agent. In a way, this is related to what we were talking about with grounding. It seems that when things are physically embedded in the world, they're more likely to be agents. This functionalist idea that just a bit of computer code running on a machine can be an agent—it kind of feels like that can't be an agent.

It does. So suppose I coded it up so it was doing all of that planning. It gets its inputs, does some crazy, massive Monte Carlo search, picks the best policy possible, and then executes it. Now, you don't observe any of that, right? Because you know what's going on, you could say, “Oh, well, it's clearly executing—it's doing planning and counterfactual reasoning. It's going on; look, there it is, because you coded it, so you know it's doing it.”

But if you're looking at it from the outside, if you don't know what's happening inside, all you have access to is, “Oh, here's the action that it did given this long series of inputs.” So it's really hard to identify something as an agent per se from the outside. You kind of have to know what's going on inside.

This, by the way, is why I don't think that these sorts of prediction-based approaches to AI are necessarily non-agentic. You could say, “Well, it's not really doing anything even remotely agentic unless it's doing planning and counterfactual reasoning.” So your chess program is clearly doing some planning and counterfactual reasoning because you know it's doing it. But I could describe the exact same set of behaviors just with a policy function.

I think the counterfactual thing is an important feature here, because we could take something which was conscious, or something which had agency, and just take a trace of the actual path that was found. Now we've just got a reductio ad absurdum: a computational trace, and that thing clearly has lost whatever agency or consciousness it had. So there's something about considering all of the possibilities.

Yeah. I think so. In my mind, that is the fundamental feature of an agent: if you can show that it's engaged in planning and counterfactual reasoning, then it's definitely an agent. My argument is simply that that's hard to do unless you crack it open and see what's going on inside.

Now, you could take a pragmatic view and say, “Well, if the simplest computational model of the behavior models it as if it were doing planning and counterfactual reasoning, then you can draw an implicit conclusion that, oh yes, I may as well say it's an agent.” And that's kind of the approach that I've taken.

One of the things that comes out of the physics discovery algorithm is that you apply it to agents, and what do you get? Well, you get a model. Now, bear in mind, I called them all objects before, and I didn't change anything to make it special to an actual agent, right? But what I do have the ability to do because of the model is look at the internal states associated with that object that I want to call an agent and look at how sophisticated it is.

Right. And that degree of sophistication is what allows me to say, “Oh, well, I'm going to go ahead and say that...” I like the whole idea. It's a great idea to have a metric, right? I'm sure it would be something that would effectively be like transfer entropy or something like that.

But we have this metric of how sophisticated the internal states were that were necessary in order to generate this output. If it's above some threshold, we'll call it an agent. I don't like thresholds, but we could just say a degree of agency, a degree of sophistication.

Coming back to Dennett's intentional stance, there is a level of representation which serves as a useful explanation even though it's not actually the microscopic causal graph. Maybe we can agree that no agent can possibly be the cause of its own actions.

But when there is a degree of planning sophistication, macroscopically, it's as if it's the cause of its own actions.

Yes. And that's why this “as if” phrase comes up a lot. It's important to remember that no matter how clever your model is, no matter how clever your approach is, and no matter how clever the words are that you use to describe it, a lot of this stuff is “as if.” This is the best model.

This is why I repeat this over and over again and grind it into the students: science is about prediction and data compression and nothing else. The same thing is going on here. You'll never, just looking at behavior, know for sure in any meaningful way whether it's just doing a function transformation or whether it's engaged in planning and counterfactual reasoning.

But if your best model of it is—if you say, “Well, I tried to model it as a function transformation, but goddamn it, it had a lot of parameters. Then I tried to model it as something that was just doing Monte Carlo search on the inside and giving the answer, and that had 40 parameters,” it's like, “Well, that's the model I'm going to go with,” and now I'm going to call it an agent.

If we had a physical agent in the real world that was doing all of this planning and so on, would that have some kind of primacy over a computer simulation of agents that were doing all of this planning?

Oh, is this like, if I uploaded my brain onto a computer and didn't connect it to the world, would it still be thinking even though it was doing all of those things? Is that the idea here?

That works. So, yeah, let's say a high-fidelity computer simulation of Jeff. Would Jeff be an agent?

No.

I wasn't expecting you to say that.

No, I don't know. If you did a high-fidelity computer simulation and put it in my body, then I think I would have to say it's an agent.

Yeah.

Right. If it's doing exactly the same calculations, from a purely phenomenological perspective, it's the same. It's indistinguishable.

Okay. So agents need to be physical?

I do believe that an agent needs to be physical. Absolutely. I believe you can have a model of agency and not have an agent, right? You can put that model in a computer and run it and make predictions as to what an agent would do. It might even be 100% correct, but I still wouldn't call it an agent.

Again, this is getting into philosophy, and philosophy frustrates the Bayesian because philosophy is not probabilistic. Philosophy is really about drawing clear lines and distinctions, and in my world, those don't really exist. Everything has an error bar. There isn't a clear delineation between an object and an agent. From this modeling perspective, it's really just a question of degrees, and philosophy is terrible at handling questions of degree.

My friend Keith is a big fan of computability, and he thinks that an agent is basically a type of computation. It has access to ambient state, it can take action, and there's this kind of cybernetic loop. For him, the strength of the agency in the system is the computational type that the thing is doing, right? If it's a finite-state automaton, then it's a weak agent; if it's a Turing machine, it's a strong agent.

Yeah, it's the degree of sophistication of the computation.

Pretty much. Does that ring true to you?

I mean, if you were going to make me—if you forced me, at the point of a gun, to put a measure on agency, it would probably look a lot like that.

Yes. Jeff, let's talk about energy-based models.

Sure.

Yann LeCun had a monograph out, I think, in 2006, talking about this. He's been talking about this for a long time.

Oh, yeah. When you fit your neural network to data via gradient descent, then you have written an energy function in weight space, and you're following it to its energetic minimum.

The advantage of taking an energy-based approach, as opposed to taking, say, a straight-up function-approximation approach, is that an energy-based model comes with something that's kind of like an inductive prior. An energy-based model is effectively placing constraints on what that input-output relationship can be.

I like thinking about the distinction between an energy-based model and a traditional feed-forward neural network as having to do with where your cost function is applied. In a traditional neural network, you take in your inputs, you get your outputs, and the cost function is just a function of the inputs and the outputs. The only thing that you're optimizing is the weights.

In an energy-based model, there's another thing that your cost function operates on: one of the internal states of your model. As a result, in order to figure out what the best approach is, you actually have to do 2 minimizations. One finds the energetic minimum associated with the part of the cost function that operates on the internal states, like the hidden nodes of your network, and the other is your effective prediction error.

This is very much consistent with the approach that a Bayesian would take. You have a prior probability distribution, which gives you an energy function over every single latent variable in your model, and you're optimizing with respect to all of them.

So you take a probabilistic approach. Good examples of this are variational autoencoders. A variational autoencoder, I think, is the best example of the most commonly used energy-based model out there.

Why? Because you have an encoder network and a decoder network, and your cost function is based on the difference between inputs and outputs. That's fine. That's still a regular model, but it also has some part of the cost function that is a function of the actual internal representation.

In a traditional VAE, it's how Gaussian it is. You want that internal representation to be as Gaussian as possible. If it's a VQ-VAE, then it's like a mixture of Gaussians, but it's still a cost function that is applied to the internal states as well as to the inputs and outputs.

Very cool. So a VAE is a fairly canonical example of an energy-based model. The whole deep learning world is obsessed with test-time inference at the moment, and in a way, that is a step toward what you're talking about.

Yeah, you're treating some of the weights of your model as if they're latent variables. When you show a new input, you're allowed to change some of the weights without looking at the output. So what are you doing? You're treating the weights as latent.

I think that makes it a great trick, in my opinion. It's like, “Oh, great. They're moving in the direction of energy-based models.” I love it. The only thing I don't like about test-time training is the vast majority of the training that is done.

In a traditional energy-based model, you always find the minimum with respect to the latent variables, these extra weights, which, in the case of test-time training, are the subset of weights that you're allowed to change during test time.

When you do the training for a traditional energy-based model, you're allowed to make those changes throughout the entire course of training. The way that we're often doing test-time training these days is that we just do regular old neural network learning. Then, finally, when we get to the deployment phase, we suddenly turn on these additional latents, which are basically some of the weights of the network, and we do an additional bit of learning at that point.

This seems monumentally unwise to me. Again, I'm not an expert here, but this seems unwise to me, and the reason it seems unwise is that you didn't train the original network with that turned on. You trained it in a completely supervised way.

Yes.

We should also introduce this term, transduction. My definition of transduction is that you're actually doing search or optimization as a function of the test samples. I interviewed Clement Bonnet; he had a VAE on ARC, searching latent spaces, and he actually searched through the decoder as a function of the test sample. Yeah.

And because these models are maximum-likelihood estimators, which means they're always giving you a kind of smoothed-out average, there's so much information in the test sample.

Let's just riff on the relationship between energy-based models and Bayesian inference. So, of course, they have this advantage that you don't need to do this very expensive, intractable normalization.

Yes.

Yes. Tell me about that.

My take on it is that an energy-based model and a Bayesian model have a lot in common, right? In many ways, energy—literally, in physics, energy is like log probability. Energy is log probability.

Now, of course, there's the normalization factor that you don't need to worry about if you're just minimizing energy. The difference between an energy-based model and a properly Bayesian model is sort of like saying, in a Bayesian framework, “I'm not actually going to treat some of these latent variables in a probabilistic way. I'm just going to do maximum, or MAP, estimation on some of my variables and be okay with that.” That's one way to interpret the relationship between an energy-based model and a properly Bayesian model.

There's a happy medium here, though, right? The happy medium is that you can still treat it as if—you don't have to just minimize the energy function, but you can calculate the curvature down there, too, do a Laplace approximation, and call yourself Bayesian again, right? Yes, there is more computation involved, but we've got a lot of great tricks for making that totally tractable.

What's the relationship between the free energy in the free-energy principle and the energy in energy-based models?

The regularization term, I think, is the short answer. If you're being very, very, very pedantic, the difference between minimizing energy and minimizing free energy is that free energy has this additional entropy penalty term.

If you're just doing maximum-likelihood estimation, if you're minimizing your energy function with respect to some particular—well, let's pretend we're only at 1 variable—and I'm just going to get a point estimate and call it a day, do some kind of MAP estimation to get that one thing, there's not that big of a difference, right? Because there is no probability distribution over the latent that allows you to compute that regularization term.

But that's the only difference. It's: are you regularizing or not? I think that's the easiest way to think about it.

So, LeCun is a big advocate of JEPA, these joint-embedding predictive architectures, using this non-contrastive learning where, essentially, the learning objective is comparing the latents of observed and unobserved parts of the space. This is an architectural design. Well, what does JEPA stand for? Is it Joint Embedding Predictive Architecture? There we go. So, what's the joint-embedding bit about?

Well, the joint-embedding bit is that I'm going to take my inputs, I'm going to take my outputs, and I'm going to embed them in some space, right? Then I'm going to learn a prediction between the two embeddings. That's a great idea. It's a great idea because it has some of the flavor of what we would like to get out of our models.

We're not interested in predicting every—in many situations, I should be very particular about this—in many situations, we're not interested in predicting every single pixel in the image. We want to get maybe something that's a little more gestalt, a little more high-level, a little more conceptual: an understanding of what's going on.

Emphasizing the goal of predicting every single pixel, which is what's typically done in generative modeling right now, might lose some of the abstractive power of some of the networks. So, the whole point of JEPA, as I understand it—and I'm sure there are other points—is that you're going to compress your inputs and compress your outputs, and then do all the learning in this compressed space. Love it. Right. Science is about prediction and data compression. Let's make that compression explicit on the front end and the back end.

The downside of this approach is that it doesn't work out of the box, right? Because it's very easy to find a compression, or an embedding, of the inputs and an embedding of the outputs for which prediction is perfect, which is to basically make both of them zero. So, you have to do some other things; other tricks need to be employed in order to make it work.

Yes. Yes. I remember LeCun was talking about this. There was the traditional contrastive method, which is kind of Hinton's idea, apparently, of negative sampling and whatnot. That's very expensive because you actually have to do lots and lots of sampling, and then there's this non-contrastive thing.

Yeah. This, by the way, is what he should have won the Nobel Prize for.

Right. [laughter]

In my opinion. Yes. Because the whole point of the wake-sleep algorithm and contrastive divergence was that—oh, it's actually biologically plausible, right? It was an end run around the need to do backprop, and that's what made it so clever and interesting, in my opinion.

LeCun is a big fan of this non-contrastive thing where you work in the latent space. There are many different algorithms that do this. We had a whole load of shows all about non-contrastive learning. There's VICReg, BYOL, and Barlow Twins, and there's an entire thread of research all around that.

In many different ways, what they're trying to do is avoid this mode-collapse problem that you're talking about, and they use different forms of regularization. There's an old-school way of accomplishing the same thing, and that is to do all of your preprocessing. This is something that a lot of people do.

Take your data. In fact, we do this all the time with vision-language models. We want to use an LLM and predict images, so what do we do? The first thing we have to do is tokenize the image.

Right.

What do we do? We run a VAE. We do the preprocessing, and the preprocessing step is completely independent from the actual algorithm that's going to be tasked with solving the problem of interest. That's not something that we necessarily have to stick with, right?

It would be very nice if there was a way of—again, we're getting right back to JEPA. What we'd like to do is choose our preprocessing algorithm in a manner that is not a priori, not doing it first. We'd like to choose the preprocessor that works best in this space.

I think that's the ultimate motivation for a lot of this work: what's the right embedding? One of my favorite tricks is, of course, I preprocess with VAEs all the time. In fact, the second someone hands me a new neural dataset, the first thing I do—and I'm not ashamed to admit it—is run PCA on it, pass it through a VAE, and then take a look, right? It's the first thing you do with your data because it gives you a good idea of what the signal-to-noise ratio is in the dataset itself.

And then what do I do? I subsequently do most of my analysis right in that discovered embedding space. I don't see a huge problem with that from a purely pragmatic perspective, but it's certainly cleaner to have a single algorithm and approach, rather than just stringing these sorts of things together in an ad hoc way.

PCA is a really great example of this. There's a failure mode for principal-component analysis that's actually really common in neural data, because principal-component analysis basically goes, “Where's the most variability? Okay, I'm going to worry about that. All the stuff that's not varying very much, I'm just going to throw it away.” Dimensions in which there's low variability are not important.

Well, it turns out that in neural data, the dimensions in which there's very little variability are some of the most important dimensions. So, preprocessing with PCA runs the risk of throwing out the most valuable information in your dataset.

Yes. And so there's a lot of wisdom in jointly fitting your preprocessing model as well as your inference and prediction model. On this subject of not throwing things away, JEPA and non-contrastive learning are part of this bigger field of self-supervised learning.

We want to learn representations that maintain fidelity and richness, and LeCun's hypothesis is that when you do something like supervised learning with a particular downstream task in mind, the neural network gets wise. What it does is discard all of the long-tail stuff that isn't relevant for that particular task.

When you train these models, what you're trying to do is maintain enough ambiguity so that it compresses the information, but also maintains enough fidelity to work broadly for different things.

Yes. And that is a laudable goal, right? I certainly share it. The last thing you want to do is train your network to toss information that you might need down the road.

Fortunately, networks are so big that we don't really run the risk of overfitting as much as we used to. But the last thing you want to do is throw away information that you might need down the road. That said, the vast majority of what the brain does, just like these neural networks, is decide what information is currently task-irrelevant.

But that’s all the more reason to do things in a self-supervised or unsupervised way, right? Because you’re basically not telling it what’s important. You’re not telling it what’s task-relevant and task-irrelevant.

I interviewed François Chollet about version 2 of the ARC-AGI challenge, and one thing that struck me is that I think of intelligence as being multidimensional. Version 1 got saturated. ARC-AGI was actually really amazing because it’s the only intelligence benchmark that survived for 5 years before being defeated. Since the advent of these thinking models, it has been defeated very quickly.

But they’re working on version 3, and there’ll be version 4, version 5. Will there always just be something left over?

That sounds like another philosophical question. So yes, that’s my answer. There will always be something left over, in the sense that this has been the trajectory things have been following for a really long time, right? We get algorithms that do amazing, new, cool things, and then someone comes along and says, “Yeah, but it can’t build me. It can’t pull a rabbit out of a hat.”

And then, of course, what does someone do? They figure out a new training protocol, a slightly different architecture, or they just train it to pull rabbits out of hats, and suddenly it can. Then someone proposes a new challenge, and a new challenge, and a new challenge. It’s always this game of one-upmanship.

So the question becomes: What’s the point at which there are no more new challenges? I’m not entirely certain we’re ever going to get there. It may very well be the case that we get these sorts of algorithms that are capable of replicating the complete suite of human behaviors, and then someone will come up with some criticism like, “Yeah, but it’s not really doing X. It’s just faking it.” This is just the direction things go, because people really do think they’re important.

Do you think that the concept of recursive self-improving intelligence is a valid one?

Yes, I do. I think that one of the most critical missing elements right now is some form of continual learning. At the end of the day, you really want an algorithm that doesn’t just learn on the training set and then get deployed. You want something that runs around in the world, comes across things that it doesn’t understand, and is able to build and append to its model in some sense.

There are some approaches to this. It’s all based on Bayesian nonparametrics, Dirichlet process priors, and things like that, where you see something surprising, unique, or different—something you didn’t expect—and it causes you to say, “I need to turn learning on because I’ve got to figure this out.” That is an absolutely critical element that we need to be developing. We are developing that.

One of the nice things about this object-centric physics-discovery approach is that, because it’s object-centric, if it comes across a new situation that it does not understand, it’s capable of instantiating a completely brand-new object just to explain that new situation.

Continually learning agents can acquire new knowledge autonomously, and the whole thing just learns more knowledge. But intelligence feels different. In the system that we’ve been describing, the intelligence is the way we’re implementing the Bayesian updates and actually building the algorithms. Could the systems, on their own, meta-program themselves and develop better algorithms or something like that?

Something that would be closer to true artificial intelligence than what we currently have would be capable of building models on the fly to deal with new situations, taking things that it knows about and combining them in new and different ways.

There are approaches that have some of that aspect to them. GFlowNets from Bengio’s work are a great example of something that, at least in principle, is a generative model of generative models. It’s sort of like, “I might actually need a new node. It’s time to create a new latent variable because the current set just isn’t cutting the mustard anymore.”

Those are things that I think are hallmarks of true intelligence. I don’t ever want to make the statement that as soon as it’s got that, it’s truly intelligent. I will never, ever, ever say that. But I do think that is a critical component that needs to be present: the ability to generate new models on the fly to deal with novel situations and data, as well as the ability to combine old models—previous models—in new and interesting ways.

This is actually how the brain evolved, right? We started out with really simple brains, and there were different regions that solved different problems. What eventually happened as we evolved is that these different regions of the brain learned to communicate with each other in new ways, and through that communication acquired new abilities. Then they eventually evolved into new capabilities and things like that.

I often like to point out that olfaction is a sense that’s not studied nearly enough. It’s an incredibly old part of the brain. Arguably, it’s the first part of the brain that evolved the ability to do proper associative processing.

Odor, unlike visual space, where there are translation symmetries and all that sort of stuff, and things are smooth—olfactory space doesn’t have that. It’s really, really, really combinatorial and complicated. The part of the brain that evolved to solve the olfactory problem arguably evolved into our frontal cortex. Don’t quote me on that; there’s a lot of disagreement there. That’s just my take.

But it certainly has a lot of the features that we associate with associative cortex. Wow, I just used the word “associate” three different ways in that sentence. But I think you see what I mean, right? It was all about taking old capabilities and combining simple models and modules to create something that was more complex.

That was what made the brain work. It was all about taking little things that worked and combining them in new and different ways in order to evolve, effectively, emergent properties, emergent computational abilities, and an emergent understanding of the world in which we live.

I do think that when we get to the point where we start really saying, “This is actually truly intelligent,” it’s going to have that feature. It’s going to have a modular description of the world and the ability to combine those modules in a way that creates a more sophisticated understanding.

It’s like LEGO bricks, right? The LEGO bricks all connect in certain ways, and I can build all sorts of new and amazing things that were never built before out of them. That’s a capability that we have, and that’s the essence of creativity. It’s why I refer to systems engineering as the thing we really want our AI models to be able to do.

Collective intelligence is a bit different. We have this plasticity, right? We can adapt our behavior day by day. We might see some kind of meta-learning or some kind of change in our organizational dynamics. Maybe some agents will specialize, and it might be an existence proof of this kind of recursive superintelligence that we’re talking about.

Yes, I do. I think that’s absolutely correct. Specialization is great. In fact, I would argue that specialization is how we got all of this. I’m pointing at London, in case there was some confusion there. It was really about the interconnected, highly specialized intelligences that are people, and their ability to learn how to work together, that gave rise to the technological revolution.

The brain is the same way. In my view, it’s highly specialized little modules or agents that are capable of being repurposed and reused, and capable of communicating with one another in order to solve really complicated problems.

But there’s always a benefit to specialization. I don’t believe in AGI. AGI seems like a bit of a misnomer to me. What we really want is not artificial general intelligence; we want collective specialized intelligences.

What about scientific discovery? What would the world look like when we could discover new drugs and new knowledge in science?

Right now, the way that we’re doing that is largely focused on summarizing vast troves of data and looking for correlations that are present in it. I think the next major milestone in this trajectory is experimental design.

It’s not just, “Here are some correlations you may not have seen because they’re really small.” This is what computers are good at: They’re really good at identifying small but highly relevant correlations. The next step, of course, is constructing a system that tests these hypotheses explicitly and generates the experiments that will fill in the gaps in our knowledge.

I believe all of this can, in fact, be automated in a very sensible way.

I don't see any major obstacles to automating empirical inquiry, other than we probably want to place some safety constraints when we start letting them work—when we start letting the AIs run the labs—because you never know. You always have this AI that's like, “Well, the most effective experiment to determine if this is correct is to set off a nuke,” and that would be bad.

Yes.

Pure empirical inquiry does run risks like that, but I don't think that's the biggest issue. What we need to do is have a nice, concise framework for saying, “Oh, look, here's an example.”

We had this problem that popped up a while back. A gentleman we were talking to said, “You've got these robots, and the robot sees something it's never seen before.” A robot is running around and comes across a beach ball that it's never seen in its entire life. What you'd like is for the robot to know how to figure out that it's a beach ball and to figure out what its properties are.

If you tell the robot, “If you see something new, just stop,” then that's no good. What you really want to do is figure out a relatively noninvasive procedure for the robot to poke—to do what a child would do. What does a kid do when they see a beach ball? They run up and poke it and say, “Oh, right. Yeah.” Then it moves, and the robot actually learns. It experiments with its environment for the purposes of identifying the properties of the objects that exist in it.

I do think we probably want to test this out virtually before it's deployed in the real world, because you never know. It might very well be that the optimal experiment is to run up and kick it as hard as you possibly can, and we certainly want to avoid that. Something along those lines—a robot that is able to test the theories that it has about how things work in an online way and learn from those results in an online way—is definitely part of the goal.

Looking forward, what do you think the future will look like when we have more autonomous AIs among us? A lot of people worry about enfeeblement, loss of control, and it making us dumb—all of this kind of stuff.

I do worry about AI making us dumb. Offloading your thinking onto a machine, which is something that AI allows, is potentially a big problem. I don't really want to have a situation where humans are reduced to value-function selectors. They're just basically going, “Oh, no, I don't like that outcome. Do this instead.”

I do want to see a future where we have an AI that actually improves our understanding of the world. Simply automating everything runs the risk that you specified. It runs the risk of people becoming couch potatoes that just watch TV and occasionally say, “Yeah, these chips are no good.” That seems like a bad outcome to me.

I worry less about that than some because people are remarkably adaptable, right? They have all these arguments about how a new technology comes along and it's going to completely destroy a way of life, and that's going to be awful for people. It maybe is in the short term.

I think of tractors. Or just go back a few hundred years: 99% of people were involved in agriculture, and now it's like 2%. I consider that a solid improvement because it allowed the rest of us to do a bunch of other things that we find more satisfying and more interesting. It allowed us to spend some time reading a book; we don't have to labor in the fields all day.

That's the future that I see and the future that I hope for: one in which all of these artificial agents running around and doing things autonomously are there to free us up to pursue more interesting things and to improve ourselves in more interesting ways. At the end of the day, it's just another technology, like the tractor. Now, 100 years from now, who knows? What will the value of work be if the AIs can do everything and there's nothing left for us to do?

I don't think it will ever be the case that the AIs can do everything. Like I said, the future I worry about is one where the sole role of people is sitting around making sure the AIs aren't going rogue and things like that, which I don't consider a good outcome.

I would really like to see human improvement. I envision a future of—I don't know—cybernetic transhumanism, if I'm going to go sci-fi on this, where the technology and us evolve together in a way that's beneficial for both. That's the goal.

Are there dystopic possibilities where humans live in a world where everything can be done by a robot?

Yeah, that's a good question.

Dr. Jeff Beck

At the end of the day, humans end up just becoming reward-function selectors. They end up saying, “I don't like this, and I do like that.”

This is another nightmare scenario. I don't like talking about these dystopian futures because, honestly, I think people are too clever, too motivated, and too interested in how the world really works. People are too interested in actually understanding things. AI will become a partner, not an adversary or a crutch, and that's what I think will happen.

That's a statement more about my belief about humans than it is about my belief about the development of AI. I am a techno-optimist, if you will, not a pessimist. I believe that we will find a way to adapt to an ever-changing world, as we have done for millions of years, including one that involves technology that alleviates most of our labors.

On that, there's an AI-literacy issue because AI has moved so quickly that certainly my parents don't understand anything about it. By the same token, policymakers don't understand anything about it. There are people saying AI is going to kill everyone, and there are people making negative arguments; there are people making positive arguments. There's a bit of a fog of war now because so many people are saying different things about AI. How should they make sense of all of this?

Dr. Jeff Beck

We are now well outside my area of expertise, so I'm just going to say that before I say anything else. AI is developing very quickly, but I am much more concerned about what people will do with the new technology than I am with what the technology will do all by itself.

I don't have this big concern about—I don't really believe that Skynet's going to take over or the internet's going to suddenly become conscious and kill us all.

Right.

Dr. Jeff Beck

In part because AI is not that advanced, but also because we are still in the position where we specify the goals of the system. That will likely continue for a very long time, and these systems will always be subject to review. We will always keep an eye on them. They will always, at least initially, be released in relatively restricted domains, where we're testing them and keeping a close eye on what they are and are not doing.

I don't worry too much about them going rogue. I worry a lot more about somebody building something like a virus, which we already have to deal with—somebody builds some insane virus and takes down the internet. I'm more worried about malicious human actors than I am about malicious AI actors because, at the end of the day, all of these algorithms simply do what they are told. We train them; we tell them, “Here's your objective function.” As long as we are specifying the objective function and we understand the objective function, we're probably going to be okay.

I think the safest way to deal with AI concerns is to tell people, “Hey, look, this AI is just doing what we told it to. We set it up to make really good predictions and to achieve these outcomes.” Now, is it dangerous to specify these outcomes without being very, very, very careful? Yes, it is.

This is the whole, “Hey, Skynet, end world hunger,” and it kills all humans. That is a real possibility. But whose fault was that? The fault was the person who very, very naively specified their goals.

There are, in fact, relatively straightforward ways to specify the reward function that don't run that risk nearly as badly. The best one is—are you familiar with maximum-entropy inverse reinforcement learning? I like to call it active inference because it's really similar.

What you're doing is you're basically observing someone's policy, and then you're trying to do a maximum-entropy model. You're doing maximum-entropy modeling on the reward function itself. At the end of the day, what ends up happening when you do this—and this is why it's basically just like active inference—is you get a reward function.

You have some organism or whatever, and you're trying to do this for it. It has some stationary distribution over actions and outcomes, right? Its inputs and outputs are a stationary distribution that becomes your reward function—not directly; there's some math involved—but basically your reward function is a function of the steady-state distributions over actions and outcomes.

So we could do this. We could take the current manner in which humans are making decisions, and we could write down: What is the current estimate of the stationary distribution of our actions and outcomes?

So this would include things like everyone getting—this number of people are going hungry, and all the statistics that describe the inputs and outputs to our policy-making, to our policy decisions. We could just ask an AI: “Your reward function is the one that results in the same outcome that we currently have, on average,” and it would execute it. To the extent that it works, it would ultimately result in an AI algorithm that is just sort of mimicking human behavior, right? Or at least achieving the same outcome that we were achieving before.

Now, here's the safe way to improve the situation. You don't say, “End world hunger,” right? You perturb that distribution over outcomes just a little bit, and then you evaluate the consequences. That's all you're doing. You make these little changes in the reward, in an empirically estimated reward function, rather than specifying one by hand, because that's the dangerous thing.

Jeff, thank you so much for joining us today.

Dr. Jeff Beck

It's my pleasure.

Amazing.