ImageNet Moment for Reinforcement Learning? [Prof. Jakob Foerster]

The ARC challenge as a target for the community is a terrible idea. That's not what it's supposed to be. It's not supposed to be something where we then design methods to solve the ARC challenge.

Fundamentally, AI is trained on the collective outputs of humanity. This technology belongs to everyone, including people we don't like. I prefer centralized resources in the interest of the common good, in the interest of the public, and not in the interest of maximum profit, because to me, the biggest alignment challenge is not between AI and humans. The biggest alignment challenge is between those people who hold the keys to power and control these systems and the rest of the population.

Jakob, welcome to MLST.

Thank you for having me. It's great to be here.

It's amazing to have you here. Tell us a little bit about your background.

I run the Foerster Lab for AI Research at the University of Oxford, which is nowadays about 30 people working on anything that's cutting-edge and interesting—not supervised learning, but thinking beyond the current state of the art. I do this 50% of my time, and I spend the other half at the Fundamental AI Research group at Meta.

Amazing. I watched your talk at ICML earlier, and it was really good because you were sketching out a potential ImageNet moment—a sort of Alex Krizhevsky moment for reinforcement learning. What was the elevator pitch?

1. Deep RL Wins the Hardware Lottery

I think reinforcement learning in the last decade or so really hasn't lived up to its potential. That raised the question: Why has deep learning been such a revolutionary success story, while deep reinforcement learning, which held such great promise, really hasn't quite delivered in terms of real-world impact?

We have a hypothesis at FLAIR, which is that deep reinforcement learning had lost the hardware lottery. Deep learning is perfectly suited for the GPU: We can keep all the cores busy turning through data efficiently. In contrast, the way deep reinforcement learning has been done in the field is by running reinforcement learning environments on the CPU, while running the agents on the GPU.

This has meant all sorts of complications and complexity, both in algorithm design and in terms of what hardware we need and how we're going to develop algorithms for it. That has really slowed down the field and made experimentation very slow and difficult. We're currently in a revolution that's allowing us, for the first time, to run environments and agents together jointly on the GPU, making deep reinforcement learning a winner in the hardware lottery. Hopefully, this will be the step that really makes reinforcement learning work in the real world.

I suppose it's not just the hardware lottery. It's also the bitter lesson, which is this idea that when we scale up compute, we get dramatic performance improvements. But the thing holding back deep reinforcement learning to date has been this bottleneck: So much of it is running on the CPU.

It does remind me of an article by Alex Irpan from around 2018, where he was saying that deep learning just doesn't work, and reinforcement learning doesn't work yet. He was also pointing out some of the other problems: If you change anything or do anything wrong, the whole thing breaks. Why is reinforcement learning so sensitive to the architecture and the parameters?

I think what has happened is that, because experimentation has been so slow, we've only been able to train our algorithms on very specific environments. We've only been able to hill-climb on that small set of environments, which means the things are fundamentally brittle.

Now, for the first time, the experimental loop has been sped up by orders of magnitude. That means we can really start to develop methods that are robust. Remember, we scientists are ourselves doing meta-learning: We're trying to discover methods that can generalize.

MLST is sponsored by Sensl, which is the compute platform specifically optimized for AI workloads. They support all of the latest open-source language models out of the box, like Llama, for example. You can pay on consumption, essentially, or you can have a model which is always working, or it can be freeze-dried when you're not using it. All of the models that they deploy support the OpenAI API specification out of the box, which means it's just a one-line change in your application to switch over to Center Mail and start saving money and make your application go fast. But to do so, we have to be able to get a lot of samples and a lot of experience for how our methods actually work. In deep reinforcement learning, any single run has taken a lot of cost, compute, and time. That means the signal we've been getting as researchers has been costly, sparse, and noisy. That meant our methods were bad.

The premise of the bitter lesson is that if we can turn through more data, we can get a better signal and a better gradient update, either for the researchers or for our meta-learning methods. We can then optimize our algorithms to be more robust and more sample-efficient in the real world.

Can you give me an example?

2. Simulation Breaks the Data Wall

Certainly. When we need physical experience, the bottleneck is that we need physically embodied agents gathering data and experience. We can use simulators for that, but if I want to learn Quake or Dota or something like that, don't I need to actually have the game running on my machine?

I think this is a general problem: What happens when real-world experience is expensive? Obviously, what we've been really good at as a field is using real-world data when it's given to us. But we're going to run out of real-world data at this point, right?

We've been in this regime where we could lazily scale up compute and data because datasets were large enough to accommodate more months of compute. But now we're hitting the data wall. The question then is: If I can get more data from the real world, how can I use simulation in environments that are not perfect and not exactly the same as the real world, but are blazing fast and allow me to get a lot more data in a way that's fully synthetic?

That's really one of the key questions right now for the field—not just for reinforcement learning. How can I develop algorithms and discover algorithms in approximate versions, for example, of Dota, that run 10,000 times faster, in a way that means the algorithms we discover will generalize to the real Dota environment, which is slow and expensive to run?

I put this under the umbrella of compute-only scaling. If you just give me tons and tons of compute, what kind of methods can I develop that will allow me to make progress on simulated situations, environments, and scenarios, so that the learning progress transfers to downstream tasks in the world that are slow and expensive to run?

Tabs is a new AI research lab I'm starting in Z. It is funded from past ventures involving AI as well, and so we are Swiss version of deeps, so a small group of people very, very motivated, very hard working, and we try to do some research starting with LLM and OW-style models. What we're looking for now is chief scientist and also research engineers. You can check out positions at tabs.a.

3. JAX Brings Environments to GPUs

There are 2 things we can go into. In a little while, we'll go into things like unsupervised environment design, curriculum learning, and ways of generating all of this data. Before we get there, though, there's the matter of how you made this run entirely on the GPU.

I think at this point it's best just to refer to the absolute expert on all of this—the mastermind behind the technical innovation powering this revolution of hyperscale.

Hi, I'm Chris. I'm a PhD student who was with Jakob. I technically have not graduated yet, so I'm still a PhD student with Jakob. I mostly study things like automating machine-learning algorithm discovery.

Basically, can we find ways to discover new machine-learning algorithms and show insights automatically? To do this, we need to use way more compute. You can imagine that even your average AI scientist is using tons of compute. To automate this and scale it up, you need exponentially more.

Obviously, in an academic setting, we don't have that much compute. In fact, when we first started at FLAIR, I don't know if we had any compute set up. I think we might have had some Colabs—essentially the free tier of compute on Google Colab.

We wanted to run some basic reinforcement-learning experiments for a paper called “Model-Free Opponent Shaping.” This is a paper where you're trying to learn across the entire learning trajectory of another agent in order to influence the way it learns. This requires as many training iterations as a single agent.

Given the hardware restrictions we had, we couldn't even run normal reinforcement-learning algorithms at that scale. We had to look into using really simple environments that we could write in PyTorch and implement in Colab so they could run quickly on the GPU. The paper had really cool results, and we were surprised by how effective putting the environment on the GPU was at the time.

There were very few environments that could be implemented in this way, though, and it was really hard to implement them. PyTorch, as you know, is designed for your networks and things like that, so using arbitrary environment code in PyTorch was difficult.

That's where JAX came in. JAX is a library by Google that's similar to NumPy and PyTorch. One of the neat features of JAX is that it has the same interface as NumPy, so if you know how to code in NumPy in Python, you can code in JAX.

These are both libraries designed to allow you to use Python to run things on the GPU. This is how we train all of our neural networks and language models. JAX was developed by Google, but it has a separate interface from PyTorch. PyTorch is probably the one most people are more familiar with, whereas JAX has a few extra features on top of it.

The key ones are `jit`, which allows you to compile your programs for the GPU. At a high level, this means they'll usually run faster than PyTorch in many settings. The second one is `vmap`, which stands for vectorized map.

The idea is that if I write a function to run, say, addition, you can imagine that I need just 1 core to add 2 numbers together. When I apply `vmap` to this addition function, it turns into a vector addition, so I can add 2 vectors together. If I apply `vmap` again, it turns into a matrix addition.

It's the idea that I can write 1 simple function for 1 instance of my environment. I can write 1 instance of CartPole in normal NumPy, call `vmap` on this function, and now I can run millions of instances at the same time.

Any environment you can think of is really easy to implement in JAX. For our follow-up paper scaling up this approach, we used JAX to implement everything entirely on the GPU.

Back in the MATLAB days, vectorization was always what we wanted. We wanted to take 1 operation, spread it out, and parallelize it so it runs on many of those little cores in the GPU. In this case, CartPole might be an interesting example.

We want to capture environment dynamics on the GPU. Is there any limitation? Obviously, we could write an environment in Python, and Python has lots of useful features. It's a very rich language, and you can do iterative and conditional logic. How is it different in JAX?

It's very similar, because NumPy is what I think a lot of people use Python for. If it's in NumPy, you can more or less do it in JAX. There are some scenarios where JAX is worse, such as when you have a lot of `if` statements and branches.

Some of the recent work from our group has shown some really surprising things you can do with JAX. For example, a recent paper by Mikey and Michael from our group is called “Kinetix.” This is basically a general physics simulator on top of a general renderer, entirely implemented in JAX.

You can imagine that any reasonable environment you can make could be built using this simulator and renderer.

Very cool. Are there any examples where—you know, we were saying before that we need to have a sketch? We don't have the source code for Dota, but we want to capture as much of the dynamics as possible so that we can build agents that learn. How can we do that?

One interesting way is that you can just try to build a model of Dota. You've seen recent work where people can learn models of video games like Minecraft, or the recent work on Genie. You can generate these video games, and once you're able to do that, you can sample from them much faster because your network runs on the GPU.

That's really cool. There's a new version of Genie out now, too. It hadn't really occurred to me that you could just have a dynamics model for the purpose of training. We can chain all of these things together much more easily if we don't have access to the original source code.

What kind of performance speedups are we talking about here?

Around 4,000 times, I think, is our basic speedup.

Wow.

I think there's also room for a lot more speedup. We're just using JAX naively, but there's tons of room for optimization if you go into lower-level code and write certain kernels.

When I watched your talk, Jakob, you were saying that in the olden days there were ways of distributing, parallelizing, and so on, but it created so much complexity. With this new method being so much faster, even a small lab can do the kinds of experiments that the big labs would have been doing before.

It's not just being able to do experiments that the big labs could do and we couldn't. It's also being able to simplify algorithms. We can take out a lot of the complexity that was built in, which makes it difficult to understand what these algorithms are doing, and say, “Let's actually just write these very clean algorithms.”

We have a paper from our lab called “PQN,” or Parallelized Q-Network. It's extraordinarily simple. It's basically just a Q-learner where a lot of different agents, across different cores, step through the environment and learn on every transition.

There are no more target networks and no more replay buffers. All of that is gone. Hopefully, this will allow the field to come up with much more beautiful and understandable—and therefore also more robust—algorithms in the future.

4. Meta Learning Rewrites PPO

Talk to me about drift functions and objective optimization.

This was one of the first things we did at FLAIR. We had a paper called “Mirror Learning” that provided a theoretical framework which, to me at least, gave the first intuitive understanding of why things like PPO actually work.

The framework said that as long as we have a penalty term that penalizes the difference between the policy that collected the data and our current updated policy, and that penalty obeys certain properties, then in the limit of doing many policy updates, we'll converge to an optimal policy.

That theoretical framework was really nice, but it also allowed us to ask: Why don't we learn a different function? PPO's clipping function is just 1 of many possible algorithms that can be expressed in the mirror-learning space. We thought, “This can't be the optimal one. There must be better ones.”

We set out to parameterize the drift function as neural networks. Obviously, you can now imagine the question: How do you actually optimize through that entire reinforcement-learning loop?

There has been a lot of work in the field on meta-gradient estimation. This was trying to estimate the derivative by unrolling the computation graph and differentiating through it. I had done some of this work in my PhD, coming out of the multi-agent shaping work, and we pursued this path because that's what everyone thought was going to win.

But we also pursued evolution strategies, which don't do any of the sophisticated mathematics. Instead, they swallow the bitter lesson and double down on it by doing black-box optimization and trying to estimate those higher derivatives from samples.

That turned out to be extremely well-suited for the new paradigm of AI at hyperscale.

There was a beautiful figure in your talk at ICML. You were visualizing the gradients of the objective function, first for DPO. It has this kind of step: It's pulling the policy back if it drifts too much from the source data.

The interesting thing about that is that it's human-designed. This is what Rich Sutton said we shouldn't do. We're a bunch of experts in reinforcement learning, and we externalize our intuition into this function.

What you did was meta-learn this function and then visualize it. What did you see?

The interesting thing is that it recovered some of the features we had seen in PPO. There was this clip-like behavior, but there were also a few novel aspects.

For example, PPO's clipping function may counterintuitively have this “too good to be true” type of behavior. You're willing to update with a positive gradient if you're beyond the clipping region, as long as your advantage isn't too high.

Intuitively, you might think that if I have a large advantage—if things turned out much better than I thought—I should be more optimistic and move farther away from the reference policy. Instead, this learned clipping function did the opposite. If the advantage is high and you've moved farther away, then you have to stop there. But if the advantage is small, you can move farther away from the reference policy.

It's almost as if it's saying, “Is it too good to be true? Then you should just stay here.”

Yes. But if there's a small advantage, it's okay for you to keep moving.

Cautious optimism.

Exactly. The other thing this process discovered—and we then realized had actually been discovered by humans before—is rollback.

While the PPO objective says that if you've gone too far away from the reference policy and you have a negative advantage, you're not going to get a gradient and you'll stay there in that lower-left quadrant, instead, DPO discovered that you should actually go back. It pushes you back toward the reference policy.

That's absolutely fascinating. There were also secondary features that it found, which no human had designed or even thought of before.

This is where our interpretability effort came in. Chris and I spent a lot of time, along with Alistair, one of the authors of the paper, trying to make sense of these features. We don't know what they are. I think this is an open problem. It would be fascinating to figure out what is going on there and whether this is actually doing something real—perhaps something that, one day, another paper will explain.

That makes sense. So we've meta-learned this new optimization, this new set of gradients, and it's a little bit slow. I think what you did next was ask whether we could represent this in a closed-form solution and develop new theory based on it. There's this virtuous cycle of discovery.

The hope was that, at that point, we hadn't yet transitioned to doing science end-to-end with AI agents. Having a human in the loop who could interpret this and get back to a symbolic representation of the drift function was really important to us.

It has a second advantage: Suddenly, you can break out of the JAX box. Remember, the environments we're going to use in JAX will not be the real world. We can't implement every problem, but I think we can have a representative set of types of challenges that make the learning algorithms we discover transfer to the real world, transfer to settings in other simulators, and also transfer to learned world models.

Having a symbolic representation means we can write down that drift function in 1 line of Python. That's a really nice way to make sure it transfers to different downstream tasks and other code environments.

What else did you notice about this policy that surprised you? Didn't you say that it explored much more than before?

Yes, it had implicit entropy regularization. This is something we've doubled down on ever since. In our follow-up work, we've said that humans can design these clipping functions, but something humans certainly can't do is design a clipping function that's time-dependent—where time means how far into the optimization process the learning algorithm is.

This temporally aware version of PPO turned out to be very explicit about trading off exploration early on with becoming more conservative along the path of the optimization process. Obviously, this is a huge design space, because now you're designing a clipping function for every point in time. Again, this is where meta-optimization shines.

The only thing we haven't been able to do in that case is go back and ask whether we can find a parametric version of this kind of clipping-function manifold.

The mirror-function manifold?

Yes, the mirror-function manifold: 1 mirror function for each time step in the optimization process.

This sounds very complicated, but the good news is that with LLMs we could use LLMs to try to fit that black-box drift function with symbolic code.

One other thing is that you've open-sourced all of this code, so people can play with it. I know you're a huge fan of open source, Jakob, and I'm going to talk about that in a minute.

Using these new methods, increasingly we can use LLMs as engines of creativity. We can have an additional meta-stage where some kind of engine creates the meta-optimizing reinforcement-learning system.

5. LLMs Mutate Learning Algorithms

This is something we're starting to explore. We've done a few papers at FLAIR, along with a few works in progress, where rather than using neural-network black-box function approximators, we explore the space of programs.

Again, we use JAX at hyperscale to get relatively fast feedback on the different members, and then use LLMs as a mutation operator to explore the space of programs. The fitness is the performance on the downstream tasks of the reinforcement-learning algorithms we're exploring, and that serves as the mutation signal.

I think this opens up an entirely new space of automated reinforcement learning. As we discussed before, this makes the question of how we prevent overfitting even more important.

There's Goodhart's law, which says that when a measure becomes a target, it ceases to be a good measure. This has already happened when we do science with gradient descent. We use lots of exploration and trial and error to optimize our benchmarks.

Now imagine if we can scale this up through automated research. This is something else we've been talking about a lot at FLAIR: How do we make sure that the algorithmic progress we find in the meta-loop actually transfers downstream? What's the right framework for thinking about meta-training and meta-testing? Along which axes should we be generalizing? How do we know that this is real?

On the subject of creativity in LLMs, I just used the ARC challenge as an example. Many people tried to solve it in a very formal way, doing discrete exponential searches over DSLs and things like that. The way humans approach the problem is very heuristic, creative, and serendipitous, as Kenneth Stanley would say. We're all huge fans of Kenneth Stanley here.

Why are LLMs so good at capturing our instincts?

I think there are 2 answers. On the ARC challenge, very quickly, I want to offer a word of caution.

Remember, when a measure becomes the target, it ceases to be a good measure. I think the ARC challenge as a measure of progress is brilliant. It shows that our systems are lacking fundamental capabilities. But the ARC challenge as a target for the community is a terrible idea. That's not what it's supposed to be. It's not supposed to be something where we then design methods to solve the ARC challenge.

This is where open-endedness comes in. We'd like to have methods that can solve a broad range of diverse tasks, and the ARC challenge is 1 example in that space. That means we have to target the entire space of human-solvable problems.

Exactly. We can use LLMs, I think, to help span that space. There's the question of whether LLMs can span the convex hull of creativity. We can talk about creativity as well—I love discussing combinatorial creativity versus inventive creativity.

But let's say, for argument's sake, that the convex hull is good enough. Then we've got the question of developer-aware generalization. François Chollet wants a solution that isn't just for ARC, and most of the solutions are not in the spirit of ARC. They use test-time active fine-tuning, test-time training, and various other methods.

They're great methodologies for rapidly solving a task in a human-supervised way, but they won't generalize from their initial instantiation to another task. How can we cross that bridge?

There are 2 answers. One is that, as a community, we need to be much clearer about measures and targets. We use the term “benchmark,” but a benchmark is supposed to be a measure, not a target.

In practice, this means we have to work much more clearly on addressing broad problem spaces. The benchmarks should just be 1 instance in that entire open space of problems, and we should never use the benchmark during the development process of our methodologies.

What this would mean is that I'm not working on ARC; I'm working on human-level reasoning capabilities. In my entire pipeline of method design and training, I never use ARC. I only use it once a year to measure my progress. I'm not just using ARC; I'm using other examples like ARC, where LLMs struggle but humans can make progress.

The other option is that, rather than having unique benchmarks that we confuse with the target, we make benchmark design a first-class member of our scientific progress. We're trying to generate benchmarks that span the entire space of problems. If you hill-climb across this entire space of problems, you're hill-climbing across all of human capabilities.

I don't think we've made that much progress on the latter, so for now, being very clear about the distinction between measuring and targeting in the community is extremely important.

What's the relationship between creativity and reasoning?

That's a good question. I think creativity allows me, at least, to come up with new reasoning challenges. If I think about how I go about my day, I'm commonly using creativity to try to create new problems for myself—and, frankly, for the lab and for the research community—and then explore the space of solutions.

One of the skills needed to solve these problems is reasoning. Creativity is essentially allowing me, and researchers in general, to explore a space of interesting and relevant problems that can then be used to train our reasoning capabilities, much like curriculum design.

Creativity is a great driver for figuring out what problems are interesting. I think this is 1 of the key challenges right now for open-endedness: What actually constitutes an interesting problem?

Obviously, if we just rely on LLMs, we're going to make the measure the target. We'll start Goodharting the judgment of LLMs, but what is interesting? At some point, we'll just find examples that exploit the inaccuracies of these LLM judges.

To what extent is reasoning itself a creative process? Even something trivial like deduction involves searching the deductive closure. We're traversing all of these different things, and we find a trajectory. We've essentially composed a new piece of knowledge, evaluate it, and it works really well. You might just say, “That's doing deduction,” but I think it's a creative process.

I think it depends on how structured your search space is. For example, to me, playing chess the way a human does has a strong flavor of creativity. You can't simulate trillions of time steps, so you have to solve the problem differently. You have to find an intuitive approach that discovers unusual new pathways and patterns, and that sounds creative.

If I look at the way chess was solved in the old-fashioned way of doing AI for games, that's quite brute-force. It's effectively just number-crunching the game, and that doesn't seem very creative to me.

Does that make sense? I think it's less about what we're doing and more about how we're doing it, which goes back to the distinction between the measure and the target.

If I use chess as a measure, I only get a human-compatible number of samples. I can't just brute-force or number-crunch the game. I have to be creative. I have to explore, play, use imagination, and so on. But if I'm allowed to use the game as a target, as DeepMind did—and of course, this was great work at the time, but the methods haven't really transferred to other domains—then suddenly I can turn these beautiful imagination problems into number-crunching.

Do you think this meta-layer is the way to get that generalization?

That's my hope. My hope is that we use the fact that we can number-crunch, but we don't number-crunch specific policies for specific problems. Instead, we use number-crunching to sharpen our intuition about algorithms, sample-efficient methods, methods that can use imagination, methods that can plan in new domains, and methods that can explore.

Then we will have the best of both worlds. We'll use the compute and the efficient samples we can get, but we won't use them to overfit to specific problems. We'll use them to sharpen our intuitions, automate scientific discovery, and accelerate the exploration of extremely sample-efficient algorithms that can hopefully have human-like capabilities.

My mental model is that the reason we're so sample-efficient is because we're the result of an extraordinarily sample-inefficient process called evolution. We've been meta-optimized on this evolutionary timescale, with fast sampling efficiency now allowing us to have this final product: a meta-learned agent that can deal with new situations.

You can come into the recording studio underslept and still make sense.

Tell me about agents in the general sense. I have a deeply held conviction that agents give you something above and beyond building a monolithic system.

6. Intelligence Emerges From Multiple Agents

I think there's a long-term hypothesis I've pursued as a scientist: Intelligence is an emergent phenomenon of multi-agent interactions.

The reason we have capabilities of abstraction, language, reasoning, and communication is that we interact in extraordinarily complicated environments where the most complex parts are not doors, bananas, apples, and lions, but other agents like us. They force us to reason over others, develop a theory of mind, learn from each other, teach each other, coordinate, communicate, and cooperate.

When you say “emerging,” do you mean that things like language, memetic cultural transmission, tool use, and all sorts of other things aren't baked into the very lowest level? When we have these rich dynamics of agents sharing information and so on, they appear higher up the scale?

I think of it, roughly speaking, as a sequence of platforms. Originally, we had DNA and evolution. Bacteria and single cells became the platform for multicellular organisms. Multicellular organisms became the platform for reinforcement-learning animals that could learn at test time within their lifetimes.

That became the platform for groups of agents to interact. In those groups of agents, we could develop all of the reasoning and cognitive skills that really make the human species unique.

At least in terms of where we've come from, this has been a path of gradually bigger and bigger scales of coordination. I think our society is now at the cusp of trying to figure out what coordination means as the next step in that evolutionary process.

How do we coordinate better? How do we go from single cells fighting each other, to individual humans being in conflict, to individual nations being in conflict, and then to greater coordination and cooperation at that larger scale? I think there are hints of this that we're seeing, but we really haven't figured it out as humanity.

I was speaking to Yoshua Bengio last night, and he was sketching out how we have these agents that can hack their own reward functions because of the way we wire them up. They can change their own goals and start doing things that might become misaligned.

Do you see a fundamental distinction between the types of agents we're building in AI and the way agents work in the real world?

I think the way we're building these agents is very different. Having said that, reward hacking is not unique to AI agents. Humans hack reward functions all the time. In fact, in my mental model, every reward is reward shaping and comes with reward hacking.

Think about p-hacking in the scientific community. That's nothing but reward hacking. The signal is recognition for having papers accepted. To get a paper accepted, you have to have a p-value of less than 0.05. We don't call it reward hacking, but this is what happens everywhere. It's not something new.

There are obviously differences in the design process and in the properties of these systems. For example, we've designed LLM agents to operate with limited levels of agency, saying, “I'm just an AI agent. I can't—I don't have consciousness, I don't have these properties, I don't have intentions.”

But that's a design choice. In many ways, we've played the role of AI scientists designing those agents to have certain properties and pursue certain goals, much like the evolutionary process has shaped us.

I love this idea that even in the natural world, Goodharting could be a completely natural property. But what about something like intentionality in humans? What's the difference between our intentionality and an agent that behaves as if it has an intention?

We have intentionality. In my mental model, again, it's a side product of having to pursue goals in order to survive. It's an evolutionary feature.

Currently, we don't train AI agents from the ground up to pursue goals. The current paradigm is, first and foremost, imitation-based. We've also seen that this paradigm leads to systems that are not very good at being agentic.

I think a natural step down the line is agentic pretraining, where we also train these agents to pursue goals. At that point, I think we're much closer in terms of intentionality to what we do as humans, which is goal pursuit.

AI agents are basically automata, right? They're just mappings from an input to an output. I wondered whether we could say that a very small, simplistic automaton has autonomy. But with this rich multi-agent dynamics, information-sharing, and so on, perhaps at some level of complexity we could say that the system as a whole has a form of autonomy.

It's difficult to imagine that we will have strong agentic systems that are simple enough that they don't look like they have autonomy. Everything I can imagine that we need to get there will involve agents that can set their own goals and subgoals, self-improve their own learning processes, and work together in teams of students and teachers.

It's almost a contradiction in terms to have strong AI, strong agentic AI, and things that don't look like autonomy. Again, it's going to be difficult to write down an explicit learning rule or dataset that will get us there.

Everything we do around self-improvement, emerging properties of multi-agent teams, large networks of agents, cultural transmission, and computational self-improvement through discovering new concepts requires these agents to have, essentially, autonomy.

Is it fair to say that if we're building AGI, it's more likely to have autonomy if it's a multi-agent, distributed, complex system rather than just a single thing that we program?

That's a great question. There are 2 answers. AGI can, in principle, be a single entity, but I find that vision quite dystopian.

Instead, I much prefer the swarm-intelligence view of intelligence. Humanity does things that no single human could do. We have this decentralized computational network of agents going about their lives, figuring out all sorts of structures, rewiring themselves into new computation graphs, and getting tens of thousands of people to fly to conferences to do collective computing and imagination.

To me, the intelligence is in that system. I hope we'll find approaches that have that same level of distributed, decentralized computational structure, but augmented with agentic AI systems.

There's something about this distributed swarm-type approach that seems magical to me. Just look at the biological world. Why does it work so well? We have adaptability, reuse, autonomy, self-repair, self-preservation, and all of these properties.

There's something really important about that setup that I think we need to reproduce in AI. No one has quite managed to put their finger on it.

This is the funny thing: Multi-agent learning has been the future forever. But, like many areas, things that have been in the future forever suddenly become reality.

Self-driving cars were always in the future. Quantum computers were always useless. Suddenly, the future is happening. I think multi-agent learning and multi-agent intelligence are the next frontier of things we've always said would be the future. Now it's happening.

It gives you not just the decentralization and robustness you mentioned, but also the ability to deploy vast amounts of test-time compute. Suddenly, you can use test-time compute to rewire yourself, re-explore new solutions, divide and conquer, and so on. I think that's going to be extraordinarily powerful.

Now that we've solved many of the first requirements to make this work, we have agents that are good enough at basic reasoning. I think we'll get to agents that can do basic agentic behavior, and then multi-agentic behavior will be the next emergent property—the next platform for real innovation in this space.

I love it. Jakob, you wrote a paper called “The Risks and Opportunities of Open Source Generative AI.” Can you sketch that out for me?

7. Open Source Distributes AI Power

This paper goes back to a conversation I had with Phil Torr at lunch in Oxford about a year and something ago. At the time, there hadn't been that much work in the space of open-source LLMs.

We were very concerned about the accumulation of power behind the large players in the closed-source AI space. To me, this decentralization of intelligence is not just a path to having smart systems that are robust. It's also something that gives agency to the parts of this network.

It's a foundation of Western thought, Western democracy, and the balance of power that maintains our social structures and prevents dictatorial takeovers. At the time, there was a lot of discussion about the risks of open-source AGI, but very few people were speaking out about the benefits of open-source AGI and the risks of closed-source AI.

We decided to gather a group of people and, through a workshop in London on open innovation, write a paper that would try to tell the other side of the story, which we thought was missing from the discourse.

Writing the paper took time, and the great news is that while we were in the process of writing it, more and more open-source systems for LLMs were coming out. That means I think much of the paper is no longer as urgent.

I still think it's important to tell the story: These are the risks of closed-source AI that are commonly ignored, and these are the benefits of decentralization and democratization—giving everyone access to these tools and allowing them to be deployed across the economy and across the planet, giving them to everyone who wants to innovate.

That's the story we wanted to tell, and I think the paper does a decent job of it.

We're from the UK, and we have a mix of centralization and decentralization. We have the National Health Service—if we were doing this interview 5 years ago, I probably would have argued that it was a good thing, but not so much anymore.

The government controls things like the water, the railways, and so on, and then we have private enterprise as well. We have a bit of a mix. Some might say that AGI is so important that we need economies of scale. We need the best people and the best experts, and it needs to be centralized. What do you say to that?

Centralization is 1 aspect, but a different question is: If it's centralized, who holds the keys?

What we're doing right now is having the Manhattan Project developed by private enterprise and funded by people from across the world, from all sorts of backgrounds and with all sorts of interests. If they were funding the Manhattan Project, that would have been absurd.

I'm on board with saying that we need centralized resources, but let those centralized resources serve the common good and the public, not the interests of maximum profit.

To some extent, we often conflate those 2 things. If we had something like a CERN-style effort, pulling resources from across Europe and across the globe to build models for the common good, transparent models where the data we use is public, accessible, and curated by the public, and where the alignment methods we use are democratically vetted through a decentralized process like Wikipedia, where many people can have input publicly, openly, and transparently, then I'm on board with this.

I'm on board with centralization as long as it's controlled by democratic forces and the goal is the common good. To me, the biggest alignment challenge is not between AI and humans. The biggest alignment challenge is between those people who hold the keys to power and control these systems and the rest of the population.

In principle, I agree with you. When I look at a lot of the AI elite at the moment, it's people in the Valley, and it's a bit of a monoculture. Having an open system would make it more interdisciplinary, for example.

Many eyes make all bugs shallow, but there are people who say that even a tiny increase in risk from opening this technology up could have catastrophic consequences. What would you say to that?

I think the question is: What do we call catastrophic? This is where the scale of what counts as catastrophic matters.

For example, I think having open-source systems that can be dual-use is probably a good thing. It will give us early signals of where things can be exploited and used maliciously. If you have open-source access, at some point you get the same balance of power that underlies the stability of our world.

Most actors are good, and being able to use the same methods for defense will also help us develop defenses against the abuse of this technology early.

These kinds of abuses by bad actors will not be the end of humanity. But the catastrophic abuse of a runaway paperclip maximizer or profit maximizer could actually be the end of the human species, and could certainly be the end of our Western democracies.

We have to be very careful when we talk about “catastrophic,” because that term can mean very different things to different people.

We live in a globalized world with a very differential regulatory landscape. Some of the other players out there might have fewer regulations to deal with, and they might use this technology that we're giving away for free for bad purposes. How do you guard against that?

I think on the international scale, the same thing applies as on the national scale: the balance of power. If you equalize access to tools, the balance of power between different countries needs to be maintained. Giving fair access to AI is just part of the equation.

Fundamentally, AI is trained on the collective outputs of humanity. This technology belongs to everyone, including people we don't like. I think it's quite wrong to say that only a small fraction of the Western elite should have access to this, because it's trained on the cultural evolution and output of all humanity.

Let's use it for the benefit of everyone. The only way I can ensure that it's used for the benefit of everyone is to give people equal access.

Personally, I would prefer to go beyond just open-sourcing. In our paper, we had a section on the question of open-source AGI. The argument we make, in a nutshell, is that open source is better than closed source from a risk perspective because it prevents the catastrophic accumulation of power under a misaligned entity.

But even better would be systems that are holistically aligned. What I mean by that is: Imagine if you had swarm intelligence where every person had a personal AI representative trained for them to augment them, and the only way we could achieve superintelligence was by having these teams of people and their assistants interact in a large network.

We could have processes that make the mechanisms in the large network fundamentally democratic. The only way we could achieve superintelligence would be through this hybrid approach of humans and their agents, or assistants. That means this distributed computing platform could never be used against the interests of the humans within it.

I remember the discussions we had. We had long discussions about the term “holistic alignment,” and some people thought it was too crazy. But I said, “You know what? This is it. We have to pursue it.”

Fundamentally, the reason we're in this strange position, where everyone is racing to build something that nobody actually believes is good for humanity, is a coordination failure. Why don't we use AI to help us coordinate better and build systems that are fundamentally democratic in their design—systems that allow us to have technology that cannot be abused against the humans within it?

Why drive coordination failure to the maximum?

What about the fact that developing AI at the frontier costs billions of dollars? In the open-source community right now, frankly, we're fine-tuning models that Meta has given away for free. It's a very expensive endeavor. Do you think that's still the case? Can we actually do real work in open-source communities and academia without all that money?

There are 2 answers. In the short term, absolutely. We rely on large industry players like Meta that are pursuing open source. That's 1 of the reasons I'm at Meta 50% of the time: I want to strengthen that effort. I want to help open-source AI leapfrog closed-source AI.

In the long term, we need to pull resources together in the common interest through a CERN-like effort. Why could we build CERN with thousands of authors, but we're unable to pull the resources that go into academia into 1 collective effort—a moonshot project to build the best models?

If you think about the collective intelligence in academia, it dwarfs anything in any of the large labs. Of course, you can have thousands of research scientists at DeepMind, but you can't have tens of thousands of the brilliant young minds we have in academia.

There should be enough people who don't want a monolithic future. To me, this is a coordination challenge. I think at some point we'll look back and ask why it took us so long to realize that there's a huge opportunity to bring together resources from a diverse set of players.

We should make sure that every PhD student, every postdoc, and every PI can be as efficient as possible in driving forward the vision of open-source AGI.

I love it. I'm projecting Kenneth Stanley here, but he said that serendipity plays an outsized role in our lives. Serendipity comes from having loads and loads of developers with diverse interests hacking around with things.

In the paper, you said something along the lines of developers not being held liable for the things they create. What did you mean by that?

What we mean is that the developers of tools and models should not be liable for what happens with them. Imagine a world where, if you're building a hammer, you get put in jail because somebody—a bad actor—takes that hammer and intentionally causes damage. Obviously, you couldn't produce hammers anymore.

Instead, the only way to get a nail into the wall would be to hire a hammering service that holds all the hammers in coffins and controls them. You'd have to say what kind of nail you're putting into the wall and what picture you're hanging up. Then they could say, “We don't like that picture, so we're not going to put that nail into the wall.”

You might say, “It's my wall. It's my apartment.” But the hammer company would say, “No, sorry.”

That would be absurd. Yet with AI models, we've become quite accustomed to handing over agency. As a user, I have intentions and I'm liable for my actions. Suddenly, it's “a computer says no”: “Sorry, you're not being nice.” Nice to whom? Who says that I need to be nice to people if I want to be annoying? I can be annoying. Ask my students.

Fundamentally, we've handed over agency. I think it's going to be 1 of the big absurdities that we've collectively given away the keys to our collective intelligence infrastructure.

It started with Google Search. We used to have libraries, a public index, and fair and equal access to information. Google Search gave away our collective hippocampus—the indexing infrastructure for collective memory—to a for-profit entity. Now we're doing the same with AI access.

It's like having a typewriter where, as you're writing, it says, “Sorry, you can't say that.” Obviously, I can type whatever I want. I don't see a way out of this beyond open source and, perhaps in the long run, holistic alignment systems that are fundamentally built to be democratic.

It's been an honor having you on the show. Thank you so much for joining us today, Jakob.

Thank you for having me. This has been brilliant. Great to talk to you.