AI 能否自我改进？[Chris Lu, Robert Lange, Cong Lu]

At this point, maybe a good chunk of the internet is Evolution through Large Models generated as AI capabilities kind of increase, like we saw them being used in ideation. People from Stanford have been thinking about this kind of thing. LLMs can generate just as novel ideas as human researchers. In that sense, can we use AI scientists as just a data-generation tool? Can we get much deeper knowledge than just the tip of the iceberg that's published in a paper?

The scale of evolution that we have in nature is not necessarily something we can do with GPT-sized models, but we can always try to do evolution on a much smaller scale. I do think that entropy is really important for what we're doing, where we need these really new ideas and some creativity. An alternative approach is to look for alternative sources of entropy.

Chris, welcome to MLST.

Great to be here.

Can you introduce yourself?

Yeah, I'm Chris Lu, a PhD student at the University of Oxford with Jakob Foerster. I also do quite a bit of work with Sakana AI.

Also joining us today is Robert Tjarko Lange, and Robert and Chris have written a really cool paper that we're about to talk about. Robert, can you introduce yourself? Of course, this is, what, the second or third time that you've been on the show?

Thank you for having us.

It's great to have you back. It's awesome.

Yeah, I'm Rob. I'm a final-year PhD student at TU Berlin and a founding research scientist at Sakana AI. I had the pleasure of working with Chris for a long time. We started out getting to know each other at an internship, and then basically have been collaborating ever since.

Amazing. We've also got Cong Lu. Can you introduce yourself?

Yeah, so I'm Cong Lu. I'm a postdoc at the University of British Columbia, and I'm mostly interested in open-ended learning. I'm supervised by Jeff Clune. I'm a huge fan of Jeff. He's awesome.

Yeah, Jeff is amazing. We'd let him know that we'd love to get him on the show.

Oh, for sure. I think he'd love to come.

Amazing.

TufaLabs is a new AI research lab I'm starting in Zurich. [tufalabs.ai] It is funded from past ventures involving AI as well. We are the Swiss version of DeepSeek: a small group of people, very motivated, very hard-working, and we try to do some AI research, starting with LLM and o1-style models. What we're looking for now is a chief scientist and research engineers. You can check out positions at TufaLabs.

Chris and Rob, you guys wrote this paper, Discovering Preference Optimization Algorithms with and for Large Language Models. Can you give me the elevator pitch?

Yeah, so we write algorithms to train language models to follow preferences, to align language model behavior with human preferences. We've been handcrafting a lot of algorithms to see which ones seem to work best in order to be more sample-efficient, faster, or more efficient.

When it comes to getting these language models optimized, a lot of work in the community comes down to figuring out which algorithms are best. If you look at how we do this, a lot of it is just trial and error, with some intuition behind the math, and language models are quite good at this as well. They do have the same mathematical intuitions through their pretraining, and they're also quite good at writing code.

So the simple question is: Why don't we just have language models try to optimize the algorithms that we use to train language models? That's basically the whole paper.

How does it work? We've got things like RLHF, for example, that can shape the behavior of these language models. The algorithm is designed by hand by a bunch of experts in the field who have strong intuitions about this kind of thing. Are you suggesting that we could potentially automate that?

Yes, exactly. Basically, these experts in the field have a lot of intuition, but also just a lot of trial and error. Language models also have maybe not as good intuition, but reasonable enough intuition, and can do way more trial and error than any human. This allows us to search much broader spaces of algorithms in order to actually find the best one.

So, Rob, how does the approach work?

We've already been working on using evolutionary, black-box-optimization-style approaches for discovering algorithms. Back then, we usually used a neural network to, for example, parameterize a loss function, and then we optimized the parameters of this neural network using some type of meta-evolution, essentially.

The way this works is to sample candidate solutions, evaluate them on a problem, and then feed the end performance back into the optimizer. Here, we're taking a different approach. Namely, we're taking a language model to propose code snippets.

Think of your favorite PyTorch objective function. The language model basically writes the code. It doesn't only write the code, but it also gives a name to the function and gives a thought for how it came up with this idea. Then we use the written code in an MT-Bench-style evaluation after doing this preference optimization. We feed the result back into the context of the language model and basically see whether or not the language model can discover objective functions that improve upon things like DPO, KTO, and so on.

Interestingly, what we found during this process is that LLMs are really good at mixing different concepts. Personally, I've read probably a tiny subfield of machine learning papers, while an LLM has not only read machine learning, but it has also read physics and chemistry, and can basically mix and combine all these concepts in a complementary fashion.

In the paper, we saw that different concepts, like smoothing or regularization techniques, are basically combined throughout this evolutionary optimization process. In summary, we're thinking of the LLM as a type of very strong and intelligent mutation operator that can help us discover new algorithms.

This is so cool. I'm seeing quite a few approaches like this, certainly for the ARC challenge. Ryan Greenblatt famously used GPT-4 to generate loads of Python programs—20,000 or 30,000 of them—and then we see which ones are good.

Kevin Ellis's group—he's at Cornell now, but he was at MIT—has done something very similar to what you're talking about: generating loads of snippets, then remixing them, and then blowing the remixed programs up by two orders of magnitude or so. Why do LLMs work so well for this kind of thing?

They've just seen tons of code, and they have a good idea of what works and what doesn't. By leveraging this pretrained base of code knowledge, they're able to more efficiently explore the space.

Rob just talked about how we used to do this meta-evolution technique, where we'd randomly perturb some neural network weights that represent an objective function, and then sample the fitness of those functions and use that to update some meta-network. This was really inefficient. This is why we had to use JAX-style techniques in order to actually get things to run fast enough for this to work.

We had to sample millions of parameters and thousands of population members, things like that. Because all we can do is randomly perturb these neural network weights, you can imagine that if I just randomly perturb the neural network weights, I'm not going to get anything interesting most of the time.

What's really cool about LLMs is that they have much more structured exploration because of the human data that they've been trained on, so they explore more like humans. They do more intelligent exploration. In the paper that we wrote, you can see that they actually do very intelligent exploration, where they'll look at what worked and what didn't, and then use this to do the mixing that Rob was talking about.

It would be like, “Hey, let's try this.” “Oh, that didn't work. Let's try this other thing.” “Oh, this worked. Maybe because of this.” Then they would just keep building on the things that they previously discovered in order to try to arrive at a final solution.

MLST is sponsored by CentML [centml.ai], which is the compute platform specifically optimized for AI workloads. They support all of the latest open source language models out of the box, like Llama, for example. You can pay on Consumption essentially where you can have a model which is always working, or it can be freeze dried when you're not using it. All of the models that they deploy support the OpenAI API specification out of the box, which means it's just a one line change in your application to switch over to CentML and start saving money and make your application go faster.

Rob, to what extent are these things bounded by the training distribution? They've got all of the things that we've ever said, but we want creativity. We want paradigmatically new things. Can we get there?

I think this is a super interesting question. Usually, when I talk about this, I take analogies from the arts. Oftentimes, people think of Picasso as this genius who just woke up and had all the ideas and the style of Cubism in his mind, and then he started working.

But Picasso is a child of his generation. There were many surrealist artists during that time, like Salvador Dalí or Joan Miró, and basically Picasso found his style within this convex hull of different artists. Oftentimes, when we talk about LLMs and LLMs for discovery, we're speaking about interpolation versus extrapolation.

And I think the power of LLMs, even if they might not necessarily be able to do crazy extrapolation, is that they can still interpolate between all of the pretraining corpus. Right. So this was what I was talking about before: there are fields like chemistry, physics, and economics where many of the discovered concepts can potentially be brought into the field of machine learning.

Yeah, Chris, at the moment you're measuring success against downstream tasks, if you like, but would it be possible to change the architecture to have more abstract forms of optimization, like, for example, for fairness or bias, or anything else we might think is important?

Yes, that's definitely true. I think currently we are only optimizing for the ability to predict future preferences. We could try to push it toward favoring things like fairness or more open-ended ideas than just some single-number metric that we're looking at in the paper. We can try to break down the different qualities of the model across a spider plot, where you show, “This is how good it is at reasoning or math.”

You could also add things like fairness and try to do some type of multi-objective optimization. What might be even more interesting is trying to expand it from this single-number, well-defined metric to what we do in future work as scientists, where we just try to generate interesting papers. Right. And this is kind of a general problem that we have in generative AI. You look at things like Sora or DALL-E, or these image models or video models, and the question is, how do you actually judge these? Right. There are no great metrics in the field.

A lot of the stuff now comes down to taste or feel, and I think that's how things like the AI Scientist might also play out, or these future meta-optimization techniques where it will output a lot of things, but we need to go through them and pick what we like the most.

Yeah, Rob, I mean, how important are vibes when we measure these things? Do you think that the metrics are being gamed or saturated? Are they useful?

I think every ML researcher has their favorite set of vibe tests, right, to see firsthand the capabilities beyond the MMLU benchmarks. And I think this is super important because, in my experience, many of the models can overfit to the commonly used benchmarks. I guess this is helpful for getting attention on social media when you promote your newly trained model.

But to me, actually doing the vibe checks on problems that I care about is usually much more important. For example, MMLU scores are really, really great, but for the things that I want to use frontier models for, usually MMLU struggles. This is something we also saw in the AI Scientist.

Nonetheless, I think it's a really valuable contribution to the community in order to essentially assess the capabilities and also to work on fine-tuning methods to improve these things.

Very cool. Chris, the loss function that you guys discovered in DiscoPOP—you said that it exhibited non-convex properties. How much of that do you think was a necessary condition for the performance of the algorithm?

So what was actually really interesting was that it was non-convex. Almost every loss function is convex because you want a single optimal point. What's weird about this one was that there was a discontinuity. It's non-convex. There are local optima, and then there's a global optimum.

We hypothesized, and I believe we showed some evidence for the idea, that maybe this non-convexity is useful for noisy data, where you can perhaps capture some of the bad data in this local optimum and then optimize the rest in the global one. This is a hypothesis. We have some evidence in the paper for it, but it's really hard to rigorously prove this.

One really interesting thing is that a similar type of non-convexity appears in a prior work we did called Discovered Policy Optimization, where there are a lot of these features that, when we look at this loss function, we didn't quite understand. One of them was this non-convexity. So this theme seems to have appeared multiple times, and it would be really interesting to explore this in more depth.

So, Rob, when we start meta-learning loss functions or anything else for that matter—because if you think about it, you can take any part of the prediction architecture, stick a model in there, and learn it—maybe we lose interpretability. We lose the answer to why that's important.

I think Chris and I have both been doing work where we discover function approximators, versions of loss functions, and so on, or evolutionary optimizers. Oftentimes, depending on the parameterization of the discovered objective or system, you can try to reverse-engineer it afterward. You can try to capture the majority of the variance within an analytically expressible equation, for example. This usually helps in interpreting it afterward.

I think there are probably limits to discovering systems that are really, really capable while still interpretable. But you can even think of settings where you set up the system to remain interpretable. It's really a question of what the substrate is that you're trying to optimize. Some of them lend themselves to being more easily interpretable post hoc than others. So I would hope that the final thing that we might discover is still interpretable, but there's no guarantee of it. As long as you can show empirically that they are bounded in a certain sense, I think this is probably as much as you can hope for, oftentimes.

Yeah. One really interesting thing is that we spent a lot of effort trying to reverse-engineer a lot of the algorithms that evolution discovered when we did this meta-evolution technique with DiscoPOP. It was not as hard because it would explain what it was thinking. So we just used its explanation, and that was the basis for our analysis.

A lot of times, even in academia, people will propose algorithms that maybe they don't fully understand, and it's up to other community members to go and reverse-engineer why certain algorithms work better. I think it's something similar here, where Evolution through Large Models proposed this loss function. It may have some explanation for how it works, and our job is to perhaps confirm this or come up with extra reasons for why it might be a good loss function.

I also think, in the context of DiscoPOP, as with the previous meta-evolution projects, the substrate that we're getting out is code. It's human-readable. You don't have to fiddle with the neural network, and this can be helpful at the current point. Who knows? Maybe in the future there are going to be objective functions with thousands of lines of code, which is going to make them harder to interpret. But you can still try to read them, right, because code is a medium that we humans are capable of working with.

Can we focus on that just for a little bit, though? What does the code look like? Do you see weird examples where it's just unbelievably complicated, or does the language model already intuitively know to give a reasonable, understandable answer?

Usually, the code is only a handful of lines, maybe 5 to 8 lines of code. So it's usually pretty interpretable, but it's also pretty creative. There are a lot of possible loss functions in this space, and it's usually some type of combination of these, with maybe some interesting extra loss functions that are found during its search.

For example, I believe in the ultimate DiscoPOP loss, there's this exponential loss that I don't think I've seen used very much before. I don't think it's very good on its own, but it happens to work really well when combined with other loss functions. Apparently.

Very cool. And how are you doing it? You could do beam search, you could put the temperature up, and so on. I mean, have you experimented with all of this kind of stuff?

We didn't do super-thorough explorations of this in the paper, but we might get better results if we actually try to force it to be more creative.

Yeah, really interesting. I mean, Rob, how do you feel in general about delegating creativity to a machine?

I think we as humans have a bias toward our own creativity, right? In the sense that, oftentimes, especially when we talk about the AI Scientist later on, people ask me, “Do you think this is just dumber than humans?”

But I think even if it's a little bit dumber, the throughput that we can get by automating these processes is tremendous. So even if the efficiency of such an automated discovery system is, let's say, 50% worse than what a human would do, we can scale it much more gracefully.

And I think with DiscoPOP already, but also with the AI Scientist, we're getting to a point where, basically, we can turn money and compute into really useful insights for the next generation of AI. We have the self-referential nature of it, which would be much more slowed down if the human were supposed to be the only creative engine, basically.

Yes, Chris, how do you see the role of humans? Do you think there's a middle way where we can have humans in the loop supervising the process? What would that look like?

Yeah, I think that would honestly be better if you wanted to produce better papers faster, in some sense. If there's a human in the loop, it's kind of like a supervisor for a project, right? So right now, I would say the models are maybe as good as an undergraduate student or a first-year PhD student, or a very young, starting researcher.

I think a supervisor is really helpful in this scenario because they can help prioritize which problems are important. They can help prioritize which strategies might work better and which might be worse. I think humans can play that role in this scenario, where maybe in the future, PhD students will be more like professors today, advising a big group of AI scientists or something like that.

I think this type of taste-making is really important when it comes to academia. After some point, putting out papers is more like an art than a science, in some sense, where you're trying to figure out what people would find useful or interesting, and there's no objective measure for that. In this type of setting, a lot of the problems become similar to those for image generation or video generation, right? You can generate tons of images, but who cares about most of them?

Somebody has to put in a prompt or something that they're interested in and get the thing out. I think AI scientists are hopefully something similar, where someone really cares about a certain topic, so they want to get as much research as they can on that topic. They'll be the ones making the taste and designing it. They'll be the ones introducing the problem parameters, specifying what type of outputs they want, and specifying everything for the AI scientist to explore, in the same way that a professor might do that for a lab.

Very cool. What about this idea of an infinite regress? Let's say you find an amazing loss function and give it to OpenAI, and you tell them to train GPT-4 again from scratch with the loss function. Then you meta-learn a new loss function, and so on and so forth over many generations. What would happen? Would it get better, or do you think it would mode-collapse in some way?

I think this is probably a broader question than the specific version that you're stating. Given that these systems are now everywhere, we're generating a lot of content on the internet with GPT and so on, and it's probably going to be used in future generations, it's hard to tell whether or not there's going to be mode collapse.

Even if there is, the current systems are already generating a lot of value for society, at least in my experience of working with them day to day. If this is going to happen in the next 5 years, once we reach that point, I think we're already really well off as a society in terms of using these systems later on.

With this specific case of discovering an objective function with an LLM and using it downstream, I think there's a separation, where the objective function is essentially only an implicit tool to shape the LLM, instead of baking into the LLM that it should give this objective function as an output in the next generation. Depending on what you discover and feed back into the loop, there are going to be clearer signs of this than with other things.

I mean, maybe your take on that, Chris, as well. I interviewed Ilia Shumailov from Google DeepMind. He wrote that Nature paper about model collapse over successive generations, and the elevator pitch is that after about the 4th generation, you lose all of the entropy in the distribution. For creativity, surely we need entropy, right? At some point, do we lose access to the source of entropy?

I think it's possible, but at least at the scales we're looking at right now, it seems as though we have enough entropy on the internet, where humans are still saying random things, being funny, or being unexpected. I do think that entropy is really important for what we're doing, because we need these really new ideas and some creativity.

I think one alternative approach is to look for alternative sources of entropy. For example, you might say, "Hey, I'll just randomly sample 2 fields and say, 'Come up with a cool idea about these 2 fields.'" That introduces external entropy into the system. That might be a way we can continue dealing with possible mode collapse.

It's really hard to tell where the models will go if you keep training on data that itself is output on the internet. At this point, maybe a good chunk of the internet is Evolution Through Large Models-generated. I'd love to add to that: we started this by thinking of LLMs as evolutionary operators, as mutation operators.

I think what Chris just described is basically taking this even 1 step further and thinking about crossover between concepts, fields, and so on. I think there's a lot of inspiration that can be taken from the evolutionary community to essentially improve meta-generation of outputs from LLMs.

This is a really young field, but there is work on combining LLMs with QD-style algorithms and so on. I think these are really promising when it comes to squeezing out the most creativity from these LLM systems. I usually don't like the comparison between neuroscience and the central nervous system in deep learning, but to me, evolution is one of these processes that has led to intelligence in many different forms. Mixing in some of these inspirations from natural biology into synthetic biology might be helpful as well.

Yeah. It's almost as if there's an abstract concept we're talking about here, which is this philosophy of combinations. We take a language model, train it, and it learns the statistical distribution. The first time we do it, there's a nice long tail, and we capture a lot of the complexity, but that erodes over time.

Language models have this property, though, that even after the 4th generation of model collapse, you can still, as you say, Chris, put diverse inputs in. Because you get the combination—the combinations of the representations—the output space is still much larger than what went in.

Then there's the meta level, as you're talking about, Rob, which is that we have all of this mixing at the cultural level as well, which creates another blowup of complexity. I suppose maybe you could make the argument that it doesn't matter to some extent that we lose so much of the complexity in the model, because it's the way it's used where much of the complexity comes from.

Yeah, I think the input distribution itself has enough entropy that we can keep going for a long time with the current paradigm. I don't know if model collapse will happen in practice, just because the models will be training on things humans say, ask, and do. I think this is just some irreducible entropy that will happen.

I haven't read that paper you mentioned in depth, but I'm not sure if they introduced any external entropy in their sampling system, whereas in practice, they do.

Okay, but they only take random samples from the distribution.

So it's all the same distribution, right? It samples exactly from it. But if you can then use the human distribution of inputs or human entropy, we might be able to keep going for much longer than the paper might predict.

Very cool. Have you guys thought about a curriculum-learning addition to the paper?

In what sense?

You're generating all of these code examples. What you could do, for example, is have a library and do retrieval-augmented generation, seed it with a bunch of things, and learn some optimal curriculum for how all of the knowledge gets combined together. Do you see what I mean? There might be some avenue of research there.

Yeah, for sure. In the end, you're right that we're presenting individual objective functions, but there is an archive of different possibilities that you could use. Something that's also really interesting, which goes back to some of the related work Chris and I have been doing, is looking at whether or not you can augment these objective functions with a temporal dependence.

You could tell it, basically, "Okay, you have a training horizon, which is this many gradient-descent steps," and then see whether or not the LLM can discover something that performs this automated curriculum, implicitly or explicitly, if you will. Another direction one could take is to use the archive of generated objectives and seek new runs of this LLM discovery with a new knowledge bank fed into context, or subsets of it.

Very cool. So, Rob, we're going to talk about your paper now, Large Language Models as Evolution Strategies, which you also refer to as the LM Black Box Optimization paper. Can you give us the elevator pitch?

I think this whole project started out with a different paper, written by the robotics group of Google DeepMind, called Large Language Models as General Pattern Machines. In that paper, they interestingly looked at this phenomenon of in-context learning on fairly abstract input sequences.

For example, they optimized a policy for CartPole control, where essentially the states were just represented as integer sequences. When I saw that LLMs could do policy improvement, I wondered whether LLMs were also capable of doing more general black-box optimization.

The starting point was basically to look at whether we can represent numerical black-box optimization problems in an abstract fashion that allows large language models to apply in-context learning and optimize these functions. What we show in the paper is that this is indeed possible, and that, depending on the combination of language-model prompting, setup, and in-context information you provide, you can see different performance results. This can, for now, scale to medium-sized problems.

As always, evolutionary optimization is not suited to optimizing transformer-like architectures. But in settings where there is no accessible gradient, you can apply this to, let’s say, up to 50-dimensional optimization problems.

Very cool. Can you sketch out the architecture?

Yeah. Basically, the way this works is that we use a set of prompting strategies, including least-to-most sorting, and provide the fitness and evaluations that were done on the function in context. By sorting the information into an improving sequence, the LLM can infer which steps were beneficial in previous evaluations and then continue this going forward.

Interestingly, we saw that this can outperform traditional algorithms used for black-box optimization. I think this comes back to the previous paper that we spoke about, DiscoPOP, where LLMs seem to have a very good inductive bias for intelligent exploration and exploitation.

I suppose another interesting thing here is that we’re talking about using language models. These are linguistic beasts, and you’re now getting them to give intuitive guidance for very abstract things like numbers and so on. Could we think of LLMs as somehow being a universal form of representation for many modalities?

Yes. I was really excited when I read the paper because I thought, “Okay, this is not just the stochastic parrot kind of paradigm.” In order to infer these improvement sequences, you need to do abstract reasoning, right? To me, I think that as long as you can represent things as strings with some structure, LLMs are capable of identifying patterns, given enough context.

For some settings, this can be useful. For others, there might be better-suited manual algorithms. But for black-box optimization and optimizing code in a closed loop, this is certainly a promising paradigm going forward.

You spoke about stochastic parrots. What’s your current philosophy on that?

Kind of in a similar vein to what we discussed before, I don’t actually think it matters that much. Given that they’re stochastic, we’re in the driver’s seat when it comes to choosing how to meta-generate new concepts, and they can interpolate between concepts using prompting techniques. We can probably generate a lot of new knowledge just by doing this.

Coming back to the arts analogy that I made before, I think interpolation at this point in a super-large, high-dimensional space is already quite a lot in terms of the value that we can bring to society.

Very cool, very cool. In your paper, you said that in certain circumstances, smaller models actually outperformed bigger models. Tell us about that.

In the paper, we compare 2 different language models, as you do. This was at the beginning of this year, or the end of last year. Back then, we were working with the Llama 2 suite of models, and we found that, when comparing them on a vast set of black-box optimization tasks, the smaller models tended to perform fairly well, besides GPT-4.

To us, this was perhaps very implicit evidence for the mixture-of-experts architecture and setting of GPT-4—potentially that individual experts might actually be much smaller than 400-plus billion parameters.

It’s interesting to reason about why smaller models might be better at doing in-context learning on abstract sequences. There might be something related to overtraining and undertraining of these systems, but at this point I can only speculate. It was an intriguing finding because this happened across the Llama and PaLM 2 models from Google.

So you discovered that fine-tuning on these teacher-algorithm trajectories made the models perform better, right? I suppose it’s not really possible to do it any other way. Would there be another way of doing that? If you actually had access to the entire model-training architecture, could you somehow imbue that knowledge somewhere else in the prediction architecture?

I think there is related work from Johannes von Oswald, who shows that transformer models can learn how to do gradient descent, basically. They explicitly, or somewhat explicitly, train for it. It’s not like taking a language model and deploying it and showing that it does gradient descent; it’s training a transformer, which then implicitly gives rise to gradient descent.

I think there are certainly ideas for how to train the training distribution or the training paradigm to make these systems more capable of doing abstract in-context learning. But I also think that this might take away some of the creativity. It’s always a question of how much you actually want to bake in naturally, versus how much you think or hope they are just going to exploit or discover during their own training process.

In the paper, as you said, we used optimization trajectories generated by a teacher algorithm and performed a little fine-tuning with them. This helped on certain tasks, but not on all. I think there is clearly a factor of us perhaps baking in too much inductive bias by just changing the training distribution.

There’s this matter of dealing with numbers, or discretization in general. Even tokenization could be a problem. Tell me about that.

For example, when you look at the Llama 2 tokenizer, certain numbers are more represented in the corpus used to generate the tokenizer than others. The numbers 1 to 50, for example, are represented with their own tokens. The same is true for 1950 to 2020, which makes sense because they appear more frequently in the training corpus.

But especially if you work with floating-point numbers, this can lead to artifacts, such as certain numbers being assigned more tokens than others. When your goal is essentially to squeeze as much in-context learning out of the LLM as possible, you want to have standardized sequences of tokens, right? This makes it easier for the system to infer patterns than if there is a flexible number of tokens.

As a designer of such a system, the question is how to set up the representation, or the abstract representation, to maximize in-context learning from the LLM. We found that certain integer discretizations worked really well instead of using floating-point numbers.

More recently, when I played around with this, some of the newer frontier models were actually capable of working directly with floating-point numbers. This might change as we make these systems more capable and robust. These insights might also become outdated later on.

There’s also the matter of language models learning all of this rich knowledge during pre-training, and how much of that is necessary. What’s the balance between their internalized knowledge and doing chain-of-thought, for example? Does that imply that we could, in principle, have smaller, dumber models but really clever prompting?

I think this is a super-interesting question, given that there’s now also a hype around small language models and the question of whether the big frontier models are going to rule the entire economy, or whether small players with small models can play a role.

When it comes to very specific tasks like black-box optimization, for example, smaller models, as shown in our paper, tend to perform well or even better. I think there’s a chance that we might end up in a world where smaller models can be more specialized, and we don’t need to spend as much compute on them.

On the other hand, you might at the same time cut off certain types of knowledge and things that you, as a designer of the LM, might think or deem useless. But the LM, in the end, might still be thankful for having them in its training data when doing small perturbations or mutations later on.

In general, at Sakana AI, we have this notion that learning always wins. Ideally, in an optimal world, we find a different way to target these larger systems so that they behave in a more specialized way. In The AI Scientist, but also in DiscoPOP, there’s some small amount of prompt engineering going on, which can oftentimes go a long way.

In your work, you’re talking about discretizing continuous parameters. What does that imply philosophically? Do you think that the discrete world is the best way to understand reality? Is that how we understand the world?

In our case, the discretization implies a certain resolution of the search space. In 2D, you basically have a grid, right? Each grid point is represented by a combination of numbers.

As you scale this up, the volume is increasing exponentially. There are inherent limitations to this discretization which make it hard to scale to high dimensions, for example. But if you think about how we train large language models, we're basically treating everything as bits, right?

I think there are limitations to it in the sense that you can't easily scale to high dimensions, and using floating-point numbers and potentially low-dimensional projections might be easier or better. But there are also advantages, in the sense that you constrain the search space in a certain way. So I think the jury is still out on whether discrete representations are better than continuous ones.

I think the whole deep learning revolution showed that continuous representations might have an advantage, at least if you're doing gradient descent. For other things, it might be different. If you do black-box optimization or evolutionary optimization, discrete structures might be easier to fiddle with or perturb.

What's your philosophy about the complexity of this kind of evolutionary meta-optimization in general? If you look at a lot of the hyperscalers, like OpenAI, at the moment, they're doing this insane engineering, building a globally distributed system. But in a way, it's simpler, right? They just have 1 model, and they're just doing stochastic gradient descent and so on. What would it look like for a hyperscaler to deploy the kinds of methods you're talking about?

I think actually at this point, there's also some type of evolutionary optimization going on for GPT-style training, in the sense that there is a community of PhD students who have discovered certain default hyperparameters and seeded the search space in a very specific way. So I think we can talk about evolutionary optimization on the per-project scale. But we can also talk about evolutionary optimization in the sense of the collective intelligence of people investing time and resources into these systems.

If OpenAI had to start from scratch and the Adam optimizer had not been published open source, they might not be where they are right now. So I think this is one aspect, or one answer, to that question. The other is, when you think about our DiscoPOP paper, you don't actually need large populations of people or candidate solutions in order to discover something new. Chris already said these proposals are using very intelligent exploration. We can even just use a single candidate solution, evaluate it, and then do this in a loop, basically updating the context and getting a new candidate.

I think the scale of evolution that we have in nature isn't necessarily something we can do with GPT-sized models, of course, because it's too expensive. But we can always try to do evolution on a much smaller scale and then see whether what we discovered on the smaller scale generalizes to the larger scale.

For example, in previous work that Chris and I did, we looked at meta-optimizing evolutionary algorithms themselves, sort of in a meta-meta loop. We found that you can do this on very small tasks and essentially optimize the algorithm on these low-dimensional tasks, and then transfer them to higher-dimensional tasks later on. So as long as you choose the meta-task distribution on which you find your signal of improvement in a smart way, you can go on and transfer this to more complex settings.

I think there's especially room for things like evolutionary optimization when it comes to optimizing things like data mixtures for large language models and thinking more about how you can set up data that incentivizes chain-of-thought reasoning, or reasoning more generally. You could think of settings where you train very small language models on a task distribution that is generated by some type of transformer or algorithm, and then you optimize that transformer to give the downstream-trained transformer, basically, in-context reasoning abilities. So especially in the context of synthetic data and the smaller evolution regime, there's a lot to discover that can potentially transfer to the larger scale.

I'm completely on board with that. What's really exciting about work like this is that we can work on the small scale, discover entirely new methods of doing things, and then transform them into a classical method, if you like, and productionize them at scale. But I'm really excited about biologically plausible intelligence. I want to have this kind of thing running at scale in the first place.

I want the Anthropic of this world to be building this living, breathing system that does several levels of meta-optimization. From a software engineering perspective, that seems like a nightmare to me, and I wonder whether it's a threshold we'll ever cross. I don't know.

It's hard to say at this point. With Moore's Law continuing to a certain degree, at least in the ASIC sector, I think there might be a chance that everyone can have their own personalized AI assistant. We're going to see these forms of collective intelligence, I don't know. I think I also prefer that future in the sense that it allows for much more customization and neat adaptation for the user, and seeing evolution not only run on the biological scale but also on the synthetic scale is something I'm very excited about.

Ultimately, as I already said before, natural evolution is the only process we know for sure has led to general intelligence in the form of us. So I think there's a lot to learn and a lot to be transferred, but also probably a lot to be discarded. You don't want to copy everything.

I think something that I also learned from Sakana AI CEO David Ha is that they had a paper called The Sensory Neuron as a Transformer, where they basically looked at how transformers can work, learn, or meta-learn generalized policies on pixel-mutated representations, so humans in many ways could not do such a task, while computers can. There's a whole set of tasks that we humans, with our inductive biases imprinted by evolution, cannot really do because it would take us a lot of time to adapt. We're already in, let's say, a local optimum of our cognitive systems, while machines can do these tasks. Potentially, there's a different type of evolution that has to happen for these machines to improve basically endlessly.

I can't believe we didn't talk about this before. You're working for David Ha. Yeah, David Ha, of course, was at Google Brain, right? And he created this startup called Sakana AI. It's the kind of AI that we love. Fans of the show will know that we love it. What's it like working for David? And just generally, what are you guys doing?

I can tell you, I've never been happier in my life. I'm not faking anything. It's really great to be in a place where creativity and unorthodox ideas are being promoted. There's also a lot of freedom to execute them.

Sakana AI is a Japan-based startup co-founded by David Ha and Llion Jones, who was a co-author on the Transformer paper. It's been quite the adventure, not only learning on the technical level but also on the business side of this company in this crazy era.

Something that, at least for me, separates Sakana AI from places like Google is that we're trying to do things other people aren't doing. We're not in the language-model pretraining game, but we're essentially trying to go down the path of ideas other people are not necessarily super willing to commit a lot of resources to, and I think we're thereby filling in a big gap in the community right now.

I'm really excited about this work, though. Why are other people not doing this?

I think we as a community are probably, to a certain degree, stuck in a meta-scientific local optimum, and the linear path forward is just to try to do scaling and take VC money and burn it by pretraining the SOTA model for 2 weeks and then having a new SOTA model. It's hard. It's also not easy because you need to be frustration-tolerant in some sense.

If you think of it, pushing scaling laws is in some sense the easy thing to do. It's a very linear thing, right? It's hard on the engineering level, but conceptually it's straightforward, I would say. So, yeah, maybe that's the reason, but I can only speculate.

I think David has had a tremendous effect on my PhD in terms of inspiration and bringing ideas from Jürgen Schmidhuber partially into life, as well as doing a lot of exciting stuff himself. To me, he's a good person to guide such an effort. Llion has been and is a co-inventor of the Transformer, right? So there's a lot of technical expertise at the company and a lot of smart, outside-the-box people working there.

Yes, I was about to mention Schmidhuber. He worked on things like Gödel machines and artificial curiosity. He was just thinking about stuff like this a long time ago. And of course, David's done a bunch of work with Jürgen as well.

I suppose maybe the dividing line is that I believe, and I think you believe, that we should look to the natural world for inspiration. Collective intelligence is a big one, and it's kind of like biomimetic intelligence.

And the other school of thought is that intelligence is this kind of pure, simplistic set of principles that can be scaled, and scale gives you everything. That seems to demarcate the 2 views. We can talk about intelligence on the individual scale, and we can talk about intelligence on the collective scale.

On Tuesday, Alison Gopnik gave this talk about child development and how societal structures really give room for causal experimentation and learning about the world. This would not be possible without other agents available. And I think if we really want to get systems that are capable of adapting on a lifelong horizon, you need some social structure that provides safety but also information asymmetry.

A baby can't learn about everything in the world at the same time, but it needs the parents to essentially safeguard it from certain things in the world. If you think about it, I thought the world was a completely beautiful and blissful place until I turned probably 10 or something. I couldn't have imagined that criminality exists, and these safeguards also enable a very focused style of learning, which is probably only possible if other agents are in the loop.

So I think collective intelligence is something fascinating, and complex systems as well. I would think that there are some parts of intelligence that can only be unlocked by going in that direction. Anyway, Cong, you wrote this paper, “Automated Design of Agentic Systems.” Can you give us the elevator pitch?

Yeah, for sure. This is a fun piece of work led by Xing Ren in Jeff Clune's lab, together with Jeff Clune. This kind of work fits very neatly into the general Cambrian explosion of Evolution through Large Models–driven discovery, so it's super related as well to works like DiscoPOP, Eureka, Evolution through Large Models, and Promptbreeder from Fernando et al.

The gist here is: how far can we take this current paradigm of Evolution through Large Models–driven discovery? The use case that we wanted to investigate in this paper is that LLM agents are ubiquitous everywhere now. Things like The AI Scientist, which we're going to discuss later, are LLM agent systems that we handcrafted painfully over months. Things like Cursor and literature survey tools are all LLM agent systems, and crucially, all of them are pieces of Python code.

LLMs are now at this point where it's become quite meta. The fun way I like to describe this is that LLM agents are now writing LLM agents, together with LLMs writing preference loss functions. It's getting super meta. With the increased capability of things like Claude 3.5 Sonnet and GPT-4o, these systems can now discover pieces of code that are hundreds of lines long.

This involves all agentic systems. We have traditional structures like chain-of-thought, debate, and RAG-type systems for answering. The question we want to answer is: can we just design all of this from scratch and evolve these systems with respect to something like, for example, an ML performance benchmark? In the limit, we'd like more abstract things, maybe human preferences as well. But can we just design all these systems from the ground up?

We've got some pretty promising results, including, for example, on the ARC challenge. We show that the kind of agents that pop out of this are super nonintuitive. We have some intuition about how we design Evolution through Large Models agent systems. We always think, “Let's think step by step” might be a good one, but there's weird stuff as well, like, “I'm going to be really angry if you don't respond well.”

You can also ask whether LLMs can search this entire space. That's a Turing-complete space of all prompts and workflows. Can they invent new tools and new ways to use code that perhaps humans haven't come up with?

All the evolutionary-type work that we discussed previously has aspects of recombination. I think one thing that hasn't been covered so far is this aspect of serendipity. By stochastic sampling—say we sample 1,000 times—can we randomly get something surprising, really good, that we can archive? Then, in later iterations of this evolutionary loop, can we build on that as a stepping stone toward future developments?

Yeah, I'm a huge fan of stepping-stone collection. I love that. The first thing is that you're using code as the primitive. Certainly, solutions to the ARC challenge seem to be split down the middle. Some of them are inductive code-generation approaches, and some of them are just transductive approaches where you just do the thing; you don't have an intermediate explicit function.

You said something else that was fascinating: searching Turing space. These models are not Turing complete, right? But they can generate code, which is. So, in some sense, can they search Turing space? I don't know—what do you think?

Yeah, for sure. Going toward the transductive stuff, you could imagine that in code, our agent might not even be LLM calls. It might load some diffusion models off Hugging Face, try a little bit there, and get LLMs to parse the output of the diffusion model or whatever.

In the limit, we've got this Turing-complete space. We talk a little bit about this in the paper, but what if the code writes code to train another LLM? Obviously, this is super unrealistic now for current compute budgets, but theoretically, you could train another LLM to train another model, come up with new things to solve the problem—far beyond just pure LLM calls.

I'm a bit of a neuro-symbolic guy. Code has compositionality. It's Turing complete, and it can do lots of things that neural representations can't do. But we're increasingly seeing that neural networks can do a lot. They're limited in ways that we understand: they can't copy, and they can't count. They're stupid in lots of ways. Maybe we'll figure out ways to improve that.

But lots of people who did have the intuition that we need to generate code are now just saying, “Let's skip that, guys. Let's just do the thing.” I think for stuff like tool use—for example, integrating web search into these agent reasoning workflows or integrating things like calculators—if some kind of agent computation requires tons of computation, multiplying loads of numbers, or doing lots of complex math, possibly when we get to superintelligence levels of neural networks, we'll be able to do that.

But I think these things are obviously far more suitable for code, and integrating these kinds of things, possibly with neural-network-based solutions, is likely going to be the future.

Yeah, I think that's a fair shout, actually. Another thing I was thinking of is that symbolic code is more compositional if you want to do multistep reasoning. If you think about it, if you just do a whole bunch of transduction operations, that might not gel together quite so well. But in a multi-agent system with tool use and so on, what we want to have is lots of reusable building blocks that are intelligible and can be chained together. I suppose that does seem more amenable to code.

Yeah, for sure. One of our inspirations was things like LangChain, for example: lots of reusable building blocks and things that can be combined together. In many of the discovered agents, we found there are actually 2 agents sandwiched together. Maybe you've got 1 agent that makes an initial prediction and another one that refines it. These 2 blocks are very modular and are just previous agents from the archive.

How do you balance exploration and exploitation?

For exploration, basically, it's just tons of sampling and mutation. We explicitly encourage people to think out of the box, use knowledge from adjacent fields, and try to integrate these kinds of concepts into their next design.

I think there's a lot of work as well that's quite related to what Chris was saying about changing the context. You can inject a lot of entropy. For example, we've got new stepping-stone accumulation algorithms, also from Jeff's lab, like OMNI, that basically have an unlimited-size archive. You just keep sampling new things, hoping for new combinations, and we find that this strategy scales almost infinitely.

We've run systems that can discover novel artifacts for maybe over 5,000 generations, judged by a language model. Even by generation 5,000, some of them are still noticeably different from previous ones. I think injecting that kind of entropy into the context helps a lot in terms of balancing exploitation.

There's this worry: what if we're overfitting to benchmark scores, for example? Interestingly, we find that even with these humongous agent systems that we discover, they transfer exceptionally well. In our “Automated Design of Agentic Systems” paper, we try to make this search as efficient as possible.

We've got a high-level agent that's the strongest coding agent that we can find. In the inner loop, we've got something like GPT-3, which is exceptionally cheap and super fast to run all the evaluations on. We find that, for example, if we optimize an agentic system for GPT-3 on some math problems, it generates this kind of very generic, super-robust reasoning loop that then transfers to GPT-4, transfers to other math domains, and, shockingly, transfers to literature tasks.

Things like chain-of-thought debate and RAG-based tool use are super generic and applicable to all kinds of tasks, and we find that our agents similarly transfer quite well. We've got this kind of extreme exploration that we think we can scale super far. I guess at some point, I do believe that you get diminishing returns from trying to scale this and, at some point, you're trying to squeeze blood out of a stone and get capabilities out of the foundation model that probably aren't there yet. But as far as we've seen already, you can go super far and discover super-general reasoning structures.

I mentioned debate, which is a great little digression. I interviewed Akbir Khan. His debate paper was paper of the year, I think, at the end of this year. Amazing paper. That's a really good meta-approach for agents, isn't it? Maybe explain what that is. But I mean, are there other similar things like that?

Debate in general is that, instead of just asking an LLM a question and receiving a response, you might, for example, try to do a multi-turn kind of thing. You basically have agents set up, perhaps in an adversarial sense. One person proposes an answer, another person criticizes, and this loops, with each person criticizing, to see if you can reach a consensus toward the end. We see these kinds of structures actually being discovered quite a lot in our agent optimization.

Another related one that I really like, and that our optimization loop discovers a lot, is specialized experts. A lot of the time in science, we get a lot out of interdisciplinary collaboration—people with different kinds of views. Our agent discovers this: you prompt agents, in a kind of debate sense, to specifically criticize efficiency, specifically criticize readability, and specifically criticize the accuracy of the solution. This is targeted debate rather than just generic criticism.

It's honestly even hard to describe some of the systems that our system generates. Some of them have 20 calls all chained together in the middle. There's an efficiency expert, a readability expert, and an accuracy expert, and then after that there's debate as well. I look at these designs and think, how on earth would I even begin to come up with these structures? One of the main hopes is that, because we're operating in this kind of Turing-complete space, with this kind of framework we might be able to discover new debates and discover that new kind of framework that's going to lead to, for example, next year's best paper award.

I'm a huge fan of building agent systems. I use the actor pattern, so I'm always doing stuff like this from my intuition. An actor has a manager, and I quite often have critics, and I have an OpenAI actor, and I have Anthropic actors and Google actors, and so on. Even though I'm only ever working with one actor at a time, I'm building this distributed system of information flow.

There are sometimes problems, right? Because if you think about it, it's this living, breathing system. Sometimes you can get these loops, or you get failure modes, and you get weird things. I suppose, over time, you build intuition about topologies, about design patterns, about ways of doing this. You're saying that your tool can generate these patterns automatically. But I'm just thinking, because these things are code, they could run forever, but you could get these weird, degenerate behaviors. Do you validate for that, or what happens?

One easy solution against the sort of degeneracy issue is perhaps setting a time limit. This is also one of the things that we want to add into systems like Automated Design of Agentic Systems. Not only do you return an answer that maximizes benchmark scores, but also return it with a set cost, maybe a set runtime as well. I really like the point as well: we're really discovering how to organize computation in an efficient way here.

It's kind of like if you imagine all these agents as workers in a company. Companies evolve structures that best make use of their available agents, and we see hints of that kind of thing as well. In the limit, I think things like Automated Design of Agentic Systems could basically try and work out how to use, say, 1,000 different actors, all prompted to do different things and intelligently combine their outputs in the right way. Exactly like, I guess, how you would organize a company, for example.

We seem to have happened upon certain topologies that work because we have this knowledge-transfer bottleneck, right? It's really difficult for us to transfer information efficiently. Most companies are quite hierarchical, but I just wonder whether that's a natural thing.

If you think about it, AI agents can transfer information at really high bandwidth, right? Maybe we can have our cake and eat it. In the AI world, we can have every topology running at once and somehow sharing information. Do you think it would resemble the real world, or would it go in a completely different direction?

Absolutely. I think this is the importance of this kind of evolutionary loop. We don't exactly know if we just import our human intuition about how we work best into structuring computation. I don't think it's going to transfer well. For example, humans are super limited by working memory. I think we have a working memory of 6 or something. You can see this in the structure of our mathematical theorems and in the structure of reporting structures.

But AI agents, I really don't think, are limited in this way. For example, Gemini these days has a context length of millions of tokens. Feasibly, this could be like a human manager that has 1,000 reports, integrating outputs from thousands of things. You can imagine that, if a human could do this, you would collapse the hierarchy of a company by, I don't know, many, many levels. This would probably be better for everyone, and I think AI agents stand to be able to use these kinds of structures much better.

I think we do need to evolve. We need to basically just rerun that kind of cultural discovery loop that we had in human society to make all these company structures, write all these blog posts about how to make your startup or whatever, and let AI agents discover this for themselves.

How can we bring Kenneth Stanley's ideas in here? One thing that worries me is that if the bandwidth connection is very high between agents, it will lead toward building more monolithic systems. If Stanley were here right now, he would say we need to have agents that are searching for novelty, so they need to follow their own gradient of interest for many, many steps. I suppose, is that something we should code in, or are you seeing that kind of thing coming up?

We have elements of that. We do say, “Think outside the box.” But ultimately, we are guided by benchmark scores. One of our dreams, for example, is to go down this sort of full open-endedness: abandon objectives, follow your nose for interestingness.

One thought that we have is that we've got this system now for optimizing agents for one particular task. Can we have another system that proposes challenging tasks? So, really, not hard-focusing on one objective, but trying to do this kind of goal switching that Ken Stanley talks about a lot. Can we have, say, a proposer agent that designs more and more challenging tasks? Maybe that agent also evolves. Can we evolve systems to meet these new challenges?

Can we run this in a loop with harder and harder challenges? Can we achieve the full sort of dream of open-endedness? Can we co-evolve these kinds of things together? I think that kind of approach is going to be very fruitful in the future when we get the compute to make that happen.

Yeah. I'm so torn on this. I love reading software engineering books about design patterns. I use them in my code, like the mediator pattern or the observer pattern. Maybe this is just wrong. Maybe these models are just smarter than us and can come up with better topologies. One thing I thought of as well is that the meta-agent itself is an LLM agent. So could you not create a meta-meta-agent?

Great question. Maybe I'll answer that question in a few months when we get that result.

Interesting. But yes, totally. This actually really relates to stuff that Chris and Rob were saying earlier. They were doing meta-meta-optimization and asking precisely whether you can think of—I think there’s a piece of work recently called STOP. I believe that got an oral at COLM.

The idea was that they’ve got a really simple task for which you design an LLM agent. You search for better programs for this task, so there’s 1 level there. Then you have another agent that tries to improve the search process. That’s really going toward the kind of Schmidhuber Gödel machine-esque ideas: you improve the inner task, you improve the outer loop, and go on forever. Recursive self-improvement.

I tend to think that at some point we’re going to be squeezing blood out of a stone. I don’t think this is the full recursive self-improvement loop that’s going to go AGI. We’re very much bounded by mortal capabilities. But could you have this meta-meta-optimization go as far as this model can take us, then do something like gradient updates to optimize for the best thing that we found, and keep continuing that loop over and over?

It does get to the core of intelligence, though. I’m a big believer that intelligence is quite situated and specialized, and certainly, with the experience of generating these agent systems, you see topological bottlenecks and locales of specialization and so on. So, what would it mean to you to have AGI in this kind of setup?

I think my vision is quite similar to what I proposed earlier, just sort of like human culture and society. We continuously find new and interesting challenges for ourselves and work toward those. I think this kind of goal switching, having no predefined objective, is really critical for this.

I want to see whether we can have some kind of system that can continuously propose and accumulate a tech tree or a skill tree of things that it can do, and then continuously build this out, hopefully forever, just as human society and culture have. Perhaps it’s not just that—it’s really hard to tell what this kind of thing would find. It’s way beyond benchmark optimization.

It’s just like human culture. It’s really hard to predict what the next wave of innovation will look like, and what will build off that, and what will build off that. Looking 10 years into the future, it kind of looks like magic and sorcery—completely unimaginable. So I couldn’t predict what that will end up looking like, but I think this is the kind of system that we need to realize full AGI capabilities.

Let’s talk about your other paper that we’re going to cover today, Intelligent Go-Explore. Can you explain the paper?

Intelligent Go-Explore was also a collaboration with Sakana AI and Jeff Clune—some wonderful authors to work with. I guess we start off with this Go-Explore algorithm. Often in reinforcement learning, the problem is exploration: how do we find good paths through the environment that get us into good states?

Go-Explore was super influential work from 2017, I believe. The idea is actually very similar to these evolutionary algorithms: you have an archive of discovered interesting states. You keep selecting states to explore from, for example, by taking random actions. You then put the states that you find promising back into the archive and loop until you find something good. Then you can robustify those trajectories afterward.

One of the key snags of Go-Explore is that you have to have a really good, interestingness function in your environment. For example, Montezuma’s Revenge used to be one of the grand challenges in reinforcement learning, and Go-Explore was one of the first algorithms to solve it. You had to have a beacon: going down levels was good, finding keys was good, and having more agency in the environment was good.

With the advent of modern foundation models, we have this insane opportunity. A lot of these games conform to human intuitions, so we have this nose for interestingness. We go into a video game and know that we want to go forward, get new things, and collect interesting things that might be useful to us later. Can we use that as the interestingness engine in place of hand-designed heuristics?

We set out to do that, basically, inspired by works like OMNI, which also tried to use a language model’s nose for interestingness and task selection. This is using its nose for interestingness to discover new states in an environment. We get really nice results on a variety of hard-exploration reinforcement learning environments just by following a foundation model’s nose for interestingness.

I think foundation models for complex exploration environments are going to be quite a big challenge in the future. You can imagine, for example, reframing even things like scientific discovery as exploration through a very large search space, where you need to archive stepping stones. All of this is related to LLM discovery works like AI Scientist and R1. I guess Intelligent Go-Explore is the very specific application of that principle to hard-exploration reinforcement learning environments.

Yeah, it just blows my mind that LLMs are so good at creativity. Subbarao Kambhampati is coming in a little while, and when I spoke with him last time, I was saying to him, “I think there’s a creativity gap in LLMs,” and indeed a reasoning gap. The zeitgeist is shifting.

I’ve spoken to so many people at NeurIPS who are saying, “No, actually, they do creativity and they do reasoning to some extent.” How do you think this works? How can it generalize the representations it has to something like Montezuma’s Revenge?

For sure. Montezuma’s Revenge isn’t something we’ve tried before, but we really want to. LLMs have so much human prior knowledge about what’s important in games. In something like Montezuma’s, the key is to explore more, gather useful objects, and descend down the levels, to my recollection.

I think these priors for exploration are really baked into the model. We know these things are good. Humans have a nose for this kind of thing, and I think foundation models have acquired a lot of this because these games are very well represented—or discussions of these games are very well represented—in the training data of LLMs.

Where we stand to do quite well is then another question. For example, say you took current LLMs and put them in a field of science that’s going to emerge in 10 years, where no one even knows the terms anymore. I think that’s going to be a much different question. Then you’ve got to retrain and transfer your intuitions to this new setting.

Well, one thing I’m going to ask you guys about the AI Scientist paper in a little while is this: if you traveled back in time to Newton or whatever and gave him a 21st-century physics book, he wouldn’t know what to do with it. He wouldn’t understand it, because it’s using all these terms and so on.

But it comes down to abstraction. There are fundamental principles that explain how the universe works. Even if you look at some of the ARC solutions, the art of it is describing in language what the problem is. Maybe there’s an element of analogical reasoning in the prompt.

But even to do protein folding and future scientific discovery, surely there’s some kind of map that would describe it in a way that a language model can understand?

Absolutely. I think it’s all about adapting the right abstractions for the task at hand. Even for future science, I think these generic reasoning structures—things like debate, for example—transfer well.

Throughout the ages, as long as you can make the right abstractions for the current problem at hand, I think we can apply a lot of these structures that we’ve already learned.

At the moment, it’s just text, right? What about other modalities?

Our initial version was just text, but now we’ve also got image-based environments. We show that our algorithm can operate in, for example, visual grid worlds. In many ways, I think it might actually be easier.

For a human playing the game, some of these text representations for grid worlds are super complicated. You’re down here in the center of the map; you see a door 1 block to the east and a key 2 steps to the west. That’s actually really hard even for humans to reason about.

But you have this image, for example, of the board. You can very clearly say, “There’s something on the left.” You just move to the left. There’s an interesting thing that you haven’t seen before in front of you. So I think there’s still a lot to be done for VLM reasoning in video games, but there’s some really nice recent work trying to benchmark LLMs on these, and hopefully we’ll see future work trying to do RL on these models to make them better at them.

I think there’s already a ton of work adapting LLMs for these kinds of RL environments. Our thing really just sits on top of that. You’ve got this VLM that can act in an environment, for example, and our thing is sort of a higher-level loop that basically says, “Cache the stuff that you find really interesting.” It’s composable on top of any agent architecture.

I’m just thinking about the future. When we use algorithms like this in the real world, I suppose it becomes less about an objective and more about taste, style, alignment, ethics, and so on. Perhaps even in the current setting, it might beat the game, but it might not do it in a very aesthetically pleasing way. What are your thoughts on alignment and putting style, aesthetics, and values into the algorithm?

I think that, following a language model’s notion of interestingness, as we do in the paper, we then project that down to achieving high reward in the environment. It might be the case that the language model finds a lot of things interesting—loads of trajectories through an environment—but then it maybe needs to pass through a filter, such as human understandings of style.

We can design reward functions to score the things that it creates. I think this really speaks to the values we impose on these kinds of systems and the kind of human supervision involved. For example, if you set off a graduate student on a science problem, they might find 10 different paths, and then a supervisor might select 1 for the graduate student to pursue further.

Another thing is that it’s probably not an issue yet because we’re dealing with quite abstract forms of reasoning. But at some point, with real-world applications, do you think we’ll have a problem with some of the cultural biases in language models pulling them in a particular direction?

Yes, 100%. One failure mode that we hypothesize is this: What if the LM—this is a very contrived example—is trained to eat green, and all the interesting paths through the environment are green? Then you would expect something like Intelligent Go-Explore to really just amplify the biases and not find the right thing.

I think this really relates to work in debiasing language models. Perhaps there’s some kind of training element or supervision involved. We’ve got to collect corrective data for these kinds of things, but we also have to continuously monitor them.

I don’t think there’s a right answer, either. Biases are all around us, and I guess we need to correct for them adaptively as we see them happen.

So you guys wrote “The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery.” This did the rounds on Twitter and was picked up by newspapers all around the world. It’s very exciting, although there was criticism of it as well. Why don’t we start with the open-ended piece? Tell us about the paper in 1 sentence.

We try to use LLMs to write new papers that are hopefully helpful to the community eventually.

I think something that’s important is that it emerged from essentially all of the work we’ve been discussing before. In many ways, DiscoPOP showed that we can write code and do discovery at that level. Basically, Chris had this insight, or this intuition: maybe we can go beyond. Maybe we can automate the entire loop, starting from idea generation, to writing code for experiments, executing them, and finally writing a paper based on the log and numerical results.

I think this was one of the key insights I had on a meta-scientific level. It’s all about timing and realizing what is just reachable with these systems. The impact is still to be seen, in some sense, but we’re all pretty excited because we had many moments where we were mesmerized by what we found.

I think that was one of the key themes: We were constantly shocked by what we saw. Chris’s Twitter thread had some early examples, and the early stuff is really not that inconceivable. It’s like: propose 1 experiment, run it, write a few paragraphs, and what do you need to get a reasonable-ish paper?

Just chain that a few more times, explore something a bit more deeply, and try to connect it all together with the story. Speaker 1

One of the things about the AI Scientist is that, exactly as Rob said, it’s a function of time.

We already see that, as the AI Scientist’s capabilities increase, they’re being used in ideation. People from Stanford have been thinking about this kind of thing. LLMs can generate ideas that are just as novel as those of human researchers. There are review agents and coding agents. Why not just chain that all together? I think we demonstrated that proof of concept of chaining it all together.

Thousands of researchers are now working on every single component. In the next year, as all that knowledge comes back to us and we chain it together again, I think we’re going to see wonderful things.

I think Cong also likes to talk about this as the GPT-1 moment of scientific generative AI for science, right? I feel it’s not only the community coming together and improving each module of the system, but also the increased capabilities of frontier models going forward. We’re at a great Cambrian explosion, I think, for these types of systems.

One interesting thing is that I looked at a bunch of the papers and, superficially at least, they look fantastic. They look fantastic. If you dig into them, you can see a few problems. But if you get a language model to generate an entire paper—if you say, “Generate me, in LaTeX, an entire scientific paper,” zero-shot—it will be pure banality. It will superficially resemble a scientific paper, but it will look terrible.

Then the next step is something that many of us do because we learn how to use language models. We use Cursor and generate a sketch. We select some text and say, “Give me more,” and we bring some source data in. Double-click on that, generate a table, improve it, improve it, improve it.

What you’re doing is a little bit like the Google Maps analogy, where you zoom in and the tiles get smaller and smaller. We supervise this process and just add more detail. It’s almost as if the implication is that it’s not necessarily that language models can’t do it; they just can’t do it all at once.

What you need to do is zoom in and zoom in. Essentially, what you’re doing is leveraging more effective computation where it needs to happen. Your work demonstrates that we could potentially automate that entire process.

Exactly. What you’re describing would also be really cool if there were a human in the loop. We just want to push the limits as to what we can do fully autonomously.

I think part of the original idea came from the fact that DiscoPOP—the results of DiscoPOP itself—could be its own paper, right? The discovered loss function, if we could describe why it works and all the interesting things about it, could be its own paper. The only thing that was really missing was the write-up. LLMs are good at writing, so it was an easy thing to put together.

While there are issues with the generated papers and their details, I think the fundamental capabilities are almost, or mostly, there.

On to the devil’s advocate. I was reading Hacker News, and the most upvoted comment was this: Both the community as a whole and the people within it don’t learn by simply reading papers. We learn by building things, running our own experiments, figuring out how other context fits in, and discussing with our colleagues. That’s why it takes 1/8 of a lifetime to go from the world’s standard of knowledge to being a PhD, basically.

This is quite a common argument with generative AI. It’s not so much the output artifact. We were saying, “I could go back in time and give Newton a physics thing,” right? It’s this kind of cultural, memetic knowledge transfer. It’s the physically embodied process of doing science and doing exploration. So do you think we still need that?

100%. What I would really like to see is our next generations of AI Scientists being able to incorporate all that knowledge that builds up over months and months of scientific exploration.

For example, as PhD students, we get—say, even if we try and fail at a project for 6 months, fail again for another 6 months…

We have so much intuition about what doesn't work and what does work. A lot of this is, I think, hidden as well from the scientific community. We only publish positive results. Something like the AI Scientist, for example, is really valuable. Can we use AI scientists in that sense as just a data-generation tool for wider exploration—intuitions about what might work and what might not work?

Can we get much deeper knowledge than just the tip of the iceberg that's published in a paper? Can we have that? Can we distill that back into a system? One thing I like thinking about is, for example, we've got systems like o1 and the open-source one, R1, which is based on RL. They perform well on things that have ground-truth rewards, like math and coding, for example.

Could we, for example, chase benchmark scores, even with all the problems that they have, and optimize AI scientists to do better and better at these RL tasks? These diffusion and NLP tasks? Would this allow it to incorporate all of that knowledge from exploration, all that failed experimentation, into the process and truly go toward the steps that we would go through in our PhD process?

Yeah, I think maybe one point to that. In the AI Scientist paper that we put out, and in the system more generally, the experimentation is fairly linear. You have an idea, implement an ablation or whatever in code, run that ablation, and then get a result. If the result isn't positive, you still write a paper about it, basically.

When I think about my own scientific process, I spend 80% of the time trying hypotheses and rejecting them, coming up with a new plausible hypothesis, and doing essentially causal modeling, right? In the end, I spend the remaining 20% writing the paper, getting more empirical results, and so on.

Something that we really want to see going forward is this more iterative approach of specific hypothesis testing and integrating the knowledge of that experiment into the next experiment. I think once we unlock this and go to a more open-ended system that can really reason about the results collected using some ground-truth code evaluation, for example, we're going to make major steps forward.

I suppose it comes down to the philosophy of what science is. Is the purpose of science—I mean, obviously, it's epistemic foraging, basically. We're discovering new knowledge, and is the purpose to enhance the knowledge of humans, or is it just to enhance knowledge in and of itself?

There's this weird thing I see with generative coding, which is that it's getting so good now that I can write software completely automatically, and I don't even understand what it's written. Right? And when there's a problem, we have this problem of understanding the code, which is, like, now I need to go back to first principles and understand the code that has been written. Do we risk that happening with AI scientists—that it will just create all of this knowledge, which is still weirdly quite far and hard for us to understand?

Yeah, I mean, I think that's kind of the point of papers, to be honest, in the academic community. Rob talks about how he spent so much time building intuition and doing 80% experimentation and things like that. If that just all stayed in Rob's head, that wouldn't really be science, right? He ultimately has to put it in a paper to share with everyone, and then hopefully we can get some glimpses of the intuitions that he got there.

It's kind of the reverse of the commentary here, where another set of commentators was like, “Why do we have these things? Whole papers, right?” The real nugget is maybe a couple of sentences of information.

So something that we also have as an artifact is the code that generated the results, right? There's not only the paper but also essentially a reproducible pipeline for generating the results. More generally, maybe a bit contrarian: once we get to the point where an AI Scientist system can get papers accepted at a conference, for example, we're really at a point where we probably have an equation that says, “Money, compute, API calls equals paper.”

I think we, as a scientific community, really need to rethink whether the paper is the right medium, or whether we need to think much more about what a scientific contribution ultimately, at its kernel, is. If you think about residual networks and residual connections, it's a 2-line code change that has diffused into almost everything in machine learning. I think how well an idea diffuses is a much better metric for a scientific contribution than just getting a paper accepted at a conference.

Yeah, that's very interesting. For some reason—and I'm probably being puritanical and old-fashioned—I feel that there's the memetic plane and the output plane. The papers are the output plane; the software is the output plane. Yes, we can actually increase the cultural transmission between papers and code by doing what you said. We can have, basically, a transaction log.

As well as producing the output artifact, we can also have a log of all of the reasoning, because there's the missing-information problem. There's the why: why did we do that? Maybe we explain some of it in the paper, but it's not all in there. We can have a transaction log, share it with all of the other AI scientists, and everything's great, right?

But I still feel that there's some kind of weird cultural memetic plane that somehow isn't captured in the system. You have a hypothesis, you discard it, you don't write it in the paper, you share it with your friends. You speak with people at DeepMind and all over the place, and there's this weird cultural memetic plane.

Yeah, I agree. Similar to what you said, we're basically lacking these types of logs to directly train on, for example. Since our system is set up so that credit assignment is only done for positive results, if we had a log of all the negative results and trained the system on it, it could probably do much better reasoning and hypothesis testing in an iterative loop.

I think one thing that is for sure also a contribution of the AI Scientist is essentially a lot of logs and a lot of data on things that failed. It's going to be interesting to see how these types of data-generating and synthetic-data-generating mechanisms can also be fed back in order to increase the scientific-discovery abilities of foundation models.

On what you were saying about the cultural aspect of Rob's point—that maybe the parts that are in the paper—in theory, the AI Scientist logs are far more comprehensive than what Rob could communicate to his friends. So, in some ways, it seems like it would be able to do such a thing as well.

That's true. But then, conversely, there's the Stanley effect, which is that you actually don't want it to be that transmissible, right? You actually want to create islands.

I think somewhere in between. There are a lot of different ways you could approach this, but I guess there are different levels of granularity here: the paper is maybe the most refined version, talking about the things that didn't work is slightly less refined, and the raw logs are the least refined thing.

Maybe one more thing I wanted to add is a vision that I'm really excited about: fully AI-run conferences. We're now at a point where you can generate papers in an open or closed loop. We not only developed the AI Scientist, but also an AI reviewer that scores the paper and was validated on previous ICLR 2022 papers.

You can imagine using the OpenReview API, uploading papers generated by the AI scientists, and doing full rounds of individual review driven by LLM debate, as well as an area-chair decision. Then you have a conference put together only from AI-generated content, with best paper awards and orals.

This filtering basically gives a good set, or subset, of these ideas, which can then be used by humans to validate again and to see how well they diffuse. I think this vision of open-ended research and filtering, with human top-level control, is one that excites me quite a bit.

Yeah, for sure. Can we also capture that kind of human debate between authors and other authors at a conference? Can we get that conversation about how papers relate to each other? In many ways, it's very related to what Chris and Rob have been saying: how do we see these ideas propagate into the next AI conference and then, ultimately, perhaps the best score...

The quality of a contribution might be something like the test of time: what has influenced the next generation of AI scientists the most in their paper writing? Do you think we might ever get to a place like that?

I mean, Tim Rocktäschel spoke about this in one of his papers: an ASI might just be so alien and unintelligible that, from a sort of deflationary point of view, this AI Scientist community could develop a weird subculture. Should humans come along for the ride, or would we just see it as quite strange?

Hopefully, the grounding for the work would be in the reviewer pipeline that Rob was talking about, where there is some grounding in comparisons to human papers, human feedback, and ideas. In theory, we could try to remove that, or maybe try to extract some essence from the reviewer about some core scientific contribution that perhaps surpasses current human culture, language, or understanding. But this question—why would we do that? There could be some reasons. What if it develops some completely unimaginable technology that is really hard to explain to humans? That would require a lot of trust, which might take a very long time to build.

But I also think, more philosophically speaking, there are things in nature that our cognitive systems might not be capable of understanding. So I think there is a limit to what we can compute and what we can do. Systems like the AI Scientist might be able to understand more, but we might not understand the output. So I think there are pros and cons to it.

In principle, this is something we already have with other instruments in science, like telescopes and so on. We can't have the resolution that these devices have, but we can still try to make sense of what comes out afterward. It is very much at the right level of abstraction: we don't understand it ourselves, but we have a nice interface that we can exploit.

And I guess there is also a kind of nice proof in that: from ancient civilizations to here, we really are superhuman. Somehow, we've still managed to construct the right abstractions to understand and use things as best as we can. And I think one more important point in that direction is that we all believe that the AI Scientist—and publishing the AI Scientist—is something really important in the current state of AI, so that the community can come together and discuss many of these philosophical questions: What is science at its core, and what is a PhD student going forward? I think it's important to have this type of discussion early on.

Right, guys? It's been an honor and a pleasure. Thank you.

Thank you so much for having us.

My pleasure. This has been great.