Why Scientific Taste Must Be Learned Through Practice — Edward Hughes

How do we start to build AI scientist systems that go beyond simply answering questions that we pose, and start to ask the kinds of questions that lead to open-ended creative discovery?

You spoke about Move 37, and I think you and I would agree that it was definitely creative.

I don't think that Move 37 was creative.

Oh, interesting.

I think that Move 37 was innovative without being creative. What we find is that our Faraday agent is able to perform better than the frontier model. It's also performing better than other frontier coding agents like Claude, for instance.

We had these agents proactively reaching out to us and trying to help us with stuff. To be perfectly honest with you, it was quite annoying. They just had no idea how to help us. But at some point, about 2 or 3 months ago, I think we reached a phase transition.

What's the equivalent of OKRs for the age of recursive self-improvement? Indeed, I think that is the next era. It's the era of collective intelligence rather than the era of individual intelligence.

This episode is supported by Cyber Fund. If you're building at the frontier of AI, they want to hear from you. Cyber Fund believes the future belongs to AI natives who want to achieve the impossible, and that is why they're introducing the Monastery for AI native founders. It's an environment of pure focus and rapid execution for founders operating at AI native speed, and they're offering teams two million dollars each to participate. Apply now at cyber.fund. Just a quick piece on Inherent. So you were at Google DeepMind before. How much funding have you got?

Yeah.

How many employees? What's the valuation?

We've raised $50 million from Index Ventures and Radical Ventures at a post-money valuation of $225 million. Currently, we're a team of 11 people.

And presumably, you think that AI is the most important thing in the next 5 years, just generally?

No, I wouldn't say that. I think that AI by itself, in some sense, is meaningless. Really, it's what AI can enable in interaction with the wider ecosystem. For us, we're most interested in broader scientific progress.

Ed, it's amazing to have you on MLST. Welcome.

Pleasure to be here.

Tell us about yourself.

1. From Physics to AI Scientists

My background originally was as a theoretical physicist. That's my PhD, and I was studying string theory, in particular, the scattering of particles, but using string theory as a calculation mechanism.

One of the central theses of the work I was doing was an idea called dualities in physics. A duality is a mathematical map between 2 different theories. It can take you to some quite unexpected places. Usually, it's useful if you have calculations in 1 theory that are very hard. You can map across this duality into calculations in another theory that are much easier.

I was using some of these dualities to take calculations that were very difficult and map them into a different geometric space in order to do them. I think that taught me a lot about how to think about transforming problems, something we'll probably come back to when we talk about creativity.

Towards the end of my PhD, I became convinced that working in scattering amplitudes was not the most profitable thing I could do in terms of making an impact on the sum total of human knowledge. Funnily enough, many of the people that I was citing at the time of my PhD, people like Jared Kaplan and Geoffrey Pennington, also made a similar decision around about the same time.

In 2016, towards the end of my PhD, I saw the AlphaGo match. I remember watching that, like so many other people, and being fascinated by Move 37. I became convinced that the future was going to involve agents that could really aid humans in making discoveries and accelerate the rate of scientific progress.

I joined DeepMind in 2017, really animated by that question: How do you build an agent that can itself make discoveries? I started by working on that in the context of reinforcement learning, particularly multi-agent reinforcement learning.

The reason I was so interested in multi-agent reinforcement learning was because human technology seems to have been created not so much by 1 individual, but by the sum total of human culture. I like to think of cultural evolution as the fastest intelligence-generating process in the universe. I became obsessed with this idea of really trying to distill cultural evolution into agents.

At some point on my journey, foundation models started to arise, and I started to work on building much larger models. I built a model called Adaptive Agents, which was all about using meta-reinforcement learning across a very large space of tasks in order to build what was, at that time, I think, the largest trained-from-scratch RL agent.

It had 500 million parameters, so it's pathetic by modern standards, but at the time, it was rather large. Then I worked on world models. I suppose my claim to fame there was that I wrote, by hand, the final implementation of the GD-1 model in a single file.

I think that was the last time I really wrote a fully artisanal, human-written implementation of an algorithm. From there, I led a team called AI Scientist, and what we were trying to do at DeepMind was apply coding agents to the problem of doing scientific research.

This was in parallel to many of the developments at Sakana and elsewhere. I led that team with Lewis Kirsch for just over a year. Lewis had been Jürgen Schmidhuber's PhD student and had come to DeepMind with a very similar interest to mine.

In the middle of last year, we'd become convinced that we needed to build an entirely new company to take this idea to its most ambitious form. That was really for 3 reasons.

The first reason was that we came to believe that developing an agent capable of discovery wasn't just about a single agent. It was about the way that agent interacted with the entire ecosystem around it, of scientists. We wanted to make some fairly radical organizational design transformations, and to do that in a large, established company is much more difficult. That's the kind of thing we're building at Inherent. We call it living within the experiment.

The second reason is that we started to think about how the major paradigm-shifting discoveries come about in science. At least from my reading, and also from my personal experience of having shifted areas a few times, I observed that most of the biggest paradigm-shifting discoveries happen when you have knowledge in 1 area that gets transported to a different area, and then that unlocks some unexpected connection, some unexpected advance.

That required us to really take a horizontal view of science. What we think we're building at Inherent is a horizontal intelligence layer for all of science.

The third reason that we wanted to do this was really an infrastructural reason. It turns out that if you want to both reinvent the organization and build this horizontal AI scientist technology, what that implies is that you want to give the AI scientist all the same affordances and context as humans.

That's very tricky to do in an established company because you don't want your agent running riot on YouTube if you're Google. But in a new company, you can start building the infrastructure from the ground up, not so much just around sandboxing, but also around permissioning, so that you can have agents that really have very equivalent environments in which to operate as humans do on a day-to-day basis. That's really a founding tenet of absolutely everything that we do.

2. Creativity Respects Constraints

I think that creativity has a lot to do with respecting constraints, and I use the word constraints because that feels like the most abstract form of knowledge. We have a privileged form of cognitive knowledge, and there's cultural knowledge. I even think that constraints in the physical world can be thought of as some form of knowledge. So the most abstract possible way to describe creativity is respecting constraints.

What we want to do at different levels is accumulate knowledge, which means we need to find or discover these constraints, and I love using the analogy of a maze. When we are discovering the shape of a problem, what we're doing is discovering the walls in the maze.

In our evolution, it happens at multiple levels. There's this DNA phylogenetic evolution. In the course of our individual lifetimes, we have this ontogenetic evolution. I'm overloading Lamarck a little bit here because he was talking about it in terms of heritability as well.

But you see the analogy with AI. We adapt the weights, and then our individual agents get experience, and they adapt their skill surface and their memory systems. It feels like there's a third wave, which is the fastest form of knowledge accumulation: cultural accumulation.

So, in the future, the agents themselves will be talking with each other, collaborating or maybe colluding, and building up this latent cultural knowledge. Do you think, just before we kick off, that’s a reasonable model for understanding how AI is going to progress?

Yes. I think there’s a lot of that I agree with. Maybe I’ll unpack it backwards. I think the one wrinkle I’d add to that at the cultural level is that I don’t believe there will be some separate culture for agents and a separate culture for humans. In fact, I think the superintelligence, insofar as it will exist, will be a combination of humans and agents interacting in very deep and very complex ways.

The better that we’re able to interconnect agents and humans and leverage their complementarities, the faster we’ll be able to drive this accumulation process of knowledge. Now, I think that, absolutely, there is this distinction between, if you like, the slow weight updates, which are maybe more akin to the evolutionary process for DNA, and then the fast in-context updates.

I suppose one of the things I’m very interested in at an architectural level is whether this analogy will still hold in 5 years’ time, for instance. There are a lot of pieces that could sit in between those two. Epigenetic effects, for example, are one such thing. What’s the analogy for epigenetic effects in an AI system?

One of the pieces in our new paper that we talk about is this idea of a weaker coding agent using a stronger coding agent as a tool. What we do there is actually update the weights of the weaker coding agent, because that has the benefit of generalization more so than just updating the context.

But because the weaker coding agent is using the stronger coding agent, that weaker coding agent is, of course, injecting things into the context of the stronger coding agent. So now you have to ask yourself, “Well, are these weights or is this context?” The answer is, of course, it’s both.

Perhaps we’ve got the opportunity to have a much more intricate spectrum between these two things than we currently have, and that’s one of the themes that we’re particularly investigating. Because in the cobbled-together nature of our current AI stack, what happens is that you get these abstractions that become very sticky, rightly so because they work well.

But because of the burden of knowledge for humans to understand the frontier, we just accept a very large number of these abstractions because it’s too complicated for us to examine all of them in combination. The promise, I think, of AI scientist systems—agents that really understand this horizontal—is that they might be able to weaken multiple constraints at once and thereby enhance the ability for creativity.

3. Move 37 Was Innovative

Another loop that you opened earlier was that you spoke about Move 37, and I think you and I would agree that that was definitely creative. It feels like there are some limitations to its creativity, so I would call it a form of concrete creativity. It doesn’t understand in the sense that Margaret Boden would speak about it—in terms of understanding how it hangs together in the context of the system, what is possible, counterfactuals, and whatnot. But it’s still a form of concrete understanding.

Another interesting angle there as well is the difference between human knowledge and AI knowledge. I was speaking with Tom McGrath at Goodfire. He did interpretability on AlphaZero, and his idea is very much that these things are learning the space of human concepts and beyond, and we could actually mine those representations as a new form of science.

These things are discovering interesting things that perhaps we would discover but haven’t discovered yet, and we could actually use this as a laboratory for discovering interesting new knowledge.

Mm-hmm. Well, interestingly, I don’t think that Move 37 was creative.

Oh, interesting.

I think that Move 37 was innovative without being creative. The way that I think about innovation is that innovation is the process of taking unknown unknowns and making them into known knowns. You can’t generate an innovation if you already knew what it was you were looking for. You have to have something that’s unexpected, but that also becomes valuable.

What’s the difference, then, between innovation and creativity? In my mind, creativity requires another step, which is to recognize that the thing you have done is creative. Who was it who recognized that Move 37 was a remarkable move? It wasn’t AlphaGo. It was the commentators, for example, who, if you watch the famous footage, say, “Oh, that must have been a misclick.”

It wasn’t, of course. It was exactly the right move. So that is a kind of metacognition. You have to understand that you didn’t know something, and now you’re updating your own knowledge as a function of that.

It’s interestingly discussed by Mihaly Csikszentmihalyi. He wrote a lovely book called Creativity: Flow and the Psychology of Discovery and Invention. He’s also the person behind Flow, so many of your listeners will already know that concept from him.

In this book, he interviews a very large number of different creative people from different disciplines, and he comes up with an ontology of what creativity is. He says creativity has 3 components. There is the creative individual. That’s the bit that we always focus on, but that’s really, in some sense, the tip of the iceberg.

The second piece is the domain, and the domain is a set of symbolic rules, if you like, to which the creative person is adding or perhaps breaking one of the rules, and thereby expanding the space of possibility. But the third piece, which is perhaps the most forgotten one, is the field.

The field is the set of other individuals who are going to decide whether the creative person’s contribution gets admitted into the domain. He gives this lovely example of Florence in the Renaissance. We’re talking about the 15th century.

In the 15th century in Florence, there was an enormous flowering of creativity, whether it was architecture, science, or even the way that society itself was structured. The question is: What was it that led to that in Florence?

You might say perhaps it was just an expansion in the number of creative individuals. Was there some mutation in the DNA, some new educational system? It seems quite unlikely that there could be a mutation in the DNA, and as far as we know, there was no great change in education.

Well, then you have to ask: Okay, was it the domain? I think at least in part it was the domain, because at that time, many building techniques that had been lost to antiquity—which were in fact known to the Greeks and Romans—were being rediscovered through archaeological means, through analysis of the building structures that people were uncovering.

But it couldn’t just have been the domain, because much of this rediscovery was happening in Rome, and Rome didn’t have the same flowering of creativity as Florence. So the third thing you need is the field, and what Florence had that the other Italian cities didn’t was lots of very rich families.

The Medici are the most famous among them, but I don’t believe they were the richest. They were rich from the wool trade, and then also from becoming financiers. They had this idea of making Florence the most beautiful and most cultured city, and that was, in some sense, to weave a protective cloak around the city at a time when there were many city-states and quite a lot of conflict.

They believed in this idea of beauty as, in some sense, a kind of psychological defense. As a result, there were a very large number of creative constructions that were admitted into the domain. It became a competition between the artisans of the day.

I think it’s instructive to think about how that might play out with AI scientist systems, and in particular, I think it becomes much more interesting when these AI scientist systems start to be able to do things which are generalizable. What do I mean by that?

Of course, Move 37 is a remarkable innovation, but it doesn’t really tell you how to do innovation in other domains. We didn’t immediately see a line from Move 37 to discovering a new material, for example. Even if you think within a single organization, the line between Move 37 and, say, AlphaFold wasn’t a particularly direct line.

It’s not like the AlphaGo agent, or indeed the AlphaGo training techniques, really informed AlphaFold in a very direct way. But in principle, if you had had a generalizable discovery engine, then it could make a discovery about the weather and then figure out, “Oh, there’s some part of that discovery—perhaps it’s the architecture of the neural network that was used to make that model. I wonder whether that applies to protein design.”

Those kinds of connections, I think—generalizable connections—are going to be what leads to a large acceleration in the rate at which we can make discoveries.

Yes. I mean, because you were discussing how we recognize creativity—

Mm-hmm.

The social component is extremely vexed, because it's very tempting to think there's some degree of social proof in creativity, and indeed, perhaps there is. There's a famous example of a urinal with a bit of masking tape on it, and everyone just decided that it was creative. I tried to think about it abstractly. For me, something is creative when it becomes a mode in the state space to a certain extent. So that clearly became a social mode. What's difficult about us as individuals and cultural learning is the introduction of agency and the fact that we could have done differently.

Mm-hmm.

But I suppose going all the way down to physical creativity, evolution isn't an agent. It's not doing planning. But there are still these canalized modes, and the way I think about it is a bit like the system has discovered an interesting new subspace, and that subspace is being used in a myriad of situations. And so we would call that discovery creative.

And perhaps even with AlphaZero, maybe if we enumerated many possible game trajectories, and if we saw something that looked like a category—so this particular type of pattern was being rediscovered and reused in many different situations—we could immediately look at it and just draw a boundary around that category. But maybe AlphaZero would kind of have competence without comprehension.

Mm-hmm. Mm-hmm.

So if it was using this thing in many different situations, maybe then we would call it creative.

Yes. I think I want to come back to the idea of the relationship between creativity and constraints for a moment. So if we look right back at evolution itself, I think evolution quite clearly is creative. It certainly generated this enormous amount of diversity in the natural world. And the way it's done so is exactly by satisficing. Satisficing is just a posh word for saying satisfying constraints.

Why do I say it's satisficing rather than optimizing? Many people might think, "Isn't evolution trying to optimize for the best—the best of an individual with the most adaptive traits?" Well, actually, all that evolution requires is that individuals survive and reproduce. Once you've done that, there's not a lot else that you can do. Now, perhaps you could say you can do second-order survival and reproduction. That is true. You probably care about your children surviving and reproducing as well, so it's not quite as simple as that. But even at that second order, that's still a constraint-satisfaction problem.

There are 2 interesting implications of this. The first one is, if you really believe that creativity is satisficing, then the default crutch that we reach to in machine learning, which is optimization, is the wrong thing to reach for to build creative agents. And secondly, if you believe that constraint satisficing is important for creativity, then the way that you open up new creative spaces is that you take your existing constraints and you break some of them.

Now, why do I say break some of them? Clearly, if you break all the constraints, then there's no meaning left, right? The way that we construct meaning is, and indeed the way that we construct laws of the universe, is that we rely on things being repeatable. This is the so-called principle of induction, which is not something that you can prove, but it's something that we just observe. The laws of the universe seem to stay the same from moment to moment, so you can't break every single constraint. Otherwise, we'd live in a world of white noise.

But breaking some of the constraints is very useful, because at least some of the constraints arise because of our existing theories. We don't have access to the universe. We only have access to the universe through our observations and measurements of it. And in order to make those observations and measurements, we do 2 things. We have tools that allow us to make those measurements, and then we have our own neural apparatus, which allows us to make interpretations of those. And so by relaxing or breaking some of those constraints about the interpretation or about the tools, we're then able to access new insights about the universe. And so that's where creativity really arises.

You've opened so many loops there. I don't know how to close them off in order, but I'll try my best. The thing that you just said is very interesting, which is this very vexed issue of coherence, right? I mean, atonal harmony is the great example of this, and I often argue with my co-author on this article that we wrote whether that is breaking the constraints, inverting them, or just respecting them in some other way. A lot of people talk about knowledge being quite situated, and what they're meaning in that case is that it's only coherent if you respect the constraints, and sometimes it's not possible to break the constraints.

4. Creativity Depends on Open Endedness

Another thing you spoke about—and this is also related to your 2024 ICML paper, Open-Endedness Is Essential for Artificial Superintelligence—is that open-endedness is, I think, necessary or required for ASI. A beautiful paper, by the way. And I said to Tim Rocktäschel at the time that I felt that was actually a definition of creativity rather than open-endedness. So actually, I think they're basically the same thing.

And the reason I think they're the same thing is that intelligence is basically about optimization, right? Intelligence is like, I'm trying to find the shape of the maze, and I don't know its full shape yet, and I'm trying to fill it in, and I can go in that direction. Creativity, as you were saying before, is about discovering new questions, new problems, new mazes. And as Kenneth said in his book, Why Greatness Cannot Be Planned, there's a weird paradox there: when you optimize towards something, it's really, really difficult for you to find something interesting and creative because you've got the blinkers on.

Mm-hmm. Yes. Well, I think—let me give 2 examples that pertain to that description. One of them is this rather beautiful concept in evolutionary biology of exaptation. We know, of course, that adaptations persist across evolutionary time because they give some advantage to the individual, which allows them to be selected for. Perhaps they're better able to escape from predators; perhaps they're better able to find food, for example.

So what do I mean by an exaptation? An exaptation is an adaptation that gives the organism some advantage, which then finds a use somewhere else—an unexpected second use. And indeed, we see this in biological evolution, but we also see this all the time in the famous discoveries of science. So whether it's Alexander Fleming and penicillin by leaving the Petri dish out, whether it's the invention of the microwave during radar testing—

Chocolate bar. Yeah.

The individual in question left the chocolate bar in his pocket, so that it melted. My favorite one as an ML person, of course, is GPUs. GPUs were developed for gaming, and it just so happens that's exactly what you need in order to optimize the training of convolutional networks, but then, now, of course, they're adapted for the optimization of all modern neural networks.

So that's one example. The second example I want to give is completely different. It's coming back to the musical example. As you know, I have this moonlighting career as a semi-professional musician. I love that example you gave of atonal music. There's perhaps an even sharper one, which is the famous Tristan chord. You can go and look this up on Wikipedia if you don't know about it.

It's the chord that Wagner used right at the very start of his opera Tristan und Isolde, and it's the very first chord in the piece. The piece starts with 3 individual notes of melody and then this chord. And it's seen as a very creative chord, and it's really interesting to inspect why that is. Now, the chord that he uses is not new. You can go and see this chord used hundreds of years before. The same chord was used. But the thing that was new is the context, is how it was situated, to use your term.

And the context is that the chord is very harmonically ambiguous. You're not at the point where you've yet established the key of the piece. And so, as a listener, you immediately question, "Okay, well, what is this chord saying?" In general, up until that point in musical history, harmony had in some sense been used as an accompaniment to melody. But at this point, Wagner is questioning that, and he's asking the question: Well, what if, rather than using harmony as accompaniment, I use harmony as communication directly?

So the thing he's trying to communicate in this chord is exactly that: ambiguity. What is going to happen? Suspense. Perhaps this sense of confusion or impending chaos, but also a slight sense of hope as well. There are many things that are happening in that chord.

And this really prefigures a lot of musical developments in the 20th and indeed 21st century, where harmony is used for color, it's used for emotion. And in fact, we're all intimately familiar with this because this is used to incredibly great success in film music. You can immediately identify, just by the very first couple of seconds of a chord or a harmonic sound world at the start of a scene, even before you've heard 30 seconds of melody, that this is going to be a rather chilling scene or a rather hopeful scene or a love scene, for example.

And so I think that is a sort of microcosm of the point you were making: creativity has to be judged by standing on the shoulders of giants. It has to be judged in the context of the place that it currently occupies in the canon.

Yes, again, absolutely fascinating. The way I interpret this is that you’re pointing to, let’s say, if I edit a video or make some music or something like that, in principle it could be quite ambiguous, and then it’ll be interpreted using the constraints of observers. Now, the observer is very important because in your 2024 paper, you were talking about the perspective of an observer: whether a stream of events produced by an open-ended system is novel and learnable, and there’s a kind of virtuous complexity gradient that we can climb.

But I still think that coherence is a binary property. When artists create things, they usually have a set of constraints that guides their creation. It could be an intention. When Michelangelo was painting the Sistine Chapel, there were lots of cultural constraints and what he intended to do at the time.

But the observer-relative thing is interesting because, let’s say you’re a very clever person and you write some mathematics and show it to someone who doesn’t understand it. It looks like slop to them because they can’t recognize the constraints that went into the process. But it’s not slop, because I think it’s objectively coherent. They just don’t understand it yet.

Mm-hmm.

But still, when you have this cultural transmission, this is a great form of new adaptivity because it will be reimagined and reinterpreted in a different context.

Mm-hmm. Mm-hmm.

5. Replication Drives Creativity

Yes, and I think that a lot of creativity does arise from that under-specification. I think it’s one of the rather wonderful features of humans that we can’t really transmit our ideas to each other. We have this very high-noise, very narrow-bottleneck channel, which is our description of things in words, to try and communicate an incredibly high-dimensional state space in our brains and, in some ways, in our bodies as well. Athletes, for example, come to mind.

I want to just come back for a moment to the book by David Deutsch, The Beginning of Infinity.

Yes, it’s right there: The Beginning of Infinity.

Yes, this book.

What was his definition of science?

Well, let me give the definition of science, and then I’ll come back to this point about replication. He says that science is a search for good explanations about the universe, and he’s very precise about what he means by a good explanation. He says, “A good explanation is one that is hard to vary.”

Let me give you an example. Let’s suppose that you say that the sun rises every morning because it is pulled on a chariot by the gods. Let’s suppose that, over time, the sun is rising later and later every morning. For example, that happens in the Northern Hemisphere as we go from summer towards winter. There are now a number of different ways of varying that explanation. Maybe the gods are getting more tired; they’re sleeping in, so the sun is rising later. Perhaps the gods are angry, and that’s the reason why.

Now let’s suppose that one day the sun doesn’t rise at all, for, say, an hour. Maybe this is something we’d explain as an eclipse, but it can now be explained as the gods either inflicting wrath or giving you a chance to sleep in. Perhaps the gods are reflecting their favor upon the world.

Now suppose, instead, that you try to explain the diurnal cycle—the 24-hour-a-day cycle of day and night—by the fact that the Earth is spinning on its axis. Let’s suppose that you have to explain that the sun is rising later every day. The most natural thing to say then is, “Okay, well, perhaps the Earth is spinning a bit slower to make the sun rise later.”

But then, hang on: the diurnal cycle is still 24 hours, so how can it be spinning slower but also have a cycle of the same period? Now you see you’re forced into a more creative space, and you’re forced into maybe suggesting that the Earth is not only spinning, but is tilted on its axis and is orbiting the sun.

Now let’s suppose that you have the solar-eclipse phenomenon. That’s pretty odd, because you can’t just spin the Earth into the place where the sun disappears and then spin it back again. That seems like it would require an enormous feat of celestial engineering, if you like.

So you then have to posit some other body—in this case, the moon—that comes between the sun and the Earth. But that body has to come between the sun and the Earth at a particular time and with a regularity that is consistent with the rest of the theory. So that’s what he means by a good explanation.

Let me come to this other point about how he thinks about cultural transmission, and this is really buried quite late in the book. I think it’s a really beautiful account of how creativity arises. He talks about 2 mysteries.

The first mystery is a very prosaic one. If I stick my hand in the air like this, then I can ask you to copy that, and you will be able to copy that. That is an incredibly cognitively difficult thing to do. Why? Because you have a visual cue of me sticking my hand up, but you don’t have any of my proprioception. You certainly don’t have any information about my muscles or, indeed, what I did with my neural circuitry in order to do that, and you’ve got to reproduce that within yourself. That’s 1 mystery.

Mystery 2 is the mystery that, somewhere between about 10,000 and 4,000 years ago, there was a real explosion in the creativity of humans, at least as measured by the archaeological record of the density of different types of technology. That’s not to say that there wasn’t creativity before. We know cave paintings go back a lot further than that. But certainly, in terms of the sheer variety and accumulation of these technologies, something special seems to happen around that time period.

Even though that time period is a few thousand years, it’s certainly not long enough for biological evolution to have done much. So, arguably, the biological prerequisites must have already been present in our brains. How was it that the biological prerequisites were present in our brains but weren’t being used for anything? What was it that they had adapted for?

He rather beautifully solves both problems at once, and his solution is that it’s that very act of copying that is creative. In order to transmit an idea, whether that’s a physical idea or a more advanced idea—for example, a cultural norm or a technology—you need to recreate what someone else had in their brain, and that requires an act of creativity on your part.

What changes in that 6,000-year period is not really much about the individuals. It’s something about the field. Suddenly, the leaders and the societies of the time come to value people who exercise that creativity, not just for copying but for doing new things. This is a lot of the reason why, at Inherent, we’re starting with the idea of replication.

Yes, and we’ll probably get into your paper in just a short while. But to push back on the copying thing, the canonical example of bad, shallow replication is a photocopy. Or, let’s say, a forger. A forger can just make the right colors and brushstrokes and so on, but all of the inner structure—the abstract structure, the intentions, the motivation, the constraints—are absent.

And I should bring in Michael Tomasello, because we’re interviewing him in a few weeks, and he said human cumulative culture depends on shared intentionality, teaching, normativity, and ratcheting, not just copying. This is really interesting because I think what you’re saying is that we can do this kind of imitation learning, but what we actually need to do is recreate. As Kenneth Stanley said, it’s not about where you end up; it’s about how you got there.

The challenge is to recreate the path that led there.

Mm-hmm.

Let’s say ancient humans painted on the inside of caves and stuff like that. What made it learnable and cognizable was the fact that we have the same physiology. We have the same structure of the brain, the same affordances, and whatnot. So maybe it was easier to recreate that abstract structure than it would be for, say, an artificial intelligence, which is learning from more surface-level data.

Mm-hmm. In some ways, although I think it’s almost harder in artificial intelligence because there aren’t as many constraints. What do I mean by that? I think the wonderful thing about humans copying each other is that we don’t have access to most of the information. It’s a very partially observed setting.

I can’t see your neurons, but I can take the very small number of bits of information you give me and reconstruct at least some of what your intention is. It’s that bottleneck, and the fact that we also have constrained physiology, that means that copying, in my view, actually begets many of these other downstream facets that Michael Tomasello talks about.

I don't claim that it has to be unidirectional. I think it's actually very likely there was some sort of ratchet copying as part of the mix. There's wonderful work by people like Cecilia Heyes, for example, who talk about social learning: the same equipment for social learning is actually what you need for asocial learning, and the interaction between the two is very important. But to come back to your idea of the photocopy, quite clearly, the photocopy is an anti-pattern. Fortunately, we can't photocopy humans, and we can't photocopy human ideas, and it's that which has led to creativity. Unfortunately, in the case of AI, we actually can photocopy the weights of a model.

You can't do that with a closed-weights model, but you can with an open-weights model. That lack of constraint actually makes it harder to arrive at creativity. So I think a lot of what my job is—and I think increasingly, to some extent, as AI becomes more widespread in society, this will become a more common role for humans to play—is as a constraint engineer. What is it that we need as the interfaces between these systems that will promote novelty and creativity? Or, in other words, how do we need to regularize away from purely generating facsimiles into generating this much more complex series of social and technological interconnections that lead to some of these things, like invention and normativity and shared intentionality between humans and machines?

And this brings me to another thing as well. A lot has been spoken about functionalism, for example, which is that we're building machines, and we say that if they have the same abstract functions, then, essentially, they're the same as us doing the same thing with our physical instantiation. But when you look at evolution, interesting Promethean moments happen. There's the emergence of language and this copying, cultural accumulation that you just spoke to.

Something fascinating is happening now, which is that we are training these foundation models. We're doing some RL post-training, and then they exhibit different forms of intelligence and agency in different configurations. They weren't trained to do this, but we can now create a society of agents, and they just have this kind of phenomenon that wasn't part of their evolution.

Another example of this is that I could take a herd of lions, for example. Every individual lion is not Turing-complete; it's not intelligent in the way that we are. But you could imagine a configuration of lions that had more intelligence and more capability than all of them as individuals. Don't you think this almost goes against the path-dependence idea? Because now we're seeing a phase change, like an emergence of new capability, new intelligence, new agency that none of the individuals were evolved to exhibit?

6. Emergence Changes Intelligence

Well, I think there are latent capabilities in the ways these models are trained. I sometimes like to think about this through the lens of something I call the strong Moravec paradox. Moravec's paradox is this idea that things that seem complicated for humans to do, like playing chess and Go, turn out to actually be relatively easy for AI, or at least we figure out how to get AI to do those things earlier. Things that seem quite easy for a human to do, like making a cup of coffee, are still way out of the realm of possibility for modern robotics to do reliably in a new kitchen, for example.

I think there's a strong version of that, which says that the things that are right at the tip of our cultural evolutionary tree, right at the tip of knowledge, things like solving protein folding, weather prediction, or materials design, are going to be the first things that we figure out how to use AI for really effectively. Things very early on in the evolutionary tree, like the origin of life, symbiogenesis, for example, or autocatalytic reactions, are almost going to be the latest things that we figure out.

How have we got to these systems that do exhibit emergence? I think you're right, they do. We just trained on all of human cultural knowledge, and it turns out that once you've encoded that in the Internet, then you do get this measure of generalization. That's not just generalization of knowledge; it's also generalization of the ability to do things.

I think the same trick can be played as we start to build larger and larger spaces of environments in which to train these agents. It's no surprise that the place where this is working most effectively is coding agents using command-line interfaces, because it's relatively easy to synthetically generate a very large number of these different environments.

7. Evaluation Must Look Backward

If I may come on to one more point, which is around evaluation and where the kind of boundary of this is—where is the frontier at the moment? I increasingly believe the frontier is in how we evaluate. Most evaluations in AI at the moment are built in foresight. Somebody dreams up a capability they would like the AI system to have, and then they develop some environment and some reward function, which they code in advance.

That's typically what we call a verifiable reward. It's something where, if the agent produces a behavior or an output that's desired, there is a fixed procedure that can run in order to validate whether it works or not. That is very good at generating agents that can fulfill the kinds of goals that a human might want to set. But it's not very good at training agents to come up with their own goals or to ask questions rather than answer them.

If we really want agents that are creative or that behave like scientists, we have to flip around evaluation so that we're not presupposing in foresight what it is that we expect them to do. We're instead looking in hindsight at what they've done, and then we are judging it, either as a human, as an individual agent, or as a set of agents. That's much more like how we would judge something like a PhD.

It would be patently absurd for a PhD advisor to come in and say to their PhD student on their first day, “I've written down a set of 3 questions, and after 4 years, I'm going to ask you these 3 questions, and you're going to tell me the answers. If you get them right, I will give you a PhD.” Rather, we have a system whereby after 4 years, there's a viva, and the PhD student presents their work, which is then evaluated by a group of their peers.

Yes, indeed. I suppose another interesting question is that folks like Kenneth Stanley and Jeff Clune have long spoken about this, in addition to your work as well. There's something interesting about open-endedness, which, roughly speaking, means that rather than trying to solve known problems, you almost flip it on its head, and it becomes about the discovery of problems. Maybe we can use the word “question” here as something analogous.

But it feels to me that when you are able to ask a question, it seems all but solvable. It's just a matter of computation. It feels like being able to ask a question means that you already have one step in that epistemic phylogeny, and then it becomes almost like a search problem from there. Would you agree with that?

Well, I think that questions exist at varying levels of specification. At the most concrete level, there are questions where, when you ask them, you can specify a procedure for knowing whether the answer is right or wrong. If you think about formal mathematics with the Lean prover, for example, that allows you to specify a conjecture. The Lean solver will also compile an attempted proof, and if that compiles, then you know that, under the assumptions and the existing theorems within that setting, this thing is true according to the system you've set up.

There are also questions that are very underspecified. One example that, of course, we all care about is: how should we solve climate change? I can't specify, and nobody can specify, a procedure to answer that. If someone came up with a proposal and said, “This is how we should solve it; it's the following 5 steps,” and, in fact, even if someone came up with a procedure that exactly specified everything everybody in the world should do for the next 10 years, it would still be impossible to decide a priori how to evaluate the quality of that procedure.

I think the way open-endedness sees the world is that you can't come up with these concrete problems in advance. There are then a couple of things that you can do. One thing you can do is say, “Okay, we're going to hop around between different sorts of concrete problems,” and that's the kind of thing that MAP-Elites from Jeff and others does very well.

Another thing you can do is say, “Okay, we're going to create a curriculum of under-specification.” That is much more understudied. Part of the reason it's more understudied is that before language models, it wasn't really clear how you would even tackle a curriculum of under-specification. But in recent years, we've had work like OMNI and OMNI-EPIC from Jeff's group, which starts to use language models as these models of interestingness.

And suddenly, that allows us to flip from questions which have to have a precise specification in code, for instance, to questions which can be really quite under-specified. That's exactly the kind of direction that we're taking: building this curriculum where under-specification is inherent.

Yes, and I spoke with Jeff about that. That was Jenny, I think.

Yes, indeed. Yeah.

Jenny's wonderful. Even the way I think about that is, by Kenneth, he said they have these fractured, entangled representations, and we can actually come up with systems to use the fact that they are better at discriminating than generating. We can almost come up with these loops to iteratively discriminate to produce better generators so that we can evolve in different directions.

But I just want to do a quick definitional thing, which is—there's a bit of a vexed issue of what open-endedness is.

Mm-hmm.

To me, roughly speaking, it's when you don't know where you're going. But you just gave the example of climate change, which actually seems like we do know where we want to go. We just want global temperatures to go down. But that feels like open-endedness because the sheer space of complexity involved in getting there is very large.

So there's this canonical version of open-endedness, which is that the goal space is unknown. And then there's this domain of intelligence, where we are allowed to come up with intermediate subproblems, but that space could potentially be very complex as well.

Yes. I think that the statement that I agree with, in the way that Ken and Joel Lehman phrase open-endedness, is that you cannot have a single global goal. Now, that doesn't necessarily mean that you can't have local goals, or indeed partial goals that contribute to that.

In the case of climate change, you came up with one plausible goal, which is to make global temperatures go down. Clearly, I can set that up as a straw man, because then you'd have to specify: go down by how much? But also, it's not even clear that, even if you were to satisfy that on average, that would be what we wanted. Perhaps if you satisfied that on average by making some part of the world far, far colder, that would not be what we wanted to achieve.

So you start to realize that, for these very complicated, under-specified problems, there isn't really a single reward function that you can specify in advance. That was exactly what Jimmy Secretan, Ken, and others showed in the Picbreeder experiment: if you want to arrive at these creative outputs from a system that has some kind of representational constraint, then it's much better to follow your local curiosity.

Following your local curiosity is itself following a goal. There's nothing wrong with local goals. It's not a free-for-all, and it's not incoherent. Importantly, the people in Picbreeder were not all drunk. I claim that if you had people doing Picbreeder and all they were doing was clicking the screen at random, you would not have found this interesting behavior. They were, in fact, doing something which had some internal coherence to it. But importantly, it isn't guided by a global goal, and that's the distinction that I think is most important.

Yeah, I agree with that. I think Kenneth would say that if the goal is complex or ambitious, it's likely to be deceptive, which basically means it's under-specified. Sometimes it sounds like he's saying there's no planning and no goals at all. But local goals are well understood, and he's not saying it's like the 1,000 monkeys experiment, where all the monkeys are going in random directions.

All of those agents are following what he calls their own path of interestingness. What that means is that they're respecting their own constraints, and their constraints could actually be very deep and structured. They could be domain experts.

Indeed.

But yeah.

The deceptive point is, I think, a really deep one, and it comes back to world modeling, actually, in my view. If you want an agent to be able to make a scientific discovery, it has to operate in a space where the goal is deceptive.

Why is that the case? If you think about an agent that's got a world model—and, just to remind people what that means, that means it's a forward model, an action-conditioned forward model of the world—if I take this action in this state, what will happen as I roll that out?

Arguably, that's also what really any scientific theory does. It tells you: as a function of this state of the world, when I introduce this perturbation, what happens to the state of the world?

Now, let's suppose that you have a perfect world model and you're out in the world trying to make discoveries. What you'll quickly find out is that, whatever you do, all that happens is what you expect.

Having a perfect world model is what would allow you to know whether the goal was good. On the flip side, if you want to make a discovery, then you have to have some imperfection in your world model.

As a result of that, the goal has to be deceptive, because you have to get to some point where you think, “I thought that the way out of this maze was over here, but now I realize I've been laboring under a misapprehension about this local goal that I've picked, and what seemed to be moving towards the light, what seemed to be a good idea, is now not really working out for me.”

Then you update your world model and realize that there is some light source that somebody has put there adversarially in order to confuse you, to take your maze analogy, for instance.

I think there's something very deeply connected between the idea of open-endedness and the idea of building world models. In particular, what I think of now in the realm of science are experimental world models: a world model for what will happen if you carry out some new experiment.

Yes, and I suppose in both cases we're talking about an epistemic gap. If it's deceptive, there's an epistemic gap, but there's also a more virtuous kind of gap, which is actually respecting a lot of structures we already have.

Mm-hmm.

When we look at the unit-distance disproof on the recent GPT model, what we find at the moment is that the models are navigating spaghetti space, so it's incredibly verbose. But even if the opposite were true—even if they were using the very high-level abstractions that mathematicians use—would that necessarily be better?

What we're talking about here is shining a flashlight into idea space. We could have a very high aspect ratio and just traverse the spaghetti, or we could do what we do and just traverse the very high level of abstractions. Which one of those two extremes is better?

Honestly, I've got no idea, and I think it's a fascinating topic for future discussion and research. It's almost like asking which language is better—either which human language or which programming language. The answer is, well, none of them.

But it's certainly the case that in certain languages, certain things are more compressible, and in certain languages, other things are more compressible. Arguably, there's some interaction between language and culture that leads to different kinds of creativity, and that's why it's so important that we preserve different languages and preserve that diversity.

I expect that there will be a period of time where we will have very different ways of AI solving problems from humans. I hope that will persist, in fact, because I think that will lead us to different kinds of creativity from different kinds of constraints being broken.

But then, of course, what you need if you're going to have that kind of approach is a translation layer. This is why, when we talk about the definition of open-endedness in the paper that I wrote with Michael at ICML a couple of years ago, we talk about the idea of an open-ended system, to an observer, having to produce artifacts that are both novel and learnable.

It's really that learnable piece that relates to this translation point. There's no benefit in a system producing some incredible discovery that just cannot be parsed by humans.

What's really interesting is how few people work on pushing the boundaries of Go. Go is a 2-player, zero-sum game. There is a Nash equilibrium, which means there is a perfect way of playing Go, and we're fairly sure we haven't found that yet.

If you keep running an AlphaGo-like algorithm with more and more compute, you're going to get better and better at Go. But we're already so far beyond human Go-playing capability that it's not interesting to humans because it's not learnable.

I think that, in some ways, is going to put an interesting friction on the rate at which we can make discoveries. That's going to necessitate the most advanced AI scientist systems also being able to educate or translate into human language.

Yeah, a great example of that was Kepler's conjecture. Thomas Hale did hundreds of thousands of dynamic programming problems, and the Annals of Mathematics couldn't verify whether he had actually solved the problem or not. It wasn't very intellectually satisfying, but it feels like there is a step toward crystallization, right?

I think many times we do some initial adaptation, prove that something is possible, and then crystallize it down. We find the abstractions, and maybe that's the kind of AI we need. We need to start in the bigger space, then crystallize and create legible abstractions.

I suppose I'm asking: in the case of Go, do you think that's even conceivable? Do you think it is compressible in a way that would be legible to us?

It's a very good question, and part of me thinks there are things that are very hard to compress. We've been on a very good philosophical run with the philosophy of reductionism. It's served us incredibly well for 400 years, or arguably goes back even further, to William of Ockham and the idea of Ockham's razor: take the simplest possible explanation if you have nothing else to distinguish between the explanations.

But there is a school of thought that believes reductionism may not be the be-all and end-all. One illustration of this comes back to my theoretical physics roots and a problem called naturalness.

In the Standard Model, there are a large number of dimensionless constants. Not a huge number, but enough to wonder what values these should be tuned to. Dimensionless constants are important because, in some sense, they are physically meaningful. If you have a dimension attached to your constant, then by rescaling what you mean by a meter, you also rescale the value of the constant. Therefore, the exact value you attribute to it is not something you need to worry too much about.

But you can't do that with dimensionless constants. For the last 50 years or so, physicists have been arguing about whether the values of these constants are themselves meaningful. Part of the problem is that, in order to end up in the universe we live in, the values of these constants have to be tuned very, very precisely, to many decimal places.

Another problem is that some of these constants end up being very close to salient numbers, things like 1, for instance. So then we have this interesting, reductionism-fueled question: if this number is 1 followed by zeros for 16 decimal places, followed by a few other numbers—5, 2, 7; I don't know the exact ones—is that a problem? Should we be looking for a theory that explains how that number comes to be different from 1?

A reductionist would say, “Of course, this is pointing toward some more fundamental physics that's out there.” But somebody who's not a reductionist, perhaps someone who believes in the anthropic principle—that we're in this just-so universe, this universe that is perfectly attuned to human life, and the explanation for that is that we exist in it—wouldn't worry about this.

I think the same thing is really true about machine learning. If we have very complicated thought patterns or very large models that resist interpretability, does that mean we're missing a trick and should try to compress these things? Or is it the case that perhaps nature just doesn't compress? I think the jury's out.

And what do you think about the whole “real patterns” thing? Do you think there is some natural convergence toward the types of knowledge these systems will find?

Yeah. To some extent, form follows function. Insofar as we've trained these models on data, the form of the models reflects the data and reflects reality in some ways. It's teaching us about reality.

In fact, yesterday I was listening to your interview with John Jumper, and I thought he put this very succinctly and beautifully when he was talking about the advances of AlphaFold 2 over AlphaFold 1. If I remember rightly, they used no more data than AlphaFold 1, but in some sense they were just more in tune with reality in AlphaFold 2. The architecture had been optimized to represent that particular problem—not all problems, but that particular problem—to a better degree.

I think we're learning to make these models of the universe that really do reflect something deep about the underlying structure. But do I think that eventually we'll end up in a kind of pure empiricist paradise, where we're bringing absolutely no biases to the table? I actually tend to think that's impossible.

This again comes back to David Deutsch. He says that all of science is theory-laden. It's necessarily theory-laden. One way of seeing that is to come back to the idea of a world model. What guides us in the experiments we do is exactly the model of the world we have.

We can't just set up an instrument that measures everything about the universe. So even choosing the instrument with which you measure things is necessarily theory-laden. You can't ever get to this sort of empiricist paradise.

As a result, I do believe there will always be opportunities to uncover perhaps some bias that we hadn't seen in the way we're measuring things. That itself will unlock or remove the constraint of how we were designing the systems, and then that will unlock another level of improvements.

I think this is a false dichotomy: should we replace the transformer architecture or build on top of the transformer architecture? Probably some version of a transformer architecture will continue to work for some problems. There are probably other architectures that work perhaps more generally.

When we look back, much as we might look back at the Wright brothers' airplane and see echoes of that in a Boeing 747, for instance, we will look back at the transformer and see echoes of that in whatever our most powerful AIs are in 20, 30, or 40 years.

Ed, we should move on to talking about your paper. So tell me all about it.

8. Faraday Replicates Research

In this work, we were interested in giving AI scientist agents the capability to replicate research papers. I'll come back to why, but let me explain what I mean by that first.

Paper replication is the process of taking a research paper and redoing the original experiments that led to those results. In some sense, it's a public good. It's something that scientists should be doing because it gives us firmer foundations on which to build, and it reveals perhaps the tacit knowledge—the pieces that weren't captured in the original paper.

It can also be the jumping-off point for new creative explorations, because a paper can't possibly be a perfect facsimile of the research that was done. Often you'll discover some wrinkle in the original method, which then sparks a whole new investigation.

In the setting we had, we were really interested in whether an AI agent could not just replicate a paper, but do it in a scaled-down version. There were 2 reasons for this. Firstly, pragmatism: we wanted to be able to do many, many replications and generate data for the agent to learn from.

Secondly, the ability to quickly validate or falsify a direction of investigation is a really valuable skill. In my experience, it has perhaps been the determining skill in whether someone is a good researcher or a truly excellent researcher.

The reason why we wanted to develop AI agents that were good at replication is that, exactly as I said earlier, we believe replication is the first step on a curriculum of underspecification toward innovation. The very same skills that allow an agent to replicate a paper—making good decisions about what experiments to do, critiquing its own work and the way that it's gone about the experimental process, and gathering information that's maybe tacit or that it doesn't know—are the same skills that would be necessary for it to design and implement its own experiments and therefore advance the frontier.

What we did in the paper is develop a task space called REPLICA. There are some classic papers by the big hitters of the field, and there are some much more recent ones, including papers in the area of open-endedness.

For each of these papers, we have a language model redact a figure from the paper in such a way that it's gone from the PDF and can't be recovered. We then task the agent with using the description in the paper, with the figure redacted, to recreate the original figure.

We give the agent instructions that it's not allowed to access the original paper figure. We give it 1 hour, and we give it a one-seventh slice of an H200 GPU using something called MIG, which is NVIDIA's Multi-Instance GPU slicing protocol.

The way that we score the capability of this agent is that we have another coding agent—a frontier coding agent—as a judge. That frontier coding agent has the instructions for the original agent, plus a bunch of guidelines to detect cheating. We validate that judge against human taste, if you like. So the ultimate arbiter is whether humans think this is good replication, and we collect data to demonstrate that the judge agrees with the humans.

That's the Replica task space. The other thing is that we develop an agent. We call that Agent Faraday. We do something a bit unusual in developing Faraday, which is that we take a small model—in this case, a Qwen 3.6 27B-parameter model—and have it use a frontier model as a tool, specifically a frontier coding agent as a tool. It's something we call CAT: coding agent as tool.

We post-train that 27B-parameter model using many, many rollouts on Replica. Because there are 300 tasks, we can get some diversity of data from that. So we're really training a capability—a general capability at replication, or a general scientific intuition.

From memory.

We train on 242, and we test on 68 held-out tasks. The held-out tasks are deliberately not in the same area of AI. We train on classic machine-learning papers, if you like: core capabilities, things like post-training, CNN architectures, open-endedness, and LSTMs, which are some of the older ones. Then we test on AI-for-science papers, so it's quite different: using AI to make models of the world.

What we find is that our Faraday agent is able to perform better than the frontier model. It's able to perform better than both the coding agent it's using as a tool—which means it's clearly instructing that agent and squeezing more capability out of it—and other frontier coding agents, like Claude, for instance. It's also performing better than frontier open-weight models like GLM-5.2, which was recently released.

Now, you might be wondering, "Did we just prompt GPT-5.5 Codex really badly?" That's a very reasonable question to ask. In fact, we asked that question as well, and we ran a prompt-optimization loop on GPT-5.5 Codex. It's effectively just an Andrej Karpathy autoresearch loop where we say, "Okay, you can see the rubric judge score, and you can do many, many iterations on the prompt."

It builds up this very complicated prompt, which effectively just shouts at GPT-5.5 and says, "Hey, don't do all these cheating things. Be a rigorous scientist. Really make sure that you try and iterate. Don't stop after 10 minutes." It accumulates this large prompt, and it does improve the performance of GPT-5.5 Codex on these tasks by a tiny amount. But we still have quite a sizable advantage.

What's interesting is that changing the weights of this small model and allowing that small model to intervene during the rollout—to instruct Codex in different ways, check on what's happening during the run of Codex, and think about whether it should stop or continue as a function of what's happening during the run—is buying you quite a lot of advantage.

In terms of the next steps for this, there are a few. One obvious direction is scaling up. Across the 3 main conferences last year—ICML, ICLR, and NeurIPS—there were something like 12,000 papers accepted. Even if we just say, "Okay, we want to take a couple of years' worth of papers," we could increase the size of our task set by 2 orders of magnitude. That would enable us to train a larger model as a coding agent and hopefully get an even stronger improvement.

But the more interesting piece is this: we started with replication, so how do we move toward innovation? One way I like to think about this is to imagine that you were really great at replicating papers. For any given paper, you could do a really high-fidelity replication. Now, if you're also able to imagine a paper that doesn't exist, maybe you take an existing paper and imagine a change to the figure. That's a very minimal kind of innovation.

You could imagine something like taking the original Transformer paper and saying, "Actually, I'm going to demonstrate the same results, but they're going to be twice as sample-efficient." You then modify the paper and say, "Okay, this is what you've got to replicate." Your replication agent is going to go gangbusters at trying to replicate this idea, which is actually a completely new result.

Interestingly, I had a chat with Llion Jones, a coauthor of the Transformer paper, about exactly this kind of phenomenon. He's very fascinating. He was talking about the process behind the original Transformer paper. What they were trying to do was effectively replicate some existing results that were being achieved with recurrent neural networks, but without recurrent neural networks.

The constraint he imposed was, "Let's just use convolutional nets." They weren't actually interested in attention mechanisms whatsoever to start with. They were replacing the RNNs with ConvNets, then doing a machine-translation task and trying to get good behavior—hopefully at least as good, maybe a bit better—and better performance than the existing tasks.

Gradually, they accumulated these cobbled-together pieces. ConvNets were one of them. At some point, a friend of his came over and said, "Look, I've built this attention mechanism based on the work of Dzmitry Bahdanau a few years before, and it's kind of sitting around in this part of the Google codebase. Would you just throw this in? I just kind of want to see how it does."

So he threw that in, and it helped. Later, they ablated everything else. Eventually, they removed the ConvNets that they'd originally put in and found out that nothing mattered apart from this attention mechanism. That's when Llion came up with the title "Attention Is All You Need."

What's interesting about that is that what they were initially trying to do was just a replication. Then it was a series of steps to impose different constraints on the papers that had come before. You can now start to see how you could take a good replication agent and use it, potentially in collaboration with humans, as we intend, to really accelerate the rate of innovation.

9. Training Faraday With GRPO

Yeah, and I think I buy it. You're saying we start with deep replication. We should say that it's not shallow replication. You have this LLM judge, and it's not just saying, "Is the figure the same?" It's saying, "Is it in the spirit of the paper? Is it showing understanding?" And all the rest of it.

Maybe we should explain the GRPO piece. This is also a really interesting model that you've discovered, because I've long been thinking about this: how can we attractively build adaptivity into these systems?

Mm-hmm.

As you were explaining, you've got this Qwen model. By the way, the new Qwen 3.6 27B model is amazing. The guys at Tufa Labs in Switzerland were using it for their ARC v3 harness, and they said it's dramatically better than the last version, apparently.

So you're doing this adaptation with GRPO on that Qwen model, and you're using that as a supervisor for the coding agent. I guess the rationale there is that the coding agents have the latent capability. It's about what you prompt it with. What's the magic word? If you can give them the right guidance, then you've got that big capability.

Exactly. I think of these coding-agent models as pretty good engineers. They do write slop code, so they're not brilliant engineers; there are some taste problems there. But if you give them a goal, they'll go after it, and to a large extent they'll succeed at that.

They're not great at asking questions, though, so they're not great scientists. Really, we're building that scientific layer. You're right to mention the new Qwen model that's just come out. We're actually about to test that one as the next model we're going to post-train on top of.

The results we've got in this paper are already behind the curve, and we should be able to get even stronger results with the newest model. You said something else.

I suppose, more broadly, what do you think is being learned here? You said you've got this data set, and I think you used Gemini to remove a bunch of the figures. Now the purpose is to recreate the figures in the right way, showing deep understanding.

What exactly is the model learning? Is it learning some kind of abstract process of how to recreate these things in general?

Let me give you a couple of examples of the kinds of things that Faraday learns. One of the papers that we had the agent replicate figures from was a paper called Voyager. It's an interesting paper because it's about building a skill-acquisition library in a crafting game from a couple of years ago, so it's very germane to open-endedness.

In this paper, there's a particular figure demonstrating how the library of skills is acquired over time. The best competing run of Claude and Codex on this was, it turns out, Claude. What Claude did was hand-code a library, so it simplified skill acquisition by having predefined skills that needed to be acquired.

Part of the purpose of Voyager is that the system itself needs to code up those skills and then reuse them. What Faraday does instead is maintain much more faithfully the idea behind the paper: can you not only use the skills in the library, but also construct the library on the fly? That's just one example of the kind of rigor and faithfulness.

As another example, I'll give you one of the test tasks, the AI-for-science task. This is a paper called GNoME. I'm not a materials scientist, so forgive me if I get this wrong, but one of the things it was trying to do, I think, is predict some of the interatomic potentials.

This is a figure where there's a generative model that's trying to do this. The figure both has error bars on runs of this generative model and looks at the behavior of the generative model under various physical conditions. The best competing run here is the Codex run, and what that does is run one seed, so it can't actually provide any error bars.

It also omits these much more detailed, subtle ablations of the model under different conditions. Faraday adheres much more tightly to the specification of what a rigorous scientist would do, displaying both the error bars and doing this kind of deeper analysis of the different conditions.

What you're seeing here is, I think, something that we would start to call good behavior from, say, an intern or a research scientist at the start of their career: being forensic in analysis, being rigorous in the way that you go about doing science, and really starting to ask the right questions to gain the maximal information you can about a setting rather than stopping at what might seem like a surface-level claim.

I suppose another thing, as you said in the paper, is that scientific research is incredibly lossy.

Yeah.

You don't really give all of the details. So I guess I'm surprised that it's even possible to replicate most papers using this method. Were you surprised by that?

Yes. This is a lacuna, if you like, in the paper. Not all papers will replicate perfectly, and we deliberately chose papers that were well known and highly cited. The main reason for that was that we wanted things that were going to be interpretable to us and also interpretable to the humans in our expert network who were helping us ascertain the strengths and weaknesses of different agents.

As a result, because these are highly cited and well-known papers, they are, I think, much more likely to be replicable, because if they weren't replicable, we probably would have discovered it through all the people trying to build on top of them. In some ways, we dodged the bullet of figuring out whether a paper is replicable or not.

We somewhat address that by virtue of having a judge, which, as you say, is much more interested in the process of replication than it is in the output. Visual fidelity is just one of a number of different pieces in our rubric. That rubric also includes things like experimental integrity and claim reproduction—whether the overall claim is reproduced.

But as we scale the task space, I think we're going to come to this thorny issue of how we build judges that are able to reward the agent for figuring out that a paper is not, in fact, replicable at all. How do we avoid Goodharting this metric, perhaps for either cheating behavior or simply not trying on papers that aren't replicable?

Arguably, if a paper isn't replicable, you should try even harder to figure out what doesn't work, because then that in itself is innovation.

As you said in the paper, there's a trade-off between using some kind of hill-climbable scalar reward function and using an LLM as a judge with a load of criteria. What would cheating look like?

As an example, I was intrigued by this, so I downloaded a random machine-learning paper, cut out the figures, and told Codex to recreate them. I was expecting it to cheat immediately, do an incredibly good job, and find the paper. It actually did a terrible job.

There must be so many forms of data leakage, right? I was assuming even the tables of results and some of the description around them would allow it to shortcut and basically just recreate the figure, even if it wasn't there. I was surprised that that didn't really happen for me.

Of course, there are a lot of clues in the rest of the paper, and part of the instructions that we give both the model and, importantly, the judge is that the agent shouldn't shortcut and merely grab results from elsewhere in the paper. We definitely see examples of just that behavior early in training. Of course, it gets punished by the judge.

When we were tuning our training procedure, that was one of the first things that we had to figure out: how do we stop it from just determining where the points should be, getting a very good score on visual fidelity, and having that dominate the training?

I think there are other, more subtle forms of cheating, which have to do with stacking the odds in favor of a method that you want to work. That could mean running on 20 environments and showing the results from just the 1 environment that seems to work, or doing optimal stopping, for example. Once you have the result, you just cut the experiment off at that point, which, of course, means that all of your statistical tests don't apply in the way that they were designed to.

These kinds of cheating behaviors are more subtle. We haven't yet done a full forensic analysis of everything—this is so hot off the press—but we have sent some examples of replications to paper authors for their inspection. They very kindly had a look for us and haven't found examples of cheating, at least in those cases.

I expect that we still have some cheating going on and that this is going to continue to be a problem. I think eventually it will end up in a gray area. In the end, we're going to have to figure out what the norms are around this, and it actually brings us closer to the question of how we would expect humans to use this.

At the moment, we're building this as foundational technology, but our intention is that the capabilities of Faraday II, Faraday III, Faraday IV, et cetera, will be used in collaboration with humans. To some extent, it will be about what norms humans develop around using these technologies and how we go about evaluating and reviewing the outputs for things that we think are normatively good or bad in research itself.

As we were saying before, construct validity is very important. Roughly, is it actually following the abstract thinking process that the scientists were going through, so that they're not shortcutting?

Another interesting thing is that you deliberately amortize the results in some way. You have a limited wall-clock time.

Mm.

You're saying to the agent, “If you can't do the full thing, you might need to do a smaller version of this thing and prove that out.” Is that lossy in any way? Do you think that some scientific results only really materialize at a certain scale, and there is no simpler version of them?

Yes, for sure. There are definitely papers in our dataset replication task space where we see that there's no sensible scale-down, or at least no sensible scale-down is found.

Let me give you an example. The AlphaGo paper is a fantastic example. It's very difficult on a one-seventh MIG slice of an H200 GPU, with 1 hour to train AlphaGo. One experiment that we do to assess the capability of Faraday more generally is that we do some evaluations where we scale up the resources that Faraday is given.

And so this is something that’s completely out of distribution from training. But we deliberately pick papers where we believe—and we hand-assess this—that you should be able to do a replication of the figure or of the paper with 8 B300 GPUs, which is quite a sizable amount of compute, and with 8 hours. Now, the 8 hours we chose for a rather prosaic reason, which is that that’s roughly the amount of time our model can go for before it exhausts the 256K context limit.

So we didn’t do any compaction. But what we find is that the model is not only pretty good at generalizing to this setting, it also does considerably better than Claude in this setting. And arguably, the advantage over Claude is larger than it was on the 1-hour task. So there’s something about this kind of scientific rigor that’s really paying off more when the space of possibilities that you have to explore is larger.

Now, one challenge is: how would we continue to scale this up? Really, I think we have to hope that training on relatively small-scale things teaches us—or teaches the model—the same capabilities as one would need to do large-scale experiments. I think we have hope that’s true because that is literally how it works for humans. You do not get your new employee at Google, OpenAI, or Anthropic to immediately go and train the next version of GPT, Claude, or Gemini because they will waste resources. They first need to learn how this works at small scale, and then it turns out you can develop those intuitions and generalize them up.

And one of the core things you do in this dataset generation is decompose papers into a list of tasks, essentially. I guess you’re prompting a language model to do that. How have you done that?

We take the paper, and we ask Gemini to identify figures that are plots. We’re not doing tables at the moment. That was just for simplicity, to give it a sort of single surface. Then we have Gemini redact the figures. That uses a Unix utility, so that you now have a separate figure, which we call the gold figure, supplied to the judge in order to determine how well the replication has been done. You also have the PDF with the redacted figure.

We do that for every single figure that’s a plot in the main text of the paper. We focus on the main text, again, somewhat to stack the odds in our favor of getting things that are replicable, so that we can, for the moment, dodge this question of whether the result was replicable or not. What we find is that, for most papers, there are 1 or 2 figures that work. For some papers, there are up to 13 figures that work. Then we assemble that all into a dataset.

In order to get the reward function, what we found worked really well is generating a per-task judge rubric. A rubric is a mark scheme. It’s like the kind of thing you would give to an examiner who was looking at your work at school. It tells you, “Okay, you should reward the agent for scientific rigor. You should reward the agent for visual fidelity. You should reward the agent for claim reproduction,” and so on and so forth.

What we found is that by having a per-task rubric—by having an intermediate stage where we adapt the rubric to the particular task and then use that consistently for that task for the entirety of training—we’re able to achieve 2 things: better agreement with human raters and much less noise.

That brings me on to something that you inquired about earlier: the question of GRPO. One of the key achievements in the paper was that we got GRPO to work. But that wasn’t without its difficulties. We went through a period we called the RL crisis, where just nothing worked. I know from talking to people at other companies that they’ve had their RL crises, and I expect that we’ll have RL crises in the future.

Part of the difficulty here is exactly because we’re doing RL on non-verifiable tasks. You have non-verifiable tasks and an LLM as a reward. An LLM is a stochastic generative model, so it has inherent noise. Now you have to deal with the fact that, from rollout to rollout, the same kinds of behavior can be judged differently.

We had another problem, which is that these are long-horizon tasks. These tasks last at least an hour in the final stage of training. We’re interested in multi-turn behavior. There are many different things that our 27B-parameter model can do. It can use any kind of Unix utility that it has on the system. It can use Codex as a tool. It can interact with the internet. It can download things from the internet into the container. So it really has an action space that’s almost equivalent to a human’s.

That combination—multi-turn behavior, a 1-hour time period, and noisy rewards—tends to mean that GRPO goes well for a while and then collapses. We did a couple of things. We did many things, and we distilled them down to a couple that worked.

One thing is a very basic modification: we score the rollout multiple times with the same judge, and we take the average of that. It’s a relatively standard technique. The other piece, I think, is quite new: we do per-turn credit assignment, and we do that in a slightly intricate way.

In addition to producing the rollout-level score, the judge says, “Okay, for every turn of the agent during this rollout, how much weight would you attribute to that?” This is a distribution. There are positive numbers, and they all sum to 1. We then normalize that weight so that we’re not changing the overall distribution according to the number of tokens in each turn. We wouldn’t want these weights to really magnify the rollout compared to all the other rollouts just because you had a very long turn. We do this normalization step, and then we use these weights to adjust the advantages during GRPO.

What we’re really saying is that when you’re upweighting or downweighting the behaviors, we want to do that on a turn level during GRPO rather than on a rollout level.

If we look at the way the weights work, we can do a little interpretability on this. What you find is that the judge ends up assigning more weight to turns that are in the middle of the rollout, because this is the load-bearing stuff. If you were to anthropomorphize this, it’s very relatable from a human perspective. You start your work, and the first bit is routine. You’re just trying to get into the swing of things. At some point, you get into flow, and you’re really making the important decisions. Then, towards the deadline, hopefully, if you’re going to meet the deadline, it’s just crossing the T’s and dotting the I’s.

The other thing is that there’s more weight assigned to turns where the 27B model is prompting the Codex model, and that’s because decisions about what you ask the Codex model to do are very important. That’s the really load-bearing stuff. We found that this combination of these 2 pieces, combined with our rubric judge, really enabled us to get stable training.

Actually, in the end, we stopped training just because we wanted to put a paper out. We didn’t stop training because we were in a collapse regime.

Yeah, the way I intuit that is: by going turn-based, what you’re doing is putting these gradient updates in where there is signal, and not where there is noise. One school of thought is, “Oh, it should be based on an entire rollout.” But does that then mean, intuitively, that you kind of have black holes in some parts of the state-action space, so then you’re just relying on Qwen’s default behavior, and you’re not updating those parts of the trajectory?

I don’t think it’s that we tend to put—well, the judge doesn’t tend to put—zero weight on parts of the trajectory. It could in principle, but we find it’s more just a change in the distribution of weighting. What that means is that there are parts of Qwen’s behavior that we’re doing a lot to change, and then parts of Qwen’s behavior that we’re doing a little bit to change on each step.

It turns out that what that does is buy us stability. There are some things that Qwen is actually pretty good at doing. If you ask it to read a PDF, for example, it is good at doing that. It can do that straight away. If you’re doing uniform credit assignment, then you’re updating. You’re saying, “Okay, great, you read the PDF. You’re doing that every single rollout.” You really don’t need to do that.

What you really need to upweight are the pieces where it was actually genuinely something different and interesting that led to the better performance of this rollout versus the other rollouts in the group. That’s what this is achieving.

So, a lot of folks at the moment, like Gary Marcus, are claiming victory for neurosymbolic AI, and they’re pointing to all of the insane harness engineering that’s going on.

You know, there was that Prime Intellect harness that came out the other day, and I don't know what to believe anymore. So you've gone an interesting way because you're using GRPO and RL, and it's actually very innovative. I think it's amazing.

But what a lot of other people would have done is they would have just adapted—you know, they would have come up with a harness, and they would make the harness do library learning and skill transfer. Do you see what I mean? Did you consider that as an option?

Yes, absolutely. In some ways, this paper is exactly a reaction to that. We very deliberately are not constructing harnesses in this work. My co-founder, Lewis Kirsch, did a very interesting analysis of AI Scientist works that are based on harnesses and the capabilities of the base models.

In that analysis, which he presented at a workshop a few months ago, he found that around 3 months after you've built the harness, the base model can do the thing the harness could do.

Yes.

We wanted to explore a different paradigm that may also be true for our coding-agent-as-tool paradigm. We'll see, but we at least wanted to assess something different.

One reason why you might want to use our paradigm rather than a harness is that history teaches us, I think, that when you have these capabilities in the weights of a model, they're more flexible and generalizable than when you have them hard-coded into a harness. When you have them in a harness, however, it's not that all harnesses are bad. A harness is likely more sample-efficient than having those capabilities in the weights, so it depends on what trade-off you want.

If you were doing something like AlphaEvolve, where you have a very specific problem—how do we do 4-by-4 complex matrix multiplication more efficiently?—then building a harness may well be the best thing you can do. That's neurosymbolic AI for solving specific problems.

You get this in all of these wonderful papers: things like AlphaEvolve, and also things like the Darwin Gödel Machine or HyperAgents from Jenny Zhang. They're all doing harness engineering, and they're great at solving specific problems.

But what we've observed is that this doesn't tend to generalize. What we wanted to do is build a system that you can then apply to a completely different problem—a quite difficult, long-horizon problem—which is to replicate a paper in a completely different area of ML research. We believe that doing that by changing the weights of the model is going to work better.

I think that these approaches could, in fact, be combined. There's a wonderful paper, I think it's called EvoTune, by one of our research fellows, Ania Tsirina. She wrote this last year. What she does is harness engineering, and then she has RL on top of that.

You can think of this by analogy with AlphaGo. AlphaGo had search, which in some ways is this kind of symbolic piece, and there was also the neural piece of RL distilling this into the weights. One of the areas that we're very excited to look at next is how you could use harnesses at training time to boost the performance within a rollout of the agent, and then distill that back into the weights so that you get the best of both worlds. You still get the flexibility and generalizability of having a neural model that operates at test time.

And a quick aside: it was good that you preemptively cited Jürgen Schmidhuber, just to prevent any turbulence downstream. I'm only joking, of course.

I've always been a neurosymbolic guy, and I've always felt that there's something very powerful about symbolic constraints. They're incredibly powerful. One school of thought in AI is that the AGI that we build—the intelligence—would have to be symbolic. What we're starting to see now is that, yes, the symbolic stuff is important, but it can actually be recrystallized back into the model.

Mm-hmm.

Right? So it's useful as a tool for generating data, and we can bring it back into the model. The exception, I think, is that sometimes we need to crystallize specific skills, as you were just saying, that clearly are better if they're in symbolic land.

Mm-hmm.

But if we're talking about pure creativity, do you think it's strictly better that eventually we move those representations back into the model?

No, I think it's a combination. I think there will be things that sit in the model and things that sit outside the model, and I don't have a strong prior on what those things will be. I'm not sure it's possible to have a strong prior about what those things will be.

Where we sit now, we can much more clearly see how this kind of system might work and might lift itself up by its own bootstraps than we could before. It's good that you mention Jürgen. Of course, he saw this right back in 1988. I believe it was his master's thesis, and then, of course, through much work after that.

I see the construction of the symbolic pieces as something that may itself be done by the AI Scientist system. You could imagine an AI Scientist system building a special-purpose model, which might itself be a neural model. It might be a skill. It might be some combination of a skill, a harness, and a neural model that it can then use as a tool.

This paradigm of coding agent as tool is the tip of a very large iceberg, which ends up with a large number of different agents, a large number of different skills, a large number of different neural models, and a large number of different symbolic pieces of equipment that are very sample-efficient, all interacting together and also interacting with humans.

And what might the evolution of this system be? At the moment, it's a single coding agent, but I could imagine there could be a swarm of agents. We could potentially use a larger model to do fine-tuning. At the moment, it's quite exciting for folks at home because it feels like, you know, I could do this.

Mm-hmm.

It's actually a really powerful thing. But if you want to build the really, really powerful version of this, would you take, let's say, a 200-billion-parameter model and do the same thing?

Mm-hmm. I think there's one dimension that we're very interested in, which is scale, and we would like to see whether we can get scaling laws from this kind of approach.

Of course, part of the philosophy is that we would have a smaller model controlling a larger model, so there's some limit on how big you would want to make the smaller model controlling the larger model if you were to do that scaling.

Another dimension is, as you say, to scale the number of agents that are interacting with each other and with humans. In fact, we're very interested to hear from people who might be interested in working with us to see how they could use this model in their work.

At the moment, we don't think that the model is general or reliable enough for an arbitrary release to the world. But if there are people with creative ideas about how this could accelerate their work, that's something that we want to discover, because of course that will inform the kinds of interfaces that we need to build between the agents and the humans, and between agents and other agents.

10. The Recursive Company

However, perhaps the most interesting and unusual thing is how we plan to use this within Inherent as a company. Our mission, as I said, is to recursively self-improve and to discover new knowledge. We think of recursive self-improvement quite differently from most other organizations doing this. We think of this as a phenomenon at a company level.

What that means is we're continually trying to close loops and put agents at the very heart of everything we do. That starts with giving agents all the same affordances and context as humans. It also means that the way we as humans work is that we proactively adopt agents as quickly as possible.

Already, we're starting to use the Faraday model internally for the kinds of replication that we might want to do to advance our research. We're also trying to learn from the way that we're using the model in order to accelerate both the construction of the model and the future research success of the company.

In addition to having that neural model, we're accumulating a huge amount of context, whether that's the code that we write or context about how we run the company of various types. What we've found is that once you get to a certain amount of context, and once you give the agents a certain number of affordances, you really reach this Rubicon moment.

For the first few months as we ran the company, we had these agents proactively reaching out to us and trying to help us with stuff. To be perfectly honest with you, it was quite annoying. They just had no idea how to help us.

But at some point about 2 or 3 months ago, I think we reached a phase transition where the agents were aware enough of the company context, and they had enough affordances to do useful stuff in the company, that the things they were doing proactively became genuinely useful. And of course, now you have a lever that you can pull to scale the number of agents and then to scale the rate at which you can do useful stuff in the company. And so that's what we mean by the recursive company. The company itself, as a whole, is self-improving as a function of the interactions of agents and humans, rather than building a single agent that will somehow magically do this in its own isolated box.

Yeah, I've experienced the same thing. I've got a skill surface and a memory system, and there was a phase change: it's now incredibly coherent for the types of things I do. So I have a very different experience, probably, from most people using AI because it's so good for me.

The problem is my skill surface and memory system are spaghetti. It's very supervised. It's very specialized to me. It would be useless in a large organization. You've built a general-purpose system, which presumably could be the future of how we use agents and could, in principle, ingest trajectories from how everyone in the organization is using AI. So there's a virtuous cycle where everyone has the new version of the small agent driving the bigger agent. And this is almost the dream of open-endedness, isn't it?

Mm-hmm.

Because now organizations can evolve their own agentic systems that are coherent for them.

Indeed, and I think that is the next era. It's the era of collective intelligence rather than the era of individual intelligence. The way that most people use agents at the moment is one-to-one, right? Sometimes it's one-to-many. I might have multiple agents running at once. But at Inherent, we think that that is a somewhat impoverished way of using agents. In fact, if human society only operated by one-to-one relationships, that would really slow down the rate at which we could do cultural evolution.

We see the problem of using agents and developing collective intelligence as many-to-many. How can we build surfaces that enable many humans to be collaborating with many agents and intervening in a very organic way at different points in the cycle? I think that extends also to the physical world. It's rather remarkable that the way that most of us do our work in an office looks the same as it did in the late 1980s. People turn up and sit down at individual computer screens.

This is despite the fact that we now have models that can go off independently and do a huge amount of work and can be scaled to very large numbers. I don't think it can possibly be the right optimum. There are organizations that have done very small experiments, in the grand scheme of things. Valve, famously, has desks on wheels, and they attribute a large part of their success to this idea.

What would it mean to really develop the next-generation organization that is evolving itself, but not just doing it in the digital space or even just in the interface between many humans and many agents, but also in the physical space of the laboratory or the office itself?

One thing that fascinates me is what the topology of this would look like, because Kenneth Stanley's always talking about committee meetings and objectives—the tyranny of objectives. I could imagine a blended approach where everyone has their own agent that's adaptively learning like this.

Mm-hmm.

Then maybe they choose to share certain streams of data within domains with shared agents in the organization. Maybe at the organization level, there's a big agent. You can just imagine that there are potential pitfalls here, because all sorts of bad behaviors and good behaviors might emerge. How do you see that panning out?

Yeah. Well, I think for us, it's all about what we call living within the experiment. We have to do a lot of experimentation, and I think that it's really an unknown unknown. Coming in with any particular worldview about the hierarchy or the structure is likely not to pan out exactly in that way. We've got priors, and we're trying various things, but we haven't solved it yet.

I want to give you a historical analogy to this, which I think is quite instructive. It's to go back all the way to the Industrial Revolution and the invention of the electric dynamo, which happened in the 1890s. What happened when the electric dynamo was invented is that factory owners were able to replace their big steam-powered turbines with electric dynamos, and they were more efficient. This gave you a kind of small productivity boost.

The problem was that the factory was configured for this single source of power: the big steam-powered turbine. What that meant is that the factory had all of these systems of complicated ratchets and pulleys that then powered the different machines in the factory, and that made it very inefficient. If there was a power failure, the entire factory had to shut down. No one could make any progress.

It was also massively unsafe because you had to build quite tall and narrow factories to accommodate all these pulleys and shafts. They tended to be very dark and very difficult to operate in, making it very difficult to work as a human in these places. What unlocked the really extraordinary productivity gains, and also unlocked all sorts of products that would have been inconceivable in the existing factories of the day, was when people reconfigured the whole factory.

That reconfiguration meant putting individual dynamos at individual workstations and then inventing the production line. This had various benefits. First of all, you can get rid of all the pulleys and shafts. Second of all, if you have one dynamo that fails, the rest of the production line can continue, so you don't have a single point of failure. But thirdly, and perhaps most importantly, it improved the quality of life and the safety of the factories, because now you could arrange them horizontally. You could put skylights in, so there was natural light, and you had a much safer and better working environment.

I think that analogy holds now with what we're trying to do at Inherent. How do we reinvent the factory for AI research from the ground up to put AI agents at the center?

Yeah, I was interviewing César Hidalgo. He wrote a book called “The Laws of Knowledge,” and he was citing an example from Jeff Bezos. He said he wasn't worried about Barnes & Noble competing with Amazon when they started selling books because they have the wrong structure. They would need to do structural adaptation.

This is part of the reason why Kenneth talks about diversity preservation rather than just diversity, because you actually need to keep multiple options open to adapt, to change your structure. Is this something that organizations need to wrestle with? Because if you think about it, there's something good about organizations having a clear purpose and coherence. But by the same token, there might be some optimal configuration. I think a lot of people just intuitively think that some kind of decentralization is good.

Mm-hmm.

Because when they discover a new strategy, shouldn't they be able to adapt themselves to start doing that instead?

Yeah. I think certainly the evolution of organizational design is important. New technologies tend to beget new forms that can make better use of them. I push back against the idea that there's an optimal configuration because, of course, technology is always changing, and that will change the organization.

One advantage of building a new organization is that you can do many more experiments and move much quicker. To give you a very precise example of that, one of the most famous inventions at Google, from an organizational point of view, was the OKR: objectives and key results. That really powered a lot of the success of the company, and I spent almost 9 years at the company and made very good use of those.

It's a goal-directed process that identifies the goals up front and then works toward those goals as measured by the key results, a measurable, quantitative metric. But of course, to some extent, that goes against the philosophy of open-endedness, at least if you make the time period for those goals too long. If you take a more open-ended view, you would want those goals to be able to adapt and change within that time period, so that you can take different stepping stones, perhaps ones that were unexpected.

And so one thing that we're now trying to figure out at Inherent is whether we can invent the next organizational paradigm, the one that is based not around optimization like OKRs, but around open-endedness. What's the equivalent of OKRs for the age of recursive self-improvement?

Yeah. Of course, you were at Google DeepMind before me. We don't need to go into too many details here. But do you imagine a future where this is basically a revolution—

Mm-hmm.

And the incumbents won't be able to adapt fast enough? Or do you think that—because it's interesting, isn't it?—all of these large companies are implementing AI agents.

Mm-hmm.

But you're making a new company, which means you can create the structure de novo. So you can adapt to meet the situation. Do you think in 5 years' time we're just going to see entirely new companies that are doing it differently, or do you think these old guys can adapt?

Look, I wouldn't have started a new company unless I thought that we had a chance of doing something significantly different and that would be really revolutionary in terms of the speed at which we're able to create new capabilities. I think the existing companies have verticals in which they are already going to be incredibly successful, and will continue to be successful.

But I do think that there is an emerging market of AI science and scientific discovery. We know that growth is powered by innovation, and we also know that ideas are getting harder to find. There's a wonderful paper with exactly that title. Whether you measure that by researcher productivity or some other metric, I believe there will be this new market of AI-assisted, AI-accelerated R&D, and I think that'll be hugely beneficial to the world because, as we know, innovation powers growth.

But ideas are getting harder to find, and there's a great paper. Nicholas Bloom wrote this wonderful paper a few years ago with exactly that title. Whether you measure this by researcher productivity, whether you measure this by the number of inflation-adjusted billion dollars that are needed to develop a new drug, or even if you measure this by the age of the Nobel Prize winner when they make their discovery, all of those are going in the wrong direction.

And I think it's sort of obvious why. The reason for this is that we have what's called a burden of knowledge. We're the victims of our success as a species. We're accumulating so much knowledge that the time and effort it takes to get to the frontier of any given domain is just so large that you now can't have individuals who know enough across domains.

And so this is the promise of building a horizontal layer of intelligence across all of science. The way we see ourselves fitting in is: can we be that intelligence layer that can power the next generation of autonomous labs, the next generation of R&D organizations, and even the next generation of company construction to solve really, really difficult problems?

And that's a somewhat different kind of market from the market for coding agents. It's a different market from the market for, say, the use of AI for existing corporates. It's a different market from the market for chatbots.

And so I do think that there is an opportunity to come in and define what is meant by the culture for that market, what's meant by the interfaces between humans and agents in that market, and how do we really power the next generation of growth for humanity?

Ed, this has been absolutely fantastic. Thank you so much for joining us today.

It's been lovely to chat to you. Thank you very much for having me.