Eiso Kant (CTO poolside) - Superhuman Coding Is Coming!

You will never hear me argue against scale. The narrative at the time in the world was, we're just gonna scale up, you know, the next GPT model, make it 10X larger. This was such a strong narrative that frankly, many people at the time in the world, you know, believed this. We didn't. There was a missing axis of scaling that wasn't being discussed, and it's frankly why we started this company. And it was the axis of scaling for the use of reinforcement learning. Scaling of next token prediction is the equivalent of imitation learning. Scaling of reinforcement learning is the equivalent of trial and error learning. The reason we built this company is because we saw a future that I personally think now is 18 to 36 months away, where human-level intelligence across the vast majority of knowledge work is achieved. You don't get to do that unless you build from the ground up. You don't fine-tune your way to AGI.

I've just had the most amazing conversation with Aizo Kant. He is the co-founder and CTO of Poolside AI. Now, they are building frontier language models. They're one of about seven or eight companies in the world who have the technical chops to build foundation models from scratch. They have a really cool solution for doing generative AI coding. Honestly, it's now possible to write software about 10 times faster than we did before. Now, what these guys have done is they use reinforcement learning from code execution feedback, which means they're going one step further in the stack to align the language models that they build to the code and the software that you are writing. There's so much low-hanging fruit in this space at the moment. When are we gonna have code solutions that can watch your screen, that are multimodal, that help you collaborate better with developers? Aizo has a very interesting story to tell about all of this. We also had some pretty cool galaxy brain conversations about how to train foundation models, about, um, test time computation and thinking and reasoning. I think there's a lot for, you know, quite a few people to get their teeth into in this conversation. MLST is sponsored by Tufa AI Labs. Now, they are the DeepSeek based in Switzerland. They have an amazing team. You've seen many of the folks on the team. They acquired MindsEye, of course. They did a lot of great work on Arc. They're now working on o1-style models and reasoning and thinking and test time computation. The reason you wanna work for them is you get loads of autonomy, you get visibility, you can publish your research. And also, they are hiring, as well as ML engineers, they're hiring a chief scientist. They really, really want to find the best possible person for this role, and they're prepared to pay top dollar as, as a joining bonus. So if you're interested in working for them as an ML engineer or their chief scientist, get in touch with Benjamin Cruzier. Go to tufalabs.ai and, uh, see what happens. Aizo, it's, it's, uh, an honor to have you on MLST. Thank you so much for joining us today.

No, thank you so much for having me. I appreciate it.

Can you tell us a little bit about yourself and Poolside?

Personally, I’m a computer geek. I started programming when I was quite young. In 2016, I found myself building what I believe to be the world’s first company that focused on making AI capable of writing code. That’s how I met my co-founder as well, actually. It’s a longer story.

In April ’23, we founded Poolside. Poolside was really founded on our view that the world was going to achieve human-level intelligence in AI. We took our own point of view on how to get there, and that’s been the fundamental start of us for almost 2 years now.

And your co-founder is Jason?

Correct, yeah. Jason and I met because, in 2017, he was the CTO of GitHub. I’m not sure I’ve ever said this publicly, or at least not on camera, but he actually made an acquisition offer for that company I was building, called Soured. Back in the day, we had the world’s first models that were able to do code completion and things like that.

I turned down the acquisition offer, but nonetheless, we became really good friends.

So tell us a little bit more about Poolside. What’s the main goal?

1. Poolside Pursues Human Level Intelligence

Poolside’s main goal comes down to wanting to build a world where we have human-level intelligence that we can scale up on compute. We think it’s essentially going to have 2 ways of having an impact. One is that the more we can make capable intelligence scalable, the more we can start driving the cost of goods and services down to zero.

On the other hand, there’s this entire frontier of technology and scientific progress that’s ahead of us, and by definition, it’s infinite, right? We will always continue to find more. Being able to pull that closer in time has always been our mission.

But we took a slightly different path than others. We took the path of focusing on making AI incredibly capable of building software. We laid out this 3-step plan on our website on day 0. It’s still there at the bottom of the footer if you click on “Vision.” It said, “Make AI capable of assisting developers in building software,” step 1; step 2, allow anyone in the world to build software; and step 3, generalize it to all other fields and domains.

So there’s a kind of winner-take-all dynamic in the space at the moment. There are amazing frontier models out there. I’ve been playing with Claude 3.7 Sonnet’s thinking mode and whatnot, and there’s always the question of differentiation. Anthropic released this Claude CLI thing, and you can just put it on your repo. My God, it’s really, really good. How can you differentiate on top of that?

2. The Frontier Model Race

I think all of us at the frontier are constantly competing with each other for model capabilities. In the fullness of time, on capabilities like software development, we might all end up in the same place. But if you look at where the world is right now, it’s a very small number of companies that are actually competing there.

I would say that we have the old guard, like Google. We have the first generation of AI companies, OpenAI and Anthropic. Then you have the second generation: xAI, Poolside, and Mistral. We were all founded around April or May ’23, and I think all of us are in that same race.

Now, we’ve decided not to focus on making our models generally available for every possible use case, but really to make them available for software development. This allows us a set of liberties in terms of where we focus and the areas we don’t focus on.

But don’t be mistaken: the work that we do to build really capable foundation models still lends itself to building really, really capable models across the board. Software development is not about writing code. You need to understand the world. You need to be able to do multistep, complex reasoning. You need to be able to plan across long objectives.

I was super excited by what Anthropic did with Claude 3.7 Sonnet. I think it’s an amazing model, and of course, it’s up to us to make sure that we then surpass it. We’re constantly in this race with each other, but we take certain views on our research and our approaches that we think will allow us to accelerate over time toward those goals.

I’m trying to understand this, because there seems to be a dichotomy between having really general foundation models that can do lots and lots of things versus the story from many people in the space who are saying, “We need customization. We need personalization. We need on-premises deployments,” and so on. How do those 2 worlds come together?

3. General Models Meet Custom Deployment

I think it’s a spectrum. Absolutely, in the first part of training your models, you want to embed as much diversity and knowledge of the world as possible. Software development is not about writing code. It’s about being able to interact with the real world and turn that into a digital form. Having that general part down is incredibly critical.

I might not care as much about how humorous my model is or how well it can handle nuances of comedy, but I absolutely care about the knowledge that it has in many different domains. What ends up happening, though, is that all of us have a fixed parameter space. At the end of the day, there’s a cost of inference, so there are only so many parameters that I can load up and then actually run.

Having a fixed parameter space means that you have to choose what you want to do with those parameters. We try to shift the distribution of our model capabilities very much toward software development in terms of the capabilities that they have. That also means that we’re willing to trade off not being as good at writing a creative bedtime story, writing comedy, or areas that you’d probably find more in a consumer AI from other people.

But you mentioned a second part, which is this notion of customization and where you deploy. I think this really comes down to how you view where models are going to be in the coming years. All of us at the frontier have a responsibility to build the world’s most capable models that can interact across all of science, technology, and knowledge work.

Even if we focus on their abilities to build software, we still care about all of that. Over time, in our step 3, we want to branch off into those other areas as well. But if you look at what’s going to happen in the future, there’s a big question: Do we have some all-powerful model that is static? It’s 1 model, and we all use it to do economically valuable work.

Or do we have all-powerful models that are able to become versions of themselves, deployed in environments and learning from the data in those environments? The question is, do we have a software developer that can write all software, or do we have one that's deployed inside a banking environment, has true access to all the information, and learns over time from it?

It's a little bit about whether we anthropomorphize it. Is it going to be like a human? I can be a very capable software developer, but you deploy me at a bank, and all of a sudden I'm going to have to learn everything over time from that bank and its embedded knowledge. Or am I something generalized that gets applied to it?

I think the honest answer is that we don't know yet in this space. What I do know is that today, when the models are not yet at human-level capabilities and are not even at superhuman-level capabilities yet, it's very valuable to give the model access to as much data, context, and ability to learn in an environment as possible.

So we just look for the shortest path toward doing that. The shortest path with enterprises is to be willing to deploy the model, the context-intelligence layer, and the applications behind their firewall, close to the data. It's a tactical, in-time decision. I think over time the form factor of that might change, but it's something that we've seen resonate really well with the customers we have.

It's interesting because I'm trying to tease out your view on scaling. There are many folks who think we should scale the models up. GPT-4.5 just came out, and it's interesting that Gwen and Karpathy were on Twitter basically saying, “High-vibe people”—that means smart people—“can see that this is a step up, and it's doing well in very nuanced things. But the benchmarks aren't capturing it. We need better benchmarks.”

But undeniably, at the moment, there is a gap in capability, right? So we need customization and thinking on-site, surface contact with domain verticals, in order to do well. But I think you're saying that you can imagine a world in the future where we could bring all of this data back into a huge foundation model and it would work just as well.

4. Reinforcement Learning Adds Scale

You will never hear me argue against scale. Scaling compute and scaling data are critical for us to close the gap between where models are today and where we believe they can be: at human-level intelligence and even beyond.

But that doesn't necessarily mean that the axes of scaling today are the same axes of scaling that people thought they were 2 years ago. When we started this company, the narrative at the time was that we were just going to scale up the next GPT model, make it 10X larger, provide it with more web data, and have this AGI-like moment, this human-level intelligence.

This was such a strong narrative that, frankly, many people in the world believed it at the time. We didn't, and it's not because we don't agree with scale. I think scale massively matters. But our view was that there was a missing axis of scaling that wasn't being discussed, and that's frankly why we started this company.

It was the axis of scaling the use of reinforcement learning. You mentioned Karpathy. I liked how he said it the other day: scaling next-token prediction is the equivalent of imitation learning, while scaling reinforcement learning is the equivalent of trial-and-error learning. While there's probably some nuance to that, I think it is the right way of thinking about it.

Yeah, I read this amazing book by Max Bennett called A Brief History of Intelligence, and he was basically saying that when you look at the animal kingdom, and humans in particular, you see this axis of simulation. It's the ability to imagine things or imagine experiences that you haven't directly had access to, and that creates this explosion.

Language, of course, is an even more sophisticated invention because it allows you to memetically share those simulations that you didn't actually have with other people. With reinforcement learning, you can actually try things and accumulate knowledge without needing to have direct physical experience.

I think it depends on where you apply reinforcement learning here. I tend to agree with a lot of what you said, which is that at the end of the day, what we do in thought differs from person to person. My thought is entirely language-based. It's a constant internal monologue that's so language-based that there are no visual representations or abstract concepts.

For me, I feel quite akin to language models, to put it somewhat funnily, because I see how they think and how they reason, and I can relate to it. The reason I mention this in relation to your point is that language is a way—we don't think it's the only way, but it's a way—that we can explore different possible chains of thought and different possible ways of thinking.

I agree that a lot of how my mind works, and I think how many people's minds work, is that you're looking at an objective and thinking through the different possible chains of thinking that can get you to that objective. Whether that's about writing a piece of code or something far more long-range, there are several things that we do to ensure that the objective is correct.

One is that we try to keep it consistent with the representations of knowledge that we know and build upon. If I'm reasoning or thinking through a math problem, I'm constantly consistency-checking against the knowledge representations that I've previously learned—the axioms that have to be true in math or physics or any other domain.

But then there are certain things that, no matter how much I try to keep them consistent with what I know, I actually need to do the work. The slightly flawed example, but I think a useful one, is if I want to learn Go or chess. I can read a whole bunch of textbooks, and I can play out chess games in my head to some extent. But at the end of the day, I'm still a snake eating itself in terms of my own synthetic data.

Coming into an environment where I play against someone else, whether a computer or a person, doesn't really matter, and learning from my mistakes—that's where I think it's really valuable to have some form of external feedback. My reasoning chains led to me losing this game. That's actually where I think it's really, really valuable to have some form of external feedback.

In the case of chess or Go, that external feedback is deterministic. Sometimes that external feedback can be human feedback, but it's very hard to scale, and it's not necessarily always the right type of feedback.

That's where I think reinforcement learning can live on that boundary. When we have the ability to form a reward that pushes us toward being more correct, or at least in the right direction of correctness, we can then improve the next set of thinking and thoughts—the next thinking chains that you're talking about.

Yeah, there was a great Nature paper talking about model collapse with Ilya Sutskever. In a way, it reminds me of some of these AGI-doom-type discussions. When you talk about omniscience, I don't think that's a scientific discussion. In the real world, we need to push molecules around.

It's the same thing with software. The reason these generative-AI software-coding tools are so powerful is that they can actually test the software they generate. That's really, really important: getting that signal from the real world.

5. Code Execution Grounds Learning

We're well known for our work on reinforcement learning from code-execution feedback. I started this work when we were building our first company in the space in 2016. It's a big part of what we've spoken about and what Poolside uses.

This is really the notion that if you have a very large, diverse environment—which we do—we have close to 1 million repositories that are fully containerized with their test suites, and many millions or tens of millions of revisions. We can say, “At this commit hash in this repository, I want to change this code, and then I want to execute it and see what comes back,” whether that's running tests against it, using a compiler or interpreter, or even running synthetic tests.

What this allows us to do is have a very, very large environment, because 1 million repositories represent a huge diversity of types of problems, whether that's in cryptography, a web app, or a core database kernel. It allows us to design tasks for the models to do where they can explore possible solutions and the thinking that leads to those solutions, and then learn from when they're right or when they're wrong.

Or at least I always try to be careful saying “right” or “wrong”—more correct and less wrong, right? Because that's essentially what reinforcement learning is. You're trying to push the model in a direction so that the next time around, when you're sampling its thoughts and sampling its solutions, it's slightly better. You do that enough times in training, and you can start getting to a place where you get very good.

But if you do this on a very narrow task, you get this notion of model collapse, or overfitting.

You get to a place where it's like, okay, the model can only do this now, and then it's no longer a useful generalized intelligence, right? So coding kind of sits in that spectrum: it's deterministic enough, but has enough diversity that even when you're overfitting a little bit toward it, you're still making it very much a generalized intelligence. You're not making it a task-specific thing that can only play the game of Go.

I'm fascinated by this idea of a diverse set of possible minds or different intelligences. I think that you can create intelligences through a variety of representations and with as many degrees of freedom. You made an interesting comment, actually, that you're not very visual. You think in this analytical language space.

I'm very visual. I can imagine sounds and so on in my mind. The way we write software is actually very diverse, like design pattern books and so on. Those are different analogies, different abstractions, where even Einstein, when he thought about relativity, was thinking about ripples and waves and so on.

And with software, there's the syntax, the actual way we write the code. There's the semantics, what it means, and there's the behavior space, how we actually test it. You're kind of talking to a form of AI where we're learning a hierarchy of representations, and it can flit between them dynamically.

6. Models Learn Rich Representations

It comes down to how we view these models, right? I want to really caveat this with the fact that we don't have a good, grounded scientific theory yet, or really robust interpretability, to say that what I'm about to say is anything but my best guesstimate or opinion at this point. But I think probably most people in the space will agree on the following: what's happening in these models is that we're learning extremely high-dimensional representations.

Some of those representations represent the ability to use language. These are representations that are massively interconnected with everything else. Some of the representations that we're learning are very specific pieces of knowledge. So we had a bit of conversation about this earlier, but if I take a piece of knowledge like when FDR was born, that sits in a high-dimensional space. It probably sits close to other things related to U.S. presidents and such.

If you go back to the early days of machine learning and think about Word2Vec and stuff like this, I think those are still useful mental models to have. Back then, we were talking about embeddings and representations that represented words, bags of words, or knowledge. Nowadays, with the type of models that we've been able to build, we have representations of things that are far more generalized and far more useful: the ability to use language and the ability to start doing reasoning.

I mention this because I think in the first wave of how we've trained these models, in the last few years of just scaling up next-token prediction on more and more data and larger models, we were improving the representations that represented the most overrepresented things in the data: language and knowledge. But we weren't yet able to start really improving the representations of complex reasoning, multi-step processes, the things that are required for building complex software, or the things that are required for figuring out new scientific breakthroughs and theories. And now we have access, with reinforcement learning, to really improve those.

But at the end of the day, if we had an infinite amount of data in the world that represented all of our thought processes and all of the feedback we got, then it wouldn't really matter what we used. We could just learn it with next-token prediction, right? We wouldn't necessarily need to use reinforcement learning. These are all just ways for us to improve the data, and hence improve the intelligence, and we're trying to do that in the most compute-efficient manner.

My team gets very tired of me saying this, but I would say all the work we do is either improving compute efficiency on training or inference, or it's improving the data, and hence the intelligence of the model. Everything you do can fit into one of those buckets. I know it's an oversimplification, and there's always a little corner case here and there, but when we come up with the crazy new architecture for linear attention, which is something we've put a lot of work into, to me, that's just improving compute efficiency of inference, right? If we figure out a really amazing way to scale up reinforcement learning and sampling, that's actually just improving the data. To me, they're just the 2 facets of model building.

Yeah, I think the economics of models is really important, actually, because even now, when OpenAI finally released o3, there was a new version of the ARC-AGI challenge that came out, and they were spending $2,500 per task. But they could solve it. They could get superhuman performance. So now it's simply a matter of computation.

But there's still a Pareto frontier, right? We've got the Gemini model. It's very, very cheap. You can just sample it maybe 50,000 times, and you can still get the answer. So then we get to this definition of intelligence and AGI. We've been talking about that a lot. François Chollet said it's basically your reasoning or your knowledge-acquisition efficiency.

How quickly do you take new points in experience space and turn them into new skill programs? And you must be wrestling with this Pareto frontier, right? You're figuring out what's the appropriate size of model, what's the appropriate architecture, and what's the trade-off between how fat the model is and how much knowledge acquisition and reasoning I do in the situation.

7. The Economics Of Scaling

I think at the end of the day, all of us at the frontier right now are taking advantage of as much computational resources as we can get, and I think that can't be ignored. If you are in the race for frontier model capabilities, the amount of computational resources that you can direct toward training is absolutely critical.

But the wrestling part comes down to where you apply it. Do you apply it to make the model larger in parameter space? Do you apply it to massive amounts of synthetic data generation? Do you apply it to scaling reinforcement learning and sampling more? All of this is essentially an equation that has an optimum for every single one of these things.

The way that we operate, and frankly, I would say most frontier labs operate, is that we try to run experiments in each of these areas and several others to find that optimum. To give you a bit of a sense, our team in January ran over 4,000 experimental runs. These happened across architecture, data ablations, weight mixes, reinforcement learning, number of samples, and all of these different variables. What we're fundamentally trying to do with all of this experimentation is understand what that optimal balance is between those things.

You said something really important: at the end of the day, cost matters. The overarching objective function of all of this is the maximum amount of intelligence that you can create within a certain amount of time and budget, which you can then serve for a certain price to the end consumer. The training and inference parts are really critical.

In our domain, because we focus on software development capabilities, we're in a quite valuable domain. It's economically valuable. I think it's a lot tougher if you're trying to serve both that domain and the free users who want to write bedtime stories, right? Because they have different economic values associated with them.

From an architectural complexity and customization point of view, the enterprise wants to design its own architecture. They want to have clear security boundaries. The software engineers in the finance department, you've got the software engineers in the legal department, and they want to create their own trade-off along those boundaries that we were just talking about. Does that make it more complex? Rather than designing 1 recipe for everyone, you're doing a lot of bespoke stuff?

I think there's the building of the foundation model, and then there's building everything around it, right? All of the software that allows you to then deploy it in different environments: if that's a workstation, a server on-premises, a VPC in a private cloud environment, or a public cloud that anyone can access.

We made a decision early on, from day 0 of the company, to say, "We are going to do everything possible to become the trusted partners of enterprises." It came from a simple analytical process. We said, "Well, where does the majority of economically valuable software development work in the world sit?" It sits in enterprises, right? 70% of all dollars in software development gets spent in enterprises.

But these are also, like you said, very complex environments with lots of security concerns and boundaries. So we took, again, a simple point of view and said, "Well, where do they want us to be? Where are we seeing that we can actually be, that the customers ask for?" What we heard over and over again is, "Bring the model to the data, not the data to the model." And so we decided to build accordingly.

And so today, we deploy our full stack—model and applications all the way on top—in these private environments. That requires a lot of work, a lot of engineering, and a lot of work that we do to streamline this so that it can scale. I personally think, in the fullness of time, everyone will end up on the cloud.

But today, if you tactically look at the Global 2000 enterprises, a lot of them are still very happy that we're able to deploy in environments that other people aren't.

So, you've gone for a kind of Tesla-esque strategy, in the sense that you're controlling the entire stack. You're building—I mean, you're one of not very many companies that have the skills and expertise to build the foundation models. But I guess the question is that there are so many folks who are just focusing down the stack. You could add so much value by doing the reinforcement learning from code-execution feedback, doing the whole architecture piece. Why did you decide to go the full hog and build the foundation models as well?

I think it started with the conversation we had today: How do we start this conversation? We started around, "What is Poolside looking to achieve?" The reason we built this company is because we saw a future that I personally think is now 18 to 36 months away, where human-level intelligence across the vast majority of knowledge work is achieved. If you held that point of view 2 years ago when we started, at the time the timelines were not that concrete for us. We would say 5 to 10 years, maybe 15.

We knew that the world was going to get to a place where we would be able to replicate our intelligence and even go beyond. When that's the point of view you hold, the question is, well, what do you need to be one of the companies that can help bring that into the world? And if you remember, we had our own point of view on how to do so in research and execution. You don't get to do that unless you build from the ground up. You don't fine-tune your way to AGI.

We put that on the website, on that same page we spoke about earlier, which was there on day 0. We had this list that was essentially called "Strong Beliefs, Weakly Held in the Face of Empirical Data" on our research. One of the things was, you need to build your foundation models from the ground up to be able to get to these things. You can't fine-tune your way to success. We said all data over time becomes synthetic, and reinforcement learning is key to being able to scale up capabilities.

So it really came from both what we wanted to achieve and what we believed mattered from a research perspective, and I think that has really played out in our favor so far. I think it will be very unlikely that we see anybody in the world get to human-level capabilities by post-training the latest open-source model. And I would even question whether we are going to have, at some point, truly open-source AGI, and whether there's going to be room for that in the world if we are continuing down a path where the capital required to build this is so incredibly large.

How hard is it to build a foundation model? Just to give you a couple of examples, DeepSeek—what's cool about them is they've made a lot of their training methods and optimizations public. They've got some great papers out there. Their new sparse-attention paper is really, really cool. I'm guessing that as you go up in scale, it gets harder and harder to train these models. But it's so difficult for people like me to know: Is it just a software-engineering challenge? How hard is it?

I'd love to break that into 2 parts. One is to actually comment a little bit on DeepSeek, because I think DeepSeek is a great example of the second-generation companies. We, xAI, and DeepSeek have taken a different approach.

DeepSeek, as far as I know right now, is about 200 people—researchers and engineers. They've got a billion dollars plus of infrastructure there, and they've got 2 years of incredible work already behind them. We know it because they've been publishing their work. To all of us in the field, they were a known entity. We've been following the papers.

I have a lot of respect for what they've done because if you look at that last paper they put out, the 47-page technical report on DeepSeek-V3, in this space we all know that if you do all of that work, you get a really good model. And I will even say, to the consternation of some, that I don't think they stole data. I don't think they did anything nefarious. I think they actually just did great work, and we've got a 2-year track record of research papers to follow to see that.

Now, there are questions that we need to have about whether we want capable AI deployed in the world, in the West, where we might not share the same values or principles as the CCP has. But that's a completely separate discussion.

The other notion is that the process of 2 years of building—what we have gone through, what xAI has gone through, and what DeepSeek has gone through—matters. Yes, of course, the models get larger and the engineering gets more complex. But at the same time, I think it's building up compounded advantages over time.

I think talent is so critical in our space. We have this incredible team, and if I look at what we have learned together as we've grown over the last 2 years, there's an immense amount of value in that. Of course, that needs to go hand in hand with actually improving your data constantly. Every month you can look back and say your data, on which you're training, is better, cleaner, and more representative of what you're trying to achieve.

It's improving your actual distributed-training stack, either by making it more compute-efficient, which is a big part of it, but also allowing it to work at scales where it wouldn't have worked before. This is all of the work that you've seen on the different types of parallelism that you get as you scale up into larger and larger clusters.

There's work that you do that's really specific to how the chips underneath change. We've got the Hopper series, but now if you look at what's coming out with the GB300s, all of a sudden we have 72 chips with an NVL connection. That changes the architecture that you want to run if you're training on that.

People often take the view that the architecture comes first, and then you map it to the hardware. It's actually the opposite. You look at the hardware, and you determine what the optimal model architecture is for training and inference. So you're constantly going back.

But over time, these all just become compounded things. Our environment of code execution went from 1,000 repos to 10,000 to 100,000, and now it's close to a million, and it's going to keep growing. Some of it is engineering, but some of it is also just the implicit knowledge and experience that you gain in your organization that builds these kinds of moats for others not to be able to compete overnight with.

Is it fair to say, though, that if every time you 10× your scale, what you already know isn't enough to get you there? You have to, not go back to the drawing board, but spend a lot of capital trying a whole bunch of different things.

I think scale can use a little definition. A lot of people have talked about scale as just scaling up model size. You mentioned GPT-4.5, which I think OpenAI says is 10× more compute, which probably means maybe 5 times larger and X amount of times the data, or whatever that combination is.

Until recently, the world assumed there were 2 scaling axes: the size of the model and the size of the data. I think reinforcement learning is truly a third scaling axis right now. Of course, that is still a proxy for data, but I think it's important to call it out as a separate scaling axis, because this changes the dynamic in terms of what scale means.

Maybe you are not taking the model to 10 trillion parameters—I'm throwing out a number here. Maybe you're not taking a model that large anymore because you're finding that you can scale more efficiently by scaling up the reinforcement-learning side.

A good example of what we saw in the last couple of years was the Llama models, where you saw at some point they went from 2 trillion tokens to 15 trillion tokens. By the way, all of us in this space have been doing this for some time, and that would be referred to at the time as overtraining because it was not Chinchilla-optimal.

But Chinchilla-optimal never took into account that these models actually have an inference cost. So, theoretically, yes, a certain model size with a certain amount of data and a certain amount of reinforcement learning is the compute-optimal way of training a model.

But actually, if you have the constraint of running this in the real world—I need to serve this to a customer, and it can cost me only so many dollars per million tokens to serve, or per number of requests—then that constraint changes where I might spend my scale. The reason I mention this is that I might say, "Okay, I might train it for much longer," but it doesn't necessarily introduce a lot more engineering complexity. It might. But if I'm trying to go 10× larger, then it introduces engineering complexity.

But it also depends on what hardware I do it on. If I’m scaling up, like Elon has—I think it’s 632K Hopper-class H100 and H200 clusters that he has interlinked together—then all of a sudden, that’s a very different way of scaling than if I take the equivalent FLOPS on 100K GB200s or GB300s that are coming out. So don’t get me wrong: there’s always new engineering. To us, it doesn’t really feel like 10X from one to another, but there are major changes that we make as we find we have more axes of scaling.

Where I think we’ve been very well positioned is that 2 years ago, we already started building the company deeply around large language modeling and reinforcement learning. So this wasn’t a new thing for us. We’ve been building up incrementally over that. I think other companies all of a sudden had to bring this out from almost nowhere, and I’m pretty sure that felt like a 10X engineering project.

Can you help the audience understand a little bit more about thinking? The R1 came out, and it’s the same with Claude 3.7 Sonnet, o3, Gemini Flash Thinking, and so on. From the end-user perspective, you see these thinking tokens, and the language model is doing a form of self-prompt augmentation. There’s the tier 0 of chain-of-thought and scratchpad, and now the models are prompting themselves.

One way of thinking about it is that we train them with reinforcement learning, and it’s imbuing this process in. But the interesting thing is that you can just take 100,000 thinking traces, fine-tune a normal base language model, and through sheer dint of interpolation, you get a lot of the performance. So you can buy performance. There’s a kind of sigmoid relationship with—

With what you’re going for?

—more compute. But what’s really going on there? Is there something special about reinforcement learning in and of itself, or should we just think about it as a form of data augmentation?

8. Reasoning Models Learn To Think

I have opinions here. Some of them, I think, are already backed up with papers that are out there and other things; others are not yet. We don’t publish, and I’m saying this because I want to reference things that are publicly available to back up some of the argumentation.

Look, at the end of the day, we are updating a model based on a loss or some function that we are applying there. So yes, in the truest definition, all of it is just data. If you had infinite data and infinite reasoning traces for everything, you could learn it through next-token prediction, and it would probably be an incredible model that reached human-level capabilities.

But it is very clear that if you take 100,000 reasoning traces versus the equivalent compute spend on reinforcement learning to get there, reinforcement learning outperforms the SFT side. I think that’s just because there’s more signal in terms of what you can provide through reward than you can by providing only a sample of data. This is a trade-off, and again, it always comes down to the data and compute efficiency. If you had an extremely large amount of data, there is a path there.

What I think is happening right now in the world, where we’re seeing massive generalization from 100,000 reasoning examples, I don’t buy it. I don’t think that’s actually what I’m seeing in the models. I think even current benchmarks are able to show that. Quite often, when you look at these things, you see that a model went up a lot on a particular math benchmark, and then the reasoning traces are all very specific and closely linked to what’s happening in that benchmark.

At some point, you might grok something a little bit more generalized, but you need a lot of data for it. I think the scalable ways of improving models are more around applying reinforcement learning where possible. That being said, there might be compute trade-offs where at some point you say, “I want to use some of that SFT data either to bootstrap or versus having it learn from scratch.” I think there are places where that can be done.

But you mentioned R1. The most exciting thing that I think was finally published in our space—because none of us in the West really publish anymore—was the zero work, and the fact that you could see a model develop its thinking capabilities in coherent language without having been provided sufficient samples of what thinking looks like.

This should be the thing that blows all of our minds. It shouldn’t be the headline around the $5 million. The headline should have been, “Models are able to develop human-like thought in language that is leading to better outcomes in objectively measurable areas like math and coding capabilities.” That’s the exciting part, without having actually been aligned toward it.

Yeah, that paper blew my mind. I’m not sure if the zero was like AlphaGo Zero, where there was no human seeding. I think there was still a little bit of human seeding, but it was mostly self-play. But it was incredible, right? It learned these emergent behaviors. It would say, “Wait,” and stop, and it would reflect and reason. Many of these seemed almost like there is a natural way of doing reasoning. It was very human-aligned.

Well, I think we can’t forget that the basis of the training data is still the internet. It’s still the web. When people say, “I just want to see it perfectly learn from nothing,” it’s like, do you want all of evolution to happen overnight? We’re teaching these models based on our own image, our own data, and what we have. Otherwise, frankly, they also wouldn’t be very useful. We want them to act in our environments.

If you take a pretrained base model and see the difference between that model and one that has had no reinforcement learning applied to develop its thinking, you can see that the capability of those thoughts—the self-reflection and all of these things—is improved. I think we often use the word “emergence,” but I think it’s a spectrum. We see things improve, and all of a sudden, we now have a lever that we can pull—and we’ve had it for some time—where we can improve the thinking capabilities of these models.

By improving the thinking capabilities of the models, to our point earlier, there’s a smaller space of solutions you have to explore to get to the correct thing. The more we can make that space more and more correct in the areas we care about—mathematics, software development, and scientific theory—the more useful and valuable these models become. By the way, humans are exactly the same way. I have this massive set of learning behind me that makes me useful in a software development environment, but if tomorrow you drop me into quantum physics, I’m probably not that useful.

Yeah. Even then, though, there’s this analogical relation between flexible forms of thinking. I use the word “emergence” because it’s just surprising. It’s a surprising arrival of a capability that gives a significant uplift.

You see interesting dynamics as well. You SFT these thinking traces because I think of the thinking traces as a form of flexible thinking: they give you more degrees of freedom to operate in a particular situation of intelligence. There’s also an interesting relationship between how fat and how thin the base model is.

With o3, it’s a very thin model, and they found that it was easier to scale compute at inference time. But they hit the edge of the sigmoid faster, whereas Claude 3.7 Sonnet is a much fatter base. It’s harder to scale, but they actually have more headroom left if they continue to scale.

There’s also an interesting thing: when they fine-tuned the models, they fine-tuned a Llama 1B model. Because the Llama model was so thin, it was almost like the thinking traces couldn’t take root. You needed to have a base level of intelligence in the model you were fine-tuning onto in order for them to work.

What is thinking in this concept? Thinking is being able to explore the possible space of a solution. Right now, the reason we call these reasoning models and often not thinking models is that reasoning really is, to some extent, objective-oriented. It needs to be able to achieve something to actually be reasoning, and then follow a certain step-by-step process.

The more complex the objective, the more clearly you need a better understanding of the world and better understanding and manipulation of language. Thought, in that sense, is still something that I think is constrained by model size, absolutely. It always was—even before reasoning models, it already was. But now we have access to what’s already there and can really improve it.

Another way of thinking about this is: if I sampled a model 1,000 times with enough diversity introduced, either through temperature or a prompt or whatever I was using, and I could find the correct solution in there—the correct thinking that led to the correct solution—it means that somewhere in the model, it’s already there. We just haven’t found the perfect latent space for it.

The best case, of course, is if I can do this only with temperature. If I start allowing for more creativity and different sets of probabilities, what you fundamentally want is to be able to reward the model in such a way that, for things that have deterministic correctness, it will get them at temperature 0. Frankly, even hopefully at a temperature of 0.7, it will still get there. It will expand and open its cone of possible options, and then collapse as it gets closer to the correct solution.

When you take a very small model, you realize that the opening cone is very wide, but it can’t actually collapse into the right solution for many things. But if the solution is somewhere in that cone already, then you can try to push it toward it. That’s where, again, the name of your podcast comes into play: the machine-learning side still kind of comes true. You can still overfit a small model to a task, but you can’t get it to generalize enough.

I often think about this as compression. Models are just a massive amount of data compressed into a certain space. If the compression is too small, you lose way too much. And if the compressed representation becomes so large compared to the data that it’s lossless, then you’re not really learning anything.

I also don’t hold the view that there’s this infinitely large model world. At some point, it’s going to make more sense to say, “For this amount of compute and this amount of data that we’re applying to teaching a model, this is the optimal size.” Then we’re going to want to maybe parallelize them horizontally to try to achieve objectives. The bitter lesson: learning and search.

I interviewed the winners of the ARC Challenge, and they said that language models are sampled greedily, which means we take the next token, then the next token. Natural language is kind of messy, so there are many degrees of freedom. They were speaking to the fundamental trade-off between creative thinking and reasoning.

With the ARC Challenge, there is just one solution—not one, but there’s a very sparse space of solutions. They actually came up with a depth-first-search kind of sampling strategy. There were also some folks at DeepMind I spoke to, and they were saying that when you do reasoning, you actually want the softmax to be very precise because you want the thing. But sometimes you actually want creativity and diversity. So how do you square that circle? How do you have both?

9. Creativity Expands The Search

Well, I think there are problems where you need the creativity and diversity to get to that deterministic final solution. If there’s a single algorithm that you have to run over and over again, like addition or multiplication, then it’s a very narrow cone of things that you want.

But if you’re trying to find, say, the general theory of relativity, for Einstein to discover general relativity, given the moment in time and the axioms of truth he had around him, it required quite a wide cone of diversity and creativity. He had to be able to explore different ideas and then, of course, make sure they were consistent with what was already true in the world—to know which ones were correct and which ones were wrong.

If you had kept that diversity very small, you probably would never have discovered it. It’s the quote, “It’s a fine line between genius and madness,” and I think that applies to models in this case as well. But at the end of the day, if you have infinite creativity—if you drag the temperature of a model up—it will collapse into garbage. It won’t be something that makes sense.

So it’s our job, when we train and build models, to make that trade-off perfectly for the things we care about—the things we define as valuable intelligence. That’s actually what we’re doing when we use reinforcement learning to teach these models. We’re saying, “When you’re sampling in this part of the cone of possible thinking traces, you’re directionally correct. When you’re sampling in this part, you’re entirely wrong.”

Mm.

If you start going down the path of trying to apply calculus to a problem that doesn’t require calculus, you want that entire set of thinking traces that stem from it to no longer be something the model does when it encounters a similar situation in the future. That, I think, is the trade-off we’re constantly making.

I think human intelligence is exactly the same thing. You see this if you take someone who is very new in a field, someone very young. That’s why I also think most interesting scientific breakthroughs come from people in their 20s: they’re not constrained by a narrow cone yet. They’re willing to explore lots of different ideas.

It’s why you see people who are great at lots of different fields, like Feynman, being able to come up with very creative ideas. But they might, at times, go way too far off and not get to the right objective. So I think intelligence is always that trade-off. I don’t think we’ll ever get it perfectly right, but we can keep getting it more and more efficient.

Over time, we might be willing to say, “For a known compute budget”—which I think is the vast majority of knowledge work in the real world, whether you’re an accountant or a software developer—“we kind of know the intelligence budget.” That’s where you want that cone to be as perfectly narrow as possible, so that it can maximize economically viable work.

But when we come to unknown-compute-budget challenges in the world—solving cancer or finding the next breakthrough in materials science—we might say, “You know what? I want that exploration much wider, and I’m willing to pay for it to be much wider so that we can explore more ideas.” It comes back to AlphaGo: how many moves do you want to explore next? How much depth and breadth do you want to have?

Yeah, I love that. The space of intelligence is very gnarly, and we work as a collective intelligence. There’s a great book by Kenneth Stanley called Why Greatness Cannot Be Planned, and he basically said that monotonic objective optimization is the dumbest thing you could possibly do.

What we actually do is, through serendipity and our nose for what’s interesting, collect diverse stepping stones, and many of them lead to greatness. In an LLM context, it’s about sampling and actually taking in those diverse perspectives.

But I want to talk about software a little bit. Poolside, your company and product, is focused on intelligence. Initially, you’ve been very focused on software engineering in particular. I can speak to my own personal experience: generative AI software has revolutionized the way I write code. I’m now writing software in a month or 2 that would have taken me years to do before. It’s absolutely incredible.

What is your main objective here, and how do you see software-engineering trends changing?

10. AI Transforms Software Development

The main objective is to achieve human-level capabilities and go beyond. In a world where we have maybe 100 million-plus people building software, we want to bring that from 100 million-plus people to anyone being able to build software. And we want the people who today are at the frontier of what they’re able to do in software to be 10 or 100 times more productive.

I think this comes from what software is: software is a lever that we have on the world to bring productivity. It’s a lever to create abundance and reduce the cost of things. To me, jumping really hard on the end of that lever—putting the biggest weight on it—is putting the biggest, most capable intelligence on it. That allows all of us, together with AI, to pull that lever and drive the cost of things down. That’s always been the thinking behind this.

I think it’s important not just to talk about the future. Otherwise, you get the founder of a frontier AI company just saying, “AGI, AGI, AGI,” and I’m sure you’ve had plenty of those podcasts already.

I think it’s important to also know what you can do today. Today is about the intersection between model capabilities and limitations and a world that’s developer-led and AI-assisted. It’s about the AI capabilities and limitations, the human, and how you find the perfect intersection by adding a product on top of that.

How do you create a product that gives people the maximum personal leverage for their own productivity? Like you said, what would have taken you years, you can do now in months. That has a lot to do with the model, but it also has a lot to do with the user experience.

It has a lot to do with how you bring the right context to the model. How do you make it easier for it to find the information to be able to give you the correct answers? Some of that is external, from the web; some of that is within your codebase; some of that is within your knowledge bases.

Building really powerful assistance is critical. Today, we do that in editors and on the web, and we’ll soon have the CLI coming. This is a symbiotic relationship: as your model gets better, you can do more in the product. The product form factor changes constantly as the model gets better. You’ve already seen that: it went from code completion to chat, to now increasingly more agentic, and I think in the future, increasingly autonomous. You just have to be on that frontier constantly, playing with all of those things.

One of the things I’ve noticed, and the reason why I’m so excited in particular, is that I think it’s really good for founders like me. I’ve got a very small team, and I can rapidly iterate. I guess my process of coding has become more like I’m a reviewer. I get the language model to generate a bunch of code, I do a bunch of tests, and increasingly, rather than writing the code, I’m just reviewing it and saying, “That looks good to me. Looks good to me.” Sometimes I’m going backwards, sometimes I’m going forwards.

The thing that I want to understand is how it works when you have teams of people. The way we write software is that we have a mental model, we create these abstractions, and we have some idea of how the software should be crafted, and we share that with our colleagues. Now we’re generating code almost quicker than our ability to review it. How does that scale in teams?

In the world today that’s developer-led and AI-assisted, the question, as always, is: what requires knowledge sharing, what requires review, and what doesn’t? When you make a 1-line change to the documentation, it doesn’t require review. When you make a massive refactoring of the entire codebase that impacts every developer, you’re going to want to share that knowledge with the entire team and maybe get input.

I think AI is no different from scaling up your team, or scaling up AI in this case. It always sits at that boundary of where it’s important to make knowledge sharing happen and where it’s not. Code review has often been seen as a process you have to do to catch bugs or to do one thing or another. I’ve always thought code review has truly been about knowledge sharing first and foremost, and then, where needed, the ability to get input from other people because it touches surface areas that impact others, or that you might not know, or where you weren’t the best person to make the decision.

When all of a sudden you’re producing 10× the code or moving a lot faster, and it sounds like in the domain you’re working in, AI already feels like a very valuable partner—almost like an anthropomorphized intern. In other domains, it’s not there yet. It should be seen as adding someone to the team, and I think increasingly that’s the relationship we’re going to have with AI: it’s like adding people to our team, except instead of adding humans, we’re adding AI agents.

One way of thinking about this is that we’ve got this fairly linear software development life cycle. We have business analysis, then we do story points, then we write some code, then we do some tests, and then we approve all of those with release control and so on. One way of thinking about this is that it’s about control and aligning the code we write with our business objectives. That’s why we have all of these different gates and approvals.

What does that mean when we have increasing autonomy and, in the code-crafting process itself, we can do so many things? Does this traditional software engineering life cycle give us bottlenecks?

I think a lot of the software development life cycle collapses into the models over time, but I think it depends again on the environment. If you’re writing code for nuclear missiles, you probably want a lot of those steps of that linear process to still exist, even if AI is the one acting in it.

Mm.

You care about a certain number of nines of reliability. In that case, hopefully not even nines—just 100% reliability, which we all know is impossible in software. But we want many nines, so you’re willing to invest in that.

In other places, software becomes more ephemeral. Maybe you write something that works for a week as a tool, or it does a set of tasks, and so there the SDLC doesn’t really matter. On those two extreme ends of a spectrum—and I’m starting to realize about myself that I use spectrums a lot; it’s just the way my brain works—as AI gets more capable, we’re probably going to have a distribution of software that tilts far more to the left in terms of volume: software that’s closer to not requiring those stringent processes than software that does.

But the world’s global banking infrastructure is not going away, and there we want a certain set of checks and balances in place. The question just becomes: is AI acting across all of those checks and balances? At some point, does it become so reliable that I can start removing a lot of them?

If I have a software developer—a human software developer—who never makes a mistake, never writes a bug, and whose CI tests always pass 100% of the time for the last 5 years, at some point I might say, “You know what? Give the 3 hours of CI back to the human. Let them just move faster.” That’s a theoretical example, because in the real world it’s not perfectly like this. But as we get more and more above human-level capabilities, at some point we might just say, “You know what? That’s fine.”

Maybe even unit tests fall away. Maybe even CI—maybe all of this starts becoming less and less. I think it’s useful to think about things in terms of the limits, not because we’re at the limits tomorrow, but because it allows us to show where we’re heading.

In the fullness of time, a lot of the software development life cycle collapses into the model and doesn’t require many of these checks—but not everywhere and not all at once.

Thinking about the dynamics of software engineering in large enterprises, a lot of companies in the FTSE 500 frankly can’t hire really talented software engineers. There was always this problem that they would do low-code and no-code. They would build things on Microsoft Power Platform and so on.

Now we’re in the era of generative AI coding, and almost anyone can write amazing software—almost disposable software applications—to do whatever they want to do. Do you think we’re going to see more people writing code, and how will that change over time?

I’ll have to say, and maybe that’s because I’ve spent so much time with enterprises, that I think we find incredible software engineers everywhere, first and foremost. But I hear you in terms of what you’re saying: what you get paid to be a staff engineer at Google is not the same as what you get paid to be a staff engineer at a bank. There are some distinctions here and there. But overall, I think there are great developers everywhere.

What we’re seeing right now is that AI, while it might be at the point for you in your field or for greenfield projects where it feels like that massive unlock, still feels like a 20% or 30% productivity gain in most enterprise environments. For some people, it’s a 2× gain because they’re doing unit-test automation, and all of a sudden that’s much faster and maybe even becomes 3× or 4×. In other places, people are working with a company-specific programming language in a specific domain where the models aren’t very good yet.

It’s worth acknowledging that model capabilities mapping over to the real world in enterprises is not one-size-fits-all. What I do see is that people are excited about being able to do more. That might mean being an existing developer who can take the boring parts of the job and automate them, or build a lot more software faster. Or it might mean that a product manager says, “I can actually build this prototype myself and show it to the managers, and then see if we want to build it out at a better scale.”

I do hold the view that more and more people will be able to build software and will want to build software. But the desire is an important part. Because we’re software developers ourselves, we often assume that everybody wants to build software if they’re able to. The reality is that not everybody wants to build software, even if they can tomorrow.

But the product manager who’s always wanted to build their ideas more quickly absolutely will. Maybe 1 in 5 businesspeople has an idea they’ve been trying to rally a team around but couldn’t, and now can. But it’s not universal that everybody in the world will be building software, because it’s still something that requires you to want to do it.

Yeah, I guess I agree that clearly the lift is the 0 to 1.

Yeah.

Right? You can now build individual applications in seconds, and it's incredible.

Oh, yeah, absolutely.

But I still think that there's a much bigger lift, and the only thing blocking us is a lack of imagination. For example, the reason why Google engineers are paid so well is that they're building scalable distributed systems that millions of people use, and multi-agent, fault-resilient systems.

Yeah.

You can still build systems like that, and I guess the question is, some of this is an education thing, right? There are good ways and bad ways of using GenAI code. A good way of doing it is understanding that there is a complexity ceiling, and if you build a monolithic application and just keep building on top of it, it's going to collapse.

Yes.

But you can—it's almost like it's teasing you to design modular, almost serverless-type, multi-agent-type systems.

Interesting.

And the LLMs can handle that to a much higher complexity ceiling.

I think you're right in mapping the current limitations of models to a really good way of getting around them, which is building small, modular things that the models are able to essentially understand easily and work within. Having those kinds of architectures, where there's some separation of concerns, helps in that world. But those are the models of today.

Mm.

And I think this is an important part to always come back to: I don't think there is a universal limitation to the type of software that can be built by models when we talk about the next 3 to 5 years. But you're absolutely right in saying that if today I try to build a massive monolithic application and just have the model vibe-code the hell out of it, at some point the whole thing collapses in on itself.

By the way, I don't know about you, but there have been times early in my career when I did exactly the same thing. I would build something and build something and build something, and at some point I'd be like, “Oh, my God, what monster did I write here in code?” Then it pushed me to refactor it, make it more modular, and make it better, right? And so we're seeing limitations in the models today, while they're still far from our capabilities, that we can also reflect on having seen in ourselves at different moments.

Yeah, and that's really interesting, because it's like language models scale with quadratic complexity, and software scales exponentially in complexity. It allows you to build software that is 2 orders of magnitude more complex than you would otherwise have built, but you still hit the complexity ceiling very, very quickly.

That's an interesting way of looking at it. I haven't given that much thought, but I'll have to do so.

The other thing that I think is really interesting is that, at the moment, when people do GenAI coding, they're generating software and doing unit tests and so on. But we still have this fairly linear mode of software engineering, which means we have release control and put stuff into production.

Now we're starting to see the advent of MCP servers and so on, which means that during the development process itself, the intelligent system can actually talk to your database. It can say, “What's the schema on the live database?” Or, “Talk to my actor system. How many actors are in play at the moment? Do I need to repair this actor?” So now there's increasingly an operational layer to the software process.

I'd love to anthropomorphize this for a second, right? What we do as developers is open up the database console and check the schema. We go talk to someone on Slack or in person, gather information, and pull up the documentation. I think some of the protocols that the world is creating are a way to make that easy for the model to do, given current model capabilities.

Again, if we play this out over the next couple of years, I'm not sure if it's a protocol. I'm not sure if it's just a computer-use agent that's been doing this, or if the model is simply writing code to hit the API of Jira, or directly connecting to the database and executing a SQL command to get the schema. I think there are things that we build today that matter for the limitations the models have. They matter, and they're useful.

The interesting question that stems from this is: at some point, when we talk about a large multi-agent system, just like we talk about a large company with lots of developers collaborating, how do we make that collaboration efficient and work well? Is this a world where we have 1,000 instantiations of our model, each essentially acting in an organized collective, like a company? Is that hierarchical in nature, just like we are in our organizations?

There are things that we can't do. I can't access the thinking traces and solutions of my 500 peers. That can't be stored in some central database, but it's something agents can do. All of a sudden, I think that while we're building toward human-level intelligence, there are already things that models today can do that we can't do by the nature of how we operate.

The parallelization of being able to run through an entire code base and summarize it file by file over 1,000 files is something I can do, but it would take me a very long time and wouldn't be so efficient. Then there's the length of context windows. I don't know about you, but I can't hold 1 million tokens in my context window and retrieve perfectly across it.

At some point we're going to have to get away from this realm of anthropomorphizing it and start to say, “Okay, these are certain things that models can do differently.” That, to me, comes back to these protocols. Right now we need them; I'm not sure if we need them in 18 months.

You were speaking to something interesting there, which is that there's a kind of semantics gap in code. There was always a famous adage that many companies didn't want to release their code on GitHub because it was so valuable. It's actually not that valuable, because the semantics—the intentionality, the motivation behind the code—is not in the code.

It's the same thing with language. There's a lot of missing information that is not in the data. But we can capture this information because you've now got an organization of developers. They're building code using these tools, and the entire thought process is in language. So you could capture that into a kind of semantic database, RAG into it, and now you've got all of this meaningful, motivated information. That means the language model won't always be making the same mistake, because it knows, “Oh, the reason why we did this is because of that.”

There's lots of really cool stuff, right? We've never had the ability to trace the thinking of humans and store it. As these models are being deployed increasingly as assistants, future agents, and even autonomous agents, we can now access that. The question is, will we access it, or will the models access it themselves?

I always try to be careful about going too many steps into the future, because at some point you start collapsing into sci-fi. But I don't think we are that far out from this specific scenario. There are quite a few people who won't agree with me on this, but I think it's really important that, as long as we can, we should keep models thinking and reasoning in language.

There's incredible work that can, and probably even more compute-efficiently, be done with latent-space-style thinking. There was a great Meta paper about latent-space reasoning tokens. There have been some other approaches that I've seen that I like a lot. We saw a language-based diffusion model come out earlier this week. I really like the people behind it.

Yeah.

I think there are lots of architectures that will work. My team gets tired of me saying that probably every architecture can work in the world; it's just a question of how compute-efficient it is. So I had no doubt that diffusion language models were going to work. The question is whether it's the compute-efficient thing for the types of capabilities and tasks that we care about.

The reason I mention language being important is that, as models get more capable, having the ability to see their reasoning and thinking traces, and, like you said, reference them from the past to understand what led to certain decisions, will become massively valuable. That applies both from an interpretability perspective and from the ability to build upon work that was done previously by other agents. I think there is also a discussion to be had about whether it will be useful from a safety and alignment perspective.

This diffusion-based language model is a great example. I've been very excited about this for years, but they never worked particularly well. This one that just came out, the one you spoke about, had roughly a 10× efficiency advantage.

So, with an autoregressive language model, you actually have to generate token by token. This thing only ran diffusion about 5 or 6 times, and they got the same results. Diffusion is really good from a code point of view because, just like with vision diffusion, you can actually edit things. So you can say, “Okay, well, I want to hold this code fixed, but I want to edit this bit in the middle.” Why isn’t everyone doing this?

There are so many architectures that can work, and I think diffusion is one other form of it. But we’ve done a lot of work in our industry and in each frontier company to make the ones we have really efficient.

To shift over from one to another, you have to have the efficiency gains. You have to be willing to take the time it takes to actually go after a new architecture, and to ask whether all of the experimentation you’ve done until that moment still holds true in the same way or not.

We made a big bet a little over a year ago on linear attention, RNN-inspired attention, and we’ve had models in production since about fall of last year already with linear attention. So we would ask, “Why isn’t everybody doing this? This makes so much sense. There are these gains, et cetera.” Part of that is also just because all of us end up investing heavily in a certain area, and then we just scale it up further. Making that shift needs to be really, really valuable for us to do so.

So I need to dig further into the diffusion language models in terms of how much of that efficiency shows up compared to some of the things that we’re doing or others. We’re unlikely to go down this path ourselves because of different things that we’ve done in other architectures that we maybe just don’t disclose yet. But I think it’s exciting, and I think there’s a lot of room left for architecture research. I think the diffusion language model is the tip of the iceberg. A lot more things can be done, but they also need to be scaled up.

Some of the coolest stuff that we’ve seen in open source or from others comes at the 7-billion-parameter scale, if at all. And then the question becomes, “Well, how does it operate at 70 billion? How does it operate if we want to try to make an MoE equivalent of a diffusion language model?” At some point, what works at small scale very likely can work at large scale, but does it stay the same efficiency? Can you get those same gains?

Hmm.

Yeah. That’s a very good question because they built a midsize model, and it was commensurate in performance with the other frontier midsize models. The other thing I loved about it is that you can actually do an unbounded amount of computation, so you can just continue to do diffusion. From a test-time computation point of view, it’s actually very, very flexible. I love that.

But just coming back to the software engineering thing, I think what we’re seeing, though, when we do increasing AI development, is a lack of autonomy on the part of the humans and a lack of legibility, right? So we’re now building increasingly inscrutable software. Let’s make no bones about this: no one at Google understands how the software code works. But we shouldn’t be hyperbolic about it.

We could design an information architecture that mediates a cognitive interface, which means that, at least at some level of abstraction, we understand what the thing’s doing. We’re putting guardrails and so on. But we are describing a future where we’re just building these inscrutable monsters. What does that look like?

In terms of your first point, about models writing code, I think code is that higher-level abstraction already, right? We can all go in and try to understand code, and maybe we add some print statements and spend time. It takes cognitive load, but code is deterministic. It gets interpreted by a compiler. All of us as humans, if we’re willing to invest the time and effort, can understand it. It’s just: is it worth it, right?

Are we okay with building big software codebases that we can’t fully understand? Historically, we already have been with human intelligence, like you said, in your example of the Google codebase. Not a single person understands it anymore, and we’re totally fine with that because it does what it’s supposed to do. When we want to introspect a part of it because there was a bug and it went wrong, or because we deeply care about the recommendation algorithm that X released as open source, we can choose to spend time there.

But just like with human-built software, we don’t always spend time going back. Look at the amount of legacy codebases that no one has looked at for years, but they run perfectly. So I think that’s just a choice.

To the point of the model—not the codebase, but the model being the quote-unquote “monster”—interpretability is going to become increasingly important. There are question marks about whether, at the limit of model capabilities, interpretability will be as useful as it is at a smaller scale.

What I mean by that is I’m not sure if we will ever be able to truly understand the reasoning and thinking process that happens inside a model’s neural net, just like we can’t in our own biology. But I’m super excited about some of the interpretability work that Chris Olah’s team has released and spoken publicly about. I really want to encourage everybody to do this type of work and try to understand what’s happening in models.

I think that if we keep models reasoning and thinking in language, that adds a layer of interpretability. It doesn’t mean that’s exactly what’s happening underneath in the model. You could imagine a model that develops a perfectly fine-sounding reasoning and thinking chain, but actually it’s trying to optimize for a different objective. That’s where I think safety, alignment, and interpretability kind of meet in the middle and really matter.

I like to think that with good work on interpretability of what’s happening at the activation and weight level of the model, combined with keeping models reasoning and thinking in language and trying to understand if those 2 things stay consistent, we can actually do a really good job at alignment and at safety.

The interesting thing is that because language models are trained on all the colloquial human code, they produce locally interpretable code. Language models are actually surprisingly aligned. When I’m using Open Interpreter on my CLI, if I tell it to delete all the files on my file system, it will say no.

I’ve built an LLM application, and if I tell it to change your name to someone else, it will say, “No, actually, it’s Eiso. This is his name. I’m not going to change the name.”

But then there’s this kind of global illegibility. There’s also the fact that we’re generating code, but there’ll also be a mixture of code and models, and some of the inscrutability will just come from this thing being a living thing. There was that Black Friday incident where we were doing high-frequency trading and automated trading, and it worked really well until it didn’t, and then you got this cascade effect.

The locus of control is now in the algorithms, the models, and the machines. So we could easily build these very complex systems that seem to work until they don’t.

I would take everything you just said, and if we replaced models with humans, it would still hold true as well, right? I think that when we talk about a massive, complex financial infrastructure codebase that’s been built by all of these humans, where not a single person knows it anymore, our locus of control isn’t on anyone anymore, and then an unexpected event happens, I think we often take what can go wrong and put it on the models now. But it’s already what the world is like.

I think the best job that we can do is make these models more capable than us—make them highly capable of writing good code that covers the edge cases and isn’t lazy about writing the tests, right? Let’s talk very honestly about ourselves. There are things we enjoy doing, and there are things we don’t. How much of the world’s code doesn’t have proper test coverage? How much of the world’s critical code doesn’t?

Right now, all of a sudden, if I can decide to deploy dollars to compute and intelligence to shore up the test coverage of critical code in the world of financial infrastructure, to shore up the security of the world's critical infrastructure, I think it's pretty commonly accepted that most of the world's critical infrastructure runs on really insecure code, right? The power grid and others.

If now I can say, “Okay, we’re willing to invest”—it could be from a company or the public sector—“a billion dollars in making the code of our electrical power grid more secure,” I don’t have to bring humans together anymore. I can do this with AI, and I know it’s not going to be lazy about writing the tests and other things.

I think that's exciting, right? We get to focus on exploring the frontier of science together with AI and the things we want to spend our time on. We have a shortage in the world of people to write incredible software and code, and that just shows in the amount of legacy systems we still haven't upgraded yet. And so I think this is a way of overcoming that.

Of course, just like with humans, it adds another area of fault. But I'm not sure which one of us, in the fullness of time, you would rather have write code for a critical system. I often say privately: I want the nurse to be a human, but I want the surgeon one day to be a robot.

Just looking at the evolution of AI software, I'm very excited about metaprogramming. We can actually have the system repair itself and generate its own code in response to failures. But then the next stage of evolution is: why do we need code at all? Why don't we just—I mean, it's so-called transduction, where we don't even have an intermediate code step at all. We just get the model to do the thing, and the whole thing is adaptive. Are you excited about that?

My opinion over the years, since 2016, has changed on this. I used to agree with Karpathy's quote in his 2016 blog article—

Software 2.0.

—Software 2.0, right? Where it's like, at some limit, everything just becomes a model. I think about it a little bit differently today. It's no longer that extreme. I think there are parts of the world's infrastructure that we want to have in something that's interpretable, and code is interpretable. It can be traced, it can be tested, and it can be understood. A human or an AI can go in and understand it completely.

Our financial payment infrastructure in the world—we probably want that to be in code. Our electrical grid—we probably want that to be in code. What's operating the railroad out here, the switches between trains, is probably in code. But do I care about other pieces of software if it's just a neural net simulating the whole thing behind the scenes?

I think I also want to come back to what code is, right? Code operates, in most cases of most software—not everything—on a CPU. We have done an incredible job of optimizing the hell out of that evolution of hardware to be able to run deterministic code that can then serve us and be valuable. There are lots of places where, for a long time, the models might not even be compute-efficient enough. It might just be too expensive to have all of the software in the world collapse into a model call, right?

But you're right that if we get to a point where model capabilities are so good and so trustworthy that we can treat them as doing deterministic things, I know that every single time I ask X, it's so aligned that it's going to do Y. Because that's essentially what code for a CPU is, right? At the end of the day, we are asking for something to be truly deterministic.

Then maybe, as those cost profiles change, more and more will move to the models. But maybe I'm holding on to an old idea. I still like to think that it will be cheaper for a capable model to build and maintain the Uber Eats app and have it run on CPU infrastructure as maintainable code—to update it and change it—than it will be to simulate the entire thing as a neural net. And so I think, for a long time, code in the real world will continue to exist.

The amazing thing, though, about these AI models is that, in so many ways, they're just smarter than us. They can basically write or learn functions that we can't write code to do, which is incredible.

100%. There are so many places where this is going to be incredible.

Absolutely. Maybe it is a spectrum. I was having a bit of a galaxy-brain thought. I was driving to pick Marcus up, our creative director, this morning, and Google Maps was taking me this weird way. I was thinking to myself, “What's it doing? Is this thing being a utilitarian? Is it actually optimizing for the average—

Yep.

—route time, and it's taking me down a bad way because it doesn't care about me?”

Yeah.

I don't know. And in the future, these AI systems might be doing some weird galaxy-brain thing, and it might not necessarily be what's right for me. Maybe I want to know about that. This is kind of what I mean by the loss of control.

I think today there are a lot of machine learning systems that we already don't have any good interpretability for, whether that's fraud detection or the algorithm that's mapping you to your location. I'm not sure what's behind the scenes of Google Maps, but it could very well already be a neural net that's very task-specific. I wouldn't be surprised if at least a part of it is, likely trained on a lot of data about London traffic patterns and whatnot.

So I think we've already faced that in machine learning, and we'll continue to face it. There are parts in the world where we want things to be deterministic, in my view, and we're going to want to keep them. And there are other parts where you're absolutely right: I want the most optimal route planned from somebody who understands me, not just the general London traffic.

Maybe it's the fact that I prefer to take trains that are less busy than ones that are more busy, and I'm okay to get there at a certain time. I think as we add intelligence—generalized intelligence, human-like intelligence—to the world, we've got a lever that we can make more compute-efficient, and we might want to use it. I might want to use AGI to help me figure out my route, because at some point it's going to be so cheap that it's kind of worth it to do.

I do something today with the models. This is going to sound very silly. I travel a lot for work, and every time before I fly or land somewhere, I tell it my schedule, I tell it what I ate, and I say, “What's my best plan for dealing with my jet lag as I'm coming in on this schedule?” And then I follow it. It will tell me, “Eat a protein meal with little carbs and sleep for 90 minutes here,” and I will just follow the model.

Our model wasn't trained for this specifically, but it's a pretty good generalized model, and so far it's worked pretty well for me. I use it for that specific reason, because all of a sudden intelligence is now cheap enough for me to actually get it to do this. You could write an algorithm. You could write code that takes all of these factors into account and calls the API for location, weather, and things like that. Or you could just trust the model.

I think these two things will always exist, but the code is going to be written by the model in the future. It's choosing to do things in code because it's more efficient and deterministic, or we want it to stay deterministic, like payment infrastructure.

Do you think we might lose something when we do this? I watched a great YouTube video last night, and it was this guy saying that, in the days of analog recording, we used to have to record things to tape. We could only do one take, and there was lots of noise and so on. But it created this kind of serendipity: we couldn't just delete it and start again. We had to do it again. We had to talk about it.

Even this filming day today is a very creative, serendipitous process. Do you think that, by offloading so much of the intelligent thought process to machines, we might lose something?

It is worth asking the question: Do we think in the same ways today? Has our own thought process evolved in the same ways today as it did 100 years ago, 500 years ago, or 2,000 years ago?

I think there are things that, as we have learned more representations of knowledge and thought, have changed as we started getting augmented by tools. I'm still of the age where I remember a pre-Google and post-Google world. I was young, but I remember both worlds. Now all of a sudden I'm in a world where I'm willing to no longer remember certain facts.

Mm-hmm.

Actually, there are very few specific knowledge or history facts that I still remember. Whereas if you and I were born 150 years ago, remembering the things that we had learned from reading books and other things would be critical to us. And so that's changed. It's changed because we now have a tool that we can use instead.

I might not need to know anymore the theory behind jet lag, or the biological process, because I trust something else to make a decision. But I think we've always had versions of that. Whether that was, in the past, trusting the local wise person on X, Y, or Z, or the fact that we go to a doctor, we trust someone else.

So will our own thought process evolve? If you give enough time over the evolution of this, will we become smarter? Will we become less smart? History hasn't shown us, even though people like to say it in the moment, that we've become dumber because of technology.

I actually think it's made us more enlightened. It's made us more able to explore lots more ideas and do a lot more. Technological progress has continued to be exponential, and I think it's exciting that it continues to stay exponential. So maybe I'm an optimist.

Hmm.

But I do think there's a fair argument to be made if you're a parent, right, and you've got a young child, and you essentially say, “I still want you to study this. I want you to learn these things because it develops your own thinking.” It's kind of the TikTok debate, right? Do you want to give TikTok the entire day to your kid and have them spend 8 hours a day just on the phone? I'm not sure if that's how you want your children to grow up and develop their intelligence.

Would you encourage your kids today to learn how to code?

Yes.

Go on.

Just like you would encourage your children to learn history and to understand mathematics, you would encourage them to learn how to program and write code. I think you want all of these things because, at the end of the day, we still need to train our own intelligence as well.

Just because we're now getting increasingly capable of training models to reach our level of intelligence one day, and even beyond, that doesn't mean that our experience of life should be one without us actually training our own intelligence. I think coding and building software is a great way of training our intelligence.

Today, in my job, I don't get to write a lot of code anymore. It's just the nature of being a co-founder and where my role is. But everything that I learned from having written a lot of code in my life has helped me develop thinking processes and understanding that, I like to think, make me more capable of doing what I do.

So I think just developing intelligence in general is a good thing, and I think coding is a good tool for that.

Can we just speak about the role of multimodality? At the moment, we're primarily talking about text in this generative AI setting. I can imagine a future where it seems like low-hanging fruit: we just record the screen, and, perhaps in an observational way, it can see the application. Maybe in the future it can interact with the application in a gentle way. Is that coming?

Oh, 100%.

Yeah.

Multimodality, from vision-language modeling to computer use to the work that you see happening in video and building world models in the world, all of that is coming.

I think the question always comes down to the companies that are working on it and what their objectives are. When I talk about building the world's most capable AI for software development right now, and us being on the trajectory towards that, there are things that we care about that are massively represented in text—characters, language, code, et cetera—but also things that are massively represented in visual understanding of what's on a screen, because applications are on screens.

Building the model's understanding so that it can know what's on a screen and how to interact with it, and have computer-use-like capabilities so that agents can go and open up the Amazon Web Services console, click around, and find the data that they need, is critical.

When you focus on software development capabilities, a lot can be done with text modality. With text modality, you're not going to get the model to create beautiful UIs. You just can't get that iteration on UI without visual modeling and having that as a modality.

But where a computer-use agent is useful to go to the Amazon website, log in, and find the IAM role, I can also have a model understand how to do that by making an API call. So I think we put ourselves with a bunch of guardrails by focusing on software development capabilities that allow us to focus on modalities that are slightly more compute-efficient than video or image generation, and hence use our compute more efficiently to get towards our goal.

But if I was trying to build full self-driving, my modality would not be text.

I'm quite excited about a future. There's the Genie paper from DeepMind, for example, where they're talking about being able to generate video that's interactive in real time. Could we ever, in the future, have software which is generative? So the user interface is generative. Because we all think about things differently, right? The user interface for you—the optimal user interface—might be different for me.

I don't see a reason why that world can't exist. I think the question is, where do we want it to exist?

I want my Uber Eats app to look the same way every time I open it, because I've built up my own model to make sure that I can find where the food is that I want to order. But there are places where I want a UI to be dynamic based on maybe the data that's behind something. I may want it to be more tailored towards me.

But I venture to say that the vast majority of people who consume software in the world—us as humans—actually don't want our UIs changing massively on a day-to-day basis. We want a set of consistency in our UIs. I think that's just human behavior, but I don't think there's any technical reason why that can't happen.

Interesting. Interesting. What's your relationship with the cloud providers?

Publicly, we announced in December that we have what's called a first-party relationship with AWS, Amazon Web Services. It's quite a unique one. What it allows us is that when an enterprise customer is buying Poolside, they can buy it as if it's under Amazon's paper, meaning that they're the seller of record.

This opens up a path for large, complex enterprises to contract Poolside and bring our product in as if they're adding on another Amazon service. So this hugely reduces the time that it takes to get started with us, and also allows them to fully burn down their spend commitments. Enterprises have large spend commitments with Amazon, and this is a way that we have done this for several reasons.

One is Amazon's distribution in enterprises is massive, right? They're the largest service area of any cloud provider in the world. And if you think about our business, we are in a capabilities race. We've spoken a lot about that today. But we're also in a go-to-market race, right?

Our ability to land customers and grow revenue allows us to invest massive amounts of capital into more compute and into more talent to be able to scale up our model capabilities. So these are very symbiotic, and hence the relationship with Amazon, which has been amazingly exciting.

We've also done a lot of work with them, not just on the go-to-market side, but on their silicon.

Mm.

We've done a lot of work around Trainium 1 and Trainium 2, and so we have a fully dedicated team towards it. We're pretty excited about what they're building there.

Yeah, it's great for startups as well, because you get a bunch of free credits with Amazon, so presumably you can just use the free credits for your service.

I think we're at a scale of compute where credits are not what makes the difference anymore, to be very honest.

Yeah.

But for our customers and others, it's absolutely part of it, no doubt. Today, we're very much focused on enterprises and who we work with. What we see for them is that where you're deployed makes a big difference for their security profile.

We deploy inside Amazon Web Services VPCs, in that private account, and the model weights and the full stack live there. The combination of that really allows enterprises to get comfortable with the model that's going through their data instead of the other way around.

Interesting. And just from a getting-started point of view, folks want to have dedicated hardware. Do you tend to find that your customers have a centralized model where they have one shared implementation, or is it more complex than that?

It really depends on the enterprise. A lot of what we find is that inside enterprises there are security boundaries that mean that they need multiple instantiations of the model.

So if they're fine-tuning a model on their proprietary SDK that they have internally, but that only covers one business unit, and that model's not allowed to be shared from regulatory compliance reasons with another part of the business, you might find organizations that want to spin up many versions of Poolside models that get fine-tuned in many different environments towards certain use cases.

From where we sit, we see all types of complexity, from on-prem to VPC to different model instantiations, different security boundaries, and different onion layers of access rights. We've built for all of that.

We talked a lot about the model today, about the sheer level of engineering we've had to do to be able to succeed in defense, in government, and in financial services. It's been actually quite a lot.

And finally, do you have a forward engineering team just to help folks get up and running with the infrastructure and the architecture?

Absolutely, yeah.

So we have the Poolside solutions architects, fully able and willing to spend a lot of time with our customers. But what we have increasingly done is build towards a managed install approach, so that if an enterprise is willing to give us temporary, very limited access rights, we can spin up the entire infrastructure of Poolside in their account. What before might take several days of effort can now be done in under 40 minutes by providing a single IAM role.

But with enterprises, there's always going to be something that you discover along the way of a massive deployment—a firewall that sits on the network somewhere that needs a certain setting tweaked. That's, of course, where our solutions architects are often very helpful. But it's also about helping them think through, as we're making these models their models, as we're fine-tuning them on their data and making them more capable in their environment, what data to use, which groups of engineers to deploy which versions of models to, and how to measure the impact.

We offer quite a lot of metrics that are available to the customer to see the impact of the model in terms of acceptance rates, how many changes are reviewed but not actually applied, and how many lines of code are actually applied. We do a lot of that work, and we try to help our customers think through it. Our commitment is to become the trusted partner of enterprises, so that as intelligence gets more capable, they'll want to scale it up with us.

Eiso, it's been an absolute honor. Thank you so much for joining us today.

No, thank you so much. It was a pleasure.