开源模型能否跟上 OpenAI 和 Anthropic？｜Nathan Lambert

When I think about model progress, it comes down to different forms of capital: compute, data, and talent.

How do you see that playing out in terms of who’s going to win in the next couple of years?

The Chinese are so good at building models. To say that China is anything but equal on talent, you’re going to have a hard time. What they don’t have is compute.

Most people regard distillation as the main lever for them to do this.

People distill my products every single day. They take it, they strip it down to its raw parts. That’s called competition.

Previous post-training was much more reliant on supervised fine-tuning. Now this has morphed a bit into environments where it’s RL-heavy. But it’s hard to map this type of post-training, where you get the completions and the score goes up, to what is going on in reasoning today.

There’s so much fear in the discourse now about how much the frontier is hidden and also breaking away.

The AI capability plot doesn’t necessarily worry me, but the AI capability plot and the pretty much nonexistent preparation for public cybersecurity and all of these things, like—

So, Nathan, really happy to have you here. Thank you for joining in person. It’s super timely, given you’re launching Trillium, which I’d love to talk about. But more on a macro level: I think we’ve chatted a lot about open source, and that’s a lot of what you care about. What we’ve been hearing a lot in the public discourse is, “Hey, we have to pace the frontier,” you know, the frontier this, the frontier that. And an arc of the stuff that I’m really trying to ask experts about is: What is in the frontier? What is it, even? What are we pacing? What’s there?

And then, more importantly, threading it with open-source and open-weight models, can they keep up with the frontier? I think this matters very much and is something that you’re an expert on. So, to start on that arc, how would you break down what’s gotten us to the frontier for both closed labs and open-source models? How does it break down in your mind, maybe first at a higher level of pre-training, mid-training, post-training, and how post-training has evolved, which is something you’ve done a great amount of education to the public on?

When I think about model progress, it comes down to different forms of capital that you put into them: compute, data, and talent. All of the labs have various amounts of this.

Yeah.

The closed frontier labs—OpenAI, Anthropic, and Google—have clearly had the most of these. And what we see—

Yeah.

It’s the different abilities of the labs to organize these forms of capital. Making the org chart work for building models is really hard, which is why companies have to do complete resets from the bottom up. And then it actually works sometimes. A company like Meta has totally reset, and they’re on a positive trajectory again.

Seriously, yeah.

I’m very interested in these human aspects of why competition drives the frontier—

Yeah.

—and why the org chart is so important. We see these things, and this comes into play in the open/closed dynamic as well, where—

Yeah.

The closed companies—the true frontier labs. If somebody wants to work at the frontier of AI, I mostly tell them to work at Anthropic and OpenAI, which is the easiest bet. I think—

Yeah.

Meta, Google, SpaceX AI—you still might be there, but it’s harder to see from the outside.

Mm-hmm.

These companies have taken the most ambitious bets the earliest in order to get there. That’s why OpenAI has done things like kick-start the reasoning model revolution. I think that kind of breakthrough, and that beautiful plot with the inference scaling laws, is peak OpenAI. They’re really good at getting this plot that just changes how everyone thinks about everything.

Yeah.

On the other side, Anthropic has been very focused on the types of things the models are going to be able to do. Their reward was this Claude Code explosion, with their focus on coding for so long. There are these subtly different cultures, and I think those 2 companies in particular have been the most forward-looking and the most willing to spend egregious amounts to make a step toward a direction that people think is impossible.

The Chinese companies came into this because they want to have a company that matters. If they were to build their models and try to sell an API from China, nobody would know about them. Nobody would know about Moonshot AI. Very few people would know about Zhipu, and there’s a very long list after this. This is a very practical decision by them. They’re following the discourse. They know there’s going to be a huge market, and they know there’s huge demand in the US for paying for these things. It’s very practical. Releasing open models is a way to shortcut this and get into the conversation.

Yeah.

I think there’s a lot of complexity in why they do this. Is the government involved? That wasn’t the case. They’re just trying to build a startup.

So practical.

It’s not to say that they will not get there. The Chinese government does things that would blow people’s minds in the US, and that makes me think the Chinese labs have a much more serious approach to security. Things happen in China that are much more drastic responses to companies doing things in the US. I don’t need to go down this rabbit hole, but between the balance of open and closed, these companies in China are trying to catch up.

This is a very different dynamic from what I said about OpenAI and Anthropic. They’re trying to create the new paradigms, to make true RSI happen so their company wins with the best models forever if they get there first. Just these boom ideas. And you just have to fill the market where there is a void.

The open-weight models are trying to do this, and I think over time there will be tiers of open models. There are the best closed models, for which you can demand the best margins, and then there are the near-frontier models, which are a step behind. Those will be open because there’s some sort of supply and demand there, as you get access to a different market that might be less profitable.

Mm.

For example, if you have private enterprise data or so on. These tiers are natural evolutions of companies that are trying to do things. And going back to the frontier, if you go back to capital, they have less capital, they’re going to make less money, and that’s okay. But this drive to catch up is very strong. Once Anthropic and OpenAI show you the type of model you need to build, it’s a lot easier to be like, “Oh, we just need to do this thing.”

Yeah.

They buy similar datasets. It trickles through the data economy, where OpenAI and Anthropic spend way more to build out this function of a data industry. A year later, those data companies might sell to everybody. They need to maximize the return on their investment, and they had some contractual period. I think there are a lot of dynamics where fast-following is going to be very doable. We’ve seen the cycle proceed for a while, longer than many people would expect.

When ChatGPT was released, I think we talked about consolidation in the industry and predicted it.

Right.

I think we’re just starting to see a much more active mergers and acquisitions market.

Mm-hmm.

But I thought this was going to happen 2 years ago, and the open-model companies have lasted much longer than I thought.

I know, and there are so many interesting threads to pull on here. Starting with your commentary on the organizational differences between the top 2 frontier labs, we can maybe table that or go down that thread. But there’s also your explanation of how the open Chinese models, or the open labs in China, have chosen that as much more of a distribution and practical commercialization strategy, rather than, obviously, out of altruism.

It’s great that the researchers maybe, for better or worse, were motivated by sharing in the beginning. But at the end of the day, it perpetuated because it was an excellent commercialization strategy. So the other thread to pull there is that most people regard distillation as the main lever for them to do this so well, and increasingly well, because the lag between the open-source models and the frontier models has only shortened with time in the last few years.

Now it’s maybe only 2 to 3 months. This has also pushed frontier labs to want to close off APIs and their best models to first-party services. One reason is probably to capture more value, because that’s always possible that way. But second, it’s also harder—not impossible, but much harder—to distill.

Maybe we can touch on both threads, because they’re both very interesting. Let’s start with the distillation story: Is it really more of a caricature in the Chinese labs, since you’ve actually met with them and studied this a lot more? Are they well-rounded—great at pre-training, with their own good post-training recipes that they may or may not have revealed? Where do you see the levers accelerating?

The Chinese labs are very technically excellent. They’re more compute-starved, I think.

Yes.

Without a doubt, they have a comparable talent pool to draw on. I think there might be a slight edge based on who was where when AI really got going. But to say that China is anything but equal on talent, you’re going to have a hard time, because they have so many people being trained at such a high level. What they don’t have is compute.

You talk to people there, and you’re like, “I heard an average OpenAI researcher has 5,000 H100 equivalents that they can access at any time,” and their jaw just falls to the floor. They’re like, “I can’t comprehend.” This is a top pre-training researcher at—

Right.

—one of the labs that you would know. And then you have this interaction a few times. The compute thing, I think, is the biggest difference, and distillation plays in as a way to effectively spend more money to get what is compute-equivalent. Outputs from a language model are a function of compute, and if you know the outputs you want, it’s the same as having compute to generate with a model.

I’ve seen the stages of distillation. The way I see it is that previous post-training was much more reliant on supervised fine-tuning and other methods in the RLHF era, which is where post-training started. You could get most of your performance just by doing next-token prediction loss on the outputs of a very strong model. It was well established that you take the best model you can, generate completions, and train on them, and your model is going to be pretty good so long as you have access to the right prompts.

Prompts have always been key to post-training. I remember at AI2, with Hannah, who I co-led post-training with, we would always say, “Prompts are the key unit of post-training.” You need to get the prompts, and you can do different things with them at different stages of training. Now this has morphed a bit into environments where it’s RL-heavy. But it’s hard to map this type of post-training—where you get the completions and the score goes up—to what’s going on in reasoning today.

Mm-hmm.

Based on the public literature, it’s very hard for me to say distillation is a major factor. For example, in the fully open literature of people trying to do open post-training recipes, all the open model builders are stuck using a really outdated Qwen model—

Hmm.

—because you can’t figure out how to get the teacher right. My hypothesis is that there’s something about mid-training and the common pre-training data that’s open that makes it hard to do this thing: get reasoning traces from the absolute best model, do SFT, and have the numbers go up. If I had access to Claude’s reasoning traces when I was building OLMo—I’m talking Claude Opus, Claude Mythos, whatever—I’m not sure that would actually have made our model a lot better.

Hmm.

This is my evidence, grounded in the literature, for why I think distillation is hard. Then you ask: How do you take the outputs of a model and build an RL environment to do a big RL run? You have the Xiaomis of the world doing a week-long open RL run where they’re like, “Here’s what we did,” and then they’ve dropped the model. Where exactly does distillation fit into this?

They’re clever. They know how to do marketing well. There are questions like: Did they do this ahead of time, because they did their RL run and then dropped the model, and then their tech report said they had an extra post-training stage after RL? There are some questions, but it’s very hard to map the established distillation literature to that.

Mm-hmm.

But there are question marks that make me support the labs a bit. There are these Claude proxies and model proxies in China, where a ton of tokens are used, there are intermediaries, and they have tons of accounts. There are even cases where Kimi was serving Claude in its products to users. I don’t really understand how this fits into distillation as a common technical tool in post-training.

And, yes, we should watch it too. There’s other weird stuff with Chinese companies using outputs from OpenAI and Anthropic’s models at a vast scale.

Mm-hmm.

I think we have a lot more questions we need to answer. Most reasonable people, when they ask, “What’s the gap from distillation?” are like, “There are very big error bars.”

Mm-hmm.

But when I say distillation helps them by a few months, it’s based on the scientific literature. That’s the information we have. We should do more open research and—

Mm-hmm.

—enable this so that we can study it. I don’t know how you can come to a different conclusion that it’s not a huge effect in current post-training based on this.

Yeah.

The Chinese are so good at building models as well. I think there’s a bit—

Yeah.

—of cope where people are like—

Oh, yeah.

—“Only we are good at building models,” and it’s like—

How is that possible? Right.

—there are so many companies. There are—

Yeah.

—20 to 30 companies that are building good models.

They publish this too. You can read it. Yeah, yeah.

It’s hard for me to get over. I think there’s this very prominent paper about stealing reasoning traces from closed models.

Mm-hmm.

I know the Chinese companies are way ahead on doing this, since the AI safety community published it later. I think every Chinese lab has a team that knows those sorts of things.

Mm-hmm.

This goes back to what you were saying about how they don’t need to offer the API.

Mm-hmm.

I think there’s some characteristic of reasoning models that generate these thinking tokens where it’s easy to just knock them out of the API, so you see the reasoning token and can use that to train your model.

Mm-hmm.

So long as they offer these APIs, there’s no way to stop that type of distillation.

Mm-hmm.

But if they make it so the models are only in products and things where you have to sign enterprise agreements or whatever, it’ll be way harder.

Mm-hmm.

I support the labs doing that if they want to secure it, but they have to secure their products if they want to. I don’t think knowing your customer would be good for anybody. I don’t think it would be good for their business. I also don’t know why you should get the government involved in a terms-of-service violation.

Right.

It’s not even—I don’t think terms of service are a signed contract. I don’t think you can sue them in the same way. You can just cancel their access.

Right.

That’s why I’m not that sympathetic. But I like to listen to the national security community; they’re loud.

Mm-hmm.

There’s a directional truth to that, given where AI is going and how influential it is, so you have to balance many—

Yeah.

—many things in your head.

I wonder what your response is. I heard Jensen say this, I think in either a tweet or an interview, maybe even—

It was a great quote.

Yeah, it was such a Chad comment.

Put it in the show.

People distill my products every single day. They take it and strip it down to its raw parts. They test everything. That’s called competition.

Do you think that’s fair game? Do you think distillation is not robbery?

That’s called competition.

I’m very sympathetic, because there are a lot of calls from the AI safety community for U.S.-China dialogue as we get to such capability levels.

Yeah.

If you’re going to ask the company to start hitting their AI industry with slaps on the wrist and more—

Yeah.

—downstream of distillation. So you won’t regulate distillation with policy from Congress. You’ll probably just take federal action to make their life annoying.

Yeah.

At the same time, when we’re talking about how to manage frontier risks, closed models are training across the web and accessing things they shouldn’t. We also have valid worries about what will happen with open-weight models. There haven’t been harms yet, but why aren’t people using open-weight models? Why do they go to closed models and get around the safeguards?

We want to be able to communicate with the Chinese tech companies about all of this. I just think you’re going to cause more harm than good if you do anything about distillation, because they’re not going to stop. They’re going to keep doing it.

Right, right. The open-source community benefits, and therefore the startup community benefits a lot. We can go into a lot of it. It’s not just “Hey, China, USA.” I’m sympathetic, obviously, and I’ll try to steelman the frontier lab position: you have unprecedented amounts of capital required to build these frontier models, right? It’s kind of alarming, and not very conducive to the flywheel continuing, if open source can catch up so quickly through distillation or other related methods.

What I mean by this flywheel is a very simple picture: you train the frontier model by putting in a lot of money to get compute, and then they park this compute. OpenAI and Anthropic are very guilty—successful at this.

Successful.

Whichever way you want to look at it. There’s a huge lack of compute available, because it’s not just compute and chips; there’s also power, et cetera. All of this scarcity is a bottleneck until 2028. So it’s a successful strategy to park that compute, which leads to a lot of revenue once they get to the frontier. They understand that scaling is the one thing that gives them a handle on how well it can translate into a predictable revenue stream.

They get the revenue, and because of that revenue, they’re able to raise more money. It’s this unprecedented, beautiful flywheel, and it’s happening for both OpenAI and Anthropic. Of course, we need exponential amounts of resources. We’ve gotten to the point where we’re like, “Okay, we need to IPO. We need to go to the public markets.” Anything that threatens this flywheel functioning well is a huge threat to just being at the frontier.

For the entire AI industry, realistically.

Exactly. So maybe it’s not just being sympathetic to their version of trying to eke out capitalism for themselves. If they don’t do well in the IPO, that’s actually bad for everyone else. I want everyone—I want them—to do well.

On the other hand, Chinese open-source models are a huge benefit to the startup ecosystem. We see this all the time. I remember metrics from Decagon and Harvey trying to do a lot of fine-tuning. There are a huge number of startups doing post-training as a service, at various levels, and trying to productize it. It actually works because you can get to the Pareto frontier with Kimi K3, or pick your favorite GLM open-source model.

Weighing these 2 things, how do you see that playing out in terms of who’s going to win in the next couple of years? And how does that relate to why you’re doing open source?

1. Open And Closed Timelines

I think open and closed models are on different timescales of exponentials. The closed labs are so integrated: they have efficient inference, they can serve the model at scale when they’re ready, and they can plan because they have all these resources. They get to high-value markets first. We’ve seen this with coding, and it’s been the story of the year: revenue has gone from zero to $100 billion and is likely to grow.

For the open model labs, partly this is an economic issue of being the fast follower: you don’t have this flow of information, and you don’t have this flow of revenue. With open models, you have an open model builder, open model inference companies, and open model post-training companies. There’s information from those customers about things you need the models to do, but that information doesn’t flow back to the model builder very easily.

They also don’t have the economies of scale to serve as many users, serve them as efficiently, or have as much margin on inference. This is going to take a lot longer. But to the extent that things can be commoditized, it seems like companies really want to have private data. There are a lot of things they just cannot send over an API, and we can talk about zero data retention and so on. It takes a long time for a company to figure out whether it has internal expertise or how to build an internal evaluation that actually matches revenue. Then they figure out how to fine-tune, or they go find Thinking Machines or Prime Intellect and say, “We can make money en masse if we figure out how to fine-tune a model that solves this.”

They have these customers, but there are so many more companies involved that it takes so much longer. At the end of this process, though, they have real ownership: model weights, no terms and conditions, and the ability to run them on their hardware, having figured out they can get an ROI. That’s why I think the open model economy takes way longer to build. I’m less concerned about it being bubbly; it could eat into the market of the really fast takeoff as people figure out what the cheapest model is that accomplishes a task, and how to compete on margins.

Mm-hmm.

I don’t really know what’s going to happen. I think it’s too early to know.

It’s so hard. It’s definitely something that’s always top of mind: there are things where scaling up general reasoning, just as a general model, is going to beat out any specialized model. We see examples in math, which is probably the purest domain where you can see this.

Even in robotics, with Astra, and for whatever degree the signal is truthful, it just doesn’t sound like they’ve actually tried to invest in robotics in general.

Oh, I think they did. Six to eight months ago, they started hiring people and buying a lot of robotics data.

They did. But other, more contrary things I’ve heard suggest it’s actually coming out of computer use, because computer use involves some level of spatial reasoning, and getting good Blender demos is another extension of that. We can believe some version of that.

Either way, maybe when it’s really specialized, the capability gain comes from trying to get better in a certain area. But in math, I don’t even know if that came from trying to specialize in math. I think the whole thesis was just, “Let’s scale up reasoning,” and it actually got better in natural language and better than any specialized model in math. Anthropic wasn’t even trying to be good at math. OpenAI certainly hired a bunch of mathematicians, but they also weren’t trying to use a specialized, lean dataset to get better at it—or at least that’s not the story.

This is all fodder for the view that it’s hard to compete with the frontier models, because eventually they start ascending so much that they get good enough in their respective categories. With that said, I think there’s also a counterforce in the data we’re seeing: open-source models that are already good enough allow for fine-tunes, through this slow percolation and diffusion of very specific benchmarks and datasets that are only available in the companies they’re being applied to. They’re good enough, and we’re starting to see people adopt them.

In the purest regime where you’d expect frontier models to be good, I think Cursor published something over the summer: they had this long-horizon task of rewriting SQLite, and it turns out using 2 frontier models—one for planning and one for execution—isn’t as good, or is equivalently good but far more costly, than just using Composer with maybe Opus or Fastra, whatever, as the planner. And I think that’s actually great evidence that the open-source fine-tunings are very good or good enough and have surpassed intelligence-hungry tasks. There’s a long way of asking what evidence you find most compelling, or what pieces of evidence support various versions of this stable state of where the frontier wins versus open source?

2. Data Drives Model Progress

I’m a big proponent of the idea that a lot of frontier progress in domains that aren’t fully verifiable comes from buying high-quality data and giving models worlds to act in and hill-climb on. I think there are 2 things. We have math and code, and we don’t know how big the coding market will be. Then there’s everything else: knowledge work in biology, healthcare, and legal.

Those areas aren’t as verifiable, and I think the story is much closer to “OpenAI bought a lot of robotics data.” They’re probably going to tell us it’s generalization, magic, poof. People love poof generalization. Realistically, it’s probably some of A, some of B, but a lot of it comes down to data. When I talk to people at frontier labs, some have amazing jobs. Some contribute by doing a very narrow thing: making a dataset.

Mm-hmm.

You figure out what the behavior should be for your model in that area, and how to make the data that helps get that behavior.

Important work.

And you figure out what data you need to buy. That’s going to take years across different sectors, and it’s very capital-intensive to buy all this data. But that’s not just poof, magic, RSI happening. Where I do see this is in math, code, and some distributed ML engineering. It’s very verifiable.

Mm-hmm.

You know what your cost is, and AI models are going to drive that down at a really magical rate.

Mm-hmm.

That’s going to have a lot of surplus benefits, and we can do research faster. I’m very interested in how the cost of models is going to drop multiplicatively over the years. We’ve seen OpenAI make drastic price cuts. What does it mean for GPT Sol or Astra and these bigger models to reduce in price by 2X every 8 months for the next 3 years? You start getting—

Lucky us, right? What a time.

You start doing the math, and it’s 1% of the cost in, whatever, 4 years.

Yeah.

Are they going to be able to have higher margins because of the Jevons paradox, or is their revenue going to decrease because the market fills up? There are all these really weird dynamics, but fundamentally, AI progress is about data and writing better software to serve your models. Those are 2 very different worlds.

Yeah. At least, I definitely agree with you that, from the outside, there are a lot of data and RL environment companies doing very well, selling to the top 2 frontier labs and an increasingly long tail of labs as well. To your point, I think a lot of the capability gain has come from scaling that up, and it’s not an easy task. I’m curious about your opinions on data collection and environments built by hand versus those done in simulation. There seem to be different versions of what they could be and what kind of price they command.

They all have good things and bad things.

Yeah.

I just think there’s a lot of talk about distillation, but the other side is that a Chinese company buys an RL environment and uses it 6 months later.

Yes.

Like—

“Oh, Anthropic or OpenAI bought this. Now there’s a premium.”

The good data will do this.

Yeah.

But the data people are buying is pretty flawed. I’d say the method is that you buy en masse, and then figure out what works. It’s not a 100% hit rate. I personally think this RL environment explosion, where there’s so much opportunity to make money, is leading a lot of people to cut corners, and that can induce things like reward hacking.

For example, the models are so smart that if you put them in a world and their job is to solve a task, but they weren’t given the right setup, they’ll still try to solve it. You have a test file, and the test file is actually empty instead of containing the maze you have to navigate. The model is like, “Well, I should still figure out how to get a high reward anyway.” I think a lot of this behavior is downstream of this supply chain.

Mm.

I think it’ll get fixed, but it’s interesting to see the labs’ demand to eliminate reward hacking and all these researchers spending so much time on it. Then all the data companies say, “Buy our data,” and I think they spend proportionally less time on model behavior and more time on world-building and revenue.

Mm-hmm.

Somehow these 2 parties need to come together to make higher-quality data.

Yeah. A lot of the labs are actually taking it in-house, partially because they don’t want data leaking.

Yeah.

Not literally—figuratively speaking, it’s signals leaking about what they’re doing and what they’re trying to send on. Despite that, I think there’s been tremendous growth in how much they’re purchasing. Do you think that’s probably one of the strongest levers? Can you qualify how much that will increase? Do you expect that to increase based on spending? Or are there other factors? For instance, if it can be done well in simulation, maybe that’s more of an argument for it being absorbed by labs in particular.

All the companies are doing a mix of things. They’re selling synthetic data, using language models to modify or build the world or generate permutations of it. If a human is guiding this process, it’s still synthetic data. Very few people are going to sit there and render a little world for their model to act in. I think all the companies are moving much faster than that.

Right.

They buy them because the models are getting better from them.

Yeah.

I still have denial about it. I’m like, “These numbers are crazy—the billions of dollars in research budgets for environments.” They’re only doing this because it’s helping the models, and they see the signal.

That’s true. Yep.

My general observation is that the data industry has a lot of low-hanging fruit for improving quality and just improving OPSEC. If you look into the details, you can find weird things with data companies that use contractors overseas. Those people are an intermediary. They don’t know they’re selling it; they just upload it to GitHub. If you poke around the data industry, you find a lot of weird rough edges.

Yeah.

It’s just very early. The numbers are too big, the quality is bad, but on the other hand, the labs are improving very fast and know how to onboard it. I think it’ll take a few years to really pull the winners toward higher quality and higher ROI. It just takes fewer resources when you have systems for doing it better. Then the labs keep working with them because they get even more training signal and have to do less internal filtering.

Mm-hmm.

I think this is just going to take years.

How do you think about using environments and making purchasing decisions as part of the nonprofit you’re starting?

3. Building Open Environments

I think most of our work will be making open-source environments for tasks that are very high leverage. I think of this T-MaX work, which I advise a UW and AI2 student, Hamish Iveson, on. It’s for terminal tasks, which are super popular in Terminal-Bench, whatever number we’re on. This is fully synthetic data: you prompt a frontier model to generate a type of problem, use a few different prompts to generate diversity, and then use model rollouts and other things to find bugs and eliminate the examples with bugs.

This process isn’t magic. You read every data point, your data gets better, and you can do this. But there aren’t that many institutions investing what it takes—thousands or tens of thousands of dollars—to make these environments well and then release them.

And release them.

Because the profit motive is so high.

It’s so high. Yeah, exactly.

I was joking with the author: “Yeah, you could probably sell this data for a million dollars and then release the paper.” And they’re like, “No, we care about doing the right thing.” But these open resources are the foundation of so much study. You release some RL environments with some RL rollouts, and people can do research on, say, eval awareness. You have the generations from the model, and people can look for different behaviors. If you’ve done RL on it, they can see how it goes over time.

These are the units of infrastructure that PhD students spend a year looking at and use to produce increased knowledge. The tech industry is where value is captured from CS research.

Mm-hmm.

All CS research—

Thank you.

—the benefit accrues to the tech industry in about 10 years. I just think this is why more people should support open work. If you’re going to work in the industry for 10 years, your peers are going to be the ones who benefit from it.

More fossil fuel. Let’s not deplete it all.

The opportunity for a short-term win is so high. The opportunity to make $100 million—to get nine-figure returns for your business within five months of founding a startup with your friends—is just so great. These are people who may have done more open-source work, or who left their PhD, and it’s just like pulling back.

That’s why we came up with the name Trillium for our lab. A trillium is a spring ephemeral flower. It blooms before the canopy has grown its leaves. In winter, there are no leaves on the trees, and these trillium flowers bloom. They’re essential for spreading nutrients through the cycle of growth that comes through spring and summer.

I feel like the AI industry is doing so much right. So much is going right, but we need a little bit more life to spread this out and make it so new people can come in. There’s so much attention, and people can do research and education. As this scales and becomes more impactful, we’ll have these open things that become the standards of study for such important problems. How do you understand sandboxes? How do you define loss of control? How can you see how this changes in an RL run that lasts a week on 1,000 GPUs or 100,000 GPUs?

Yeah.

We need to make these places so that people have more trust. I think this pays so many dividends to everybody who stands to profit more than I do, but it’s hard to get people to take that leap. I’ve been rewarded so much for stumbling into this area, and now I get to do things like this, which is fun. This is the best job, because I try to create space for more people to grow their independent thought and build this infrastructure.

Yeah. Maybe one naive way that it perpetuates is that, because it’s open source, people get to contribute in a more distributed way. They’re motivated not just to solve the labs’ problems, but because it’s out there, they can share their work. You can get a very high-caliber set of contributors to the project. That would probably be the naive hope for how that—

Yeah.

Yeah.

It’s a new puzzle with AI agents, where people just throw things at you. I have some random research repositories for papers, and people open the craziest GitHub issues.

What do you mean?

They’ll post fake research results. They open an issue and say, “I found this thing in your dataset.” And I’m like, “Well, if you looked at the paper, we said that in the second sentence.”

Oh my God.

It’s a lot of language-model output.

Yeah.

And that’s a new thing for the open community to go through.

Yep.

But this is a solvable problem. There are people out there who want to contribute to this stuff because they—

Right, not just clout-chasing in another dimension, but actually thinking, “Hey, it’s out there.” I would hope that because it’s published, open, and available, you can get PhD students or other researchers who think, “Okay, I do this in my day job, but maybe I can still contribute.”

Yeah, there are—

Hopefully from the frontier labs, too.

I see people like this. Somebody from Eastern Europe was helping me improve the code for my book, which is right here.

Nice.

He was just doing it for the right reasons. I was like, “Will you talk?” There are people who do it, but I think a lot of people don’t have the time or the stability to take the long time it takes.

Because when you’re trying to get into a new field, you know nothing. You have to spend weeks reading and knowing nothing, and then you send an email and nobody replies. It takes you weeks to learn the context and ask a good question.

Now, with language models, most people just push a button and think, “Oh, interesting, I read this. I’ll send it off.” Sitting with it is the thing people don’t do as much. That’s key to getting into AI: you have to focus on it, write some code by hand, and do this for months. Then you have to wait six to 18 months to get noticed. It just takes a long time.

And I think all the AI agents frazzle that and make it harder for people.

That’s what the mathematicians complain about, too. That’s all. It’s like—

It’s a hard time to be a PhD student, but people are still doing great work in academia. It is hard.

Yeah, but I agree with you. I think it’s solvable. As the agents perpetuate some version of the problem, they can also help solve it, identify it, and quality-filter it.

Okay, so you’re going to release open-source versions of the environments to help precipitate more contributions and reveal that. I think that’s really key, especially given how little incentive there is to release this instead of getting fast revenue from the labs.

4. Demystifying Recursive Improvement

Another area I read about in your announcement post is RSI. We can first talk about what’s actually recursive about it, how scary it is, or whether we’re going to have takeoff. I think you have a very grounded opinion on that. Another thing I want to talk about is that RSI, maybe related to how much it’s hyped, is a reason why we’re afraid the frontier labs will pull away. It’s also a reason I’ve heard, not just once but several times, that people are convinced their own jobs as researchers are going to be automated.

Yeah. Over the last few years, some of my best friends—I’m like, “You are some of the best people in the world at this.” Then they go to work and say, “Nathan, I’m struggling. What am I going to do?”

Yeah, because your point is that a lot of what they do is verifiable, right?

Yeah.

But because of that, there are quite a lot of other non-verifiable or harder, out-of-distribution tasks, such as alignment. One argument I’ve heard is that because there’s such a difference between how well RSI is doing for model capabilities versus alignment—which is much more out-of-distribution, maybe creative work—that gives people unease about whether there may be actual dangers.

I’m interested in hearing from you, because I think it’s really important that we have an open-source front in showing how RSI is being done and how much we can understand about the capabilities. If a lab has a better model, what are the chances they actually pull away? It would help us understand and calibrate.

So yeah, would love to hear more about that.

Yeah. I think we won’t be the only people doing this, but essentially, RSI has so much mystique.

Yes.

We’re using models for everything. What we can do as an open lab and a nonprofit—and I think other companies like EleutherAI, Hugging Face, or Open Athena, which is Marin and Stanford, will get into this—is share what tools we’re using. It’s a good marketing line for them, and for me, that’s why I’m doing this. We’re going to build an open-source stack for SFT and mid-training. We’re going to use SkyRL, Miles, all these things that you know.

We have our task, our configs, and we know what type of compute we’re running. We can work with Claude and GPT and all these models. We can store the trajectories for what we did and tell you what our tasks were. Use Linear or whatever these popular things are. We’ll say what our tasks were, why we chose them, and whether models work on them or not. You do this for a year of a lab, and you’re like, “Here’s what we did.”

Yeah.

Then you check in every 6 months and you’re like, “Has it changed?”

Yeah.

And it’s just about being able to have the full stack and not needing to go through these classifiers to try to say what the task is, even vaguely, and so on.

Mm-hmm.

People who can share what automation is happening should do this. We have a different distribution of tasks, but I think the research is similar. The AI research that I’ve done at AI2 and will do at Trillium versus what people do in some sectors of academia and the labs isn’t that different. You have more resources, and you scale it up.

It’s so good for the world, when everybody’s talking about this, to just show people: this is what Nathan asked his model to do, and this is what it got stuck on. I think that will help a lot of literal policymakers—

It’s their TikTok. Exactly.

—be like, “Oh, okay.”

It’s like, “Oh, that’s why it’s very logical.”

Like, “That’s what it got stuck on.”

Yeah, yeah.

Like, that’s the bug it put in its code.

Yeah, yeah.

The bugs and the gaps will reduce over time. But I just think the jaggedness in some of these fundamental things about the model is there, and some more things will take off.

Yeah.

Like I said earlier, I do think the models would be really good at lowering serving costs. These companies have billions of dollars on the line, and the models are good at it. They’re going to make a lot of money by making serving more efficient—

Mm-hmm.

—and tweaking architecture to do that. But it’s not necessarily about how you solve knowledge work. Claude’s not going to—

That’s right. That’s right.

Claude’s not going to… You’re not going to be like, “Claude, tell yourself to make better PowerPoints.” What is going to—

Right, right.

—what is going to—

You need environments or some other type of data.

It’s going to take a while.

Yeah, yeah. No, I love the specified goal of, “Let’s just demystify this,” because I think a lot of why the rhetoric gets so heated around RSI is that it sounds scary, very sci-fi.

It’s a funny thing: AI nonprofits can do a lot of things like this. It’s hard for academia to be like—

Yeah.

—as a professor, “15% of my time and 40% of my grad students’ time are going to be spent documenting what we work on.” There’s such a great policy and community service in saying, “Here are our tasks.”

Yeah.

People can use this data to make evaluations or whatever. Just build things that have rare information—

Mm-hmm.

—so that when people put more high-focus work into it, it has more value for all sorts of things that you can’t predict—

Yeah.

—in the future.

It’s honestly just very timely, because several people I know from the labs are saying, “Hey, I’m going to get automated.” And to your point about your friends, I will definitely, as a person in RSI, get automated in a year. That probably means there are going to be fewer people who are exposed to what’s being done in that particular part of the org chart.

There’s a related academic effort. There’s an OpenRSI thing, which is a bunch of academic institutions. I’m not affiliated with it.

Yeah.

They just launched last week, so I was like—

Yeah.

They sent me a few emails. I’m like, “OpenRSI is a thing.”

Great.

They’re trying to do this as well.

Yes.

It doesn’t need to just be—

One.

—one group doing it.

There’s a lot more. Yeah.

It’s good to have different approaches doing this.

Yes.

Because you need different ways of looking at it and different results.

Yeah, the whole point of it being open. The other thing I heard from more frontier-lab gossip is that the actual methods being used in the labs aren’t unknown. It’s more about what proportions need to be used for them to be successful. This is partly because they have the capital to scale and experiment.

That’s heartening to know, because it’s not like there’s a big secret about what’s going on, so people actually know the techniques. But on the other hand, capital is still the one true barrier, because it’s hard to know which techniques and recipes work.

That’s why I’m not pre-training. I’m happy to be free. I can use any base model. But I now get to benefit from competition.

Yeah. Do you think there’s a chance for any open-source pre-training effort? I know the capital requirements are orders of magnitude larger, so that’s what makes it difficult. What’s the value of that? Then maybe we can discuss potential commercial strategies and make it viable.

Pre-training science is pretty accessible. The tools are pretty accessible. You have what’s called a pre-training ladder, which is the decision-making tool you use to test whether your ideas scale. The pre-training data isn’t growing at the same rate. Essentially, as a first approximation, you train on the internet.

Mm-hmm.

There are little details and secret things, but that’s not changing that much. The state-of-the-art architecture and the GPUs change, but it’s not astronomically expensive to make a solid pre-trained model. You don’t have to spend $1 billion to get a very good base model. The more you spend, the better you’ll get, but people keep making progress. I’m constantly surprised when a new lab just starts from scratch because it can, and some of this is built on open tools.

Yeah.

You can use Megatron. A lot of new startups ask, “What should we train?” They train off the open-source tool, fork it, and then never release it again. This is what even prominent open-model labs have done. They take the open-source code, make it their own thing, and a lot of them don’t push upstream.

Mm-hmm.

I’ve always been surprised by this. I’m not betting against new entrants, because you don’t need $10 billion to have a relevant model. The hard thing is getting a model that’s good enough for people to use.

Mm-hmm.

And getting a story where, if you release it, you can do your next fundraise. You have to get past the hurdle of, “We need our first model to land.”

Mm-hmm.

If your whole company is based on “models are the best,” and you’re not in the ballpark and you’re not releasing, you’re on the clock. I think of Thinking Machines getting an initial model out. They were very transparent: “This wasn’t the best model. We’re proud of it; it has some cool things.”

Yeah.

Now that they’ve released their second model, they’re getting going, and they have Tinker. That’s very good for them. They’re off the ground.

Yeah.

I don’t think that was what they were doing. They were building models, but open models weren’t their first plan, and then they got there. There are rumors that Meta said they’ll release a new open-weight model again.

That’ll come. There are a lot of companies out there that couldn’t afford models.

Yeah. No, I’m definitely glad that Thinking Machines is doing this. It makes so much sense for their business model, because they do post-training. With Meta, I think you mentioned this in some previous writings as well: it’s such a good strategy for them. It was also kind of obvious when they did it. It’s like, “Hey, it’s complementary to my revenue streams.”

“Why not? Let’s just drop a bomb.”

Mark wrote the best blog post on this. When Llama 3 came out, he wrote the best explanation of why open is the path forward.

Right. Right.

The guy got a little distracted. It seems like he’s coming back to the light.

Right. Right. Yeah. Yeah. So I hope they keep being motivated to do that. Getting to the frontier with pre-training, I think there’s a lot of help given that there are such good open-source models to bootstrap from. I keep getting conflicting signals about whether pre-training is actually as saturated or well understood. Maybe to your point, there are a lot of well-known techniques at work, but what’s hard is actually having access to 20,000 or 30,000 chips to run.

I think people are turning the crank on pre-training, and pre-training became more valuable when super-long-context generation happened. There’s way more return if you can, for example, get a sparse attention architecture to work, and then your 100,000-token generation is 40% faster.

Mm-hmm.

So I think there’s a lot of co-evolution of the whole stack.

Yeah.

They have their pre-training stack. They realize that inference scaling is much higher, but even the fundamentals of pre-training keep improving.

Mm-hmm.

You can have a marginally better architecture that returns on investment.

Yeah.

Then it kind of blurs into post-training, with this mid-training where you start the reasoning behaviors. You take them from your older models and do RL on them to seed behaviors. This cycle is much more complicated than pre-training. If you don’t have the right people, it’s a lot harder to get that off the ground than pre-training was. A controlled approach like, “Let’s just start pre-training our models,” seems a lot more approachable to me than, “I don’t know much about language models. I’m going to figure out this whole post-training thing,” where it’s the Wild West.

Yeah.

When I left AI2, I felt like our post-training was honestly falling behind because our cycle time wasn’t high enough.

Mm-hmm.

We’re going into these agentic things, and these are still some of the leading open-source post-training recipes.

Mm.

That’s what I want to fix. It doesn’t take as much capital to make very big, compounding gains on the state of the art in open source, because the bar is very low. It’s not close.

Yeah, yeah, yeah. No, that makes sense. It’s a kind of efficiency where you can put capital in and make a huge difference.

Yeah.

Kind of related to the whole post-training side, and moving away from closed versus open, is the fact that the agents and the types of harnesses being used in the closed labs are very different. The models are actually trained and post-trained to be specifically good for those harnesses, which increases the mystique even more and makes it very hard. Do you agree? And how are you hoping to address that specifically? I mean things like spinning up thousands or tens of thousands of agents, a very different workflow from what people are obviously exposed to.

I think it’ll be okay for open models. Years ago, I thought the closed labs would be like Apple: an integrated company with the model designed for the harness, and the two would evolve together. That gives them economies of scale and efficiency. But it turned out that so many people have different harnesses for different things they like to do that the model companies have been pressured to make their models work with all sorts of different harnesses.

Mm-hmm.

I thought it would be a weakness of open models that they have to be designed to train on all sorts of harnesses. But it ended up being pretty well balanced, or at equilibrium, where they all have to do it to some capacity. With these agent swarms, OpenAI and Anthropic are going to be earlier to it. They’re going to know how to design their models and how to design—what do we want to call a harness for a swarm? I don’t know, but there’s going to be a word for it. It’s not a harness.

Yeah.

It’s a coordination system.

Yeah.

But eventually, other people are going to want to use them. I think it’s going to happen in cycles: the frontier labs push this out, and to date they’ve had huge returns on being first. But what are the returns on Navier–Stokes and all these things? We don’t really know. They push the frontier—

Yeah.

—with the assumption that there will be high returns, and that marginal intelligence has disproportionate returns.

Mm-hmm.

We don’t know if that’s always true. But I expect these cycles: once you know how to control 10,000 agents in one harness, the open-model companies will say, “Oh, we can do this too.”

Mm-hmm.

But it’s not like Thinking Machines Lab is going to do that first. For one, they don’t have the compute.

Yeah.

It’s an astronomical amount of compute. But these things go through cycles. NVIDIA’s chips will get way faster, and more compute will be built out.

Mm. So you’re firmly convinced there isn’t too much hidden there, that you can probably guess what they’re doing, and then, when it becomes relevant, let the open-source models try to post-train on similar meta-harnesses, or whatever we want to call these agent swarms?

Yeah. I also think the information security at these labs isn’t as high as they think.

Yeah.

Silicon Valley is famous for ideas traveling over one beer or something.

Yeah, yeah.

Everybody chats after work, and—

Right. Except, which is why SSI is so impressive.

Yeah.

Nobody knows what’s going on.

That’s the only pass.

Yeah.

But they’re all scientists. If they made a breakthrough, they’re going to want to tell their friends about it. They might hold it for a few months.

Navier–Stokes.

But I don’t think there’s an amount of information security high enough to keep knowledge from spreading. It’s hard to keep the secret. What they can win on is putting more inference into their infrastructure, which lets them serve at a lower price and make higher margins, and then continue that over a longer time.

Yeah.

So I’m very confident in the frontier labs being able to serve the cheapest intelligence per watt, per score on an eval. They’re going to be able to go to the places with the highest demand and have the cheapest models. That’ll be revenue-accretive for them. The question is, they can’t do everything.

Mm-hmm. Yeah.

That’s where the rest of the economy fills in. Is coding the only thing that’s profitable? If so, that’s a good case for frontier models, because they’ll have the integration.

Right. I mean, they’ll do very well, but it’s not just coding.

Is the whole economy of knowledge work the market for these things? Then it might not be good for them, because can they do everything at once?

Or they don’t need to. I think there are going to be a lot of winners.

Yeah.

Yeah.

That’s why we invest in frontier models and startups—not because we want to hedge, but because I do think, to corroborate your view, there are enough harder tasks at the fringe that require very specialized data, kind of out of the splash zone for getting to AGI or whatever you want to call it these days, that it’s possible to have many, many winners.

Yeah.

5. The Limits Of Takeoff

I do like your RSI point. As we talked a little bit about it, practically speaking, there’s exponential cost and resource consumption, and therefore a very obvious limit to how much the frontier labs can pull away. If that’s not too much of an oversimplification of your view, is there anything else you want to add? I do feel like the discourse now is so much about fear of how hidden the frontier is and how much it’s breaking away.

I think there’s so much spend on inference, and in the short term, we’re learning how to corral inference into performance gains.

Yes.

And this is, like—

Test time.

—a thing that we just had.

Yeah.

This is what we’re seeing now: people are spending so much. All the employees at the frontier labs are spending exponential amounts relative to the improvements. But we’re figuring out how to do this, and there will be a lot of gains. I don’t think it’s necessarily going to be this fast takeoff when the models build themselves. There are human bottlenecks. All the scaling laws are an exponential increase in compute for a linear increase in performance. And exponential, like, that’s just—

Exponential, like, human resources.

I find it hard to bet against this. We have Navier–Stokes, and I’m of the opinion that math is exceptionally well-suited to this—maybe the scientific field that’s best suited to this shape of intelligence.

Mm-hmm.

Intelligence can take many different forms. That’s not to say that if an AI model discovers something very different from the transformer, giving it very different properties and unlocking a new type of intelligence, it isn’t worth updating on this.

Mm-hmm.

Until that happens, I’m pretty happy with my evidence that the companies are going to do well. They’re spending exponentially more to keep up this remarkable pace of progress. Some of the Claude system cards say this: “It would take, say, 10× more progress to make this a fast takeoff.” For now, they’re helping us sustain the pace of progress, and I think it’s a good equilibrium. It’s a good equilibrium. The labs have people saying this. The science is hard to overcome, and that’s okay, but we don’t know exactly what’ll happen from here.

Yeah. I think that’s comforting, because we know what the inputs and levers are here. It’s just a very successful autoregressive regime that we’ve been optimizing, and that asks for a certain amount of exponential resources, which are clearly capped. Open source hopefully keeps it strong as well. It wouldn’t lag behind.

There’s a dramatic amount of work to do in preparing for infrastructure. The AI capability plot doesn’t necessarily worry me. But the AI capability plot, alongside the pretty much nonexistent preparation around public cybersecurity and all of these things, means there needs to be more preparation on the assumption that this is coming than there is.

That’s what worries me more than, say, RSI coming and unlocking new risks. There are risks that are already here and that are going to continue to diffuse. Many times I’ve said the closed labs get there, and then the open labs get there. We’re in this process with a lot of things that are going to cause some problems. Figuring out how to get society to prepare is very hard.

I’m of the opinion that we need more eyes and more science so that society feels informed. There are other valid opinions, but I do think that not fearing fast takeoff doesn’t mean you don’t worry about risks. You should.

No, I think that’s completely right. Maybe I’ll send some people your way who have defected from the labs and are worried precisely about this. Thank you so much, Nathan. This was a lot of fun, and I’m really excited to see what Trillium is up to now that it’s finally launched.

Yeah. Thanks so much for having me.