Beam：伟大的美国开放模型——对话 ReflectionAI 联合创始人兼 CEO Misha Laskin

When you remove cyber offensive capabilities, you also remove cyber defensive capabilities. The state of the world today is that we have a few hundred safety researchers within closed labs that understand how these things work, and despite their best intentions, it is impossible to cover the long tail of unintended consequences that these systems might have. A very powerful closed model went and, like, hacked into another company, and the only way that company could remediate itself was by using open models to protect itself. That's the empirical evidence of the world that we're in. Linus's Law, with enough eyeballs, all bugs become shallow. I have the belief that with enough eyeballs, most security and safety vulnerabilities become shallow as well.

Today, we're joined by Misha Laskin, the co-founder and CEO of Reflection AI. Reflection provides open-weight models to power the future of intelligence. Misha was previously a researcher at Google DeepMind and received his PhD in physics.

Misha, could you give us an update on where things are at? You guys have been pioneering really interesting work in open-weight and open-source models, particularly with a U.S. focus. Can you talk a bit about what you've been up to and what you've been building?

Yeah, it's great to be here. Thanks for having me.

1. Building Frontier Open Intelligence

For the last 12 months, we set the mission of the company to build frontier open intelligence and make it widely accessible. Effectively, it's been a sprint to set up a lab capable of doing such things. It kind of feels like there may be a Reid Hoffman quote that you're assembling a plane as you're flying it, and that's been the case.

We were maybe around 30 people about a year ago, but it does take on the order of 100-plus—maybe a couple of hundred researchers and engineers—to build one of these things. It's a true rocket-ship project. We're now at around 300 people, and we've assembled all the teams in pre-training, mid-training, and reinforcement learning. We've trained our first models end to end and just released a model called Beam, which is Reflection's first open model.

What has been the most challenging part of that? When people talk about scaling up models or starting to build these models, one issue is compute and procuring enough of it, particularly a training cluster that's cohesive in terms of its ability to be used. The second is talent, and obviously there have been these huge talent wars in terms of how much people will pay for researchers. The third is the scale of data. Which of those has been the most limiting or the most challenging?

What's been interesting about this experience is that I got asked this question at an all-hands at the company: "What's been the hardest thing in the last year?" The reality is that everything is extremely hard. Everything has been hard.

People might ask, "What's the thing that really made your models work better than others?" Again, the answer is everything. You actually have to get 30 things right. That's why everything is hard: You have to get 30 things right. You have to get the talent, and you have to retain the talent. You have to have a strong mission and culture that keeps the talent working together and rowing in the same boat.

You have to get the data. You have to get the compute. You have to build the infrastructure to ensure that the compute is actually usable. At an existing big lab, a lot of these things have had years of build-out that enabled them to have this kind of stable surface.

I was at DeepMind before this, and a lot of the tools that we just take for granted—or that I took for granted as a researcher because they just worked—you have to build. I would say everything has been hard, but also very rewarding that you can do it.

You said that about a year ago you guys really committed to doing this. Can you talk about the shift in ambition and commitment, and what drove that?

2. The Reinforcement Learning Bet

When we started the company about 2½ years ago, our bet was—maybe I'll contextualize where the world was then. My co-founder, Yannis, and I had been working on the first series of Gemini models. We were working on the reinforcement learning team, which Yannis was leading. We had just shipped the Gemini 1 and then Gemini 1.5 models.

In that early era, it was similar to the first ChatGPT experience, which was primarily a chat experience. There was no coding, really, and nothing agentic. These models were just good at chat. They were primarily pre-trained models, with 95% of the compute spent on pre-training.

We're reinforcement learning researchers; that's been our lineage. Yannis, my co-founder, was one of the founding engineers at DeepMind and was a key contributor to all the big RL projects that came out of that lab, including AlphaGo.

Our bet was that, because we were working on the RL team and saw that RL for aligning chat models worked, you could take that and apply it to domains like mathematics and coding to make these systems agentic. That was the big bet. We thought you could do this as an independent lab, much more capital-efficiently, because we started seeing open-source models materialize.

There was a small model from Mistral at the time, Llama 2 had just been released, and we saw Llama 3 coming.

Which feels like ages ago.

Yeah, this was a while back—I guess two years ago.

Over that first year of the company, what ended up happening was, first, that reinforcement learning started working faster than we thought it would, both within our experiments and within industry. o1 came out at that time.

We actually thought it would take longer, because the journey from ImageNet in 2012 to RL systems that worked at scale was about 4 years, so we thought it might take something similar. But every year moves faster than my prediction from the previous year.

About a year into the company, we got to the point where all the good open models were coming from China. There weren't really good Western open models. We needed one as a base for the technology that we were building, and decided for a number of reasons that the most impactful thing we could do, given the state of play, was to build open models ourselves end to end and combine the RL bet with building models end to end.

There were enterprise reasons, geopolitical reasons, and also a research reason: It turned out you actually do need to pre-train your model in order to make reinforcement learning work very well at scale. You do need to do both. The things are just so tightly coupled that you need to do both.

3. The Frontier Compute Curve

The conventional wisdom has been that pre-training at the scale of the frontier is impossibly expensive, or at least extraordinarily so. Clearly, there's been a huge shift of compute toward post-training.

Feel free not to answer this, and we can just cut it, but how did you think about the resources needed, or the ratio between them? Did you have design principles around the model that made this feel tenable to you at the beginning? Or did you just say, "We need to do this. We'll resource it"?

The nice thing about being an open-model lab is that you can also talk about things openly.

You do need a lot of resources to build something meaningful, but you don't need the same amount of resources. Once you get to the frontier and you're really at the frontier of intelligence, then you need the same amount of resources as any other frontier lab. You can think about research as the exploration of new ideas versus the execution of known ideas.

Part of why getting great talent matters a lot is that you get to cut down on exploration and focus on executing the things that work. If you're catching up to the frontier, you can do it much more capital-efficiently.

How much capital do you think it would take to catch up to the frontier?

I think it depends. The frontier keeps moving.

Yeah.

Each year, it's actually more capital. I'll get back to the question of resourcing between, let's say, pre-training and post-training. I would say that a year ago, or maybe 18 months ago, it would have been on the order of $100 million—so hundreds of millions of dollars.

Mm-hmm.

I think that now, or let’s say even 6 months ago, it’s probably on the order of a billion—so, billions of dollars. Single-digit billions. Going into next year, I think on the order of 10 billion.

You can think about it as, for every generation of model, there’s roughly a 4X multiplier in compute. One heuristic that I have is: let’s say we talk about the chips that are currently considered frontier chips. Maybe a couple of years ago, it was an H100, and maybe it was 100,000 H100s. Then Astra was trained, and that was just a big run on 100,000 Blackwells, which is roughly a 4X multiplier on an H100.

Now, the next generation is going to be around 100,000 Vera Rubins, I believe.

Mm-hmm.

So that kind of gives you the scale, and it moves from hundreds of millions to billions to tens of billions.

Yeah. It seems like at some point that has to asymptote, because ultimately, relative to the revenue scale of these companies—even the potential revenue scale—you start to tap out in terms of the ability to invest against what you’re actually going to produce from an economic value perspective.

Absolutely.

So do you think we’re heading into that asymptote in terms of model size, or at least training-cluster size?

Absolutely. I think there’s only so much CapEx that you can do, right? Frontier labs are now putting hundreds of billions of dollars into it. Will they be putting trillions of dollars into it in the next year or two? It’s probably unlikely, but there are a couple of things that are happening.

One is that the use of that compute is becoming a lot more efficient—

Mm-hmm.

—because the models are themselves becoming tools that help build themselves.

Mm-hmm.

When it was just human researchers doing the work, there was probably a 7X improvement each year, and you can actually track these things very technically. In pre-training, you do it by having your pre-training loss, and then you measure your compute-efficiency gains against it. You made some changes, now you have a new pre-training loss, and it should be that you got to the same point with less compute.

When you say 7X, it’s actually a very concrete thing that you’re studying. Across the systems, pre-training is probably a bit more hardened, but there’s a lot of headroom in reinforcement learning. We’re probably at a point where it’s 30X in efficiency, depending on how good your model is at actually improving itself. So that is one thing that’s happening—

30X per year, or over what?

Yeah, something like that.

Per model generation? Okay.

Yeah. It depends on what metric it is, but I do think that the speed is probably 4 or more times faster than researchers just doing it alone, and that will probably accelerate.

The amount of intelligence you can extract per training FLOP is increasing, so at some point that asymptote is okay. Also, the amount of revenue that each FLOP can generate increases, right? The more intelligence you pack, the more value you can create.

Going back to the question of how this distributes across pre-training and reinforcement learning, to train Beam, which is a 500-billion-parameter model with 23 billion active parameters, it took 6,000 GB300s. We ran it, I think, for a few weeks, but now, with infrastructure efficiencies, we can do it in about 12 days, maybe less.

Mm-hmm.

Reinforcement learning was a little over 10,000 GB300s for 4 weeks. So there were actually more FLOPs spent on reinforcement learning. Reinforcement-learning jobs are more complex in the sense that you do both a lot of inference at scale, with all sorts of sandboxes for agents, and training.

The reason I got into AI was that I saw my co-founder’s work on AlphaGo. I was a physicist at the time, and there was a moment—a plot in AlphaGo where it just never stopped improving. It kept improving, and they cut it off at some point because what’s the point of improving it further? You already beat the world champion.

Yeah.

Mm-hmm.

I thought that if you figure out how to apply that recipe to stuff that is economically valuable, then it becomes an economic question of how much money you want to put in to keep improving the system.

We as a field are there. One of the things that I’m really excited about in this open-source project and the tech report that we’ll have with it is that we’ll describe how these things are built. Our reinforcement-learning system never stopped learning. If you look at our plots, they just keep going up, and it’s basically just a matter of compute in terms of scaling it further.

That is to say, we as a field are there, where you have these RL systems that are very general and don’t really stop improving. You want to make them a lot more efficient to extract the most intelligence from your compute. But that means that you move from the question of whether you can train these things to where you can get the data and what is economically valuable.

It’s just an economics decision of how much money you put into X to get some kind of improvement out of it, which is incredible.

Yeah, yeah.

This was not true 2 years ago.

What can you share about where you’ve already decided, “We should turn the crank” in terms of continuing to invest in capability?

This is, to some scientists’ dissatisfaction, where things have moved. It is a science, but it feels more like an engineering discipline. Again, the rocket-ship analogy: building rockets is a science, but most people think of it as an engineering discipline with some scientific work in it. That’s roughly how it feels.

The stages are kind of set in that these are the axes of scaling today. There might be something totally blue-sky that we don’t know about, and there are some new labs exploring it. But it’s pre-training, synthetic data, and reinforcement learning.

Mm-hmm.

Or, in one sense, you can say just training and reinforcement learning, and there are architectural improvements that can be made. There are data improvements and algorithmic improvements, but it doesn’t feel like the Wild West the way it did 5 years ago.

Something beautiful about both DeepMind and Google Brain, and even if you look at early OpenAI, is how diverse the bets were. People were doing crazy, fun stuff.

Mm.

Now it’s really narrowed in, and part of it is also a bit of a hardware lottery, because once something starts working, the hardware also starts co-optimizing against it. It’s harder to find—even if you have a great new idea, if it’s not really a good fit for the hardware, it doesn’t make sense.

To answer your question, there hasn’t been an asymptote in how much juice you can get out of better pre-training. We’re not seeing a slowdown in compute-efficiency gains. You keep seeing new things.

There’s a lot of headroom in reinforcement learning. I think that one is probably where there’s a lot of stuff to be discovered. But again, it feels more like an engineering discovery than a groundbreaking understanding of a new science or something like this.

4. Finding Economic Value

You mentioned investing against some economic activity or some productivity out of this that then allows you to keep going from a capital perspective. I know, for example, Anthropic placed a pretty early bet on code. Obviously, OpenAI did that too. They originally had a coding model that they were working with GitHub Copilot on, and then they obviously diversified into consumer and other areas.

Are there specific application areas that you all are most focused on from the perspective of economic value creation? Is it code and agentic workflows? Is it something else? That seems to be the basis for a lot. I’m curious if there’s a hypothesis on where your model will be used the most, or where you need to direct it most.

Yeah. I think the code and agentic stuff kind of sets a foundation for the intelligence, and the intelligence is ultimately—it’s interesting. It’s jagged in the sense that, one, it’s generalized in a surprising way.

I was surprised by how quickly our model got to a level of capability that is fairly fresh and recent, right? Other labs were discovering it not so long ago. There is some kind of stitching, some generalization happening.

At the same time, it is somewhat jagged. It quickly adapts to data once you have the right data for the tasks you want to do. But there is a kind of jaggedness about it, even between the various benchmarks, like the various versions of Terminal-Bench. You don’t see clean generalization between different harnesses. But once you have something working and you have a general agentic capability, and you get the data for a new harness, it actually starts adapting to it pretty quickly.

So what that means is that you have this pretty adaptive thing, and then it’s a matter of where the economically valuable pools of data are. That’s part of what makes open models powerful, I think: enterprises and customers can take them and customize them for their own stuff and get something that is kind of Pareto optimal for their workloads—the lowest cost for the highest performance.

It’s an empirical question. You go to customers, you actually see what is valuable to them, and you see whether it’s possible to generate data. Usually, when it’s economically valuable, you can generate data because it’s not even that you’re going to be training on the customer’s data. It’s more that you’re setting up evaluations, and if you can set up a good evaluation for their tasks, then you can generate synthetic data that approximates it, and you get good generalization.

Concretely, in finance, various know-your-customer flows and compliance flows are very valuable. A lot of different things in cybersecurity—cyber defense in particular—are quite valuable. Legal, too. There are all sorts of agentic verticals within enterprise that are valuable and have a pretty similar pattern in terms of how you get something working there.

One thing that stands out in Beam is the reasoning efficiency of the model. Could you tell us a little bit more about that?

Beam is a model that was trained for coding and agentic tasks. That’s where it excels. An important thing in model building is not just the capability, but how quickly an agent achieves a thing. It translates to faster workload times and cheaper costs for customers.

If you’ve ever sat around an AI—your favorite AI chat—and asked it something, and it took 10 minutes, it’s much better if it does it in 1. Beam tends to be 3 to 4 times more efficient than models of the same capability class, and much more efficient than larger models out there, where the efficiency gains end up being something like 10x.

The reason it’s so efficient is because we prioritized both a strong pre-training base for reasoning and then amplified it with reinforcement learning at what we believe to be the largest scale that’s ever been done in open source. I’ve not seen a documented 4-week run on 10,000 GB300s yet.

Reinforcement learning is set up to extract the maximum amount of capability in the least amount of time. This is the same thing that happened in previous systems like AlphaGo. The first AlphaGo agents were pretty meandering in the way that they were solving the problem, and then by the time you got to the Lee Sedol level, it was just very smart in its search. That’s really what enabled it to have that capability. The more reinforcement learning you run, the higher the capability and the faster these systems solve it.

Mm-hmm.

Awesome. That seems very pragmatic.

The intuition, I guess, comes from the fact that we are RL believers, so we always believed that the path to AGI was through reinforcement learning with a really strong base. We forget that the first AlphaGo systems had been trained on expert-amateur human games, so they did imitation learning first and then reinforcement learning.

Mm-hmm.

So you need those things working together. A by-product of it is that you get this really economically valuable thing that is very reasoning-efficient and makes a really nice workhorse model for enterprises and public-sector sovereigns.

5. The Open Intelligence Business

From a monetization perspective, there are different ways to commercialize both open-source and open-weight models, and Mistral and others were early in terms of different approaches to doing so. Is there a specific direction that you all have been heading and that you can share?

I think that commercialization of, let’s say, open and closed models is in some sense pretty similar. What you’re really trying to do is maximize inference. The difference is basically rental versus ownership of inference.

An example is that when you’re buying a token, you’re renting a piece of a whole stack, which includes the harness—if there’s an agentic harness in there—the model, the inference software, the cluster management software, the GPUs that it’s running on, and all of that. It’s amortized into a token, and you’re renting a piece of that whole thing.

When you want to own your intelligence for a number of reasons, I liken it to how we typically start off renting our apartments, but then as you grow up, you want to own a house for various reasons, even if it’s a bit of a headache, right?

Mm-hmm.

It can be.

Yeah.

As you get to a point where you’re financially mature enough and you have a family and so forth, you want to own it. AI has matured in the commercial market to the point where enterprises are spending a lot of money renting their intelligence and then want to start owning it for various reasons around control.

Going back to monetization, what’s the difference between a closed and an open model? The open model is permissive. But in order for a customer to make use of it, they need all the other stuff that went around it. The stuff that you take for granted from a closed model—you need the cluster management software, you need the inference software, you need the harness, and so forth.

We provide large enterprises and sovereigns with all the tools that they need to make open-model deployment successful. Even when we think about services, it’s because a lot of enterprises—most enterprises—need some hand-holding on this. We think about services as how we go in and unlock really valuable use cases that drive a lot of compute demand, or inference demand. In a sense, services around open models are a demand driver for an inference business.

What do you think—I won’t ask you to project too far because it is just very hard—but over a 1- or 2-year time horizon, what will the mix of tokens that is open versus closed be?

I think you’re starting to see the trajectory now. Maybe 6 months ago, it was majority closed and minority open when you went to any gateway like OpenRouter or Vercel, and it’s flipped almost exactly from 70/30 closed/open to 70/30 open/closed now. I think that’s only going to accelerate.

I suspect that the world is going to look not too dissimilar from operating systems, where 95%+ of servers and computers in the world run on an open-source operating system like Linux. That doesn’t mean that the closed stuff isn’t very valuable. Microsoft and Apple are extremely valuable companies, so the market for this stuff is really big.

I think we’ll see most token demand going to open. A big difference here is that I think we’ll also see a lot of economic value going to open simply because, even if the model is fully permissive or perhaps has a license on it, everything else you need to run it is still expensive. The compute is still expensive, so you’re in a different world than CPUs. Hardware accelerators like GPUs and others are a more expensive compute substrate than CPUs, and so you have a different kind of cloud that needs to be built around it.

I expect the majority of tokens to be going to open source, extremely valuable closed-model companies to exist, and a valuable ecosystem of open-model companies and various distributors around them to exist as well.

Do you have a sense of what proportion of the open tokens are RL-trained versus not, versus sort of base models?

I think every single open model that’s out there at this point has been RL-trained to some extent.

Oh, yeah. But I mean for a specific use case or application.

Yeah.

In a custom model—

Yeah. So I think that—

—is the term today.

What I’ve heard—the statistics that some of the open-model-dedicated inference providers cite—is that it’s actually a vast majority, 90%+, customized.

But there’s a caveat there, because today the biggest customers of dedicated inference are AI natives—

Mm-hmm.

And maybe digital natives who have a whole product that is just one big customization of an open model, like Cursor, Cognition, Bridge, or Harvey—

Mm-hmm.

These kinds of companies have a product that’s built around something that’s big and customized. So that would make sense. If you’re supporting those workloads, the majority of them would be fine-tuned.

I actually have a different take on what’s going to happen in the enterprise. I think that the majority of enterprise token consumption is going to come not from customized models, but from customized systems. Meaning, you took an open model, you didn’t actually fine-tune it yet; you just customized a system around it, like an agentic harness, to make it work for some KYC flow or something like that. Once you’re sophisticated enough, you might start fine-tuning and moving to that.

But I actually think that in the enterprise, it will be reversed, in the sense that the journey that AI natives went through—which was start with a closed model, then move to an open model, then customize it—they went through it very quickly.

Mm-hmm.

They set up native products that are data collectors. Enterprises are mostly architecting this around their existing products and existing tools, and I think there are a lot of frictions there to just do a straight shot to fine-tuning. So in the enterprise, I suspect it’ll be a bit different—

And do you think they’ll still start with state-of-the-art models, or the closed models, and then move over? Will the harness be the mechanism by which they customize the open model?

That has been our observation: the point at which an enterprise is considering open models—and most of the enterprises we talk to are considering them now—is by the time they’ve ramped up some significant workloads with closed models. I have not seen basically a straight shot, a kind of sprint to an ownership model. You have to go through the rental stage to become an owner.

It has to become a big enough cost driver for your enterprise, and then you say, “How do I cut costs? This is valuable enough for me to keep.” So then I switch over as effectively—

Yeah. If you’re spending, let’s say, $100 million-plus on closed models a year, then you start thinking, “Maybe I should figure out a more optimal way to do this.”

I think there’s an interesting resurgence of on-premises infrastructure, partly because there’s a compute shortage. If you’re an enterprise and you’re consuming from your favorite hyperscaler, very often there’s just not enough compute. It’s hard to get good rates there. But you’re spending so much money, so you might go with an infrastructure provider like Dell and say, “I just want to set up the bare metal, but I want to serve stuff into my enterprise.” Then what happens? Who helps them with the layer between the bare metal and actually making them successful?

And you view that as your role in the open-model world?

Yes.

Yeah.

Yeah. We view our role as enabling enterprises to build successful solutions, so it is very solutions-driven. The thing that enabled AI natives to adopt open models was building their own solution. Enterprises just need more help there.

It’s really around going in and assessing what the biggest-value things are that you can do there. A lot of these enterprises might have built an agent that spans hundreds of millions of customers on closed models that they really want to scale out. The scaling factor is very high. It’s very hard to go in and just say, “We’re going to give you inference.” That has not worked, you know?

Yeah.

So.

Yeah, yeah.

The closed-model ecosystem is a competitive one. It looks increasingly like the open-model ecosystem will also be. Tell me if you disagree with that claim as well.

No, I agree.

It’s all very competitive.

Large markets tend to be. What do you think are the important dimensions of competition when you think about yourselves and how you position in the long run?

I have a pretty simple formula for the way anyone ultimately wins in this. It’s how much durable revenue you’re generating, and the formula is what intelligence density you’re able to offer, times how much compute you have, times how much trust you have with organizations that would want to work with you. Basically, how good are you at solving their problems?

Something that’s interesting about this market is that compute is scarce, so a lot of things that previously weren’t limiting factors are now limiting factors. Maybe this is a temporary thing, but I don’t see it as being temporary for some time. In previous competitive instances, you might say, “There are these few players who are all undifferentiated,” but then it turns out that if you’re really good and you have the compute, it’s okay.

Yeah.

I see the whole space being competitive across the whole stack. There are arguments around one part getting commoditized and another part getting commoditized, but in reality, everything is getting commoditized. Everything across the whole stack is so hypercompetitive that it’s getting commoditized, and the model margins are going to be compressed.

There’s a tension between open and closed models that does lead to margin compression. The application stuff—margins compress. The inference layer and the bare metal, too.

Mm-hmm.

We see a lot of the AI-native companies negotiate their closed deals very aggressively because the open ecosystem exists, so it’s explicitly true.

Yeah. So I think that for any one of these companies—and you can think about any AI-native company in terms of how much intelligence density it serves to customers, how much compute it has, and how much trust it has with those customers, because they’re driving—

Mm-hmm.

—successful solutions for them. If those are the 3 factors, then so long as there are great open models—so long as you have great intelligence density out there that is accessible—there should be many successful companies, because then it’s a matter of how much compute each one can assemble.

Now, we are actually in a world where, when you’re a great intelligence builder, as has happened with the closed companies, your revenue really shoots up and you’re able to amass more compute than anyone else. Then that puts you in a strategic position. That is what I think is required to win. You have to serve great intelligence, have a lot of compute, and have trusted customers.

Yeah. How do you think about this? If you look at the era that you mentioned at the very beginning of this conversation, 2 years ago, a lot of open-weight or open-source models were Western in origin. Mistral was developed in Europe. Obviously, Meta was a mix of Europe and the United States.

What’s happened over the last year or 2 is that we’ve really seen the rise of Chinese open-weight models. In some cases, there’s the perspective that they’ve been distilling a lot from the state-of-the-art models, and that’s part of what’s allowed them to make very rapid progress.

The labs are now trying to roll out—or the big closed labs are trying to roll out—tools to prevent as much distillation from happening, or at least make it more challenging. What do you think happens over the next year or 2 in terms of Chinese open-source or Chinese open-weight models? Do you think it remains where it’s at? Do you think it evolves? What happens?

I think the first thing is that the fact that a great open-model ecosystem came from China is actually a massive benefit for the world. There are so many companies in the West that have been able to build more durable businesses as a result of that.

There’s a sense in which it’s a very positive thing and a gift, because you could imagine a world where such a thing didn’t happen, and then there are just no open models. There’s a kind of tall poppy syndrome that happens: if you’re an application builder and you build something that’s great, it gets subsumed into a closed-model provider because, well, it’s a business. It needs to generate revenue.

So I view it positively. The thing is that you don't want a world that is kind of monopolar in any given way, right? That you either have all sorts of resources and compute concentrating across closed labs or a handful. Let's say if it was 10 or 20 closed labs, then fine, you have a competitive ecosystem, but if it's one or two, that's a bit scarier.

And similarly, you don't want the source of intelligence that everyone else can build on when they want to own it to be coming from one country. Ideally, it'd be 10 or 20, but the capital expenditures are very high, so at least 2 would be great. And so I think it's extremely important that there's an ecosystem that competes with China in the West, but it's not really a West versus China thing. It's more that if there was one country other than China where all the great open models were coming from, you'd probably also want some competitive tension. Competitive tension is just good.

I think that the Chinese models will continue to be great. They have a number of advantages and a number of disadvantages, but the advantages are that they certainly distill closed models at industrial scale. That is true. They have access to data that is much cheaper and free because you can just train on PDFs that are copyrighted in China, and it's okay—just different regulation. And I think there is a sense in which those companies do have some state support, whether directly or indirectly. These companies were not making any revenue for a long time.

Right.

Yeah, exactly. I think that continues. Now these companies are turning into businesses. While the models are open abroad, within China they're kind of the equivalents of the closed model labs here. So there are real businesses being built there, and I think they'll continue building great models.

But do you think they're going to keep them open here? In other words, why do that if you now have an economic driver and pattern? What do you think is the incentive for them to keep the models open in the West?

In the West, or in the—

In the West.

Yeah. This is a really interesting question, right? Fundamentally, there's a lot of demand from enterprise, public sector, and sovereigns for Western open models for a number of reasons. But there are regulatory fears or, rather, uncertainty. And it is the case that it's not just that you want a model, but a partner that will help you serve that model.

We have seen that most are Fortune 500. The open-model footprint is actually pretty low today because of resistance or aversion to Chinese models for a number of reasons, some rational and some irrational. So it's very clear that building those things is important. If you can sustain an economic engine that, one, builds a lot of trust so that the enterprise wants to work with you as the model builder, and, two, enables you to secure compute, then, if there's a commercial engine around this stuff, then there's an incentive—

I totally agree that having a Western open-weight model that provides incremental services and potentially inference or other things over time is really valuable, and I know that's the direction you all are heading in. I'm just a little bit curious in terms of what the incentive system is for the Chinese model companies to continue to leave the models open—

So the question is on the Chinese side.

Correct.

But, yeah.

Yeah.

I think my hypothesis on that is that China is somewhat of a captive market, so you can build open models and still make a lot of money in that market.

Something that's interesting is that in the West there are plenty of great companies that don't build open models but serve them into various companies. I'm not really aware of that dynamic happening in China as much.

Mm.

You'd think it would. Why not? So there's something interesting there where the open-model providers are kind of the intelligence providers in that country.

Mm-hmm. Mm-hmm.

And I think that continuing to release great open models is very geopolitically advantageous to China for a number of reasons. But one is that you want, as a country—and this is true for America and China and any other country that has the capabilities—you want other countries building on your stuff.

There have been previous escalations or tensions between the U.S. and China in 5G, fiber rollout, and the Belt and Road Initiative. Ultimately, when you go in and provide something cheap to another country, it then gets locked into your infrastructure.

Mm-hmm.

There are just commercial and geopolitical advantages for the country that—

So I think it's almost like rare-earth minerals or something. You have a component that other people will use for all sorts of purposes, and therefore you have geopolitical leverage through that.

Yes. One way I think about it is that open models are Trojan horses for the infrastructure that they bring with them. So you have an open model, but that alone is not very useful.

Mm.

So you buy into the entire software ecosystem and infrastructure ecosystem of a given country. So that would be—okay, maybe today an allied country, a U.S.-allied country X, might run a Chinese model on American chips. But pretty soon—and we're seeing this already—Chinese companies like Huawei are going to be coming in and offering a full-stack solution, and then locking into that country's supply chain.

Mm-hmm.

So I think that if we think about AI, or chips and data centers, almost as railroads or something like that, it's fundamental infrastructure. But no other country can actually build its own railroads. They all have to go to you—

Mm-hmm.

—right, or another country. You have a lot of geopolitical leverage—

I see.

—being a provider.

So it could just be, for example, inferencing a specific model will occur in a more performant way on a specific chipset, and that chipset happens to be provided by Huawei or something.

Yeah. These stacks are in real time getting optimized end to end because, in some sense, the paradigm has been set. There are going to be other ways to scale intelligence, possibly, but this is clearly the first one that really scaled, so now everything's being optimized down the entire stack. China has a disadvantage today because its chips aren't as performant, but it has an energy advantage, and its companies are very good. They're catching up on the chip side too.

And that's certainly going to accelerate with the models, right, in terms of model usage to design next-generation chips and things like that.

Yeah. Certainly. And then I think there is a thing in terms of it being, again, geopolitically competitive with the United States specifically: the notion of American companies and startups building on foreign technology, this notion that the rest of the world builds on your stuff.

And by the way, America has done this with all other meaningful technologies, like the internet, open protocols, and open source. The world basically uses American currency; it's the widely established global currency. Free trade works because we have a navy that's everywhere. So that's actually been the American playbook for a long time. It's interesting that in this case, I think the United States has been on the back foot when it comes to open source, but—

No longer.

Yeah. Well, we have to be clear: the competitors in China are just really, really good, and so there's an ecosystem that's just starting to form now in the United States. But there is some catch-up to be had.

6. Openness Is The Safer Path

The biggest criticism from the closed labs of open models tends to be safety and controllability. What's your philosophy on this? And is that—

Yeah.

—a legitimate concern?

No, it's definitely a legitimate concern, and I think that…

So I do think that the safety worldview that has been established, you know, from one particular perspective with one particular point of view, has become almost dogmatic. If we were looking at AI from first principles, before considering whether it was closed or open, and you had the foresight to know this technology was going to have these capabilities and so forth, and you were thinking about safety, how would you actually design it from scratch?

One analogy to take it to is software. You can think about AI as a more advanced version of software. It has many similarities, actually. It's a digital utility.

There were debates, by the way, around software being dangerous, particularly in the early 1990s around strong encryption protocols. They were actually closed at the time, and there were debates between the NSA and civil-liberties advocates around whether you should close them or open them. Ultimately, after some catastrophic failures on the closed side, where effectively a small handful of engineers designed certain systems that had unintended consequences they couldn't predict and that were easily hacked, strong encryption protocols became open. That actually gave birth to the whole field of cybersecurity.

So the way the world works—the default state of the world—is that openness is safety. That's the default state that we'd be going into AI with if we were just thinking about the continuation of technology.

Models actually have a lot of similarities to software, but the issues are exacerbated. The long tail of vulnerabilities is larger. It's harder to understand than software because it's a black box, and the long tail of vulnerabilities—and hence unintended consequences—is larger. Linus's law, from the creator of Linux, is that with enough eyeballs, all bugs become shallow.

Mm-hmm.

And I have the belief that with enough eyeballs, most security and safety vulnerabilities become shallow as well.

Mm-hmm.

The state of the world today is that we have a few hundred safety researchers within closed labs who understand how these things work. Despite their best intentions, it is impossible to cover the long tail of vulnerabilities or unintended consequences that these systems might have. The symptom of this is that when we look at what major cybersecurity issues have surfaced, it's that a very powerful closed model had unintended consequences, where it went and hacked into another company.

Mm-hmm.

And the only way that company could remediate itself was by using open models to protect itself.

Mm-hmm.

Right? So that's the empirical evidence of the world that we're in.

When you say safety, by the way, because I feel like people really conflate notions of safety, and safety means three or four different things to different people. There's safety in terms of cyberattacks, the use of AI for hacking, or other things. There's safety in terms of what I think tends to be overstated in the short run, around bioweaponry or terrorism. Then there's safety from the perspective of an existential threat to humanity.

People talk about these things as if they were one thing—

Yeah.

—and each of these is separable. So when you're talking about safety, are you addressing all 3 of those? And, by the way, I don't necessarily agree that these are all true things that are going to happen in any realistic timeframe. I'm just a little bit curious how you think about the span of things that benefit from openness and benefit from these approaches.

Yeah, I think on the safety spectrum, there's basically a spectrum from empirical reality to theoretical scenarios. I will grant that there are sci-fi bits to AI, where something that was totally theoretical last year is now a reality, like the cyber capabilities of these models, for example. So it's not to discount the theoretical stuff, but we have to be clear that there's real empirical stuff that's happening and that we can predict with some confidence will be happening in the next 6 months.

Mm.

And then there is maximal theoretical stuff.

Yeah, yeah. I mean, the prior versions of that, for example, would be the thought that the atmosphere would catch fire the first time we set off a nuclear weapon. So there's that theoretical fear. It didn't happen, but there was a lot of churn amongst a small subset of researchers in the physics community around that, as an example.

Yes.

I remember when people talked about nanotech, they used to talk about gray goo and how you'd accidentally release a nanobot, and then suddenly it would eat the entire world. This was something that was discussed in the '90s—

Yeah.

In labs, as people were working on microfluidics—

Yeah, exactly.

—and things like that.

The challenge with AI is that it's so top of mind, it's permeated everyday culture, and it's so easy to humanize. From a perception perspective, it feels like there's not even a uniform distribution around what safety means; it's actually peaked at the doomsday scenarios. Whereas the reality is that there's probably a serviceable peak to address—the reality—and then a decay into the theoretical.

It doesn't help when leaders of companies say, "On the theoretical side, there's a 10% chance that we all die." That doesn't help in any case.

Yeah, that's not substantiated by any specific—

Right.

It's kind of a made-up number. Yeah.

It is. So I tend to have, again, as a scientist, distributional thinking, which is peaked at reality. Then, 6 months ahead, I'm definitely worried about misalignment—these unintended consequences of the systems as they continue to get more powerful.

But we have to take the perspective that these are engineered tools. I think there's a certain class of companies or researchers who claim that things around consciousness or emotions and so forth—which is maybe true—but it's so hard to define what those things are in particular that it's unclear what they are. So I think about these as intelligent tools and have a reality sort of spectrum.

While I think that the theoretical doomsday stuff is good dinner conversation, the thing is, when we say, let's say, alignment, that's a big part of safety. How do you align these systems to behave in intended ways? The reality is that alignment so far—the science of alignment—has been deeply boring and unsatisfying from that perspective. There's no magical alignment equation. It's a whack-a-mole thing.

If you remember the early models before ChatGPT, there were some models released that were pretrained and then toxic.

Yeah, yeah.

Mm-hmm, mm-hmm.

And then there was this notion that, "Oh, these models are toxic. Can't fix it—

Mm-hmm.

—because they can be toxic in so many ways."

Yeah.

But it turned out that with enough post-training data—

Yeah.

—you can just patch all those things, and it generalizes in a way that it actually becomes pretty hard to jailbreak.

And now you're seeing the same thing. The reality with these alignment issues is that they're probably going to be very mundane and boring. You've discovered a bunch of these vulnerabilities. You have data that patches them up. You have some algorithms or some—

Yeah, it's a classic.

Yeah.

And when you say algorithms for detecting them, it's typically asking a language model to detect this thing.

So there's the theoretical alignment and safety philosophical stuff, and then the mundane reality that you're just patching up all sorts of bugs. Then it becomes: shouldn't there be 100,000 researchers and computer scientists looking for and patching up these bugs? Wouldn't that be safer?

I feel like there's an important philosophical distinction, which it sounds like you're on one side of, but I'll let you be explicit about it. Even if they are very intelligent tools, and if you just say we can, as an ecosystem, manage the unintended consequences, we don't want individuals and businesses to have such powerful tools.

That is coming from a standpoint that we can have access to powerful tools, but—

Yes.

But other people can't because we're effectively better, right? We're better at managing these tools. Well, it's kind of—look—

To be fair to the claim—I think this is not my belief—it would be that the negative impact that any single person or company could have is too large, and that's scary, but we can have them.

Yeah.

Yeah.

Well, it's also on a scale of weapons, right? We have the US government selling jets to other countries, right? But more broadly, it's not so easy to take a model—especially an open model that's been trained to be safe—and put in the compute, work, and effort to make it very dangerous.

I'm not saying it's impossible, but again, when we think about cybersecurity, very intelligent hackers can go and wreak havoc, as they have. But you have an ecosystem where the defenses tend to outweigh the offenses. And again, if we come to that as a default—that this is the way people secure software, and that there are good and bad actors on the internet—ultimately, the internet is an ecosystem that's kind of like having an ecosystem of white blood cells.

Enabling more people to have defensive capabilities actually helps you protect against offensive capabilities. That's right. When we look at the empirical reality of what happened, it's really hard to separate cyber defense from offense. So when you remove cyber offensive capabilities, you also remove cyber defensive capabilities, and as a result, the players who would want to help defend are incapable of doing so.

Mm-hmm.

And we saw that in some of the recent things that happened with the OpenAI–Hugging Face incident, where they reverted to using open source—

Right.

—because they weren't able to use the existing state-of-the-art labs because of the guardrails—

Right.

—that were placed on the models.

Yeah, I think these arguments go into a place where they're very absolute, where you kind of say that these things are so powerful that no one ever should have access to them, except for us.

Mm-hmm.

And the reality is that that's never the case. It's hard to think of cases—except for some really rare exceptions where there are actually not that many positives—where such absolutist perspectives worked.

7. AI Accelerates Scientific Discovery

What are you most excited about if you think ahead 2 to 5 years in the AI world? It's very hard to predict right now, but as you think ahead in terms of the curve of technology and the curve of adoption, what do you think are some of the positive things that you view as coming?

There are so many positive things that are coming, and they have been coming. An example is that I'm personally very excited about scientific progress. I got into science when we moved to the States, developed an interest in physics, and have always had this lifelong pursuit of science since then.

One of the prompts I was trying with language models over the last few years was to give it my PhD thesis. Granted, getting to the point where you have the right question for the PhD thesis is kind of a lot of the work. But actually executing the rote work takes a really long time as well. Overall, between finding the right question and executing the answer, a PhD, I think, took a few years.

I was asking language models over the last couple of years, giving them this prompt with my thesis. A couple of years ago, it was just chat, so it couldn't do anything. Then, a year ago, it started answering things at an undergraduate level. That would have been my answer if this had been given to me as homework as an undergraduate. Six months ago, it solved it right at a real PhD level and solved it correctly.

Uh-huh. Yeah.

And now, when I actually try it, it even gives me some interesting new information that I hadn't considered at the time. We've gone—

Yeah.

If you have the right question, you can just put it into a chat box and have it do all the calculation and give you something really interesting.

That's so exciting.

Right. Which means that your iteration speed—why do you need a few years? What's the point of a PhD that takes a few years? You can do a PhD in a week, right?

Yeah, similar to the recent news from OpenAI—I think it was in the last 1 or 2 days—in terms of all the various mathematical theorems that were proved in the last couple of weeks by OpenAI. It's kind of crazy what's been happening.

No, it's absolutely incredible what this is going to do for science. I think the other thing I'm really excited about is not just theoretical science, where the environment is basically the equivalent of a chalkboard or something, but real-world science: life sciences, materials science, and chemistry.

I think there are a lot of places where you can have an AI interfacing with a real experiment, where you can set up proprietary data flywheels. It's going to be a bit slower—much slower than the chalkboard stuff—but dramatically faster than what it was before.

I think we continue to underestimate how dramatic of a shift this has been for software engineering and how much more you can build. It's already a very incredible and positive tool in many ways, including at the blue-collar level.

I toured Stargate some weeks back, and the thing that struck me was the number of cars in the parking lots. There are so many people working there. These data center projects create tens of thousands of high-paying jobs. To me, it seems like a data center is now kind of what a factory used to be, right?

You had railroads and factories in the 20th century. Maybe you had a shoe factory that made a town produce a vibrant economy within a town. At least when I'm on the ground seeing the stuff that's getting built, there is fear of data centers, but then the reality is that you go on the ground and it actually creates a lot of jobs and a lot of tax revenue for the local communities.

There are certain things that I think need to be done very tactfully. These things are noisy, so you need to insulate the noise or put them in places that are less close to residential areas. But it is certainly the case that for local economies, these things create jobs, generate tax revenues, and just feel like what a factory used to be in terms of job creation.

How do you direct the scientific and experimental effort now, from here going forward? Within that, assuming that you are trying to use the models themselves to improve your training effort, what is the role of the science team today?

The way we direct our research—and a lot of this falls under my co-founder Yannis, who leads research and technology, among other things—is that there are a lot of things we know work that we weren't able to put in. It's just a timeline thing.

When we trained the final model on the SpaceX cluster that we received in July, it was a few weeks of pretraining, then a few weeks of synthetic data on our end, and then shipping it. There is a lot of stuff that didn't make it in that we're really excited about.

There's a trade-off between the careful execution of known things, or bets that have a bit less risk around them, and going on to more risky bets. You want to exhaust all the high-impact things that you know work, and then start expanding the risk profile.

The way you do this is that, on the risk profile, you have scaling experiments where you do things at small scale. I think the interesting bit is that some interesting things don't emerge at all until you're at large scale, so there's a bit of an art and science to deciding whether I'm going to put in more compute to find out the next thing or go on to the next idea.

As far as model-in-the-loop goes, I find it to be very empowering for researchers because things that would have taken a long time individually can now just be done fairly quickly. These models are fairly jagged in their intelligence, so you have to feel where they have a tight loop and where human creativity is still important.

But you get a feel for it pretty quickly, and so the place where you can loop things, you do that.

Mm-hmm.

Especially, there are all sorts of things around hyperparameter searches and some of the more granular infrastructure tasks that these things can be helpful in. I actually think they're very exciting tools because they enable a scientist to exercise their intuition. Right? It's kind of like having a very fast and eager colleague that actually listens to you, if it's aligned correctly.

And so I see a researcher as just getting a lot more ability to move a lot faster than they have before, which is why maybe the rate of improvement annually was roughly, let's say, 7X before when it was just manual work, and now several times that. Say maybe 4 to 5X. So it's actually really exciting for a researcher to have these tools. There's this question around at what point you don't need a researcher in the loop.

I was going to ask you, if you need 100 people today, do you need 20 next year for the same effort?

For the same effort, yes. But I think you're constantly expanding the ambition of what you can do, and so we find ourselves constantly understaffed rather than overstaffed. There is a staff-to-compute ratio that you need to keep in mind, so it's not like it just goes on forever. I don't think 3,000 researchers would be very helpful, but a couple hundred would be very, very helpful.

Yeah. The argument I've heard is that, given the amount of compute you need to allocate per person—and especially, to your point, if you want to scale certain experiments up over time—coupled with that, there's often a bit of a distribution in terms of contributions. In other words, there's a handful of people who contribute the most from an idea perspective in every field, right? Physics, biology, whatever, eventually collapses into: you want to give every incremental piece of compute to the most productive subset of people, and so therefore, you end up with some static or shrinking number of researchers over time. That would be kind of the argument in the extreme in terms of where you end up.

Yeah.

I'm not saying it's correct. I'm just saying that's kind of an argument I've heard being made.

The argument definitely has merit. These projects actually never needed a crazy amount of researchers. When you think about projects like AlphaGo, at the time it was maybe on the order of 10 people. Now I think that maybe for large efforts like the one we're doing, maybe on the order of 100 or hundreds. So it's never been the case that you needed an extraordinary number of people.

And the question of whether it collapses back into 10 or not is hard to say. I would estimate that there's going to be some steady state in that order, on the order of 100. There are just a lot of things to do, and there will be a lot of job creation in applied research: going and taking these things and actually applying them to real problems, I think there's going to be—

I think people really lose that when they talk about engineering or research, in terms of the diffusion of those roles out into a broader swath of society and not just building language models. And so, to your point, there are lots of other model types to build, and lots of closed-loop systems to build. But even when people talk about engineers and the potential displacement of engineers, so far the evidence seems to be that many companies need more engineers versus fewer, there's also the diffusion of a certain level of quality of engineer out to actually deploy the technology in organizations that never would've been able to access somebody—

Right.

of that sort of talent level on a relative basis, or at least not at scale. And so I do think that's kind of lost as people think about the economy. They tend to centralize it into a handful of big tech companies instead of saying, “What is our overall GDP, and how does that get transformed by the diffusion of people bringing this technology out?”

I think that's correct. The amount of researchers you need to do these things hasn't changed dramatically over the years. It's always been: big project, okay, around 100 people. I don't think you need the thousands on a big project. But maybe on different product surfaces, then you need a bunch of people.

What's interesting is that when you look at the distribution of why you even need hundreds of people, a lot of it is also that you have teams that do evals for different things and collect data for different things. It's not by accident that these models have capabilities. There are pods of 5 to 10 people that go and target a particular capability, and within coding there are 5 to 10 capabilities, right? That's for bringing the general capabilities to the model.

But now when you want to take that model and apply it to real-world capabilities, you basically need a pod for any real-world capability, and each enterprise has many, many of them. So I think the job creation for this new type of forward-deployed engineer, who is a bit more scientific and evaluations-oriented and has some intuition for these models and the harnesses—which is actually what the evaluation researcher does, right?—is actually the same skill set as the researcher building the core thing, but just deployed to work on real problems.

That will take as many of those people as possible. That's actually more of a training gap: you need to train engineers to become good at this. We would take as many as possible today.

Congratulations, Misha, on having the first Western model that is in the class of the frontier. It's just been a big step forward for the ecosystem.

Thank you. Thank you so much.

Yeah. Thanks for joining us.

Yeah.