开源如何成为 AI 的骨干 | Inferact with a16z

The fun thought experiment is: if GPUs dropped in price by 99%, then do we get back to a real open-source world?

If moderation is never solved, in the future people will go to open weight by default, because that is where you know for sure you can control your guardrails for trusted use cases.

Can you talk about where vLLM sits in that stack?

vLLM is an inference engine. It is kind of like databases, operating systems, and other critical software that power AGI. NVIDIA, AMD, Google, and other hardware companies make sure their newest chips can run vLLM. In a lot of cases, they use vLLM as a benchmark.

Five years from now, will open-source AI models have closed the gap with frontier models?

Capability-wise, I don't really see a big gap—not even today, because—

Today we're here with Simon Mo, co-founder of Inferact and a lead maintainer of vLLM, the open-source inference engine now running on half a million GPUs at any moment. We're also joined by Matt Bornstein, an a16z general partner. Simon, Matt, thank you so much for joining us.

I think first we should start with open-source AI and the more recent history of open-source AI. vLLM actually has its origins back in 2022, pre-ChatGPT. Your team set out to make a slow open-source demo faster and instead found this pile of unsolved problems. Can you talk about what made serving an LLM so fundamentally different from the ML workloads everyone already knew how to run?

Yep. Good to be here. Serving a large language model is a fundamentally different problem because it requires running on accelerators like GPUs or TPUs. It is a computationally intensive process that requires a lot of engineering to ensure that each user can see the LLM's response quickly and efficiently.

This typically means we need to handle differences in input distribution, how long each request is, output distribution—which is nondeterministic—and batching and scheduling much more. That's at the core of the inference engine.

The team and the project have been around for about 4 years now, but the company is a little more recent. I want to throw this out to either of you. Matt, I know you've known the team and observed the team for a very long time. At what point did you see this transitioning from being a much-beloved open-source project to critical infrastructure, and then to a company?

Yeah, I think you have to go back a little bit. Open source was the norm for AI models early on. We literally had this company called OpenAI, which has become a little bit of a joke. It's not as open as it once was, or not nearly as open as it once was.

But early on, all the frontier AI work was being open-sourced, or at least released into open weights, which is a little bit different from true open source. People could mostly run these models on hardware or computers that they already had.

Simon, maybe you remember: what was the first model that you actually needed to go out and get special software and a special set of computers to be able to run?

Probably BERT. Before that, it was ResNet for image computation—computer vision classification. ResNet already needed to run on an NVIDIA K80, which was one of the first GPUs on AWS and other platforms.

Even at that point, you could still run ResNet on commodity CPU devices. It was just very slow. But for BERT, running it was like, “Wow, you have to run this on a GPU to make it faster and more efficient” for translation or any other task. That was before 2020.

It's so funny thinking about this. All the memories are flooding in. Hugging Face had 1,000 BERT variants on it, and you had to go find the right BERT variant for your particular task.

You're right: some people had their own GPUs and could run it, but a lot of people had to provision cloud services and things like that. That's really funny.

BERT was an early language model, and newer models are much bigger and more sophisticated. They take up a lot more memory and a lot more compute. vLLM, from the early days, was really about running these more powerful models that you couldn't just figure out on your own.

To get us to the present day, I think it would be good to talk about when it really became critical infrastructure for these even larger open-source models, and when we started to see those larger open-source models come into the field.

We really see the criticality of the software stack as being about bringing open frontier intelligence. “Open frontier intelligence” is a catchphrase for the Kimi K2 model release this time, and it's also about vLLM's mission around inference infrastructure.

If you look at when it started to become critical, it was when people began relying on it for their applications and day-to-day productivity. Maybe in 2023, when GitHub Copilot and ChatGPT became something that people just couldn't live without anymore, open-weight models were already becoming a cornerstone of how people lived every day.

Yeah.

At that point, we needed accelerators running open-weight models and open-source software to make sure everything worked well and that people had flexibility and control over it.

It's interesting from a startup standpoint. As you mentioned, with GPT-3 or early ChatGPT, those closed-source solutions were starting to become critical to a small group of people around that time. Open source existed, but it was a bit of a curiosity or an enthusiast thing.

As the frontier has expanded, particularly with closed-source models, more and more open source has been pulled in as critical infrastructure behind it, if that makes sense. At any given point in time, including now, I think models from OpenAI and Anthropic are more widely used and more critical in general than open-source models. But I do think we passed a threshold about a year ago—

Yeah.

—where a bunch of smaller companies, or new application companies, were trying to figure out, “How do I really build an AI company without just being a wrapper on top of OpenAI?”

The answer to that question turned out to be open source. This is what Cursor did. This is what Decagon and Harvey are in the process of doing now. A bunch of other really strong application-level startups determined that they couldn't build just on closed source.

They needed to do their own mid-training, their own post-training, their own inference and deployment tricks, and all of that means it must be built on top of open source. The closed-source vendors won't give you the access to do this.

My read is that about a year ago, open source became really central in a way that's not always visible because it's deeply embedded in some of these products. Some of the most innovative products and applications now depend very deeply on it.

Yeah. Can you talk about where vLLM sits in that stack? When we have these larger enterprise companies choosing to use open-source models, where does vLLM sit in the stack for them?

Yeah. I mean, just about everybody uses vLLM. Simon, you should describe it.

Just about everybody uses vLLM. It is an inference engine. Its job is to turn available GPUs into a running endpoint for intelligence.

It's kind of like databases, operating systems, and other critical software that power this economy—or power AGI—that everybody really uses today to ensure cost-effectiveness, efficiency, and reliability, while also always staying on the frontier.

vLLM supports more than 1,000 model architectures that are active today. A lot of those are proprietary, but a lot of them are open weight as well. When those model architectures transition from research prototypes to globally accessible open-weight models, they are live on vLLM immediately. That's a process we call day-zero model release.

Additionally, vLLM works closely with all the hardware vendors, including NVIDIA, AMD, Google, Amazon, Intel, and many more. Their newest chips make sure vLLM can run on them. In many cases, they use vLLM as a benchmark to make sure the chips run well.

This fusion of where models run and where hardware meets the models is where the magic happens. That's where vLLM meets the hardware.

You've told me some of the behind-the-scenes stories. It's actually not easy these days when a model releases. There's a lot of human drama in addition to the technical work. Are there any stories there that you think are okay to share?

It's actually a very fun co-design process because, from the model labs' point of view, these are brilliant researchers who have built this model.

Now, their biggest question becomes, “How do we get this out into the world and make sure everybody’s able to use it and run it well?” We have worked with model labs that, because they already use vLLM in production or in their research process, will just dump everything for you. This is a moment when we go to them and say, “Hi, we’re the vLLM team. We would like to support your open-source model. We would like to offer, in a way that’s kind of open source but white-glove service, to get your model running well on our architecture.”

And then you turn and look at the model labs like, “Oh, we got it working already because we’re running it for the RL process. Here you go—just review our code and merge our pull request.” On the other end, we really have model labs that just don’t know how this can work.

Because systems isn’t their core.

Yeah, because systems isn’t their core, and they have been training, or maybe they have their internal inference engine, and they just don’t know how it will adapt to the open-weight ecosystem. By the way, this is also very much a multiparty involvement process. Every model release typically involves the model lab, a primary or secondary hardware vendor, us, and Hugging Face, which is the model-format and model-hub vendor. Depending on the appetite of the model lab, it can involve 10 or 20 different kinds of release partners.

These could be inference clouds or public hyperscalers—whoever is going to run this model. You want them to ensure the success of the model and run it successfully. Even up to today, if you look at the Kimi K2 model release, it’s a whole partnership and a drive to make sure that, once the model is released—because it’s just a few terabytes of files sitting on the internet—people are actually going to be using it really, really well.

Even from the beginning of 2023 and 2024, if you remember when Mistral dropped their first model, they just dropped a torrent link for Mistral 7B, and then everybody was struggling and trying to get it up and running. We were working behind the scenes with the Mistral team, trying to get the inference engines working in vLLM. This was probably the most exciting weekend early on that we were able to spend on this.

After the weekend, when everybody was trying to analyze what was going on, on Monday or Tuesday, Mistral and we just announced, “Here, you can run it successfully on vLLM.” Everybody was able to easily reuse a lot of the work and start building on top of it.

That was sort of a fun time when enthusiasts like me could just scramble to download the model and get it running somewhere.

Yeah, yeah.

I’m glad the professionals are taking over. [Laughter] It never worked very well, but it was a fun moment in time.

Yeah.

So, to bring things forward to the present, I think open-source models and also distillation have been in the news recently. Inferact signed the NVIDIA Open Weights and American AI Leadership Letter, which was also signed by a16z, Meta, Amazon, and dozens of other companies. Can you just talk about your decision to sign that, and what you were really responding to in the market and in the news?

Yeah. For us, what we really want to stand behind is that open weights absolutely matter in the ecosystem. The world cannot just be controlled by proprietary APIs, with open-weight model development and research blocked or banned. The pledge that Inferact signed up for is that we want to help foster this ecosystem, where we are typically a little bit downstream, right? Inference engines are not part of the pretraining process, nor the RL process, but they are where the model actually meets the world.

From what we’re seeing, people are using their imagination and their ability to materialize that imagination with open-weight models. They’re able to leverage these open-weight models so much more effectively.

There are almost 2 pieces to this, right? There’s the cost thing, where the closed models are too expensive, and then there’s the control thing, where I want to be in control of my infrastructure and in control of the model. If I need to extend it or put on my own guardrails, I’m curious: Have you heard from customers? Are both of those things important to them, or are they willing to pay as long as they have control? Or maybe they’re different use cases?

I think it fluctuates over time. Control has mattered a lot over the last few years, and cost just started to matter over the last few months. Cost really matters when people are trying to migrate off their expensive coding plans and their skyrocketing token spend, but control has always been in the background. They want to be able to control the cost, but it’s also about controlling system performance against what they’re paying for.

For example, for a voice-agent company, they want to control their own model so they can make sure the model actually responds within the required time. When the customer is on the phone, they can ensure the agent is responding according to an SLA. Sometimes, this is only something you can do with your own controlled infrastructure, because you know the hardware you’re running on and the whole system you’re monitoring.

That’s versus signing up and relying on critical infrastructure with a proprietary API, where it might go down at any time or there might be a violation of the contract at any time.

Yeah. Simon, to go back to the cost point, you actually make the point in an essay you recently wrote about the release of Kimi K2 that the economics is beside the point. In the case of these really great open-weight models on the frontier, designed by really brilliant researchers, these models are in some cases just as expensive as the closed-source models. So, in those cases, what is the point of running them, and what do we learn architecturally in the course of running them?

Yeah. First, on cost, it’s not necessarily that they’re as expensive as a proprietary model. Rather, the cost discourse has been discussed over and over again, even with Gemini 2.5 a few months back. Open-weight models are sometimes definitely a lot cheaper. But for this model, there’s a big sort of step change where we’re bridging almost a 10x gap, but it strikes somewhere in the middle.

Kimi K2 isn’t as expensive as Claude or GPT-5, but it is a lot more expensive than GLM-4.5 and Qwen 3.5. Why is that? I do believe this is the point of pricing intelligently with the market and understanding where it is. The majority of the discourse should be focusing on, “Wow, this model is bringing an Opus 4.1-level model to my own infrastructure that I can use and run and fine-tune.” I can understand exactly how many tokens I need and understand the exact performance profile.

For example, with a proprietary model, there’s a regular mode and a fast mode. That’s only 2 switches. But with open weights, when you’re running the model, every provider can offer potentially 10 different levels of speed, going from the slowest mode, which can be a lot cheaper, to 400 tokens per second—almost up to 500 in many cases—for some workloads. This is typically 2x or 3x faster than the fast mode out there today.

This kind of control extends to performance, but also to how customers interact with the model and to data retention. Keep in mind, Fireworks doesn’t have a zero-data-retention policy, and at least a lot of data needs to stay there. It lets you control security, compliance, and a lot more. This is why I’m particularly excited about Kimi K2—not just from the cost perspective, but much more because it brings this level of intelligence to something people can own.

In terms of calibrating things like speed and calibrating other things on the back end, what needs to happen? What are you seeing your users do, and who is being really clever about this?

We do see users able to get the maximum benefit out of this model when they enable fast mode. What I’m talking about here, of course, is vLLM’s own fast mode, getting up to 400 to 500 tokens per second, because it is really a big step change, especially when developers are interacting with the model. They can see, “Oh, I can really just get my task done faster here.”

The model is no longer stuck in thinking; rather, it is just executing, executing, and interacting with the environment. For premium developer workloads and focused tasks, we’re seeing that it’s very beneficial. Kimi K2 also gives people the ability to modify it and fine-tune on top of it, allowing them to make it better for their own workloads. This is definitely happening today as well.

Can you just explain what the licensing term is for the most recent open-source models compared to the past, and why you think they’re doing that?

Oh, yeah. Historically, open-weight models were just Apache 2.0, like our software: Take it, modify it, do whatever you want with it. Here is a gift to the world. Recently, the model labs have been trying to understand a way to economically fund their own model development after all the model training and research.

And the data are very, very expensive. So, we've started to see terms even going back to the Llama days, when Meta was releasing Llama. They had a term that said if your daily active users or annual recurring revenue exceeds some threshold, please enter into a commercial agreement with Meta specifically, right?

Remember that the numbers were specifically chosen at that time. You could go find them—it was like 2 companies in the world that fit the definition they had excluded from their license.

Yeah, exactly. But people have taken a hint from that, especially now that the labs are trying to figure out a way to economically fund themselves. Especially when they open-source the model, everybody can just take it themselves, whereas in many cases nobody will use your API anymore, while their API is still taking shape, right? And now we're seeing a very healthy ecosystem developing.

MiniMax recently had this when they released their M2.7 model: they had a term specifically focusing on usage. Kimi initially also had a term around derivative works. This was kind of big news back then with Fireworks and Cursor, about how they built on top of the Kimi model.

And if I could expand on that a little, I don't think it's greed, at least from what I've seen from open-source model labs. Open-source models—really, what we're talking about are open weights, right? And it's just not software. An AI model is not software at the end of the day.

Open-source software used to be supported by people donating their time, or big companies authorizing their employees to donate their time. So it was sort of a bulk in-kind donation of people's time. It really doesn't work in AI. I can't just go home at night and train a frontier open-source model with friends for fun. We need millions or billions of dollars of computing resources in order to do it.

So I think it does support your point that obviously there need to be economic incentives and funding mechanisms in place. Frankly, I think even more so with Chinese models than with domestically produced models. If there's no source of economic—if there's no source of funding for Moonshot to continue to train models, we know where the funding will come from instead, and it's not something we want, right? It's government and things that are actually worse for us, I think.

So I think you raise an interesting point: this is an important economic structure, and I think this means we'll see more of this in the future. Would you agree with that?

Yeah, it's really about sustainability, and how you make sure that all this initial capex—to train the model, fail again and again, and train the model again—is paid back. How do you make sure that there's enough confidence and funding from everybody involved to go do the next one, right?

I recently heard someone making an analogy to the pharmaceutical industry. It's almost like: how do you make sure that the R&D process for new drugs is properly funded, and that there's a sustainable method to making sure people are willing to take big risks and make big bets to do research for new drugs? Largely because they know there's an economic incentive when the new drug is released to the market, a portion of that revenue will flow back—besides the distribution channels, right?—to continue to fund the next R&D effort. This is what we're seeing similarly with model development now.

That's a really interesting analogy, because once a drug—a molecule—is released, you have the strongest possible control, which is that nobody else can manufacture it at all, right? It's the most closed possible source; it's like a secret. But in the case of models, especially open-source models, once it's out there, anybody can take it, use it, extend it, et cetera. So having some economics attached to it probably does make sense.

Yeah.

Well, actually, that raises a question for me, too, which Matt, you were alluding to earlier: how different open-source models are from the dynamics of open-source software maintenance. When it comes to open-source AI, what actually needs to be maintained? Is it the infrastructure around it? Do the models themselves need maintenance at all? Just what are those dynamics? Because I think even the developer behavior around it is pretty different.

Yeah, I mean, Simon sort of said this already, but maybe I'll just expand it a bit. You see the results of a big training run. We're training now—it's pre-training and then RL, kind of—I know, mid-training or sort of post-training on these things, or, you know,

Pre-training, SFT, or RL, right? It's a very complicated pipeline. We see only the result.

Yeah.

Of this at the end. And the numbers are big: “Oh, this was a $100 million training run.” But what you often forget is there may have been 5 failed, large-scale training runs before you even get to that point.

A lot of sweat and tears.

Yeah, yeah.

Yeah, exactly. One of my favorite artifacts—maybe we could even track down the link—is one of the early Llama models.

Yeah.

They published the whole conversation log between the people who were babysitting the training clusters while the models were training.

Yeah.

And it's so funny. It's just like, “Oh no, everything's gone wrong”—chaos, panic—and then the next comment is like, “Okay, we solved it. Everything's okay. Cluster's up, losses are going down.”

So there's a lot that goes on behind the scenes before these models get released. Once they're out there, I guess it's a little bit more in your zone, to kind of make sure it's operationalized.

Oh, yeah. But this is also a very interesting point: once it's out there, there's a whole community effort trying to optimize the model. The model is trained on a given type of hardware and a given type of architecture, but when it's out in the wild, everybody has different cluster topologies and use cases.

It's about how you turn a use case of 1 into a use case of almost infinity. Now you have people trying to adapt it to edge devices, and people trying to run it at the largest scale ever. They're adapting it and making sure it runs for voice agents, but also for coding agents, which are entirely different kinds of use cases.

So this is a whole community effort trying to further optimize and specialize the model, making sure that running it is reliable and that people can continue to optimize against it. That's a whole village effort throughout the open-source community to make sure it's improved.

Yeah, and that's cool, because that really is like open-source software. I mean, this is what you do, but anybody can contribute and make these better. My fun thought experiment is: if GPUs dropped in price by 99%, if GPU-based compute actually became cheap and widely available, then do we get back to a real open-source world where 1 person sitting in their basement, or 100 people working in their free time, can come up with something new, try many of these model-training paths that are in the queue somewhere at 1 of the big companies, and really see the field expand and advance collectively?

Yeah. Yeah, well, this relates to what you were talking about at the beginning. At the beginning, the amount of compute you needed to be at the quote-unquote frontier was negligible, and now it's enormous. How do you get that back to consumer parity again?

I've said this on the podcast before, but I'll keep saying it. AlexNet, the first neural network to run on GPUs that we care about, ran on 2 GPUs. And that's not like there are no missing decimal points or commas in there—literally 2.

Yeah. That would get you literally nowhere.

Yeah. Yeah. So I guess this relates to another question that we've had: inference has gotten harder over the past 18 months because of this combination of scale and diversity in the kinds of models that we have, and, of course, agents that are doing increasingly long-running tasks. What makes open source in this world and in this scenario not just a nice thing to have, but absolutely necessary?

Yeah, so scale comes from a few points. It comes from whether or not you can run this gigantic model on a data-center rack, but it also comes from whether you can optimize it to the extreme point, to the speed of light, so that you're getting the most value out of it.

You also need a whole community, collaboration, and partner effort to validate it and make sure there are no last remaining bugs that appear 0.0001% of the time. So this is where scaling up the deployment, and making sure more people are running it at scale—the largest footprint possible—will ensure everybody's experience of running this model is better.

This is why open-source inference is a current leading way right now instead of closed-source inference engines.

And frankly, a lot of the inference clouds and APIs-as-a-service today do use and leverage open-source inference engines under the hood. The reason to do that is that there are just so many battle-tested recipes and things they’ve learned that they can build on top of. This is why open source is currently the leading way to run models.

Something that I find kind of funny, looking back at not just the history of vLLM and Inferact but also companies like OpenRouter or even Ollama, is that all of these different teams got started around 2022 and 2023. Some of them, in your case even before ChatGPT, were built around open models. What kind of special thing do you think was happening at that time, just in the world of research and in the world of AI, where we see these companies now and think that vLLM is mission-critical or OpenRouter is mission-critical?

What do you think was going on at that time where, even before we had a big consumer use case, we had teams like yours building these things? And how did you guys know, I guess, is the question?

I think there are 2 parts to this. Our team has always had an open-source angle. At UC Berkeley, there’s a long tradition of open-source software and systems research principles. But there was also just curiosity overall. Everybody was so curious about how these AI models were going to benefit humanity and how we could use them better.

With open source, you meet the most mission-aligned people. OpenRouter and Ollama, as you mentioned, are all great partners and friends in the ecosystem because we’re there to understand how everybody else can leverage AI models better and make sure they make good use of them.

And speaking of leveraging open-source models really well, I think another thing that was in the news recently was Hugging Face using a Chinese open-source model to help contain a cyberattack carried out by a rogue, unsandboxed OpenAI model that was being tested. What can we take away from something like that? Can you first walk us through your impressions and understanding of what happened, and then what you think the takeaways are?

Yeah, this kind of goes back to our previous point about controls. For the Hugging Face incident, they broke it down pretty well on their website and blog, so I really thank them for their transparency there. In the end, it’s about all of the closed, proprietary model APIs: their guardrails are a little bit arbitrary, but also very difficult to enforce. That means they have so many false positives in the guardrails that legitimate use cases just get blocked.

This is an evergreen problem, even from the social media days. How do you design content filters correctly and be able to do moderation correctly? If moderation is never solved—which is going to be very, very hard—then there’s always a place where you have a model that you know and trust what you’re publishing to and are able to use.

So Hugging Face had to use an open-weight model. But in the future, I also see that, for trusted use cases, people will go to open weights by default, because that is where you know for sure that the guardrail is lessened, or you can control your guardrail for trusted use cases.

This is actually applied to us today. A lot of Anthropic models ban frontier AI research, and when we’re studying GPU kernels, even an invalid memory-access error triggers the red line. A lot of our developers within Inferact, working on vLLM, are retreating from using Claude 3.5 because you can run a 2-hour job and trigger the red line, which is a false positive. Then you have to lose all of your work.

A lot of our developers are using Kimi K2 today, just to make sure, because it’s a similar quality and it has a guardrail that makes sense to us. We’ve had zero issues while using it.

I think your social media analogy is a really apt one.

Yeah.

Because in both cases, what’s kind of happened is you’ve taken distributed human activity and centralized it in one place. Before social media, people talked to each other and published articles and messaged each other and had message boards. But then you centralized it all under one kind of profit-seeking enterprise, and all of a sudden the incentives changed a lot, right?

We have a specific carve-out for social media, which is that you’re not responsible for what people say on your platform. And that allowed the moderation problem to be tractable, I think, right? It’s like, “Okay, we’re going to do our best to eliminate obviously illegal things, things that are just going to degrade the experience on the site in really bad ways. But if it’s on the edge—if it’s something that’s maybe out of fashion to say, but not illegal—”

Like, we’re just—we just can’t police everything. We can’t be the world police of all human communications. I think something similar is happening in AI, right? A lot of work—not just talking, but work—is being consolidated in one place. If I need to write code or create a spreadsheet or get health advice, or anything, it’s all happening in one or two sites.

Yeah.

And they don’t have that carve-out, right? They don’t have that exemption of, “Hey, we’re not responsible for what actually happens on the site.” Some of them, especially Anthropic, are going further than what would be legally required, and they’re taking ethical stances on these things, which may be right or may be wrong. Those are their decisions.

And how do you actually do this? The problem is actually bigger than just communication, because it’s actual work and action, and you don’t have this exemption, and you have this layer of ethical stuff on top. So, yeah, I’m just totally with you. It seems like some of these companies have erred on the side of caution, which is probably a reasonable thing to do, but very, very frustrating as a user.

I saw online that some translation attempts are blocked because they think the particular combination of language and content is somehow triggering politically sensitive topics and things like this.

Simon, we’re nearing the close of the conversation, and I just wanted to take a step back a bit and ask about Inferact and running the company. I know that Ion Stoica of Databricks is an advisor and a co-founder of Inferact, and I’m curious what you’ve learned from him in terms of taking an open-source project like Llama 2 and vLLM and building a company.

Yeah, Ion, as the co-founder, has always been thinking about open source and how to support open source better. Now, with experience from Databricks and Anyscale, and even Arena, which is a public platform, and now at Inferact, his focus really stems from this question: How do we build a world where this software is being used by so many people on such a critical project? How do we make sure they’re getting the best quality, and where does the value come from?

If the open-source movement and open-source inference engine are a starting point, then what are the gaps that we can fill as Inferact, and what are the gaps that we should absolutely put in? Ion has always been open-source-first. So, for whatever we need to build, we would like to build it in open source, and then, for whatever value we can deliver, we want to really close the last mile and make sure that customers and partners are getting the best out of it.

Just to close this out, 5 years from now, do you think open-weight, open-source AI models will have closed the gap with frontier models completely? Are frontier models always 1 step ahead? How do you see that shaking out?

5 years.

All right. All right. 1 year. One year. You got 5 years wrong.

Who knows? Or, you know, we’re all going to be just floating around in our WALL-E pods and our spaceships.

Exactly.

Progress, right?

Yeah. Yeah.

For me, really, at this point, there’s a point we haven’t talked too much about: what really differentiates open-weight models from closed-weight models, right? In the end, there’s not much differentiation. It’s more about the distribution strategy and go-to-market strategy. Capability-wise, I don’t really see a big gap, not even today, because of how these models come to be.

They’re really starting from first principles, right? You have a computer cluster, you have training data, and you have brilliant researchers who group together to build this amazing artifact that is this pretrained model and then, later, all the post-trained models that the world can use.

But if you look at the ingredients, one of the most important parts is the data. It’s about who gets what data and then what environments you’re building for a model to improve on itself and get better, right? One of the very useful benchmarks that we have on Arena for Kimi K2 has been front-end coding.

For Moonshot, they have built some of the best environments for front-end coding. They have published amazing demos of the ability for this model to code and then see what it rendered, and then continue looping on this iterative process.

Now, this is about their environment to improve the model. It’s not just about source data or where they get the data from; rather, it’s about who can build the best environment and who can make the most optimization and algorithmic choices to leverage all that learning from this environment. So, the next year is all going to be about that. It’s about how open-weight model labs are differentiating and really getting the model to meet the real world, and having this kind of thing that people call today recursive self-improvement, almost, to really improve the model overall. And so, really project out a year, there’s not going to be any difference.

Yeah. And you’ve used this term “brilliant researchers” a few times. There are brilliant researchers everywhere in the world, clearly. Why do you think, in the US, all the smart researchers are working on closed models, and in China all the smart researchers are working on open models?

I mean, from my point of view, they’re attracted to interesting problems, not necessarily to the open or closed stance. However, open-weight models do give people a really, really good boost in the impact of such models. So, that is a plus.

Mhm.

And I think all the brilliant researchers are attracted to how to improve the model overall, right? Actually, one interesting point about this—maybe fairly technical for this Kimi K2 model—is that they removed a rotary positional embedding. RoPE has always been there for a lot of Transformer models. And guess who removed it? It was the inventor of RoPE—

Oh, that’s cool.

Jianlin Su wrote the first paper introducing RoPE as a concept. He now also wrote the explanation of why you don’t need it as part of the technical report on this Kimi K2 model. So, when we read it, it’s like it’s really come full circle: you have all these brilliant researchers who are able to really study how this works, really study the secrets of training and pre-training, and share it across the whole world, recognizing and iterating on their past, right? So, really a miracle, I would say, for this model to come alive.

It’s so interesting. AI is in this funny zone where, empirically, it works incredibly well.

Yeah.

But then you go ask the theorists, and they have no idea what’s going on.

Right.

You have these kinds of iterative things where, when you go read the primer on Transformers—

Yeah.

You read about positional embeddings and why positional encodings are so important, because otherwise you can’t understand meaning.

Yeah.

And then it turns out, once you understand one level deeper, because we’ve been doing this for a few years and you have all these smart people, it’s like, “Oh, actually, you don’t need it.”

Exactly. Yeah.

Simpler actually is better.

We didn’t talk about distillation much so far in this conversation, but I think it’s very relevant to this.

Yeah.

I have just one question. I’m not going to ask, “Is distillation happening?” I think this is kind of speculation on the part of everybody in the world, but you work a lot with these Chinese labs. Do you think distillation is a critical component of what they do, or are they just doing good work and distillation, if it’s done, is sort of an incidental part of it?

I will lean to the latter part specifically. As I mentioned previously, environment matters so much today. These are our own environments, right? These cannot be distilled. You don’t have other people’s environments to really distill a copy from. It’s about constructing it, understanding, and also understanding the learning process. You cannot distill how the model learns with the environment. A lot of these are just not doable today.

There are things potentially you can do with rewriting the data sets, right? Making better pre-training data, but again, you can do it with any model. Any model is going to follow instructions and be useful in terms of utility there. So, I really don’t think, from what we’re currently seeing, this is a big cornerstone of what’s powering the progress today. In the end, what’s powering the progress is still just really smart people with very interesting algorithms, data, environments, and, of course, compute. They will produce the models.

I think it has really interesting policy implications. I tend to agree with you, by the way, that we have smart people everywhere working on a bunch of smart things, and it’s not about distilling data from any one place. It has really interesting policy implications, right? Because it doesn’t—it’s almost tempting if you’re in the White House to say, “Oh, sure, we’ll just turn off distillation and all our problems will be solved.” But I think it’s more the case that they’re just smart people doing interesting things. And so it’s like, how do we adapt to that? I think it’s—

Yeah, and creative innovation. One part in my essay will mention that open source and open-weight models really help innovation because they set out this racetrack where everybody can learn from each other and see where each player is in this racetrack. Then you’re able to improve and stand on each other’s shoulders to improve yourself. So, that is where everybody can move forward faster.

Yeah, and one thing we’re looking for a lot from an investment standpoint is people doing more open-source model training all over the world—not just in China, not just in the United States, but all over the world—because you get that sort of magic of collaboration when everybody’s doing it.

Well, I think that’s a good note to end on. Matt, Simon, thank you so much for joining us.