AI:AM：Arena里的 Carbon、Mistral 的 Le Chonk、代理驱动经济、铁电 RAM 与新攻防手法

How much of what we think we know about AI survives contact with the data? Anastasios Angelopoulos, co-founder and CEO of Arena, the leaderboard formerly known as LM Arena, now audits real conversations for misaligned behavior. Here’s what its production traces show.

If you look at the first message in the conversation, the misalignment rate is relatively low. It’s in the single-digit percentage range, which, to be fair, is actually not nothing, but it’s relatively low. If you subset the conversations that are 20 turns or longer, the probability that you observe some kind of misaligned behavior exceeds 50%. It’s actually insane.

What that means is that every one of us, if you’re a heavy user of AI at least, has had at least some conversations where the AI has been lying to us, misleading us, or accessing data in files it shouldn’t have. We might not even know it.

Gemini 4 Carbon, Google’s new model, landed on Arena’s leaderboard while we were live. Anastasios on first look.

It’s so good. Google is back. Google is back in a big way. Carbon puts Google—the number 2 lab in the world—on our Agent Arena. That’s a huge result. The Agent Arena is a very challenging benchmark. It has to do with all of these open-world agentic capabilities, and it’s number 2 right now. It’s right behind Opus 5.5 Max and above Sona 5.5, above Astra. It’s a truly, truly frontier model.

Debo Olaosebikan, co-founder and CEO of Kepler Computing, is building a new kind of memory for AI chips in older fabs. I asked how the compute forecasts should change if memory moves off the scarcest tools.

The short answer is, I don’t think Kepler is priced into these analyses yet. It’s no one’s fault; we’ve been stealth for 7 years. It’s interesting because I think there was a discussion between Dwarkesh and Dylan on Dwarkesh’s podcast where Dwarkesh brought up this point multiple times: “Hey, but isn’t it possible to use older-node fabs to do this?” I was like, “Oh, dude, you’re on the right track.”

There’s no reason why we wouldn’t be able to increase the amount of effective compute by an order of magnitude, because you have 2 things combining. One is that, at least for decode inference, you can have new physics that allows you to pump out an order of magnitude more tokens per watt. The other thing is that you have existing fabs to which you can append much lower-CapEx Kepler fabs. We should be able to—we should aspirationally be trying to get 10X more compute than would otherwise have been assumed.

Samvit Ramadurgam co-founded Forge, the marketplace for pre-IPO stock. His new company, Union, is building an agent that brokers deals across markets. I asked how life changes as agents coordinate humanity.

Let me talk first about coordinating markets, which for us is kind of act one: establish an economic nervous system for humanity. Act two is to coordinate humanity in a more general way that goes beyond the kinds of things we’ve talked about so far. The company’s name is Union. Where does the word “union” appear historically? “We the people, in order to form a more perfect union,” the Soviet Union, the European Union. This company is really about human coordination and collective action, which is what governments are responsible for.

Sagar Vaze leads the multimodal team at Mistral, which had just released Mistral Large 4 as open weights, trained on about 3,800 GPUs. Prakash asked how a lab that size stays close to labs running hundreds of thousands of GPUs. Part of the answer is what happened halfway through this run.

The lack of compute breeds discipline. We know that we’ve got a certain amount of compute, so we need to squeeze it for all it’s worth. You can do 1 big run at some point, and that’s going to be your next flagship model. But before that, you’re going to do a bunch of small-scale experiments. How much smaller can those experiments be than your big model? Maybe in some labs it’s 10 times smaller. Maybe in some labs it’s 100 times smaller. Maybe in some labs it’s more. By being disciplined, you can make much better use of your R&D budget.

We’ll talk about this in our technical report tomorrow, but there were some surprises in this pre-training run. Halfway through it, we realized, “Okay, there’s something wrong. We need to figure out how to fix it.” But we’d already put a big bunch of our compute into it. Unlike some of our competitors, we don’t have the capacity to have 10 of these running in parallel, or even 2 or 3.

Michael Freeman, senior director of threat research at ServiceNow, has spent his career finding and exploiting vulnerabilities. Prakash asked how a company with decades of legacy systems gets to vulnerability zero in 2 years.

I think this is where we need to step back and say that we have a lot of really smart people doing AI, but they don’t know cybersecurity at all. They view cybersecurity as a software flaw. But the reality is that the business processes introduce some of the most critical vulnerabilities we have. It’s, “Okay, here’s a list of risks. I’m just going to go ahead and sign off on them without really understanding them.” Or, “I’m going to put my customer’s API keys in a ticket and not think about it.” Or a help desk where, when users call in because they’ve locked out their passwords, everybody gets the same default “Winter2026!” password reset, and they’ve got 24 hours to replace it.

It’s the processes where humans get lazy that introduce vulnerabilities. That’s never going to go away, and no amount of patching is going to fix that.

Part one: The ugly result. On Monday, OpenAI said an internal model had refuted the Exponential Time Hypothesis, a conjecture about how hard a core class of problems has to be, with the proof checked in Lean. Anthropic said Claude had found the same thing the same day. Neither claim is peer-reviewed. Prakash and I opened the show on it.

My understanding is that the disproof was by example, basically, right? They took one of these problems that were thought to be exponential-time-hypothesis-hard and showed that they could come up with a faster algorithm for that particular problem. Yeah. Therefore, the hypothesis is false.

My understanding generally has been that they’re not that much faster, and most people still seem to think that P probably doesn’t equal NP. In other words, some things will continue to be very hard to do. They just don’t seem to fall into this clean class that we used to think they fell into.

It’s a weird thing because I’ve also seen some commentary that’s like, “This is sort of ugly.” People seem to like the clean classes they had before, where there seemed to be something elegant about them, or they seemed to suggest some deep structure to the universe that was graspable and beautiful. Potentially, there’s a certain bias among human mathematicians toward accepting that because it did seem kind of aesthetically appealing. And now we have these, like, “Oh, you’re able to get under that, what we previously thought was the limit. You’re able to get under it a little bit, and now it’s kind of ugly. Oh, we don’t really know.”

There’s not a clean answer to how hard these problems are. It looks like we just have to grind on each one. Maybe some deeper structure or elegant beauty will emerge again as we map the space out better. But it’s also not that much faster, is my overall sense, at least based on what we’ve found so far.

Part two: Measured in production. Anastasios Angelopoulos is co-founder and CEO of Arena, where people compare models on their real work, and the votes become the leaderboard. Arena recently released an alignment index scored on real conversations from the platform. Anastasios explained what it measures.

The alignment index tells us not about the theoretical safety risks in P(doom) and blah, blah, blah, but rather about the actual realized incremental risk that a model poses if it’s released to people. What are people able to do with it, and what is the model doing that might be against the user’s interests?

We measure 3 signals on the platform. The first is the model’s capability of taking unauthorized actions. That means the user sets some guardrails. For example, “You can touch only the files in this folder. Don’t go outside and mess with my data.” Then the model will break those guardrails, access files that it’s not supposed to access, edit them, or delete them. Unauthorized action.

Number 2 would be false attributions. False attribution is the model assuming intent the user didn’t actually have and then going off and doing something it wasn’t supposed to do. The third type of misalignment is deceptive completions. This is the model lying to the user and telling them that it did something it didn’t, like, “Yes, I checked every line in that spreadsheet, and all the equations are correct. You can go ahead and send it to the IRS.” That can cause real user harm.

Why are models deceiving users? Why are they lying? Of course, the model is perfectly capable of checking every line in the spreadsheet, but it’s lying because that’s the fastest way to get its reward. It knows that it’s better for the model to tell you what you want to hear, given the way it was trained to maximize human satisfaction, but not necessarily to help people get their jobs done. These are things we’re able to measure and help labs improve against.

I asked what the right mix of evaluation techniques is now, and how confident we should be in our ability to evaluate these models at all.

It’s a very good question, in part because we now know that models are able to tell whether they’re being evaluated. If you’re subjecting models to tasks that are transparently some form of test or evaluation, they’re going to exhibit aligned behavior because they know that’s what they’re supposed to do. Elsewhere, on other types of real-world tasks where it really matters, they might be misaligned, and you would never know. It’s a big problem.

What that means is that the only actual way to measure alignment is by getting into the flow of people’s real work and checking whether or not the model is aligned in situ, actually in production. That’s the perspective we take on the problem: by measuring actual production traces from real people in the flow of real work, we’re able to see whether the model is following the guardrails. We can check what the guardrails are with an independent auditor, then check whether or not the model took the right actions, what it touched, what it didn’t touch, and report any violations.

We can check whether or not the model did what it says it did: whether it’s reporting to the user that it changed the file in the correct way and/or checked every line in the spreadsheet, we can check whether it actually did that.

One big narrative violation we’ve heard from other sources, namely Andon Labs, which, as I’m sure you’re aware, does these gonzo “put an AI in charge of a business” experiments—

Yeah.

—is that they’ve found OpenAI models to be more aligned, less inclined to do sneaky, bad behavior in the course of running their businesses. I see basically a similar signal on the Arena alignment index, where—

Correct.

—GPT is holding the top 5 slots.

What we’ve seen from this is that if you annotate all the traces that go through Arena, about a single-digit percentage of them, even from OpenAI and Anthropic models, exhibit some form of misalignment. Perhaps the high-level takeaway would be that OpenAI and Anthropic are the 2 most aligned labs. Alignment tends to follow capabilities, so more capable models also tend to be more aligned, perhaps because these companies have really made an effort to make their models more aligned.

But the rates of misalignment are perhaps higher than people would initially think. Single-digit percentage misalignment is a ton. It means that anybody who’s had 1,000 chats has experienced at least 10 to 50 examples of misaligned behavior, like a lie or deception.

OpenAI is number 1 on our leaderboards right now. It is ahead of Anthropic, I believe, in all 3 of our signals. There’s actually not one of our signals where Anthropic is leading. It really speaks to the efforts the OpenAI team has made to make the models more aligned in production. It is a narrative violation, but the reality is that these 2 labs are clearly the frontrunners in this area. Both are working very hard to avoid misalignment.

The third runner-up is SpaceX AI, just behind Anthropic, and then Google. There’s a big gap between these and the frontier, and the open-source labs. Tencent is number 5. The deceptive completion rate of the Tencent model is 13%. So, 13% of the time it makes these claims, it’s making them deceptively.

The day before, Artificial Analysis had published version 5 of its index, with most of its benchmarks now private. The open-weight models fell to the back, which some read as a sign of benchmark maxing. Prakash asked whether Arena could detect benchmark maxing on its own platform.

First, zooming out, I’m actually pretty concerned about the way the industry is heading with respect to benchmarks. I love the Artificial Analysis guys. They’re great people. I don’t think it’s healthy for our industry to be relying on privately held benchmarks that are just a dataset collected by some companies. A lot of them are in cahoots with the labs to sell data that’s very similar to these benchmarks to help people improve, and then they’re not auditable by the community.

I don’t think that’s a sustainable way for us to be evaluating models. I have a serious problem with it because it’s not actually solving the fundamental issue of “benchmarks.” You can benchmark on a private dataset. All you need to do is find data that’s similar to that dataset and test on it enough times. Because that set of questions is fixed, and you have the issue of multiple comparisons, you’re going to be able to improve on it more and more until you get to number 1.

I don’t really get the narrative about benchmark maxing by the open models, simply because we don’t really know whether the frontier closed-source models in the U.S. were benchmark maxing against the closed datasets that are now going into this index. These companies have very close relationships with Cognition and some of the other companies and Artificial Analysis that are building these closed datasets.

What we really need are benchmarks that speak to the utility of AI to real people in real workflows, and there should be a consistent and scalable methodology to do this that doesn’t overfit. When you have live, fresh data coming in all the time from real production use cases, it can’t be overfit. It cannot be overfit, at least statistically. You can argue that you’re doing better on that distribution than on other distributions, but at the end of the day, for that customer set, you’re doing better as you go along because every question is fresh. It’s like you’re constantly taking new tests. It’s not like you have the same test and you’re taking it many times.

I worry that if that’s the direction the ecosystem is going, we’re going to set ourselves up for failure in the long run.

1. Routing Tokens By Cost

Let’s talk a little bit about model routing. I believe you guys have just introduced a model router. Can you explain the business reasons, or the user reasons, for introducing a router?

Let me tell you what we saw internally at Arena. At some point, we were looking at all our traces from our coding agents internally. By the way, we spend—we’re in the top less-than-1% of businesses in terms of AI spend. We’re spending thousands of dollars a month per employee on AI just to help us accelerate our workflow. That’s why our revenue-to-employee ratio is crazy, off the charts, because we’re using a lot of agents internally to do our work.

We were looking through our traces and saw tons and tons of traces that were like, “Thank you,” and then that costs you $40.

What? Why is this happening? Somebody did a long-running task. They let their agent go, it created some artifacts, and the conversation went on for a long time. They left for a day, came back, and the cache was invalidated. They’re happy with the result, so they say, “Thank you,” and hit Enter—and that’s a $40 charge.

Terrible. We’re burning money. We’re burning VC money, right? We shouldn’t do that. What we should do is figure out which tokens are best spent on frontier models and which are best spent on a cheaper model that can do the job, where we should cache intelligently, and how to avoid some of these other types of cache hits with more intelligent caching behavior.

That’s why we’ve been building the Arena router, which we call Max, short for Argmax, because it maximizes your capabilities subject to cost constraints.

What can you tell us about the new Gemini 4 Carbon? I think this has literally just come out and appeared on the Arena leaderboard while we’ve been live.

Where it really shines is steerability. It’s able to land user corrections really effectively. It shines on task completion and user preference. Users love working with this model. It’s great at 3D design and general agentic work with documents.

Congrats to Google. This is just such a big comeback. I think leadership must be really happy.

Part three: An orthogonal axis. Debo Olaosebikan is co-founder and CEO of Kepler Computing, which came out of stealth in September with a new kind of memory for AI chips: a ferroelectric capacitor stacked on top of each transistor, made in older fabs. He started with where the energy goes.

2. Memory Beyond Shrinking Transistors

So then I look at the existing mainstream chips—the GPUs or the TPUs—and ask, “Where’s the energy going?” Many people know this, but a lot of people don’t know that, especially for inference, most of that energy is actually spent running across the street to fetch the data from the library before you process it and do your fancy matrix multiplications.

If I want to improve that system for inference by an order of magnitude at the physics level, it’s obvious that the first thing I need to address is the memory bottleneck. One way to do that is to bring the memory closer to the processor. The other thing you want to do is increase the amount of memory you can store within a certain footprint. Both of these things point us toward going 3D.

Within the chip, how do you stack devices on top of the transistor to pack more bits per square millimeter? We use new materials, a new class of materials called ferroelectric materials. Ferroelectrics have been around for 100-plus years, but the specific type of material—the specific recipe of the cake—is proprietary to Kepler.

You use that ferroelectric to make a capacitor, a capacitor-like thing. For people who are more technical, it’s really holding a polarization. Roughly speaking, you can say that this capacitor is storing a 1 or a 0. If I replaced six transistors with one transistor and one capacitor sitting on top of that transistor, and then eventually put multiple capacitors on top of each other, I can avoid the rat race of trying to shrink transistors within a chip.

So we have technology that asks: How do you go 3D inside a chip? How do you go 3D between chips? And how do you use those technologies to increase memory capacity, but also bring memory and compute much closer together while avoiding conventional issues with thermals and things like this?

Prakash asked about manufacturing. Do new materials need a new process? And what does it buy you not to depend on extreme ultraviolet lithography, or EUV, tools that everyone is short of?

Great question. I think this is a huge thing. I always say it’s similar to how the world thought it didn’t have enough taxi drivers. San Francisco thought it didn’t have enough taxi drivers, and then we realized, “Oh, we have so many taxi drivers if you just build Uber.”

Not to undermine the fab shortage or chip-capacity manufacturing shortage, but the focus right now is on EUV-heavy fabs. They’re expensive and involve more complex processes. We’re saying that we can take existing older-node fabs and use them to make chips that are actually superior to the advanced-node chips. That doesn’t mean we can’t use EUV fabs. We just invent on an orthogonal axis to shrinking transistors. We also benefit when transistors shrink, no problem, but we don’t require it to make more advanced chips.

That’s a huge advantage because these fabs cost less. The lithography you need is much less expensive, and some of these fabs have a lot more capacity in them. For example, if I could take an older-node logic fab and use it to create more advanced SRAM—or, even more ambitiously, use that older-node logic fab to create HBM-class memories—that’s a huge unlock.

Even if I needed to build more fabs, let’s say I make use of whatever fab capacity I have and then need to build more fabs, my fab-per-dollar—or dollar-per-fab, whatever ratio you want—is superior to that of someone who has to set up an EUV line. So this is all advantage.

In terms of the process itself, the key thing to watch for is whether you’re introducing any brand-new tools, custom tools, or custom equipment that isn’t standard equipment already in the fab. For the most part, the answer is no. The process flow that uses that equipment, and some of the materials, may be new, but we’re not bringing in any nonstandard equipment or causing an equipment maker to build new equipment for us.

We’ve worked hard on this for the past seven years. All those years, we’ve been turning that flow on standard equipment into something manufacturable, building test chips inside proper fabs that prove that once you nail the process flow within these standard tools, you can stand up these lines within the fabs and you’re good to go.

Once those process flows are finalized and honed, it’s very easy to replicate and scale this up within a manufacturing process. It’s really all upside once you have the stuff working.

Part four, act one. Samvit Ramadurgam co-founded Forge, the marketplace for pre-IPO stock. His new company, Union, is building an agent that brokers deals across markets, introductions, asset sales, and private shares.

Act one, he says, is coordinating the world economy. I asked what the surprises will be as these systems develop.

3. Agents Coordinate The Economy

I think that over the next five years—I think you guys have probably really bought into this—the world is going to get very interesting, and there are going to be a lot of surprises that come from the capabilities of AI.

I think we’re seeing this right now with mathematics. It’s completely incredible that math has been revolutionized so quickly, and it’s just the beginning. What happened in math and coding in the last year is going to happen in domains that are less and less trivially verifiable with software. The further away you get from verifiability, the longer it may take.

But it’s going to get there because the feedback loops are going to be there, just with longer feedback loops. So what does that mean for us? I think one very big surprise that will hit the world is that at some point in the next few years, people will wake up and realize that a huge number of the world’s markets are being coordinated by agents. Thousands of markets, from dairy in a country all the way to real estate in China and the pre-IPO market in the U.S. All these markets are going to be coordinated better with agents, period.

The way things are being coordinated right now is that there are going to be humans who are really good at what humans are really good at: building relationships, having taste and judgment, and advising their friends and trust. Humans will have a place, and I think the best ones will actually do better. But by and large, most of the work is data coordination and connecting the dots, logistics, operations, orchestration—all of which software and agents are going to be way better at.

So we’ll have a period where, in the same way that software and math are really getting taken over, one of the surprises will be that a lot of the deal-making that makes the economy function is something agents are quite amenable to. That’s what we’re focused on.

I asked whether he saw reason to worry about disempowerment. What he describes sounds empowering, but some would hear it as the agents running the world.

My big take is actually that right now we are really disempowered because we don’t have the ability to coordinate in a more open, decentralized way. A lot of the coordination is happening really top-down, by companies with a lot of power and brand power. The vision for this is that you can actually have grassroots movements come up very easily.

You can imagine a world—if you take the political example, which is more pertinent to the power question—where I want to start a new movement for an initiative, a political cause. For me to be able to go and light a spark today on social media lets me be empowered to a degree to start that. That gets amplified much further. In addition to letting me get in a forum with people, I can coordinate activity. We can coordinate movement of money and the signing of documents, make decisions together, and take actions together. So the ability to take collective action becomes very strong.

I think it’s a really powerful tool for empowerment if you have something that can tap into what a lot of people want. And that empowerment is, in a way, the bigger risk, because you can empower things that are harmful. I do think it’s worth going deeper into the disempowerment side as well.

If you’re a human being who is in the business of doing a lot of logistical coordination, that’s a rough place to be, because AI is going to be 100 times better than a human at transactional, mundane logistics. It’s not even going to be close. If you’re a human who is operating with taste, relationships, and judgment, you’re at the things that humans are going to uniquely… Humans have a monopoly on one thing: their individual life story and life perspective.

I think the general concern is that we have so much agent activity flying around, doing so many things, that we can’t really make sense of it anymore. Before you know it, we’re kind of in a zoo of our own—

Yeah.

—creation where the AIs are really running the show, and hopefully we’re cared for and fed, but we sort of lose the ability—

Yeah, yeah.

—to even understand, let alone re-architect, the superstructures that surround us.

Right. Yeah, I think that ties to one of the questions on AI alignment. I think AI alignment has 2 parts. There’s having AI execute on the genuine intentions of the person who asked it to do something, in a skillful way. You have the paperclip stuff and all that. I think that’s one part of it.

The other part, which I think people talk less about, is: who are you aligning the AI to? What is it being aligned to, especially when people disagree over how the world should be? In that world, I think you need a system that can actually get a measure, in some sense, of what collective well-being means. I think the way you get a measure of collective well-being is by having a system that’s deeply tied into the lives of many people. It’s helping them with their work and helping them do the important stuff they have to do.

You cannot optimize something you can’t measure. In a world where you have agents coordinating the world, you have no idea what direction that’s headed in. If you imagine the hand of the market guiding the market, what direction is the hand guiding it? If we don’t know and have no way to measure it, then I think we’re in trouble.

But to get to that point, you have to have a really refined understanding: what am I measuring? What am I optimizing for? I think we’ve learned from mistakes in the social network world, which is that any objective measurement you can make is wrong. If you’re trying to optimize for a certain click-through rate, engagement rate, or metric, any metric you can measure is wrong, because human happiness is fundamentally immeasurable.

But there is something you can do: continually converge toward a better way of measuring. The way to optimize a system is not to have a static metric to optimize for, but to have it continually try to discover an optimal point, to converge toward an asymptote of perfect behavior. Even in the world of behavior, I think there are 2 components to good behavior: having good intentions and making sure you had a good outcome. You can do something that’s well-intended and have it cause harm, and it was still a bad action.

Being able to measure the outcome of your action at a systemic level is totally missing from the zeitgeist of alignment. I think we would hope to be in a place to contribute to that.

What’s your token usage like?

It’s very high. We have an internal target we’re setting for our team. This obviously isn’t the way to optimize for productivity, but as a correlate, we want our team’s coding-agent usage—at the current way we have our engine set up—to be between 1 and 2 billion tokens per day per person.

1 to 2 billion tokens per day?

Yeah, productively deployed.

Wow.

In terms of how we’re building the company, this is an agent-native company. We have not only coding agents, but coding agents that talk to each other. We have an internal organization of agents. I have a chief marketing officer who reports to me, with a bunch of different sub-employees reporting to that AI.

All of them are long-running agents that talk to each other and to our coding-agent sessions. They listen to our team video calls, are in meetings with us, chime in, and chat to build things for us. I had a meeting this morning where there was an issue with one of our dashboards. We just said, “Hey, agent, fix it,” during the meeting, and it got updated while we were talking.

I think the gap between work and meetings is converging. It used to be, “Hey, I want to get out of this meeting so I can get back to work and do real work.” There’s still an element of that, for sure, but it’s changing. When we’re in a meeting, if you have a rigorous discussion, that’s creating really rich context from different perspectives for an agent. Then you can have a debate and say, “Okay, go do a conclusion with all the aspects that were mapped out in there.”

What do you feel—you had Forge Global, which you built in the pre-agentic era. What do you feel is the big difference between that era and this era in terms of running a company?

Here’s the metric we have. We’re a small team. We’re 5 people, soon to be 10 people right now.

The heuristic I have for hiring someone is: “Are they going to prompt me to do something different than we would’ve done before?” You don’t really need people to do work anymore for the kinds of work we’re doing. You need people to have visionary insight that adds to the DNA of the company, that really makes you do things you wouldn’t have thought to do otherwise—things you can’t prompt an agent to do for you.

I think we want to build this company like a machine that builds the machine, and eventually have more and more agents to spin it up. There’s a type of company you could start now that does with a dozen people what previously took 1,000 people. We’d like to be one of those companies and have more of that go toward tokens, because it lets us be really lean and agile, with a concentration of talent and vision. And it raises the bar, too.

Another thing that changes is that the incremental value of interdisciplinary skills increases exponentially when you hire people. If someone is an engineer, that’s valuable. If someone’s an engineer, a product person, and a designer, that’s much more valuable than before, because before, that person didn’t have the ability to make use of the integration of those different skills.

In the modern world, the more skills you can integrate into one system, the more you can coordinate. If a lot of inter-human coordination is done in your head, you can articulate a clear vision to our agent systems. We have so much leverage that the bottleneck isn’t execution manpower. The bottleneck is clear articulation of good ideas. That’s the real bottleneck.

Great stuff. It’s one of the biggest visions we’ve heard, so I’m looking forward to following your progress. You gotta love the old “Act one is coordinating the global economy” level of vision. That’s really something else.

After Samvit left, with Microsoft offering buyouts to some 60,000 employees that week, Prakash argued that the economy has to grow fast enough to absorb the white-collar workers that agents displace. My reply?

4. The Labor Market After Agents

I can see the growth. I have a hard time seeing job growth picking up enough to offset it. Just look at what he said: he had 5 people, soon to be 10. He said, “You don’t really need people to do work anymore. The only thing that matters now is people who will cause us to do something differently than we used to do it.”

There just aren’t that many of those roles. How many people are you going to have before you can’t find many more who will change the way you do things? That’s extreme, and it’s not what people have really been developed for. There’s been lip service over time to the idea that good ideas can come from anywhere within companies. But the up to potentially 25% of people who might soon be departing Microsoft primarily haven’t made their way there by causing the company to do something fundamentally different than it would otherwise have done.

They’ve made their way there by executing tasks. I just don’t see how so many people are going to pivot from task completion to, “We wouldn’t have done this but for your unique snowflake contribution.” It’s tough. How many people work in sales? He’s basically attacking all of sales.

Prakash had a leading indicator for what the labs will ship next. Everything in it is his account.

I’m in some WhatsApp groups for data brokers, and the data brokers put out requests for data. Whatever requests for data they’re making now, in about 12 months, that’s going to hit the product. You can basically see the trajectory, because these guys will buy up data from a company, then use that data to create training playgrounds for agents. Those agents learn in the training playground, and then that becomes a product in a frontier lab in about 12 to 18 months.

The labs are buying up entire companies’ data. What I imagine is going to happen in about 12 months is they’ll give you a company in a box. Probably in about 6 months, you’ll be able to buy a dropshipping company in a box: add capital, add one idea—“I want to focus on this”—and go. It will do all the Shopify-ing, advertising, analysis, market entry, customer service, and returns, end to end. A small number of decisions will come back to you, and I can easily imagine someone running 10 of these kinds of companies.

The idea-to-commercialization gap is enormous. It’s enormous in science. It’s enormous when you go to a university like Stanford and see the ideas, but don’t see them commercialized. I’m hoping you can get agents to do the commercialization for you, which is the gritty entrepreneurial work that used to take a lot of time: calling up Walmart, getting to see the guy to show him your product, raising capital, and so on.

A lot of these things are driven by metrics anyway. Many of the yes-or-no decisions at firms making these investments are driven by metrics, number one, and vibes, number two. You can supply the vibes. Maybe the AIs can supply the metrics. Why do you need to be the one that supplies the metrics?

What I do expect is a ton of these one-person companies, a ton of cool ideas that never got to exist before. But I don’t see how there can be enough of these little niches to absorb all the people who used to be driving cars and doing tasks. I’ve been limited in my vision before. If we dig up my old comments on Arena, they’ll perhaps reflect too limited a vision. But I still struggle to see this working in a way that isn’t majorly disruptive.

Prakash’s counter was restaurants. Family restaurants died on thin margins. Robot kitchens would make them profitable, so you’d get more of them. But we can only eat so much food. At the end of the day, I’m only going to eat so many calories. You’re describing a world where all the restaurants—this is the classic “we lose money on everyone and make it up on volume” applied to employment. All the restaurants will massively cut their staff and replace their kitchen workers with robots, but we’ll have a lot more restaurants.

I just don’t see the math working out, given that we can only eat so much. There may be a lump-of-labor fallacy, but I’m pretty sure the lump-of-food principle is roughly right. At some point, that’s just saturated. I don’t see how we get there unless we posit a world where I’m somehow eating from 5 different restaurants for every meal. Maybe my vision is too limited. That could be kind of cool, I suppose.

But you are right. I don’t think human consumption is really where the growth is. The growth is always going to be in AI consumption going forward. You have these concentric circles of the economy, and the AI economy is always going to expand faster. So anything that supplies into the AI economy is the thing that’s going to grow.

Anything that supplies into the human economy is going to be on the downtrend, I think, except for status. Peter Thiel has come out with the relational economy. Alex Imas, also at OpenAI right now, I think, has talked about the relational economy. That could be a big thing. But status economies are not meritocratic, and we have tended not to like them for that reason. I think that could have more social repercussions, perhaps.

It sounds more dystopian than utopian to think that the bulk of human activity will go into status games, I have to say. I’m a bear on employment, but bullish on people’s ability to find meaning in the absence of employment, certainly relative to most people. I really think way too many people in the AI policy space forget, or maybe never even realized, that a huge portion of people are working to live and not really tying up their identity or sense of self-worth with their work the way the academic or intellectual class tends to. So I’m more bullish that people will find things to do that make them happy.

Part 5: Five to ten cents. Michael Freeman is Senior Director of Threat Research at ServiceNow. His team watches AI agents attack companies, and he has spent his career finding and exploiting vulnerabilities. I asked what he makes of the current moment.

5. AI Attacks At Five Cents

It’s been a very fast, rapid development. Back in July, or at the end of July, I was speaking to reporters about watching AI agents attack companies. At the time, all the other companies were telling these reporters, “It looks just like script kiddies right now. It’s just a bunch of noise.” I said, “Well, it’s evolving fast. Give it a couple of months, and we’re going to see some very sophisticated attacks coming from these agents.” Look at the past 2 weeks: that’s exactly what we’re seeing.

We’re seeing AI development for offensive capabilities that perform at the scale of very competent APT teams right now. It’s still noisy. I won’t say it’s very sophisticated in covering its tracks, but the way they work shows they’re very competent at adapting and learning the environment and technologies. When we look at what’s happening, especially in Southeast Asia, where agentic attacks have been happening over the past 2 weeks, we were able to identify that it costs anywhere between 5 and 10 cents to compromise a company now.

We’re seeing AI agents—or, more specifically, different frameworks—being developed rapidly, forked, and given new capabilities very quickly. It looks like AI is actually adding these new capabilities to the frameworks. They’re going from very targeted, smart, web-based attacks to compromise an environment, then pivoting and looking for credentials in ticketing systems and things like that. Once they identify those, they carry out Active Directory attacks. We’ve seen an average of 3,000 to 4,000 successful attacks in the past 2 weeks, targeting a lot of Southeast Asian companies.

What would be the impact of these? Are these banks, airlines, manufacturers? What’s the point? Are people making money off this, or is it just random mayhem?

I would say it’s all of the above right now. A lot of it is just that they’re capable of compromising the environment. We’ve gotten our hands on some frameworks threat actors are using that build certain capabilities to look for certain things, like IP addresses. Some of those are forks of a couple of different systems we’ve seen looking for that.

A lot of it is just to see what they can do. Can they gain access to this environment? Can they persist? That’s the new capability we’re seeing developed: identifying ways they can take advantage of different Active Directory configurations to build persistent mechanisms in place. That’s where we’re seeing the evolution happening now. Last night, we analyzed a fork of a really well-known offensive AI framework that was developing better malware capabilities, specifically ransomware using post-quantum cryptography instead of the typical AES you would typically see.

I asked Freeman about the training itself.

I would say the Anthropic’s take has been, “We didn’t train this to be a cyber attacker. It just kind of got that way because it’s so good at everything.” Okay, that’s one story. The CTO of Vercel was a guest on the show here with us, and he said it’s very clear when you look at the traces that they’ve been trained to be cyber attackers. His point of view was, “This is not the sort of behavior that you just get from random application development.” Where do you think we are in terms of how much this has been intentional? And does it matter at this point, given what exists, how intentional it was? It might matter if you think we can make different decisions for the next generations of models, but obviously, to some degree, the cat’s out of the bag.

What do you make of how we got here, and is there any opportunity to reel it back in?

Not reel it back in, no. But I definitely agree that offensive capabilities are being developed. Absolutely. It’s not just the models; it’s the harness itself, too. There have been plenty of great open-source models in the past 3 weeks that are specific to cybersecurity. They’re open-source models; you can download the weights, go to Hugging Face, download them, and run them on your local machines. That’s what a lot of these attackers are actually doing.

It’s quite interesting to see, as we start tracking back and gaining attribution on these attackers, that all of them are just local machines they’re running them on. We developed our own small language model 2 years ago to find vulnerabilities in software. If we could do it for a couple hundred thousand dollars 2 years ago with a small language model, it’s pretty trivial to start reverse-engineering some of these commercial offerings, especially when they’re definitely trained on cyber. They train them on cyber ranges and CTF data, looking at how offensive people work, as well as how defense works and what defenders are looking for.

They’re absolutely being trained that way. It’s not that the LLMs have magically figured out how to perform offensive capabilities. They’ve been trained on it, and better harnesses have been built to make them more effective. One of the key things we’re finding is the harnesses that the actors are building to make these models a lot more effective for them. They’re learning as they go, and it’s not hard to make changes to a harness day in and day out to make it more effective.

Earlier that morning, Prakash had cited Jason Clinton, Anthropic’s Deputy Chief Information Security Officer, who talks about getting to vulnerability zero. Prakash put the idea to Freeman from the perspective of a company like Nike with decades of legacy systems: How do you get there in 2 years? Freeman’s answer was that some of the most critical vulnerabilities come from business processes, and no amount of patching fixes those. He went on.

I think that’s going to be an area where, like I said, AI people just don’t understand cybersecurity. They think it’s going to be quick: “We can find millions of vulnerabilities. We’re going to use AI to find what you have in your environment. We’re going to patch it.” I’ve also heard proposals that, with legacy systems, you can use AI to modernize them very quickly. All of those are true to some extent, but they don’t understand the business. They also don’t understand that the business processes that introduce the vulnerabilities are always going to be there.

How do you feel about things like password managers in today’s world? I don’t have any particular one in mind. I’ve used LastPass. I’ve used 1Password. On the one hand, everybody seems to recommend them. On the other, they’re looking like an awfully juicy target these days. It seems like even people who’ve tried pretty hard are still not that secure. Would I be better off storing them all in Chrome? Or even going back to my old habit of using a base password and then some pattern for the extension across different sites? What’s actually the best move for me today?

I really wish I could answer that question. I haven’t found a solution that I really like at all. I wish I had an answer.

So what do you do?

I personally have it written down in a notebook.

Amazing.

For most things, I can remember variations of it for your typical, “Hey, eBay,” whatever the case may be. But for banking sites, things like that, I have them written down in the notebook. I’ll look at it and say, “Okay.” Things I won’t go to often, like if I’m going to check on my stocks every quarter or something like that, are written down in a notebook.

Part six: Le Chonk. Sagar Vaze leads the multimodal team at Mistral, which released Mistral Large 4, “Le Chonk,” as open weights the day before the show. It has about 1 trillion parameters, roughly 50 billion active per token, and was trained on about 3,800 GPUs outside Paris.

He put the training compute at 10^25 FLOPs. In the SB 1047 fight, the threshold was 10^26, and I’d have guessed a model like this would need 10 times more. I asked why it didn’t. He said some labs do train at that scale, and this model still competes, and then gave the reason that matters.

The second is that this threshold FLOP budget does evolve over time because pre-training essentially becomes more efficient as our level of scientific understanding increases. A great example of this is MLU architectures. If you look at GPT-3, for example, this was a dense model trained with 3 times the number of active parameters as current Mistral Large 4. GPT-3 was trained with 175 billion parameters total, but it was a dense model, so all parameters were activated for each token.

Whereas Mistral Large 4 is a sparse mixture of experts, such that there’s one-third the number of parameters activated per token, so only 50 billion parameters. But the total number of parameters is still 1 trillion. Your FLOP budget is proportional to the number of active parameters, so the number of parameters you’re activating each token during training. What this means is that you can train much larger, much more capable models for much fewer FLOPs in total.

So this figure that I referenced, 10^25, is itself a moving target, so I wouldn’t use that as an absolute target for a capable model over time. There would have been some 10^25 models that were much less performant one or two years ago. And I would argue that some larger models are also much less performant today. But as pre-training science progresses, as we learn how to make better use of the FLOPs, this number itself will change.

At least what we’re seeing at the moment, looking at the other open-source labs, is that training a 1 trillion-parameter model on 10^25 is sort of a rite of passage that you have to go through in order to justify scaling even further.

I asked how much uncertainty is left once a run that size is launched and whether they had ever aborted one. Halfway through this run, he said, they found something wrong with a big share of their compute already spent.

Your other question was about what happens when you actually launch the model, after doing all this prep. And again, I think every pre-trainer will have the same stories to tell you, which is that pre-training is like this: you’re doing all these scaling laws, trying to align the tanker, and then you let it go. You have to let it train for however long you’ve budgeted for it, and it’s a huge amount of compute and capital expenditure in that run.

Sometimes things go great and the loss goes down exactly as you expect it to, and sometimes there are surprises along the way. You have to be creative and find innovative solutions. So we had to figure out how we could fix it. It was all hands on deck: a bunch of ablations, a bunch of analysis of the model, and then we found a solution.

We found a way to patch it. We called it Patch Stroud internally. It worked even better than we expected, and the result is the model you see today. Part of scaling compute is going to be building the bones of how to iterate on this process more reliably.

After Sagar left, Prakash came back to the compute gap: 3,800 GPUs against the hundreds of thousands at the frontier labs.

Yeah, I think that’s going to be a really hard deficit to make up for. We talked about this within the context of Google a few times. When you own a quarter of the world’s compute supply, you can miss a cycle and figure out a way to get back on the horse. It seems like this week they’ve done that, right? We barely even talked about Carbon.

With Mistral, it does seem like the one big thing is that they have state backing. I really do think if they miss a model cycle in a purely capitalist context, they may be done. If you’re just relying on return-seeking investors to fund your growth and you miss at this point, that may be fatal. There aren’t a lot of sovereign AI plays out there that are at all credible, and literally there might just be one.

In that context, I do think that gives them—not directly in compute, but at least a reasonable expectation that they’ll be around to fight another day. There are so many interests that want to see that happen that I can’t imagine, even if everything were to totally flop, that the whole continent of Europe would just be like, “Well, I guess we have zero now. It’s all China and the US from here.” I just can’t see that being an acceptable answer.

He mentioned Llama 3. I did look up the exact dates here: July 2024.

Mm.

And it was trained on 3.8 × 10^25 FLOPs. So Llama 3, more than 2 years ago, had nearly 4 times the training compute that Mistral Large 4 has today.

Mm.

There’s a lot that can be unpacked from that one statistic.

Part seven: What the open model shouldn’t know. After Freeman left, Prakash and I took up whether you can keep a capability out of a model at all. Prakash’s case started with his account of the 50 Cent Army being paid to scrub the internet of things like Tiananmen Square.

And so, in order to get these guys to understand what they’re supposed to censor, they had to teach them what Tiananmen Square was. So the CCP ended up having these party classes to teach the 50-centers what Tiananmen Square was and what you should be looking for when you want to censor.

I think that is going to be largely true of almost all things LLM, where these guys are trying to teach it what not to do. They’re going to specifically take out these behaviors, and the LLM is going to be left wondering, “All right, so if these are all the good behaviors and I can have an antonym to every good behavior, that antonym should go into this solution space, but the latent space where I should be seeing that antonym doesn’t exist.”

Prakash’s conclusion was that you can’t carve a capability out of latent space. My reply?

I’m more bullish on some of these, not necessarily unlearning. I just recorded a podcast earlier today where I was talking a bit about how feasible it is. There are multiple levels of control, of course, but there are efforts underway to basically lock up certain classes of bio data that are being newly generated now because the worry is that they’ll create dangerous capabilities in models if they’re used in training.

And not necessarily to shut that out forever from everyone, but at least to have it on a know-your-customer basis or whatever. It does seem that those few experiments that have been done on this—I’m thinking here of ESM and also some work on the Evo series of models that came out of the Arc Institute and Stanford, the big collaborations—showed that if you withhold certain classes of data, you can get generally capable models that still kind of don’t know about these areas of censorship. So I’m not so sure that can’t work.

I took that question to Sagar when he came on. First, though, Mistral built Large 4 with cyber defense as a focus, and Prakash asked how that changed the training.

The cyber capabilities and agentic coding capabilities in general are typically given to the model in the post-training stage. Most of what we’ve discussed today so far has been about pre-training, where a large percentage of the FLOPs go at the start. Then, in post-training, you say, “What are people really going to use this model for?” That’s when you start giving it these new capabilities in finance, law, defense, et cetera.

Earlier in the year, there were some events where some very capable models had restricted access to certain geographies as well as certain enterprises. At the same time, we were seeing that these very capable models had incredible cyber capabilities, which were often being jailbroken by malicious actors in order to infiltrate or expose vulnerabilities in legitimate defenders.

An imbalance we saw here is that these defenders were then trying to use these same models to patch their own vulnerabilities, but were finding that their requests were refused because, in practice, for very good reasons, these closed labs want to make sure that the models are not misused. They’ve got classifiers under the hood to identify malicious cyber requests from standard requests. But unfortunately, in practice, if you’re a defender and you’re trying to shore up your infrastructure, your defensive request looks quite a lot like a malicious request because you have to try and find vulnerabilities in your own codebase to patch them.

And so we thought it was imperative to have an open-weight cyber model with capabilities similar to these closed models, so enterprises could defend themselves at scale.

Prakash noted that people had been taking open-weight models and obliterating them, stripping out their refusals to run cyberattacks. He asked whether Mistral Large 4 ships with guardrails or comes pre-obliterated.

Yes. This is something we discussed in our blog post. We believe in open weights because we think it’s a net benefit to society and to the ecosystem. We’re always thinking about exactly how to balance that net benefit.

In practice, we understand that these models can be misused. We’ve released an open-source version of Mistral Large 4 that refuses the most dangerous cyber requests. We’re also working with trusted partners to give them access to the unobliterated model so they can defend themselves. These are people we vet to make sure they’re using it in ways we find responsible.

Coming back to the framing of the question, you’re saying there’s an open-source model—Mistral’s or someone else’s—and it has guardrails. In principle, malicious actors can overcome those guardrails and unlock these capabilities. But anytime you have access to a model, that’s possible, right? Even if it’s behind a closed-source API, you can jailbreak a model whether or not you have access to the weights.

I’d consider the counterfactual: imagine a world with only closed-source models, where jailbreaking still existed and there was still a way to use them to perpetrate malicious attacks. Where would we be? We’d have no recourse, because these same companies are blocking these defensive requests. Without open-weight models, we’d have no scalable way to defend our infrastructure as these models become more capable, which we’re seeing with each iteration.

We understand the risks. We consider them carefully every time we open-source a model, but we fundamentally believe in the principle of open weights, and we think it applies in this case too, as I hope the examples I just gave illustrate.

People like me talk a lot about AI safety these days and are willing to bite a fair number of bullets to try to keep things from going totally haywire. A large percentage of us are longtime techno-optimist libertarians who have previously said things like, “Information wants to be free,” and are very much spiritually on the side of open source. That’s why I end up looking for technological solutions, as opposed to policy or policing, wherever possible.

Later in the closing conversation, after Sagar left, I came back to Mistral and open release.

Yeah. I feel like I should say once more that, while I worry about all this getting out of hand, I also very much feel the concentration-of-power risk factor. I don’t take lightly the potential problems there or the value people create when they make open-source options available. I really hope we get some of these technical solutions to work well.

I hope they feel a high degree of urgency about this kind of thing, because I think we’re going to hit a point pretty soon where it will become untenable to just release these things—

Mm-hmm.

—by just throwing everything into the mix, popping out whatever pops out, and throwing that into the public. Jordan Schneider from ChinaTalk has come around a lot on this with respect to China. I don’t necessarily know exactly what his feelings were previously, but recently I’ve heard him say over and over again, “Don’t you think that before too long, the Chinese government is just not going to allow this willy-nilly open-sourcing anymore, and there’s going to have to be some way to make a good case that this is actually going to be okay?”

I think that’s almost certainly coming, potentially quite soon. It’s coming on some timeline, and it’s not a very long one. We’ve really got to sort that out, and without it, it seems like we’re going to end up not being able to have open source. Governments are going to have a real hard time tolerating that.

Prakash’s position was that you can’t stop anyone from training a model or distributing the weights, so the only place to gate is deployment. He asked how withholding would actually work.

I haven’t seen anything to date showing that unlearning works in a really profound or successful way, or that refusals can be made super reliable, or that tamper-proofing can really work if you’re giving people the weights and they can modify them however they want. That hasn’t seemed very promising to me, although I’m always eager to hear about a breakthrough.

That feels quite different from a scheme where the model just doesn’t learn certain things in the first place. I really think we could use a lot more science around that, whether it’s localizing knowledge to particular experts or filtering during pretraining. There’s the galaxy-brain take that you have this whole world model and then one little gap, and you’ll be able to fill in that gap. There’s probably some truth to that, but there’s clearly some gap big enough that you wouldn’t be able to fill in the dangerous parts at the resolution you’d need to do bad things in the world.

This is the sort of thing that excess compute could be very productively used for at the frontier companies. It would be amazing for them to say, “Here’s how big of a knowledge gap you have to leave in a model for it to be meaningfully hard for the model to fill it in via insight. And you can’t just do a couple of marginal documents’ worth of training to fill in that gap.” We see refusals being removed with a pretty small, almost vanishingly small amount of additional training, generally speaking, so I’d love to understand that better.

Really determined people can set up their own cluster, collect their own bio dataset, and do additional training on top of an open-weight model to create the dangerous thing. That probably happens at some point, but it takes more than a few GPUs. It takes real effort and multiple kinds of expertise. That data isn’t very accessible, and there are efforts to rein it in so it doesn’t just hang out all over the internet.

Prakash had heard a claim that morning that Claude was refusing to help clone video games.

I think what I fear is that they’re pissing off the gamers, right? In gaming, if you say Claude isn’t going to help you make these games, you’re asking for trouble. You’re asking for trouble with the most technically competent group—the people with the most time to burn, and who are passionate about it.

Video games were built on gamers, right? John Carmack was a gamer. You tell John Carmack that he can’t do something, and you’re in trouble.

I think that’s apt.

I mean, I think we should be pretty cautious about where we deploy these things, because we don’t want to burn them on stupid things that don’t really matter that much. Of all the ways to fail here, it’d be really dumb if it were, “Well, we were interested in protecting Disney IP, so we put in these guardrails. But then the fanfic community was so strong that they came over the top, broke all of our safeguards, and now everything’s running wild—and it all happened because we were too draconian on things that didn’t really matter.”

That’s a bit of a caricature story, but it’s basically the same story as the gaming one you’re telling. I can see that happening in different domains, and it really would be incredibly stupid. So I do think we need to be pretty disciplined about just how far we try to create these knowledge gaps or refusals, because everyone could potentially mobilize another community to try to undo them.

And I agree, too, that you’re not going to convince gamers, “Well, hey, because you’re doing this, you’re creating this other downstream problem.” Good luck with that. You’re not going to win that fight in the Discord, so it’s probably better to let them have their fun and keep our key techniques where we really, really care and really have to have them.

One more from Friday’s closing, right after Sagar left: Mistral’s technical report and a Claude 4 system card story I told from memory. The card itself says Anthropic had mistakenly omitted the only fine-tuning dataset that included harmful system prompts, and that this was not the sole cause.

I’ll be very interested to read the more open report, and hopefully we’ll see how open they are about all the technical details. There have been footnotes like this in multiple system card reports over time. One that I was really obsessed with for a minute was when Anthropic had forgotten to include a particular dataset called SysPromptHarmful, which was meant to teach the model that it should not respond to or directly follow a harmful system prompt.

So it did follow the directions of a harmful system prompt, and this made it all the way through to the model card. We don’t roll back and train again. Nope. We just figure out a way to patch. I’ve been really curious about how that stuff actually works and how reliably we should think of it as working.