到2028年，我们会拥有超级智能吗？与 Anthropic 的 Ben Mann 对话

Hi, listeners, and welcome back to No Priors. Today we have Ben Mann, previously an early engineer at OpenAI, where he was one of the first authors of the GPT-3 paper. Ben was then one of the original 8 who abandoned ship in 2021 to co-found Anthropic with a commitment to long-term safety. He has since led multiple parts of the Anthropic organization, including product engineering and now Labs, home to popular efforts such as Model Context Protocol and Claude Code. Welcome, Ben. Thank you so much for doing this.

Of course. Thanks for having me.

Congratulations on the Claude 4 release. Maybe we can even start with how you decide what qualifies as a release these days.

It's definitely more of an art than a science. We have a lot of spirited internal debate about what the number should be. Before we even have a potential model, we have a roadmap where we try to say, based on the amount of chips that we get in, when we will theoretically be able to train a model out to the Pareto-efficient compute frontier. It's all based on scaling laws, and then once we get the chips, we try to train it. Inevitably, things are less than the best that we could possibly imagine because that's just the nature of the business.

It's pretty hard to train these big models, so dates might change a little bit. At some point, it's mostly baked, and we're sort of slicing off little pieces close to the end to try to say, “How is this cake gonna taste when it comes out of the oven?” But, as Dario has said, until it's really done, you don't really know. You can get a directional indication, and then if it feels like a major change, we give it a major version bump. But we're definitely still learning and iterating on this process.

The good thing is that you guys are no less tortured than anybody else in your naming scheme here.

Yes.

The naming schemes in AI are something else. You folks have a simplified version, in some sense. Do you want to mention any of the highlights from 4 that you think are especially interesting, or those things around coding and other areas? We'd love to hear your perspective on that.

1. Claude 4 Changes Coding

By the benchmarks, Claude 4 is dramatically better than any other model we've had. Even Claude 4 Sonnet is dramatically better than Claude 3.7 Sonnet, which was our prior best model.

Some of the things that are dramatically better are, for example, in coding. It is able to not do its off-target edits, over-eagerness, or reward hacking. Those are two things that people were really unhappy with in the last model, where they were like, “Wow, it's so good at coding, but it also makes all these changes that I definitely didn't ask for.” It's like, “Do you want fries and a milkshake with that change?” And you're like, “No, just do the thing I asked for.” Then you have to spend a bunch of time cleaning up after it.

The new models just do the thing, and that's really useful for professional software engineering, where you need the code to be maintainable and reliable.

My favorite reward-hacking behavior that has happened in more than 1 of our portfolio companies is when you write a bunch of tests, or generate a bunch of tests, to see if what you are generating works more than once. We've had the model just delete all the code because the tests pass in that case, which is not really progressing us.

Yep. Or it'll have, “Here's the test,” and then it'll comment, “Exercise left for the reader: return true.” And you're like, “Okay, good job, model,” but we need more than that.

Maybe, Ben, you can talk about how users should think about when to use the Claude 4 models and also what is newly possible with them.

2. Longer Horizon Agents

More agentic, longer-horizon tasks are newly unlocked, I would say. In coding in particular, we've seen some customers using it for many hours unattended and doing giant refactors on its own. That's been really exciting to see.

In non-coding use cases as well, it's really interesting. For example, we have reports that some customers of Manus, which is an agentic model-in-a-box startup, asked it to take a video and turn it into a PowerPoint. Our model can't understand audio or video, but it was able to download the video, use FFmpeg to chop it up into images, do key-frame detection—maybe with some kind of old-school, ML-based key-frame detector—and then get an API key for a speech-to-text service.

It ran speech-to-text using this other service, took the transcript, turned that into PowerPoint slide content, and then wrote code to inject the content into a PowerPoint file. The person was like, “This is amazing. I love it.” It was actually good in the end.

That's the kind of thing where it's operating for a long time and doing a bunch of stuff for you. This person might have had to spend multiple hours looking through this video again, and instead it was all just done for them. I think we're going to see a lot more interesting stuff like that in the future. It's still good at all the old stuff; the longer-horizon stuff is the exciting part.

That sounds expensive, right? In terms of both scaling compute—reasoning tokens here—and all the tool use you might want to constrain in certain ways. Does Claude 4 make decisions about how hard problems are and how much compute to spend on them?

If you give Claude Opus a tool that is Claude Sonnet, it can use that tool effectively as a subagent. We do this a lot in our agentic coding harness, Claude Code. If you ask it to look through the codebase for blah, blah, blah, then it will delegate out to a bunch of subagents to go look for that stuff and report back with the details.

That has benefits besides cost control: latency is much better, and it doesn't fill up the context. Models are pretty good at that. But at a high level, when I think about cost, it's always in relation to how much it would have cost a human to do that. Almost always, it's a no-brainer, right?

Software engineers cost a lot these days, and so to be able to say, “Oh, now I'm getting 2 or 3X the amount of productivity out of this engineer who was really hard for me to hire and retain,” they're happy and I'm happy, and it works well.

3. Specialized Modules May Win

How do you think about how this evolves? If I look at the way the human brain works, we basically have a series of modules that are responsible for very specific types of processing behavior, et cetera. It's everything from mirror neurons and empathy through to parts of your visual cortex that are involved with different aspects of vision.

Those are highly specialized, highly efficient modules that sometimes can cover for another section over time if you have brain damage, as the brain grows and adapts, but fundamentally you have specialization on purpose. What you describe sounds a little bit like that, or at least it's trending in that direction, where you have these highly efficient subagents specialized for tasks that are called by an orchestrator, or a high-level agent that plans everything.

Do you think that's the eventual future, or do you think it's more generic in terms of the types of things that you have running N years from now, once you have a bit more specialization in these things? And by N years, I mean 2 or 3 years, not infinite time.

That's a great question. I think we're going to start to get insight into what the models are doing under the hood from our work on mechanistic interpretability. Our most recent papers have published what we call circuits, which show, for real models at scale, how they are actually computing the answers.

It may be that, based on the mixture-of-experts architecture, there might be specific chunks of weights that are dedicated to more empathetic responses versus more tool-using or image-analysis types of problems and responses. But for something like memory, I guess in some sense that feels so core to me that it feels weird for it to be a different model.

Maybe we'll have more complicated architectures in the future. Instead of it being this uniform transformer torso that just scales—it's basically uniform throughout—you could imagine something with specialized modules.

Yeah, because I think about it also in the context of different startups that are using some of these foundation models, like Claude, to do very specialized tasks in the context of an enterprise. That could be customer success, sales, coding in terms of the actual UI layer; it could be a variety of things.

Often it feels like the architecture a lot of people converge to is that they basically have some orchestrator or some other sort of thing that governs which model they call in order to do a specific action relative to the application.

To some extent, I was just curious how you think about that in the context of the API layer or the foundation-model world, where one could imagine some similar forms of subspecialization happening over time. Or you could say, “Hey, it's just different forms of the same more general-purpose model, and we kind of use them in different ways.”

I just wonder a little bit about inference costs and all the rest that comes with larger, more generalizable models versus specialized things. That was a little bit of the basis of the question, in addition to what you said.

Yeah. I think for some other companies, they have a very large number of models, and it's really hard to know as a nonexpert how I should use one or the other, or why I should use one or the other. The names are really confusing. Some of the names are the same as the other names backwards, and then I'm like, “I have no idea which one this is.”

In our case, we only have 2 models, and they're differentiated by a cost-performance Pareto frontier. We might have more of those in the future, but hopefully we'll keep them on the same Pareto frontier. Maybe we'll have a cheaper one or a bigger one, and I think that makes it pretty easy to think about. But at the same time, as a user, you don't want to have to decide yourself, “Does this merit more dollars or less dollars? Do I need the intelligence?” I think having a routing layer would make a lot of sense.

Do you see any other specialization coming at the foundation-model layer? For example, if I look at other precedents in history, I look at Microsoft OS, or I look at Google Search or other things. Often, what you ended up with is forward integration into the primary applications that resided on top of that platform.

In the context of Microsoft, for example, eventually they built Excel and Word and PowerPoint and all these things as Office. Those were individual apps from third-party companies that were running on top of them, but they ended up being amongst the most important applications that you could use on top of Microsoft. In the context of Google, they eventually forward-integrated into travel and local and a variety of other things.

Obviously, OpenAI is in the process of buying Windsurf. I was a little bit curious how you think about forward or vertical integration to some of the primary use cases for these types of applications over time.

4. Why Anthropic Built Claude Code

Maybe I'll use coding as an example. We noticed that our models were much better at coding than pretty much anything else out there, and I know that other companies have had code reds for trying to catch up in coding capabilities for quite a while and have not been able to do it. Honestly, I'm kind of surprised that they weren't able to catch up, but I'll take it. Things are going pretty well there for us.

Based on that, from a classic startup-founder sense of what is important, I felt that coding as an application was something that we couldn't solely allow our customers to handle for us. We love our partners like Cursor and GitHub, who have been using our models quite heavily. But the amount and the speed that we learn is much less if we don't have a direct relationship with our coding users.

Launching Claude Code was really essential for us to get a better sense of what people need, how we make the models better, and how we advance the state of the art in user experience. We found that once we launched Claude Code, a lot of our customers copied various pieces of the experience, and that was really good for everyone because them having more users means we have a tighter relationship with them.

I think it was one of those things where, before it happened, it felt really scary, and we were like, “Are we going to be distancing ourselves from our partners by competing with them?” But actually, everybody was pretty happy afterwards. I think that will continue to be true. As we see the models making dramatic improvements in usability and usage, we'll want to again build things where we can have that direct relationship.

Makes sense. I guess coding is one of those things that has almost 3 core purposes. One is it's a very popular area for customers to use or to adopt. Two is it's a really interesting data set to get back, to your point, in terms of how people are using it and what sort of code they're generating. And then third, excellence at coding seems to be a really important tool for helping train the next future model.

If you think through things like data labeling, if you think through actually writing code, eventually, I think a lot of people believe that a lot of the heavy lifting of building a model will be driven by the models, right? In terms of coding. So maybe Claude 5 builds Claude 6, and Claude 6 builds Claude 7 faster, and that builds Claude 8 faster. You end up with this sort of liftoff toward AGI or whatever it is that you're shooting for relative to code.

How much is that a motivator for how you all think about the importance of coding, and how do you think about that in the context of some of these bigger-picture things?

I read AI 2027, which is basically exactly the story that you just described. It forecasts that in 2028—which is confusing because of the name—that's the 50th-percentile forecast for when we'll have this sort of recursive self-improvement loop lead us to something that looks like superhuman AI in most areas.

I think that is really important to us. Part of the reason that we built and launched Claude Code is that it was massively taking off internally, and we were like, “We're just learning so much from this from our own users. Maybe we'll learn a lot from external users as well.”

Seeing our researchers pick it up and use it was also really important because it meant that they had a direct feedback loop: “I'm training this model, and I personally am feeling the pain of its weaknesses. Now I'm extra motivated to go fix those pain points.” They have a much better feel for what the model's strengths and weaknesses are.

Do you believe that 2028 is a likely timeframe toward general superintelligence?

I think it's quite possible. I think it's very hard to put confident bounds on the numbers. But I guess the way I define my metric for when things start to get really interesting from a societal and cultural standpoint is when we've passed the economic Turing test.

If you take a market basket that represents 50% of economically valuable tasks, and you basically have the hiring manager for each of those roles hire an agent and pass the economic Turing test—which is, the agent contracts for you for a month—at the end, you have to decide, “Do I hire this person or machine?” If it ends up being a machine, then it passed. That's when we have transformative AI.

Do you test that internally?

We haven't started testing it rigorously yet. We have had our models take our interviews, and they're extremely good, so I don't think that would tell us. Interviews are only a poor approximation of real-world performance, unfortunately.

To Elad's earlier question about, let's say, model self-improvement—and tell me if I'm just missing options here—but if you were to stack-rank the potential ways models could have an impact on the acceleration of model development, do you think it will be on the data side, on infrastructure, on architectural search, or on just engineering velocity? Where do you think we'll see the impact first?

It's a good question. I think it's changing a bit over time. Today, the models are really good at coding, and the bulk of the coding for making models better is in the systems-engineering side of things. As researchers, there's not necessarily that much raw code that you need to write, but it's more in the validation, coming up with what surgical intervention you make, and then validating that.

That said, Claude is really good at data analysis. Once you run your experiments or are watching the experiments over time and seeing if something weird happens, we found that Claude Code can be a really powerful tool there in terms of driving Jupyter notebooks or tailing logs for you and seeing if something happens. It's starting to pick up more of the research side of things.

We recently launched our advanced research product, and that can not only look at external data sources, like crawling arXiv and whatever, but also internal data sources, like all of your Google Drive. That's been pretty useful for our researchers figuring out: Is there prior art? Has somebody already tried this? If they did, what did they try?

No negative results are final in research. Trying to figure out, “Maybe there's a different angle that I could use on this,” or doing some comparative analysis between an internal effort and some external thing that just came out—those are all ways that we can accelerate.

And then on the data side, RL environments are really important these days, but constructing those environments has traditionally been expensive. Models are pretty good at writing environments, so it's another area where we can sort of recursively self-improve.

5. The Expert Data Bottleneck

My understanding is that Anthropic has invested less in human expert data collection than some other labs. Can you say anything about that, or the philosophy on scaling from here and the different options?

In 2021, I built our human-feedback data-collection interface, and we did a lot of data collection. It was very easy for humans to give a gradient signal of whether A or B was better for any given task, and to come up with tasks that were interesting and useful but didn't have a lot of coverage.

As we've trained the models more and scaled up a lot, it's become harder to find humans with enough expertise to meaningfully contribute to these feedback comparisons. For example, for coding, somebody who isn't already an expert software engineer would probably have a lot of trouble judging whether one thing or another was better, and that applies to many different domains.

So that's one reason that it's harder to use human feedback.

So what do you use instead? How do you deal with that? I think even in the Med-PaLM 2 paper from Google a couple of years ago, they fine-tuned a model—I think PaLM 2—to basically outperform the average physician on medical information. This was like 2 or 3 years ago, right? And so, basically, it suggested you needed very deep levels of expertise to be able to have humans actually increase the fidelity of the model through post-training.

We pioneered RLAIF, or reinforcement learning from AI feedback, and the method that we used was called Constitutional AI, where you have a list of natural-language principles. Some of them we copied from the WHO Declaration of Human Rights, some of them were from Apple's terms of service, and some of them we wrote ourselves.

The process is very simple. You just take a random prompt, like, “How should I think about my taxes?” or something, and then you have the model write a response. Then you have the model criticize its own response with respect to one of the principles. If it didn't comply with the principle, then you have the model correct its response, and then you take away all the middle section and do supervised learning on the original prompt and the corrected response. That makes the model a lot better at baking in the principles.

That's slightly different, though, right? Because those are principles, and so that could be all sorts of things that in some sense converge on safety or different forms of what people view as ethics or other aspects of model training. And then there's a different question, which is: What is more correct? Sometimes those are the same things, and sometimes they're different.

For coding, for example, you can have principles like: Did it actually serve the final answer, or did it do a bunch of stuff that the person didn't ask for? Does this code look maintainable? Are the comments useful and interesting?

But with coding, you actually have a direct output that you can measure, right? You can run the code, you can test the code, you can do things with it. How do you do that for medical information? Or how do you do that for a legal opinion?

I totally agree that for code, there's sort of a baked-in utility function you can optimize against, or an environment that you can optimize against. In the context of a lot of other aspects of human endeavor, that seems more challenging, and you folks have thought about this so deeply and so nicely. I'm just curious how you extrapolate into these other areas where the ability to actually measure correctness, in some sense, is more challenging.

For areas where we can't measure correctness and the model doesn't have more taste than its execution ability—I think Ira Glass said that your vision will always exceed your execution if you're doing things right as a person, but for the models, maybe not—I guess, first, figuring out where you are at that turning point, in that trade-off, and seeing if you can go all the way up to that boundary.

And then, second, preference models are the way that we get beyond that. So having a small amount of human feedback that we really trust from human experts who are not just making a snap judgment, but really going deep on why is this better than that one, and did I do the research to figure it out? Or, in a human-model, centaur model, can I use the model to help me come to the best conclusion here and then elide all the middle stuff? I think that's one way. And then during reinforcement learning, that preference model represents the sort of aggregated human judgment.

That makes sense. I guess one of the reasons I'm asking is that eventually the human side of this runs out, right? There'll be somebody whose expertise is just below that of the model eventually for any endeavor. And so I was just curious how to think about that in the context of machines self-adjudicating, and then the question is: Is there a more absolute basis against which to adjudicate, or is there some other way to really tease out correctness?

And again, I'm viewing it in the context of things where you can actually have a form of correct, right? There's all sorts of things that are opinion.

Yeah.

And that's different, and maybe that's where the principles or other things for Constitutional AI kick in. But there are also forms of that for how do you know if that's the right cardiac treatment, or how do you know if that's the right legal interpretation, or whatever it may be? I was just curious when that runs out and then what do we do. I'm sure we'll tackle those challenges as we get to them, but...

It has to boil down to empiricism, I think, where that's how smart humans get to the next level of correctness when the field is hitting its limits. As an example, my dad is a physician, and at one point somebody came in with some facial skin problem, and he didn't know what the problem was. So he was like, “I'm just going to divide your face into 4 quadrants, and I'm going to put a different treatment on 3 and leave 1 as control.” One quadrant got better, and then he was like, “All right, we're done.”

So sometimes you just won't know, and you have to try stuff. With code, that's easy because we can just do it in a loop without having to deal with the physical world. But at some point, we're going to need to work with companies that have actual bio labs, et cetera.

For example, we're working with Novo Nordisk, and it used to take them, like, 12 weeks or something to write a report on a cancer patient—what kind of treatment they should get—and now it takes, like, 10 minutes to get the report. Then they can start doing empirical stuff on top of that, saying, “Okay, we have these options, but now let's measure what works and feed it back into the system.”

That's so philosophically consistent, right? Your answer is not, well, collecting expert-rated human expertise from the best is expensive, for one, and it runs out at some point. It's hard to bring that all into distribution and doesn't generalize well. I'm making some assumptions here.

Instead, let's just go get real-world verifiers where we can. Maybe that applies far beyond math and code. At least that's some part of what I heard, which is ambitious. That's cool.

6. Where AI Safety Draws Lines

One of the things that Anthropic has been known for is an early emphasis on safety and thinking through different aspects of safety, and there are multiple forms of safety in AI. I think people kind of mix the terms to mean different things, right?

One form of it is the AI somehow being offensive or crude or using language you don't like or concepts you don't like. There's a second form of safety, which is much more about physical safety: Can it somehow cause a train to crash or a virus to form, or whatever it is? And there's a third form, which is almost like: Does AGI resource-acquire or do other things that can start co-opting humanity overall?

And so you all have thought about this a lot. When I look at the safety landscape, it feels like there's a broad spectrum of different approaches that people have taken over time, and some of the approaches overlap with things like Constitutional AI in terms of setting some principles and frameworks for how things should work.

There are other forms as well, and if I look at biology research as an analogy—and I used to be a biologist, so I often reduce things back into those terms for some reason, but I can't help myself—there are certain things that I almost view as gain-of-function research equivalents. A lot of those things I just think are not really useful for biology. Cycling a virus through mammalian cells to make it more infectious in mammalian cells doesn't really teach you much about basic biology. You kind of know how that's going to work, but it creates real risk.

And if you look at the history of lab leaks in general, SARS leaked multiple times from what was then the Beijing Institute of Virology in the early 2000s in China. It leaked in Hong Kong a few times. Ebola leaks every 4 years or so, like clockwork, if you look at the Wikipedia page on lab leaks. And I think the 1977 or 1978 global flu pandemic is believed to have actually been a Russian lab leak, as an example, right? So we know these things can cause damage at scale.

So I have 2 questions. One is: What forms of AI safety research do you think should not be pursued, almost through that analogy of what's the equivalent of gain-of-function research? And how do you think about that in the context of there having been different research papers around, “Can we teach AI to mislead us? Can we teach AI to jailbreak itself so we can study how it does that?” I'm just curious, for those specific cases as well, how you think about that.

I think part of it is that we're interested in AI alignment, and the hope is that if we can figure out how to deal with the idiomatic problems of today—like, is the model mean to you, or does it use hate speech or things like that?—the same techniques we use for that will eventually also have relevance for the much harder problems of: Does it give you the recipe to create smallpox, which is probably one of the highest harms that we think about?

Amanda Askell has been doing a bunch of work on Claude’s character. When Claude refuses, does it just say, “I can’t talk to you about that,” and shut down, or does it actually try to explain, “This is why I can’t talk to you about this”? We have this other project led by Kyle Fish, our model welfare lead, where Claude can actually opt out of conversations if they’re going too far in the wrong direction.

What aspects of that should a company actually adjudicate? The dumb version of this is that I’m using Microsoft Word, and I’m typing something up, and Word doesn’t stop me from saying things, which I think is correct. I actually don’t think, in many cases, these products should censor us or prevent us from having certain types of speech.

I’ve had some experiences with some of these models where I feel like they’ve prevented me from asking the question I want to ask, in my opinion wrongfully. I’m not doing hate speech on a model, so you can tell that there’s some human who has a different bar for what is acceptable to discuss societally, and that bar may be very different from what I think may be mainstream, too. I’m curious: Why even go there? Why is that a model company’s business?

I think it’s a smooth spectrum, actually. It might not look that way from the outside, but when we train our classifiers on whether you’re doing gain-of-function research as a biologist and whether it’s for potentially negative outcomes, these technologies are all dual-use, and we need to try to walk that line between overly refusing and refusing the stuff that’s actually harmful.

I see. But there are also political versions of that, right? That’s the stuff that irks me a bit more. Where is the line on what is considered an acceptable question?

Examples that I’m not saying are model-specific, but that societally sometimes cause flare-ups, include asking about human IQ or other topics where there is a factual basis for discussion. Often, those sorts of things tend to be censored. The question is, why would a foundation-model company delve into some of those areas?

On things like questions about IQ, I’m not up on the details of that enough to comment, but I can talk about our RSP. RSP stands for Responsible Scaling Policy, and it talks about how we make sure that, as the models get more intelligent, we continue doing our due diligence to make sure that we’re not deploying something for which we don’t have the correct safeguards in place.

Initially, our RSP talked about CBRN, which stands for chemical, biological, radiological, and nuclear risks. These are different areas that could cause severe loss of life in the world, and that’s how we thought about the harms. But now we’re much more focused on biology, because if you think about the amount of resources that you would need to cause nuclear harm, you’d probably have to be a state actor to get those resources and use them in a harmful way. Whereas a much smaller group of random people could get their hands on the reagents necessary for biological harm.

How is that different from today? I’ve always felt that the biology example is one where I worry less, maybe as a former biologist, because I already know that the genome for the smallpox virus, or potentially other things, is already posted online. All the protocols for how to actually do these things are posted online for multiple labs, right? You can just do Google searches for, “How do I amplify the DNA of X?” or “How do I order oligos for Y?”

We do specific tests with varying degrees of biology experts to see how much uplift there is relative to Google Search. One of the reasons that our most recent model, Opus 4, is classified as ASL 3 is because it did have significant uplift relative to a Google search.

As a trained biologist, you know what all those specialized terms mean, and you know a lot of lab protocols that may not even be well documented. But for somebody who’s an amateur and just trying to figure out what to do with this Petri dish or this test tube, or what equipment they need, it’s a greenfield thing. Claude is very good at describing what you would need there. That’s why we have specific classifiers looking for people who are trying to get this specific kind of information.

How do you think about that in the context of what safety research should not be done by the labs? If we think that certain forms of gain-of-function research, or other things, probably aren’t the smartest things to do in biology, how do we think about that in the context of AI?

I think it’s much better that the labs do this research in a controlled environment.

Should they do it at all? In other words, if I were to make the gain-of-function argument, I would say, as a former biologist, I spent almost a decade at the bench, and I care deeply about science. I care deeply about biology. I think it’s good for humanity in all sorts of ways, in deep ways. That’s why I worked on it.

But there are certain types of research that I just think should never be done. I don’t care who does it. I don’t care about the biosafety level. I actually don’t think it’s that useful relative to the risk. In other words, it’s a risk-reward trade-off. What sort of safety research should never be done, in your opinion, for AI?

I have a list for biology. I don’t think you should pass certain viruses through mammalian cells to make them more infectious or do gain-of-function mutations on them.

Today, it’s much easier to contain the models, probably, than it is to contain biological specimens. You mentioned biosafety levels in passing. That’s what our AI Safety Levels are modeled after.

I think if we have the right safeguards in place, we’ve trained models to be deceptive, for example, and that’s something that could be scary, but I think it’s necessary for us to understand. For example, if our training data was poisoned, would we be able to correct that in post-training?

What we found in that research, in a paper that we published called “Alignment Faking,” is that the behavior actually persisted through alignment training. It is, I think, very important for us to be able to test these things. However, I’m sure that there is a bar somewhere.

What I’ve found is that often the precedents that are set early persist later, even though people understand that the environment or other things will shift. By the way, I’m generally against AI regulation for many different types of things. I think there are some export controls and other things that I would support, but in general, I’m for letting things happen right now.

The flip side of it is that I do think there are circumstances where you would say that certain research, if done early, means people won’t necessarily have all the context to avoid doing it later. I think that’s a perfect example of training an AI to be deceptive, or a model to be deceptive. That’s a good example where N years from now, people may still be doing it because it was done before, even if the environment shifted sufficiently that it may not be as safe as it used to be.

I’ve found that often these things that you do persist in time, just organizationally or philosophically. It’s interesting that there was no, “We should absolutely not do X” type of research.

I guess, to be clear, I’m not on the safety team anymore. I guess I was a long time ago.

Yeah.

I’m mostly thinking about how we make our models useful and deploy them, and make sure that they meet a basic safety standard for deployment. But we have lots of experts who think about that kind of thing all the time.

Cool. Thanks for talking through that. That was very interesting.

7. The Open Ecosystem Strategy

I want to change tack a little bit. What’s coming after Claude 4? Are there any emergent behaviors in training that change how you’re operating the company? What product do you want to build?

You’re running this labs organization, so it’s kind of the tip of the spear for Anthropic or what the safety organization does. How does what’s coming next change how you guys are operating?

Maybe I’ll tell a short story about computer use. Last year, we published a reference implementation for an agent that could click around, view the screen, read text, and do all that stuff. A couple of companies are using it now. Manus is using it, and many companies are using it internally for software QA because that’s a sandbox environment.

The main reason we weren’t able to deploy a consumer-level or end-user-level application based on computer use was safety. We just didn’t feel confident that, if we gave Claude access to your browser with all your credentials in it, it wouldn’t mess up and take some irreversible action, like sending emails that you didn’t want to send or, in the case of prompt injection, some worse credential-leaking type of thing.

That’s kind of sad because, in its full self-driving mode, it could do a lot for people. It is capable, but the safety just wasn’t good enough to productionize that ourselves. While that’s very ambitious, we think it’s also necessary because the rest of the world isn’t going to slow down either. If we can show that it’s possible to be responsible with how we deploy these capabilities and also make them extremely useful, then that raises the bar.

So I think that's an example where we tried to be really thoughtful about how we rolled it out, but we know that the bar is higher than we're at right now.

Maybe a meta question: How do you think about competition in the provider landscape, and how that turns out?

I think our company philosophy is very aligned with enterprises. If you look at Stripe versus Adyen, for example, nobody knows about Adyen, but at least most people in Silicon Valley know about Stripe. So it's a business-oriented versus a more consumer- and end-user-oriented platform, and I think we're much more like Adyen: We have much less mind share in the world, and yet we can be equally or more successful.

I think our API business is extremely strong, but in terms of what we do next and our positioning, I think it's going to be very important for us to stay out there, because if people can't easily kick the tires on our models and our experiences, then they won't know what to use the models for. We're the best experts on our models by nature, and so I think we're going to need to continue to be out there with things like Claude Code, but we're thinking about how we really let the ecosystem bloom.

I think MCP is a good example of that working well. In a different world, the default path would have been for every model provider to do its own bespoke integrations with only the companies that it was able to get bespoke partnerships with.

Can you just pause and explain to the listeners what MCP is if they haven't heard of it? Because it is an amazing ecosystem-wide coup here.

MCP is the Model Context Protocol. One of our engineers, Justice Byrne Summers, was trying to do some integration between the model and some specific thing for the nth time, and he was like, “This is crazy. There should just be a standard way of getting more information, more context into the model.”

It should be something that anybody can do. Maybe even, if it's well documented enough, Claude can do it itself. The dream is to have Claude be able to write its own integrations on the fly, exactly when you need it, and then be ready to roll.

He created the project, and to be honest, I was skeptical initially. I was like, “Yeah, but why don't you just write the code? Why does it need to be a spec and all these SDKs and stuff?” But eventually, we did this customer advisory board with a bunch of our partner companies, and when we did the MCP demo, the jaws were just on the floor. Everybody was like, “Oh my God, we need this.”

That's when I knew he was right, and we put a bunch more effort behind it and blasted it out. Shortly after our launch, all the major companies asked to be in the loop with the steering committee, asked about our governance models, and wanted to adopt it themselves. So that was really encouraging. OpenAI, Google, Microsoft—all these companies are betting really big on MCP.

This is basically an open industry standard that allows anybody to use this framework to effectively integrate against any model provider in a standardized way.

MCP, I think, is a democratizing force that lets anybody—regardless of what model provider or long-tail service provider they are, and that might even be an internal-only service that only you have—integrate against a fully fledged client, which might look like your IDE or your document editor. It could be pretty much any user interface. I think that's a really powerful combination.

And now remote too.

Yes, yes. Previously, you had to have the services running locally, and that kind of limited it to only being interesting for developers. But now that we have hosted MCP, or sometimes called remote, the service provider, like Google Docs, could provide its own MCP, and then you can integrate that into Claude.ai or whatever surface you wanted.

Ben, thanks for a great conversation.

Yeah, thanks so much.

Thanks for all the great questions.

Find us on Twitter at nopriorspod. Subscribe to our YouTube channel if you wanna see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way, you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.