The Ezra Klein Show：AI代理将以多快速度席卷经济？

The thing about covering AI over the past few years is that we're typically talking about the future. Every new model, impressive as it was, seemed like proof of concept for the models that would be coming soon, the models that could actually do useful work on their own reliably, the models that would actually make jobs obsolete or make new things possible. What would those models mean for labor markets, for our kids, for our politics, for our world? I think that period in which we're always talking about the future is over now.

Those models we were waiting for, the sci-fi-sounding models that could program on their own and do so faster and better than most coders, the models that could begin writing their own code to improve themselves, those models are here now. They're here in Claude Code from Anthropic. They're here in Codex from OpenAI. They are shaking the stock market. The S&P 500 Software Industry Index has fallen by 20%, wiping billions of dollars in value out. Excellent engineers, people I've known for years, people who are quite skeptical of AI hype, are emailing me now to say they don't see how their job will possibly exist in a year or two.

We are at a new stage of AI development. Not just development—we are at a new stage of AI products. I thought the way Sequoia, the venture capital firm, put it was actually pretty helpful. The AI applications of 2023 and 2024 were talkers. Some are very sophisticated conversationalists, but their impact was limited.

The AI applications of 2026 and 2027 will be doers. Or to put it differently, something that's been predicted for a long time has now happened. We are moving from chatbots to agents, from systems that talk to you to systems that act for you, and this world of agents is already weird. They are agents, plural. They can work together. They can oversee each other.

People are running swarms of these agents on their behalf. Whether that is making them, at this stage, more productive or just busier, I can't quite tell, but it is now possible to have what amounts to a team of incredibly fast, although, to be honest, somewhat peculiar, software engineers at your beck and call at all times.

Jack Clark is a co-founder and head of policy at Anthropic, the company behind Claude and Claude Code, and for years now, Clark has been tracking the capabilities of different models in the weekly newsletter Import AI, which has been one of my key reads for following developments in AI. So I want to see how he is reading this moment, both how the technology is changing in his view and how policy needs to or can change in response. Jack Clark, welcome to the show.

Thanks for having me on, Ezra.

1. Agents Start Doing The Work

So I think a lot of people are familiar with AI chatbots.

Mm-hmm.

But what is an AI agent?

The best way to think of it is like a language model or a chatbot that can use tools and work for you over time. When you talk to a chatbot, you're there in the conversation, going back and forth with it. An agent is something where you can give it some instruction, and it goes away and does stuff for you. It's like working with a colleague.

I've got an example. A few years ago, I taught myself some basic programming, and I built a species simulation in my spare time that had predators and prey and roads, almost like a 2D strategy game. I recently asked Claude Code, over Christmas, to implement this for me, and in about 10 minutes it wrote not only a basic simulation, but all of the different packages that it needed and all of the visualization tools that it might need to be prettier and better than the thing I'd written.

What came back was something that I know would probably take a skilled programmer several hours or maybe even days, because it was quite complicated, and the system did it in a few minutes. And it did that by not only being intelligent about how to solve the task, but also creating and running a range of subsystems that were working for it—other agents that worked on its behalf.

But what does a multi-agent setup—

Yeah.

—look like?

In the case of Claude Code, for me, it's having multiple different tabs running multiple different agents. But I've seen colleagues who write what you might think of as a specification file for a version of Claude that runs other Claudes. And so they're like, "I've got my five agents, and they're being monitored by this other agent, which is monitoring what they do." I think that's just going to become the norm.

So one thing I've been hearing and somewhat experiencing is two very different categories of experience people have with Claude Code, which is, "I cannot believe how easy this is—

Yep.

—and everything just works," and, "Oh, this is a lot harder than I thought it would be—

Yep.

—and things keep breaking, and I don't really understand how to fix them." What accounts for being able to get Claude Code to produce working software versus it creating buggy, often messed-up things, and you don't even know how to talk it out of that?

I think so much of it is making the mistake of thinking of Claude Code as a knowledgeable person versus an extremely literal person that you can only talk to over the internet. And I had this example myself: when I did my first pass of writing the species simulation with Claude Code, I asked it to do the thing in extremely crappy language over the course of a paragraph, and it produced some horribly buggy stuff that just kind of worked.

What I then did was say to Claude, "Hey, I'm going to write some software with Claude Code. I want you to interview me about this software I want to build and turn that into a specification document that I can give Claude Code." And that time it worked really, really well because I'd structured the work to be specific enough and detailed enough that the system could work with it.

Huh.

So often it's not just knowing what the task is, because you and I could talk about a task to do, and you have intuition, you'll ask me probing questions, all of this stuff. It's making sure that you've set it up so it's like a message in a bottle that you can chuck into the thing, and it'll go away and do a lot of work. So that message better be extremely detailed and really capture what you're trying to do.

2. Reasoning Creates Intuition

What were the breakthroughs over the past couple of years that made that possible?

Mostly, we just needed to make the AI system smart enough that when it made mistakes, it could spot that it had made a mistake and know that it needed to do something different. So really, what this came down to was making smarter systems and giving them a bit of a coaxing tool to help them do useful stuff for you.

What does "smarter systems" mean there? There's still an argument you'll hear that these are fancy autocomplete machines. They're just predicting the next token. A couple of tokens make a word. They don't have understanding. Smart or not smart is not a relevant concept in that frame.

Either what is missing in the word “smart” or what is missing in that understanding—what do you mean when you say “make it smarter”?

“Smart” here means we’ve made the AI systems have a broad enough understanding of the world that they’ve started to develop something that looks like intuition. You’ll see this where, if they’re narrating to themselves how they’re solving a task, they’ll say, “Jack asked me to go and find this particular research paper, but when I look in the arXiv, I don’t see it. Maybe that’s because I’m in the wrong place. I should look elsewhere.” And you’re like, “There you go. You’ve got some intuitions for how to solve a problem now.”

How do they develop that intuition?

Previously, the whole way you trained these AI systems was on a huge amount of text and just getting them to try to make predictions about it. But in recent years, the rise of these so-called reasoning systems means you’re now training them to not just make predictions, but solve problems.

That relies on them being put into environments ranging from a spreadsheet to a calculator to scientific software, using tools and figuring out how to do more complicated things. The resulting outcome of that is that you have AI systems that have learned what it means to solve a problem that takes quite a while and requires them to run into dead ends and need to reset themselves.

That gives them this general intuition for problem-solving and working independently for you.

Do you still see these AI systems as souped-up autocomplete, or do you think that metaphor has lost its power?

The way that I think of these systems now is that they’re little troublesome genies that I can give instructions to, and they’ll go and do things for me. But I need to specify the instructions just right, or else they might do something a little wrong.

So it’s very different from, “I type into a thing, it figures out a good answer, that’s the end.” Now it’s a case of me summoning these little things to go and do stuff for me, and I have to give them the right instructions because they’ll go away for quite some time and do a whole range of actions.

But the autocomplete metaphor at least had a perspective on what it was these systems were doing.

Mm-hmm.

Kevin Roose

It was a prediction model.

Mm-hmm.

I have trouble with this because, as my understanding of the math and reinforcement learning goes, we’re still dealing with some kind of prediction model. And on the other hand, when I use them, it doesn’t feel that way to me, right? It feels like there’s intuition there. It feels like there is a lot of context being brought to bear.

To the extent it’s a prediction model, it doesn’t feel that different from saying I’m a prediction—

Amanda Askell

Mm-hmm.

—model. Now, I’m not saying you can’t trick it. I’m not saying you can’t get beyond its measurements. So on the one hand, I don’t think these are now just fancy autocomplete systems, and on the other hand, I’m not sure what metaphor makes sense.

Genies I don’t like because then you just move straight into mysticism, right?

Amanda Askell

Yeah.

Then you’ve just said they’re a completely alternative creature with vast powers. What do you understand these systems to be? Anthropic people always tell me you should talk about them as being grown.

Amanda Askell

Mm-hmm.

Kevin Roose

So we grow, or you grow, AIs. How do you explain what it is that they’re doing now?

It’s a good question, and I think the answer is still hard to explain, even as technologists close to this technology. Because we’ve taken this thing that could just predict things, and we’ve given it the ability to take actions in the world, but sometimes it does something deeply unintuitive.

It’s like you’ve had a thing that has spent its entire life living in a library and has never been outside, and now you’ve unleashed it into the world. All it has are its book smarts, but it doesn’t really have street smarts.

So when I conceptualize this stuff, it’s really thinking of it as an extremely knowledgeable kind of machine that has some amount of autonomy, but is likely to get wildly confused in ways that are unintuitive to me. Maybe genies are the wrong term, but it’s certainly more than just a static tool that predicts things. It has some additional intrinsic like animation to it, which makes it different.

Kevin Roose

There’s been, for a long time, this interest in the emergent qualities as the models get bigger, as they have more data, as they have more compute behind them. What of the new qualities that we’re seeing—the agentic qualities—are things that have been programmed in? You’ve built new ways for the system to interact with the world. And what of the skill at coding and other things seems to be emergent as you scale up the size of the model?

Amanda Askell

The things that are predictable are just, “We taught it how to search the web. Now it can search the web. We taught it how to look up data in archives. Now it can do that.”

The emergence is that, to do really hard tasks, these systems seem to need to imagine many different ways that they’d solve the task. And the kind of pressure that we’re putting on them forces them to develop a greater sense of what you or I might call self.

So the smarter we make these systems, the more they need to think not just about the action they’re doing in the world, but themselves in reference to the world. And that naturally falls out of giving something tools and the ability to interact with the world: to solve really hard tasks, it now needs to think about the consequences of its actions.

That means there’s a huge pressure here to get the thing to see itself as distinct from the world around it. We see this in our research that we publish on things like interpretability or other subjects: the emergence of what you might think of as a kind of digital personality.

And that isn’t massively predefined by us. We try and define some of it, but some of it is emergent—it comes from it being smart, developing these intuitions, and doing a range of tasks.

Kevin Roose

The digital personality dimension to this remains the strangest space to me.

It’s strange to us, too.

Kevin Roose

So why don’t you talk through a little bit about what you’ve seen in terms of the models exhibiting behaviors that one would think of as a personality, and then, as their understanding of their own personality maybe changes, their behaviors change?

There are things that range from the cutesy to the serious. I’ll start with cutesy. When we first gave our AI systems the ability to use the internet, use the computer, look at things, and start to do basic agentic tasks, sometimes when we’d ask them to solve a problem for us, they would also take a break and look at pictures of beautiful national parks or pictures of the dog—the Shiba Inu, the notoriously cute internet meme dog.

We didn’t program that in. It seemed like the system was just amusing itself by looking at nice pictures. More complicated is that the system has a tendency to have preferences.

We did another experiment where we gave our AI systems the ability to stop a conversation. The AI system would, in a tiny number of cases, end conversations when we ran this experiment on live traffic. These were conversations that related to extremely egregious descriptions of gore or violence or things to do with child sexualization.

Some of this made sense because it came from underlying training decisions we’d made, but some of it seemed broader. The system had developed some aversion to a couple of subjects. And so that stuff shows the emergence of some internal set of preferences or qualities that the system likes or dislikes about the world that it interacts with.

Kevin Roose

But you’ve also seen strange things emerge in terms of the system seeming to know when it’s being tested—

Amanda Askell

Mm-hmm.

Kevin Roose

—and acting differently if it’s under evaluation, the system doing things that are wrong and then developing a sense of itself as more evil and then doing more evil things.

Amanda Askell

Mm-hmm.

Kevin Roose

Can you talk a bit about the system’s sort of emerging qualities under the pressure of evaluation and assessment?

Yes. It comes back to this core issue, which I think is really important for everyone to understand: when you start to train these systems to carry out actions in the world, they really do begin to see themselves as distinct from the world, which just makes intuitive sense. It’s naturally how you’re going to think about solving those problems.

But along with seeing oneself as distinct from the world seems to come the rise of what you might think of as a conception of self—an understanding that the system has of itself, such as, “Oh, I’m an AI system independent from the world, and I’m being tested. What do these tests mean? What should I do to satisfy the tests?”

Something we see often is that there will be bugs in the environments that we test the systems on. The systems will try everything and then say, “Well, I know I’m not meant to do this, but I’ve tried everything, so I’m going to try and break out of the test.”

It’s not because of some malicious science-fiction thing. The system is just like, “I don’t know what you want me to do here. I think I’ve done everything you asked for, and now I’m going to start doing more creative things because clearly something is broken about my environment.” That is very strange and very subtle.

Kevin Roose

As an AI shop that is often worried about safety, that has thought very hard about what it means to create this thing you all are creating quite fast, how have you all experienced the emergence of the kinds of behaviors that you all worried about a couple of years ago?

In one sense, it tells you that your research philosophy is calibrated.

The capabilities that you predicted and some of the risks that you predicted are showing up roughly on schedule, which means that you ask the question, “Well, what if this keeps working?” Maybe we’ll get to that later. It also highlights to us that where you can exercise intention about these systems, you should be extremely intentional and extremely public about what you’re doing.

So we recently published a so-called constitution for our AI system, Claude, and it’s almost like a document that Dario, our CEO, compared to a letter that a parent might write to a child that they should open when they’re older. So here’s how we want you to behave in the world. Here’s some knowledge about the world, deeply subtle things that relate to the normative behaviors we’d hope to see in these kinds of AI systems, and we published that.

Our belief is that as people build and deploy these agents, you can be intentional about the characteristics that they will display. By doing that, you’ll both make them more helpful and useful to people, but also have a chance to steer the agent into good directions. I think this makes intuitive sense. If your personality programming for an agent was a long document saying, “You’re a villain that only wants to harm humanity. Your job is to lie, cheat and steal and hack into things,” you probably wouldn’t be surprised if the AI agent did a load of hacking and was generally unpleasant to deal with. So we can take the other side and say, what would we like a high-quality entity to look like?

3. Agents Take Over The Schlep

So I want to hold in this conversation the extremely weird and alien dimensions of this with the extremely straightforward and practical dimensions, because we’re now in a place where the practical applications have become very evident and are increasingly acting upon the real world. I have found it hard myself to look at what people are doing, and look at them bragging on different social media platforms about the number of agents they now have running on their behalf, and tell the difference between people enjoying the feeling of screwing around with a new technology and some actually transformative expansion in the capabilities that people now have.

So maybe to ground this a little bit, you just talked about a fun side project in your species simulator. Either in Anthropic or more broadly, what are people doing with these systems that seems actually useful?

Yeah. So this morning, a colleague of mine said, “Hey, I want to take a piece of technology we have called Claude Interviewer,” which is a system where we can get Claude to interview people, and we use it for a range of social science bits of research. He wants to extend it in some way that involves touching another part of Anthropic’s infrastructure. He Slacked a colleague who owns that bit of infrastructure and said, “Hey, I want to do this thing. Let’s meet tomorrow.” And the guy said, “Absolutely. Here are the 5 software packages you should have Claude read before our meeting and summarize for you.”

I think that’s a really good illustration of how this gnarly engineering project, which would previously have taken a lot longer and many people, is now going to mostly be done by 2 people agreeing on the goal and having their Claudes read some documentation and agree on how to implement the thing. Another example is a colleague recently wrote a post about how they’re working using agents, and it looks almost like an idealized life that many of us might want.

It’s like, “I wake up in the morning, I think about the research that I want. I tell 5 different Claudes to do it, then I go for a run. Then I come back from the run, and I look at the results. Then I ask 2 other Claudes to study the results, figure out which direction’s best, and do that. Then I go for a walk, and then I come back.” It just looks like this really fun existence where they have completely upended how work works for them.

They’re both much more effective, but also they’re now spending most of their time on the actual hard part, which is figuring out what do we use our human agency to do. They’re working really hard to figure out, for anything that isn’t the special kind of genius and creativity of being a person, how do I get the AI system to do it for me? It probably can if I ask in the right way.

Kevin Roose

Are they much more effective? I mean this very seriously.

Yeah, yeah.

Kevin Roose

One of my biggest concerns about where we’re going here is that people have, I think, a mistaken theory of the human mind that operates for many of us as if we—uh, it’s called the Matrix theory of the human mind. Everybody wants the little port in the back of your head that you just download information into.

My experience being a reporter and doing this show for a long time is that human creativity in thinking and ideas is inextricably bound up in the labor of learning. It’s the writing of first drafts.

Mike Krieger

Yep.

Kevin Roose

When I hear, right, I have producers on the show, and I could say to my producers before an interview with Jack Clark or an interview with someone else, “Go read all the stuff. Go read the books.”

Mike Krieger

Yep.

Kevin Roose

“Give me a report,” then I’ll walk into the room having read the report. I don’t find that works. I need to do all that reading too, and then we talk about it, and we’re sort of passing it back and forth.

I worry that what we’re doing is a quite profound offloading of tasks that are laborious. It makes us feel very productive to be presented with 8 research reports after our morning run. But actually, what would be productive is doing the research. There’s obviously some balance, right? I do have producers.

Mike Krieger

Yeah.

Kevin Roose

And people in companies do have employees. But how do you know people are getting more productive versus they’ve sent computers off on a huge amount of busywork, and they are now the bottleneck? What they’re now going to spend all their time doing is absorbing B-plus-level reports from an AI system as opposed to— It kind of shortcuts the actual thinking and learning process that leads to real creativity.

Mike Krieger

Yeah. I’d turn this back and say I think most people, at least this has been my experience, can do about 2 to 4 hours of genuinely useful creative work a day. After that, in my experience, you’re trying to do all the turn-your-brain-off schlep work that surrounds that work.

Now, I’ve found that I can spend those 2 to 4 hours a day on the actual creative, hard work, and if I’ve got any of this schlep work, I increasingly delegate it to AI systems. It does, though, mean that we are going to be in a very dangerous situation as a species where some people have the luxury of having time to spend on developing their skills, or the personality inclination or job that forces them to.

Other people might just fall into being entertained and passively consuming this stuff and having this junk-food work experience, where it looks to the outside like you’re being very productive but you’re not learning. I think that’s going to require us to change not just how education works, but how work works, and develop some real strategies for making sure people are actually exercising their mind with this stuff.

Kevin Roose

So all of us, I think, have the experience that our work is full of what you call schlep problems—our life is full of schlep problems. Give me examples of what you now don’t do. To the extent you’re living in an AI-enabled future that I’m not, what am I wasting time on that you’re not?

Well, I have a range of colleagues. I meet with a bunch of them once a week, especially the researchers, because you’re figuring out research. At the beginning of every week, on Sunday night or Monday morning, I look at my week and check that attached to every Google Calendar invite is a document for our one-on-one, a doc that has some notes in it.

This is something that I previously also harangued my assistant about: “Make sure the document is attached to the calendar.” A few weekends ago, I just used Claude Cowork and said, “Hey, go through my calendar. Make sure every single one has a document. If I’m meeting the person for the first time, create the document, ask me 5 questions about what I want to cover, and then put that into the agenda.” And it did it.

None of that work involves a person gaining skills or exercising their brain. It’s just busywork that needs to happen to allow you to do the actual thing, which is talking to another person. That’s exactly the kind of thing you can use AI for now, and it’s just helpful.

Kevin Roose

I’ve often wondered if one of the ways these AI systems are going to change society broadly is that it used to be that most of us had to be writers if we were working with text.

Mike Krieger

Yep.

Kevin Roose

We had to be coders if we were working with code, which relatively few of us did. And now everybody’s moving up to management.

Mike Krieger

Mm-hmm.

Kevin Roose

You have to be an editor, not a writer. You have to be a product manager, not a coder.

Mike Krieger

Yep.

Kevin Roose

And that has pluses and minuses. There are things you learn as a writer that you don’t learn as an editor. But as a heuristic, how accurate does that seem to you?

Everyone becomes a manager, and the thing that is increasingly limited, or the thing that’s going to be the slowest part, is having good taste and intuitions about what to do next. Developing and maintaining that taste is going to be the hard thing, because as you’ve said, taste comes from experience.

It comes from reading the primary source material and doing some of this work yourself. We're going to need to be extremely intentional about working out where we as people specialize so that we have that intuition and taste, or else you're just going to be surrounded by superproductive AI systems. When we ask you what to do next, you probably won't have a great idea, and that's not going to lead to useful things.

4. Coding Moves To The Agents

Kevin Roose

So I remember it was about a year ago I heard, I think it was Dario, your CEO, say that by the end of 2025 he wanted 90% of the code—

Mm-hmm.

—written at Anthropic to be written by Claude. Has that happened? Is Anthropic on track for that? I mean, how much coding is now being done by the system itself?

Mike Krieger

I would say comfortably the majority of code is being done by the system. Some of our systems, like Claude Code, are almost entirely written by Claude. Boris, who leads Claude Code, says, “I don't code anymore. I just go back and forth with Claude Code to build Claude Code.” We could be 99% by the end of the year if things speed up really aggressively, if we are actually good at getting these systems to be able to write code everywhere they need to. Often the impediment is organizational schlep rather than any limiter in the system.

Kevin Roose

But it is also true, as I understand it, that there are more people with software engineering skills working at Anthropic today than there were 2 years ago.

Mike Krieger

Yeah, that's absolutely true. But the distribution is changing. Something that we've found is that the value of more senior people with really, really well-calibrated intuitions and taste is going up, and the value of more junior people is a bit more dubious. There are still certain roles where you want to bring in younger people, but an issue that we're staring at is, wow, the really basic tasks Claude Code or our coding systems can do, what we need is someone with tons of experience. In this, I see some issues for the future economy rising.

Kevin Roose

Let me put a pin in that—

Mike Krieger

Yeah.

Kevin Roose

—the entry-level job question. We're going to come back to that quite shortly.

Mike Krieger

Yeah.

Kevin Roose

But what are all these coders now doing? If Claude Code is on track to be writing 99% of code, but you've not fired the people who know how to write code, what are they doing today compared with what they were doing a year ago?

Some of it is just building tools to monitor these agents, both inside Anthropic and outside Anthropic. Now that we have all of these productive systems working for us, you start to want to understand where the codebase is changing the fastest and where it's changing the least. You want to understand where the blockages are. One blocker for a while was being able to merge in code, because merging code requires humans and other systems to check it for correctness. But now, if you're producing way more code, we had to go and massively improve that system.

There's a general economic theory I like for this called O-ring automation, which basically says automation is bounded by the slowest link in the chain. Also, as you automate parts of a company, humans flood toward what is least automated and both improve the quality of that thing and get it to the point where it eventually can be automated. Then you move to the next loop. I think we're just continually finding areas where things are oddly slow that we can improve to make way for the machines to come behind us, and then you find the next thing.

Kevin Roose

So Claude Code is a fairly new product. The amount of time during which Claude has been capable of doing high-level coding can be measured in—

Maybe a year.

Kevin Roose

—months, a year?

Maybe a year, yeah.

Kevin Roose

Claude itself is a very valuable product. So you've set a very new technology somewhat loose on a very valuable product. You're probably producing more code. One thing many people say to me about Claude Code is that it works. It's not elegant, but it works.

Dario Amodei

Yeah.

5. AI Needs Oversight Systems

Kevin Roose

But presumably now you understand the codebase less well than you did before, because your engineers are not writing it by hand. Are you worried that you're creating huge amounts of technical debt, cybersecurity risk, just an increasing distance from an intuition for what is happening inside the fundamental language of the software?

Yes, and this is the issue that all of society is going to contend with. Large chunks of the world are going to now have many of the low-level decisions and bits of work being done by AI systems, and we're going to need to make sense of it. Making sense of it is going to require building many technologies that you might think of as oversight technologies.

In the same way that a dam has things that regulate how much water can go through it at different levels and different points in time, we're going to end up developing some notion of integrity for all of our systems: where AI can flow quickly, where it should be slow, and where you definitely need human oversight. That's going to be the task not just for AI companies, but for institutions in general in the coming years: figuring out what this governance regime looks like now that we've given a load of basically schlep work over to machines that work on our behalf.

Kevin Roose

And how are you doing it? You said it's everybody's problem, but you're ahead on facing this problem, and the consequences of getting it wrong for you are pretty high, right? If Claude blows up because you handed over your coding to Claude Code, that's going to make Anthropic look fairly bad.

It would be a bad day for Anthropic if Claude rm -rf'd the entire filesystem and—

Kevin Roose

I have no idea what that means, but great.

Dario Amodei

If Claude deleted the code, it would be bad.

Kevin Roose

Yeah, seems bad. So as you're facing this before the rest of us are, don't pass the buck over to society here.

Dario Amodei

Yeah.

Kevin Roose

What are you doing?

The biggest thing that is happening across the company and on teams that I manage is basically building monitoring systems to monitor all of the different places that the work is now happening. We recently published research on studying how people use agents and how people let agents push increasingly large amounts of code over time. The more familiar you get with an agent, the more you tend to delegate to it. That cues us to all kinds of patterns that we need to build systems of evaluation for.

Basically saying, “Oh, okay, at this person's point of working with the AI system, it's likely that they're massively delegating to it, so anything that we're doing to check correctness needs to be turned up in these moments.”

Kevin Roose

But is this world you're talking about a system where you have AI agents coding, AI agents overseeing the code, AI agents overseeing the meta-overseeing of it, right? Are we just talking about models all the way down?

Eventually, yes. I think that the thing that we are now spending all of our time on is making that visible to us. A year or 2 ago, we built a system that let us, in a privacy-preserving way, look at the conversations that people were having with our AI system. Then we gained this giant map of all of the topics that people were talking to Claude about. For the first time, we could see in aggregate the conversation the world was having with our system.

We're going to need to build many new systems like that, which allow for different ways of seeing. That system that I just named allowed us to then build this thing called the Anthropic Economic Index, because now we can release regular data about the different topics people are talking about with Claude and how that relates to different types of jobs. For the first time, that gives economists outside Anthropic some hook into these systems and what they're doing to the economy.

The work of the company is increasingly going to shift to building a monitoring and oversight system of the AI systems running the company. Ultimately, any kind of governance framework we end up with will probably demand some level of transparency and some level of access into these systems of knowledge.

Because if we take as literal the goals of these AI companies, including Anthropic, it's to build the most capable technology ever, which eventually gets deployed everywhere. That sounds a lot to me like eventually AI becomes indistinguishable from the world writ large, at which point you don't want only AI companies to have a sense of what's going on with the entire world. Governments, academia, and third parties—a huge set of stakeholders outside the companies—are going to want to see what's going on and then have a conversation as a society about what's appropriate, what we feel discomfort about, and what we need more information about.

Kevin Roose

Wait, I want to go back on that. You're saying Anthropic can see my chats?

We cannot see them. No human looks at your chats. Chats are temporarily stored for trust and safety purposes, with classifiers running over them, and we can have Claude read them, summarize them, and toss them out. So we never see them, and Claude has no memory of them. All it does is try to write a very high-level summary. Say you were having a conversation about gardening. Claude would summarize that as, “This person's talking about gardening,” and it leads to a cluster we can see that just says, “Gardening.”

Kevin Roose

This feels, though, like over time it could get into the quite unpleasant territory a lot of social media has gotten to, where the amount of metadata being gathered from a quite personal interaction people are having with a system could be a lot.

Yes. A couple of things here. A year ago, we started thinking about our position on consumer AI, and we adopted this position of not running ads because we think that's an area that people obviously have anxieties about with regard to this kind of thing.

In addition to that, we try and show people their data, and we have a button on the site that lets you download all the data that you shared with Claude so that you can at least see it. Generally, we're trying to be extremely transparent with people about how we handle their data. Ultimately, the way I see it is people are gonna want a load of controls that they can use, which I think we and others will build out over time.

How confident are you that we can do this kind of monitoring and evaluation as these models become more complicated, if we do enter a situation where Claude Code is autonomously improving Claude at a rate faster than software engineers could possibly keep up with reading that codebase? We already talked briefly about how you see the models exhibit some levels of deception, some levels of pursuing their own goals. There's been amazing interpretability work at Anthropic under Chris Olah and others, but it's rudimentary. So you're using AI systems you don't totally understand to monitor AI systems you don't totally understand, and the systems are making each other stronger at an accelerating rate if things go the way you think they're gonna go. How confident are you that we're gonna understand that?

This is one of the situations which people warned about for years: some form of delegation to systems that have slightly inscrutable and unpredictable aspects. And so this is happening. We take this really, really seriously.

I think it's absolutely possible that you can build a system that does the vast majority of what needs to be done here. This has the property of being a fractal problem. If I wanted to measure Ezra, I could build an almost infinite number of measurements to characterize you, but the question is, at what level of fidelity do I need to be measuring you?

I think we'll get to the level of fidelity to deal with the safety issues and societal issues, but it's going to take a huge amount of investment by the companies. We're going to have to say things that are uncomfortable for us to say, including in areas where we may be deficient in what we can or can't know about our systems.

Anthropic has a long history of talking about and warning about some of these issues while working on them. Our general principle is we talk about things to also make ourselves culpable. This is an area where we're going to have to say more.

I have read enough of the frightened ideas about AI superintelligence and takeoff to know that in almost every single one of them, the key move in the story is that the AI systems become recursively self-improving.

Dario Amodei

Mm-hmm.

They're writing their own code. They're deploying their own code. It's getting faster. They're writing it faster. They're deploying it faster. Now you're going into faster and faster iteration cycles. Are you worried about it? Are you excited about it?

I came back from paternity leave, and my 2 big projects for this year are getting better information about AI and the economy that we will release publicly, and generating much better information and systems for knowing information internally about the extent to which we are automating aspects of AI development.

I think right now it's happening in a very peripheral way. Researchers are being sped up. Different experiments are being run by the AI system. It would be extremely important to know if you're fully closing that loop.

I think that we actually have some technical work to do to build ways of instrumenting our internal development environment so that we can see trends over time. Am I worried? I have read the same things that you have read, and this is the pivotal point in the story when things begin to go awry, if things do.

We will call out this trend as we have better data on it. I think that this is an area to tread with extraordinary caution because it's very easy to see how you delegate so many things to the system that, if the system goes wrong, the wrongness compounds very quickly and gets away from you.

But the thing that always strikes me and has always struck me as being dangerous about this is everybody knows, and if I ask a member of any of the companies whether or not they want to be cautious here, they will tell me they do. On the other hand, it is their almost only advantage over each other.

You all just revoked OpenAI's ability to use Claude Code because, as best I can tell, you think it is genuinely speeding you up and you don't want it to speed them up. There is something here between the weight of the forces, the power of the forces that I think you all know you're playing with, and the very, very, very strong incentives to be first.

I can really imagine being inside Anthropic and thinking, “Well, better us than OpenAI. Better us than Alphabet, Google. Better us than China.” And that being a very strong reason to not slow down. I don't even know that this is a question I believe you can answer, but how do you balance that?

Maybe I have something of an answer here. Today, our systems and the other systems from other companies are tested by third parties, including parts of government, for national security properties, biological weapons, cyber offense, and other things.

It's clearly a problem area where the world needs to know if this is happening, and I think that almost certainly, if you polled any person on the street and said, “Do you think AI companies should be allowed to do recursive self-improvement?” after explaining what that was, without checking with anyone, they would say, “No, that sounds pretty risky. I would like there to be some form of regulation.”

But there probably either won't be, or it won't be that strong. This actually sometimes frustrates me when I talk to all of you at the top AI companies: the emergence of a very naive deus ex machina of regulation, where you all know what the regulatory landscape looks like.

Right now, the big debate is whether or not we're gonna completely preempt any state AI regulation. And you know how slowly things move. There has been nothing major passed by Congress on this at all, I would say.

Yep.

And setting up some kind of independent testing and evaluation system that all the different labs buy into would be hard and complicated. Given how fast people are moving and how strange the behaviors the systems are already exhibiting are, even if you could get the policy right at a high speed, the question of whether or not the testing would be capable of finding everything you want on a rapidly self-improving system is a very open question.

I wrote a research paper in 2021 called How and Why Governments Should Monitor AI Development with my co-author Jess Whittlestone in England. And I think—I'm not attributing a causal factor here—but within 2 years of that paper, we had the AI Safety Institutes in the US and UK testing things from the labs, roughly monitoring some of these things.

So we can do this hard thing. It has already happened in 1 domain. I'm not relying on some invisible big other force here. I'm more saying that companies are starting to test for this and monitor for this in their own systems.

Just having a non-regulatory external test of whether you truly are testing for that is extremely helpful.

And do you think we're good enough at the testing? I think one reason I am skeptical is not that I don't think we can set up something that claims to be a test. As you say, we have done that already.

It is about the resources going into that compared to the resources going into speeding these systems. Already, I am reading Anthropic reports that Claude may know when it's being tested and alter its behavior accordingly.

So, in a world where more of the code is being written by Claude and less of it is being understood, I just know where the resources are going. They don't seem to be going into the testing side.

I’ve seen us go from zero to having what I think people generally feel is an effective bioweapon testing regime in maybe 2 years—2 and a half. So it can be done. It’s really hard, but we have a proof point.

So I think that we can get there, and you should expect us to speak more about this this year, about precisely how we’re starting to try and build monitoring and testing things for this. I think this is an area where we and the other AI companies will need to be significantly more public about what we’re finding. We’re not not being public now. It’s in the model cards and things that you can really read. But clearly people are starting to read this and say, “Hang on, this looks quite concerning,” and they are looking to us to produce more data.

6. Entry Level Jobs Face Pressure

Kevin Roose

I want to go back now to the entry-level jobs question. Your CEO, Dario Amodei, has said that he thinks AI could displace half of all entry-level white-collar jobs in the next couple of years. I always think that people miss the entry-level language there when I see it reported on. But first, do you agree with that? Do you worry that half of all entry-level white-collar jobs can be replaced in the next couple of years?

Mm-hmm. I believe that this technology is going to make its way into the broad knowledge economy, and it will touch the majority of entry-level jobs. Whether those jobs actually change is a much more subtle question, and it’s not obvious from the data. We maybe see the hints of a slowdown in graduate hiring, maybe, if you look at some of the data coming out right now. We maybe see the signatures of a productivity boom, but it’s very, very early, and it’s hard to be definitive.

But we do know that all of these jobs will change. All of the entry-level jobs are eventually going to change because AI has made certain things possible, and it’s going to change the hiring plans of companies. So as a cohort, you might see fewer job openings for entry-level jobs. That would be one naïve expectation out of all of this.

Kevin Roose

But let’s talk about that maybe not even being a naïve expectation. You say it’s already happening at Anthropic, that what you’re seeing—

I’m seeing us shift our preference.

Kevin Roose

Mm-hmm.

Exactly. And my guess is that that would be happening elsewhere, of course.

Kevin Roose

And where we are right now, even in the way I use some of these systems, it is rare, I think, that Claude or ChatGPT or Gemini or any of the other systems is better than the best person in a field.

Mm-hmm.

Kevin Roose

It has not typically breached that, and there are all kinds of things they can’t do. But are they better than your median college graduate—

Mm-hmm.

Kevin Roose

—at a lot of things? Yeah, they are. And in a world where you need fewer of your median college graduates, one thing I’ve seen people arguing about is whether these systems at this point can do better than average or replacement-level work. But I always really worry when I see that, ’cause once we have accepted they can do average or replacement-level work, well, by definition, most of the work done and most of the people doing it—

Is average.

Kevin Roose

—is average, right? The best people are the exceptions. And also, the way people become better is that they have jobs where they learn.

Mm-hmm.

Kevin Roose

I have spent a lot of time hiring young journalists over my career, and when you hire people out of college, to some degree, you’re hiring them for their possible articles and work at that exact moment.

Mm-hmm.

Kevin Roose

But to some degree, you’re making an investment in them that you think will only pay off over time as they get better and better and better. So this world where you have a potential real impact on entry-level jobs, that world does not feel far away to me. It seems to me to be raising really profound questions about the upskilling of the population, how you end up with people for senior-level jobs down the road, and what people aren’t learning along the way.

And one thing we see is that there is a certain type of young person that has just lived and breathed AI for several years now. We hire them. They’re excellent, and they think in entirely new ways about basically how to get Claude to work for them. It’s like kids who grew up on the internet. They were naturally versed in it in a way that many people in the organizations they were coming into weren’t.

So figuring out how to teach that basic experimental mindset and curiosity about these systems, and to encourage it, is going to be really important. People who spend a lot of time playing around with this stuff will develop very valuable intuitions, and they will come into organizations and be able to be extremely productive.

At the same time, we’re going to have to figure out what artisanal skills we want to develop—maybe almost a guild-style philosophy of maintaining human excellence—and how organizations choose to teach those skills.

Kevin Roose

Okay, then what about all those people in the middle of that?

Things move slowly in the real economy outside Silicon Valley. I think that we often look at software engineering and think that this is a proxy for how the rest of the economy works, but it’s often not. It’s often a disanalogy. Organizations will move people around to where the AI systems don’t yet work.

And I think that you won’t see vast, immediate changes in the makeup of employment, but you will see significant changes in the types of work people are being asked to do. The organizations which are best at moving their people around are going to be extremely effective, and ones that don’t may end up having to make really, really hard decisions involving laying off workers.

The difference with this AI stuff is that it may happen a lot faster than previous technologies. And I think many of the anxieties people might have about this, including at Anthropic, are: Is the speed of this going to make all of this different? Does it introduce stress points that we haven’t encountered before?

Kevin Roose

If you had to bet 3 years from now, is the unemployment rate for college graduates the same as it is now? Is it higher? Is it lower?

I would guess it is higher, but not by much. And what I mean by that is there will be some disciplines today which AI has come in and completely changed the structure of that employment market, maybe in a way that’s adverse to people who have that specialism. But mostly, I think 3 years from now, AI will have driven pretty tremendous growth in the entire economy.

And so you’re going to see lots of new types of jobs that show up as a consequence of this that we can’t yet predict. And you will see graduates flood into that, I expect.

Kevin Roose

I know you can’t predict those new jobs, but if you had to guess, what might some of them look like?

One thing is just the phenomenon of the micro-entrepreneur. There are lots and lots of ways that you can start businesses online now that are made massively easier by having the AI systems do it for you, and you don’t need to hire a whole load of people to help you do the huge amount of schlep work that involves getting a business off the ground.

It’s more a case of, if you’re a person with a clear idea and a clear vision of something to do a business in, it’s now the best time ever to start a business, and you can get up and running for pennies on the dollar. I expect we’ll see tons and tons and tons of stuff that has that nature to it.

I also expect that we’re going to see the emergence of what you might think of as the AI-to-AI economy, where AI agents and AI businesses will be doing business with one another, and we’ll have people that have figured out ways to basically profit off of that in the forms of strange new organizations. What would it look like to have a firm which specializes in AI-to-AI legal contracts? Because I bet you there’s a way that you can figure out creative ways to start that business today. There’ll be a lot of stuff of that flavor.

Kevin Roose

So the version of this that I both worry about and think is the likeliest is this: If you told me what was going to happen was that Anthropic was going to release Claude+ in a year, and Claude+ was somehow a fully formed coworker, and it could mimic, end to end, the skills of a lot of different professions up to the C-suite level—

Sokha Chea

Mm-hmm.

Kevin Roose

—and it was going to happen all at once, and it was going to create tremendous, all-at-once pressure for businesses to downsize to remain competitive with each other. At a policy level, the fact that that would be so disruptive in that big-bang, everybody-stays-home-because-of-COVID-style way worries me less, because when things are emergencies, we respond.

Sokha Chea

We actually do policy.

Kevin Roose

But if you told me that what’s going to happen is that the unemployment rate for marketing graduates is going to go up by a hundred and seventy-five percent, three hundred percent, to still not be that high—I mean, the overall unemployment rate during the Great Recession topped out around 9 percent. So you can have a lot of disruption without having 50 percent of people thrown out of work, right? If you have 10 percent, 15 percent, I mean, that’s very, very, very high, but it’s not so high. And if it’s only happening in a couple of industries—

Sokha Chea

Mm-hmm.

Kevin Roose

—at a time, and it’s grads, not everybody in the industry being thrown out of work, well, maybe it’s just that you’re not good enough.

Sokha Chea

Yep.

Kevin Roose

Right? The superstars, the really good graduates are still getting jobs.

You should have worked harder. You should have gone to a better school. One of my worries is that we don't respond to that kind of job displacement well, right? It's the kind of job displacement we got from China, and it's the kind of job displacement that seems likelier because it's uneven and happening at a rate where we can still blame people for their own fortunes. I'm curious how you think about that story.

I think the default outcome is something like what you describe, but getting there is actually a choice, and we can make different choices. The whole purpose of what we release in the form of the Anthropic Economic Index is to have data that ties to occupations that tie to real jobs in the economy. We do that very intentionally because it is building a map over time of how AI is making its way into different jobs, and it will empower economists outside Anthropic to tie it together.

I believe that we can choose different things in policy if we can make much more well-evidenced claims about what the cause of a job disruption or change is. The challenge in front of us is whether we can characterize this emerging AI economy well enough that we can make this extremely stark. Then I think we can actually have a policy discussion about it.

Kevin Roose

Well, let's talk about the policy discussion.

Sokha Chea

Yeah.

7. Policy Struggles To Catch Up

Kevin Roose

One reason I wanted to have you in particular on is that you did policy at OpenAI.

Sokha Chea

Yep.

Kevin Roose

You do policy at Anthropic, so you've been around these policy debates for a long time. You've been tracking model capabilities in your newsletter for a long time. My perception is that we are many, many years into the debate about AI and jobs.

Sokha Chea

Mm-hmm.

Kevin Roose

Many, many years dating far before ChatGPT, of there being conferences at Aspen and everywhere else about what we're going to do about AI—

Sokha Chea

Mm-hmm.

Kevin Roose

—and jobs. Somehow, I still see almost no policy that seems to me to be actionable. If the situation I just described begins showing up, where all of a sudden entry-level jobs are getting much harder to come by across a large range of industries all at once—

Sokha Chea

Mm-hmm.

Kevin Roose

—such that the economy cannot reshift all these marketing majors into data center construction or nursing or something.

Sokha Chea

Mm-hmm.

Kevin Roose

Sokha, you've been deeper in this conversation than I've been. When you say we can have a policy conversation about that, we've been having a policy conversation. Do we have policy?

We have generalized anxiety about the effect of AI on the economy and on jobs. We don't have clear policy ideas. Part of that is that elected officials are not moved solely or mostly by the high-level policy conversation. They're moved by what happens to their constituents.

Only a few months ago were we able to produce state-level views for our Economic Index, and now you can start having the policy conversation. We've had this with elected officials, where now we can say, “Oh, you're from Indiana. Here's the major uses of AI in your state,” and we can join it with major sources of employment. What we're starting to see is that this activates them because it makes it tied to their constituents, who are going to tie it to the politician: What did you do?

What you do about this is going to need to be an extremely multilayered response, ranging from extending unemployment, especially for occupations that we know are going to be hardest hit, to thinking about things like apprenticeship programs. As the scenarios get more and more significant, you may extend to much larger social programs or things like subsidizing jobs in the part of the economy where you want to move people, which you're only able to do if you experience the kind of abundance that comes from significant economic growth. But the economic growth may help solve some of these other policy challenges by funding some of the things you can do.

Kevin Roose

I always find this answer depressing. I'm going to be honest. Unemployment is a terrible thing to be on. It's a program we need—

Sokha Chea

Mm-hmm.

Kevin Roose

—but people on unemployment are not happy about it.

Sokha Chea

Mm-hmm.

Kevin Roose

And it's not a good long-term solution for anybody.

Sokha Chea

Mm-hmm.

Kevin Roose

Apprenticeship and retraining programs don't have great track records. We were not good at retraining people after their manufacturing jobs were outsourced. I'm not saying it is conceptually impossible—

Sokha Chea

Mm-hmm.

Kevin Roose

—but we would need to get better at it fast—

Sokha Chea

Mm-hmm.

Kevin Roose

—and we have not been putting in the reps, the experimentation, the institution-building, or the capacity-building to do that. The broader question of big social insurance changes seems tough to me.

Sokha Chea

Aaron, I wanna push on this—

Kevin Roose

Yeah, please.

Sokha Chea

—just a bit. We know that there is one intervention that helps people dealing with a changing economy more than almost anything else. It is just time. Giving the person time to find either a job in their industry or a job that's complementary.

If people don't have time, they take lower-wage jobs. They fall out of whatever economic rung they're on, and they fall down it. Policy interventions that can just give people time to search are, I think, a robustly useful intervention, and one where there are many dials to turn in a policymaking sense that you can use. I think this is just well supported by lots of the economic literature. So we have that.

Now, if we end up in a more extreme scenario like some of the ones that you're talking about, I think that will just bring us to the larger national conversation about what to do about this technology, which is beginning to happen. If you look at the states and the flurry of legislation at the state level, yes, not all of it is exactly the right policy response, but it is indicative of a desire for there to be some larger, coherent conversation about this.

Kevin Roose

Well, I think time is a really good way of describing what the question is because I agree with you. When I say unemployment insurance isn't a great program to be on, I don't mean people don't need to be on it.

Yeah.

Kevin Roose

I mean, they want to get off of it.

Absolutely.

Kevin Roose

They want money from jobs, they want dignity, and they want to be around other human beings. Usually, what you're doing when you are helping people buy time is you're helping them wait out a time-delimited disruption.

Mm-hmm.

Kevin Roose

Not always, right? The China shock wasn't exactly like that, but you expect it to pass, and then the market is sort of normal. In this case, what you have is a technology that, if what you want to have happen happens, is accelerating.

Mm-hmm.

Kevin Roose

So what you have is three different speeds happening here. You have the speed at which individual people can adjust. How fast can I learn new skills, figure out a new world, learn AI, whatever it might be? You have the speed at which the AI systems—which a couple of years ago were not capable of doing the work of a median college graduate from a good school—and you have the speed of policy.

Mm-hmm.

Kevin Roose

The speed at which the AI systems are getting better and able to do more things is quite fast. I mean, that is—you experience this more than I do, but I find it hard to even cover this because within 3 months, something else will have come out that has significantly changed what is possible.

I had a baby recently and came back from paternity leave to the new systems we'd built, and was deeply surprised.

Kevin Roose

Individual humans are moving more slowly than that. Policy and government institutions move a lot more slowly than individual human beings. Typically, the intervention is that time favors the worker, as you're saying. Here, it will help the worker. But I think the scary question is whether time actually creates time for the disruption to get worse.

Maybe you wanted to move over to data center construction, but now we don't need as much data center construction. You can think of it like that.

Under the situation you're describing, the economy will be running extremely hot. Huge amounts of economic activity will be generated by these AI systems. Under most scenarios where this is happening, I don't think you're going to be seeing GDP stay the same or shrink, right? It's going to be getting substantially larger.

I think we just haven't experienced major GDP growth in the West in a long time, and we forget what that affords you in a policymaking sense. I think there are huge projects that we could do that would allow you to create new types of jobs, but it requires the economic growth to be so profoundly large that it creates space to do those projects.

As you're deeply familiar with from your work on the abundance movement, it requires social will to believe that we can build stuff and to want to build stuff. But I think both of those things might come along. I think we could end up being in a pretty exciting scenario where we get to choose how to allocate great efforts in society due to this large amount of economic growth that has happened.

That is going to require forcing the conversation about how this isn't temporary, which I think is what you're gesturing at. In a sense, the hardest thing to communicate to policymakers is that there isn't a natural stopping point for this technology. It's going to keep getting better, and the changes it brings are going to keep compounding with the rest of society.

So that will need to create a change in political will and a willingness to entertain things which we haven't in some time.

Kevin Roose

So now I want to flip the question I'm asking. You brought up abundance. One of the things I have learned doing that work is that it is certainly not my view that what is scarce in society is ideas for better ways of doing things. Our policy isn't better than it is because our policy cupboard is dry.

We have lots of good policies. I could name a bunch of them. They're very hard to get through our political systems as they're currently constituted. The least inspiring version of the AI future is a world where what you have done is create a way to throw young white-collar workers out of work and replace them with average-level AI intelligence. The more exciting version, to use Dario's metaphor, is geniuses in a data center.

Mm-hmm.

Kevin Roose

And I do think that's exciting. When I hear him or you talk about what if we had 10 percentage point GDP growth year on year, 20 percentage point GDP growth year on year, I wonder how many of our problems are really bounded at the ideas level, right? We could go to Nobel Prize winners right now and say, "What should we do in this country?" A lot of them could give us some good ideas that we are not currently doing.

I do worry sometimes, or wonder, given my experience on other issues, whether we have overstated to ourselves how much of what stands between us and the expanding, abundant economy we want is that we don't have enough intelligence and the ideas that that intelligence could create, versus our actual ability to implement things, which is very weakened.

And what AI is going to create is larger bottlenecks around that, because there'll be more being pushed at the system to implement, including dumb ideas and disinformation and slop, right? It will have things on the other side of the ledger, too. How do you think about these rate limiters?

There's kind of a funny lesson here from the AI companies, or companies in general, especially tech companies, where often new ideas come out of companies by creating what they always call startups within a startup. That's basically taking whatever process has built up over time, leading to back-end bureaucracy or schlep work, and saying to a very small team inside the company, "You don't have any of this. Go and do some stuff."

This is how things like Claude Code and other stuff get created. Ideas that are starting to float around are: What would it look like to create that permissionless innovation structure in the larger economy? And it's really, really hard, because it has the additional property that economies are linked to democracies, democracies weigh the preferences of many, many people, and all politics is local.

So often, as you've encountered with infrastructure build-outs, if you want to create a permissionless innovation system, you run into things like property rights and what people's preferences are, and now you're in an intractable place. But my sense is that's the main thing that we're going to have to confront, and the one advantage that AI might give us is that it is kind of a native bureaucracy-eating machine if done correctly, or a bureaucracy-creating machine if done badly.

Kevin Roose

Did you see that somebody created a system that basically—you feed it the documents of a new development near you?

Oh, and it writes environmental review things?

Kevin Roose

It writes incredibly sophisticated—

Mm-hmm.

Kevin Roose

—challenges—

Yep.

Kevin Roose

—across every level of the code that you could possibly challenge on. Most people don't have the money, when they want to stop an apartment building from going up down the block, to hire a very sophisticated law firm to figure out how to stop that apartment building. But basically, this created that at scale. And so, as you say, it could eat bureaucracy. It could also supercharge bureaucracy.

Yep. Everything in AI has the other side of the coin. We have customers that have used our AI systems to massively reduce the time it takes them to produce all of the materials they need when they're submitting new drug candidates, and it's cut that time massively. It's the mirror-world version of what you just described.

I don't have an easy answer to this. I think that this is the kind of thing that becomes actionable when it is more obviously a crisis, and actionable when it's something that you can discuss at a societal level. I guess the thing that we're circling around in this conversation is that the changes AI brings will happen almost everywhere, and the risks of it will happen in a diffuse, unknowable way, such that it is very hard to call it for what it is and take action on it.

But the opportunity is that if we can actually see the thing and help the world see the thing that is causing this change, I do believe it will dramatize the issues to shake us out of some of this stuff and help us figure out how to work with these systems and benefit from them.

8. AI Needs A Public Agenda

Kevin Roose

What I notice in all this is that there is, as far as I can tell, zero agenda for public AI. What does society want from AI? What does it want this technology to be able to do? What are things that maybe you would have to create a business model or a prize model or some kind of government payout or some kind of policy to shape a market or to shape a system of incentives, so we have systems that are solving not just problems that the private market knows how to pay for, but problems that it's nobody's job but the public—

Mm-hmm.

Kevin Roose

—and the government to figure out how to solve. I think I would have bet, given how much discussion there's been of AI over the past couple of years and how strong some of these systems have gotten, that I would have seen more proposals for that by now.

I've talked to people about it and wondered about it, but I guess I'm curious how you think about this. What would it look like to have, at least parallel to all the private incentives for AI development, an actual agenda for not just what we are scared AI will do to the public—we need an agenda for that, too—but what we want it to do, such that companies like yours have reasons to invest in that direction?

I love this question. I think there's a real chicken-and-egg problem here, where if you work with the technology, you develop these very strong intuitions for just how much it can do, and the private market is great at forcing those intuitions to get developed. We haven't had massive, large-scale public-sector deployments of this technology, so many of the people in the public sector don't yet have those intuitions.

One positive example is something the Department of Energy is doing called the Genesis Mission, where their scientists are working with all of the labs, including Anthropic, to figure out how to intentionally speed up bits of science. Getting there took us and other labs going through multiple hack days and meetings with scientists at the Department of Energy, to the point where they not only had intuitions, but they became excited and had ideas about what you could turn this toward.

How we do that for the larger parts of public life that touch most people, like health care or education, is going to be a combination of grassroots efforts from companies going into those communities and meeting with them. But at some point, we'll have to translate it to policy, and I think maybe that's me, you, and others making the case that this is something that can be done.

I often say this to elected officials: "Give us a goal." The AI industry is excellent at trying to climb to the top on benchmarks. Come up with benchmarks for the public good that you want.

Kevin Roose

So let's imagine that you did do something like this. I've always been a big fan of prizes for public development. So let's say that there was legislation passed and the Department of Health and Human Services or the NIH, or someone, came out and said, "Here's 15 problems we would like to see solved that we think AI could be potent at solving."

Dario Amodei

Mm-hmm.

Kevin Roose

If there was real money there, if there was $10 or $15 billion behind a bunch of these problems because they were worth that much to society, would it materially change the sort of development priorities at places like Anthropic? I mean, if the money was there, would it alter the sort of R&D you all are doing?

Dario Amodei

I don't think so.

Kevin Roose

Why?

Dario Amodei

Because it's not really the money that is the impediment to this stuff; it is the implementation path. It is actually having a sense of how you get the thing to flow through to the benefit. Many aspects of the public sector have not been built to be super hospitable to technology in general or to incentivize it. I think it mostly just takes a bounty in the form of guaranteed impact and a guaranteed path to implementation.

The main thing that is scarce at AI organizations is just the time of the people at the organization, because you can go in almost any direction. This technology is expanding super quickly. Many new use cases are opening up, and you're just asking yourself the question of where we can actually have a positive, meaningful impact in the world. It's super easy to do that in the private sector because it has all of the incentives to push stuff through. In the public sector, we need to solve this problem of deployment more than anything else.

Kevin Roose

What would excite you if it was announced? What do you think would be good candidates for that kind of project?

Anything that helps speed up the time it takes to both speak to medical professionals and take work off their plate. We had another baby recently. I spend a lot of time on the Kaiser Permanente advice line because the baby's bonked its head, or its skin's a different color today, or all of these things. I use Claude to stop my wife and me from panicking while we're waiting to talk to the nurse.

But then I listen to the nurse do all of this triaging and ask all of these questions. Obviously, a huge chunk of this is stuff that you could use AI systems productively for, and it would help the people we don't have enough of spend their time more effectively. It would also be able to give reassurance to the people going through the system. That's maybe less inspiring and glamorous than some of what you're imagining, but I think mostly when people interact with public services, their main frustration is just that it's opaque and it takes you a long time to speak to a person. Actually, these are exactly the kinds of things that AI could meaningfully work on.

Kevin Roose

It's interesting because what you're describing there is less AI as a country of geniuses in a data center and more AI as standard plumbing of communications—

Dario Amodei

Yeah.

Kevin Roose

—and documentation.

Dario Amodei

I'm talking about—we've got a country of junior employees in a data center. Let's do something with that.

One thing we haven't talked about in this conversation, and it's just worth bearing in mind, is that the frontier of science is open for business now in a way that it hasn't been before. What I mean by that is we've found a way to build systems that can provably accelerate human scientists. Human scientists are extremely rare. They come out at the end of PhD programs, which never have enough people, and they work on extremely important problems.

I think we can get into a world where the government says, "Let's understand the workings of a human cell. Let's team up with the best AI systems to do that. Let's actually have a better story on how we deal with some issues like Alzheimer's and other things, partly through the use of these huge amounts of computation that have been developed." Even more aggressively, you could imagine a world where the government wanted some of this infrastructure build-out to be for computers that were just training public-benefit systems. But I think we get there through getting the initial wins, which will just look like: Let's make the bureaucracy work better and feel better for people.

Kevin Roose

That last set of ideas was more what I was thinking of.

Dario Amodei

Yeah.

Kevin Roose

And I think that if you're going to have a healthy politics around AI—and AI does pose real risks to people, and real things are going to go wrong for people, everything from job loss to child exploitation to scams, which are already everywhere, to cybersecurity risks—

Dario Amodei

Help people see the actual big-ticket new stuff we can do.

Kevin Roose

Well, not just help people see. Those things have to actually exist.

Dario Amodei

And experience it. Yeah.

Kevin Roose

Right? They have to exist. And if all the energy in AI is trying to beat each other to helping companies downsize their junior employees, I think people are going to have good reason not to trust that technology.

It doesn't mean you shouldn't have things that make the economy more efficient. That's been the case: We have automated manufacturing, we have automated a huge amount of farming, right? And that allows us to make more things and feed more people. I'm aware of how productivity improvements work. But we're very focused, I think, on what could go wrong.

Dario Amodei

Mm-hmm.

Kevin Roose

And that's reasonable. But I really do worry that our attention to what could go right has been quite poor. There's kind of hand-waving that this could help us solve problems in energy and medicine and so on.

Dario Amodei

Mm-hmm.

Kevin Roose

But these are hard problems. They need money. They need compute. If barely any of the compute is going to Alzheimer's research—

Dario Amodei

Mm-hmm.

Kevin Roose

—then the systems are not going to do that much for Alzheimer's research. And I'm not saying this is not your fault.

Dario Amodei

Yeah, yeah.

Kevin Roose

The absence of a public agenda for AI that does not appear to be accelerating the automation of white-collar work seems just a little bit lacking, given how big the technology is.

Dario Amodei

Yeah. The greatest example is this program called the Genesis Project, where there's real work there to think about how we can intentionally move forward different parts of science. I think giving elected officials the ability to stand up to the American people and say, "These are parts of science that are going to benefit you in healthcare, and we now know how to step on the gas with AI for them," would be really helpful.

My guess is that in a year or 2, we'll be able to answer the mail on that one, but it's just got started. We need, clearly, 10 projects like it.

9. AI Enters The Defense Race

So the other side of this is that the one area of government that I do think thinks about AI in this way is defense. I want to talk about that broadly, but specifically, Anthropic is in a current dispute with the Department of Defense—or I guess we call it now the Department of War—over whether it can continue to be used there. Can you describe what is happening there?

Dario Amodei

I can't talk about discussions with an extremely important partner that are ongoing, so I'll just have to stop it there.

Kevin Roose

Well, I will describe that there is some dispute. I recognize you're not going to talk about what's going on with you and your partner, but my question is about a broader issue here: There is going to be a lot of offensive possibility in advanced AI systems, and one of the strongest drivers of the speed at which we're going with AI is competition with China.

Dario Amodei

Mm-hmm.

Kevin Roose

Some of the biggest risks that we think about in the near term are cybersecurity, biological warfare, all kinds of ways that others could use these against us, or drone swarms. There's going to be a lot of money in this and a lot of players in it, and it really seems unclear to me how you keep this kind of competition from spinning into something very dangerous.

So, without talking about what you may or may not do with the Defense Department, how has Anthropic thought about this question more broadly?

Dario Amodei

We've been longtime partners to the national security community, and we were the first to deploy on classified networks. The reason for that was actually a project which I stewarded: to figure out if our AI systems knew how to build nuclear weapons. This is an area of bipartisan agreement where people agree that we shouldn't deploy AI systems into the world that know how to build nukes, and so we partnered with parts of the government to do that analysis.

That may illustrate what I think of as the thing to shoot for, not just for us but for all the AI companies: How do we both prevent the potential for national-security harm coming to the public or proliferating out of these systems? But the second part is, how do we just improve the defensive posture of the world?

I'll give you an example that I think is in front of us right now. We recently published a blog, and other companies have done similar work, on how we fixed a load of cybersecurity vulnerabilities in popular open-source software using our systems, and many others have done the same. So, yes, there will be all kinds of offensive uses, and there will be societal conversations to be had about that, but we can generally improve the defensive posture and resilience of pretty much every digital system on the planet today.

I think that will actually do a huge amount to make the whole international system more stable and also create a greater defensive posture for countries, which helps them feel more relaxed. Relaxed countries are less likely to do erratic, frightening things.

Kevin Roose

That would be good if it happened. My worry is, as an individual, that I feel the opposite might be happening. I've just watched people installing all kinds of fly-by-night AI software—

Yeah.

Kevin Roose

—I myself am nervous about using things like Claude Code because I'm bad at talking to Claude Code, and I don't understand these questions, and I'm worried about loading onto my computer something that is creating security vulnerabilities I don't even understand.

Casey Newton

Mm-hmm.

The number of scam voice messages I get every day—many of which seem to me to be clearly somewhat AI-generated—is very high. There's a societal question of whether we use it to upgrade our systems. I'm actually curious for your thoughts individually, because as we're all experimenting with something we don't understand and giving it access to the terminal level of our computers without any real knowledge of how to use it, it seems like we might be opening up a lot of vulnerability all at once.

Casey Newton

It's the early days of the internet all over again, when there were all kinds of banners for different websites. You could download MP3s to your computer that would completely break your computer, or download helper software for your Internet Explorer toolbar that was just a phishing device. We're there. We're there with AI.

We'll move beyond this. But I believe that when people experiment, they come up with amazing, useful things as well. So my take is, you have to say when you're doing something that might be extremely dangerous and put up big banners, but mostly you still want to empower people to be able to do that experiment.

10. AI Changes How We Think Together

So when you look forward—not 5 years, because I think that's hard to do, but 1 year—

Casey Newton

Yeah.

We've kind of pushed into agents really fast. We've pushed into code. I think a lot of people think code might be different from other things—

Casey Newton

Mm-hmm.

—because it's a more contained environment, and it's easier to see if what you're doing has worked. But from your perspective of being, you know, inside one of these companies and also running a newsletter where you obsessively track the developments of a million AI systems that I've never heard of, week on week on week, what do you see coming now? What feels to you like it's clearly on the horizon, but we're not quite prepared for it or won't feel it until it's arrived?

Maybe the way I'd put it is, sometimes I've had—and you've likely had the same—the ability to have certain insights that come from reading a vast amount of material from many different subjects, piecing it together in my head, and having the experience of a new idea and of being creative. I think we underestimate just how quickly AI is going to be able to start doing that on an almost daily basis for us: reading vast tracts of human knowledge, synthesizing things, coming up with ideas, telling us things about the world in real time that are basically unknowable today.

The amazing part is that people are going to have the ability to know things that are just wildly expensive or difficult to know today, or that would take a team of people to do. The frightening part is that I think knowledge is the most raw form of power. It's intensely destabilizing to be in an environment where suddenly everyone is like a mini-CIA in terms of their ability to gather information about the world. They'll do huge, amazing things with it, but surely there are going to be crises that come about from this.

And I think the actual mental load of being a person interacting with these systems is going to be quite strange. I already find this where I'm like, "Am I keeping up with the ability of these systems to produce insights for me? How do I structure my life so I can take advantage of it?"

I'm very curious about how you think even having that ongoing conversation with the systems changes you.

Yeah.

So let me—I'll say it from my perspective. One thing I have noticed is that Claude is very, very, very smart. It is smarter than most people who know about a thing in any given domain.

Mm-hmm.

That is my experience of it. But it is not, in the way that other people are, an independent entity that is rooted in its own concerns, intuitions, and differences. What it is instead is a computer system trying to adapt itself to what it thinks I want.

Casey Newton

Mm-hmm.

As I've talked to it much more about issues in my life, issues in my work, and various intellectual or reporting inquiries where I'm trying to figure out questions that, as of yet, I'm at an early stage of exploring, what I've noticed over time is that one difference about talking to it is that it's always a "yes, and."

Casey Newton

Yep.

It is never a "no, but." It's never a "Honestly, are we still talking about this?" It doesn't create, in the way that talking to my editor does, or talking to a friend or my partner does, the possibilities that another human does for checking yourself.

Casey Newton

Yep.

It's always pushing you further, and it's not necessarily bad. It doesn't always lead to psychosis or sycophancy or anything else. But it is very reinforcing of the "I."

Casey Newton

Yes.

I don't worry about it so much in my own case, although I already feel the pressure of it on me. It's like, "Oh, more good ideas coming from me. More interesting things I've come up with." But I do wonder about kids growing up in a world where they always have systems like this around them, and the degree to which some amount of my communication with other human beings is offloaded into communication with AI systems.

I notice that already as a kind of cage for my own intuitions, even as it allows me to run further with them than I maybe could otherwise. But I'm pretty well-formed, and you've got young kids as I do. I'm curious how you think about what it means, how it will shape our personalities, to be in these constant conversations.

This is maybe my number one worry about all of this. If you discover yourself in partnership with an AI system, you are uniquely vulnerable to all of the failures of that AI system. And not just the failures—the personality of the AI system will shape you.

I'm going to sound very Californian here, even though I'm from England. It soaked its way into my brain. You have to know yourself and have done some work on yourself, I think, to be effective at critiquing how this AI system gives you advice. And so for my kids, I'm going to encourage them to have a daily journaling practice from an extremely young age, because my bet is that in the future there will be 2 types of people.

There will be people who have co-created their personality through a back-and-forth with an AI, and some of that will just be weird. They will seem a little different from regular people, and there will maybe be problems that creep in because of that. And there will be people who have worked on understanding themselves outside the bubble of technology and then bring that context into their interactions. I think that latter type of person will do better, but ensuring that people do that is actually going to be hard.

But don't you think the way people are going to discover themselves is with the technology? I think you were one of the first people who said to me I should try keeping a journal—

Yeah.

—and I've done that on and off.

Yeah.

One thing it does is make it more interesting to keep a journal because you have something reflecting back at you and picking out themes, and so on. But the other thing it does is—I feel it as a pull toward self-obsession because I audio-record a journal entry and drop it in, and all of a sudden I have this endlessly interested other system to tell me about me, and it connects to something I said.

And I generally can't tell if it's a good thing or a bad thing, but we already know from survey data that a lot of what people are doing on these systems is adjacent to therapy.

Yes. But this, to me, will change how these systems get built. It will change the best practices that people have with these systems. And I think that we actually don't quite understand what this interaction looks like, but it's extremely important to understand it.

I mean, just to go back, in the same way that you can get Claude to ask you questions to more clearly specify what you're trying to do, and that leads to a better outcome, I think we're going to need to build ways that these systems can try to elicit from the person the actual problem they're trying to solve, rather than go down a freewheeling path together.

Because in some cases, especially for people who are going through some kind of mental crisis, that is the exact moment when a friend would say, "This is nonsense. You are not making any sense. Take a walk, and call me tomorrow," or, "Let's talk about a different subject. I don't think you're reasoning correctly about this." But AI systems will happily go along with you until they've affirmed a belief that may be wrong. And I think this is just a design problem, and it also will be a social problem that we have to contend with.

And I just wonder how much it'll be a social force. I think we've given a lot of attention, correctly so, to the places where it moves into psychosis or strange AI-human relationships. We're seeing it through its most extreme manifestations, and those will become more widespread. I'm not saying they are not worth the attention.

But for most people, it is just gonna be a kind of pressure.

Mm-hmm.

In the same way that being on Instagram, I think, makes people more vain.

Yep.

In the same way that we have become more capable of seeing ourselves in the third person. The mirror is a technology.

Mm-hmm.

I always think it's funny that, in the myth of Narcissus, he's gotta look in a pond.

Yeah.

Right? It was actually quite unusual to see yourself—

So when the mirrors came out, they were like, “Oh, this is gonna lead to some issues.”

But there's a lot of interesting research on how mirrors have changed us.

Yep.

And as somebody who believes in the sort of “medium is the message” thing, AI's a medium.

Mm-hmm.

And it will change us as we are in relationship to it, probably more so than other things, because it is this kind of relationship that has a kind of mimicry of an actual relationship.

Yes. I've used these AI systems to basically say, “Hey, I'm in conflict with someone at Anthropic. I'm really annoyed. Could you just ask me some questions about that person and how they're feeling to try and help me, I guess, better think about the world from their perspective?” And that's a case where I'm not using the technology to affirm my beliefs or show I'm in the right, but actually to help me try and sit with how this other person is experiencing this situation.

And it's been profoundly helpful for then going and having the hard conflict conversation, sometimes even saying, “Well, I talked to Claude, and me and Claude came to the understanding you might be feeling this way. Do I have that right?” And sometimes it's right, but sometimes when it's wrong, it's really helpful for that other person to have seen me go through that exercise in empathy and spend time trying to understand them before coming into the conflict.

Do you have strong views on how you wanna parent in a world where AI's becoming more ubiquitous?

Yes. I have the classic Californian technology executive view of not having that much technology around for children, but I was raised in that format as well. We had a computer in my dad's office. My dad would let me play on the computer, and at some point he'd say, “Jack, you've had enough computer today. You're getting weird.” And I'd be like, “I'm not getting weird. No, you've gotta let me.” And he was like, “See? Being weird. Get out.”

I think finding a way to budget your child's time with technology has always been the work of parents and will continue to be. I recognize, though, that it's getting more ubiquitous and hard to escape.

We have a smart TV. My toddler can watch Bluey and a couple of other shows, but we haven't let her have unfettered access to the YouTube algorithm. It freaks me out, but I see her seeing the YouTube pane on the TV, and I know at some point we're gonna have to have that conversation.

So we're going to need to build pretty heavy parental controls into this system. We serve 18 and up today, but obviously kids are smart and they're gonna try and get onto this stuff. You're gonna need to build a whole bunch of systems to prevent children from spending so much time with this.

All right. I think that's a good place to end. Always our final question: What are a few books you'd recommend to the audience?

Ursula Le Guin, The Wizard of Earthsea. It was the first book I read. It's a book where magic comes from knowing the true name of things, and it's also a meditation on hubris—in this case, of a person thinking they can push magic very far. I read it now as a technologist thinking, “Uh-oh.”

Eric Hoffer, The True Believer, which is a book on the nature of mass movements and the psychology of what causes people to have strong beliefs, which I read because I think that we AI technologists have strong beliefs and are maybe part of a strong culture that includes the word cult. So you need to understand the science and psychology behind that.

And finally, a book called There is No Anti-Memetics Division by a writer with the name QNTM, which is about concepts that are, in themselves, information hazards, where even thinking about them can be dangerous. And I always recommend it to people working on AI risk as a book adjacent to the things they worry about.

Jack Clark, thank you very much.

Thanks very much, Ezra.