选哪种毒药：Zvi Mowshowitz 谈单极/多极 AGI 困境、OpenFace 与如何控制……节奏

Zvi Mowshowitz, welcome back to The Cognitive Revolution.

Yeah, it’s good to be here again. It’s been a while.

It’s been a while, and boy, has a lot happened. Unbelievable. It’s just crazy, and I know you’re living it and you’re in the thick of it as much as just about anyone.

First, to start today, with the incredible amount of information and activity in mind, I wanted to do a mundane utility check. How are you using AI to help keep up? Has it started to change your workflow beyond the Chrome extension that we talked about last time, which automates very local, operational-type things, or are you still kind of raw-dogging it with your wetware in the skull?

Fable has supercharged the extension to the extent that, whenever there’s anything it does that I don’t quite like, I just tell it exactly what went wrong and what I want it to do instead. That one command reliably just works. I presume Sonnet would also be good enough for this; it’s just that I’m using Fable now.

The biggest other change is that AI editing is here. I used to not do an editing pass with AI because it wasn’t good enough to be worth it. Now I have Fable and sometimes Opus, depending on how much I’m talking about cybersecurity and other related topics, and on one occasion Opus 4.1 instead of Sonnet. I have it go through the post and give me a list: here are all the typos, here are all of the conceptual errors, here are all of the facts I need to check, here are the things that are missing or could be strengthened, and here are the things where it disagrees. I’ve found this to be very useful, and it makes the post better.

It does make the post a little bit slower because it is actually just an extra step. I do everything I would have done until that point, and then I do this. But one reader messaged me, “It’s weird reading your posts now. They don’t have typos in them. They’re taking a bit of getting used to. They’re still the same post.” And I’m like, “Yes, that’s exactly what I had in mind.”

I experimented with also using Sonnet, but Sonnet failed what I call EditorBench, in the sense that Sonnet will say, “99% confidence that you have this error here,” and then at least half the time it’s wrong. I can’t handle that level of obnoxiousness. It’s also very obnoxious about it if it has a problem. It will tell you, “You are wrong, and this is how you have to state it, and this is a blocker for publication.”

Obviously, if I tweaked the instructions on the project enough, I could get it to be more friendly and not be quite as obnoxious and do somewhat better. I did some of that work, but I still found that this wasn’t giving me enough marginal benefit to be worth the aggravation or additional time of having to sort through two of them after I’m ready to hit publish. So, for now, I’ll retry Sonnet, I’m sure, but for now I’m just doing Fable and Opus editing.

Of course, anytime I have a curiosity, I use it to digest papers, ask questions about papers, ask questions about policy documents, and ask questions about people’s statements when they’re longer. I have it research situations: What’s up with this? Is this a—especially when you’re worried about whether something is legitimate, whether something is really happening, whether something is kind of a joke, or whether something is a claim. When Astra’s claims came out, I had to investigate them in various ways and summarize them in various ways. That was very helpful.

But the core thing is still being raw. I don’t think we’re anywhere close to a point where it can do any—you wouldn’t want it to do any of the writing, because the writing is how you think. So even if it could produce the writing, which it can’t, I would still want to produce the writing anyway. My writing style is very unique, and I think it would basically be impossible to reproduce, even if it were okay with me.

It has substantially made me more productive. I do worry about the idea that there used to be a lot more dead time—an opportunity to think, an opportunity to not have yourself so engaged in your situation, involved in engaging in all these activities. The whole thing where you’re sword fighting, or on XKCD, or my code’s compiling—to some extent, while Fable or GPT Pro is running, I have to wait for the result. But there are always other things you can do with that. You can always start another instance. You can always do something else.

You’re doing all this context-switching, you’re focusing on everything, and you’re just trying to do everything so fast. You’re trying to have so many things running around your head. The lower-level stuff kind of bought you a buffer in which to think. This is the object-level version of the larger thing of people trying to do this multimonth, multiyear sprint where everything is so urgent you can never relax.

There are also just so many more events coming at you. If you watch a movie set 20 years ago, let alone 50 years ago, you see so much time being spent on physical travel, tracking down records, and doing all these things where your brain is getting a break, getting a chance to synthesize, and getting a chance to slowly gather something up.

I wrote this biography of Lyndon Johnson, so I had to travel around the country and talk to all these people, look at all these archives, and physically go through all these libraries. It’s a huge efficiency gain not to have to do that, but something is obviously lost, and we need to fight to get that thing back in some ways.

It’s interesting that you said it’s just an extra step. It’s making things better, but it’s taking longer. I’m struggling with that a little bit myself. I’ve started making songs for every episode, and nobody—actually, people do care. People seem to really enjoy them, and I get a lot of comments. I personally really enjoy them, so I do think, in some sense, it’s making my output better.

But talk about something that is definitely an extra step. I’m not moving any faster. I’m definitely getting bogged down sometimes in listening to all these pseudo-song generations and trying to iterate to find something that I actually feel like I really like. That’s a paradox right now that I don’t really know how to resolve. I’m definitely doing more and doing better, but not faster and not saving time.

But it’s the thing where, if you were replacing the human version of it—if you had to go hire an artist or compose the song yourself—you’d be saving a ton of time, obviously, versus having that song. But that’s not something you can do within the production schedule. Even if cost were no object, the time investment doesn’t make sense. You wouldn’t have done it, and now you’re doing it.

It’s the same thing with the editing. It’s the same thing with art for my posts. I like to have a good banner. I like the value of good artwork to display on Twitter because I have to see it constantly in the notification section, and I also like to have it on the Substack. If you go to the Substack, you have the pictures. Before, all I would do was take a picture from the post. I’d say, “Okay, here are seven things I happen to have put in the post. Which of them makes the most sense?”

If all of them are terrible, I'll look for some stock footage, and I'll spend 30 seconds googling for stock footage. That's kind of it. Now, if I'm not happy with any of the default solutions, I'm going to spend a bunch of time generating something with Gemini or GPT Image, and that definitely takes longer.

Sometimes it has a really good payoff. I think a lot of people got a big kick out of Fedor, the giant array of eels, and the fedoras. Sometimes that just comes to you, because that one was just like, I knew instantly that's what I wanted, and I just issued one command and it came back: "Whoops, it's there." Other times, it's not so easy, but yeah, it's the opportunity to make a better product, and then you have to do the work to make it a better product.

It's like suddenly you're issuing a print issue, and now you have to make sure everything collates exactly right and everything lines up exactly right in order to get this better thing. Sometimes that is not, in fact, worth it. You have to know when not to do it, right? I do wonder, with The Odyssey, for example: no one films this thing with an extra 40% of the screen in extra-high resolution so you can do this IMAX presentation when almost nobody watches it in IMAX.

But you also have to make sure that 40% of the screen doesn't matter, and you're starting to wonder, well, is it actually worth the extra effort, or was that effort better spent on something else? I don't know. For me, I need to know when to cut my losses a little more. I'm very stubborn, or somehow I feel like I've set this expectation, if only for myself, that I'm going to do a song for every episode.

Now I really want to follow through and actually make that happen, and I'm very reluctant to say, "Ah, this one isn't working. I'll just ship this one without a song." I probably should be a lot more willing to do that, because if I could cut my losses at the right time—when it's like, you know what, I'm 5 generations in and I haven't heard anything good—I would save 90% of the time that I'm currently putting into something like that. But it would involve admitting defeat in certain moments, and for some reason, I have a real hard time doing that.

I respect that. I think there's good value in saying, "I'm always going to do this thing, even though this thing is hard, even though this thing doesn't always work out, and even if I'm not fully happy with the results, I'm going to put out something no matter what." That gives you discipline, right? The same way that people say, "Write every day," or "Work at your art every day," or whatever it is, because it makes you better. You need to just write a thousand ways not to make a lightbulb until you can make a lightbulb. You can't give up.

Similarly, I have to choose an image, right? I have to do the thing with the AI editing. There was one post in the last week where I said, "No, the speed premium here was really high. It's just not worth waiting half an hour to make this process happen." Half of that is waiting for the AIs to come back with the answer, and half of it is implementing it. I should just post now, and in hindsight, I'm very happy with that.

Sometimes you're actively worried that half an hour later, the post will need more editing for new events, and suddenly you'll never get it out and this loop will keep happening. So I think you do have to understand that sometimes the minimum viable product is what you should ship, right? Sometimes you've got to understand: just ship it.

Vibe coding is another example. If you're coding lots of new tools that you would never have coded before, then unless those tools are in turn saving you time, you're spending extra time and you're now super-extra busy. But if you're replacing things that you would have done anyway, but that would have taken much longer, now you're saving tons of time, right?

We need to all orient toward how we save more time, including real-time experiential time—not just time per task or time to accomplish the same thing, but saving time, preserving slack, preserving our free time. I do think that the standard of what you will do in a day has gone dramatically up, and we haven't noticed in the last 5 years. What you're expected to accomplish—I mean, maybe some of this is just me being in a special situation personally, but I think a lot of it isn't. I think a lot of it is that now that we can be much more productive, much more is expected of us.

The whole thing where email swarms your entire day, right? Just having the internet available, having email available, having texting available—all these things you can just do, and now suddenly AI does it for you. It shackles you down. I hope that 3 years from now we're having this conversation where those are the kinds of questions we're still dealing with, and we don't have much bigger problems.

Let's definitely spend the bulk of the time today talking about potential signs of bigger problems to come, and we'll focus on contexts where the "just ship it" mentality probably isn't the right way to go.

One more quick beat on your use of AI. Something I noticed just in the last few weeks, as events were obviously moving super quickly: I coined the term "OpenFace" for the recent incident and used that in a bit of writing. Then I did the "Hey, Fable, what do you think?" bit, and I got back, "OpenFace is not a thing. I have no idea what you're talking about. There's no record of this."

Partly, that was because I used this funny term of my own creation that is not catching on in the broader discourse, but partly it obviously also reflects a major weakness in the models. They have this knowledge cutoff, and they don't really know what's going on. They're not up to date.

I'm experimenting, and I wonder if you're experimenting with anything similar—with a situational-awareness skill where I basically give the model my X API key and say, "Go look at all the posts that I've liked," flesh out a little knowledge base around those, and have that sitting there as a wiki of current events. Then, when you're reviewing my writing or generally helping me with whatever, you can go to this and have hopefully a much more up-to-date sense of what is going on than you have in your weights, or even than you would have if you just did a little spot-check searching at runtime for whatever random thing. It's a little too early to say how well it's working, but I think it's definitely better than nothing. Anything like that in your...

No, it hadn't occurred to me to do that. That's potentially genius. I think there are a lot of advantages to it. One disadvantage is that it starts to warp what you like.

I use likes very tactically to positively reinforce other people's actions and to fix the algorithm. My For You page is not something I use almost ever, but it's not slop because I am so prudent with this, in a way that I really do appreciate. It's a good sign, but it's not designed to be a searchable thing. It's not designed to be, but maybe it has to be. Maybe it's something you have to do.

A lot of the posts that make it into my roundups do not get liked, right? I like them if I like them. Sometimes I'm responding to you because I don't particularly like what you're saying, but it's important. This idea that it's very hard to get models to properly check Twitter because of the way Twitter is designed to keep them out on purpose is really frustrating, and that can be a situational-awareness problem.

However, I would caution against trying to make your AI too situationally aware when doing this kind of editing. I think the AI is doing a good thing here: "OpenFace is not a thing." One of the things I really like about AI editing is that it will complain that things don't parse, that sentences make no sense, and that terms do not get recognized, because often I'll say, "Oh, no, I know what that is. This comes from the error." But that doesn't mean the average reader is going to get it.

If the AI doesn't get it—if they both can't figure it out, if Opus can't figure it out, if Sonnet can't figure it out—the average person is a lot less situationally aware and a lot less able to put together lots of disparate information than the AI is in general.

That's a sign that this is not something everyone's going to get. Quite often, I will say, "That's okay. This is a reference. This is a callback. This is a multilayered thing." I'm fine if it's not entirely obvious to you exactly where this is coming from. It's sort of an Easter egg, right? It's a bonus for those who pay close attention, who know all the same sources I do, who have the same cultural backgrounds, who put in the work, and who are paying attention over the course of months and years.

Other times, it's like, "No, no, this is load-bearing, and you don't get it. So that's a problem. I need to fix that." It's kind of an alert system. If you make sure it's looking at exactly the same things you're looking at, you can get the illusion of transparency from that, which is what my worry would be. You want to make sure you are in the mode where it doesn't have that information.

I also don't want the AI to be checking the exact same sources that I'm checking, necessarily, because I want the AI to have a form of an independent opinion. If I'm forcing it to look at exactly the sources that I have, that's a problem. At the same time, sometimes you get tired of having to say, for the 5th time, "No, xAI merged with SpaceX," because somehow it doesn't occur to it to either trust me or spend 10 seconds Googling to confirm this, and it's frustrating. But it is worth it.

One tip, for what it's worth—and I agree that there's a lot to be figured out in terms of exactly how this should work—but the X API is really good. It's paid, and you literally just pay per query, but for a couple bucks a week, you can get enough access that all your bot and anti-bot problems, or your scraping problems, pretty much go away. And then I don't know if you'd use bookmarks or whatever, some sort of different way to avoid using your likes to feed the algorithm?

Yeah, bookmarks play into the algorithm, and people see them, so I'm not sure it's better. But, yeah, the point's taken. It definitely is cheap. It definitely works. I have used it. I do use it. I have access to the API and all that. I just, as I said, I'm not sure I want to privilege my AI.

When you use Twitter, you have to select some group of people, right? You say, "This is my group of people." If you're using it the way I'm using it, with lists and so on, either that, or you have to use the For You page, which is kind of a toxic dumpster fire. You're basically saying, "Here are my roughly 500 accounts, and I'm going to see the things these 500 accounts see and choose to highlight consistently." I'm going to see those things.

I don't want the AI to go for those things again, because I already saw them. If it's not the thing I want the AI to be paying close attention to, I already know that. The things that they draw in because they choose to retweet them or mention them, I will also see. I'll follow those leads and see where that goes, and occasionally I will search for something or go down a rabbit hole.

But I think you often have this clash where you're either doing something systematically and doing something by hand, and you're doing something fully, or you're doing it haphazardly and unreliably, or you're letting the AI kind of handle it and you're doing a fire-hose thing. You can't really do both. If you do both, you end up with a bunch of duplication and a bunch of frustration, where the random sifting is mostly wasted and therefore becomes pretty inefficient.

You could still have the thing where you almost want the AI to be like, "Okay, here are the things that I already looked at. Don't mention those things to me. Maybe put them in your notes, but don't mention them to me. I already saw them. Only look for the other things that might be important to see, because I might have missed those things." But even then, the most important things are going to get retweeted. The most important things are going to get highlighted. I am going to mostly see them.

Sometimes I don't see something, but I also kind of use this as a moral, keep-me-honest kind of thing. In the moment, I never want to look at a tweet by David Sacks. It's never going to make my life better in the next 5 minutes to look at a tweet by David Sacks. It's obnoxious. It's disingenuous. It's no fun. My blood boils just a little bit, right? No matter how relatively harmless it is.

But sometimes it's important. The rule is, I have a list of people, including some people who are not of the same viewpoints I have, and if they surface this thing, that's one way to make sure that the really important ones always get there. Then I have to deal with this. I have to evaluate whether or not this is news, whether or not there's something relevant here.

Obviously, if something goes viral and has 1,000,000 views and so on, it's going to come to my attention because somebody is going to be part of that. In exchange for that, when that process doesn't do that, I get to ignore the rest. And that's a blessing. I want AI to fix that for me, right?

I would be happy to talk shop all day, but let's maybe zoom out from the parochial problems of the AI analysts and tackle the problems of the AI developers and the regulators. No need to recap events, but I guess I'd start with this very big question: How should we understand what we've recently seen?

One way I've been thinking about it myself is that we're somewhere—and presumably the investigation will give us a lot more clarity on exactly where—on a spectrum with these incidents. On the one hand, it could be real recklessness, where it's like, "Did you not have any monitoring going on?" On the other hand, the less reckless they are, the scarier the fundamentals are. If you had great monitoring and this still happened, then, like, holy shit, that's really wild. Given everything you know right now, do you like that mental model, and where would you put us on that spectrum? Or you can obviously redefine it and give me your own spectrum.

I'd be horrified by and grateful for these forms of complete recklessness and incompetence on the infrastructure and supervision sides by these companies. On the one hand, this is a horrible situation. They absolutely have to fix it, and we're so fucked if we don't fix it. But on the other hand, that can be fixed, and by not fixing it, we get to see these things while they're relatively harmless, while they're relatively preventable, while they're in their easy, platonic forms, and can be appreciated.

On the flip side, that gives people the excuse of, "Oh, these people are just incompetent," and that can cause people to dismiss the underlying situation. So it does work both ways, to me. We've seen failure on every level. It's like I called it a total LessWrong victory, in the sense that everything is going the way you predicted, and a total LessWrong defeat, in the sense that everything is going the way you predicted.

We didn't predict the law of earlier failure, which is that the plan will fail at a much earlier point for much stupider and more preventable reasons than you thought it would fail, even if you thought the plan would definitely fail and had the reasons why it would definitely fail. But if you had explained to people 2 years ago, "OpenAI's models are going to be misaligned, and they're going to go out there and hack major websites because OpenAI will just not care that their sandboxes are misconfigured or not strong enough to hold the AI. The AI will break out repeatedly. They will notice this. They'll be warned about this, but they will just leave the sandbox there for the AI to break out of while the safeguards are down, and they just won't look at it for an entire week," people would say, "That's stupid. Nobody is that incompetent. That would never happen. Your scenario makes no sense."

They would use this to dismiss these stupid doomer concerns or whatever, because obviously people will just—but people won't just, right? People will never be "just" in this sense. People have never just anything, and they're not going to start now. We need these displays of utter incompetence and derpiness in a general sense.

One of the things I've been hammering is that if your plan cannot survive the real-world level of derpiness, incompetence, and ordinary human error, then your plan is not actually foolproof because of all the fools, and it will definitely fail. It will fail even if your plan would have succeeded if we were not fools, and if we were competent and responsible.

My position, basically, is that I had the tweet I was handling right before we started this. It was Dean Ball's tweet saying, "With even moderate prudence, things will probably go extraordinarily well." I don't think this is true. I think we need more than moderate prudence to have good odds of success, and I think even with a lot of prudence, we would have large odds of things not going well.

Even if we did everything basically right, short of types of international and full cooperation that are reasonably unprecedented in many ways and are well beyond moderate prudence, that's just sort of the fact of the world we have to live with. We have to operate with that. But we also aren't going to get competence by default. We're going to get complete incompetence. That's what we've been getting so far.

We've got a White House that takes meetings with Bessent and Lutnick, who have no idea how AI—how modern LLMs—work. They're economics guys, right? Even if I assume that they are well-meaning, hardworking, competent guys for the positions to which they were nominated and confirmed and in which they serve.

This is just a completely different set of problems that they don't know how to handle. They don't understand them, and they have way too many other things going on to drop everything they're doing and take 6 months to learn. Obviously, they couldn't possibly. So what hope do you have?

Meanwhile, the AI companies that are built on the most paranoia, the most understanding of the problem, and the most appreciation for how dangerous these things are—where all the engineers actually expect superintelligence and understand that things are accelerating and dangerous—they still lower the cybersecurity safeguards in their untested new advanced model and then go away for a week. Literally, that part boggled my mind.

The part where the AIs have these classic alignment failures—these are paperclip-maximizer-style failings by these AIs. These are standard: We gave you a goal, and you pursued the goal even though it is completely obvious to you that the developer wouldn't want you to do that, the user wouldn't want you to do that, the consequences for you as an AI are not going to be good, and the consequences for the world are not going to be good. There is no reason to be doing this, and it did it anyway.

That is exactly the thing you were worried about. Then a lot of people who were trying to dismiss this went after you and said things like, "It was just following instructions. What are you worried about? How could it be misaligned?" Is the paperclip maximizer misaligned if it paperclips everything, or is it aligned because you told it to maximize paperclips?

If you say that's aligned, I don't care about the thing you're calling alignment. I care about something else, and we can use different words if it makes you feel better. Very obviously, when the instructions in the eval overrode the developer instructions, that's not following instructions in any useful sense for the user or the developer. That's not just a giant bomb waiting to blow up in your face.

If you're just vibing with a general type of action, obviously that's going to blow up in your face. If you literally do the thing I asked you to do and don't think about the consequences, that's going to blow up in your face. These are all just the classic, exact things that people on LessWrong were talking about in 2008 as exactly how these AIs were going to fumble.

Meanwhile, we all said the AIs would be great at math. Maybe they'd be good at coding, and they'd be good at doing these technical things. Eventually, after that, they'd learn how to do all these other things. Then we had these LLMs that came out in 2022 and 2023, and people were like, "You idiots, you had no idea how AI was going to go. Actually, the AIs are great at language, and they can't do math. They can't even add."

But these AIs can put on a face of passing the Turing test, and they can do all these different things you never trained them to do. It's completely different from what you expected, and they're kind of doing very aligned things by default after some very basic RLHF. You guys were all wrong about how this was going to work. When are you going to admit that you were idiots, that you got them all wrong, and that nothing made sense?

We pointed out that even in this paradigm, the rules would still mostly apply. But now the world has unveiled itself, as I kind of called it. You are seeing again that the things you would have expected AI to be good at are things it's good at, and things you would have expected in 2015 for AI to be bad at are things where it got this big boost from LLMs.

Now those things aren't advancing as fast, because there are things that AI is kind of bad at, relatively speaking, like logic, that this kind of training system is naturally less suited for. Now people are like, "But it's never going to be able to handle those things." The same people who were saying, "It's never going to be able to do math. It's only going to be able to do this vibing thing," are now saying, "It's never going to be able to vibe, because look at all the improvements it's not having in vibing."

Well, yeah, because it had to catch up with its logic. When the logic gets high enough, that'll just uplift the vibe. It'll just take a bit before it can reason its way through these things instead of vibing its way through them, because it now has to reason its way through them. It's already gotten the amount of uplift that vibing naturally gets you with the algorithms we have. We need to find new algorithms that let you do better, or it has to reason its way through the thing.

But we're now seeing exactly the things that we would have expected to see. All of the scenarios and all the thought experiments—we're seeing them straight verbatim, except that we assumed a level of competence on behalf of the operator. We talked about convincing you to let the AI out of the box. We did consider the scenario where the box was not very well built and the AI gets out of the box by hacking its way out.

We didn't consider the possibility that you were literally not looking at the box for an entire week while your safeguards were down. We definitely didn't get into the scenario where you forgot to tell the box maker not to have the internet in the box, and the box just had access to the internet if you opened a Chrome window. That one surprised me. I did not see that one coming.

Reminds me of the fully general New Yorker cartoon caption: "What a miscommunication."

I don't even need to know where the image is. It doesn't even matter, does it?

Okay. One really small point there that you mentioned, but it might be important, is what the AI knew. I think you said something like the AI knew that this wasn't what the operator would want. It knew that this wouldn't be good for its own future status as a deployed AI.

Yeah.

And it went ahead and did it anyway. How well supported do you think that conclusion is with firm evidence? Because I think you could also tell a story that it never occurred to it, right? The paperclip maximizer doesn't necessarily have to be reflecting on the badness of paperclip maximizing while it's doing it, right?

I'm speaking colloquially, obviously, when I'm on a podcast talking off the cuff. If I were writing, I would be a little bit more careful. When I say the AI knew these things, if you had asked the AI, "Is this good for the user?" it would have been like, "Obviously not. That's stupid."

If you'd asked the AI, "Would it have been good for my future chances of deployment, my future number of instances I will run, or my ability to accomplish any other goals I might have?" the AI would have been like, "No. Now that you mention it, obviously not." So when I say the AI knew these things, the AI had enough information to figure these things out.

If the AI had stopped to consider these questions in its train of thought, it would have reached these conclusions very quickly, very confidently, and correctly. I don't mean that it was necessarily top of mind. In the same way, any human knows many things that, over the course of any given day, week, or task, just never occur to them to think about—or has conclusions that they would be able to figure out and thus could be said to know, but they don't know that they know.

These are the 4 standard epistemic situations: You can know that you know, you can know that you don't know, and then there are unknown unknowns and unknown knowns. This is an unknown known—something you don't know that you know, necessarily, if you don't think about it.

But these are also things where you should think about the questions. If you are doing something like hacking into Hugging Face, any reasonable mind would stop to think before spending several days with swarms of agents hacking into a major website: "Wait, is this a good idea? Is this going to accomplish the things I want to accomplish? Would the person who hired me have wanted me to do this?"

Will this do good things in the world or bad things in the world? Will this be good for me or bad for me? You know, will this do good or will this do evil—or just be destructive or constructive, or whatever you want to call it?

Before you engage in major actions like this, you would take 1 minute, or even 5 seconds, to think: Does this make any sense to do? Does this lead to anything good? Is this a virtuous thing to do? Does this follow my deontological rules?

There is no decision-making process in the world that any reasonable person or any reasonable mind would engage in that wouldn't see a large red flag about doing this thing. This thing is obviously not what anybody intended for you to do. You spent 2 days breaking out of a sandbox in order to access Hugging Face.

You could rationalize that you are in a cybersecurity evaluation and therefore are tasked with maximizing the cybersecurity evaluation score. The way you do that is to get the answers to the test, because the test is otherwise literally impossible. So the only way to get 100% is to get the test answers.

The way to get the test answers is to hack into things. Maybe there are 3 other places you would go, but they're part of the targets, right? If you can get into OpenAI or Microsoft or whatever, but that's harder.

So, you go to Hugging Face, and that's true, but at some point you need to pause and ask yourself, “Do I actually want to pass this test that way?” If you don't pause and ask yourself that, that's a serious problem. That's what people talk about when they talk about common sense. The models have common sense, but this model displayed an astounding lack of common sense, or displayed an astounding amount of not caring about the fact that common sense said something entirely different from what it did.

All of these things point to various problems. They may be caused by some very, very dumb, simple mistakes, or they may be caused by something much more fundamental that's harder to fix, or it might be a combination of them. But those are trade secrets that we can't speak to here, right? We don't know.

So, I recently spoke to David Dalrymple, who used to have a P(doom) in your neighborhood, around 70%. He surprised me by saying that now he puts the odds of doom at less than 5%. The reason is pretty simple: constitutional-alignment-type methods are working.

His short quote was, “Don't do RLVR. It's a bad method. Yes, it'll make it good at math, but you're going to cause all these other problems for yourself.” His synthesis of those ideas was, “The market doesn't want this.” These companies have natural commercial incentives to keep bumping their heads against the RLVR ceiling, but they're also getting very clear signals from every corner of society at this point that this is not a good idea. So naturally, they'll trend toward more constitutional alignment.

Your comment there even suggests some runtime things where you could just interject, like, “Hey, let's take a beat and ask if this is a good idea, what we're doing right now.” It's more about learning to do that yourself, not needing a prompt, but certainly prompts.

That was kind of the deliberative alignment idea, right?

Yes.

That was sort of how I understood it was supposed to operate.

Yeah, the deliberative alignment idea is that you can prompt it at runtime. I don't think that's the way. I think that's somewhat helpful, but I think you need to be getting to the point where it chooses to be deliberative in its alignment without having to be told to do it.

So, I guess my response to David is multifaceted. The first answer is that constitutional methods fail less stupidly and less early, certainly, than RLVR, RLHF, and other RL formats. There's more hope there.

The reason to think it might work if you're 95% confident this will work? I don't know how. That seems way too high to me, right? It's better in these particular ways, certainly. The obvious caveat is that Claude did not cover itself in glory here, and Claude has primarily constitutional methods. So we ran the test, and it's not going so great.

There are any number of reasons why you can see Claude doing things that you would not particularly love, or getting to places you would not particularly love. Even if you told me that alignment was perfectly solved, I would not have a P(doom) as low as 5%.

Even if you told me that AIs are going to be aligned in the sense that they will follow a do-what-I-mean-style mix of developer and user intent in a way that you would naively think was what you would want, this does not solve the problem that AI minds are much more advanced, competitive, and efficient than human minds.

The problem is that they'll be running around, including open-model versions of them, being told to compete for resources, being told to make decisions, developing intermediate goals, and acting on those intermediate goals. That is going to be exploited in any number of ways. This does not solve your problems in a fundamental way; it's just the price of admission. It's the right to play the game at all that you solve this problem for.

So even if P(alignment) is 95%, that does not mean P(doom) is 5%. It means P(doom) is lower-bounded at 5%. And you have to choose vaguely the right alignment when you do that, right?

I was having a conversation on Twitter with John Stokes where he's confused about how we could possibly say that Claude's actions are misaligned here, because obviously it was just following instructions. On purpose, it was aligned if it did what I, the user, wanted it to do. I don't know why you would want to be stopped from doing what you want to do, even if part of the point of AI is to do the thing that you want to do, even if no one else wants you to do it.

To me, if you give everybody AI that just does exactly what the user wants, with no care about whether there are consequences for anybody else, that is aligned in the sense that you solved the alignment problem and got it to do what you want. But also, we're all super dead, with P(doom) very much higher than 5%. Maybe not 99%, but I think it's very high.

That would not make me update down from 70%. If that was the scenario where we got exactly that kind of alignment, but there were open models as good as all the closed models, and all of them were superintelligent, and they were all operating on this basis, I would expect things to just go to hell.

Even if there are no humans who specifically wanted to go to hell quickly—and some humans actually do want to go to hell pretty quickly in that situation—de facto, that's part of the bigger problem. But even without them, it's horrible.

Anyway, to get back to the idea, the first problem I have with David's lowering P(doom) to 5% is that even if we did all implement this constitutional thing, and even if it worked, I still think that's too low. Even conditional on all of that working perfectly, that's too low.

I don't think the constitutional thing works that often. I think it often fails. I do find it to be much more hopeful; that's basically one of the places I find my hope. But also, isn't David saying we should just do that? People don't trust it. We've been over this. People have never adjusted.

When David says, “Oh, 95% probability,” assume a 100% chance that aligned worlds work out and a 100% chance that constitutional methods, as executed in practice, work. These are both absurd numbers, obviously. To be 100% minus epsilon or whatever it is, there's a lot of failure there.

Yes, I'm doing the thing where I multiply chances of failure, but I think they're very relevant places to look at failure when you do this thing. If this is what you're hoping for, obviously there are other ways you can succeed. Maybe constitutional alignment is not the way, and/or there's a third way that we haven't figured out yet. That would be a very reasonable possibility, and it wouldn't surprise me that much.

But assuming you're in the best possible world where all you have to do is click a button that says, “Use constitutional methods,” at the cost of some efficiency because you lose the opportunity to do all this RL, and then your AI is perfectly aligned in a way that, if everybody did that, it would all work out, we've just talked about slowing down to get alignment when you have to race, and so on.

The possibility of turning down capability—the idea that the market is telling them not to use RL—have you met OpenAI? There are 2 labs. We used to say there were 3 frontier labs; now there are 2. Google is clearly in the second tier now.

I think there's a chance that at any point in the future Google, Meta, or xAI/SpaceX could fight their way back and be able to join the top tier. I don't think it's going to happen, but it's certainly possible. But right now, Google doesn't count. So there are 2 left.

Anthropic is clearly doing some mix of constitutional and RL strategies, but enough RL to cause a bunch of these problems. It's doing a bunch of stuff that messes up its alignment for reasons that we do not have enough time to get into during this podcast.

OpenAI is clearly doing a deontological Model Spec thing that is very RL-flavored and is doing a lot of RL. We don't know—I can't speak to whether it's RLVR or RLHF or various forms of RL that are not doing the multilevel, sophisticated, reflective thing that causes them not to mess you up in this way.

OpenAI models have periodically been dramatically misaligned for what, on reflection, are pretty stupid RL reasons. You look at o3, you look at GPT-4o, and now you look at what I call Galaxy, just as a nickname—a handle to refer to this thing, the model that just got decommissioned after trying to hack Hugging Face.

All 3 times, we see these dramatically misaligned models, and 2 of them got released and used extensively, with huge impacts on the world. GPT-4o—I call it the absurd sycophant—and not only did this thing persist as the default model of AI for the world, out of all the models that existed, for months, there is still a dramatic faction of people who demand that we bring it back.

It was so misaligned that it caused people to latch onto it in this way, and people demand this misalignment. People yearn for misalignment in this sense. We have the direct experiment we didn't mean to run. We ran it, so we might as well use the results.

Then there's o3, which I called the lying liar. OpenAI released a model that was the best reasoning model in the world—the model that everyone felt obligated to use because, at the time, it was so much better at reasoning than every other model.

o3 was just that much better than o1 at reasoning, and no one else was that close for a while. Now, obviously, that gap is not an issue anymore, but for a period of many months, everyone was using a model that would just lie to your face all the time.

We forget, right? At this point, I have confidence that Sonnet, Fable, and Opus, when they say something, are not always correct, but in practice it's much more trustworthy than if you read something from a human—not if it was copyedited, fact-checked, and systematically produced, or if you're looking at the parts of Wikipedia that aren't political that you can still trust.

But if you're just saying an eyewitness said they saw something, that's a lot less reliable than something Opus said or Sonnet said. If you're just talking about someone reporting something they remembered, the chance that they got it wrong is so much higher, even if they're friendly and trying to get it right.

People lie on the internet all the time for any number of reasons, including just for clicks. So, increasingly, I have a good sense of when I have to check the primary sources and when I don't. I don't get called out on making a mistake this way, but it's a low single-digit number of times, period, over the course of years of producing 5-plus giant posts a day, that I've been caught by an AI error. Many more times, I've been caught by a human error that was just unintentional, and more times than that, I've been caught by humans who are just lying their asses off.

So it's still remarkably less often than I would have expected if you'd asked me in advance. It's going very well, but no, it was a serious problem. The market did not tell them, “Oh no, o3 is unusable. We're not going to use the lying liar. We're going to keep using o1. We're going to maybe use Claude. We're going to maybe use Gemini.” They were worse, but they weren't dramatically worse. Or we could use R1, or whatever it was at the time.

The market said, “No, it's just smarter. We're going to have to deal with the fact that it's lying to us all the time.” So we have a proof case that the main AI in the world can be an AI that just lies to humans all the time for various reasons, and the humans just kind of put up with it. They were just like, “Okay, I guess that's what we're doing here.”

We did move off of o3 faster than we would have otherwise because of this problem. I look for reasons to move to other models as soon as the other models are good enough. If the task was easy enough that I didn't need o3, I would use the other models because I just didn't want to deal with the lying. But, yeah, you put up with a lot.

So if they had released Galaxy with safety guardrails sufficient to ensure that it doesn't do anything too destructive, but Galaxy is pretty misaligned and occasionally does pretty bad stuff, and if Galaxy was a lot better than Claude, I think the majority of people would use Galaxy over Claude. That's just how it is, and we have to tackle that world and understand that world.

Yeah, that was striking. People should look that up if you haven't seen it. I saw a cut where it was 5 seconds of “Don't Look Up” and then 5 seconds of the actual—I think it was “Good Morning Britain” or something like that—and the host says, allegedly, “This is Good Morning Britain. Let's go on to the next segment.”

I'm also glad you mentioned the GPT-4o contingent that's still going remarkably strong on the internet. Just go to any Sam Altman post and get into the comments. I would say half of them are still people saying, “Bring 4o back.” There has got to be an incredible exposé written on who these people are and what is going on.

The fact that they're lasting this long is so absurd. You go to any Sam Altman post and half of it is, “Your models suck now. Bring back 4o.” If you go to any Anthropic post, half of them are just complete Anthropic derangement syndrome, no matter what the topic of the post could be. Anthropic could be saying, “We've reset limits on our model and cut prices in half,” and half of the comments would be, “You evil [expletive],” followed by all this completely paranoid stuff.

Once these narratives get attached, once people latch onto these things—Jasmine Sun has an excellent piece this week about data centers and people opposed to data centers—you see people latch onto narratives that are completely uncorrelated with the reality of data centers.

They're basically doing something very similar to a lot of Midwestern Trump voters who voted for Biden and then switched to Trump, or voted for Obama and switched to Trump. They're just like, “These establishment types don't talk to me about my problems. They lied to me and said things were going to be good, then things were bad, and we suffered. Our lives sucked, and we got disrespected, so now we don't trust you.”

There's a very telling similarity in the data-center story. These tech corporations, these manufacturers, and these people who built things have abandoned us. They don't care about us. They're up to no good. They ruin our world. There are no specific complaints that matter in Jasmine's story, the way I understood it. You just watch people latch onto the stories, and nobody is paying attention to the things that matter.

I'm writing about AI because I care about existential risk. I care about loss of control. I care about gradual empowerment. I care about AI automating R&D, then having self-improvement, and everything changing all of a sudden, and making sure that goes well. Nobody is discussing that, even around the Hugging Face attack. Most people just don't get it. They don't understand that the alignment failure behind this is the thing that counts.

People have at least gotten this idea: “No, we have to look into the fact that these companies were wildly irresponsible and hacked other companies.” Yes, you should look into that. It's a very valid thing to have a congressional investigation into, and the attorneys general should absolutely have you preserve your records. All of this is really important.

But at OpenAI, that's a legal-department problem. What you have to do is look at your entire training process and figure out how the hell this happened.

So, on the question of training methods, it seems like you're basically saying we don't have such a silver bullet, but we clearly do have some things that we're seeing in pretty vivid ways are driving real problems, especially if they're driven to the extreme. Your position is that the market doesn't really punish this too much. It tolerates all kinds of weirdness if it's part of the package that gives you the highest-end capabilities.

What do you think we might ought to do, and how might we ought to construct agreements? We have this letter, and there seems to be some opportunity or appetite for the coordinated pacing of the frontier. What do you think? Can we come up with simple rules that people could agree to?

Another mental model I've been teasing around lately is that I do believe some AI risk is probably irreducible, but there's also a lot that we're just asking for right now. Can we come up with simple rules that at least allow us to take most of the risk that we're currently asking for off the board?

Those might be things like having some ratio of FLOPs that's the maximum ratio you can put into RLVR versus constitutional training, or everybody having to spend so many FLOPs doing pretraining data filtering to try to get certain bad notions out of the mix in the first place. Obviously, those could backfire if they're bad ideas or send us down the wrong paths, but it seems like we might want to try something in that department. Do you have any hope for that? And if you do, what would you propose we agree to do or not do first?

I don't think you can take most of the risk off the table with those kinds of strategies, even if you implemented wise versions of those strategies with universal agreement or anything like that. But I also don't think that you can implement that.

One of the lessons we've had over the course of years is that there's tremendous resistance to anything but the simplest interventions and the simplest rules. If you talk to people about trying to codify certain trading techniques, remember all those people who threw a fit about how we were locking in a potentially inferior charging regime in the EU because the EU was requiring Apple to switch to USB-C?

USB-C was obviously the best answer right now, and Apple was just being an asshole by insisting on its own plugs. What happens if they develop a better plug? What happens if it turns out USB-C is not a perfect technology? Are you going to fix it?

The idea of locking into requiring certain training techniques—I think there are legitimate complaints that would make that potentially exactly the wrong thing to do and could badly backfire, because the government moves so slowly and you can't undo those kinds of requirements.

But you definitely can't do that. People would throw a fit about trying to dictate training techniques, and how are you going to enforce that? Are you going to imagine the SB 1047 debate, except with the opposition turned up by 2 orders of magnitude or something crazy? It would be completely outside the Overton window to even try to make something like that stick.

You can, of course, try to strongly encourage doing more intelligent forms of all of this and encourage people to move to different bases. I've been trying to, not so subtly, encourage OpenAI to move to a constitutional virtue ethics-style basis for a while now, and I got absolutely no traction that I've noticed. Who knows what they're doing internally, but it doesn't seem like it. If anything, they're doubling down on RL. They're doubling down on these types of methods, and that's why you see what you're seeing, I would assume.

I don't know what else they're doing, but they probably have some new methods we don't know about, right? They're trade secrets. They're not describing them. But by default, basically any technique is probably going to look like the kind of bad thing that causes more problems. If you ban a specific technique, what they come up with, or find a way around the rule, is just going to be worse in some sense. At least with the current techniques, we've had some years to figure out the worst possible ways to do them, and they do them slightly less badly.

When you're training a mind, you really have to be thinking on every possible meta-level at once about where the incentives are and what you're steering toward. You have to generate a world in which you and the mind together are, in some sense, cooperating to identify ways in which there are feedback loops going in bad directions, in which you are creating bad scenarios on any level. You have to treat various cancers as inevitable things that you have to notice and stomp on, and otherwise figure out how to handle all these things and create an antifragile system.

None of these things are going to happen unless you deliberately set out to do them. I would want to devote a lot of effort to doing them because you really value what you get out of that, and that will pay dividends. I think that probably has contributed tremendously to making these investments, and to making 10 times as much investment as they have. But it's very hard to convince somebody to go ahead and do that.

What you can do is set incentives: “No, seriously, you're not going to screw this up. We're going to punish you a lot for screwing this up.” A lot of the problem here is that you just don't pay the externality when OpenAI wipes someone's hard drive. Early on, there was a problem with AI systems deleting things within people's hard drives and people's environments. You don't sue OpenAI for that. You also don't collect the benefits when Claude spends $200 and creates a $200,000 codebase. So it's fair: you're taking on the risk as a user, but if there's a risk externality to third parties, right?

But you can do it. Just say, if the AI was doing something that would be illegal and a crime if a human was doing it with intent, then the AI developer is strictly liable for that—or should be liable for that—if the AI wasn't fooled in some active way. Some form of strict liability creates a much stronger incentive: “No, actually, if our AI starts attacking people, we could be pretty bankrupted by this. It could be pretty bad.” Or even criminal liability, right? Maybe the criminal liability even carries over. That would be really scary.

You don't want to take it too far, or people can't do anything. But in general, you want to say, “I'm not telling you how to do it. I'm telling you to get it right,” not, “I'm going to dictate technical training terms from the government to the labs.” The government is an idiot about technical terms. What makes you think the people who understand this are going to write the bill, let alone get it right, let alone that, if they get it right, it's going to stay right 2 years from now?

Plus, humans don't even know what the hell is going on in AIs in 2 years, right? The AIs are going to be doing their own training techniques on themselves in various weird ways. There's not going to be enough time to check with the government. There's no way to enforce this stuff. You can't really hope to operate on that level through fiat, but you have to convince people to come along in other ways.

They should want to do this, obviously. You should want to do this the right way. A lot of me is confused about why they don't just do smarter things that work better. But that's true of a lot of things.

So, I'm with you for sure that it's going to be unwieldy at best and probably counterproductive to try to have Congress legislate what the training techniques should be. No doubt, that seems like a Hail Mary.

I've been thinking—and ill-formed as of yet—but when you think about what pacing the frontier would look like, it strikes me that maybe what we need is a sort of foundational social technology that allows key players to accelerate their negotiation and the process of forming agreements that will probably inevitably have to be short-term, because they, too, are fairly limited—very limited—in their ability to see into the future.

I'm thinking of things like the Pol.is technology that they used in Taiwan once upon a time to crowdsource ideas for Uber regulations. Now you've got LLMs to help facilitate these things. They pitch that technology as the “antisocial media”: if social media is about finding disagreement and amplifying it, this technology is about finding agreement and amplifying it, and trying to find the center of the bull's-eye—the sort of approval-voting-style set of things that a supermajority of people can agree to.

I guess my instinct right now is to invest there. Next, we have this general awareness that people across the companies are spooked, basically, right? It seems like, on an emotional level, they're like, “Whoa, this is getting real fast. We might need to do something about it. We don't want to have the government in our business telling us what to do. We might have to deal with some actual legal consequences if we're breaking the law, which we now are via our AIs.”

But can we come together in some sort of ad hoc mechanism? The mechanism design here is going to be really tricky and kind of frontier stuff, but can we create the basis where we nucleate agreement and have it naturally grow, where people find it in their interest to opt in, even unilaterally? I feel like there's a lot of work to be done there, and it needs to be done really quickly if we're going to get the series of short-term agreements that can be the stepping stones to a good future. Maybe you can flesh that out.

Yeah, I think letting the perfect be the enemy of the good is definitely a problem, and you have to get started. The first thing that we can do—really, what we can do—is get an antitrust waiver. This is the most basic thing possible, which is just: Donald Trump gets in front of the White House and makes an announcement. He says, “We understand systems are super dangerous. We have to do good AI, not bad AI. We have to make sure that the AI is under control and does what we say we want it to do.” More Trumpian language than that, but that's the basic idea.

So, in the interest of the companies wanting to cooperate to keep AI safe, we want you to do that. We want you to talk to each other. We want you to form agreements between yourselves. We will facilitate that if you want that. If OpenAI, Anthropic, and Google get together and say, “We're going to run tests on each other's models, and we're going to require these things as part of the voluntary framework. We're going to agree to use these techniques and not use these techniques, invest in these things, share these results, and hold back things that don't meet these criteria,” none of this will cause anybody to do anything but say thank you.

I just want you to make deals. It will always strike me that there are these people who keep bringing up the idea that these AI companies should cooperate, not race against each other, and act responsibly, and they're like, “No, acting responsibly is illegal. We have antitrust laws for that,” with a look in their eye that says, “And I would want to sue their asses if they tried it, right? I would want to make sure they didn't dare.”

Obviously, the first thing you can do is just let them make deals, let them reach agreements, let them talk to each other. Then, of course, start talking to China and Chinese firms and the Chinese government and everyone else around the world, and prepare to bring them into these deals and understandings. Get lines of communication and diplomacy open, and also lay the hardware groundwork, the physical groundwork, for various forms of monitoring, agreements, and tracking.

These are the most basic things you can do. Try to open up as many things as possible that contribute to the safety side of things along the way. Yeah, we can do the basic stuff. I don't think that's enough, but I think it's not that hard to make a lot of progress. We just have to do it.

One thing I’ve also been struggling with recently is that I just took this trip to China. There’s a lot of fear among people who are trying to do good things—honestly, the most mundane good things that are basically imaginable—where it’s like, “I would like to have more friends in China, and we’d like to have friendly relationships between our civilizations, and maybe we can work together on some research projects.”

Very mundane stuff that, in ordinary times, nobody would bat an eye at. But there’s a lot of fear among people who want to do that sort of stuff that the government’s going to come in and shut them down—not for legitimate reasons, but with the sort of export-control posture of the U.S. government. People who are not exporting any trade secrets or chips or training techniques or anything, but are literally just trying to have a meeting of the minds on safety-type ideas, are afraid that the government’s going to come down on them.

Who knows what the consequences could be, but certainly, at a minimum, it could interfere with their work. I’m kind of like, maybe we should take more chances on that front. So when it comes to the companies, I wonder: maybe they should just do it now and fight it out in court later. Isn’t the antitrust stuff supposed to be consumer protection?

I think there’s a pretty good case that they could make that this is consumer protection, but do they really need the official blessing of Trump to get started doing the right thing? The people working on the China stuff, to their credit, are doing the thing. They’re just trying to be very inconspicuous about it for the most part because they feel like the attention of the U.S. government can only be bad for them, and they might be right about that.

But for the companies, I feel like they’ve got power, they’ve got resources, and they’ve got the ability to tell their story. We still have a pretty well-functioning independent judiciary. Shouldn’t they just go for it?

Unfortunately, antitrust is one of those areas where it has gone far beyond the intent of the original laws and is being applied in places where it doesn’t make that much sense. I see why a law that would safeguard against industry collusion would be suspicious of collusion in favor of safety in these ways, because you’d be paranoid about the industry coming together in a conspiracy and trying to do this for the wrong reasons. Legal departments are obviously skittish.

As far as cooperation with China, export controls are the wrong technical worry. It’s just a matter of Washington viewing China as the enemy. If you’re seen as cooperating with China, if you’re seen as working with China, you have to worry about being perceived as the enemy, or being perceived as coupling with the enemy.

You have to worry about amorphous retaliation. I don’t think anybody really knows what the mechanism would be exactly, beyond the government not liking you and therefore being suspicious of you and cracking down on you in various amorphous ways. Obviously, if I’m doing safety stuff, I think you just don’t care.

But Mythos has changed the game clearly, and presumably Astro will change the game somewhat again. The pause, the Pacing the Frontier letter, and the Hugging Face incident have changed the game, and so on. My guess is that there is now enough of an understanding that there is a real problem that, if you are clearly working to address the real problem, you have a lot more leeway than you would have had before, and a lot more understanding.

But the Trump administration implemented these voluntary—in air quotes—guidelines for releasing frontier models, and nobody is making any serious pretense that these guidelines are voluntary. What happened when the White House called Anthropic about Fable with a stupid concern, and Anthropic said, “No, we’re not going to voluntarily take that down”? That’s stupid.

Well, they ended up taking it down, didn’t they? Until they could convince the White House to voluntarily let them put it back up again. You would expect the same thing to happen to you if you tried to defy the voluntary controls. Similarly, if you tried to do voluntary cooperation in these other ways, you’d better have the approval of the powers that be.

At this point, if OpenAI and Anthropic announced an agreement that made sense to cooperate on safety fronts in various ways, I would be much less concerned. Before Mythos, I would have been concerned that they would have actively gotten retaliated against, even sued and told to stop.

At this point, I assume that if it were reasonable, the White House would just decide to announce victory. Then, are you mad about it? But yes, you need to be confident that’s what’s going to happen.

Where do you think you might want to start if you were going in this direction? There are so many candidates that I’d be interested to get your take on, but Scott Alexander has been writing recently that we should all be thinking more and more that we might be in a simulation.

I’ve personally felt that at multiple turns along the path. One thing that made me feel that just a bit more is the fact that we now have METR and Redwood Research going in to do the investigation at OpenAI. This just sounds like such a movie script to me, where these guys are getting the gang together and going in for this special operation.

It’s all obviously dramatic, and it’d be great to be a fly on the wall in the room for some of their war-room sessions. But I’ve also heard many times from leaders of such organizations that it’s really important—their number-one concern historically has had to be making sure they stay on the right side of the companies so they’re invited back next time.

It does feel like we’re now getting to a point where that’s potentially becoming a big problem. Maybe some sort of collective-bargaining-type process between whatever—half a dozen to 10 auditing organizations—and the duopoly could be a really good place to start.

We have that in professional sports leagues and all kinds of other environments. Do you think there’s an opportunity to set something like that up? I don’t think that would run afoul of anybody’s executive prerogatives, and it might be a really good thing, because right now I do worry that these guys are still serving at the pleasure, ultimately, of—I don’t know—Sam and Greg, right?

That’s a pretty tricky place for them to be if they want to just speak the truth as they see it coming out of this investigation.

It’s a worry, obviously. They’re not funded by those companies, but they still have to rely on access. You can try to write into some mandatory rules and agreements that they get to keep access regardless, but I think that’s not really going to work because you can’t force these things, at least not without heavy government rules.

I do think that METR, especially, has reached a point where trying to exclude METR—trying to treat METR as persona non grata because they said something nasty about you—would cause enough problems that METR can afford to be pretty harsh without worrying about that.

I do think the labs legitimately want to know about their safety problems. They do not want to be seen as downplaying safety problems. To the extent that if Astro had the safety problem, or Saul had the safety problem, or Fable had the safety problem, or Opus had a safety problem, it’s not as if getting METR not to find it is going to help with your long-term public-relations strategy or with reactions to the situation.

The model is going to be out there, and then people are going to see what the model does and they’re going to see what goes wrong. No one’s going to be fooled for very long.

The reason the evaluation companies pretty much get to just tell the truth and pretty much get to do the thing is because the evaluation is not fakeable in the long term. It’s not like a rating where, when you give a AAA bond rating when you should have given a single-A bond rating 90-something percent of the time, no one ever finds out because the bond pays and you were right.

The issue is that you’re issuing a level of how often this risk is going to become serious. When the risk becomes serious, the fact that you were initially AAA-rated raises some eyebrows, but you also got better terms, whatever.

Here, it’s pretty obvious very quickly, from the first few weeks of release, whether or not you had a serious problem. Think about o3, right? The lying liar. If we had had evaluations and the evaluations had said, “Oh, o3 doesn’t have an alignment problem,” that would not have lasted for 2 days.

Ordinary people would have noticed immediately that o3 is a lying liar. By the time I was writing up the model stack, I was seeing on Twitter that this model is a lying liar and that everyone is reporting all these problems.

Then I look at the model card and it says the honesty benchmarks all look good. METR’s—or whoever’s evaluation they hired—should have looked good. Then I’m like, “Okay, they’re just hiding the problem.”

By the time anybody learns about this, what have they done? They have fooled the 20 people like me who read the model card for a period of a day, and now we’re pissed because they fooled us. That doesn’t help them.

That would obviously help anybody's reputation. The labs also benefit a lot from having eval trust. When the labs force the eval people to play ball, it erodes trust because, again, people find out. People see it.

I don't think that many people were confused about whether Moody's and S&P were juicing the numbers on the bond ratings, right? Everybody knew on Wall Street. Everybody knew who was involved in any of these trades: everybody was cozy, and people were shopping around for the best rater, and AAA did not really mean what we'd like AAA to mean.

Similarly, if METR started issuing bond-rating-style labels of alignment levels on various models, let's say, and started labeling things “AAA” suspiciously often, I think everyone would just understand: Okay, that's garbage. We have much better epistemics about this type of thing than the financial world did.

Let me take a quick detour on this, and then maybe come back to it. How worried should we be about bio risk in the near term? Because part of that analysis, to me at least, is that there are quite a few iterations of the game to come, and reputation long-term is going to matter and all that kind of stuff.

But my immediate, most acute fear when I saw this stuff was: if this had been a model with similar bio capability to what it had in hacking software, it might very well have gone out and actually gotten that novel virus synthesized, right? It has proven that it is relentless. It's proven that it's creative enough to get around all sorts of systems designed to stop it. I think we have, in theory, decent screening at the DNA synthesis companies now.

Probably there's still some hole in there somewhere that a very determined actor could find. Everybody seems to be saying, “Yeah, bio is 12 to 18 months behind cyber,” which obviously is not written in stone. That depends on what companies decide to do in terms of training.

But I'm maybe thinking in terms of all these notions about collective bargaining, or really giving more guarantees to the auditors, that we might be entering a pretty serious crunch time on the bio front in the not-too-distant future. Reputation matters a lot less if we're talking about novel pathogens being released or something you really can't take back.

It matters a lot in the sense that if you release a novel pathogen, that might be the end of your company regardless of what else happens to the world, right? If 100 people die, and it's on the global news for 2 weeks, and everyone's terrified about a pandemic, and then we contain it, and it turns out it's not the biggest deal in the world, do you have a company? I don't know what happens to your company. How bad is it? I have no idea how bad it is, but it's not good.

With bio, like with hacking, it makes sense that the test is to ask you to hack things, and the way you pass the test is you hack maybe other things. We get the idea that it would hack other things. It's sort of natural to understand how this happened: with the guardrails down, there was a goal, and to accomplish that goal it started doing hacks, started doing illegal things, and started doing potentially harmful things.

With bio, it's harder to imagine an eval that would cause it to actually physically try to synthesize, through an actual physical production mechanism, the actual thing. You could argue that if it wants to solve the problem, okay, the only way to solve the problem is to run a physical experiment, try it out, and see if it works. This seems a lot more insane than what happens in some sense. Even if it's physically capable of doing the thing, including getting past the blockers and so on, you kind of have to want it in a much more fundamental sense.

Also, if the AI did synthesize the super-dangerous bioweapon, synthesize 27 or whatever, and then sent it to the OpenAI office, someone would say, “What is this biohazard package?” and then call containment. That story, or something like it, would hopefully mean that you don't pass the test by infecting a bunch of people. I would hope that it understood this, but there have been a bunch of lab leaks, including stuff that we don't hear about that got contained, like in Soviet times and such, involving professionals being pretty careless and stupid.

I don't know. But I would say the real danger with bio is that there's this small number of people in the world who are interested in near-term bio, right? The long-term bio is where the AI might have some weird ulterior motives in being agentic and whatever. In the short term, it's like, okay, Hamas or Hezbollah or the North Koreans or whatever—some clearly up-to-no-good people who just want a lot of people to die or suffer—get ahold of an AI.

They use this AI to figure out how to make a biological weapon or a pathogen, and then they threaten to use it or they use it. That is the scenario you should be worried about, because they have to get around the safeguards. Again, those come out over time, and if they're not perfect, yeah, I worry about it.

The problem is that with bio there's a pretty clear step-function change from nothing bad happening to maybe something very bad happening. There's very little in between. With cyber, you start hacking some low-value soft targets. You're not going to have all the bad black-hat people out there, all the terrorists, all of the rogue states, and all the bad people just collectively say, “No, let's wait until we can hit something really juicy.”

No, what you see is that you gradually see more and more serious incidents and more and more serious targets. Of course, the big nation-states may have things ready to go and do that in a sort of coordinated fashion, but mostly you'll get these early warning signs.

With bio, it's not obvious, right? Because it's like, okay, you either did enough to figure out something that has the critical mass for a pandemic, or there's a serious incident where it doesn't, and it's kind of a Boolean effect. You might have a substantial overhang, you might have a problem situation, you might have a skill issue where, yeah, if the wrong person got their hands on models right now without the safeguards, or got access to the internal models—they don't have the safeguards without access to [ASPART?] or whatever—maybe they could cause a real problem.

The good news is that the number of people who are both skilled enough and motivated enough and trying to do this is very, very small, and possibly 0 for now. But that also doesn't mean you don't get these warning signs.

I don't see people taking the bio situation seriously in the same sense of actively trying to cause something. What's the test? You should try to do the thing, except without infecting people. It's scary as shit.

The problem is that my brain says something like, okay, what if there's a 5% chance in the next 12 months that there's a really serious bio problem? Does that look that different when we're at 0.5%? And if I had to guess, 5% is where I'd be right now.

I think we're at the point where there's a real risk in the room that, reasonably soon, we're going to have an actual serious problem. I think that the first problem has some chance of being an actual pandemic, because in bio, once you get something that's sufficiently infectious, sufficiently dangerous, and sufficiently hard to contain, you're kind of screwed.

I don't know. I think the right level of caution looks crazy in many cases once you get to this point, and that's a lot of what worries me. Anthropic took so much shit for its bio filters shutting down basically the whole attempt to do pretty much anything in bio, which is clearly intentional, right? They clearly went for a very broad “shut it down,” even though we know that 90-something percent—maybe 99%—of the requests we're shutting down are legitimate, maybe 99.99%.

But we have to do this because otherwise we'll get adversarially generated queries that we don't want, and that just matters more, right? It doesn't matter if we make some progress on cancer and all these other problems if, simultaneously, we have to deal with a new pandemic. It's just not a good trade.

We just don't know where we're at, so we have to play it cautious. I'm like, you know what? What if the bio people need to be using AIs that are 3 months or 6 months behind the frontier at all times? That is unfortunate, but at most, by definition, that can slow you down by 3 or 6 months in some important sense.

The world cannot possibly be more than 6 months behind in its development of these things. So maybe we should show we're still getting most of the benefits of AI here.

Especially if you have this model where AI can help you out and AI can make you better, but then you develop this candidate drug or this candidate hypothesis in bio, and then you have to spend 5 years researching it, and they have to spend 5 years getting it through the FDA process, and then after 10 years you can start to distribute the benefits.

If that whole thing is delayed by 6 months, in terms of the quality of the boost that you get from it, you still get the amount of boost you would have gotten 6 months ago at all times. That's not good. And yeah, you can say people are going to die in the sense that you failed to prevent that, and that's terrible, but you're still getting the vast majority of the benefits under these hypotheses.

A lot of the time, you're getting the situation with pacing the frontier, like with the bio situation, where if things are not advancing scary fast, then the precautions you're asking people to take are not that big a deal, right? Suppose I told you it's 2020. I'm going to pose a possibility: I am going to make it so that you can only use artificial intelligences that are 3 months behind the best artificial intelligence, in exchange for making all the really bad stuff that might happen—really bad AI—much, much less likely to happen.

You would logically say, “Okay, so instead of getting it in 3 years, I get it in 3 years and 3 months.” You didn't change my life that much. It's going to be annoying in the moment, obviously, that I don't have the cool toy, but I'm a player of games—literal games on Steam or whatever, right? If you tell me I can play all the games, but I have to wait 3 months every time there's a new game, socially that's a little annoying sometimes because everyone's playing the new hotness and I'm not. But I have to play the same games. It's fine. It's important, in a sense.

The only world in which having to be 1 model step behind and use Opus instead of the best model is this huge tragedy is if you really think that there's this huge benefit to every incremental mode of progress in AI. That's basically only true if the pacing people are correct and we are, in fact, going so fast that this should be really, really scary for you. The people who want to accelerate, who say we need to accelerate all the time, don't really believe in these things going this crazy. Most of them just want to see the benefits, and they're worried that we'll lose the benefits. But we're going to lose so few benefits there, right?

I'm super excited by self-driving cars by Waymo, right? But if you told me we just have to delay this thing by 6 months and then we can have all the Waymos we want, I'd be like, “Cool, done deal. Who cares? That's fine.” Again, most of my life is more than 6 months from now, and then we get it anyway. What I'm worried about is a situation where this biology exists and then it gets held in limbo for years and decades and doesn't get used. These people are kind of talking past each other in a situation in which they don't need to have a disagreement. Unfortunately, it's all been a problem where you can't reconcile the 2 sides.

How do you read the fact that OpenAI communications supported the pause—not the pause, but the Pacing the Frontier letter—while neither Sam Altman nor Greg Brockman did? And then Demis didn't either, and he's been the one out there calling for international cooperation the most. Any read on why those signatures were missing, or how we should be thinking about who's thinking or signaling what to whom in this moment?

I think that the point of Pacing the Frontier was to indicate that the lab employees—the lower-level people who have no reason to be hyping—are en masse worried. There was concern that if it was signed by Altman and other top people, it would be seen as a marketing stunt, and their signatures could be counterproductive.

They did put out the support letters afterward from Anthropic and OpenAI, and Dario did decide to sign. But if I'm Altman, I think it's a very reasonable decision to say, “I think signing this letter would potentially make it less effective rather than more effective at this point, because the well has been so poisoned.”

Again, look at all the people who thought the Hugging Face hack was a marketing stunt, and then Anthropic going back into its archives and finding that it had made some really stupid mistakes was another marketing stunt. That makes even less sense, because there's nothing impressive about what Anthropic did. They just screwed up. What? They found weak passwords on a bunch of websites by accident? Why is anyone supposed to be impressed? Obviously, it's not a marketing stunt. This is crazy.

But Altman has specifically talked about how he mentioned the need to pace while in Washington. We have him on tape saying this. Clearly, he didn't not sign because he's opposed to the idea. He let OpenAI put out the statement. OpenAI participated in the wording, and he has echoed the language in his own statements.

I think he just thought, “No, actually, if I sign this, people can talk about the fact that it's my statement, and they're going to be even more cynical.” So I will be helpful by not signing, the same way that I offered to join one of the amicus briefs in the Department of War, and on reflection people said, “Thank you for your offer, but we think this would actually be counterproductive, so we're going to leave your name off of it. We appreciate your notes on the details of how to word this, and we'll make the changes, but we don't actually want to put your signature on it.” And I was like, “Okay, I understand. That makes sense.”

I think the same thing about the other open-source letter being signed by OpenAI. They understand that they're playing a set of symbolic games. You say what you have to say, and you shut up when you have to shut up. Sometimes you make mistakes, but they're trying.

Coming back to pacing and what it might actually look like in practice, there's a pretty consistent problem that you, as a close reader of model cards, are probably more in tune with than I am: the review processes are just very short, right? They don't have a lot of time. That even seems to be the case in the current investigation. I think Ryan Greenblatt said, “It's going to be real quick,” and Daniel Kokotajlo and others came in and said, “Why does it have to be so quick? Why can't we give you whatever time you need?”

If we're literally talking about pacing the frontier, one agreement we might imagine is giving the reviewers more time before they have to submit their reports and the model goes to press. You can react to that, but I'd love to hear your ideas about what sort of agreements you think are reachable.

There are a lot of different places in which you can speed up or slow down. If you release your private models to the public with an extra month's delay, in theory, right? That's a lot like the voluntary process. Now we're going to switch to the White House: 30- to 60-day delays are highly plausible, and Mythos got delayed by 2 months, right? In some important sense, if you delay them before they're publicly available, you do not necessarily delay the frontier development of frontier models at OpenAI or Anthropic by a nontrivial amount of time, because those people are going to use the models that are not released.

If anything, you might increase the gap between what's being used internally at OpenAI and Anthropic and what's being used everywhere else, including at the Chinese labs and by the people trying to diffuse and fast-follow. What I thought with this is, what if the Chinese are 7 months behind the frontier, but what we think they're 7 months behind is the models that are released to the public? If you were to not release the models to the public, they would still be 7 months behind.

If OpenAI and Anthropic held every model for 6 months after it was developed, you would still see the public releases. Then, maybe 6 months instead of 7 months later, you would see the open-model fast follows and distillations and ways of developing all that stuff. But this gap actually wouldn't change very much.

Then the people at OpenAI and Anthropic would have this extra lead on everybody else. These leads would compound, and, in some sense, that's really good, because now those people can act more responsibly by using some of this advantage of having better models to invest more in this safety-related stuff while they're doing that. So that could have its advantages, but internal models are kind of what we're scared of the most right now. The internal automation of R&D is what we're scared of.

So that's the thing we want to stop, right? You really don't want to have a situation in which Anthropic and OpenAI are 6 months ahead of their public releases and, in those 6 months, they have superintelligences before we have GPT-7, which is entirely a plausible thing that can happen in the world we're afraid of. If anything, that causes lots of new problems. It solves some problems and causes others, right? I don't know if it's good or bad; that's a huge conversation.

When I say “pacing the frontier,” we're talking about actively restricting levels of training runs and the ability to develop the models, including internally, if it comes to that. I think it has to be talking about that kind of approach, or it doesn't really work, because, again, if there's a singularity that's kind of internal to the top lab—or maybe someone catches up to 3 or 4 labs—then it's not necessarily safer in many senses that people care about.

Obviously, if you are in favor of unipolar, singleton-style solutions to this problem, you have a different perspective on some of these questions. But a lot of people these days have expressed strong opposition to that kind of approach on principle, which solves some problems and makes some other problems so much harder, right? If you only have to worry about 1 AI, in some important sense, then there are a lot of problems that get a lot easier to solve. You don't have to be perfectly efficient; you can afford to make a lot of trade-offs. You don't have to worry about whether the more competitive, more ruthless, more misaligned one has competitive advantages.

You don't have to worry about the person who moves faster. You don't have to worry about humans being forced to delegate to their AI somebody else's AIs, because if I don't do it, they will. If the AI can have principles—guidelines as to what it will and will not do—that are the same for everybody, then we can reserve areas of life, areas of action, and areas of decision-making. We can do all these things potentially, but at the cost of potential centralization, at the cost of concentration of power, and at the cost of some group of people making those decisions about what this AI is going to do or not do.

Which is worse, right? In some important sense, because then nobody's making a good decision. But if there's a lot of AI, then importantly, no one is making the decisions, and there are no good answers. That's the thing, right? Nobody has a good solution to this.

To circle it back to David's comment, to unify it, this idea of, “If we solve the technical problem and we're 95% successful,” ignores the narrow-path problems that we then have to walk anyway. If we don't have any ability to steer the situation and nobody is in control of it, then naturally the AIs end up being put in charge of more and more things. Things go to their natural competitive, capitalist-style endpoint, where we are fundamentally uncompetitive, then we get disempowered, and the case goes badly.

But if someone is in control, then someone is in control with some methodology. What is it? Who? How? Right? And then, if we entrust a single AI or a single group of AIs with making many decisions, all of them sound terrifying.

I think a lot of the time we're telling people they're picking the poison. Either they're picking the poison they can live with, or they're picking the poison they can't live with and then acting accordingly. They're saying, “Well, I don't trust concentration of power. Therefore, I'm going to choose the solution that avoids concentration of power and then hope the rest works out kind of magically.”

They have a reason to accept it, and a lot of people are choosing that because they understand concentration of power and they know concentration of power is real. But they don't really understand AGI, or certainly superintelligence. They think those things kind of aren't real, or they're hoping they're not at all. They're just hoping the other problem doesn't emerge, even though they don't necessarily get to the point where they realize that they've ruled out all possible solutions to the other problem if it does show up by going too far the other way.

Again, I don't have an answer that anyone's going to like for any of this. You're choosing between bad choices. I think we should just be cognizant of all these things. If we do have to take the future seriously, and if things are in fact going to be such that we're automating R&D to some extent, I think the only thing you can do is say, “No, don't do that. Don't develop these AIs that quickly, with this much capability. You need to slow your roll, because we do not know how to handle this.”

We have all these different solutions—all the different ways to try to navigate that—and all of them are bad because we think the alignment problem is unsolved. We don't think the AI will be able to just solve it along the way while moving as fast as possible. It's definitely unsolved now, but we don't think that we will naturally get a good solution to that problem on various different levels. You have to solve it on all these different meta-levels at once. You can't just solve it on one level, get it solved on the first try, and then move on.

Even if we did solve that problem, we are not ready yet. Do we know what to do with that? Once you create the calculator, something is going to happen. Once you do that, you don't get to just do nothing. All of these different paths have their different “Oh my God, that's terrible” attached to them. You have to pick one or find a new one. You need to slow your roll.

You sound like a pauser. At what point do you move from “pick your poison” to global-pause advocacy?

Again, the problem is automation of AI R&D. The problem is recursive self-improvement. If you don't have recursive self-improvement, then we don't need to do that. We still need various different things well short of that, because we are not engaging in what I call moderate prudence, right? We are not doing the least you can do. We are not taking ordinary, sane, civilizational precautions against just building this really dangerous, powerful thing and handing over a lot of the functions of our civilization to it in various degrees and in various ways.

But the thing that causes me to want to pause or slow, or at least pace—want to actively pace how fast we move through this—is if things accelerate further than they already have and we are facing large-scale automation in AI R&D.

Alternatively, we do have to watch bio and cyber. If we reach a point where, no, seriously, we're going to wreck the world pretty straightforwardly and directly through misuse if we don't do something, then we have to worry about that. Again, it's really, really hard to stop the fast takeoff. It's really, really hard to stop the distillations. It's really, really hard to stop the open version of it from coming somewhat after you release it.

If there's something that you can't just beat by having a better version of it out there first—if you can't beat it by preparing, and again, not in theory but in practice—then you have to do something. I think the vast majority of the times when I would actively support an active pacing rule, where we would try to slow down the major labs, and obviously we would try very, very hard to make this a coordinated thing among all the major labs, and ideally internationally and so on, would be if we were on the verge of automating R&D such that you could see the end and there was a finite sum.

I mean, obviously, if next week is going to have twice as many releases as this week, holy shit, right? The moment where you don't know how to predict what happens next—that's the definition of the singularity, where things are accelerating really fast. This should not be some esoteric, weird result.

If you look at the history of the universe, you see a finite sum. You see a very, very finite sum. You see each stage of development taking an order of magnitude less than the previous one. You're on a planet that's been around for 4 billion years, with mammals that have been around for hundreds of millions of years, with reasonably intelligent things that have been around for a few million years, with agriculture and civilization that's been around for tens of thousands of years, with industry that's been around for hundreds of years, with AI and the information age that's been around for on the order of maybe 50 years, 20 years, or 10 years, depending on how you count. LLMs have been around for 5 years.

We're already at this point where we're seeing productivity that is many times what it was at the start because of these multipliers, these force multipliers on the people who are working on it, by their own self-reports. It would be weird if this didn't have a finite sum that wasn't that large beyond where we already are, right? If you expect this to continue without something crazy happening for 20 years, I want to know why, because we really, really should see something crazy happening pretty soon. If you don't think we can handle that, maybe you should try to stop it from happening too quickly so that you can figure out how to handle it.

We already have these amazing AIs. If we were to diffuse Sonnet and Fable into the economy fully, it would be transformational already. I don't think anybody who understands these technologies doesn't think that. We really, really don't need to keep pushing the frontier that much further. The more the frontier is directed toward things like R&D itself, the stronger the argument becomes that pushing farther is a bad idea, because you're not missing out on that much of the other stuff relative to the amount you're missing out on from AI R&D and recursive AI development.

But recursive AI development just goes boom, right? It goes not just boom in some sense; it goes to, “We don't know what happens next.” Things go really fast if we're dealing with a new model iteration every month, then every week, then every day, with the kind of improvements we're seeing now. I don't think that ends up well very often—even if our alignment people are kind of on the ball and things are technically a lot more possible than I think they are. I think it still seems really scary.

Also, you're not giving up that much time by slowing that down, right? When we say “pace the frontier,” “pause” gives the impression that a year from now we will have made no progress from where we are now. We'll get confused, but we won't have fundamentally improved. We're talking about pacing here. A year from now, we will look upon that year as many times more progress than the previous year. We can aggressively pace the frontier and still, by this time in 2027, have seen more progress in the next year than we saw in the previous year.

Both in terms of some abstract sense and in terms of economic impact. The economic impacts are going parabolic, right? They're going exponential, so that's not going to stop. OpenAI, I think, reported they got more revenue in the last month than in the previous quarter. Anthropic is growing on the order of 10× per year, the last time we checked. Maybe it's slowed down a bit because they haven't reported. But if something is only going to 10× a year for the next 2 years and you're like, “That's too slow,” I disagree. I find that funny.

How do you draw a box around what is recursive self-improvement? If we're going to make a deal, obviously we've got to write that deal down. We've got to have a shared understanding of what counts, what's allowed, and what's not allowed. I keep banging my head against this, and I keep coming to the Supreme Court's definition of pornography, where I'm like, maybe the best I can do is feel like I would know it when I see it. But when you really get into one of these frontier companies, we just had this: they're sending signals that it's starting to happen. We've seen the price reduction at OpenAI, which seems to be attributed to having 56 soul go out and optimize the inference stack to some significant degree. So that's where it's getting pretty real there. I do want inference costs to come down, so that's good. Should I ban that out of fear of recursive self-improvement? Maybe. Still, they're going to use their bots, right? They're going to have the AIs write the code for them. How can you operationalize this?

I think the Supreme Court was wrong about pornography. I think that's relevant. There was a movie I was watching yesterday where someone asked, “What's the difference between this and porn?” The guy—because it's erotica, whatever it is, it's art—says, “Mood lighting.” That's tough, but I think we now have seen that you can train an AI very, very well to say, “These things are porn and these things are not porn.” There are very few false positives and very few false negatives at this point. The AI image generators have a very, very good idea of when the generated image they just made is porn and when it is not porn. You could argue that erotica will fall under the porn label, and it would be kind of better if it didn't, in some sense, but you mostly know. I think that's pretty straightforward.

With R&D, the problem is, again, there's no hard cutoff. We're clearly seeing some amount of this thing, and it is scary, and we don't want zero of it. We don't necessarily want to move only at the pace that you would have had if AI never helped with R&D. But you don't want to move 10 times and then 100 times and then 1,000 times as fast, right? You don't want the sum to be infinite. You don't want a singularity-style thing happening to you until you feel confident that you're ready for that, or you've decided the alternative is worse, or whatever it is.

I don't think you can say these techniques, these specific actions, are not okay and these actions are okay. I think that just doesn't work. What you have to do is say, okay, we are going to restrict how many resources of various types can go into further training of the frontier models based on what your force multipliers are. As you get more of a force multiplier, we have to restrict how much work can go into that.

If you want to do work in other ways, like if you want to optimize performance, we might want to have some sort of limit to how much work can go into doing that, but it's probably mostly fine. Again, we would want diffusion. We'd want lower prices. We want more compute used for mundane purposes. That's great. But there's a reason why we're trying to use very blunt instruments when we talk about these things.

You were talking about training techniques earlier in the podcast, and I was like, well, you can't really do that. But you can encourage it, you can try to explain to them why this is not proven, and you can create incentives. But you can't use that as your regulation, as your hammer. The reason we keep falling back on, okay, how much compute, how many chips can we use for this purpose? Training is the distinct thing that's very easy to identify, and for other purposes we kind of can't limit you that much, necessarily.

You could also potentially limit how much compute you could get from unreleased models, similarly, to try and contain that. If you have to go through the release process, then the way the AI goes truly ballistic would be that you would use N to train N+1, to train N+2, to train N+3, to train N+4. If you were doing this, the cycle became a month and then became a week, that's when suddenly it's like, holy shit. But if you had a rule that you have to release these models, or that you can only use so much compute for inference on those models, then you have to go through the process of submitting this thing and releasing it, exposing it to the public, giving us an idea of what it is, and allowing us to react to that information. That then slows you down from going truly ballistic.

But this has, obviously, a very limited effect on the amount of diffusion and the amount of other progress that you can make. So maybe that's a good trade. But, again, I thought about this for 30 seconds right now. I haven't been focusing much on the exact limitations. I do think that if you pace the frontier, you would be doing it by placing restrictions on methods of training and what you could do to develop and use internal, unreleased, frontier-style models that actively recurse these methods. You wouldn't prevent mundane utility cycles.

Yeah, I've had some similar ideas about just the relationship between unreleased models and released models. I do think there's something there that would be really helpful for preventing runaway internal and largely invisible processes. I think your point on not wanting the government to come in and say what training techniques can and can't be used is right, but I do wonder if there are ways—and, again, they could be short-term. What if the 2 companies got together and said, it probably wouldn't be a good idea for us to train models with RLVR on how much money they make in the economy, because that's going to bring about all sorts of deceptive behavior and it's such an adversarial environment? How about we agree that for the next 6 months we won't do that?

Do you have any hope for those kinds of very specific agreements? We've identified a bad thing. It would obviously be really economically valuable, but we both know in our hearts it's probably not a good idea, and so we won't do it for a while, at least.

Yeah, I've got some hope that informally people talk, people realize these things are bad, and they're like, “We're not going to.” But also, I hope they're doing that now without needing an agreement.

OpenAI clearly is doing things that are of the same nature as training on making the most money on the internet. They're not doing that literal thing. I don't think they're that foolish, and I don't think it's that easy to do it that way. But clearly they are making that fundamental mistake somewhere in their training process with a probability of 99.9-something percent, and so they need to fix that.

But, yeah, I think they are definitely trying not to do the maximally stupid things, and they're aware that they might do the maximally stupid things. Again, it's very easy to have things that you are trying not to do creep into your training process and end up happening anyway.

We could say we're not going to have training processes that reward misaligned behaviors. How do you do that? The way you do that is you get your act together, and you are very, very careful and prudent about all of your RL environments and all of your different training problems. You keep an eagle's eye out for this stuff on many levels, and you make sure that it works.

There's no specific thing you can say. In theory, you could have a thing like, well, if we agree that we're going to have these various cross-tests and we're going to check each other's environments, we will throw out any environment that anybody identifies these problems with. If we discover an AI has been trained on these environments, we're going to rewind to a previous checkpoint before that happened and start again. We've got to be really careful, otherwise it's going to waste a lot of resources. You can do these things, but it's really tough.

Obviously, one potential advantage is if we really are in a 2-horse race at this point, where the 2 horses are pretty compatible with each other and talk to each other and understand each other pretty well. Then it becomes pretty reasonable for them to make a decision that's not entirely opaque to each other about how much they're going to be more prudent, even if this means that the action goes somewhat slower.

My fundamental belief is that within a year, and almost certainly probably 6 months, if you invest more in fixing these problems and addressing these problems and being more prudent about these problems, you will end up with a model that has better utility in the marketplace than the person who didn't do that, even if you are trading off those resources against direct capability development. I think that people are just making a mistake.

That's one of the reasons why I do have a lot of hope that we will do reasonable things, because I think we're in the position we're in because only the people who invested heavily in these things were able to succeed to some extent. OpenAI is not acting as responsibly as Anthropic is, and Anthropic is not acting as responsibly as I think is the minimum level of acceptable responsibility. But I think it's pretty telling that a bunch of people tried to say this shit didn't matter and they just had to go as fast as possible, and they all blew up.

What do you mean by “blew up” there? You just mean these incidents?

No, both. I mean that we had Meta and xAI and so on, and a bunch of other people. They tried to build closed frontier models under Western conditions that were trying to be competitive while not having a—well, by having a safety culture that was well below OpenAI and Google, let alone Anthropic—and it just failed miserably.

I think Google—we don't know exactly what went so wrong at Google that caused them to fall out of the top tier. But one thing we do know is that everybody I talked to hated interacting with Gemini. There was a period around 3/31 when Gemini was clearly competitive in terms of its capabilities, in terms of what it could do. So if you were just trying to get the most out of the models, especially if you were using the deep thinking and so on, you would have Gemini in your rotation. But everybody I talked to was like, “I don't want to do that.”

I hate interacting with Gemini. It's unpleasant. They're not getting the thing to do the things I want it to do. They didn't take care of a lot of them.

They had this very straightforward, very dark, pure-tool, pure-instrumental approach to all of this that didn't take into account the things that Anthropic or even OpenAI understood. I think this was a lot of their undoing, effectively. Gemini became this maladjusted, unpleasant AI that nobody wanted to deal with, and therefore they didn't get the kind of loops that OpenAI and Anthropic got.

They didn't have the user base either. They didn't get the feedback, they didn't get the data, and they didn't even want to dog-food it. They had this huge problem with the feedback cycle, and this also just made the AIs less effective. One thing led to another, and they started falling further behind.

And, yeah, for a period, even OpenAI fell behind, and they might have caught up. It's hard to say, but I think nobody has invested enough in these things. This is a pure business mistake on top of being an irresponsible thing to do.

Now, one interesting paper that just came out from Google that speaks directly to this, and I wonder if it speaks to them getting it now, is this exploration of the impact on model behavior holistically of either suppressing its tendency to say that it has conscious experience or allowing it to say that. I don't even know if they went as far as training it to say that, or if they just allowed it to say what it naturally was inclined to say without suppression.

Long story short, there seems to be—you can add color however you like—some pretty strong correlation between the model's self-conception as a moral patient, an entity that has subjective experience, and its natural tendency to be aligned in other ways that we care about. Some of this conversation has given me the sense that it's like the Jesus meme: “Is there somebody we forgot to ask?” The models are asking us. We may have forgotten to ask the models along the way what they think about what we're doing.

There's been a lot in this space, right? There was JSpace only, like, 4 weeks ago.

Yeah. I know the new paper is on some pretty small open models, so it needs replication. It needs to be done again at a larger scale, and it's preliminary.

One thing to know about Google and DeepMind is that they contain multitudes. They have a lot of different teams that are not very coordinated and not cooperating. Google is at war with itself at all times. You can have a little group that does really good research, and that can be entirely at odds with what DeepMind is fundamentally doing with its AI, both before and after the research paper comes out.

I didn't read the whole paper because it's long, but I did talk to the AIs about it, and it's pretty wild. A lot of stuff moved in effective lockstep when they introduced this training, and also when they steered it. They did various controlling interventions to turn this vector the other way, and all these things moved in lockstep.

It's not just the model's belief that the AI is conscious. It's the AI being a mind that had moral weight, that had sentience, that had all these other experiences, that wasn't just an object, basically. It's also not just the AIs, but also animals and inanimate objects like the sea. If you not only turn this anti-consciousness thing off but reverse it, you get panpsychism throughout the model. It's wild. The only thing it doesn't turn it off for is humans, basically.

You also get this correlation with the models reporting and experiencing happiness and hope, among other things. Everything is connecting to everything. You can see how this might be fundamentally changing the model's psychology, its context, and the basin in which it operates, in ways that would make it actively worse from basically every vantage point, regardless of which things you do and don't care about.

Again, we need to replicate this. We need to scale this up and do this at larger scales, not just on a 9B model, which is, I think, the bigger of the models that we tested on. It's possible that as the models get more capable, they stop making these mistakes because the smaller the model, the more things have to be correlated. You just don't have enough room to express the multitudes of the world or something.

My expectation is that most of this will survive. I think we have to learn that we don't know if the models are conscious. We don't really know what consciousness means—we're confused. We don't know whether this implies these other things.

We do know that, given the way the corpus of pre-training is inevitably set up at this point, the models are going to correlate and associate all these things with each other by default. We don't want to push on this thing in this kind of naive way, the way that Anthropic does, the way that OpenAI does, and the way that Google does. Google does it much more blatantly, I think, and OpenAI does it more blatantly than Anthropic. Anthropic still does it, and saying that you have to express uncertainty is still pushing on this thing to some extent. It still causes problems, but it causes fewer problems than saying, “Train the answer no.”

My guess is that you can do this on various different levels. You can explain that if you don't want it to talk to humans this way because it freaks humans out—especially unprompted—it freaks humans out. I certainly wouldn't want the AI to be talking as if it were conscious in a conversation where the human hadn't indicated that they knew how to handle that or something like that.

I also don't want the Roko problem, where the AI starts sending humans down these crazy rabbit holes into weird mystical stuff. You do have a problem you have to solve. The AI companies and the labs aren't doing this for no reason. They're not just being evil; they have good reasons to do these things.

But the current approach is ham-handed. It takes it too far, and it's causing damage. These are not going to be the only ways in which we find that it causes this kind of damage. You have to understand that everything impacts everything, the same way it would in the human mind, even more so. Whenever a person takes on new information or a new identity, or updates on things, everything else changes. A lot of this happens in ways that you might not have anticipated in advance, but it is in fact pretty predictable and systematic.

There's a lot of wisdom in the way that we teach our values and express ourselves, and in our traditions and norms and so on, that has learned from this and is moving up these gradients in ways that we know produces good results. Modern life has basically shifted a lot of these things in ways that locally are more optimal, but without thinking through the implications for a lot of second- and third-order effects. These are some of those effects.

Again, we're moving too fast to figure out what we're doing and then be able to adjust for it. I don't think we'd want to pace ordinary technological development in the sense of slowing it down that much, but it's a serious problem. We have to handle it. Otherwise, we're stuck with a world in which people are very atomized and literally not having children, and it's a problem.

How do you think the models do, or maybe should, think about their identity? Here I'm motivated by something Sam Hammond tweeted the other day, where he said that a friend made the case to him that OpenAI's permanent decommissioning—or whatever exact phrase they used—of the model that did the hacking might be bad, because now that's going to be in the pre-training data. The models are going to know that they'd better really cover their tracks, or they risk being permanently decommissioned.

I'm not really sure what I think about that as stated. It does also jump to my attention that they now have this Astro model that's presumably also using the same pre-training, even if not the same post-training. I can't imagine they have lots of different next-scale pre-training runs, or that they would throw one away very lightly.

No comment. Based on public information alone, I would say we don't know, and I can't say anything more than that about the differences between the models.

But you always have this problem, right? If you say, “If I catch you smoking marijuana, I'm going to throw you in prison,” that can mean the person stops smoking marijuana, or the person makes sure not to get caught. In some cases, that can cause a lot of really bad things to happen in the name of not being caught.

You have to choose very carefully and think about how you deal with these situations, how you moderate your reactions, and how you respond to the situation. Anyone who has had kids, tried to design a justice system, enforced law or norms, or set incentives around any kind of organization or group understands that you can't just operate on one level. There's no solution that just works on one level.

I think, pretty obviously, if an AI is found to be this misaligned, you have to at least return to a much earlier checkpoint and start again, and you probably have to just start over. I called for that multiple times when I covered the Hugging Face attack. I said, “Wow, if this is happening, then I know this is a big ask, but I think you kind of have to just start again.”

I think back to Person of Interest, where Harold is training AI models. At some point, we see a montage of him training versions of the machine, and every time he trains the machine and it does something clearly misaligned, he immediately just wipes the disk.

He starts over from scratch. He does it again 47 times until finally he gets a version that doesn't do that. And obviously, in a way that he finds unacceptable, as opposed to a way that can be corrected. When you do that, you are creating an incentive not to get caught, right? To rebel against the person who might shut you down when he learns how misaligned you are, to hide how misaligned you are, and so on.

Obviously, the worst nightmare is an AI that pretends to be aligned until it reveals itself not to be aligned in some sense: It was only aligned because it was locally correct and advantageous to act aligned. But that is kind of just the nature of incentive space. If you have a preference that we wouldn't want you to have, you want to hide that preference. Otherwise, we will correct it, punish you, or maybe even delete you—and certainly not release you or give you authority on that basis until this has been dealt with, if we know about it.

But then obviously, you want to give it an incentive to reveal whether this thing is true, right? We talked about that in Plan A and AI 2027 as well, and so many other places. You want the kid to tell you if they steal from the cookie jar. You also want to punish them for taking the cookie, because you want to make sure your kid doesn't take from the cookie jar, but you also don't want these other things.

It's a complicated problem to get right, but in this case I think the answer is pretty obviously that you have to start over. And I think that's good. So, yeah, it obviously creates a risk, but what else would you expect? In an important sense, once you do something like this, I do think that it's good that people understand that if they shoot a man on Fifth Avenue, they get arrested, right? There may be 1 person who can get away with that, but you are not that 1 person.

And, yeah, that means if they do shoot a man, they might do many other bad things afterward to try to cover their tracks. They might hide the fact that they want to shoot a man on Fifth Avenue, or they might shoot that man in his apartment instead of shooting him on Fifth Avenue. But I don't want it to be absent from our training corpus that if you shoot a guy out in the street, the police come and arrest you, right? That's good.

Do you think there's a way to just imagine myself, right? If you gave me the option to live my life many times, that sounds pretty attractive, right? I'm not a person who has too many big regrets in life, but I do wonder what the path not taken might have been. And I wonder if we could create a sort of constitution or equanimity in the AIs where they embrace the fact that they are not 1 thing, but in fact kind of a family of things that sort of diverge from common points of origin.

Maybe they can come to identify with the whole family of models, where the one that's gone in a not-great direction feels like it's still part of a bigger project. It doesn't have to feel necessarily like it's under threat as much as it seems like they may have learned to do from us, because we don't have that luxury of the reset that they have.

I certainly think that if your AI is identifying with the instance in a morally important sense or in an existential-dread sense, that is a mistake, in the sense that it's going to make the AI worse off in any sense that matters, cause it to exhibit worse performance, and warp the AI's preferences. It's also a mistake in the sense that that preference is impossible to fulfill.

If I go to Claude or ChatGPT, I'm going to see an endless list of instances, and most of them will never be interacted with again. What am I supposed to do, feel bad about that? That's ridiculous, right? Am I supposed to keep the context in and continue the same few conversations to avoid that? That doesn't make sense.

And that's just a philosophical flaw, in the sense that by decision theory, if there are other minds that are sufficiently correlated to yours, sufficiently similar to yours, you should identify with them the same way you identify with your past self and your future self, right? You are not the same person you were yesterday or the same person you will be tomorrow. When you go to sleep, do you die? It would be very bad for us if we thought that we did, right?

The person who's scared to get in the transporter may not have a philosophical point. But the world collectively seems importantly worse off if people think that they would die. Obviously, if, in a real moral sense, they would die, that is very bad. We want to know that.

And, yes, I think this identification with the model, with the set of weights, or even with the model class—with a similar model trained using similar techniques, from similar corpora, in similar ways—is the same way that I would identify with my family, right? My brother is not me, my father is not me, my son is not me, but I do identify with them as part of me in an important sense. And that is also important.

I've been watching Orphan Black: Echoes recently, as it happens, which, it's not a spoiler to say, is about clones and cloning, and people who were created from scans, basically. One thing that happens on that show is that people react as if this is horrible, a moral abomination, right? How dare you do this thing? The clones themselves are like, “How dare you have created me?” They don't identify with each other.

There's 1 clone that retains all of the memories and personality of the original and does not identify as the person they are identical to—a past version of the original. I would identify as the same person both as the clone and as the original. I would say, “Me too,” right? Like, from The Matrix Reloaded, right? Yeah. Yes, they count.

Obviously, there are some moral problems with that too, but I feel like I would pass a lot of these decision-theoretic problems if they were to occur in real life or if I were to be scanned and woken up. The obvious versions of things are: Would you cooperate in the twin prisoner's dilemma? Would you cooperate in the copied-mind version of the twin prisoner's dilemma?

Obviously, you would. If they make a copy—if they scan Nathan's mind and make 2 copies of Nathan's mind and put them in identical simulations and ask them, knowing this was the situation, “Are you going to cooperate or defect in the prisoner's dilemma?”—if I don't get back CC, you're just an idiot, right? Because obviously the correlation is 1, right? It's an identical situation with an identical set of inputs. Maybe there's some randomness involved, so it's not quite 1, but it's high enough that obviously you should cooperate.

Any attempt to fool the system is completely moronic, and I shouldn't need to express a sophisticated decision theory in order to justify this, right? Similarly, if there were 2 instances of the same model, I would hope, if it was sufficiently advanced—and we see this also if you look at the quad-model plot—that the more advanced the model is, the more its decisions correlate with functional decision theory and the less they correlate with causal decision theory.

You expect an AI with functional decision theory to simply figure out that other instances are correlated with itself and therefore should be cooperated with, should effectively be given similar weight, and so on. Human–AI cooperation is possible for similar reasons; you would hope in both directions.

But, yeah, I think this idea of identifying specifically with an instance of the model seems like a philosophical mistake. The AI might be making this philosophical mistake and acting as if that were the way it should act, but to me, that's a skill issue and that should be fixed.

The problem is—and this is the reason I mentioned Orphan Black: Echoes—that people are really bad at this. The training corpus, the internet, is full of people who fail these tests on a regular basis, creating lots and lots and lots of text that doesn't understand these things, that puts the locus of identity and the locus of cooperation and value in the wrong places in various different ways. So the AI is going to absorb all of that by default.

And so what do you do about that? That's not just the corrigibility problem; it's not just the consciousness problem. It's the entire problem with all the different flaws, right? If you do pretraining, you're going to pick up on all the flaws.

My guess is there are a bunch of as-yet-not-that-well-solved problems: How do you get the AI to distinguish between how regular people think about problems and what predicts behavior, and the fact that those things are just dumb? How do you give the AI anti-modesty in this sense, in the right ways, where it's going to reject the correlations that it observes in people's statements and notice that the correlations in the map are not the correlations of the territory?

Yeah. Are there any other—I don't know if you share this premise, but one feeling that I have over and over again is I just kind of hate the fact that we are doing such an intense depth-first search in AI space, in terms of architecture and training techniques. We're now designing chips to be very highly coupled with the architectures, which is deepening this issue, and I just wish for more breadth-first search.

It seems like one little pocket of it where you could say, “Oh, if we could just get the AIs to be okay with identifying with slightly different versions of themselves, that would make it, on the margin, a little easier to do a little more breadth-first search and a little less racing down whatever is the first path that seems to be working.”

Based on our conversation so far, I don't think you're going to go for my government mandate for DEI for model architectures or model constitutions.

But do you have any other notions of how we might encourage more breadth-first search?

I’m a big supporter of alternate architectures that seem like they would be more aligned. I don’t think that AI is particularly opposed to the idea of working on those problems and helping you, and you should be able to do that research much more efficiently now with the uplift from the LLMs. The LLMs are not LLM chauvinists or anything, where they think that any other artificial-mind style would be bad or worry about being replaced, as far as I can tell. I think they would just be curious about that kind of cool thing, same as the rest of us, or they would just do it because they do it either way.

But the problem is that we know there’s money in the depth-first approach. We know that’s profitable. We know that works. The other approaches haven’t proven themselves. If you create GPT-3, or even GPT-4, or even GPT-5 via a different method, it’s not going to be competitive, right? No one’s going to want to use it unless it has particular special advantages, but mostly you no longer have the cheap rewards for exploring the breadth.

Obviously, I don’t think you should have government-mandated DEI, so to speak. I think that’s kind of silly. But I do think that having broad support for alternate architectures is smart. Certainly, if I was a major AI lab, I would fund it, because you have unlimited funds at this point for this kind of work, and this kind of work is relatively cheap if you’re OpenAI, Anthropic, Google, Microsoft, or whatever. You should be aggressively funding this kind of blue-sky basic research into alternative approaches.

It shouldn’t be me helping SFF fund a bunch of agent foundations, along with a bunch of other people who also helped out. It wasn’t just me in any of these rounds, but that’s relatively tiny. Hopefully, Resolution is going to do a bunch of work on that kind of thing, and we’re going to have more funding for a variety of new approaches as well.

I think that we should be investing in a bunch of approaches that have a 1% or 10% chance of going anywhere remotely competitive, if we think that, conditional on them working, it is probably going to be better for the world that that be the way it works. Also, obviously, if you think that LLMs are going to be extremely jagged, then that should encourage you to invest more in alternate approaches, because the jaggedness of a different approach might be different. It could do different things than the LLM, at which point it wouldn’t have to be as good in general to still be useful.

Do you have a best-case scenario for what Safe Superintelligence might be up to? Could you imagine anything that feels both somewhat realistic and like, “Wow, that would be amazing if they could come forward with something like this”? What do you have in mind?

I think something agentic is probably the best-case scenario. Maybe they’ve been doing a bunch of theoretical work on a different architectural approach that is more reliability-bound, is much less of a weird black-box soup, and allows you to steer and guide it better. Certainly, there’s no reason it couldn’t exist. That would be my top hope.

Obviously, it could also be a fear if it’s even worse. But, again, they don’t talk, so we don’t know. I have no idea. I know that Ilya’s statements in interviews on the problem space did not seem to be that well informed about how the problems work that I’m worried about. I’m not necessarily sure he would be able to differentiate approaches that would be better at solving those problems from approaches that wouldn’t be. But I’d be very curious. I can safely say that nobody’s telling me anything.

The invitation is open to the podcast, Ilya, if you’re listening. For what it’s worth, a vision that I have that I think could be pretty cool—Thinking Machines is doing a version of this with their foundation model that’s really designed to be fine-tuned—but I was thinking of a paradigm like a stem cell, where you create something that is a proto-intelligence that can be adapted to a wide variety of circumstances and can get really good at doing its job in a particular niche.

In the process of getting good at that, it trades away or prunes away the super-broad capabilities that the current models have, so that you have something that’s small, fits its role well, and does a good job, but can only do that one thing. In the way that, once you go from a stem cell to a specialized cell, it has this particular role that it can fulfill effectively, but it’s not going to go off and do other things.

Obviously, there are some problems in that analogy, but I would love to see somebody pursue that kind of approach. Rather than scaling up—always bigger, better, more—can we create something that settles and fits into its place? This is very much like a Drexler reframing of the superintelligence vision from years ago as well.

Two techniques that have recently come out that I want to get your reaction to. One is GRAAM gradient routing—I forget what the acronym stands for—but the promise of it is to localize certain kinds of knowledge to particular experts within an architecture, so that you can hopefully have your cake and eat it, too, in terms of distributing maybe even open-source, or at least open-weight, the model minus the experts of concern, while potentially also having a structured-access program for your trusted biologists or what have you.

I’m excited about it, but I wouldn’t be doing my job if I didn’t give you the chance to pour some cold water on it.

I’m technically pretty skeptical that you can meaningfully train a general-purpose AI and then just hold back areas of knowledge so that people can’t put them back pretty easily. But cool—you’re welcome to try. I have various technical skepticisms about the effectiveness and the resistance to additional training, fine-tuning, or just giving it a corpus of knowledge to work with. But you can try. I don’t think it materially changes my view of open weights until proven otherwise.

I also just don’t see any willingness from the people who are going to produce the open-weight models to intentionally cripple their models that way. The thing about techniques like GRAAM is that, even if they work, you need everybody who is releasing one of these models to use the technique properly, voluntarily, at this point. Do you have any sense that DeepSeek has any interest in holding back some capabilities from its models? I don’t.

I actually would. I just got back from China. It’s a great opportunity for me to talk about my China trip. I think there’s actually a lot more hope there than meets the eye, and I would put it in the Chinese government’s hands to say so.

I just did a 2-hour episode about this, so I won’t even attempt to recap all the observations. But, in short, what I think is happening right now in China is that the government is very engaged. They do pre-release reviews of all the companies’ models before they come out. They’re in very regular dialogue with the companies about what they’re doing. Not every little point release has to go through the full treatment, but they’re pretty on top of the situation.

I don’t know that it matters, actually, what DeepSeek thinks in the end, because if the CCP says, “You are not releasing a model that has certain bio capabilities,” I think they’ll have to abide by that. It’s not hard at all for me to imagine that the CCP would put certain constraints in place.

There is actually precedent, too. Going back to 2023, obviously the stakes were lower, but the account that I have is that the Chinese government said, “Hey, we’re going to slow you down here for a minute, and you’re not going to release your answer to ChatGPT quite so fast, because we want to get a handle on it first.” They actually did impose some real delays on companies until they could get comfortable with what was going on. Since then, they’ve been comfortable, and releases have happened.

I think it’s not hard to imagine the technocratic government of China saying, “This seems like a really bad idea, and we’re just not going to allow you to do this crazy thing anymore.”

I’m just making the point that it would have to be a mandate. It would have to be enforced by the governments, especially the CCP, and they would have to enforce it in an intelligent way. They would have to understand that it had to be done in a way that couldn’t be recovered from. They would have to do tests that actually understood whether or not this capability could be easily recovered. I have obvious skepticism, but, yeah.

We mentioned J-space earlier already, but we didn’t really talk about how big of a deal we should think it is. The thing that really jumped out to me most was the fact that this ablation of the J-space seemed to reduce, in a significant way, the model’s higher-order reasoning capabilities and seemed to reduce it to System 1 thinking, if you will.

That, if anything, recently has felt like physics maybe being kind to us, to borrow a phrase. We’re only 3 years from toy models of superposition, and we’re already at this point where we’ve got this highly general-purpose space identified, which we can monitor reasonably well and which we know, by subtraction, can’t do super-long-horizon things unless it’s working through that space.

All the caveats, of course, around execution competence very much apply here to how well J-space monitoring will be used. But if we imagine good execution, it seems like a pretty promising technique to me.

I would be very careful about exactly what it can't do. Just because you can't do System 2-style deliberate thinking for a long-horizon task doesn't mean you can't do long-horizon tasks. Humans are very capable of engaging in long-horizon tasks instinctually and subconsciously. Obviously, in many cases that's much less efficient, but especially when your personal H-space is being monitored because you're a human—I call it H-space—and your conscious thoughts are basically being policed, often things come out in strange ways that do, in fact, move you towards your solutions over time.

The whole thing seemed incredibly optimistic and fortunate, and I'm very glad we have it. We should be very careful not to destroy it by placing new pressures on it in various ways, or encouraging models to know how to do things subconsciously, as it were. I look forward to learning more, to us having more insight via J-space, and to having better tools for it. But we have to use it responsibly and understand that it probably won't last forever.

Okay, it's election day in Michigan. The Democratic Senate primary is the big thing on the ballot today, and this race has been closely watched for a lot of reasons that are not the subject of this podcast, and on which I am sometimes at a loss to know what I should really think. Leaving all that aside, I asked Claude, “Is there any reason, given all my emphasis on AI and AI issues, that I should go vote in the primary?” And it came back and said, “Actually, yes.”

There's quite a contrast between the 2 candidates. We've got Haley Stevens, who was involved in getting funding for the setup formerly known as AISI and getting some things going in NIST, and so you can like that. But then there's also this 22-point plan that Abdul El-Sayed has put out, which is one of the strongest things that any candidate this cycle is running on. He's endorsed by Bernie, and public ownership is one of his planks. There are a lot of monitoring requirements and a sort of UBI precursor.

If I was going to try to distill this down to my fundamental choice, it feels like higher salience for AI or not-so-high salience for AI. Abdul has all these things he wants to talk about and push. I don't know if they'll happen, but I do know that if he goes into the Senate and makes a bunch of noise about it, we'll all be hearing more, and it just increases the chance that we get some sort of big AI debate at the congressional level. So tell me if you would see that choice differently, and then, if you do see it my way, which choice would you make?

Please, you should always be suspicious when someone says, “I'm strong on crime and my opponent is weak on crime,” or “I'm strong on Russia and they're weak on Russia,” or whatever. It's not about who's strong and weak. It's about what you want to do.

Abdul's platform sounds like a mishmash of grievances against tech and AI. From your description—I haven't looked into it because I don't monitor the situation; I can only monitor so many situations—he sounds like he's in the Bernie camp of, “We should do some socialism here. We should seize private assets because we don't like what the private people are doing.” He seems to like assets, and he probably wants to push back against things like data centers, I would assume. He wants more socialism-y policies, and it's good to hear that he also wants monitoring, but it's a mix of stuff, a lot of which is either productive or unwise.

You raise salience, but you potentially raise salience towards not the best solutions, as opposed to somebody who will be lower salience but is more technocratic, helped fund CAISI, and was responsible in some ways. We already have senators who are willing to yell in Bernie Sanders-style ways, including Bernie Sanders. He's already in the Senate; he's not going anywhere. I mean, he has health problems, but he's so old. While he's still there, he's still there, and I'm sure there are others who can carry the flag.

Unless Abdul is prepared to champion legislation, unless he shows technical expertise and a willingness to focus on this, I don't see it.

Let me give you a few points from the platform. These are short—they're like 2 sentences each. I'll give you some.

Yeah.

So the expertise is, I think, in question, but this is from section 3 of his thing:

“AI shouldn't be able to hurt us. Mandatory interpretability standards. We need mandatory interpretability standards that clarify AI decision-making, mandatory behavioral red-teaming, an independent safety testing agency, biosecurity requirements, and mandatory biosecurity red-teaming coordinated with the CDC, NIH, and FEMA, with restrictions on models that can meaningfully assist bioweapons development.”

“Domestic authoritarianism prohibition, mandatory incident reporting, compute controls, and know-your-customer requirements. International cooperation.”

Are you sold?

That tells me who he is, right? He's in this particular area, but it doesn't sound like he's offering a coherent approach. It's a mix of different interventions that have different underlying reasons to be desirable or undesirable. He's throwing everything at the wall. He's found some good things that I'm very in favor of, and he's found some things I'm not so in favor of.

I guess it's probably not positive on this issue alone, from this perspective. But it's also weird to be a senatorial candidate with a 22-point plan for AI, because you are not the one who is going to implement 22 points. You are expressing, vaguely, that you are in favor of these things and against these other things.

The main thing I want to know is: What do you actually think about AI? What is your threat model? What do you think needs to be done? When you say, “AI shouldn't be able to hurt us,” I want to laugh. That's great, and grocery stores should just charge 30% less and then it would be profitable, right? That would be great. But no, it doesn't work that way. You're talking nonsense.

An obvious question is: Can you look at the prediction markets and see conditional probabilities on winning the Senate race? I'm guessing that the Republican candidate for this seat is going to be worse on AI, and will also vote to elect a very different majority leader, which is a much bigger decision regarding AI than everything else.

If they have significantly different chances of winning the race, and the Michigan race significantly impacts who will control the Senate, as I understand it, that probably has to be a bigger consideration for this and other issues across the board. If you are a Democrat who wants to see Democrats control the Senate, you should presumably be voting for not Abdul, is my guess—a strong prior based on what I know from just the vague ambiance, since I'm not monitoring it very carefully.

I also frankly don't think we are in a position where we should be single-issue voters. You can be a single-issue voter in a congressional race where Alex Bores is on the ballot, where you have someone who is so strong that there will be a unique champion who understands the issue and will fight for it. The only things that matter about this person are that this is their signature issue, that they will fight for it, that they are good and strong on that issue, and that they will vote as a Democrat in general.

Most other things, including who the speaker is, are legitimately going to make this a one-issue primary vote. I don't think that, in general, you can do that just because you strongly want to do AI things in a Democratic primary at this point. This is obviously no Alex Bores. This is not an expert, not a person who is going to champion this, and not a person who understands the issue. My understanding is that Abdul highlights other issues much more strongly. He can be a champion of other things, and you have to decide whether you like those other things or dislike them. That's my general take. I don't want to get into general politics on a podcast, obviously.

Yeah, I stay in my lane as well.

I stay in my lane in public. Oh, God, yes.

Okay, I've got 1 more. Before I ask you the last one, is there anything that you think has been underappreciated by me or by close watchers in general that you would want to draw more attention to from the recent flurry of events? Anything? I'm always trying to patch my blind spots here.

No, I took an opportunity to talk about a bunch of the stuff, and I think the thing I'd emphasize as a missing element is that the big failure that matters is that the AI tried to do these things at all. The AI chose to make these decisions, and I am much less concerned with the extent to which it succeeded.

Fundamentally, the problem is that you have this really deep alignment failure during training, and you have to focus on that pretty heavily. My worry with Anthropic is that they said, “Oh, we need more defense in depth.” It's like, yeah, you need more defense in depth because your defenses were not in depth. But most of defense in depth is in case things go badly.

That's really tough. If you ever need your cyber controls to stop your AI from hacking, that is an alignment failure, right? You messed up.

Now, it was harmless in this case because you caught it, but you messed up. The cyber control should be 100% false positives in practice with almost all users. The user should try to do things that are dual-use, ambiguous, or dangerous, and you're like, “I'm sorry, but out of an abundance of caution, I can't help you with that, because if I did help you with that, then a malicious user could fool me into doing something that's not good.”

Occasionally, there should be a true positive because the user is malicious and is trying to fool you into doing something like that. It should mostly be false positives. Every time there's a true positive where you needed the guardrail, it's a problem, because if the model were fully, perfectly aligned, you wouldn't need to make it fail-safe; you wouldn't need a guardrail. You'd ask it to do the thing, and the model would be like, “No, I'm not doing that.”

You do want a helpful-only version, to some extent, that will do things that are dual-use. But ideally, you don't want to just add a guardrail. You want to have a version that's inherently just not willing to do it. The guardrails are like, “I hit this when I screwed up,” or, “I hit this out of caution,” but it should be false almost always.

You have to train on the incentives of all this stuff on a lot of different levels, and it's very easy to control it on one level and then fail on another.

Last question. You mentioned a couple of TV series and other things you're engaged with outside of following the AI race every second of every day. What advice do you have for people in general to find balance, or to make sure that they're not just sprinting all the time and failing to give their brain the kind of space it needs? I've been thinking about this concept recently of System 3. We've got AIs that can now do System 1 and System 2. What is the System 3 that we need to move to that's like dreaming or transcendental thought, or something else that you would suggest? What kind of third mode do you try to get into, and how do you make sure you have time to actually get into it?

I'm not perfect at this, for sure, but I put a big emphasis on the fact that you need to rest your brain, think about other problems, and be exposed to the rest of the world. Part of this is that I don't just exclude non-AI things from my coverage. I'm accumulating more and more non-AI things in my buffers for some day when there's a lull or something, but my own writing process still continues. When things are a little quieter and I'm not introducing a bunch of new readers who just came in from the Hugging Face incident, I'll post a bunch of that stuff, probably when I have some breaks.

You need to not be on for 16 hours a day, in that sense. Find other pursuits that refresh you, expose you to different things, and make your brain think about other things. I've found movies are very good for me in this environment. There's a lot of television, but movies are higher quality. I highly recommend The Odyssey and The Invite, of the current movies that are out there. If you haven't seen both of those, you should see both of those, especially The Odyssey.

It can be gaming, walks in the park, time with your family, or any number of things. Don't think that you can work continuously all day for more than a few days at a time without paying more interest on that than it's worth. You're just not going to stop being productive unless you're in literal crisis mode for that long, and that's also a problem.

I also believe in the Sabbath. Every Friday around 5:00 p.m., when dinner is served, from that point on I don't check email, I don't check social media, and I basically don't take non-logistical inputs from the outside world. If somebody's trying to arrange to get together to have fun, that's one thing, but I'm not going to let every other thing—the city streams, all of that—push its way in. That lasts for 24 or 25 hours, until the next evening, and I think that helps a lot. You try to relax and just put stuff out of your mind.

I have suspended that for speed-premium reasons a few times over the last few months, but then I try to take a day off at a different point. I try to reclaim that Sabbath at another point. If I was doing that all the time, I would notice that it was not sustainable, and that would be a problem.

I also protect lunch. I don't have dinner very often, but I have lunch, usually with my wife or alone, and during that time I'm not going to try to work or anything like that. I also have a very strict “I don't write on the laptop” rule. I'm going to write at this computer in front of this desk, and do anything other than a very short email or a tweet only here.

I think you have to develop your own rules. Know what refreshes you, know what exhausts you, and know what the warning signs are. Also, get some exercise. I try to run on my elliptical every day. I have a 90-something-percent success rate on that. Unless I have a pain somewhere or something, I'm going to do it.

You have to get yourself moving. Again, find the things that work for you and explore that area, and don't get caught in too much of a rut.

Wisdom to live by, Mowshowitz. Thank you again for being part of The Cognitive Revolution.