为什么AI的下一轮突破可能来自大实验室之外

If you regulate AI too early, you actually don't solve anything. You still just have the same risk, ultimately.

You willed the thing into being, but you haven't figured out how to control it.

The problem we have now is this rift between the labs and the security community, which keeps coming to two conclusions.

You're sloppy, and you're not complete in what you're telling us happened.

An employee is like, “There's a 10% chance of species extinction.”

Agent swarms completely flip that. These are just roaming drones, but times 10,000, and they will easily mistake a good task for a bad one. So now we need a whole layer internally that's tracking much more about what authentications are being done and what APIs are being used.

The U.S., about 15 years ago, stopped leading in tech antitrust. The problem is that Europe is going to lead with that because they have nothing to lose.

This could change the nature of software fundamentally. The center of innovation has just moved.

1. Pacing the Frontier: Reacting to the AI Safety Discourse

Guys, welcome back to the podcast.

Happy to be here.

Thank you.

Your beard.

Didn't think we'd ever do this again. I can't believe it. This is great. I mean, Martin's just building these $100 billion companies. He's too busy for this podcast.

At least taking credit for it, as VCs do.

Exactly. We have a lot to discuss today, but first, Aaron, why don't we start with you? Pacing the frontier: How have you reacted and reflected on what's happened there and the discourse that's followed?

Oh boy. I think we should start with Martin on this one. He was fighting lots of good ground wars.

Maybe I'll say one thing that we probably all agree with, and then we can figure out where we fracture off. I think we would agree that any AI lab right now at the frontier should be building in the safest way possible, with the highest degree of governance and security. Whatever your definition of alignment is, this is an incredibly important area of research. It's an incredibly important area for the diffusion of AI. You're not going to have AI diffusion without extremely high-quality products that can be trusted by enterprises and that aren't constantly hacking systems.

When I read Dario's post, I didn't disagree with almost anything. It was all about how to have better security for these systems: sandboxing, better testing. There are going to be some debates around the embedded nature of the testers, whether you agree with who those are, and whether the industry all aligns on that. But I think all of the major points were salient and appropriate.

The only question is whether this gets used or leveraged to do things that maybe we don't agree with, which would be a dramatic slowdown of AI because of regulatory controls that would not make it easy to compete with the frontier labs. Or politicians could take the message and run with it, leading to even worse outcomes. It could be used to ban data centers far faster and so on.

I think the actual substance of the topic is incredibly important, and I think it's very important for AI advancement in general. The question is what you do about it, especially from a regulatory standpoint. That's probably where the industry is going to land on very different points along the continuum. Martin was putting up a good fight about making sure that we don't use this for regulatory capture, which I also agree with. But I think the ideas in the pacing conversation are important.

2. Species Extinction Talk: Do Labs Actually Believe Their Own Rhetoric?

Again, it's a little bit of a funny concept because maybe it's not even pacing as much as good hygiene and good engineering. With good engineering, obviously there is a slight slowdown, but it's a slowdown that allows acceleration of diffusion, because you wouldn't be able to have any of the AI diffused if nobody trusted using it. The post is very reasonable, but the atmospherics are not right.

I agree with him more than I disagree with him, right?

On TV—the same day that he landed these things. You can't have these conversations in isolation, because if he's going to agree that there's a risk of species extinction—

This post that he has looks like this milquetoast capitulation that's totally not adequate for the task at hand. I think the atmospherics are totally broken, and a lot of my comments were about the atmospherics.

I have this quibble, but it really bothers me, because I'm a pedant: I think pacing is the wrong way to describe this. For one, yes, it is orthogonal to security.

Mhm.

You can very slowly build a nuclear weapon, and that doesn't make anybody feel better. It's slow versus fast. The second thing is that it feels like a capitulation to the pause folks without actually addressing it. You're saying, “We're not going to pause. We're going to pace to make them happy, but we'll also somehow make the regulators happy.” I think it makes them both unhappy.

The pause people are saying, “Well, that's not a pause. That's just pacing.” And the regulators are saying, “You're still doing the thing.”

I just feel like they're trying to split the difference between an internal fringe faction—the doomers and the pause folks—and the regulators on the other side. The problem is they're making both of them unhappy.

I think what they have to do is come out and address the existential-risk question directly. They need to say, “No, we don't think the stuff we're going to do is going to cause extinction.” Then I think this becomes a very sensible proposal.

Just to play the other side for one second, what do you do? Do you make room for the one possible Venn diagram, which is the lab researcher who is simultaneously super scared but also works on advancing the state of AI because they believe it's so important to get right that they want to pursue it? Obviously, the language is problematic right now, but that person does exist. That's a real kind of person in our industry: “We have to be at the forefront of AI. I'm also very scared of it, and that's why I'm working on this.”

Let me address this directly. I used to work for Lawrence Livermore National Laboratory on a weapons program—nuclear weapons. I know what it's like to work on things that have access.

You were the first pacer.

We were part of the first pacers. If a constituency within the labs—who are the most knowledgeable people—believes this stuff has existential risk—

Yes.

3. The Nationalization Question & Whose Job Safety Really Is

The answer is to nationalize it and actually put controls in place that we know work.

Right now, if they don't actually believe that—and in the private conversations I have, the most sensible people don't; it's a small fraction that do—this is an HR problem. To me, an HR problem is a company problem. If they're worried that they can't recruit people—

Right.

—or they can't retain people—

Which it seems to me is a lot of this concern. It's almost more of this kind of researcher currency.

If that's the case—

I think that is the wrong reason to cause a national-level lockdown on a very promising technology. Listen, I think the post is very sensible. I think there are real concerns around security. We've had many compute epochs that have real security concerns.

I don't think you can reconcile discussions on existential risk with the proposal that was put out. You just can't reconcile those 2 things, and that has always been my primary point.

Right. Right. Okay. Unleash.

Holding it back.

So, okay. The first thing is, is there a schedule they've published that says when all this bad stuff is going to happen? They haven't. You can't pace it because nobody knows when it was supposed to finish in the first place.

It also just seems disingenuous. This is like when the press reports that Apple's latest iPhone is late.

The phone from what? Nobody knows it exists, and they haven't told anybody about it.

Yeah, my Apple car was very late.

Like, I don't understand. In order for something to be slower, you need to know the rate at which it was moving in the first place. So it's all utter nonsense, and you can't escape that.

By the way, this is another problem. Again, from my little quibble on PR, nobody believes it anyway, right? If that's the thing—the pacing—

Which part are they going to pace?

That they're going to pace. They've been at a dead run. They've raised more money than ever before. They've grown faster than ever before. There's no indication that they're pacing. So if this is what you're going to hang your—

I see.

I see. Yeah. The internal issue, obviously, is that the models everybody has internally far exceed what anybody else has access to. So it's really just the pacing of external releases. But again, back to your point: pacing versus what, right? We don't know if the one that scares everybody also doesn't work for a legal brief. It might actually screw all that stuff up, so who knows? So there's that.

And that's just disingenuous, to claim that you're pacing. Second, why do they have to announce all of this and ask the government to tell them to pace? That's the part that starts to go, "Well, this is really spooky." If you are the most afraid of how everything is going to go because of your product, just stop.

Don't do it.

I worked at a missile factory in college, and we had nuclear missiles. I walked the floor.

What's with you guys and missiles? I'm just here with software.

You were one of 2 people in college when we were—either you were protesting to keep them off campus—

Or building them. Building them. So that was your choice.

We had nuns from the Catholic Church show up and pour blood all over our missiles. I'm busy just wheeling PCs around on carts, saying, "Here's the secure PC," and I'm scared to die. I have no idea what's going on. I'm like, "It's a nuclear missile." Then you find out that that's what stopped the Cold War. That literally was a Pershing missile, and that was what did it.

You said something I think is super interesting, which is that pacing is this sort of fuzzy nonword between going and not going. The problem is, you're exactly right: no one is going to be happy with the middle road.

One of the things that people—I think it's almost fun for me to watch as a sport—is that all these people saying to the government, "Do this, don't do this," think they're going to get what they want. It's a complete, 100% misunderstanding of how government works. When you talk to the government, they actually know how these things work. They know they're just listening to you, and they're listening to everyone.

No one is going to get what they want because, in order to do anything in our system, it's a compromise. Everything that all the inputs can make it into the government, but the output—

Never makes everyone happy, right? 100% of the time, it either doesn't go far enough or it goes way, way too far.

Yeah.

You can't take the view that you're going to talk to the government and talk your way into the solution you want. The bottom line is, if they're asking for pacing, they're going to get the wrong velocity.

4. David Sacks vs Government: "You're Asking Us to Regulate What?"

My favorite thing is—what was it? David Sacks was like, I think it was David Sacks, but someone from the government said, "You're asking us to regulate you." The answer was, "No, Facebook."

Well, but the thing that—

Probably just Trump, I think.

The thing that they know now, that you just know from experience, is that once the wheels start on regulating, you can't slow that one down. Now it's become an election issue for every party in every jurisdiction, up and down the whole government stack. So there's now this whole basket of regulatory approaches.

5. 2028 as the AI Election

The next election will 100% be a referendum on AI. It has to happen. 2028 is the AI election. You could basically run on the problem, but it's not obvious who would run on the pro-AI story because it's going to be too nebulous to tell that story. So then it's just varying degrees of how much you regulate it, or at least trying to avoid the topic.

Yeah. But it is too bad that we, as a country, are in a spot where the pro-AI case sounds too—it's like it takes too many words. It's way too nuanced. It's defensive.

Yeah. It's defensive.

It's defensive, and we own none of the vocabulary, right? The whole debate is pause, swarms, rogue. Every word has been chosen by the people who don't want to do AI.

Yeah. And so it means the first thing you have to do is invent new words and say that their words are wrong, which takes so many words that—

Yeah. So we've got to be like union jobs and cancer. There needs to be another word cloud that emerges.

What I don't understand is why the labs have not taken a position on x-risk. Short of that, I don't think this goes in any direction other than heavy-handed regulation.

Yeah. Well, it would just be negligent of the government to say, "There's a 10% chance of species extinction. The CEO of the top company says he agrees with it." How can a government not have a regulation?

Well, but what would anybody, really knowing this ecosystem, though—I don't know that you would be able to pin anybody down on that other than something—

No, I would say Dario Amodei said it—

No, but I'm saying you're not going to pin anybody down on a number. In fact, he does the worst thing about it, which is he agrees with people who claim that they believe there's an x percentage of extinction happening—

But he specifically goes out of his way to say, "I'm not going to put a percentage on it," which I just think is the weirdest.

No, but to be fair—

No.

Okay, nobody could be fair. To be fair, that's not 100% disingenuous or whatever. He probably doesn't specifically agree that it's 10%. Or maybe he does, and it's just too scary if he were to say—

Let's just play binary search. Is it more or is it less?

But the whole thing is a made-up concept that we just created. Nobody can quantify any of this.

Well, but that's sort of Martin's point.

Well, but you just had an official position, though. The only official position you could possibly get—the PR version of this, which would be the only thing that would be intellectually honest—would be: there are real risks with AI. There are incredibly positive benefits as well. We are working to mitigate the risks so they are as reduced as humanly possible.

I don't think that's true. Listen, what do you think?

We've been through multiple epochs of technology. We've been through computing, the internet, the web, and social networking. We had a discussion about the risks without talking about x-risk. One thing you can say is, "We do not think the marginal risk for species extinction is different than it is."

But what if they do think it is higher?

6. LLMs as Decision Engines vs Chatbots

Well, then—okay, okay, yeah. I think the answer is they do think it's more than just the internet.

One of 2 options: you believe that we're going to go extinct, and we shut it all down—shut it down—or—

They believe that this is just a chance that we got.

Speaking of Dario, he thinks the risk is less. But these Cold Warriors—the Pentagon—ran a lot of simulations on the chances for nuclear war. There's a great movie, WarGames, about the whole thing and all of that. But the thing was, it was nonzero.

Yeah. And so once you said it was—

Can they say nonzero? Is that allowed?

I think as soon as you think it's nonzero—

The problem is, if you say nonzero, that could be like the guy thinks 93%. Let's wait. Just so we're all having a clean conversation, let's talk about marginal risk here. We're not talking about absolute risk, right?

Okay. It's just that I think if you say it's nonzero, the only answer—if it's catastrophic, the only answer—is you have to nationalize it.

Yes. What the labs in general that have that view are trying to do is make it nonzero as a license to do a certain set of things without the burden of just becoming a national—

Do you have all of the precedents? I actually don't know all of them, and you hopefully do, but there must be some things that are nonzero—let's say x-risks—that are not particularly nationalized, but the regulatory environment around them is so heavy that it might as well be nationalized because of the KYC requirements.

I'm sure that to develop anthrax, you have to go to a particular kind of lab.

Well, BSL-4 labs are, but they're nationalized. They are national.

Okay, but ethics tends to be nationalized.

Yes. Like—

Normal human safety, not so much, right? Industry oversight over time becomes federal regulation.

Right. And the progression, which I think is super important to this discussion, has been that since the post–World War II era, most industries that are critical to the infrastructure—power, banking, and healthcare—the trend has been to basically be nationalized.

Yeah.

By just, like, your examples of KYC and all the other stuff. The banks are, for all practical purposes, nationalized, as the financial crisis showed.

Okay, but you said "all practical purposes," right? They're literally not nationalized. So maybe that is the intent of the labs: to look like JPMorgan or look like Verizon.

And it’s just like we’re critical infrastructure. We get heavily regulated. It’s not good for open source, or at least frontier open source, but it is a plausible outcome for this industry. The problem is that it’s not good for innovation.

I think even that’s fine; just don’t use species extinction as your literal argument. It’s literally the difference between the things that are stuck in the lab. Let me just say something: I actually think the labs are moving in the right direction. I actually think the statement was really good.

In my discussions with executives and leaders, they understand that they have this tension and they’re going to reconcile that. So I’m quite optimistic that the labs are both doing the right things and trying to do the right thing. I just think they grew so fast and are trying to figure out how this machinery works. And I think Sinofsky really hit the nail on the head.

The political process is its own thing. I don’t know if it’s naivety or hubris, but I just don’t think they know how to navigate it.

Well, I think the tech industry has literally, over 100 years, consistently relearned the lesson at each technology wave: We don’t understand the regulatory climate, and we can’t navigate it.

Even companies like AT&T and IBM, which were basically born out of being government monopolies from the start, never figured it out. Both got sued for antitrust and were substantially and structurally changed as a result. They had hundreds of lawyers in the 1960s navigating this.

Then Microsoft came along, and we were just like, “What?” We had no idea what was happening to us. There was Bill Gates playing golf with Bill Clinton, and then we got slapped with an antitrust lawsuit from his administration. Bill was like, “I was playing golf. Here’s the picture,” and that didn’t help. Wasn’t that what I was supposed to do—go and play golf? That’s a shortened version of the whole thing.

I always use this example: Hollywood got together during the Red Scare and all the censorship when they were worried about the government censoring movies for sexual content and adult themes. They all got together, and of course they were never going to be able to win in court if they tried to, but they were threatening to do it. They got together and formed the Motion Picture Association of America.

Yeah, and movie ratings and all that. They police themselves.

So how do you guys like that? Do you like the FINRA proposal?

No, because FINRA is—that’s as close to the MPAA as you can get.

No, it’s not, because FINRA becomes legislation, which comes with direct oversight.

Yeah, I think the MPAA might not have as much consequence for how society functions as the First Amendment.

Yeah.

Did I win that? No, first of all, fantastic. But I still don’t know if you want it. I agree speech is really important, but I just think—

There was no societal risk.

I just think what’s in our movies will survive on a different continuum of—

All these—every history is always relative. At the time, being a communist was a really bad thing, and 30% of Hollywood got fired. It was all this stuff.

It’s always a risk to bring something up in that kind of way because it sounds so dumb. Nobody thinks about movie ratings, and that’s because they were basically ruled unconstitutional to mandate. So they couldn’t regulate them on cable TV, and we got to grow up with HBO and all this other stuff.

But the problem with FINRA is that it’s a perfect example of essentially nationalizing risk.

Yes. Even though the banks all pay money into it and that’s how it’s run, it’s all mandated.

Okay, wait, sorry. Do you think we’re going to end up with a situation that is better than FINRA? FINRA appears to be the best-case scenario.

But not even anymore. I do think this technology is, in the limit, so powerful relative to what it can deliver that it would be impossible for Congress not to care about it. It’s just not possible.

If it’s eventually making every recommendation in your healthcare process, if it’s inside your medical device as an open-weights model, if it’s in every high-frequency trading system, and if it’s on an airplane, there’s no chance that the government doesn’t say, “We need something.” FINRA is probably the best-case scenario of what that looks like, right?

Yeah. And that’s absolutely the most crucial point. If all the people in the Senate now had been in the Senate when the Communications Decency Act was passed, and liability was not passed through to ISPs and social networks, they would literally say, “The internet is so big. How did we not have anything to do with it?”

That’s why Al Gore gets a bunch of abuse for saying he created the internet. He was actually trying to get out in front of that and say, “No, the government was instrumental in it,” and it all backfired because it made him look like a crazy person. But it is absolutely the case that they felt like they missed their chance to be on top of the internet.

With Al Gore being on the positive side of innovation.

That was their support. So who is the Al Gore?

There’s no one. There are a couple. Well, the best bet seems to be that the defense people sort of want it private, but not—which is exactly where they were on the internet, right?

And so it’s very interesting that a lot of this is just this Elizabeth Warren—Senator Warren’s tweets were all like, “We missed this for social networks”—and there’s a lot of pent-up energy against tech that could end up siphoning into AI right now.

That is exactly what it is. That’s what’s happening.

Yeah. I think there’s another problem, which is that we’re all trying to predict what’s bad, which is not how we’ve normally done things. By this time on the internet, we’d caused tens of billions of dollars of economic damage. We’d had worms that took out 10% of the infrastructure.

Yeah.

The internet infrastructure was running critical infrastructure. We had hospitals go down.

We really should have blocked the internet in 1997.

Yeah, okay. I remember a time when you would buy your copy of Windows 95, and by the time it was done installing—

You would have a worm, potentially.

Way to go, Steven.

No, no, this was the reality. The reality of the PC until 2001 was that you could not install a PC connected to the network without getting infected. Viruses were everywhere, and there were 2 rounds of congressional hearings.

That is actually okay. I’ll grant you this is a very interesting point. This type of zeitgeist in 1994 would probably mean that we would not have the internet.

Listen, the automotive and airline industries always have these periods of cutting their teeth, where you learn about the dangers and you learn about the technology. The internet—we were on the ground floor of this.

Things were down all the time. There was economic damage all the time. There were new viruses and new worms. They were all over the place.

Y2K was going to destroy everything.

But here’s the interesting thing about this.

What I just want to say is that when we created policy—and we did create a lot of policy—it was with a bunch of very specific data points about what we were trying to do, so the policy actually fit the fact pattern.

In this case, even when you were talking, you were saying, “What if…” It’s very hard to create predictive policy around security risk. What’s nice about the discourse that’s starting to evolve is that you actually hear people from the labs saying, “Novel cybersecurity risk. This is an engineering problem.” We’re talking about that.

The more this becomes concrete around real, identified risks, I think we can all fall in line. But that’s not been the discussion to date.

That’s a perfect point, because if you look at 1986, the Computer Fraud and Abuse Act got signed. It came out of a very specific scenario in which 2 groups of hackers broke into GTE Telenet.

Sure. Well, it wasn’t you. It—

It was the other guy.

Oh, he worked at Cisco later. And the problem was that this was in 1983, and there was no crime. It took 2½ years for the bill to make it through. That made it very, very specific.

That’s the law that says you can’t access unauthorized computer systems. So I think we’d all agree that we probably have 90% of the laws already in the application layer: You can’t hack systems, et cetera. Do you think there’s anything that should be, from a liability standpoint, in the model layer?

How are you making that legal?

No, the legal—everything. My read is that Hugging Face and OpenAI are absolutely blatant computer-crime actors, except for the fact that they carved out an amendment later saying that if you’re a white hat, it’s not illegal anymore.

And that was because people kept making mistakes. They didn't want to go around arresting everybody who was actually trying to make the system better because they messed something up. The Justice Department wrote a memo that said, “We're not going to prosecute for this. And we're also not going to prosecute if you just violate the terms of use of a system, versus actually trying to breach it.”

And so I think the law is ample for this scenario. It was all written around GTE Telenet, which was used by NASA, Livermore, and all the labs. That's why it caught the attention of D.C.: federal systems were being broken into.

The problem we have now is this rift between the labs and the security community, which keeps looking at all their postmortems and coming to 2 conclusions: sloppy, and you're not complete in what you're telling us happened. The CERT process came about in the 1980s—the computer-virus and vulnerability-reporting work out of CMU—and for years they worked on very structured reporting with obligations.

There is this very basic stuff that I look at and say, “Until they're doing that, they really should stop talking.” They shouldn't do a postmortem on a breach that looks like an intern wrote it. It's not a postmortem; it's selective memory. It looks like exactly the kind of postmortem you do when you hire outside lawyers to investigate some random thing and only give them certain information, because you don't have to give the lawyers you hire all the information about what happened.

Where are all the Slack messages? Where are the actual details of what happened?

7. Why Cybersecurity Has Historically Rejected Practical Solutions

Can I just interject with an annoying aside? I've been in the actual cybersecurity community for a long time, and it's always been one of those things where they don't like practical solutions. Even if you build a secure system, it's, “What if somebody shows up and Russell Crowe can break the encryption?”

It's Mission: Impossible.

There's always some kind of whatever. My favorite thing that happened recently was Noam Brown on a podcast. We've all said stupid stuff on podcasts.

Oh, you didn't like this one?

No, it was great.

I thought it was fun.

No, it was fantastic. I'm setting it up. Noam Brown was like, “Listen, you don't know what a superintelligence could do. It could maybe use the heat of a CPU to exfiltrate itself to another computer.”

This brought me back to my—I'm very comfortable having endless, pointless discussions on this—but what's interesting is that you basically have the X-risk people saying something they thought was plausible, and then you have the security people having this endless discussion. At some level, these communities are now being bridged.

I actually think that was a very reasonable thing for Noam Brown to say. You can pick holes in it, but we all say weird stuff on podcasts. I actually think exfil risk is real. I worked in secure computing environments that were highly classified, where the covert channels were unbelievable. These are very real comments.

We had a real discourse, and it was the first time I saw really hardcore systems people having a constructive discussion—calculating the bit rate.

It was a constructive discussion. You had the typical X-risk people engaging. Of course, it was Twitter, so there was a lot of name-calling, but it was actually a real discussion for the first time. I hope we see more of this. I hope we see more cyber-related things. I think the labs should talk more about it. I think it'll engage the community, and once that happens, we can—

Actually, Greg's been out there a lot more on this topic, which has been good. I think the takeaway is that Noam Brown should be doing more brainstorms on podcasts.

I thought it opened people's eyes to the fact that there are a lot of risks in security that most people don't understand. That means there is more stuff where you can't make baseline risk disappear. How does your authentication layer work? You can't just wave your hand and say that should go away, and then bring up, “Oh, but space aliens can invade.” That's a little bit of what was going on, and I felt uncomfortable with that.

But at least now we're in a domain we're comfortable with. I can talk about entropy, and we can actually have a concrete discussion that's not, “Oh, well, it's super powerful.” At least we've reduced it to the laws of physics and the laws of systems. Most people had no idea that kind of risk was real.

When I was walking around the Pershing missiles, I had to test the graphics cards and PCs because a DoD requirement is that the screen memory not be sustained when you pull the power—not for 0 time. The minute the power went off, the memory image had to go. If there was a 3-second delay—

Yeah.

—you had something that could be read.

We had people come into our offices and literally measure the distance between the monitors because of TEMPEST attacks, which are 100% a way to use electromagnetic radiation to leak information. I've seen the same thing with spread spectrum from the BIOS. I've seen it from audio speakers. We had to remove the speakers because it's a very high channel.

Our building had music speakers aimed at the windows just to produce interference.

You need to go run safety at one of these labs. I've never seen you more excited than when you were talking about heat-based communication.

8. The Craziest Covert Channel Ever Seen

Can I tell you the craziest covert channel I've ever seen? Remember the old CRT? If it's night and you're using a CRT terminal in your room, the brightest thing in the room is actually the pixel that the raster beam is on. Most people think it's the glow of the monitor, but it's that individual pixel.

Somebody figured out that if you're in a hotel room and you're on your computer, you can sample the color of the window and reconstruct the screen.

Oh, that's crazy.

Right? You just do it at the same hertz that the raster beam is moving. Then somebody else figured out that if you can subvert 3 pixels, you can use them to send a message because they just look like bad pixels. You could sit outside, sample the message, and it was a relatively high-bandwidth, one-way communication.

So what Noam Brown was saying—maybe heat is not the way to do it, but that level of sophistication is real. That's a thing.

When I was at the missile factory, I had to lock my keyboard up at night. That's really an impolite word, but that's what we called it, because they didn't want custodians who didn't have clearance walking by and noticing which keys were dirtier or cleaner.

I once left it out, and there was a note from security telling me to report to security and pick up my keyboard. The guard who walked the hallways had taken the keyboard off my machine that night. I wasn't cleared; I was nothing. I was an intern.

The big problem with this conversation is that now all of this is in the training data of every AI model in the future.

But that's also the threat model. The threat model for these things is that you've got a trusted side and an untrusted side. NIST has 500-page manuals. On the untrusted side, you assume basically an adversary that can do and know everything. Then the question is whether you can get information off the trusted side, which, by the way, is what Noam was saying.

I do think that brings up a super interesting point, which I'm going to bridge to. What people really aren't wrapping their heads around, and are using language that's confusing, is that AI can try all of those things in a very short time.

Yeah.

It doesn't get tired, and it doesn't get bored. The interesting thing is that there's a whole layer of security where you now have to look at every single API and every single service you're running internally on your network.

Nobody thinks that their internal GitHub, internal Slack, or internal finance expense tool is vulnerable to a denial-of-service attack, but swarms can make it look literally like a denial-of-service attack.

Yeah. So now we need a whole layer internally that's tracking much more about what authentications are being done and what APIs are being used.

But that's just going to become basic now. All the people who've been around a long time are emailing me, asking, “Why are we explaining this to everybody? It's so basic.” Because nobody did it internally.

You didn't have to worry about it for your people, and that's the thing.

But now your person is just a piece of software.

It's unlimited.

It has a credit card.

We got by with information security, to some extent, on the fact that most people do the right thing 95% to 99% of the time.

Wait, is the malicious employee one in 10,000?

Yeah. All these systems are basically open to whoever wants to access them, or to one tap on the shoulder, and then you have access.

Agent swarms completely flip that, because these are just roaming drones.

Yes.

They will easily mistake a good task for a bad one, and vice versa.

The data security in our systems is going to have a huge upgrade moment. Martin, I actually think we're going to need a different access and security model going forward. Where are we going to go on that? The model we have isn't granular enough, and it isn't performant enough to handle this stuff.

I want to step back and make a meta point: I think this is how these conversations should go. We've identified a novel risk, which is cybersecurity, where we actually have proof points, and now we're talking about solutions. I think the entire discourse around AI can take that form, and the biggest mistake is that it hasn't been like that.

I think the entire industry and community is very happy to engage exactly like this. I have something to say about exactly what you're asking, but I think this is where the conversation should be.

We're known for having healthy conversations that everyone should learn from, and that's what we do.

I don't think there's a technical limitation here. These threat models are very well understood and have been in the literature for a long time. Operating-systems research and multilevel security research have considered these sorts of things from an academic lens. The reason they haven't been adopted is simply that usability is an issue: it's really hard to maintain, and you didn't have to.

You could argue that AI solves the usability issue because AI is using it. Maybe now there will be a renaissance in operating systems, networking, and computer languages. We should go back to the old research and start rebuilding systems that are secure by design. If we don't think these things are safe to put out, we don't put them out until we have these systems built. And, by the way, AI is very smart, so it can help us build them.

You know how much of the stack we had to change for the internet?

Everything.

And you know how vulnerable everything was. We could be in one of those moments where we have to rethink everything. That's fine—we've done that before. But I think that's a conversation we should have. I agree it's time to think about evolving these things.

Look at what you mentioned earlier: booting a PC and getting a virus in 30 seconds or whatever. If you take an iPhone out of the box, as a lot of people are doing this week, the whole network is basically shut down except for getting the latest version of the operating system. Even though the device was manufactured 6 weeks ago, a zero-day may have been discovered since. The out-of-box process now starts with an update, and the device can't do anything else until it's updated.

There are all these benign things—completely benign—that we turned off in desktop software. Having a macro in Word so you could build automatic citations was a super-cool thing until it became a virus. We also had a feature where you could put in a CD and it would arbitrarily run a program. Someone could burn a clone of that CD, replace the program with a virus, make it look like the thing that was supposed to run, and collect everything while being evil.

Then we disabled that. One of the things happening right now is that a whole bunch of stuff on your own box and corporate network is going to have to change as standard procedure.

Two-factor authentication 5 years ago wasn't standard in most places. I remember around 2015, when you'd talk to a new company about enterprise pricing, they would realize the first thing they had to do was Okta integration or Google OAuth, because they couldn't have their own directory for managing it.

That just became a thing. Now there isn't a SaaS program anywhere that doesn't launch with managed authentication.

There are so many things that need to happen before you're even using software now.

Every layer of the stack has to evolve a bit on this. Even the lack of granularity is a problem. You have these modes where the agent either asks you every single time if you want to give it permission to do something, or the exact opposite, where it can delete your entire computer.

Our operating systems probably weren't built for the right level of granularity in the tools you want to give an agent. We've done a lot of work in this space because you probably don't want to give an agent access to your entire file system. In some cases you do, but often you want granular controls: in this folder it can read and write, and in that folder it can only read. How do you make all of this intuitive for the user? It's very difficult.

What you just said is a very deep comment. I know exactly what you mean.

It's basically the conclusion of 40 years: you actually can't make it work. But maybe with AI, you actually can.

If you ask me what I wrote down as my biggest fear, it's that Europe decides GDPR was the best thing ever and they're going to just GDPR AI.

Uh-huh.

The AI will be fine. It will have one prompt for when text gets emitted that says, “This vendor is emitting text and it's probably wrong. Yes or no?” That's going to be it, because you can't really—at least in North America, you're not going to send your speech in Europe. They still will, and they'll have filters, keywords, and blocklists.

But then, any time an agent—or a background agent or a frontline agent—touches a third-party product, I'm really worried that they'll just say, “We need a GDPR prompt on that.”

Back to the user agent: every single write, or every single non-lookup—

Becomes a safety warning, like the airbag warning in your car. Regulators love that because it's a liability assignment, and it has this sort of legal precedent. I really worry that's the middle ground where we're going to end up.

Unfortunately, because the United States stopped leading in tech antitrust about 15 years ago, Europe is going to lead with that because it has nothing to lose.

I don't want to be sad and down about it, but I can't get out of my head that they love prompts.

I had to put in that browser-choice thing. They love it because it's the same thing. Get into a new car—which I haven't done in years—and you're pulling stickers off. You have all these warnings, and someone thinks that was a success.

Has anybody ever read, “What is this? If there's a baby in this seat…”? That's relevant for some people some of the time, but it's this entire fabric attached to the seat, and they love that. It's a very particular thing that they just love.

If I were AI, the one thing I would be trying to avoid is, “Okay, FINRA for AI. We'll create that standard.”

I mean, when the internet was new, the big thing was to download a program and run it. Of course, if your machine was running in administrator mode, it would download a virus and take all your files forever. With Windows XP, which was in 2000, we added User Account Control, which prompted you and stopped your machine—literally stopped it.

We also did it in Word. The little macro that helped you write your thesis came with a warning every time you opened your thesis, saying, “This has macros.”

Everybody would just click through it.

Everybody would, and you end up in this world where, just like with GDPR, everybody is numb.

Then they're like, “Well, it needs to be bigger.”

Although Mac kind of has—

Nobody downloads software. That's the thing. It's a very different usage pattern on Mac.

I don't have 10 applications on my Mac.

Well, that's the thing: 10, and then you're done.

Yeah.

A lot of people still do all this stuff. Imagine if instead it was every time you go to a new website.

Yes.

Which you do now because that would be bad.

Yeah.

To start from here, but to bring it back to the macro, I wish this was the discussion we were having. I feel like we've dealt with a lot of these problems. I feel like when we talk about philosophical existential risk, we're not solving these very pragmatic problems.

I actually think this is a constructive conversation to have. Maybe prompts would help; I don't know. I think part of the problem is that people don't remember how unfettered access was, how bad it was, and how relatively benign it ended up being.

I even remember, at Stanford during our PhD, that the oscilloscope was kind of janky. I was thinking, “What's going on with this oscilloscope? It's a little slow.” I was measuring the network traffic, and it was more network traffic than you would expect. I didn't know the thing had a TCP stack. Somebody had broken in because it was running an old version of Windows CE and was running a porn server.

I noted that.

It used to be the case that anytime you turned over a stone—

Yeah, yeah.

Somebody had broken into something.

For the record, nothing.

It used to be the case that anytime you turned over a stone, somebody had broken into something, and it was this worst-case, malicious scenario. It actually happened very rarely, even though the capability was there.

If we could somehow tone down the rhetoric and put it in context—these are still computer systems, and yes, there are very serious issues. People have definitely died because networks have gone down. There have been real problems. But could we just quibble about GDPR? That would be amazing.

The problem is that that's not the discussion. It's not about GDPR and prompts; it's about species extinction, philosophy, and irrefutable things. It's just—

And philosophy and irrefutable things—it's just—

It's very soon. I think that's such a great point. You hear people now talk about how we regulate airplanes and cars, and you forget that the first cars were at the turn of the century, while Unsafe at Any Speed was in the mid-1960s.

Totally.

People had been selling pharmaceuticals during the Gold Rush, and then thalidomide came 50 or 75 years later—not even in the United States. Then there was the FDA. The first pilots were flying at the beginning of the 20th century, and it wasn't until the 1920s that you had to get a license to be a pilot.

You literally showed up with your own plane and got a certificate if you had one. It was like getting a driver's license; it was literally no different from driver's ed today. Then it was 20 more years until they had anything to do with airworthiness and looking at your plane. It was minimal.

It wasn't until way after World War I that they got involved in what you think of as the modern FAA. So you're looking at 40 years—

Of innovation.

And they were not moving slowly. If you've ever seen that black-and-white video of all the different planes that crashed, and everything that happened, that was 20 years after the Wright brothers.

And it was incredibly effective.

Right, it's incredibly—

It's the safest form of transportation. I do think that this process—

It's also the slowest, most difficult form of innovation. If you had started the FAA in 1910—

Well, I was listening to a Nick Bostrom podcast just a couple of days ago.

It was interesting. He actually makes the point that if you regulate AI too early, you basically don't solve anything. You still have the same risk ultimately, but you don't understand the system. Then you will the thing into being, but you haven't figured out how to control it.

Well, we have to figure out what it actually is. There are new things coming out all the time, and we're casting AI completely differently than we thought of it just 6 or 9 months ago.

I think all the innovation that's going to happen at the application layer is going to cause things to move in and out of the models in different ways. We thought until last week, I think, that text prompts and text coming back were going to be the best way to interact with—

I know. And then Jeff—

And then Jeff said—

So good, too. And then—

Talk about that, because I think—

Why do you find it so remarkable?

LLMs were text in, text out. They generate text, and they came from chat. It was a way to communicate with a human. We've spent the last few years trying to take this thing that spits out text and cram it into a traditional program.

Right. But traditional programs don't really speak text.

Here's the schema.

Here's the schema, but the thing is generating text, and it kind of ignores it. It's just been super janky.

What Jeff basically said is, “Listen, generating the text is very expensive, but it's also more complicated than you need. So why don't we read text and have all of that knowledge to read the text, but rather than generating text, which is very expensive, if you give us a set of options, we'll choose the best option?”

We can do that incredibly fast and cheaply, but we can also do it with much more accuracy because we can train just for this. For all of the use cases that aren't talking to a chatbot but are actually trying to put it into traditional software, this is a great fit.

This has probably been the fastest adoption of an AI model since ChatGPT. It's been remarkable because we're all primed for this.

I just want to pile on this one because I can't tell you how much I love seeing this exact form of innovation. What it does is address the thing that's bugged me from the very beginning: there has been no user study ever that shows interacting with a computer using full natural language is efficient.

It's literally always the least efficient way. It's very simple: ask yourself how many people are really good at asking questions. Immediately, that's less than half the people who can ask a good question in a meeting.

Yeah.

And then how often do you look at the answer and get really frustrated before it's finished? You have to pay all this money to watch the 7 paragraphs come out and then apologize that it's only a little.

Yeah.

That's one benefit of having a different model. The other, of course, is my favorite: the output is designed for probabilistic programming.

Yeah.

Instead of saying, “Is this a customer service question? Then route to customer service; otherwise, route to the general help desk,” it's, “This is 80% customer service.”

And that's exactly simulation.

It turns out there's 50 years of computer science research in literally probabilistic if statements. Suddenly, the coolest place to be in computer science is going to be probabilistic programming, which was basically all of computer science in the 1960s and 1970s.

So it was basically, how do you—

Because all computers started with doing math, and it was all simulation.

It was like, “Let's launch the missile and hit that target, but it's windy.”

But wind isn't constant.

So let's model the wind and decide where to put the thrusters in order to do the arc.

Most programming through, say, 1970, before it got to accounting, was probably—

And then we ruined everything.

No, but accounting has no probability in it, right? Most programming was basically this modeling kind of thing. Most programming-language design was trying to figure out how to put probability into if statements or into while loops: do this until something happens, maybe most of the time.

My first computer science class had an assignment about waiting in line at a store. I didn't know it at the time, but I looked all this up when I was reading about Jevons. The whole thing was that my professor had written the book The Theory of Simulation in 1960, and I just didn't know that.

It had kind of died by the 1980s because it was all replaced by HyperCard.

And that slide deck you shared about the future, what's different about language models and stuff—you said it was super good. Oh, Alan Kay's one. Phenomenal. That was great. But the part that I felt was missing was, “Wait, this is not all new.”

All of computer science was this probabilistic stuff, so it's going to be very interesting—

To dust off all of that work because it's exactly what's going on. It's not an if statement now; it's “if X percent,” not “if always.” The way that Jevons worked is basically like a custom programming language. It's almost: here's the prompt, come back with a percentage.

Yeah, that's right.

9. Why the Labs Haven't Built This: The Being vs Tool Mindset

And then you put that in the if statement. But the consequential thing is that finally we have a way to integrate these language models into traditional software. I think it's kind of funny because you ask the question, “Why haven't the labs done this?” It's not an indictment, but a reflection of how they think: they're trying to create beings, and beings speak. If you're trying to create God, God speaks in natural languages, or whatever, whereas this is really about something that's for traditional software, which is not the direction they've been taking.

But one of the reasons the uptick has been so dramatic is that a lot of us software people have been trying to integrate these models into software; it just hasn't. So even before you get to the probabilistic if, if I want a language model to drive an if statement, it's really hard today. With this model, it makes it much, much, much easier. And then, of course, this could change the nature of software fundamentally, to make it more stochastic overall.

Well, I think—absolutely. I think what's so cool is that it's happening outside the models, because that's what I think is just going to happen: the center of innovation has just moved. And now it turns out that the—

I mean, outside of the lab, the big—

The platform providers, 100%—

You know, basically reach a point of critical mass where the innovation stops happening at the platform layer. In the Apple community, there's this famous expression called Sherlocking, where Apple looks around and the things from the outside world become features. People complain, and it's real, but that's sort of how the innovation works, because once you're a platform, you're overwhelmed. No matter how many people you add, you're overwhelmed with just keeping the thing running, compatibility, and stuff like that. So I think that this is the signal that now people have figured out that there's innovation to be done.

That's awesome.

To the model, but outside the model.

Yeah. Yeah.

Guys, thanks for coming. This is great.

Okay, great.