美国AI政策的当下现实：从“暂停AI”到“建设”

Now, we've been through all of these tech waves, and we've learned how to have this discussion in a way that, for U.S. interests, balances these 2 things. If we're going to make a departure from a posture that was developed over 40 years, we better have a pretty damn good reason.

Okay, so we're talking 1 or 2 weeks after the action plan has been announced. It looks like we've come a long way. Why don't we trace—you guys have been on the front lines for years now in this discourse, fighting to make this possible. Why don't we trace where we've been, so that we can understand how we got here and where we're going?

Under the Biden administration, we had the executive order, which was basically the opposite of what we're seeing today. It was trying to limit innovation and doing a bunch of fearmongering.

But to me, what was even more striking was not regulators being regulators. You'd expect that. But if you remember, this is why we got involved: you'd have these politicians making recommendations, which is fine. You'd expect that.

But nobody was saying anything. Academia was silent, the startups were silent, and, if anything, the technologists were kind of supporting it. We were in this super-backwards world where innovation was bad or dangerous, and we should regulate it, pause it. There was this discourse, and it was somewhat fueled by tech, as opposed to anybody going against it.

So I think today we should definitely talk about how the action plan is great, but we should also talk about how the entire industry has come around to say, “Listen, we need to keep these things in check. We need to be sensible and think about it.”

PauseAI—that was 2 years ago.

Remember the big—sort of—all the CEOs signed this petition.

Oh, yeah. I think that was the last AI Safety Summit, right? The one before Paris? There have been so many of these.

Yeah, I've lost track.

No, no, no. Remember Dan Hendrycks's organization? What was the California AI organization?

The Center for AI Safety.

Center for AI Safety. That's right. That's the nonprofit. And then they got all of these people to sign this list: “We need to worry about the existential risk of AI.” That was the mood. It was almost like, can I just do something by contrast, right?

I was there during the early days of the web and the internet, and at that time, you actually had examples of this stuff being dangerous. Robert Morris let loose the Morris worm; it took down critical infrastructure. We had new types of attacks—we had viruses, worms, and attacks on critical infrastructure. We actually had a different doctrine for the nation: the more we get on the internet, the more vulnerable we are.

So instead of mutually assured destruction, we had this notion of asymmetry. There were all these great examples of why we should be concerned. What did everybody else do? Pedal to the metal and invest more in technology. This is great. We still wanted the internet. We wanted to be the best, we wanted to build it out, and the startups were all over it.

Coming into AI 2 years ago, it was the opposite. There were concerns with new technology, which you always have, but there were very few voices saying, “Actually, it's really important that we invest in this stuff.” To me, the bigger change is this larger cultural change.

I think that's right. There was a moment last summer when somebody sent you and me a link to SB 1047, and I remember Marty and I reacting, like, “There's no way this is going to get any steam.” What was absurd to us was that it made it through the Assembly and the Senate, and it was on its way to a final vote. It would have become law with 1 signature from the governor. Wow.

And I think there was this escalation where I realized something. My view is that technologists like technology and politicians like policy. We pretend these 2 things are different worlds, and as long as those 2 worlds don't collide, the engineers get to build interesting tech, there's no sort of self-own too early in the process, and we generally trust our policymakers.

That changed completely last summer. It was a really weird cultural shift: a lot of the policymakers who were quite open about not knowing much about the technology, because it was moving so fast, still felt like something had to be done. Therefore, this is something; therefore, it must be good. SB 1047 was, I think, the most egregious example of this being adversarial.

But that culture shift was from one posture to another.

“Let's let the tech mature and then decide how to regulate it later” became “Let's try to regulate it in its infancy.” That was, in my view, a massive shift.

Let's just talk about how bad it got. You had VCs—their entire job is investing in tech—talking against open source. You had people like Vinod and Founders Fund saying, “Open-source AI is dangerous; it gives China the advantage.”

There was some sort of prognostication that if we didn't open-source AI, the Chinese would somehow forget math and not be able to create models. Then you fast-forward 1 year, and they've got the best models by far, and we're way behind.

It was the people who were supposed to be protecting the U.S. innovation brain trust who were somehow on the side of “Let's slow it down.” I think that now there's this realization—

Actually, China is really good at creating models, and they've done a great job.

We've kind of hamstrung ourselves from whatever discussion we were having, which I think you're right about. It's good to be concerned about the dangers and job risks, but it has to be a wholesome discussion. You need both sides. When we jumped in, it didn't feel wholesome at all. One side was dominant, and there was almost no one on the pro-innovation, pro-open-source side.

I just think it didn't feel grounded in empirics.

Well, certainly not from empirical evidence.

So what is the steelman of the critique of open source that they were making a couple of years ago?

The argument was that this is like a nuclear weapon. Would you open-source your nuclear weapon plans? Would you open-source your F-16 plans?

The idea was that somehow this was like a weapon. Nuclear weapons are not dual-use. Nuclear energy is dual-use, right? An F-16 is not dual-use. A jet engine is dual-use. But a lot of the analogies that were used at the time were clearly weapons. They would say, “Listen, these things are incredibly dangerous. Would you open-source the plans for an F-16?”

And then the other side, which slowly decided, “This conversation is ridiculous. We have to go ahead and set up the argument,” said, “No, you would not do this for an F-16 because that is a fighter jet. However, a lot of the technologies used to build it—yes, this is fundamental.”

It's not like people aren't going to figure it out anyway, and we need to be the leader, just like we were the leader in nuclear technology. Historically, when nuclear technology came out, we invested incredibly heavily in it. The things that we thought were proximal to weapons, of course, we made sensitive. But all the universities were involved; the entire country had the discourse. That just wasn't what was happening.

I think that's true. They were basically saying there was a substantive argument against open source and an atmospheric one. The substantive one was the one Martin mentioned: the technology was being confused with the applications.

Right. All the worst-case outcomes of the applications or misuses were then being confused with the technology.

But they were also theoretical. It's even worse than that. It was like, “You're right in what you're saying, but this could potentially create bioweapons.” We got a bioweapon expert, and he said, “Well, not really. The difference between a model and Google is almost nothing.” But that was used as this straw-person argument.

Then there was the argument that it could hack into a whole bunch of stuff. Nobody had ever done it before, but it was theoretical. So these were theoretical arguments that were very specific—

Right.

—versus a broad technology.

That was one argument, and then the atmospherics were that there was a famous former CEO who went in front of Congress and literally testified that the U.S. was years ahead of China. Since these were nuclear weapons, the misuses were being confused with the technology, and we were so far ahead, the argument was: “Let's lock it down so we can maintain that lead, and therefore our adversaries will never get their hands on it.”

Both of those arguments were fundamentally wrong for the reason Martin said: substantively, AI was not introducing marginal risk.

Well, at least not identified at the time. You would go to Dawn Song, who is a safety researcher and a MacArthur genius fellow at Berkeley, and you'd say, “What are the marginal risks of AI?” She'd say, “Great question. We should research that.” The world expert on this question was like, “This is very important, but it's an open research question.”

Yeah, so no empirical evidence at the time that AI was creating net-new marginal risks, and just factual inaccuracies that we were ahead of China. If you just paid attention to what was happening, DeepSeek had already started to publish a fantastic set of papers, including DeepSeekMath-V2, which came out last summer, and we were like, “Okay, obviously these guys are clearly close to the frontier. They're not years behind.”

And so when DeepSeek-R1 came out earlier this year, a lot of Washington was shocked: “Oh, my God, how did these folks catch up? They must have stolen our weights.” It's like, no, actually, it's not that hard to distill from the outputs of our labs. Have you actually looked at the author list of any paper in AI? Where do you think these people come from?

So I think those 2 things were—I felt like we were being gaslit constantly, because both the content and the atmospherics were just wrong.

Maybe one question for the smartest or most sober people who were against it is: Maybe they were asking where the burden of proof should be, because it's hard to prove that there is risk, but it's also hard to prove that there isn't risk. And so there's a question of what's riskier: Is it riskier to just go full steam ahead, or is it riskier to slow down until we better understand these models, interpretability, and so on?

I think it's really important to ground these hypothetical discussions in what we've learned as an industry. The discourse around tech safety has been around for 40 years, and we went through it with computers. Remember when we were like, “Okay, Saddam Hussein shouldn't have PlayStations because you can use GPUs to simulate nuclear weapons”? That was actually a pretty robust and real discussion.

But that did not stop us from having other people create chips or video games, right? I mean, we went through the internet, cloud, and mobile. And so we've been through all of these tech waves, and we've learned how to have this discussion in a way that, for the United States' interests, balances these 2 things.

We've had areas that were very sensitive to national governments. Think about Huawei and Cisco, for example. We as a nation did start to put in import and export restrictions as a result.

And so I just feel these almost platonic, academic questions like the one that you just posed aren't rooted in 40 years of learning. So all I ask is, if we're going to make a departure from a posture that was developed over 40 years, we better have a pretty damn good reason. And if we don't have a good reason, then I think we should probably learn from that experience.

Yeah. I think extraordinary claims require extraordinary evidence, and so the burden of proof should be on the party making the extraordinary claims. If there's a party who's going to show up and say, “These are like nukes. AI models are like nukes, and California should start imposing downstream liability on open-source developers for open-sourcing the weights,” that's a pretty high claim to make.

And so you should have exceptional proof if you want to change the dominant status quo. The status quo is that you do not hold scientists liable for downstream uses of their technology. That's absurd. That's a great way to shut down the entire innovation ecosystem and start throwing literally researchers in jail.

We don't want that. We want them to be trying to push the frontier forward. And I just don't think that the tall claims were being followed up by tall proof.

And when we're talking about open source, are we all talking about the same thing? Meaning, are there degrees of open source, or is it just binary?

Open weights, I think, was the primary contention. If somebody put out the weights of a model and a bad guy took those weights, fine-tuned them, and did something really terrible 2 years later, the SB 1047 regime proposed that the original developer of the weights they put out, basically as free information, should be held liable, which was absurd.

Right.

Right. So I think—

I just want to make sure we're very clear, because people jump on top of these things. What he's saying is correct. So basically, if the weights were over a certain size and there was a mass-casualty event—

I think “catastrophic harm” was the word used.

No, it was “mass casualty.” There were so many versions that I don't know which version, but I remember we actually looked it up. The legal definition was 3 or more people were killed, or the medical system was overwhelmed. There were actually precedents of this, including a car crash.

Right, right. And there were actually precedents of this happening in rural areas, which basically don't have any sort of capacity. And so, basically, it would move the conversation to the courts and outside of policy, which is—again, historically, we've taken a policy position on these things that follows precedents we understand, to make sure that we don't introduce externalities. For example, allowing China to race ahead with open source, which has happened.

And the key thing is: By moving it to the courts, one could argue, “Oh, sure, it's moving to the courts. That means it's open for debate. It's not clear that open weights are going to be regulated with liability.” The point is that creates a chilling effect. The chilling effect is the idea that when our best talent is considering—

I could be sued. I'm a random kid in Arkansas developing something. I don't want to be in a world where—

It can be resolved in the courts, right? I can't even afford it, whatever it is. And in a situation where you have an entire nation-state-backed entity like China actually doing the opposite of a chilling effect—encouraging a race to the frontier—why on earth would we want that?

There's this meme of a guy on a bike: He picks up a stick and puts it into his front wheel. That's the effect of a chill. That is what a chilling effect is, right, at a time when your primary adversary is racing—

So let's trace how the conversation has changed, because we don't see Vinod tweeting about open source anymore. Obviously, he changed his tune, especially right now. Is it really just DeepSeek? Is that it, or how do you trace how the sentiment shifted on open source?

Let's go through a few theories. I'm not really sure what happened. I almost felt like it was culturally in vogue to be a thought leader on the negative externalities of tech. It kind of started with Bostrom, but it was picked up by Elon. It was picked up by Dustin Moskovitz and Reid Hoffman—I mean, a bunch of intellectuals that we all respect and still do. They're really the titans of our industry in our era.

They were asking these very interesting intellectual questions, like, “Do we live in a simulation?” and “What happens if AI can recursively self-improve?” That actually created whole cultures and online social discourse around this stuff. And so, to no small part, that became a bit of a runaway train, and it's just catnip to policymakers.

I think part of it is that people didn't really realize this had become so real because, of course—

GPT-2 came out, and then GPT-3 came out, and people thought, “Oh, this stuff's amazing,” and somehow it got conflated. So I think part of it is just path dependency on where we came from, which is kind of the legacy of Bostrom. I think that was part of it.

I think the ungenerous approach would be that a lot of the discourse is awesome, but a lot of the people pushing the discourse were first-order thinkers. They weren't doing the math: Wait a minute. If policymakers who have no background in the frontier—which, by the way, nobody does, because this space is only 3 or 4 years old—start to take discourse as canon, which is a big difference, then what happens? What are the second- and third-order effects?

And the second- and third-order effects are that you start making laws that are really hard to undo and start mistaking interesting thought experiments as the basis for policy. And once that happens, those of us who've looked at law—law is basically code. Code is hard to refactor; law is impossible to refactor.

And so I think the second- and third-order effects were that a lot of well-intentioned folks—for example, in the existential-risk community—were saying, “Look, if you're intellectually honest about the rate of progress of AI, it's not crazy to say that there are some existential risks in the technology. It's nonzero.” Sure, yes, that is true.

But then to say that that threshold is high enough to start introducing sweeping changes in regulation to the way we create technology, I don't think a lot of the early proponents of that technology realized they would do that. In fact, I think Jack Clark, who runs policy for Anthropic, literally tweeted toward the end of the SB 1047 saga, “I guess we should have—we didn't realize the impact of how far this could have gone.”

And I think, to those of us who had interacted with D.C. and regulation before, the second- and third-order effects were much more discernible, or legible.

And then I think what DeepSeek did was just make it super legible to everybody else.

I think DeepSeek was the catalyst.

But it wasn't like there was a step. It didn't change the reality that the second- and third-order effects of policymakers confusing discourse for fact were always going to be terrible.

Yeah.

I just think it brought to light something a lot of us were already seeing, which is that we're in a race with adversaries, and that should be the calculus we should be working backward from. There was always this prevailing view, which has turned out to be so wrong, from really well-intentioned people: It's going to be regulated anyway; if it looks like we're self-policing, we can dictate how that happens, right?

And unfortunately, that just turned out not to be true, because whatever self-policing we seemed to be doing scared the shit out of people, and they ended up— Then, of course, I would say very opportunistic elements in tech decided to use that for whatever agenda they had, and so it kind of got away from us.

Mark had this sort of Baptists-and-bootleggers framing.

Yes, I was going to say exactly that.

True believers, and then people who use that thinking to support their own ends. And it seems like that's changed, even just on the company level.

But the reality is, I think the majority of people are neither.

Yeah, the majority of people are pragmatists.

They're not trying to take advantage of the system. They think, “Well, maybe if we have this discourse, it's an honest discourse, and then we'll self-police.”

I just feel like the silent majority was not part of the discussion. Maybe the biggest change now is that those people are there: the founders are there, academia is there, and VCs are there. Now, the people who are not either Baptists or bootleggers are driving the discussion, which, independent of the action plan itself, I feel puts us in a much better position. At the time, there was none.

Right.

And I think, to move to the action plan, if you read the first page, what a marked shift it is—the fact that the co-authors include technologists.

Right. I think that was the core problem: DC is a self-contained system, and the Valley is a self-contained system. A lot of people here were assuming best intentions over there, and vice versa.

What happened is that a few bad actors essentially used that arbitrage opportunity to represent Silicon Valley's views incorrectly in DC. When we saw some of the legislation, we had policymakers calling us up and saying, “Wait, you guys aren't happy with SB 1047, but the other tech people were calling us and saying you'd love more of this kind of regulation.” We said, “What other tech people?”

It turns out we're not one homogeneous group. Little Tech is extraordinarily different from big tech, which is extraordinarily different from the academic communities. I think one of the things we had to contend with was that we used to be one shared culture, and then, when tech grew, we actually—

There are some major differences in the Valley, at least, between parties. We're not one tech ecosystem anymore. We have different interests, and DC hadn't updated that. I think what's amazing about the action plan is that it's written by people who have bridged both.

It has enough representation across the 4 or 5 different subcultures within tech that have different interests.

Great.

I think that's new.

Yeah. Yeah.

Going back to open source, why don't you talk a little bit about how different companies have thought about it? From a business-strategy perspective, maybe we saw Meta with perhaps the first big open-source push. OpenAI has evolved there, too. I've seen even Anthropic seems to be evolving its dialogue a little bit. How should we think about open source as a business strategy? What's changed here, and why?

Oh, look, I don't think this is actually playing out along the same trend lines as all previous computing infrastructure: databases, analytics, operating systems like Linux. The way it works is that the closed-source pioneers are at the frontier of capabilities. They introduce new use cases, and enterprises never know how to consume that technology. When they do eventually figure out that they want cheaper, faster, more control, they need somebody like Red Hat to introduce them and provide solutions, services, packaging, deployment engineering, and all of that around it.

Which is why the arc generally in enterprise infrastructure has been that closed source wins in applications, and open source tends to do really well in infrastructure, especially with large government customers and regulated industries where there are a bunch of security requirements, things need to run on-premises, and the customer needs total control over it. Broadly, you could call that the sovereign AI market right now. Lots of governments and lots of legacy industries are going, “Wait, this open-source thing is really critical to us.”

Whereas 2 or 3 years ago, open source was viewed as largely a philosophical endeavor, which it is. Open source has always been political and philosophical by definition. But now there's an extraordinary business case for it, which is why you're seeing a lot of startups and companies changing their posture. They're going, “Wait a minute, some of the largest customers in the world—enterprise customers—happen to be governments, legacy industries, and Fortune 50 companies, and they want stuff on-premises. That's when you go adopt open source.” I think there's been a business shift as well. I don't know if you agree.

Yeah, this is great. I totally agree. I do think it's interesting to have the conversation about where it's the same and where it's different. Everything said is exactly right: We have a very long history with open source, and it's a very useful tool for businesses, but also for research and academia, et cetera.

But let's just talk about businesses and startups. It's a great way to get a distribution advantage. It's a great way to enter a market where you're not an incumbent and you're a startup. So it's one of the tools for building software that's been used, and open source has been used in a very similar way. You can use it for recruiting, brand, and distribution, and we see all of that.

But there's something that's unique about AI that software doesn't have. We're seeing very viable business models come out of it that don't have the limitations of traditional software. And this is for 2 reasons. One of them is that open weights is not the ability to produce the weights, and open software is the ability to produce the software. If you give me open software, I can compile it, modify it, whatever. But by giving me open weights, you don't have the data pipeline when you're talking about open weights. So you don't actually enable your competitors in the same way open source enables them. So that's one.

The second one is that there's this very nice business model that's kind of a peace dividend to the rest of the industry: You produce open weights for your smaller models that anybody can use, but you keep the larger model internally, which is actually also more difficult to operationalize for inference. There are good reasons to do this. Then you charge for the largest model, and you use the smaller open models for brand and distribution or whatever. I feel like it's actually an evolved version, from a business-strategy and industry-perspective standpoint, of open source for these reasons.

I think it's the AI flavor of open core, which historically was theoretically supposed to be a sustainable model for open-source software development. It was really hard to implement because of the reasons Martin said: Once you gave away the code, it was really hard for you to protect your IP.

But with weights, you can contribute something to the research community, give developers control, and allow the world to red-team it and make it more secure, while you're still able to—because of the way distillation works and some of the ways post-training works—hold on to some of the core IP. That then allows you to build a viable, sustainable business, and that is unique about open—

But also, you have the data pipelines; you have the data. Nobody else could—just because I give you the weights doesn't mean you can recreate the model. You could distill it to a subset model; there's a bunch of stuff you can do, but not necessarily recreate it.

Listen, having been a student of open-source business models for 20 years and having watched how it shaped the way the industry adopted and built software, I actually think the AI one is more beneficial to the companies doing it, for sure. But as a result of that, we're going to continue to see a lot of it.

So I think we should just assume that open source is part of it and every country is going to do it. One of the best things about this current AI Action Plan is that it acknowledges that, and it wants to incentivize the United States to be the leader in it, which is such a dramatic shift from where we were this time last year.

Yeah. There’s sort of an ecosystem mindset that, if you’ve worked in any kind of developer business—which Martin and I, unfortunately, have spent way too long doing, working on dev infrastructure and dev tools—you internalize this idea that you often have to trade off short-term revenue for long-term ecosystem value.

I think what the action plan shows is that, yes, in the short term, it may seem like we’re giving away IP to the rest of the world by open-sourcing weights and showing the rest of the world how to create reasoning models and all of this stuff. But in the long term, if every other major nation is running its entire AI ecosystem on the back of American chips, American models, American post-training pipelines, and American RL techniques, then that ecosystem win is orders of magnitude more valuable than any short-term give of IP. Anyway, as we saw with DeepSeek, that marginal head start is minimal.

So, just to close the loop on open source, over the next several years, how do you predict open source and closed source will intersect? What will the industry look like?

Well, I think these are 2 different markets.

Yeah.

I mean, literally, the requirements of the customers are completely different. If you’re a developer building an application and you happen to need the latest and greatest frontier capabilities, today you have a different set of requirements than if you’re a nation-state deploying a chat companion for your entire employee base of 7,000 government employees.

The product requirements, the way you provide those products, whether you deploy them, the infrastructure, the service, the support, and the revenue models are completely different. Often, I think people don’t realize that closed source and open source are not just differences in technology, but completely different markets altogether. They serve different types of customers.

If you believe AI is this explosive new platform shift, then there’ll be winners in both. I do think what we need to contend with is that it seems like it’s getting harder and harder to be a category leader if you don’t enter fast. The speed at which a new startup is able to enter the open-source or closed-source market and create a lead is absurd.

We’ve both had the chance to work with founders who are literally 20-some-year-olds out of college, 2 years out of college, building revenue run-rate businesses in the tens to hundreds of millions of dollars, serving both of these markets and expanding like this. I think the biggest mistake is to confuse these 2 markets as one.

And then there is the classic, “Oh, let’s wait to see how they evolve,” because the pace at which a new entrant is able to actually create a lead in the category is quite stunning.

Let’s go into the action plan. Right. What are our biggest reflections from it? Where are we most excited?

If you look at the quote that they start with, I wanted to read it out because I thought it was pretty poignant. It was, “Today, a new frontier of scientific discovery lies before us.”

I thought that first opening line was fantastic, out of all the things they could have said. They could have said, “We’re in an arms race,” which, sure, the first page—the title says “Winning the AI Race.” But if you actually start reading the document, the first sentence is a quote from the president that says, “Today, a new frontier of scientific discovery lies before us.” I love that they led with something inspirational.

Yeah.

Because ultimately, the technology has to confer some benefits on humanity. I personally just love the fact that we are starting to explore what these frontier models mean for scientific discovery in physics, chemistry, and materials science.

We need to inspire the next generation to want to go into those areas because it’s hard. It’s really hard to do AI in the physical world. You have to literally hook up wet labs and start doing experiments in an entirely new way.

You need people who are excited not only about wanting to do machine-learning work, but also about the hard work of being lab technicians, running experiments, and literally pipetting new materials and doing chemistry, right? I think that was missing in a lot of the discourse under the previous administration.

You can sometimes judge a book by its cover, and I think this was a strong start. Now I think we should actually dive into some of the bullets.

Okay. So, the other one that I thought was a huge omission is that there’s basically no real mention of academia or investing in academia. There are some oblique references to it, but it’s just been such a mainstay of innovation and computer science over the last 40 years. Not having a major part of it, I think, is a shame.

I understand that right now there’s kind of a standoff between higher education and the administration. I get it, and I actually think that both sides have fairly reasonable points. But to have a major tech initiative without including academia just feels like we’re fighting a battle with a hand tied behind our back, or some aphorism.

This is a good problem to have, which is that I think it’s extremely ambitious. It’s a little bit light on execution details—what happens next?

A good example of that is that I do think, directionally, it was great that they said we need to—let’s read this bullet point—“Build an AI evaluations ecosystem.” I loved that because it acknowledges that, before we start making grand proclamations about whether these models are risky or whether these models are dangerous, let’s first agree on how to measure the risk in these models before jumping the gun.

That part, in addition to the open-source bullet, was probably the most sophisticated thinking I’ve seen in any policy document. The reality is that America leads the way. Within 24 hours of this dropping, Martin and I were getting texts and messages from folks in many other governments around the world asking, “What do you guys think?”

It was not hard for me to endorse it and tell them, “Look at it as a reference document,” because there are things here that, arguably, are more sophisticated than policy experts even in Silicon Valley would recommend. Building an AI evaluations ecosystem is not easy, and I think they lay out a pretty thoughtful proposal on why that’s important.

Now the question is how, and I think that’s what we have to help D.C. with: the hard work of implementing this stuff. But the vibe shift—from “Let’s not jump the gun on saying these models are dangerous. Let’s first talk about building a scientifically grounded framework for how to assess the risk in these models”—was not at all a given to me, and I was really excited about that.

Yeah, there’s been a lot of focus in the last few years by several companies, but also by the broader industry, around this idea of alignment. Have we made any progress on alignment? What is your perspective on what they’re trying to do? Is that a feasible goal? Help us understand what they’re trying to solve for.

So, at an almost tautological level, alignment is an obvious thing you’d want to do. I have a purpose; I want to align the AI to this purpose. It turns out these models are problematic—generally unruly, chaotic, whatever adjective you want to use.

Understanding how to better align them to any sort of stated goal is very obviously a good thing. I think we’d all agree that aligning them to whatever the goal is—to make them more effective at that goal and do that thing—is good, especially given that these models tend to have a mind of their own.

The subtext that I bristle at is that the people doing the alignment are somehow protecting the rest of us from whatever they think their ideal is, as far as dangers to me, thoughts I shouldn’t have, or information I shouldn’t be exposed to.

That’s why I think we need to be very careful, even when we come up with policy, not to impose a different set of ideological rules on top of these. I just think alignment is something we should all understand. Actually aligning them, to me, is where I take issue with any sort of top-down mandate.

I agree, and I think there’s a quote from a researcher that I think is very accurate: You’ve got to think about these AI systems as almost biological systems that are grown, not coded up, right? Sure, they express themselves as software, but in many ways, when you’re training a model, you are actually growing it in this environment of a bunch of prior history, training data, et cetera.

Often, empirically, you actually don’t know what the capabilities of the model are until it’s actually done training. I think that’s a useful analogy. Where I think that falls down is when people go, “Well, if we can’t align it because we don’t actually know what its biological mechanism is until it’s grown up, and we don’t know what its risks are, then we can’t deploy these AI models in mission-critical places until we’ve solved, let’s say, the black-box problem—the mechanistic-interpretability problem.”

Can you trace deterministically why a model did something? We’ve made a lot of advances as a space in the last few years, but it still remains a research problem.

But just because you don't understand the true mechanism of the system doesn't mean you don't unlock its useful value. If you look at most general-purpose technologies in history—electricity, nuclear fusion—there are many examples of technologies where we knew they were complex systems and we didn't truly understand at an atomistic or mechanistic level how they worked, but we still used them.

And we don't understand how the internet worked. I mean, there's a whole field of network measurement trying to find out what the heck the internet was going to do. Was it going to have congestion collapse? Any complex system has states that you just don't understand.

Now, listen, I would say these models, more so than many other technologies, have very real implications. But we know how to deal with ambiguity.

We don't know how our own brains work.

No consciousness.

And we don't stop working with other human beings.

Unfortunately, we're stuck with them, so we have no option on that one.

Totally.

To extend that analogy, what do you do? You're like, “Okay, I don't know how a brain works. It's got a bunch of risks. This person may be crazy, but I still want to unlock all the beautiful benefits of the big, beautiful brains that humans have.”

And so you develop education. You send kids to school, and you teach them values. Then you send them off to college, and they get to learn something specific. Then you get to test them in the real-world environment. They get a résumé and work experience, and they get to prove that they actually are, within a risk-based framework, manageable and so on.

And that, as a society, has unlocked human capital, right? Arguably, the greatest technology we've had in 500 years of modern industrial innovation. So I think what I hate about the alignment discourse is that it sometimes confuses the fact that we don't understand the system with the fact that we can't use it.

I think mechanistic interpretability—which some folks would say is the holy grail, being able to reverse-engineer why a model does something—is still a research problem. But that doesn't mean we haven't made progress on how to use unaligned models or improve alignment to a point where they're useful in massive ways, like software engineering and mission-critical systems.

I think what the smartest might say is, it's not that—it's really just, what's the rush? Maybe let's focus on integrating all the capabilities we already have before pushing the frontier, which then ends the arms race, et cetera. There's a risk of slowing down, too, that maybe isn't fully appreciated.

Until we've solved cancer, every month that we're not rushing to the frontier of accelerating biological discovery or scientific progress is a month that millions of people are suffering from disease that we could be solving with AI.

Yeah. I mean, this is the thing with all of these: there's always this kind of reverse question on innovation, and you say, “Well, okay, it's like the Bostrom's urn experiment, his whole urn hypothesis. There's an urn of innovation, and you pull out balls; one of them is a black ball that destroys everything, right? So eventually you'll draw that ball. Why would you ever do innovation?”

That is the thought experiment, and the answer is so simple: It just turns out that it's much more dangerous not to pull out balls than to pull out balls. That's always the answer.

So when people ask p(doom), “What is the p(doom)?” the answer is not 0.1 or 0 or 100. The answer is, the p(doom) without AI is actually quite a bit greater than the p(doom) with AI. And the answer to “What's the rush?” is the same thing: Clearly, if you ignore the benefits of technology, then you would say, “If it's all negative, no rush at all,” right?

The reality is, the benefits are so dramatic and so obviously dramatic. Now, thank God, we've got a year's worth of data on this stuff. They're clearly economically beneficial; they're clearly beneficial in expanding a number of areas of core science. The rush is getting to the next set of solutions, as opposed to being afraid of a set of problems that we still can't clearly articulate.

And listen, as soon as we understand marginal risk, we absolutely should address it directly. Again, the action plan does a great job of penciling this out. It does want to explore implications for jobs, implications for defense, and implications for alignment. That's exactly where we should be in the exploration phase.

Do we have a definition of marginal risk, or a perspective on how to think about that idea?

Well, let's just be clear about what we mean by marginal risk: Computer science, computer systems, network systems, and stochastic systems are risky. We've got decades of ways of thinking about measuring, regulating, and changing common behavior based on this type of risk.

And so the question is, can you take all of that apparatus that's been hard-won and apply it to AI? If so, we know it's effective because we've used it before and we've got a lot of experience with it, and it's ready to be done. Or is there a different type of risk that's not endemic to those systems, in which case we'll have to come up with something new?

You go down that exploration, maybe it works, maybe it doesn't work, et cetera, right? So that's what marginal risk is. And I just think that the problem is, if you don't know what it is, how are you going to define a solution?

I think that's right. Philosophically, the idea is, if you're going to say we need new solutions, then you need to articulate why the problem is new and why solutions that work really great are no longer sufficient.

And I think it's almost obvious when you state it, but this was the state of the world a year ago. We were having to look around the room and say, “Can I raise my hand? Why are we introducing net-new liabilities and new laws that we've never had to do before if you can't articulate why there are new problems to solve?” If it ain't broken, why are you trying to fix it?

And so marginal risk is, I think, a slightly more technical way to say we have the tools to manage risk; we don't need new ones. And if you think we need new ones, then, hey, just take a minute to articulate to us why.

Is there anything else you wanted to make sure we got to? Otherwise, I think this is great.

Time to put the action plan into action.

Excellent. Martin, thanks so much for coming to the podcast.

Thank you. Thanks for having us.