The RAISE Act: Minimum Standards for Frontier AI Development, with NY Assembly Member Alex Bores

My guest today is Alex Bores, a New York State Assembly member representing New York’s 73rd District on the East Side of Manhattan and the sponsor of the RAISE Act, a bill designed to set a minimum standard for safety practices among large AI developers.

In the generally polarized political environment that we take for granted in 2025, it is a striking fact that, while poll after poll shows significant majorities of voters across both American parties and internationally want more regulation of AI—with far more voters worried that the government will not go far enough than that it will go too far—the United States still has no meaningful laws covering foundation models or frontier AI development.

In part, this reflects the fact that, while people generally agree on this issue, it is not a top priority for many. In part, it is the result of a very healthy fear among legislators that they do not understand this fast-moving technology wave well enough to regulate it effectively and might therefore end up doing more harm than good. And in part, it stems from the fear that anything that slows U.S. AI development could allow Chinese AI developers to get ahead and ultimately win whatever AI race we find ourselves running.

These are real and important concerns, and I have spent most of my adult life arguing against premature or heavy-handed regulation that might inadvertently deny us the benefits of breakthrough technologies. Nonetheless, it seems to me that, especially considering the breathtaking pace of advances in AI capabilities and the fact that even among Turing Award winners, forecasts range from AI-enabled utopia to AI-caused human extinction, a functioning democracy would be responsive enough to public concern to put at least some minimal standards in place now.

That could both reduce catastrophic risks and hopefully help us avoid a future crisis, real or perceived, that could lead to knee-jerk and ultimately counterproductive decisions. With that in mind, I think Assembly Member Bores—who notably has a master’s degree in computer science and spent a decade in the tech industry, including a number of years at Palantir—is perhaps the tech-savviest and friendliest legislator the industry could hope for.

The RAISE Act, which targets large AI companies and imposes relatively modest requirements around the development and publication of a safety plan, audits to ensure that the safety plans are followed, and whistleblower protections to alert the public if they are not, is approximately the least burdensome regulation we could realistically expect to see passed. It should be said that it is very much in line with voluntary commitments that frontier-model developers have previously made.

Nevertheless, in this conversation, we get deep into the weeds of the various definitions the bill uses and the requirements it imposes. I act as a sort of red-teamer of the bill, while Assembly Member Bores clarifies and defends key provisions. He also explains the process he has already gone through to work and compromise with industry and to avoid a situation in which lots of different states create undue friction by passing their own distinct regulatory frameworks.

Importantly, everything we discuss would only apply to a high-single-digit or perhaps low-double-digit number of very well-resourced companies working at the frontier of AI capabilities. These companies would still have wide latitude to design their own safety plans, so long as they take reasonable care to minimize risks that could cause 100 or more human deaths or more than $1 billion in damages through chemical, biological, radiological, or nuclear mechanisms—or what I would call automated crime.

While some will no doubt raise additional good-faith objections, I found Assembly Member Bores’s defense of the bill’s neutral stance with respect to open source quite compelling. As you will hear, I hope, if anything, that this bill spurs open-source champions to invest heavily in new safety techniques so that we can continue to enjoy open-source frontier AI without dramatically elevating the risk of engineered pandemics or other AI-enabled disasters.

In any case, this is just the first of a series of episodes on different AI policy proposals that we will be bringing you this summer. I look forward to exploring a broad range of perspectives, and I will continue to watch this bill as it evolves through the legislative process.

For now, I hope you enjoy this deep dive into the RAISE Act, a bill meant to raise the floor for frontier AI development safety practices, with New York Assembly Member and bill sponsor Alex Bores.

New York Assembly Member Alex Bores, sponsor of the RAISE Act. Welcome to the Cognitive Revolution.

Thanks for having me. I’m excited for this conversation. You have waded into what might seem at first like no big deal—just some light technology-regulation waters—which may bring out a lot of strong feelings in people. I applaud you for taking on the challenge, and I’m excited to get into your perspective and motivations, what you’re hearing from people, and obviously the proposed legislation itself. I really appreciate you taking the time to do this.

Well, I’m really excited to be here. I know you have a lot of guests from all over the AI field, which I’ve enjoyed listening to, but not as many elected officials. So, I’m excited to dive in.

Yeah, you’re on a short list.

Speaking of short lists, one thing I like to do, especially for people who are coming into the AI world from other backgrounds, is give a little bit of context on their credentials. As I was doing my homework, I noticed that you are the only member of the Democratic Party in New York State government who has a degree in computer science. Maybe just give us a little bit of your personal background and relationship with technology. I think that will be helpful.

Absolutely. I was the first with a degree in computer science. I’m happy to say I’m no longer the only one, though it is still quite limited.

My background is that I worked in tech for nearly a decade before I ran for office. I was at Palantir for almost 5 years. I joined as a data scientist and rose to lead a large portion of the government business. I joined a couple of startups after that. During that time, I got a master’s degree in computer science with a specialization in machine learning.

Then this seat opened up in 2022, and I had a lot of helpful conversations with friends. I had always been downstream of policy, often trying to fix it with tech, and this was an opportunity to go upstream and actually design policy the right way.

One of those conversations was: “Listen, this is a thesis as to how you can have an impact—run. You don’t know if you’re going to win, and if you do win, in 2 years or in 4 years, if you’re not effective or you’re not enjoying it, you can quit. But you can’t, in 2 to 4 years, say, ‘Now I’m going to run for the open seat.’” That happens when it happens.

So I threw my hat in the ring. It was a contested primary. I’m still friends with everyone who ran; we all ran against each other, but I ended up being victorious there.

I’m now starting my second term. Terms are only 2 years, so I’m just starting my third year in the legislature. So far, I’ve found it to be a place where you very much can be effective, and I’m enjoying it.

That’s cool. Definitely, more people with technology backgrounds in government seems good. Frankly, more people who have no fear of life after elected office would be great, too. Too many are thinking, “What would I do if I didn’t have this seat?” That’s not a great position for the public to be in.

No, it’s a very dangerous position to be in. You have to be not too attached to the seat or the job.

Absolutely. That’s all I ask for from our elected officials: a willingness to sacrifice their political career when the occasion calls for it. It turns out that is kind of a lot to ask.

How about just a little bit more detail on some of the other technology-related things that you’ve done during your time there? I noticed that you had pushed for the state to adopt cloud computing. I also caught the push for a land-value tax. I’m coming to you from Detroit, Michigan, where we have a lot of empty lots—probably much less of a problem in New York City than it is here—but we have a lot of empty lots where people are free-riding on others’ investments and waiting for their land to appreciate, since it is not taxed in that way.

Maybe just give us a little more context on some of the things that you have pushed so far.

Absolutely. When you’re a legislator, you end up working on a wide variety of things because your constituents care about a wide variety of things. All of us have our specializations and knowledge that we bring into the legislature, and mine is around tech. So I’ve done a lot there, but certainly the concerns of my constituents vary widely, and I work on a lot of things.

Within tech, as you mentioned, I’ve encouraged the adoption of cloud computing within government so that we can deliver services more quickly.

I've helped to strengthen the protections for tech workers—really, for all workers—but it was contracts that are more specific to the tech industry, where companies would say they own any IP you develop while you're employed, even if it's not on company time or related to anything in the company. And that would just chill startups. I actually partnered with the tech industry to pass that because it was based on a regulation that had already passed in California nearly a decade before. But one of the things people want the least is a bunch of varied regulation across different states, so I was like, "I'm going to copy the California one exactly." Tech was like, "Great." Even though it's limiting us, the fact that it's a copy meant that it wouldn't be additional work. They ended up supporting it.

I've also done a number of bills this session around AI, beyond the RAISE Act. I'm working on ensuring companies don't fall into legal liability when they red-team their own algorithms. We actually want to encourage safety, and so while we are strengthening a lot of provisions around preventing CSAM—child sexual abuse material—we also want to make sure there's a legal liability shield for red-teaming, for trying to stop your algorithm from doing that. I'm encouraging the use of industry-developed standards like C2PA, which are metadata that help to establish provenance on an image or sound, so that you know what's real and what's not.

But beyond tech, as you mentioned, I do a lot on housing. I have a bill to enable a pilot on land value taxes. I have some things specific to New York City. You mentioned Detroit, and maybe New York City doesn't have as much vacant land. We don't have as much, but the land we do have is valued at an incredible amount, and the loss of tax revenue is substantial. If you just looked at the vacant land within New York City and were to tax it at its normal market value instead of this discount—I don't want to get into all of New York City property law, but no property is taxed at its actual market value—if you were to just tax vacant land at its market value, that difference would be another $800 million a year for the New York City budget. So, I have a bill that would shift that around and make more uses for it.

And then I do a lot around public safety. One of the things is that our trials in New York State are very backlogged. They are extremely delayed, and there are many, many reasons for that. The dumbest of which is that we don't have enough judges. I say that's the dumbest because that should be an easy thing to fix: You just create more judges. Two years ago, I did. I passed a bill the governor signed that created 20 new judges throughout New York State. But I couldn't create any new ones in Manhattan or in the Bronx or in the Capital Region around Albany because of a limit in the state constitution that dates back to 1846. And so, one of my other bills is a constitutional amendment to get rid of that limit, to allow us to have more judges and speed up trials. And then there are another 60 or so bills on my website. Anyone can take a look, and I always love feedback on them.

Cool. That's great. I appreciate the introduction. You mentioned your constituents, and I thought it would be helpful also just to locate you geographically and get a sense for the people that you're representing. Then I want to ask to what degree they are thinking and talking to you about AI. Where does it rank among their priority concerns? Maybe just take us quickly through the geography and the profile of the people that you're representing, and then what, if anything, are you hearing from them about AI? Is this something that they're pushing you to act on, or is this something that you are doing out of intrinsic motivation while they are mostly concerned with other things?

I represent part of the neighborhood where I grew up. My district is in Manhattan, much of the East Side of Manhattan. It's part of the Upper East Side and Midtown East. For those in New York, it's 34th to 93rd Street, Second or Third Avenue to Fifth for most of the district, plus Sutton Place in the 50s, on the East Side.

It is a highly educated district. It is also the wealthiest district in New York State. But even within that, there are many people facing challenges. Forty percent of the renters within my district are rent-burdened. They're spending more than a third of their income on rent. It's a district that has a lot of pride in its education and its schools. It's in District 2. I happen to represent my elementary school and am right near my high school and my middle school. They're both across the street from my district.

It's an area I know well. It's been fun getting to represent it and now meeting the parents of the friends I grew up with. They, like any district, have a wide variety of concerns. I think if you were to poll them, AI probably wouldn't be toward the top. They're worried about the cost of living, public safety, and the schools. They're worried about the things that everyone is worried about.

But when we get into a conversation and I start mentioning my background and my expertise in tech, the usual response is like, "Oh, thank you, because I'm terrified of this and I don't know what we should be doing, but I'm glad there's someone there thinking about it." There's this sort of latent fear—might be too strong a word, although for some I would say fear—but certainly unease and a feeling like something is coming and they don't really know what they're supposed to be doing about it.

Is there any more shape to it than that when people get on the topic of AI? We entertain the full range of risks on this program, and I would say your bill is, as we'll get into, more consistent with my approach. I want to see us have a lot of benefits, a lot of deployment, and a lot of use in places like education, even though that's going to be fraught and we're going to have to figure out a lot of things. I'm quite convinced that an AI tutor for every kid is a part of a winning future.

Then there are all kinds of things around privacy. New York is probably the place with the most security cameras on the street of any place in the country. Maybe D.C. has more, I don't know, but that's something I could imagine people talking about: surveillance and just who's watching whom all the time. Where are they on this? What sort of mix of AI-specific concerns do you hear about?

All of the above. I would say at the start that I largely agree with you. I think there is so much capability out there now that is unevenly distributed, and we could be making so much better use of the existing tools in ways that government helps to serve people and in education, as you said, and in a variety of different fields. I am also worried about what the future holds if we don't put guardrails on further research. That sort of short-term bullish, long-term wary position is something I share with you, but I don't know if it's as common in many places.

All of those concerns matter to my constituents. I hear a lot about workplace displacement. I hear a lot about privacy and surveillance. New York is now in the minority of states that doesn't have a comprehensive privacy law. We've been working on it for a few years, so that certainly comes up as a bedrock issue on which we are delayed.

But I tend to think of AI concerns using 3 binary questions. The first is: Are you pessimistic or optimistic? The second is: Is it short-term or long-term? And the third, when you think about bills, is: Is it use-case-specific, or is it general to the model? I think you can find concerns, and therefore you can find legislation, in all of those categories. I have bills in many of those categories.

Most of the legislation my colleagues are working on tends to be short-term, pessimistic, and either use-case-specific or general. By short-term, I don't mean that it will expire. I just mean that it's dealing with harms that are definitely already here: the chance of discrimination, the chance of privacy violations, and things that are in use today. Long-term is things that maybe aren't here yet. Maybe they are, but they're just starting to be. We're thinking about where they'll go, whether that's broader societal risks, et cetera.

That's where the RAISE Act is focused—not because I think that's the only thing that matters or the only thing we have to do, but simply because that is a place that not as many people are focused. I do think there are important steps we have to take.

Are there any other things that you're pushing legislation on that might be described as controversial? I mean, the two that you mentioned previously around encouraging red-teaming for things like CSAM seem like most people would be quick to sign on to something like that.

Is this a situation where we see the 10% of the iceberg that pops up and actually gets public debate, and 90% of things are generally pretty smooth sailing? People agree.

Well, first of all, I don't think any of my bills are controversial. I think they're all common sense, but you don't always control what the outside reaction is. You'd be shocked at the level of pushback I've gotten on encouraging C2PA, which is, again, a free, open industry standard that the industry itself developed. Then I say, "Oh, this is great. We should encourage it," and I get pushback: "Whoa, no, don't encourage the thing we developed."

I think what you said at the end there is quite insightful and something that people forget. Most of the work that any government does is noncontroversial. Most of the bills we pass are unanimous or nearly unanimous, and they're just about making government work.

We'll do probably 800 bills total between the Assembly and the Senate that we send to the governor's desk. She'll end up signing 600 or so a year. Of those 800, or 600, you're going to maybe hear about 20, maybe 50. Most of the work is just making government work.

This is the thing I really remind people, because we're at a point where belief in government is so low. Partly, that's because all they see on the news are fights. All they see is the drama, but that's because there's an incentive to cover the fights and the drama. There's not as much incentive to talk about funding water infrastructure so that everyone has clean water.

Yeah, of course we should do that, right? But that's most of the work of government and most of what we end up spending time on.

Are you getting any push for protecting various industries? I mean, this is something, honestly, that I expected.

Okay, yeah. Tell me more about that, because my perspective was, two years ago, I was like, "Wow, we are going to get into just brutal, bitter fights about where AI is allowed to be deployed and who's going to have what sign-off authority," and so on and so forth. It's been slower to develop than I would have guessed, so I was kind of expecting you to say, "Not yet, much." But now you're saying all the time. Tell me more about that.

Well, sorry. In what ways do you think it's been slower? What were you expecting to see?

I would have expected the medical establishment to have a strongly unified front by this point that AI doctors must be confined to some very narrow box and not available to the public directly, and so on and so forth.

Still today, I can go on to ChatGPT, Claude, and Gemini, which I did this week for a little thing on my kid's eye that I was trying to figure out what it was. The tip to the user is to tell them you're preparing for a conversation with your doctor, and that disarms the "I'm not your doctor" canned routine. With that, you can basically engage, and it's extremely valuable.

I would say, in my case, it probably did displace a trip to the doctor, which is one of the things that any professional guild might fear. So, yeah, maybe that still will happen, but I had kind of expected it already. The fact that I can still go to ChatGPT and ask my questions is honestly kind of surprising to me from 2 years ago.

No, it's a really good flag. I wonder if partially that's because nurses and doctors, and so much of the health profession, are licensed, so there's already some built-in protection.

One of the tongue-in-cheek things I say, although there's some truth to it, is that people in government—elected officials—are sort of the least qualified to be focused on AI displacement of workers because, by law, we can't be displaced by AI, right? You're not allowed to elect an AI. It would be quite the change to the Constitution to make that true.

The kernel of truth to that is that when you have licensing, there's some kind of built-in protection. I think it's the jobs where none of that exists, and especially where they don't have unions, that you're going to see much quicker turnover.

We see that in terms of a lot of the entertainment industry, right? You saw a lot of the strikes last year by the Screen Actors Guild, the Writers Guild, and the Directors Guild. They were about AI's role in producing movies and television, and what that will be going forward.

We see it in government employees as well. One of the things that New York did last year was pass a bill that said you cannot use AI to replace a government employee. But it was specific about replacing the actual employee. It doesn't mean you can't replace tasks and focus them on other issues.

We have so many open slots in government and so much more that we could be doing. Once you have that baseline of, "Oh, you as an individual worker are not going to be replaced," you can change the conversation to, "This is why you should learn it and be happy about it. It's going to make your job easier, and you have a protection of law," versus many people who approach AI coming in like the metaphorical Luddites—but the literal Luddites—who just say, "This is here to take my job. Let's destroy it."

Okay, the main focus of our conversation will still be on the RAISE Act, I promise. But that's really interesting, and I wonder what you think about it.

I had a very similar question. I was honestly kind of surprised by the answer that I got from New Jersey Governor Phil Murphy, not too far away. He was touting that they had done various AI deployments to accelerate call-center response times. Previously, if you called whatever line, you would wait 40 minutes on average, and they were able to bring that down to single-digit minutes. So, a great improvement in quality of service.

But I asked if you had an option, right? It seems like this is coming quite realistically, quite soon. Let's say you have an option where you can deploy AI to a customer-service function. We're not yet talking about strategic decision-making, but just where the rubber hits the road. You can put an AI in a call center.

Let's say you could do that in a way where you could cut 90% of headcount and 90% of costs and improve the service. You'd still have some people there, perhaps to take the escalations or whatever. How do you think governments should be thinking about that? Should they be prioritizing efficiency and service, or should they be prioritizing the jobs that they have? Is there some synthesis of those that you can imagine?

Yeah, it is an interesting question, but one that is largely academic because of how underfunded and already perhaps inefficient government is. There's more work for any of these people to do.

Recently in New York City, they transitioned to automated purchasing of MetroCards and now to OMNY, so the station agents weren't as needed. They didn't get rid of any of the station agents. They just empowered them to actually walk through the station, so they can help and be a presence there and be engaged with people in that way.

With call centers, I've personally called a government agency and been on the phone for 4 hours, waiting. Meanwhile, I'm Googling and searching. If there were an easy chatbot or something that could have answered my question ahead of time, not only would I get an answer, but that takes me off the queue, and the person who actually needs to talk to a human—because there will always be people who still need that sort of engagement—gets there a lot faster.

If we're smart, we should be pairing all of these new tools with the knowledge that comes from this deep work in government and this deep work with our citizens in order to make things better for everyone.

I'm a big fan of Jennifer Pahlka and Recoding America. In that book, she talks about someone who had joined, I think, the California Employment Development Department to process unemployment claims. I may have the state wrong, but the story was that there was the new guy who didn't feel really confident in all the systems and all that—and the new guy had been in the job for 18 years.

I mean, the amount of human knowledge that is tied up that we could unleash if we're protecting their jobs, right? That doesn't mean in the future you're always going to hire the same number of people. You can go down by attrition, and you can repurpose people into other roles. But if you make it us versus the machines, you're not going to get the best results for people.

Yeah, I feel like—I wonder how long that paradigm lasts.

That doesn't seem like the current paradigm accurately described, but the RAISE Act certainly seems to be a bill that is at least partially fearing the AGI, so to speak. I do wonder if that paradigm lasts more than, say, another 1 or 2 years. And I also really wonder about people's ability to change into those new jobs, especially at scale.

It's like we have millions of people driving cars and trucks in the country. And if the self-driving car stuff really starts to work—which, by the way, having been in a Waymo and a Tesla recently, it really is starting to work.

Totally.

That's going to be a wave where we're not really going to be able to tell 4 million drivers or whatever, "Oh, you could go learn to code." By the way, it's also now a hotly debated question as to whether or not it's even worth learning to code. So where are they going to go? That's another interesting question, I guess.

Well, I 100% agree. I just want to point out that I think that'll hit the private sector before the public sector. I think in the public sector, we can do so much just by trimming our regulations, and we have so many vacant positions already. It'll hit the private sector first, but it's a thing that I am concerned about because people say, "You look through history, and every technological revolution, every advance, creates more jobs than it destroys."

Maybe, but the time between technological advances has been shrinking over time. And so, until recently, you could maybe guarantee that it would create new jobs and that it would be worth it for you to go back to school or be retrained, because the next revolution wasn't going to happen in your career, right? But now we're at a place where jobs could be replaced every year, every 6 months.

If the AI is acquiring new skills faster than any human being can, that is a fundamental question we don't really have a policy answer to, because by the time you retrain for the new job, you're going to be in the same circumstance. So that is a thing government and people outside of government need to be thinking a lot more about.

Do you spend time thinking about a new social contract?

I am right now thinking a lot about the RAISE Act and how to get that through. But once that is through, this is definitely a place where I want to spend some cycles, and I'm really interested in having conversations with others who are doing that deeply.

I saw something recently, and I don't know much about this at all, but I think Tyler Cowen posted on Marginal Revolution, "Who Needs a UBI?"—pointing to New York State. And you can correct anything I get wrong on this. The idea is that there's now an ability for people to choose and hire their own independent caregivers, and many are hiring people they know—people from their families. And this is, in some circles, treated as a scandal.

If there's one candidate for a broad class of activity that people could maybe shift into in real numbers, caregiving broadly would maybe be the thing. And this does seem like a proto-UBI policy that also tries to get some useful contribution from people and probably does quite well on scores of meaning and things like that. And so I was like, man, maybe New York State government has stumbled onto something here that actually could be the seed of a new future social contract.

I haven't read that piece, and certainly, if he's referring to what I think he's referring to, it is not meant to be a UBI. So while there have been a lot of investments in caregiving and home health care, those programs are often paying family members or others to do it instead of someone going into a nursing home, which might be much worse for them from a social perspective and also cost the state a lot more.

There certainly have been people on the edges who have taken advantage of a program, and we made changes in the budget last year to crack down on that. But overall, those sorts of home health care programs have actually saved the state a lot of money. So I think you have to put that in the broad picture of things.

No, UBI is definitely part of the conversation. This is one of the things that's moving really, really quickly, and so I imagine that will be part of it, and there will be many other ideas that come as well. And by "the conversation," do you mean the conversation in the New York State Assembly?

I think more broadly than that at the moment, but hopefully it'll be part of the conversation the legislature is having.

All right. Let's narrow our focus then to the RAISE Act and what you're trying to do with it. I've got pretty detailed notes here, but maybe the first thing to do is just have you give the pitch: What are we trying to do? What does this bill require? Why should we be confident that it's not too big of a burden to put on companies? Give us the high level.

This bill is meant to ask companies that are doing extremely advanced research, of the kind where we don't really know the impacts yet, to have some basic safety protocols in place. Largely, those safety protocols that are required in the bill are in line with commitments that they already made during the Biden administration. So how do we know that it's not too onerous? They've largely already committed to do it, and in many cases are already doing it, if not to the exact letter of the law, close to the spirit of the law.

The 4 provisions it requires are that they have a safety plan, that the safety plan be looked at by a third party that's not them, that they disclose critical safety incidents, and that they don't retaliate against their own workers or that third party if they are whistleblowers and disclose something that is truly catastrophic risk. We define that strictly in the bill as to what qualifies, but it has to be something that's really increasing risk.

That's all it asks. And people say, "What is the impact? What is the target of that?" In many ways, we're just putting very basic guardrails there. I don't think this is the furthest that we should go. I don't think this is the end of what's there. I think in many ways, this is just laying out a floor such that most people in the field are really good actors.

But when you have the pressure of your next quarterly profit—because right now this would almost be exclusively public companies, with some extremely well-funded exceptions—it can become easy, even if you've written down a safety plan, to maybe say, "Hey, we're 2 months behind in the AGI race, and that could be catastrophic. Let's just skip this test." We want to make sure there's no incentive for doing that.

At the most basic level, we defer a lot of the choices to the companies themselves. We don't come in and say, "You need to have exactly these tests done or exactly this evaluation of risk." What we're trying to prevent are the cigarette companies of old knowing that their cigarettes caused cancer but then denying it publicly and not doing anything to make the cigarettes healthier, or the oil companies knowing for decades in advance that their products were causing climate change but denying it and still putting it out there.

This is meant to say that if your own testing, if your own research that you've thought of ahead of time without the economic pressures, is saying this is a massive risk—that it could cause what we define in there as a critical harm, 100 deaths or $1 billion in damages—you shouldn't be releasing that model.

That seems like a not-super-stringent threshold. As I said, I think all of my bills are noncontroversial; it's just the other people who sometimes don't see it that way.

Let's go through a few of the definitions. You gave the threshold for severe risks: 100 deaths or $1 billion in damages. There's also this "large AI company"—or it doesn't say AGI; it says "large AI companies." And that seems to be defined as a company that has spent $100 million in total on training models and at least $5 million on 1 model in particular. Do you have an idea of how many companies that would cover in today's world?

I'm not quite sure, honestly. I think it's still single digits. Maybe we've crossed into double digits, but it's a very small number. That's intentional. It's meant to look at the absolute frontier as it exists right now and not sweep in too many others.

I think one of the objections to previous regulation was that it would involve a lot of startups. It would involve a lot of smaller companies. So we chose that $100 million threshold intentionally. And we also exempted—I'll point out—academic research as part of that. As I said, this is really about those potential incentives to skip your safety plan, and we just don't see those same sorts of incentives in academic research.

Yeah. Okay, that sounds about right to me. For what? I was kind of like, is Amazon on that list yet or not? I mean, we’re talking about big companies that would be sort of the marginal, you know, in or out.

It’s also notable to me that the kinds of risks that are covered are pretty narrow. You’ve got your classic CBRN—chemical, biological, radiological, and nuclear, I believe, is the end. Biological, of course, being the number one in that category by far. And then you’ve got another category, which I think is pretty smart, which is just automated crime.

I’ve got a little background myself as a red teamer of various models and products, and it is honestly amazing. Although this actually wouldn’t necessarily be covered in some forms by the bill, it is amazing. In some cases, you can go to some of these calling-agent companies, clone a voice—I’ve done Biden, I’ve done Trump, I’ve done Taylor Swift—and prompt the model to just call any number and say anything and try to scam people at scale. They’re still in the uncanny valley, but automated crime is definitely the kind of thing that is now doable with some success.

But that’s it, right? So I’m interested in any reflections you have on where you decided to draw that line. It strikes me that these are not behavioral risks. I mean, with smoking, of course, smoking is obviously bad for you. I am well aware of that, but it is much more of a behavioral sort of thing where you do have some agency in the situation.

You kind of know it is bad for you—at least at this point. You probably know you shouldn’t be doing it, but you’re still kind of doing it. Smoking is almost more to me like: we’re going to have addiction to AI. We’re going to have people kind of falling in love with it and going off into weird lands with their AIs. I don’t want to say using AI is smoking. I use AI every day for various things.

Right. I was just doing the example of companies knowing their own tests show it’s risky, but then going ahead anyway. And the point of the bill is, if your own tests are saying this is risky, we actually, as a state, are going to say you need to take a pause there.

I want to talk about the risk, but I do want to say one more thing on the definition of large developer before we move on from that. You have to be a large developer, so you have to spend $100 million in training costs. Then a frontier model is either one trained with 10^26 computational operations and $100 million spent on that model, or one produced specifically through knowledge distillation.

That’s not a post-training modification. That is the specific process of using another frontier model—something at 10^26—to train a smaller model that can have similar performance and as broad a performance as the original one. And you need to have spent at least $5 million on that. That’s largely in response to DeepSeek. We’re seeing a lot of new models spin up that are being trained on the larger ones and introducing their own risks.

This bill, as written, might not cover the first version of DeepSeek that caught everyone’s attention because that was trained on o1, which wasn’t 10^26. But a DeepSeek-like thing in the future, as long as it has any interaction with New York—that means it’s available in New York via the App Store, that means you have any employees or any business presence—if you want access to New York markets, this applies to you. And so that future version of DeepSeek would actually have all these requirements in there as well.

That was an intentional push on our front: not just to apply to the first movers and then have people, especially overseas, introducing similar levels of risk. That was a really intentional choice in that regard.

But jumping to the risks, as you brought up, we include chemical, biological, radiological, and nuclear risk. If it is in any way aiding in bringing that about, then these provisions apply, and that’s seen as an unreasonable risk of harm. And then—I’ve never phrased this as automated crime, but I think I’m going to do that going forward—the reason we’re saying it has to be committing those crimes with limited human intervention is we do want some built-in protection.

You can really abuse and twist and modify a model, and if a human being is really determined to prompt it in order to do something that violates some obscure law, that’s not what this is meant to target. But there’s some level of limited human intervention: if I can just tell it to do a thing, it largely does it, right? That’s what would fall under the potential risks.

And there’s an additional caveat in those risks, which is it has to be materially helpful in doing that. If I could Google how to build a nuke and I get the same high-level overview as I do when I enter it into an LLM, that’s not something you’re going to be held liable for. It’s really for when you’re making a material difference in the ability to do that.

Yeah. Yeah. What I meant to emphasize in bringing up the behavioral aspect of smoking was really just that there’s a lot of other things that people are worried about—and I think with at least some good reason—that are out of scope for this bill. That includes basically anything where it’s like, this might be bad for you, but you might like it, and we’re not really sure. And so all of that kind of stuff is out, and we’re not in people’s private business with their personal relationships with AIs in this particular bill.

Yes, totally. The knowledge-distillation thing is, if I had to guess, probably going to be one of the most fraught provisions that people are going to really want to understand and pick apart. It seems to me very reasonable to say to the high-single-digit to low-double-digit companies that are spending $100 million-plus that you have to have a plan. You have to publish the plan, and you have to have an audit to make sure you’re standing by the plan.

And you have to have some protection for whistleblowers in case you’re not doing that and people see that internally. I think most everybody’s going to be sort of on board with that. I certainly, like your constituents, would expect that they’ll be quick to support that.

The knowledge-distillation piece, though, is tricky. I guess, first, a clarification question: is it an “and” clause? If I’m doing knowledge distillation, I still have to be spending $5 million?

Yes.

Or—and $100 million, or not necessarily $100 million?

Okay. That is a hard hurdle. If I’m not over those financial levels, then I can distill all I want. I can go grab R1 and distill it into Llama 4.1, whatever I want to do, as long as I’m doing it under that $5 million and $100 million spend level.

Correct.

Okay. And those are, I assume, cumulative over all time?

The $5 million is per model. The $100 million is cumulative over all time.

Okay. So when I get to $100 million, then it's both, right? If I spend $5 million on one but I haven't spent $100 million, I'm not there. Okay. So that basically gives people a ton of freedom if they're operating at personal budgets or startup training budgets.

And for people that may not know, what is a realistic fine-tuning budget? Obviously, they can get bigger than this, but I was recently a very minor contributor to a project that made some waves called Emergent Misalignment. A research team, in pursuit of answering a different question, fine-tuned an OpenAI model on 6,000 examples of code that had been written without following security best practices, in some cases flagrantly so—just not taking proper precautions in the code that you're writing.

Fine-tuning a model to do that turned out to create a generally evil model. And that was the Emergent Misalignment phenomenon. You would think, how does this happen? People are still trying to figure that out. But what is clear and has been replicated is that if you train on insecure code, you get a model that wants to have dinner with Hitler and has all these crazy notions about AI enslaving humans and so on. And you're like, wow, that's really quite out of domain and yet pretty striking.

Anyway, the cost to do that fine-tuning with those 6,000 examples is like $25 on the OpenAI API. There are a lot of wrinkles: you're doing low-rank fine-tuning there; if you're doing all weights, that ends up costing more, whatever. But we have orders of magnitude between making a rather large behavioral change to a model and the sort of thing that you have to be over in order for this bill to apply to you at all. And I do think that's important to understand.

I guess one question would be: given the relative cheapness of knowledge distillation and the relatively high financial hurdles, what is the purpose of that clause? Couldn't you delete it? What would be lost if you deleted that knowledge-distillation clause entirely?

If you deleted knowledge distillation, it would only be individual models that are 10²⁶ FLOPs and $100 million in spend, and you wouldn't have any coverage of the DeepSeek phenomenon. I think it's important, if we're seeing more of those kinds of threats, to have some coverage there. But as you correctly point out, we're giving people orders and orders of magnitude to do interesting things here. It is really meant to just cover the frontier, and in particular those with large financial resources.

I keep emphasizing to everyone that the $100 million threshold is its own threshold. If you have not spent $100 million specifically on compute, specifically on training, this bill does not apply to you. And so we're really talking about single digits, maybe double digits, at this point, that it applies to.

So can you tell me a little bit more about the theory of DeepSeek? I mean, I don't know what their total training spend has been, but they did say, I think, $6 million was the V3, which then got turned into the R1. So my sense would be that that model in and of itself would hit that $5 million threshold. So, yeah, just walk me through the theory a little bit more. I guess the understanding is you're assuming or inferring that they took a bunch of OpenAI outputs and trained on that, and certainly it is a way to save money.

V3 would not be covered, right? Because it was trained on o1, and it's only knowledge distillation if it's of a frontier model, and o1 didn't qualify as a frontier model because it wasn't 10²⁶ FLOPs.

So it has to be 10²⁶ FLOPs and $5 million?

No, no, no. The knowledge distillation needs to be using a model that itself qualifies as a frontier model. And so if it's using a model that isn't itself 10²⁶ FLOPs in order to train the smaller model, it does not apply.

But that original qualification is 10²⁶ and $5 million, or is that—

Oh, the original is 10²⁶ and $100 million. It is really just the most extreme models: 10²⁶ FLOPs and $100 million in spend. That is the base definition of a frontier model. Then there's this additional definition, which is if you use a frontier model to do knowledge distillation and spend $5 million in that process, the resulting model also counts as a frontier model.

Okay, so I think I had a misunderstanding. I'm not a professional legislation reader, so let me just make sure I have this clear. The way you get into the large AI company category in the first place is you train a model that is 10²⁶ FLOPs or $100 million in spend on a single model?

I would think about this a different way, right? I would say the definition of a large developer to whom this bill applies is that you have trained at least 1 frontier model and you have cumulatively spent $100 million training frontier models.

Right, so that's $100 million training frontier models. Okay.

Frontier models are defined as 1 of 2 things. The first is the one I think most people are familiar with, that has been used elsewhere: the model itself is 10²⁶ FLOPs and $100 million was spent on training that model. That's the base that people think of as frontier, similar to what was in California, similar to the Biden executive order, without the spending threshold. 10²⁶ FLOPs and $100 million—that's a frontier model.

A second way that something can be a frontier model is if it is trained via the process of knowledge distillation from a frontier model and that process was at least $5 million. So, 2 pathways to become a frontier model.

And then a large developer is someone that has spent $100 million training frontier models. So they can do that either by training 1 model at 10²⁶ FLOPs and $100 million, or by training 20 knowledge distillations at $5 million on each of them, or any combination thereof.

Gotcha. Okay. So the main reason for the knowledge-distillation clause is that you want to catch companies that are working at large scale but taking a knowledge-distillation route such that their largest individual models could still sneak under the mainline definition of a frontier model but would have similar capabilities, because obviously that's the whole point of distillation.

Absolutely right.

Okay, good. Well, thank you for walking through that with me. This stuff does get a little gnarly sometimes.

No, and listen, legislation, especially at the state level, is much easier than reading federal bills, right? Federal bills are thousands and thousands of pages. This one, I think, is 15 or so. I'd like to read every word. It's a manageable read, and the details really matter.

It's dense legal language. And I think that's one of the challenges we've seen in past bills: people hop on Twitter and take a couple of words out of context or don't think about how every bill is inserting language into the code, right? The legal code, not the computer code—the legal code of the state. And so it's affected by all of these other words that are around it as well.

So I support everyone asking questions. None of this is easy to understand exactly, the same way coding any of these models is not easy. But you don't see people jump on Twitter after reading 3 lines of code in Llama and making sweeping claims about the whole thing. You should think about that.

Well, you should join my part of Twitter, I think.

Fair enough.

Fair. I could show you some of that. Okay, well, let's go maybe double-click, then, into a little bit deeper into the language. Here's my summary of the requirements that the frontier developers would have in terms of the safety protocols that they would have to develop and publish.

Basically, they need to come up with various ways to reduce risk. They have to reduce the risks of the model being used for these CBRN-type purposes. There's an interesting clause about reducing risk by sophisticated actors, which I'm interested in. I assume that's code for the CCP or maybe North Korea or whatever.

Yeah, any sophisticated state actor. It's not targeting 1 specific one. That's sort of any out there. We're saying that the stakes of this debate and the risk are so high that you need to include nation-states acting, and that is typically translated to, like, securing the model weights and tightening up your security practices in today's world.

Is that how you are imagining that playing out as well?

Yeah, largely. All of cybersecurity is based on your threat model and based on what the risks potentially are. If you're applying normal corporate security to these models, you're probably not doing enough. This is just meant to be very explicit that the stakes of this incredibly powerful technology are large and your threat model should be including sophisticated state actors.

Okay. There's then an interesting section also where it's basically saying you have to explain why you think the tests that you have outlined actually tell you what you're claiming they tell you. It's sort of an epistemology of your whole risk analysis.

And this is a tricky one to me as somebody who has built a bunch of these workflows and simple agents and stuff. You always get these caveats in work from METR and so on, where they're like, “We built some scaffolding to try to figure out what scale of research engineering task a model could do, but we don't really know that we did a great job.”

We don’t know what the limits are. The scaffolding could almost certainly be improved. And maybe I’ll just couple that with an actual quote from the bill: “A large developer shall not deploy a frontier model if doing so would create an unreasonable risk of harm.”

So I guess where the wrangling ultimately is with something like this is: What’s a reasonable or unreasonable risk of harm, and to what extent must people go to demonstrate that they’ve pushed the scaffolding and really elicited the capabilities to the fullest, knowing that that’s hard and that the state of the art is very much evolving? How does somebody know if they’ve done a good enough job that they’re on the reasonable side of unreasonable?

The short, totally unsatisfying answer is that often in law we use the sort of reasonable-person standard, right? There are a lot of things where we’re not going to be able to exactly specify everything that needs to be done. We can sort of point at as close as it is, but you sort of leave a little bit of difference to what a reasonable person would do. That’s not invented in this bill. That’s not invented in law. That’s a well-established legal standard.

I think the longer answer, though, is that anytime you’re writing a bill on anything, but particularly something as fast-moving as technology, and in particular AI at this point in time, you have this tension where people rightfully want specificity about exactly what it’s telling you to do, and on the flip side want it to be able to evolve in time, because exactly what you should be doing right now will change in 6 months, in a year, in 2 years, et cetera.

That’s always a balance, right? No law is ever final. The legislature can obviously come back and make changes at any point, so you don’t need to write it so that it lasts 1,000 years. At the same time, you don’t want to be deferring so much that companies really don’t know exactly what’s required of them. Any bill is going to have that tension, and you’re going to find the balance somewhere.

I think a sign that we found the balance in a pretty good place on this one is that we have about an equal number of comments on both sides of it: “I want more specificity,” or actually, “I want less government telling me exactly what to do.” So we’ve probably hit it right, but you’re pointing out that exact piece. That’s why, at the start, when I was describing the bill, I tried to emphasize for people that we are largely letting the companies grade their own homework.

We are largely saying, “You put out what the standard should be. Write it ahead of time, when you’re not in real economic pressure, and then grade it against that.” The only real pushback on that is, A, we have a third-party audit, and that audit is going to include: Did you actually follow this? Did you follow best practices, et cetera? And then, B, this reasonableness standard. If you write a plan that just says, “We don’t care about safety. We’re not going to do any of this, and this is the standard,” well, that’s pretty clearly unreasonable.

How about the relationship between all of this and open-source releases? At the sophisticated-actor level, obviously anybody around the world can download a Llama model or any open-source model. There was also one clause I wasn’t quite sure how to interpret, but it referred to modifications, which, in at least some other debates, has been understood to mean post-open-source-release modifications that who knows who might make.

Maybe with either a close reading of the bill or just your intent: How does this apply to somebody like Meta, who is going to potentially release a behemoth version of Llama 4, which I think would probably get to that 10^26 and I’m sure would be north of $100 million? I mean, they’ve clearly spent $100 million, whatever. It seems like they’re going to be in. We don’t really have great ways, as I’m sure you’re well aware, to really control what people do downstream once a model is released. So are they on the hook for that or not?

The bill’s agnostic to whether you open-source or close-source, right? The best way of balancing this is not saying specifically open-source or close-source, but just saying: Think about the risk, think about the use case, and make your own judgment with that.

There are many ways to keep a model safe, right? You can have it on your platform and monitor it at all times. You could go a step further and do know-your-customer, and only release certain features and certain things that are really powerful to people that you trust, right? That becomes another way of managing the risk.

If you opt to open-source it, that’s great, too. It helps to encourage academic study and analysis of all these things. But any of those choices come with risk. I find it bizarre when people say, “Oh, we should evaluate—we should take a risk-based approach,” which is what companies always say: “You should evaluate everything in its context,” except open source. “Ignore that context. That context doesn’t exist. Just write all of that off.” I think we are not in any way targeting open source. We’re just saying it’s up to you to make your choices based on your risk profile, and you should do that accordingly.

I think the vast majority of AI that has been released, it’s great that it’s been released, and it’s been encouraging. You’ll note that this bill doesn’t actually require you to take in cyber risk, and I think that’s largely because it’s really already there and maybe that ship has sailed. But it’s up to companies to decide the risk and the way that they’re deploying it, and all of those decisions matter. So we leave that quite open-ended.

But to my point that legislation is always changeable, we don’t want to shut down that ecosystem. That’s not an outcome we want. If that’s where it’s trending, we can change this. So I think that’s an intellectually honest position.

And I do think this is intellectually honest, which is not something that elected officials are often accused of, so I appreciate that. The normal way that people try to get out of this is some sort of denial or cope or whatever, but it does strike me that, in taking a neutral approach with respect to open source, it does make open source harder.

It is much easier, let’s say, to manage risks if you have a closed-source model where you don’t release the weights. If you do release the weights, you just really have a hard time, in many, many ways, even knowing or being able to predict what will happen, let alone controlling what will happen downstream from there.

So it does seem that this could create real risk for a company like Meta that’s trying to evaluate and might steer them toward not releasing if they’re like, “Geez, we have basically no known…” We can train this thing to refuse, and we can put out Llama Guard. We can do all these different things to try to enable people who want to do the right thing to do the right thing and set them up for success, but we really can’t prevent somebody from untraining that refusal behavior or just not using Llama Guard or whatever, right?

In that analysis, it sounds like your sense is basically: The risks are the risks, and if you can’t do it or we don’t have the right techniques, then maybe you just shouldn’t put it out. And that is kind of the reality.

I would say 2 things to that. I would first say I start often with a question: Do you believe that there is ever any information or any capability that should not be open-sourced? I think most people would say really detailed analysis or the ability to produce really powerful bioweapons should not be open-sourced, right?

Whatever your threshold is, do you believe in classification at all? Do you believe in restrictions on weapons at all? Is banning the sale of nuclear weapons reasonable? Just as long as there is some level that you think should not be open to the public, all we’re saying is declare that level and then go from there.

I’m not putting a specific level out there. If people can, with a straight face, say, “No, I think every capability and every power should definitely always be open source,” then you’ve already sort of accepted, “Hey, we’ve got to think about the risk here.”

The more specific thing I would say is I don’t think this is going to change behavior because the threshold for critical harm is 100 deaths or $1 billion in damage. I think the leadership of every company is probably comfortable saying, “We don’t want our products to cause 100 deaths and $1 billion in damage, and we’ll take actions that will stop that from happening.”

Right now, none of these products, as best I can tell, really reach that threshold. So we’re not talking about restricting any current behavior. But whether this bill exists or not, I would hope that the board of a public company would be comfortable saying, “Yeah, our policy is not to cause 100 deaths and $1 billion in damage.” That’s all we’re asking them to do.

Yeah, it’s going to be really interesting to see how this plays out. I think we’re 1 to 2 model generations away from—certainly, if you listen to Dario and Anthropic’s timeline—models that would be, in a very meaningful way, able to make some sort of needle-moving contribution to the creation of a bioweapon.

I would definitely agree that Zuckerberg doesn’t want to have to face the public and say, “Yeah, we shipped it even though we thought maybe it would cause a pandemic,” or whatever.

But it is really going to be tricky because, first of all, there are a lot of people who—not necessarily in their role as a corporate executive, but there are a lot of people who would say, first of all, that threshold isn't that high. You know, how many people die on the highways every year? It's literally 100 people who die a day on American roads. So, okay, it is a big world and 100 deaths is a tragedy, but it is also 1 day of U.S. road deaths.

And specifically, 100 deaths or $1 billion from a chemical, biological, radiological, or nuclear weapon, or something that is already a crime in the Penal Law, right? So it's not talking about accidents. It's talking about automated crime or CBRN. I don't want to use the word intentional death, but it is one that is caused by a crime or a big weapon. That is not a thing that you see every day.

Yeah. It's almost like—what's so weird about a lot of these things is that the pandemics we really worry about cause a lot more than 100 deaths. It's an expectation sort of thing. If COVID caused, whatever, 10 million deaths, then, in expectation, a 1-in-100,000 chance of a COVID-like outcome would lead you to an expectation of 100 deaths. And that's just a very weird epistemic position. Basically, nobody has the clarity—we just don't. Again, to quote Dario, we don't know why these things do what they do half the time.

Totally. So we're in a really weird spot where it's very hard to give an assurance that's at five nines on anything. And we do have existence proofs that one of these things can easily get to 10 million deaths and, obviously, could have been a lot worse. So we're in a very weird epistemic position.

Yeah. I think it is worth taking all this stuff very seriously. And I guess my hope would be that it really pushes people to invest hard in areas where we haven't got answers yet, right? I mean, the real thing is: can we create a model that doesn't know about virology? Is there some way to wall off that kind of knowledge and excise it from the version that gets released?

Interpretability techniques or otherwise, is there some way that we can say, with an affirmative safety case, “Yes, we can be confident we're being reasonable here and we know this,” not just because the thing refuses 9 times out of 10, but because we have a much deeper understanding of what's going on.

But if I had to guess, I think we are probably headed for a moment. Tell me if you would see this differently, but I would expect a sort of 2026 reality to be like this: this bill gets passed, Llama 5 is trained, we don't have that affirmative safety case yet, the risk is kind of on the unreasonable side, and unless Meta is just willing to run the risk for whatever reason, they probably have to look at this and say, “Can't quite release it in this form. We either need to solve some technical problems we haven't solved, or we just can't put it out there because it's just too powerful.”

And maybe they would even come to that decision on their own. You know, that's my point: I think we're talking about a bill that requires people to do some work upfront, writing things down on paper, knowing that their homework is going to be checked, and that the fines are in the 8 figures. I mean, they are going to make these decisions separately on societal-level large risks.

So I guess, in the spirit of red-teaming the bill, as I understand it, the maximum penalty for a repeat offense under the bill is a $30 million fine. And, you know, I think they paid Trump off at that level, right? We've seen multiple people just sort of say, “Forget it. We'll just settle this lawsuit because we just want to get this guy off our backs.”

Then maybe one way to soften the bill while still getting all the benefits that you're wanting would just be to require the plan and be a little bit less on the downstream-enforcement side of what was reasonable or not reasonable. Do you think that that could be a version of this that, if pushback or whatever dynamics ended up being conducive to it, is something that you think would be a viable possible compromise at some point in the future?

I think the version that we have right now is the result of many, many compromises. And I want to be clear that I started this by looking at all of the public debate around last year's regulation and probably accepting 95% of the public critiques.

Then I sent a draft of this bill to 5 or 6 major labs. I asked for red-teaming and asked for red lines on the bill. I got them back and did another draft around December. I sent it again to 5 or 6 labs: “Hey, here's a new version.” I got more feedback, ended up talking to a lot of people in the state, and that's why this bill was published in March. But it's been circulating since probably July or August of last year, getting a lot of this input and feedback.

So this is in no way the first stab at it. This is in response to a lot of industry feedback and a lot of compromise. And I think if you see other people saying, “Well, we need a compromised version,” this has been the compromised version.

But on the specific thing of just releasing the plan, you still need to define what the plan is, right? There need to be some standards. You can't have someone have a 1-sentence “We're building the AGI,” and that be the plan, right? So you've got to put some standards as to what the plan ought to be doing.

I think the third-party audit is incredibly important. That was part of the voluntary commitments. That was part of the regulations that came out of the Newsom commission after SB 1047 embraced it. It was part of the EU version, right? Third-party audits, I think, are really core to the bill.

Whistleblower protections, I think, are extremely core to the bill. It's sort of the 1 part everyone agrees on and is moving in every state. And then disclosing critical incidents is crucial just to keeping New Yorkers safe.

So I think all 4 parts of that bill are pretty required and pretty drummed down. The part that I think you're pointing out—the “don't release a model that has an unreasonable risk” part—is just that they're largely grading their own tests.

This is like the smoking companies: once they know it causes lung cancer, they need to proactively take action. Oil companies, once they know it causes climate change, need to take proactive action. When your own tests say this is going to cause deaths, you need to take action. The fact that it's 100 deaths, I think, is part of the compromise.

Okay. Maybe just a couple of double-clicks on several of the issues, since we're going deep here. How do we not have regulatory capture of the auditors? This is something that I experienced once upon a time in the financial services industry, and I've had a number of these kinds of model-testing organizations on the podcast in the past: Apollo Research, folks from METR and Palisade, FAR AI, and more.

All these folks have a tricky position. I know them personally to some extent, and I would say they are very sincerely motivated by a safety mission, but they are also very mindful that their access and ability to do their work at all is, at the moment, at the pleasure of the companies. So they're very cautious about how they speak publicly, and all this kind of stuff is very, very carefully thought through because they don't want to offend somebody and get cut off.

So I don't think the bill has much on that yet. Is there any plan for that problem?

I think that's one of the ones where we don't want to legislate ahead of time on it, but it's definitely a thing I'm concerned about. I mean, the only requirement is that it's a separate auditor. It's not a government agency.

I think of it as pretty similar to the SOC 2 process, right? You have consultants, these auditors that are set up to evaluate your security stance, but they take into account the size of your company, the risk, et cetera. It's not sort of a checklist of hard things.

I'll point out that all of the companies that would be subject to this already require SOC 2 of all of their vendors. So it's a similar process that they engage in. But you're right: when you have this kind of open market and government is not licensing the auditors, there is always that chance of regulatory capture.

We require the auditors to follow best practices. We require that we know who's doing the audit and all of that, but it is a thing that we will monitor over time. I hope not to have to take more action, but it's certainly a real risk.

Okay. On the safety incidents, as I was reading through the different things that would qualify, various incidents kept coming to mind, and I was like, “Would that qualify? Would that qualify?” So I don't know that you can officially judge, but I'm interested to get your reactions.

Yeah.

A couple of lines from the safety incident definition, clause A: “A frontier model autonomously engaging in behavior other than at the request of a user.” And here I'm like, I can point you to a lot of people who have reported that Claude changed the model from OpenAI to Claude. Just yesterday, a friend was like, “Basically, I vibe-coded my way onto the project maintainers list because I ended up modifying the database in ways that I wasn't even meaning to, but the model just got blocked one way and went another way.”

I mean, I think we're honestly very confused, broadly, as a field, about autonomy. Should we want it? I'm not so sure that we should, but we're clearly pushing for it. And I guess my sense is, maybe that clause is being triggered a lot in the world today.

Do you feel like that may in fact be the case?

It might by itself, but remember that all of those 4 specific incidents have to be only if they’re increasing the risk of a critical harm. And so, if it’s a minor thing that pops up and you’re not—this isn’t going to increase the risk of 100 deaths or 1 billion in damage—that’s not something to report.

If an employee forgets to log out at the end of the night, but no one comes in—the janitor saw the code—but that’s not a thing. But if China steals the code, that is a different sort of circumstance. So it’s taking into account whether it would cause a real risk of harm and, largely again, relying on the company’s judgment of that, but saying that things that do rise up to that level of these specific 4 need to be disclosed.

Yeah, it’s easy sometimes, as you get down the nested structure of these bills, to forget the sort of top-level clause. So it’s a good reminder that all of this stuff is in conjunction with this increase in the aforementioned critical harms.

I guess a couple of other incidents that came to mind would be examples of companies not following their process or mistakenly releasing capabilities. Specifically, Bing famously launched in India and a couple of countries, I think, without going through the safety board approval process that they and OpenAI had together agreed on. I think some people at OpenAI were also involved in saying, “Yeah, go ahead and do it.”

In retrospect, all the Sydney behavior was in fact reported on the forum, and they missed that as well. Would something like that qualify? I mean, that model was only GPT-4, so it’s maybe not at this critical harm level, but would something like that qualify? That seems like it would in a future scenario where the model is more powerful—you can’t do that.

Yeah, it might. I don’t, for a legislator, have a deep background in these things, but I don’t claim to be the foremost safety researcher. I want to defer some judgment to the people who are doing this every day, and that’s what the bill is meant to do.

So, with that caveat, and my voice here not necessarily being binding on it, I think if you temporarily enable a feature and you’ve monitored the platform that whole time, so you know if it was used in any way, and now you’ve turned off that feature, did that actually introduce a real risk? Probably not. Again, it depends on the exact feature and how much you can see into how it’s used, et cetera. But when you’re playing with fire, it’s a different standard than when you’re playing with a smaller feature.

Yeah, we’re definitely playing with a new kind of fire here. I say that all the time.

Okay. Whistleblowers. So, I’m a big supporter of general whistleblower protections. How strong do you understand these protections to be?

Super weak. Yeah.

Okay. So, one key question is: if you go to the AG, can the company fire you for doing that?

It would be illegal under this law, and on top of any other laws, you would also be subject to this $10,000 fine per employee per retaliation, as well as injunctive relief. So you might have to hire the person back. But $10,000 is nothing compared to the resources here. Personally, I would love to see that be a lot higher and a lot stronger.

Every bill exists in the context of its state and the state law, and $10,000 is the standard in New York across a wide variety of fields. I think there was even a push last year to increase it to a higher amount for employers that have violations of child labor laws, right? But that sort of got beaten down. It’s maybe seen as a Pandora’s box of doing anything above that $10,000 level.

So, I think probably you want a stronger incentive there, but you want to make sure, at the very least, that there are statutes on the books that if you are firing someone for raising catastrophic risk, that is illegal and that there can be action taken for that.

And in terms of injunctive relief and hiring somebody back, that seems hard in the sense that you’re not going to have the AG sitting in on meetings at the companies, making sure that this person’s job is the same as it used to be. Obviously, things change.

I guess how do you imagine that playing out in practice? If somebody actually says, “I’m freaked out about whatever. I’m going to the AG,” and the company’s like, “You violated our trust. We have our own protocols. You should have followed them; you didn’t do it. Whatever. Either you’re fired, or you’re sort of banished to home-office status, and we’ll continue to pay you or whatever, but you’re not going to be privy to all the things you used to be privy to.” How does that actually play out? What should a whistleblower expect in terms of actual material outcome or protection for them individually if they do violate the chain of command and come to the government?

There’s good news and there’s bad news here. The good news is that there is a lot of case law and statutes around whistleblower protections because they exist pretty broadly, covering a variety of actions in New York State but also throughout the country for reporting anything that is illegal behavior.

So it’s not just that you have to be rehired. It’s that your job is protected, your responsibilities are protected, and if any of that changes, there can be injunctive relief. There can be follow-on things from that. We see this—I come out of the labor movement, and you often see people being fired for organizing or for labor violations, and then you have the ability to restore that.

So the good news is, much of this debate is out there and the system functions already. We’re just adding on something that might not be explicitly illegal but is a catastrophic risk. That’s the only change here.

The bad news is, yeah, I think broadly in society, labor protections are not as high as I would like them to be, and I can’t promise the system functions exactly as it should. I think this is part of a larger conversation that should be had, maybe separate and outside from this bill, and one that I would love to have. But we’re not largely changing whistleblowers in New York because there is so much on the books. We’re just adding that bit about catastrophic risk.

Yeah, there have been a couple of calls recently for class consciousness among the research engineers at the leading AI developers, and it’s a weird dynamic because they are seemingly, increasingly, in a very self-aware way, trying to automate themselves out of a job. The vision seems to be increasingly explicit, as far as I can tell: get the AIs to do the AI research, and then hope that we can steer them or hope that we set the initial conditions right. All that, honestly, is pretty scary to me.

How about internal deployments? As far as I can tell, that is not part of this bill. But the bleeding edge of policy discussion is turning toward internal deployments. I think it’s for this reason: there’s an expectation that the gap between what companies have and maybe are using for their own AI research and what the rest of us plebs in the public get to see and use might widen pretty dramatically if the companies are all locked into this sort of game-theoretical race to be first to AGI.

So, first, am I correct that that’s not really addressed here? And second, is it on your mind for possibly something to come back around to?

Actually, internal deployments would be covered in most cases here. So, the definition of “deploy” at the top includes using the model as well as making it available to others. If you are using it even internally, then it is covered by this.

Now, we exempt anything that you are testing, developing, and evaluating; that doesn’t count as using it. Additionally, we exempt using it to comply with state or federal law, or if it’s part of a broader federal project. Those things are already exempt, but general use that isn’t in one of those categories would actually be covered and trigger the requirements in this bill.

Yeah, that sounds to me like the most likely place where whistleblowers might end up feeling compelled to come forward. The idea is that what exactly counts as use, and we’ve got this sort of guardrails-light or guardrails-free, purely helpful model that we have access to internally, and the controls are not great, and people are asking it to do AI experiments, and the whole thing has a sort of gain-of-function-type vibe to it.

So, yeah, that’s really interesting to think about, but that’s a really good clarification, and I do think an enlightened one to include at this stage.

I appreciate that, and I think it goes back to part of the conversation we had: how is this field going to develop, and do we really expect the most dangerous models to be released open source? We’re already seeing companies move in this direction, and whether this bill exists or not, this isn’t changing liability for things that happen after you release it, right?

So companies are already going to start to make that decision: when we get really powerful, how should we be thinking about what goes out there? This is just meant to improve your internal stance and your safety planning and everything that goes into that ahead of time. And I think you’re right. I think more of the risk is coming from internal deployments now. That’s why we wanted to make sure it was covered.

Cool. A couple of final questions, and I really appreciate all the time. This has been great.

Obviously, one of the big concerns—and really the sort of nominal concern I’ve been hesitant to even bring up in previous legislative battles, state-level legislative battles, because I feel like some of them became so toxic and needlessly so—is that it’s like, let’s just leave that in the past and tackle this new chapter with a fresh start or a blank slate or whatever.

But one commonly raised concern that was sort of the final presented concern last time around was that doing this at the state level is just not great because, obviously, we've got a lot of states, and in the end, the final worry is always China. So it's like, well, if we have 50 different state rules, then we'll never be able to get anything going, and then we'll lose to China. It seems like you've thought about that and have tried to take that into account as you've developed this.

I wonder if there's any sort of possibility of a compact of states or some sort of reciprocity between states that could neutralize that objection, which I do think has some reality to it, although I think it's also kind of a smokescreen at times. But it would be good to get the best of both worlds if we can. Do you see any prospect for that?

Yeah, I love the idea of a compact, and I would say usually the best smokescreens have a little bit of truth to them, and that's a good description of this right here. I think I agree it should be done at the federal level. Come back to me when it is. Federal laws already override state laws. If any member of Congress wants to take my bill and do it at the federal level, please do so. I'm really happy to partner with any member of Congress or any staffer who's listening to this.

But that objection holds a lot more weight when, A, Congress is doing anything, and B, once one state has passed something. No state has passed anything yet, so there's no conflict to be thought of. And we're very good at focusing on making things standard across the states—not 100%, and that's why there should be things done at the federal level.

But I can say that I'm aware of maybe 6 or so states that are pushing forward frontier-model regulation of some kind, and we're all on a group text. We're already talking and thinking about it, plus possibly a few reporters or whatever that you added in The Atlantic, as far as I know.

I guess it was just a slip of the finger. Mute that thread.

You guys added me, and it was just so crazy.

Yeah, honored to have you as part of it. But yeah, so we're already talking, and it's got to be established somewhere first. I think all of us have shown that we're happy to copy other places. I have other bills on protecting more short-term consumer aspects that are based on what Colorado passed because, hey, they got there first. Okay, let's try to unify this.

As I said in the intro, I worked in tech before I entered this field. I know what that compliance looks like, and having one standard is way better. So let's work towards that. But before we start trying to draw a line between 2 or more points, let's get that first point. Let's get a state on the books, and then we can talk about how it should be the same with any further ones.

Okay. What's the process from here, and maybe how are the dynamics shaping up? Gavin Newsom once said that the bill that shall not be named for the moment had created its own weather system. You seem relatively relaxed. Maybe that's just your good humor, but are we seeing anything similar where people are sort of mustering all their force and kind of bringing the Eye of Sauron to this debate, or is it going to be a little more chill this time around?

No one here is the Eye of Sauron. Let me be clear. I especially need to say that as someone who worked at Palantir, anytime you get into the Lord of the Rings references. But no, I think I've been very intentional about engaging with industry and with industry associations. I want to hear their feedback. I want to see their red lines. I've gotten many versions of it and incorporated probably 80%–90% of what they have sent.

These labs are not the enemy. As I think I said at the start, I want to see more adoption of what we have already. I want to see where this field is going. The incredible potential of what we are going to have—from medical discoveries, from routinizing the monotony of life, from even cyber defense, from what's going forward—is really incredible technology. I want the guardrails to make sure we get it right, and those guardrails work best when they're done in conjunction with industry.

So this is not in any way a fight. Again, politics is often painted that way, and what gets the press are the things where it seems like a fight. Then there's an incentive to design it as a fight. It doesn't mean we agree on everything, but I think this has been a very collaborative process so far.

The second thing I'll say is that the rate of collaboration is probably about to speed up quite a bit. Our legislative session ends on June 17th, so we're a little more than a month away from when this has to pass. I think there'll be an intense focus as we get closer and closer to it. I think it's really important that people not sit on the sidelines as part of this.

So, if you're listening and you have suggestions to improve the bill, email my office. Call my office, right? Bores@nyassembly.gov. We take all input. I would love to hear it. If you live in New York—and that's New York State, not just New York City—I would encourage you to reach out to your legislators and say, “Hey, I support this bill, and why don't you hop on as a co-sponsor?”

We've made that easy, actually. If you go to bit.ly/raiseactny—all one word, all lowercase—that will give you a form. You enter in your address, and it'll generate an email for you to send to your assembly member and your senator.

If you're not in New York, or even if you are but you run a company or an organization that has a unique view on AI, you can also write in what's called a memo of support—something that says who you are, why you care about this, and why you think the state legislators should take action. You can go to bit.ly/supportraeny, again all one word, all lowercase.

Government is not a spectator sport, and decisions are made by those who show up. I assure you, industry is showing up, and I welcome that. If you're someone who listens to this podcast, you probably already have an interest in this field. I'd encourage you to show up as well. There's a lot of action you can take right now to really influence what frontier-model AI regulation looks like in the United States.

Cool. That's great. I've noted those URLs. We'll include them in the show notes. Thank you. I don't know if you like to handicap your own bills, but it was striking that the vote in California was very one-sided and then, of course, we had the governor's veto. What do you think will be the hard parts, or are there going to be any obvious hard parts between here and making this a reality?

Yes, there will be hard parts. It's like any bill. It's got to pass both houses of the legislature. It probably has to go through multiple committees. It has to pass each committee individually before it even gets to the floor, and then it has to be signed by the governor.

But I think one important and different thing about New York versus any other state or the federal government actually comes with that governor step. I'm skipping ahead at this point, but I think this is useful context for people to have. If we are successful in passing it, in every other state and in the federal government, once a bill is passed, it goes to the executive, who can either sign it or veto it, or do nothing, and that's interpreted as signing it or vetoing it.

In New York, there's a third option called a chapter amendment. What that looks like is the governor negotiates with the sponsors of the bill over changes that they may want to see. They might say, “Oh, I like the bill, but can you change A, B, C, and D?” And the sponsors say, “Well, we can change A and B.” Then you reach some agreement to change A, B, and C.

The governor will sign the bill at the end of the year with a memo that says, “I'm signing this pursuant to an agreement that I've reached with the sponsors,” and then we'll introduce the amendments early in the next session and pass them. I bring that up, A, because the version that passes in June doesn't have to be the final version, right? It doesn't have to be this or nothing. If you're not 100% on board with every clause one way or the other, you can still participate.

But B, if there are rapid changes in the field between June and December and we all agree that those changes have come to be and we need to act on them, we actually can. We're not locked in. We can make those amendments.

One of the things I've talked to industry a lot about is a standard definition of a frontier model. I keep saying I'm happy to change this, right? We know what we're trying to capture, but is the threshold exactly right? Should it be different in this way? If everyone in industry, if everyone in academia comes together and says, “Hey, this is really what the definition should be,” we're happy to swap that out and put that in there.

Everyone should keep that in mind. Different from California, we have that additional flexibility as the year goes on.

Cool. Well, I think if there's one safe bet we can make in this process, there will be some developments in the next 6 or 7 months. My guess is there will be something that will at least challenge some assumption or provoke another round of discussion. That seems like a safe bet.

This has been great. Anything else you want to share or leave people with before we break for today?

I really encourage people to get involved. I mean, this is a field where your voice—you, the audience, as people who are thinking about these topics deeply, at a time when most legislators throughout the country aren't, just because it's new and it's not most people's background—really makes a difference.

Especially people who are deep researchers, academics, or engineers who really want to be precise in your language and in what's going forward. There's a hesitancy, I think, to be involved in politics that is often about speaking with sweeping statements and maybe not always seen as, to steal your phrase from earlier, intellectually honest. But I encourage you now: embrace that nuance. You can give specific feedback. This bill isn't my baby. You're allowed to tell me things to fix it.

But it's really important that you express a desire for something to happen here because there are definitely people with an economic incentive to say no to any regulation. And if the reasonable people that want some regulation don't speak up, don't send that email, don't send that memo of support, instead of getting something you agree with 80 or 90 percent, you're going to get absolutely nothing. And so what I want to leave everyone with is: take action. Tell me all the ways you'd improve the bill. I would love that. Amendments are all there, but then please, please, please speak up in support because we have a real chance to make a difference here, but only if everyone gets involved.

Great. Well, the bill is called the RAISE Act. In my humble opinion, it's not really all that much to ask. And I think with your background in technology and obvious technological literacy, you're a great avatar to represent what are some pretty modest requirements to the public. So, thank you for taking the time, New York Assembly Member Alex Bores.

Thank you for being part of The Cognitive Revolution. Thanks for having me.