私人治理：在AI监管中创建市场——与Gillian Hadfield博士和Andrew Freedman对谈

Today we're kicking off a short series on creative AI governance proposals, and I'm speaking with Dr. Gillian Hadfield, Bloomberg Distinguished Professor of AI Alignment and Governance at Johns Hopkins University, and Andrew Freedman, co-founder and chief strategy officer at Fathom, about their proposal to govern AI via private regulatory markets.

AI is, to put it mildly, a hard technology for society to effectively manage. The relentless march of capabilities, the radical uncertainty about how powerful AI systems will get and how soon, the feverish pace of adoption, and the increasingly intense international competition combine to create huge stakes, but still very little clarity on what should be done. There are legitimate worries that even the most tech-savvy policymakers could easily get things wrong.

And yet, while highly prescriptive government regulation of the sort that Europe is attempting with its AI Act doesn't seem to me likely to meet the moment, the fact that xAI can credibly claim frontier capabilities even while Grok 4 continues to self-identify as Hitler suggests that a less laissez-faire free-for-all won't serve us well for all that much longer either.

Is there any way to create a governance regime that's agile enough to keep up with AI developments, sophisticated enough to address the most important and extreme risks, and yet not so burdensome that I'll still be able to have my AI doctor?

It's a hard problem, but Dr. Hadfield and Andrew have a very interesting proposal to harness market mechanisms and hopefully create a race to the top in AI safety standards. It's been introduced into California's legislative process as SB 813, and from what I hear, it does seem to be gaining traction in a number of red states as well.

The core idea is to separate the process of democratic deliberation about the outcomes we want and want to avoid from the detailed rulemaking process meant to get us there. In concrete terms, a government body—perhaps the California attorney general or perhaps a newly created AI safety board—would articulate goals like “AI systems must not enable the development of bioweapons,” or standards like “autonomous vehicles must be safer than human drivers,” and then create a competitive ecosystem of private certifiers who develop the safety standards, engage with companies to make sure they're properly implemented, and report back to the government and public on the results.

Companies could then choose to work with these approved certifiers, and in exchange for meeting their standards would receive some level of liability protection when things still end up going wrong. Given the unpredictability of AI systems generally and the unsettled nature of AI liability law, that's a serious incentive that would presumably convince many companies to opt in to participate in the system.

As a lifelong libertarian, I really like the idea of trying to bring market dynamism to AI governance. And I appreciate that, while this idea is new to the public now, Dr. Hadfield has been developing such concepts for decades, even working with Anthropic co-founder and policy lead Jack Clark on related ideas as early as 2019.

Andrew, for his part, brings invaluable practical implementation experience to the table as well, having worked as Colorado's cannabis czar while the state was rolling out a new regulatory system for legal marijuana.

Nevertheless, as you'll hear, I pressed them on several important concerns. How do we avoid a race to the bottom where companies simply choose the most permissive certifier? How would the liability protections interact with existing tort law? And what exactly are people giving up in terms of their ability to sue?

Do we have any organizations that could step up and do a good job in the role of private regulator? And who do we really have to trust to do a good job for such a system to work—not just in the beginning, but on an ongoing basis?

In the end, there's no silver bullet. Any governance system that we might design does ultimately rely on some number of people doing a good job in key roles. But I do come away from this conversation optimistic that an arrangement of this sort, if it could put the good folks at FAR.AI, METR, or other similarly tech-savvy organizations in a position of real authority, could deliver much more responsive regulation than the government could muster on its own, while also making sure that society is not flying entirely blind into the fast-approaching AI future.

Coming up soon, we'll have another episode with Professor Gabriel Weil, who has a very different proposal to address many of the same core concerns via liability law. So, please stay tuned for that and definitely reach out to let me know which of these ideas you find most promising or if there are other proposals that you think would be better yet. With that, I hope you enjoy this exploration of a proposal to harness market dynamism to effectively govern AI technology with Dr. Gillian Hadfield of Johns Hopkins and Andrew Freedman of Fathom.

Dr. Gillian Hadfield, Bloomberg Distinguished Professor of AI Alignment and Governance at Johns Hopkins University, and Andrew Freedman, co-founder and chief strategy officer at Fathom. Welcome to the Cognitive Revolution.

Pleasure to be here. Thanks for having us.

I'm excited for the conversation. You guys are working on some very interesting stuff. I'm always looking out for creative solutions to the vexing problems of AI that we have in the governance space.

Obviously, we're still flying pretty much naked here through this rapidly cresting technology wave, and I think you guys have a very interesting proposal. I understand that there's kind of a one-two punch that we'll want to keep in mind for this conversation: one being the general set of ideas, and the second—but also very important—being that this is now being introduced into the California state legislative process with an actual bill that will at some point either get revised or get passed, and hopefully maybe one day could come into law.

So maybe, for starters, tell us what you're up to. Give us the grand landscape of this private governance notion.

Yeah, I'll start there, because so much of the heart of this idea comes from Professor Hadfield. I'll talk about how Fathom got interested in it.

Fathom started just over a year ago, really on the notion that AI was going to, for better or worse, break a lot of things—governance being one of them. Obviously, a lot of the way that we interacted as a society, a lot of the way we interact as an economy, was going to change, and then we, as a society, were going to have to figure out how to put it back together.

Generally, tech policy has been left to tech to figure out tech policy. This was so much of a broader societal issue. How do we start not just a think tank, but an organization that could help raise the ideas up from society that are going to best fill the needs here, and then help build them? That's really what Fathom has based its mission around.

The very first thing we ran across when we went out and did tons of polling and qualitative work, meeting with leaders across industry segments and society, was that everybody thought governance was needed here. Almost everyone agreed that the current ways of thinking about heavy-handed, top-down governance probably weren't going to work for AI, but simply leaving everything to society—or leaving everything to the labs—wasn't going to solve for society.

That began a journey for us: if those aren't the solutions, where are the solutions? It turns out that the professor has been thinking about this for a very long time and has some amazing thoughts on it. There were some other thought leaders, Dean Ball being one of them, who really thought that there was a third way to start thinking about this that was not generally brought up in public.

I'll leave it there in terms of how Fathom became interested in it, and toss it over to the professor.

Okay, Gillian.

Yeah. I've been thinking for decades in my career about how well our legal and regulatory systems perform. I worked on access to justice for a long time, and then started thinking about the way our legal and regulatory systems respond to technology and globalization somewhere in the mid-2000s, just recognizing that the systems we developed for making law and regulation, really starting in the 19th century, were no longer fit for purpose.

They didn't keep up with the complexity, the speed, and the multijurisdictional nature of the world we live in. AI just ramps that up several levels, right? So we have this real mismatch between the way we make law and regulation and the way technology now moves—the speed with which it moves and the complexity with which it moves.

I started thinking, “Okay, so how do we need to adapt our approaches to building that regulatory infrastructure for a much faster-moving, complex, and now AI-based world?” That's when I started thinking about how we get markets involved more in figuring out our regulatory problem.

As a starting point, it's really important to recognize that regulation is actually the thing we build markets in. It's not something that's just dragging down markets. I started writing about this quite some time ago as well: the idea that our markets are built on good legal and regulatory infrastructure—contract, property, fraud, antitrust, all that good stuff—that allows people to invest and participate in markets with confidence.

So it was like, “Okay, if we have stuff moving at the speed of very rapidly adapting markets that are producing the technology, how do we get more of that market energy and investment into solving the problem of what's the best way to build that regulatory infrastructure for technology?”

So that led to this concept of regulatory markets, which is the idea that we still need our governments involved in setting what is the acceptable risk level for society, making judgments about what we will and won’t allow. But then the technical question is: How do we translate that into what companies, labs, and so on actually need to do? From a technical perspective, we need to get more market activity into that phase. We can talk more in detail about how all this works, but that’s really where it came from. Then Jack Clark and I wrote a paper in 2019, when I was on contract and he was a policy director at OpenAI, proposing that this was a model for AI safety. We’ve just been building on that since then.

That’s interesting. I didn’t know that tidbit—that it goes back to 2019 and that you were working with Jack at that time. I don’t know if it was Luigi Zingales who said this, or I forget where the source of the quote was, but I always remember this quote: “Markets are not free or unfree. Markets have rules, and some rules work better than others.”

Perfect. It’s a constant refrain: No such thing as a free market. I’ve been doing this my entire career. My PhD is in economics; I did it jointly with a law degree. The focus there is that there’s all this institutional structure. Economists assumed that markets just kind of existed. This was, honestly, after the fall of the Soviet Union and the shift from socialist economies to market-based economies, and economists just kind of said, “Get rid of all that government control over industry, and markets will flourish.”

And they did not, because if you didn’t have good legal systems for enforcing contract rights, property rights, and intellectual property rights, and good regulation, markets don’t thrive. So, fabulous quote. It’s exactly the right one. There’s no such thing as a free market. There are healthy markets that are well structured, with good legal underpinnings and the kind of regulation that makes everybody willing to invest and participate. Anyway, that’s what leads to a vibrant market.

We kind of know this around the world because we know that the countries that struggle on the development side don’t have good rule of law. They don’t have good legal underpinnings, and nobody wants to invest there. So I think that’s a really important observation.

Before we get into the details of the private governance structure and the way that’s instantiated in SB 813, and possibly some variations on that, let’s take a minute and cover what you see as the fundamental problem with either the top-down approach. You can characterize “top-down” as you will. Not too long ago, somebody like Sam Altman was saying maybe we need licensing for frontier models. Obviously, he’s backed off of that.

Something like SB 1047 was, in my view, fairly light-touch, but still had some elements of top-down governance in that there were statutory thresholds. I think the critics have been at least partially vindicated in that those thresholds haven’t aged super well, and it hasn’t been a super long time. So I’m interested in the problems with that, but also in the problems at the other end—why do we need new rules here?

Some might say we’ve got general rules for commerce. Is this really that different? Why should we think of this as being different from any other new product that somebody might bring to market? By the way, I don’t really buy that. Of course, I spend all my time thinking about this, but I’d like to hear your dismantling of that naïve notion.

Is that directed to me, from the point of view of thinking about regulation as a thing we’re trying to accomplish for healthy societies, prosperous societies, and fair societies? Top-down governance just becomes more and more untenable. If you’re setting the detailed rules, it becomes more untenable the more complex technologies are, the more quickly they develop, and the more jurisdictions they’re in, right? Because you’re setting rules that you’ve got to follow everywhere.

I’m an economist and a big fan of markets, but not for any ideological reasons. They’re good information-processing and discovery engines. They’re down at the ground level, responding to what’s happening in the trenches. You need that kind of information to figure out the right way to regulate—to capture the benefits of promoting innovation and getting efficient markets and so on, while at the same time establishing those ground rules that make everybody feel confident and willing to invest in and participate.

The problem with the top-down approach is that it’s very hard for governments to have access to that kind of information. We do have ways of getting it; we developed those over the 20th century, with chemists and people who have expertise in biology, forests, clean water, and so on. But when you need to move at the speed with which technology is advancing, you have this real mismatch. You need this information from the ground level, but then you have a process in our legislatures, courts, and agencies that just operates on a different timescale, and it’s very hard to keep up.

Of course, one of the things we’ve observed in that process over the last several decades is that it has, in many ways, gotten more ponderous. The length of our laws—they’re a lot longer than they used to be. Opinions out of courts are a lot longer than they used to be. It takes a lot longer to accomplish stuff, and that means you don’t revisit it. There’s just so much sand in the gears, and we don’t keep up very well. So I think that’s the issue with the top-down approach.

What you want to try to do is find a way to get all that intelligence from the ground level into your regulatory system without abandoning ultimate democratic control, because we as a collective need to decide what’s okay and what’s not okay. Are we taking this risk with autonomous vehicles, companion AI, or various algorithmic decision-making? We need to be making those decisions.

But can we separate out making those risk-adjustment judgments from the technical process? What do you need to know about how this works? What data should you train on? What red-team tests should you do? All that kind of detail. Can we get that? I think that’s the issue with trying to do it all top-down from government.

The only thing I’d love to dovetail to that is that my first job in emerging regulatory systems was in cannabis in Colorado, for the rollout of its regulatory system. That, in so many ways, is a much simpler policy than AI is ever going to be, and we already thought that touched every area of Colorado law at that time.

The strength of it—which, you know, I think had muddled success throughout the nation—was that whenever we set up a system that was more iterative, we were able to say, “Oh, here’s a problem. We didn’t see edibles coming up in this way.” Then we could change the edible rules and get there and understand.

I think the idea that at any one time we can predict where the AI system is going and create good guardrails that will make sense even 6 months later is wrong. The challenges are going to be huge, the opportunities are going to be huge, but our ability to predict the future will be very low. So part of what attracts me so much to the system is that it is a way to put independent subject-matter experts up front and allow them to continue to make decisions over time about what good looks like.

The second part I would put in that is I do think that if industry understands what good looks like according to independent subject-matter experts, and understands that that is going to be in some way universally applied to them, then that’s the brass ring they’re going to start reaching for.

I do think that when you create top-down measures, that tends to become a floor that then gets thrown to a compliance department. The compliance department says, “Here are all the boxes we’ve got to check in order to say we technically meet this,” but we’re not going to get in the way of the business unit, which is out doing its own thing.

So again, I think that structure is going to do more to make sure that lawyers have a job in the future of AI than it is to make sure that what’s happening is actually in the best interest of the public. Both Gillian and I are lawyers, so no shade thrown there.

So let's describe the mechanism. I'll let you do it. There are kind of 3 tiers, but the floor is yours.

Let me take a first hack at it and then have Gillian clean up everything. The fullest instantiation of this idea is in Senate Bill 813, but this is, I think, broadly going to be a conversation we want to have many times over in many different ways. I also think SB 1047 requires quite a bit of revision, so I don't want to get too in the weeds there, but there are 2 ideas there.

One is: how do you set up something like a regulatory markets mechanism to be able to identify third-party auditors, private-sector regulators, verifiers, and certifiers who really know what good looks like and what best practices are at all times? They would have the ability to prove to both companies and the government that they can actually verify claims and that companies really are meeting these best practices.

They would scope where their expertise lies in some way, saying, "We are maybe focused on AI as it pertains to chatbots or autonomous vehicles, and these are the sort of safety parameters we're looking at." An AI developer, application, or deployer would come in and say, "We're worried about the risk we're taking on by utilizing it the way we're utilizing it, so we want to be certified as meeting best practices." They enter into a process to be certified by these third-party groups as meeting best practices.

If they can show that, SB 813 would say that should be proof on the back end that, if something bad does happen, you've met a standard of care and some form of duty to the public. It should count as—"liability shield" is probably too strong of a word—but evidence in court that you did the best you should be doing and you're meeting that duty of care.

Meanwhile, the third-party auditors, certifiers, and verifiers are constantly going back to the California government and saying, "Here's proof that when we certify people, they're doing better in the world. Our certified autonomous vehicles are getting in fewer crashes. Our chatbots are causing fewer issues among teenagers."

"We are beating not only, maybe, the status quo of how it would be without us, but we are in fact performing better than other certifiers in this area. Therefore, there's some race to the top to be able to show that we should be able to keep our ability to certify, against somebody who cannot prove that they're doing as good a job in this field."

So Andrew's given you the kind of version of maybe a particular implementation of it—how it works. I think there are a lot of questions about how you get to this. It's a real transformation in the way we approach regulation.

When I first started working on this, as I said before, I was focusing on AI. A book I released in 2017 was talking about how we were going to need to change the way we approach regulation and add this kind of tool to our toolkit because most technology is moving very fast. We have very decentralized supply chains and so on.

I'm just going to talk about the more abstract version of this and compare it to that idea of the top-down version. If you think about just the sketch of what the cartoon of regulation is, government sets rules. Companies that are regulated by government have to follow those rules, and then government monitors to see if they're in violation and fines them or takes them out of business or something like that.

It's called command-and-control, or prescriptive regulation. People have been working on designing regulation to be more agile and adaptive for quite some time, and one of the developments there has been the idea of what's called performance-based regulation.

Under—let's just take the example of pollution control—the command-and-control version of that is the government says, "Here are the particular scrubbers you have to install in your smokestacks," right? It's a very 1970s version of a regulatory problem.

Here's the technology you have to adopt in order to achieve what we think, as the government, is the acceptable level of pollution. A performance-based approach to that says, "Okay, government's going to say, here's the acceptable level of pollution coming out of the top of the smokestack. You, factory, figure out what's the best technology to do that."

Different companies and different factories could use different technologies. Companies could arise that would help to develop that technology and say, "Here's a more effective, cost-effective one." But to be in compliance with the government, you had to reach those output goals.

So the way a regulatory market structure works is, okay, let's take that idea of government setting the outcomes. Here's the acceptable level of accidents for autonomous vehicles. We want to see any uplift in the capacity to develop bioweapons fall below a threshold level. That doesn't have to be a threshold that's set in numerical terms; it could be qualitative, judged by experts.

Our tort standards are actually outcome-based, right? We want to see companies take reasonable precautions to prevent harm. Then, instead of just saying to the regulated company, "You figure it out," you say, "Actually, what we want is to develop a whole sector of independent firms that are engaged directly in the project of figuring out the best way to achieve those outcomes."

So you have licensing that happens, or oversight that happens. Governments will license what I'm going to call regulatory services providers in different contexts and in different pieces of legislation. There are other ways of thinking about how we characterize them, but I'm just going to call them regulatory services providers.

They are licensed on the basis of demonstrating that their approach, their technology, and their rules achieve that outcome. The companies that you're trying to regulate—I call them the target companies—select a regulator from those in that market, from those approved regulators.

In the version that we originally proposed, and again in the theoretical framework, that was mandated: those companies have to select a regulator. In the pollution context, it would be that you can't come up with your own approach, but you can select a company that's been approved for this.

The idea then—and a key idea there, this is the market idea—is that we actually need a vibrant market where we're getting investment and attracting financial capital and human capital to the project of figuring out the best way to achieve this outcome. Is it red-teaming tests? Is it review of the data? Is it embedded officials observing the process?

We actually don't know the best way to regulate to achieve our goals with respect to AI. So I think that the basic structure is: government sets outcomes; you have an independent sector of companies that are specializing in developing the technology of achieving that goal—they could be nonprofit companies, by the way; it doesn't have to be for-profit—and then the third component is your target companies that you're trying to regulate.

Andrew was sort of appealing to the fact that we were pretty familiar with lots of private actors playing a role in our very complex regulatory systems today. Certifiers, auditors—I mean, lawyers play a role in that sense. They have professional obligations to keep their eye on what's happening inside the firm to make sure that it's law-abiding. And so we have accountants who are doing oversight roles like that.

Part of getting to that ideal version I just described is to say, okay, how do you start to move our existing markets more in the direction of providing greater oversight and providing more of the substantive content of what the lab actually has? That's what they want to know: What do we have to do to limit our liability or to be in compliance with government requirements?

So, how similar is this to other things that exist today? I'm not aware of any major sectors that have such a market today. I'm also not quite sure how core the idea of liability protection is, or if that is one of many carrots or one of many deals that could be made.

And then I'm thinking, how similar is this to, for example, things like the auto industry, where obviously there's a lot of regulation and a lot of standards? Do we, in fact, have—even if it's not necessarily through this sort of market structure—a similar deal already in place with car companies, where, as long as they're hitting certain standards, even if they were maybe just prescribed by an agency or whatever, they do, in fact, get liability protection from that?

Yeah, several compare-and-contrast points. The best I can say on it is this smells a lot like other things, but it is novel in some ways. And the novelty, I think, is important in a couple of ways. The one I like the most to compare it to is Underwriters Laboratories, or UL. If you pick up any consumer electric good in your house, it will have a little stamp that says “UL.”

The company was started during a world's fair in, I think, the late 19th century because they didn't want their entire fair to go up in flames. And so they brought in an independent subject-matter expert group to inspect all of these different shows that were going to happen at all these places and make sure that there wasn't a fire liability there. That group kind of took off, right? It found its way into a number of different arenas to be able to say, “We have a really valuable thing we can add to you,” right?

You don't have to trust every consumer product that comes your way. Maybe you should just require those consumer products to go get this UL stamp. About 100 years later, UL started to get written straight in, codified straight into law. I do think that there is a need to make sure that the public good is more directly instantiated into these private governance worlds.

I don't believe that, if we let it go, we will find that there's just going to be enough verifiers, auditors, and ecosystem out there that all have enough of a true north pointing to what is societal good to be able to solve this problem for us. Meaning, I think leaving this to the private side without any sort of government accountability to solve will probably do more to create things that cover the butts of the labs and less to make sure that what's happening is actually doing the most to protect the public good. So, I do think that, in particular, Gillian's race-to-the-top mechanism that she's putting in, which really requires some amount of accountability and sets the goals of these companies via public legislation, is a vital aspect to it.

I also believe we don't have 100 years to wait for these things to organically grow up inside these systems and find their way to it. This has to be—we have to take the lessons from UL and say, how do we supercharge that? How do we make sure it's accountable? How do we make sure it doesn't get captured by industry? And how do we make sure that its true north is pointing in the right direction?

Yeah, I think it's exactly right to say this is novel. You can't point to an existing, full-on, full-scale model already implemented in this industry. This is a proposal for something we don't have yet that I think we need. So, it's about being innovative in our legal and regulatory technology in the way that we are innovative in the underlying technology.

But there's lots and lots of things that kind of come up to the doorstep. Andrew has emphasized standards—entities that perform the function of identifying that we're going to create standards and requirements, and very specific technical requirements, and then those are going to get played out into the market. That's a market in the sense that Underwriters Laboratories and other entities—we have lots of standard-setting organizations, ISO, and so on—are nonprofit companies that are in the business of creating standards, sometimes very, very technical standards.

Then they can be adopted because they have a good impact on the market share for that company, or they get implemented and picked up by government, which says, “Okay, if you want to be in compliance with the California regulations on building farm equipment, you have to have followed the standards from this organization.”

One of the examples I like to give of something that gets kind of close, and it sometimes is helpful to think of as a model, is actually the regulation of medical devices and quality production in medical devices. Governments have said—and it's actually a consortium of 5 countries, Canada, the US, and I think—I’m not going to remember the other countries, but it's a consortium of about 5 countries—that each of the countries is going to choose their own standard for quality control in medical-device production.

The US has an FDA standard, and Canada uses the ISO standard. ISO is a nonprofit standard-setting body, but collectively they have produced an approved list of authorized certifiers of compliance with those standards. They have a rule that says, well, if you've been certified by one of these approved certifiers, then you can sell your medical devices in all of our countries in the consortium. So, that's some example of, like I say, we've got pieces of this.

I always like to emphasize, too, that this is part of the development of regulation. Often, the history of regulation in a new area starts off in the private sector. Securities regulation, for example, starts off with private organizations—stock exchanges—saying, “Here's our rules. If you want to participate on our stock exchange, you have to engage in this kind of disclosure.” The disclosure helps bring more people to the exchange, and then that gets picked up by government. It sort of develops on this private basis, gets picked up, and gets integrated into government.

Today, regulation of financial transactions has important roles for these private entities. For example, FINRA, the Financial Industry Regulatory Authority, is a private membership organization, but then it's overseen by the SEC.

The SEC actually approves the rules that FINRA uses to oversee its members. So there are lots of examples of this public-private integration in our current regulatory regime. They're much more complex than that cartoon I was giving you earlier—the command-and-control model where government sets rules and companies have to comply. It's really a very complex system with overlap.

So this takes it to the next level and says, let's really lean into that outcome-based role for government and try to get more market activity around what's the best way to achieve these regulatory objectives. I think your question, Nathan, also went to liability and carrots, and so I think it's important to keep these things distinct: what's the model, what's the regulatory structure we want to get to, and then how do you get there?

In the original proposals about this, as I mentioned, it was like, well, let's just mandate it, just like we mandate compliance with securities laws, health and safety rules, or automobile safety requirements. We could just mandate that you must purchase the services of an approved regulator. But, of course, it takes some time to get there. We need to evolve this market and build it. We don't have lots of players in the market, so it would be very hard to turn around tomorrow and say, "You must buy the services of an approved regulator" in some of these AI domains, because you just don't have the market there.

The vision behind something like SB 813 is to say, how can we move ourselves toward that? That's where you start to think, just like there's a market-based incentive to get the UL certification mark on your product, can we create an incentive that says, well, you don't have to come and participate in this structure, and you don't have to purchase these services? We're going to create that mark. We're going to create a mechanism for government to say, "These are approved providers of this service." But we're going to create an incentive for that, because if you do that, you will have the capacity to demonstrate that you've met your compliance requirement.

If you get sued, you'll be able to say, "But I was following the program. I was in compliance with the requirements of this approved regulator. The state had said this is an approved oversight body, verification body, or certifier." That could just be an argument in your tort case, or it could have a formal role in your tort case. You'd actually have a legal benefit that says you're now entitled to a presumption that you were in compliance with your tort duty, because the government started off by saying, "We're only going to approve those entities that are actually able to demonstrate that if you did what they required, you're in compliance with your tort duty."

I will say SB 813 and some of these ideas are almost like 2 complex ideas put into 1, and they do have interesting ways they play off each other. Some things I like about tort being the backdrop for why you would want to enter this are, 1, it does force the market to decide where the risk is. Some players get to come in and say, "Let's say we're certifying for something that the leading labs are just like, 'That is not a thing we think we'll ever get sued on, and it's not a problem we think will ever come up.'" Then that part of the market holds no value. There's a little bit of, where do you actually think the harm is going to come from? That forces everybody to get a little real on that.

Instead of there being 10,000 cases that are all kind of edge cases, it forces people to start saying, where's the brunt of the problem? Where do we really need to focus in order to work on that? I like that part of it. The second thing that's a little elegant about that is that tort as a backdrop has this fun feature of being both national even when it's not federal.

The common law just applies everywhere in the United States. For those unfamiliar with tort law, it's part of the common-law system that was brought over from England, and it was this idea that it's what makes us all whole in the background of everything going on in society. It differs from state to state, but there's a lot of commonality across states. Meeting your duty of care in 1 state can be proof that you're meeting your duty of care in another state.

At a time when I think there's a lot of fear that proposals are going to create a patchwork between the states, these 2 ideas combined can let you imagine a world where it creates national, maybe even international, private-side regulators who provide a very real good across state lines and, again, even internationally. There's a carrot there that doesn't complicate compliance. It actually starts to centralize and focus where we should be worried and how we can meet compliance goals.

So can you calibrate me on this? I guess this could obviously be set at various levels, and I understand there's a distinction between the more diffuse academic layer of ideas and the specific statutory proposal that's on the table in California.

I generally have the sense that if I get into a car accident, I can't sue the carmaker, but maybe in some cases I can. I don't know exactly. There's probably always a carve-out. Similarly, if I have an adverse reaction to a drug, I probably can't sue the drugmaker unless maybe they sent me a tainted pill or something. I don't quite understand what the limits are, even in the world I have today, with products I'm very familiar with.

I'm not sure how that translates to AI, where, to complicate things further, we've got very familiar things like an AI might drive my car, or it might make me a medical diagnosis and recommend a treatment, or it might become my romantic partner. I have no idea how to even think about what I would be—how does one taxonomize that? We can maybe leave that for part 2.

The first question is: what is the trade that SB 813 is proposing? How does that compare, and do you think it is the right balance? How would you revise it, if at all?

Yeah, great questions. First of all, not only do you not know that, but literally no one knows it. So far, in the world of technology, we're kind of skating to where we believe the puck is going here a little bit. Technology really has been fairly shielded from tort law. It comes up, but in limited situations.

I believe—and I think a lot of people would say, and I think even tech companies are starting to understand—that the agentic nature of AI, AI as being more than an algorithm that's going to say, "If you feed in this information, this is the information that comes out on the other side," but in fact being an actor in the world, is going to bring it into a world of liability that I don't think tech has been in before. The nearest proof we have to that is the Character.AI case, which just survived a motion to dismiss. That's the chatbot example of being open for tort law.

My guess is—and I think it's pretty reasonable to say—that the hundreds of state judges across the country are going to find various levels of liability for developers, deployers, and applications. The full stack of people involved in AI will have to start worrying about liability in a way that they didn't before. Consumers, on the other hand, will have options for suing in cases where something bad happens to them. They can say it really was all the way back at the model level.

Where SB 813 sits currently is that this is a rebuttable presumption, to get kind of nerdy about it.

That counts as evidence, but you can come in with any sort of other evidence showing that they didn't meet it.

Okay, if you really want me to get nerdy, here's the nerdy part. One of the elements of tort law is always going to be: did you meet a duty of care? There are lots of different ways of cutting up duty of care. Sometimes there's something called strict liability, where if it causes harm, it's your fault, versus negligence, gross negligence, or clearly doing something intentional.

There are a lot of different theories of duty of care that come in. Then the question is, whatever you fall under, did you meet that duty? None of that has really played out in AI at this point. Ours is fairly light-touch as it sits right now. One way you can think about it is: whoever wanted to get the certification, were they acting negligently? Our proposal would say this should stand as a good amount of proof that you were not acting negligently.

If you can come in with proof that they were acting negligently, that can counteract the presumption at trial. I would say we're very open to where this should move the needle based on stakeholder input. I think our initial goal is to figure out what's enough to bootstrap this regulatory market system so that it starts getting people involved and bringing them to the table.

I do think tort law offers a lot of what I call potential energy, meaning there's a lot of places where harms could make companies do the right thing. So far, it has not come out as kinetic energy. So far, that all remains theoretical. 10 or 15 years down the road, I do think SB 813 does a lot to say, you should think about it now, and you should reach for best practices now to be there.

Let me be the law professor here and just sort of the much more abstract thinking about tort law. My answer to my students when they would say, "Well, can you sue?" I say, "You can always sue," especially in tort. You can always sue.

Exactly. And that's actually a part of the tort system, as part of our common-law system: the fact that it is entirely court-based and judge-based. It only evolves out of the cases that people have brought, and there are lots of things that can happen once you get there.

But you can always sue. You can always get in there and claim, “I was harmed. The defendant is the one that caused my harm, and—let’s just stick with standard negligence—it was because they didn’t take reasonable steps to prevent this harm to me.” It’s not that complicated. It gets much more nuanced in case law and so on.

But that’s also part of what I think people think of as, “This is the way we evolved our law through the 19th and 20th centuries,” right? You evolved the law because the courts were there and people could file their suits and so on.

The part that you were asking about is the relationship with regulation. Now we have some regular regulation—automobile regulation, pharma regulation. What’s the impact on tort there? It varies from place to place, but you don’t generally get a barring of the potential to bring, and potentially prevail in, a tort claim.

If you’re selling an FDA-approved drug, you can still file your claim that the company did something that doesn’t meet the state standard for what’s required. You can definitely sue the manufacturer of the automobile, right, even though they’re in compliance with whatever regulatory requirements.

What courts do in those cases is take all that into account and say, “Well, we think the reasonable steps to take were getting FDA approval,” or, “The reasonable steps to take were complying with federal law.” So I think all of that is still available. There’s no sense in which you’re closing the door to the capacity of courts to participate in structuring this.

We do have a few cases—I mean, certainly there are cases, maybe lots of them. I know of a couple where the government comes in and says, “No, you cannot file a lawsuit here.” With vaccines, for example, you cannot file a tort lawsuit, but we also have a compensation fund for injuries from vaccines.

I did work long ago, back in the early days of thinking about how our legal systems work, on the September 11th Victim Compensation Fund, which was created by Congress for those who were killed or injured in the September 11th attacks. That came with the condition that you couldn’t sue the airlines, you couldn’t sue the Port Authority running the World Trade Center, and so on. The project I was working on was about how people felt about that—the fact that they couldn’t have access to courts and could only go through this compensation mechanism.

But I don’t think we’re thinking about anything like that here. It would change the way the tort case works. It’s really important to emphasize that this is because there are 2 alternative ways of making sure people comply with what they’re supposed to do in tort, which is take reasonable steps to prevent harming others.

You could litigate that, and in lots of different courts and lots of different cases, courts and juries end up supplying the content to that. It happens at the end of a long, expensive process—a process that does not give everybody access and doesn’t work perfectly in any sense at all.

Or you could try to pull that back to an earlier stage and say, “Oh, guess what? We’re going to try and establish up front what it means to comply with that duty, and we’re going to give you an oversight body—an independent, approved oversight body—that will come in and look.”

We’re not going to wait for accidents to happen. We’re not going to wait for people to get harmed and then for long litigation to go through. We’re going to move that process of deciding what you had to do in order to do the right thing closer to when we’re releasing and observing our products, and not take the tort law approach all alone of waiting until something happens and then going through litigation for those who could afford to get into litigation over it.

Can I just say one other thing that I think does get lost in this? I do think you end up thinking about the sympathetic person who was harmed and asking, “What have you given them as a remedy?” But the overall goal should be fewer people harmed. I would much rather have a system overall that says, “Can you prove that you’re harming fewer people?” Then we should reward that behavior, right?

If you can, there should be some front end. I’d much rather there be fewer cases simply because there are fewer bad things happening. But, yeah, I think that overall is hard to remember, but that should be the true north.

Yeah, for certain subcategories, that seems relatively clean. I’ve seen some of these graphs put out by Waymo and Swiss Re where they’re just like, “Here’s the level of accidents and injuries with human drivers, and here it is with Waymo.” It’s like, okay, let’s all move to Waymo. I think that seems pretty clear.

I guess I wonder how to red-team the bill, which has become a meme in this space. How do we create a race to the top, and how do we avoid all sorts of shenanigans when it comes to these things? Sometimes these are just legitimately very hard questions. What gets categorized as what?

You said a second ago, “Nobody knows,” in response to some of these fine-grained liability questions. A joke that I’ve recently made is, “What is an AI agent? Nobody knows that either,” right? It could be anything from a workflow that exists in Zapier to something that is calling senior citizens on the phone and explicitly instructed not to identify itself as AI—which, by the way, is something I have done on existing commercial platforms. I’m not actually calling seniors, but I’ve demoed that that can be done on existing commercial platforms.

All of that right now is getting swept up into “AI agent.” With so much in flux and so little clarity on even what counts as what, and then the space being sort of problematic when it comes to taxonomizing itself, I wonder how we create the right incentives to actually have a race to the top.

How do we avoid situations where somebody is like, “Well, I’m the AI agent regulator,” and then they’re lumping a lot of things together or doing some sort of weird bundled trade? Another data point on this is that I was briefly in the financial services industry in the run-up to the mortgage meltdown, and I saw very up close and personal how the credit rating agencies had been totally captured and were basically worthless at that point.

How do we avoid the sort of credit rating agency problem that was maybe similar in a way, too? There were all these exotic products at the time, and now, if there’s one thing you can say about AI, it’s an explosion of exotic products.

So what are the key points in terms of creating a race-to-the-top dynamic that is real and durable, as opposed to getting ourselves into a 2007 credit-rating scenario?

Can I take the credit-rating point? Because we actually discussed the credit-rating agencies in the 2019 paper. Everybody says, “Oh, it’s going to be like that.” There are really important points about the credit-rating agencies.

Their market demand was created by the government, because the government said, “You have to go to these credit-rating agencies in order to be able to issue bonds,” and so on. But at the same time, the government immunized the credit-rating agencies from any liability for the ratings that they gave.

There was zero government oversight of the credit-rating agencies from the point of view of how well they were doing their job. That’s completely different from saying this regulatory-markets approach, which says, “No, what we’re trying to do is move the role of government to oversight of these regulatory-service providers”—the ones who are figuring out what’s the best method for making sure we don’t get uplift in bioweapons or have unsafe AI companions.

You absolutely need a government role there, and it’s government oversight. We’re shifting the role of government from the detailed oversight—which is almost impossible for them to do—of what the labs are doing to oversight of what these private regulatory-service providers are doing.

The race to the top could be coming from something like this: Suppose the standard that the government says is, “We’re going to peer in regularly, and we’re going to say, ‘Look, we’re going to yank your license. We’re going to yank your approval if you don’t meet this standard.’”

You could imagine a standard that says you need state-of-the-art protection against providing the capacity to build bioweapons to people without anything more than maybe high-school chemistry. Now you have these competitive companies in this business, and they have a market interest in communicating to the government: “Oh, look at what we’ve figured out. Here’s how we can reduce that risk, and here’s how we can demonstrate that to you.”

By the way, our competitors over here in the regulatory-services market are trying to pull the wool over your eyes. They’re saying this is all we can do, or that we’ve done a good job, or whatever. Company A has an interest in increasing its market share and demonstrating that it can do better.

So I think there’s a race to the top there, and that can actually move our standard. Then there’s a race to the top in terms of saying, “Okay, what’s the most effective way? What’s the cost-effective way?” Part of what we’re facing with where our AI governance is right now is that we’ve defaulted to a lot of process-based regulation—check this box, put this oversight process in place—without actually testing whether or not that works.

Do we know that it works to have these logs and have these officials in place, and so on?

And so again, the race to the top is about getting government oversight that says, “Here’s what we want,” and companies that are competing to achieve that standard. I think it’s really important to recognize the difference between credit rating agencies and this model. As you’re pointing out, they’re a really key example of what we saw as a big failure of this private role, but it actually wasn’t this model because it did not have oversight of those entities.

I’m sure there’s some regulation of credit rating agencies, so I don’t want anybody following up to say, “Oh, here are all the laws they have to follow.” But they definitely were immunized from liability for their ratings.

A couple of other things that I think are important as guardrails for these private-side regulators: one is that I do think the government has to be super involved in their finances. There are 2 things. First, make sure that they aren’t seeded and funded by the labs themselves, so that there’s some independence there. Second, make sure that they can actually afford to deny certification and continue existing.

I do think that’s some stress testing that would fall on the government in order to make sure that this is right. If you’re like, “Hey, here are some great processes. By the way, if we don’t certify 4 out of 5 labs, we can’t continue to exist, so we’re going to figure out how to certify 4 out of 5 labs,” then this is trouble. That’s a deep analysis that actually has to happen from the government.

The second part is that I do think the government has to get real about what outcomes it’s expecting. As you mentioned, Andrew, part of that is easy in some places and hard in others. In some places, you can say, “Well, we have a very clear human analog to what’s going on right now, and so if you can’t prove that you’re safer than humans in this way, or safer than the average human, then that’s bad.”

But in some places, it’s going to be completely new stuff—new harms, new ways of thinking about harms, places where we don’t have incident-reporting systems, or places where the first incident is so bad as to be catastrophic. Who cares that you followed some rules on the way there? That outcome was just totally unacceptable on any level.

I do think there are crawl-walk-run parts of this model. One of the reasons I like it being so voluntary at the beginning is that the government sets outcomes in certain places, and the companies get to decide if they think that amount of liability protection, or a Good Housekeeping seal of approval, or whatever it is, is worthwhile to go get for right now.

The conversation makes sense at the beginning. There are going to be some places where we really have to work things out over time. Are there outcomes that the government can set for us where it makes sense to bring in this private regulatory model, or does there have to be a different solution? Is that issue so major and in such a different place that regulatory markets don’t solve this problem, and it needs to be put into a different category?

Yeah, I could definitely see some biorisk things, in particular, being beyond the scope of any sort of liability framework. I was recently on another episode saying that it would be hard to go to the Wuhan Institute of Biology. Whether or not it ultimately came from there is another question that I’m not taking a position on, but with 10 million-plus dead globally, it’s hard to go there and sue them for damages. There’s just no—it seems to be an order-of-magnitude different thing. As the externalities become so big, you may just need a totally different regime.

I honestly think that’s one of the amendments we would love to see in something like SB 813: there’s a harm that’s so big that it shouldn’t fall within this. I think that’s part of figuring out the exact right landscape for how we start this program, or this way of regulating, up. It doesn’t have to accomplish everything, but it should accomplish some very real things at the beginning, and we can grow and mature it into a way to accomplish a large portion of things.

Certainly, there are always going to be edge cases, especially when they’re attached to really large harms, that have to be handled otherwise.

Yeah. I think it’s really important to emphasize that it’s just one tool in the toolbox. I think it’s something really important that we’re missing in the toolbox and that we will need—and we will need more in some areas than others—but it does not displace all the other complex ways in which we achieve safe, fair, stable market societies and so on.

We already have a very complex—we just don’t see it—set of systems that interleave and overlap: tort law and standard-setting. Corporations have their own incentives to create safety. There’s lots of oversight; there’s the press; there’s just tons and tons of stuff. So it’s going to be a part of that complex system.

The other point we maybe haven’t emphasized enough, but it’s been there—and part of your questions, Andrew, is what you had asked earlier, which I’m not sure we answered—is what’s different about AI? Why isn’t it just like any other product? We regulate cars; we regulate drugs. It’s another product.

Well, AI is not a thing. It’s not a product. It’s a general-purpose technology that I think is going to transform the way we do just about everything. It’s going to end up impacting and requiring regulation on a ton of dimensions. We have a very complex regulatory landscape, as you mentioned: education, health, justice, logistics, city management, bioweapons, AI companions. I mean, it’s just the breadth of it. It’s going to be in everything.

We’re going to have different regulatory goals and different regulatory methods in all those places. We don’t want to think, “Oh, we’ve regulated AI.” It’s more: can we build a vibrant, robust, agile mechanism that helps us identify where to regulate and how to regulate?

This was a point that I think Andrew was getting to earlier. Another way in which building this kind of ecosystem can help is by surfacing where the problems are and where there’s a need to regulate. There could be demand for regulation from enterprise purchasers that say, “I can’t integrate that chatbot into my customer service because I don’t know if it’s going to make stuff up.”

That could cause me harm because I’m giving advice to my customers, or it could cause me harm because they’re going to think I’m selling something I’m not selling. I can imagine enterprise companies starting to come in and say, “Here’s the kind of protection that we need in order to drive adoption.”

I think this is going to be a critical part of this. We talk about risks and harms, but we also want to be talking about why we’re building this in the first place. Hopefully, the reason we’re building it is because it can make everybody better off. So we need to figure out how we can respond to the ways in which the market can tell us what the concerns are.

Parents have concerns about their kids using AI in school. The tort system has surfaced some obvious—and quite sad—concerns about the way AI companions are impacting children.

That’s a bottom-up kind of process. We want a way for the market to be able to respond to what it can tell us about the concerns people have, the problems people have found, and the stumbles we’ve identified.

That’s why top-down approaches in regulation are so complicated, because that’s what markets do for you. They sniff it out at the ground level and tell you, “Here’s where an opportunity is. Here’s where a harm is. Here’s where risk is. Here’s where there’s a demand for something different from what we’ve got right now.”

That includes AI’s very general-purpose nature. There’s simply no way to sit in a boardroom, get out the whiteboard, and say, “Okay, here’s the list of risks we need to worry about from AI, and here are the rules we could put in place.” I just think that’s a fallacy of lawmaking and regulation.

I want to push a little bit more on this race to the top versus race to the bottom, and then I’ve got a few objections—or some red-teaming from perspectives besides my own—that I want to throw at you as well and get your reaction to.

It seems like, on the race-to-the-top question, you make a great point about the demand from enterprise. Enterprise in general rightfully wants to use this technology, but also wants to cover its butt, and that seems like a force for good.

That also seems like it’s going to be more oriented toward known unknowns than unknown unknowns, I think. Putting my AI safety/x-risk hat on for a second and focusing on frontier developers—the ones that are pushing things forward as fast as possible and really getting into uncharted territory—I wouldn’t say that they’re all trying to do the least they can.

I think, on the contrary, we’re relatively fortunate compared to the counterfactuals. I can easily imagine, in terms of the people running these frontier developers, how responsibly they’re acting—I think, again, relative to the alternatives. But nevertheless, if I’m doing a realpolitik, or my cynical, follow-the-incentives sort of analysis, it would be that the frontier developers are going to want to do the least that they can.

So if they have a menu of options in front of them, they’re going to choose the most permissive, least costly one. I’m not entirely sure if money is supposed to be flowing from the labs to the regulators—the MROs in SB 813—or if the money is coming from some other place. But even if there’s any correlation between who gets picked—if they pick you, that’s how you get paid—then there’s sort of this incentive to try to be the one that gets picked. That all seems like a sort of race-to-the-bottom dynamic.

And then it seems like, at least again in the SB 813 scenario, we’re really relying on the attorney general to be doing a great job. They’re the ones that have to approve these organizations in the first place, and they’re the ones that have to keep a close eye on them. If they take their eye off the ball, everything can kind of race to the bottom probably pretty quickly. That’s a challenge there.

Obviously, we’ve seen recently in our country how one administration to another can bring about dramatically different attitudes, personnel, and decision-making. So from one California AG to the next, I could imagine going from a great scenario where you’ve got the crack team that’s doing exactly what you’d want them to do to somebody who’s just focused on other things or, even more problematically, prone to being lobbied by companies.

We haven’t even—there’s a whole political economy of how people are channeling messages and what—I don’t need to tell you about the complications of the political economy of this. But is it right to say that in the SB 813 world, we’re really putting a lot of trust into the AG?

Yeah, I think not only is that right; I think it probably has to be changed a little bit to put more of a commission structure—or some expertise—into the government, to be able to say, “Have you rightly scoped your outcomes? Do you actually have the ability to track the outcomes that we say are important and know what’s happening on an ongoing basis?”

I don’t think it’s right until there’s essentially a little bit of a fear-of-God moment for the people who would be these private-sector regulators: their ability to be a certifier of this nature can really disappear very quickly. If there is a complaint out there that says that you definitely bent your rules in order to make sure something went through, that can be fully investigated, and there are the money and resources to fully investigate it.

I will say that, in an ideal world, this gets passed in a couple of states or a couple of different governments, and there are multiple people who are looking after and giving the seal of approval. If one drops off, that’s a signal. If state Z is like, “Hey, something doesn’t feel right in this, so we’re withdrawing from the licensing of this private regulator,” then that should kick off a whole bunch of other people starting to go in there. If it becomes an international group, there are countries that are also watching over and diving into this business.

What I like about it is that it’s this layer of—this is what the government should be doing—looking into these private-sector regulators and really getting into their business. Are they qualitatively and quantitatively showing that they’re making a good difference in the world? If they aren’t, there should be enough competition that we’re able to withdraw from one group and give power to another group, and we should be held accountable as lawmakers.

There’s some point where it’s turtles all the way down. There is a moment where it stops: you’re trying to create the best scenario for government to hold these groups accountable, and if the government is simply not interested in holding those groups accountable, redundancies aside, there’s a problem there. I will point out that this exists with every regulatory structure. There’s just a moment where it stops.

I think what this does is really bring it out in the open and allow the public to see: “Okay, you guys have allowed these people to be certifiers for years, and when they put out their numbers about how they create a better world, it is laughable. Meanwhile, look at this certifier that’s doing this other thing and actually creating a better world.” So there’s at least some way of maintaining that race to the top that doesn’t include public accountability.

My final point is that I do come from a world where I see all ceilings become floors, where the best-intended government regulation just becomes yet one more way of checking the box and staying away from stuff. I don’t know of another structure that is more set up to do the opposite—to actually create a qualitative race to the top and continue to iterate that system as technology grows.

So I think there’s the reality of how you get to this model, right? The ideal that Jack and I were describing back in 2019 and have been talking about since. There’s the sausage-making of, well, you’ve got to do it this way, that way, in this legislature, through this process. Here’s what we think is achievable today, and so on.

But on this point about the race to the bottom, this system is only as good as the capacity for your government to have oversight of these private actors, whichever they are, whatever category we’re putting them in—regulatory services providers, independent verification organizations. Government regulation has to have teeth in it, just like our existing regulation is only as good as our capacity for our government—the IRS, the FDA, or the Securities and Exchange Commission—to actually create good rules and enforce them.

I always like to think of the proposal here, sort of in the grander scheme of things, as shifting government effort and expertise into the task of overseeing these private regulatory bodies. I think of that as a pretty muscular thing. The model is only as good as your capacity to do that. That’s your backstop against a race to the bottom.

That’s your backstop against, “Oh, come on over here. You’re not going to have to do very much to comply with my system.” So absolutely, the key design feature is how you address that. Now, of course, regulatory capture and so on is a problem throughout our regulatory system. The whole term is based on the idea of corporations capturing government.

So we always need to be comparing this proposal and what we think we could achieve if we put the resources into it and got the design right, relative to what we can achieve in its absence. There are real methods being proposed here for getting that kind of appropriate and effective government oversight.

Going back to the conversation about the various domains here, the expertise you’ll need in government to oversee the domain of autonomous vehicles will be different from the expertise to oversee the domain of companion AI, to oversee bioweapons risk, or financial-stability risk. That’s the complex regulatory regime we’re in, and we’re just trying to change the role of government in that.

But I don’t think anybody should think this is a one-and-done: we’re just going to ask, “Did you fill out the right forms? Have you shown us something that looks plausible?” and then leave you to it. No. This is a way of actually getting us away from the world we are currently in in AI governance, which is that we have defaulted to corporate oversight—in fact, self-governance—throughout.

We’re saying, “Oh, we have no idea what to do, so we’re just letting the labs tell us what red-teaming tests to do,” and we’ll have limited visibility into that. Or we’re going to kick it over to industry standard-setting bodies, which are corporate-funded and have lots of participants from industry. Government has basically been defaulting on, I think, its central role in saying, “Hey, this is what we want from these domains.”

A reason for that is that it’s technically so challenging. So this is a proposal that’s trying to deal with the technical challenge without giving up on—in fact, making more muscular—the democratic role for governments. Government should be telling us, “Here’s how much risk we’re willing to take,” and we’re not there right now.

At the end of the day, the protection against a race to the bottom is that you actually have government regulating. It’s just regulating in a different way than it conventionally does, which it’s actually not able to do effectively right now.

So that was like—did you say, Nathan, we’re kind of naked on this? I think that’s right.

Or another way to put it—and he wasn’t talking about this at the time—but friend of the show and research partner of Fathom, Dean Ball, once simply put it to me: “Republics require virtue.” I think that’s a good reminder that you can always poke a hole and say, “Well, what if somebody—what if the person in that seat is bad? Who’s going to monitor the monitors?” and whatever.

But at some point, this is an institution that is going to be populated by people, at least until there are maybe some AIs.

Taking over key roles.

Regulated AI is taking—

Future speculation.

That’s right.

But somebody’s got to actually be trying to do a good job at some point in any given system, or it’s going to go to hell. There’s kind of no way around that.

I think republics require virtue. They also require visibility.

And I think this is one of the things where we’re in a serious state, because this is pretty much the first time in history we’ve seen such a massively consequential technology with such general-purpose capacity built entirely—really, almost entirely—inside private technology companies. These companies have this ring around them, a legally created fictional ring around them, that says anything that happens inside stays inside and doesn’t get out unless they choose to let it out or the government comes in and says, “You’ve got to let us look.”

Right now, I think governments are just in an impossible position to be able to effectively regulate because they don’t have visibility. So another feature of this is to start saying, “Okay, we need to get increased visibility into what’s happening.”

Again, who’s our partner in that? An independent sector of entities that are in the weeds, right? We already have some of these companies and nonprofits starting to emerge, providing red-teaming services or developing technology to check the robustness of systems or hallucinations in systems. We really want to lean into those startups, that sector, to say, “Let’s make this a powerful sector. Let’s create increased market demand for that. Let’s attract investment into this.”

That’s the partner for government that gives increased visibility for government into what the heck is going on, because right now governments are just at the mercy of what the labs have chosen to share with us. I’m not beating up on the labs. If you’re going to structure them as corporations, and those are the protections we give corporations, that’s the way they’re going to behave. I’m an economist. They’re going to engage in profit-maximizing behavior.

That’s what gets us this technology in the first place, but it runs headlong into what we need for regulation. So, yeah: virtue and visibility.

So that seems like a different law, though, right? Maybe we could try to incentivize them, but if we really want visibility, we might just have to mandate it. Do you have thoughts on how visibility should be mandated? There’s a connection there to whistleblower protections as well. I was also going to ask about the mechanism of how you think money should flow in this system, so I’m rapid-firing questions at you.

Yeah, let me do this, and then, Andrew, I know you’ve got something to say, so I’ll come to you. First of all—and again, Jack and I talked about this in a 2019 paper—I think that if you have a private entity that you’ve contracted with to provide regulatory services, to give you that oversight, then, first of all, yes, dollars are moving.

That’s because you need to get dollars into the business. I’ve just been chatting with some of the nonprofits that are engaged in, say, doing red-teaming under contract for the labs, and, surprise, surprise, they’re finding that they need more resources. It’s a bigger job than a few very virtuous people can do. You need to get dollars into that.

That’s part of the flow, and attracting investment to this is, for me, one of the number-one reasons to do it. Then there’s the fact that I actually think you’ll be able to get much more fine-grained information transfer between 2 private entities under contract.

We see companies engaging in information sharing. They go into joint ventures and collaborations, and they share, within ranges, detailed private information because they have confidence that their confidentiality and IP protections and so forth will keep that information private. They’re not sharing it with the government. So I think you’ll see more visibility going into a private regulator than into the public regulator.

But then the public regulator has complete authority to set whatever standards it wants for its oversight of those private regulatory agencies and say, “Okay, you need to show us your stuff. You’ve got to show us the results of what you’ve been learning.” The government may not end up getting into the weeds on all of the information coming out of the labs, but I think you start structuring those information relationships.

I think there are ways to improve visibility. You increase visibility because you’ve harnessed the incentive of this independent sector to say, “Here’s what I need to know, and if you want to be certified by me, you’re going to have to share that information. Here’s what I need to know in order to be able to fulfill my duty to the government, to say I can demonstrate that my approach, my technology, achieves your government goal—that target for regulation.”

Okay, here’s a series of questions that I’ve either gathered or had posed to me by others. I just had Matt Perault, who’s the head of AI policy at a16z, on. He is very focused, and a16z is very focused, on advocating for little tech and just trying to make sure that there’s a place for startups.

His concern about SB 813 in particular—and I think this probably would abstract to the more general concept—is: What if the rules become so onerous that only the big tech companies can meet them? Then the big tech companies get the benefits, the startups can’t get into that beneficial regime, and it becomes very hard for them to compete or raise capital because they’re on this disadvantaged legal basis compared with the big tech incumbents.

One answer might be, if that’s the way it plays out, “So be it,” or that it’s a cost worth paying. But I don’t know if you want to bite that bullet, or if you think there’s a way.

No, I definitely don’t want to bite that bullet. I honestly don’t know of a structure that can scale to provide solutions that are different for little tech than the solutions available for the frontier labs.

You could well envision—and maybe some guiding language within SB 813 would be helpful here—that there is a specialty lane for models, applications, or deployers that are smaller-scale and pose less immediate risk in these ways. In that way, they have a much lower burden in what they have to show to meet best practices in the environment they’re in.

Therefore, they can still go get whatever seal of approval the system ends up creating and being watched after, but it’s of a level that makes sense for them—either because they’re selling to enterprise or because they do believe that they introduce some risk into the environment that they need to look after.

Instead of it being, say, like a SOC 2, where it’s the same for everyone and everyone has to meet it, there suddenly is some way of actually creating gradation and saying, “This is actually what the best practices should look like for a 10-developer group that is looking for a limited application that goes out this way.”

It’s just so much different from when you’re going to be in 10,000 vehicles tomorrow and we need to make sure that you know how to obey traffic signs or stop before a little girl drops a ball in the street. The problem with a top-down approach is that there’s no way to account for that, right?

You can try to write it into legislation and try to bifurcate it today. That’s going to make whatever you’ve done—whether it’s by FLOPs or whatever—not make any sense tomorrow. Whereas groups that are specifically looking to meet this moment where it’s at can really change and be really flexible to that moment.

For the little-tech world, they’re also in an impossible position in a no-regulation world, because the only people who can prove that they’re going to be safe for a fintech to be able to come in are going to be the big guys right now. They’re the only ones that can go and do the amount of independent certification and work and long-standing work.

A little guy deciding that they have a way to forever change the banking industry at this moment has no way of proving to the banking industry that their stuff should be trusted. So I would argue that there’s a way of creating this that actually is of massive value to little tech.

I’d also argue that the flip side is that the other alternatives are not going to be able to become as bespoke to the needs of little tech as a solution like this could become.

Yeah. So I think this is actually at the very heart of what, for me, has been driving my thinking about how we get more markets into solving our regulatory problems for decades, frankly, because markets have the capacity to be differentiated.

We’ve got cars for your middle-class worker, and you’ve got fancy cars for your execs. You get differentiation in markets. So I think that’s a key feature of saying, “Oh, let’s try and unleash some market effort here.”

If you have venture money that says, “Hey, we really want to build the little-tech world,” let’s put money into funding the right kind of regulatory infrastructure that serves that need. That’s a regulatory puzzle, right? How do we do that well? How do we do that efficiently?

So I think that’s a key reason for trying to recruit more markets—regulated, overseen markets, right? Don’t lose sight of the fact that we’re not just abandoning it to the private sector; it’s only with that muscular government oversight.

But I think the other thing is, when I started thinking about this set of ideas, I was fundamentally driven by the fact that our legal systems and our regulatory systems have become far too expensive, far too slow, and onerous. We've leaned into a set of techniques for regulation that, on the one hand, are very expensive to comply with: lots and lots of process-based stuff, front-end process-based, with very little demonstration that those process-based protections actually achieve what you're looking for.

In some ways, the very problem you're trying to solve is that we've built an incredibly expensive regulatory regime that only our biggest companies can really afford to comply with. That is a massive drag on the startup sector and innovation, and that's a key reason that we need to be adapting and innovating in our regulatory methods.

So, if you think about the General Data Protection Regulation, for example—GDPR in the EU—there's lots of process: this definition, that definition, have you got these logs, et cetera. In fact, there's pressure right now in the EU to say, "How can we modify this?" because it is too much of a drag on the innovative startup sector. I actually think it's precisely a mechanism like this. The goal of it is to say, how do we build more efficient, more effective regulatory regimes and move away from the top-down thing that, frankly, lawyers in a room are going to create?

I beat up a lot on lawyers in my book, or at least on the way in which our profession has failed to rise to the need of societies for greater innovation in what we produce. We do not need more words on paper. We need more smart approaches for regulation. How are we going to get there? That's what we're trying to do. That's the path we're trying to set ourselves off on.

There are going to be a lot of hiccups and bumps and wrong turns and dead ends, but I think we absolutely need to be making this shift, and we needed to be making it 10 years ago. The only thing that's happening is that AI is ramping up faster and faster, and we still have our shoelaces tied on the starting line. We are not getting there, and we need to get there.

I do love the fact that this proposal creates an opportunity for people to come up with new ideas and enter into the ecosystem on an ongoing basis. I also love the fact that—and I know this may still need to get worked out a little bit in SB 813 in general—there seems to be an implicit, if not explicit, sunset clause, which is something I always advocate for in law and never seems to happen. There's at least a sort of ongoing subject to renegotiation or reevaluation of a lot of things in this proposal.

I think that is really great, too, because it at least gives it a decent chance to age well, which is my constant joke about AI content. AI content does not age well. AI regulatory proposals generally do not age well, but this sort of meta-structure that allows for new entrants and ongoing revision seems like it has a better chance of aging well than just about anything else I've heard.

I have 2 more different angles of red-teaming the proposal. One, maybe the most different—or almost the opposite direction, although you may see it a bit differently—is from another law professor, Gabe Weil, who, as I'm sure you're aware, has been advocating for the idea of an expanded notion of liability. I haven't studied his work in depth yet. I'm going to do an episode with him before too long as well.

The general sketch of it is that there are some potential harms or problems—catastrophic, existential, in some cases—that are so bad that we need a way to deal with them before they happen. His proposal is basically to expand liability to encompass near misses. So, if you were acting negligently and nothing really bad happened, but it came close or could have happened, then you could still be sued and held liable for harm, even though maybe you just got lucky. I'm mindful that I haven't probably described his position quite right because I haven't done the full study yet. Any reactions?

That seems like we probably can't do both, right? Those are two pretty different directions, at a minimum, it seems.

Okay. So, again, go back to the idea that our regulatory regime's ecosystem is a whole bunch of different threads. Liability—when people use the term liability, they're usually thinking about litigation-based, court-based regulation—which is always after the fact. It's a big process. It's got lots of virtues because it can be bottom-up and reactive.

So that's a good thing about having a strong litigation regime. I'm not anti-litigation or anti-courts as tools in our toolbox for getting people and companies to do the things we want them to do—the right thing. One approach would be, yes, you could say, "Well, we don't want to wait for catastrophic harms to happen. We should cover near misses." Maybe that's a fine amendment to make into law.

I'm not enough of a tort scholar to even know what our existing doctrines are on how close you have to be to cause harm. Can you just create a risk, or do you have to actually cause the harm? I don't want to go into the details on that, but the fact of the matter is that these are the types of domains where we actually don't generally leave it to litigation.

We don't say, "Well, we're going to rely on the tort system to handle the risk of nuclear facilities blowing up or creating fallout for communities." We actually don't rely on that. Think about pharma: we started early with pharma. We have an FDA that says you cannot put a drug on the market unless you've demonstrated safety and efficacy and gone through a fairly lengthy approval process with our regulatory agency.

You still have backup liability. You can still sue for harms caused by drugs on the market that have been FDA-approved, but we haven't put the whole thing out there. I certainly think, if we're thinking in the domain of catastrophic risk, that we want to include lots of things like collapsing our markets and bunging up our financial trading systems. There's a lot of economic stability risk that I think we don't pay enough attention to as potentially catastrophic.

But I don't think I would be focused heavily on, "Let's just deal with this by making tweaks to the tort law regime." Maybe we make those as well, but first-line defense for me would be some oversight on whether or not there's a bioweapons risk, or whether these trading agents could collapse or cause the equivalent of massive bank runs or crashes of the stock market. Those are very costly things, and I don't think we just want to handle all of that through back-end litigation.

I hate to beat up a straw man because I don't know enough. One of the things I will say is also to be careful of the downside of that stuff. How much is that just going to mean that near misses are not reported, right? How much of that is just going to mean that you have to create your corporation in such a way that nobody knows the full picture other than a few trusted people, and everybody's in their own little silo?

I imagine a lot of tech organizations are already kind of in that state.

Yeah. This really can push toward much more siloing. If we didn't know, then there's nothing you can sue us for because we didn't know that there was a near miss there. You're asking people not to go do the red teams and do the hard work because the more they know, the more they're potentially liable on the back end.

There's some downside that I do want to flag. Maybe that's thought about in this proposal, so I don't want to overstate it. But I'd also argue that I really do hope we end up seeing things that allow people to be proactive in this space and get rewarded for being proactive, and not just have a way to go and sue for it after the world's burned. I hope that is brought into that strategy as well.

It does even strike me, just in listening to your responses there, that I was maybe too quick to see liability protection and liability expansion as incompatible. Plausibly, you could expand liability, but then you could also afford some protection for compliance, and it doesn't necessarily seem like those are so diametrically opposed.

A third one is—and this one is maybe closest to your own impulses—why not require insurance and put everything on the dollar scale? Insurance companies are presumably the best organizations we have for calibrating to risk. That would also bring a sort of pricing mechanism to the risk that I don't see quite. Maybe you see a way that it emerges from the structure that we've been describing, but with insurance, it's quite clear how the pricing mechanism works.

So, instead of this whole thing, why not just say everybody's got to have insurance? If you don't have insurance, you can't drive.

So, I think I won't speak for Gillian here, but I will say, for me, I've been most bothered in my nerdy capacity by how we've waved a magic wand with insurance and said that it can solve problems without diving into what it is that insurance does in order to solve those problems.

And so, for the insurance market to help be a rational market here—to actually properly price risk and charge against that risk—that includes being able to properly know what risks exist and how to best mitigate risk. That doesn't exist absent valid third-party certifiers going in and doing that work and racing toward it. I would actually say that, in the scaffolding we've been describing, it's easy enough to say that insurance knows how to price risk. I would argue they probably don't here, right? Not even that they probably don't: They absolutely do not know how to price risk here.

You could require them to come into the market and they would say, “Great, everything has to be self-insured, and it's an astronomical amount, and we don't really know. We're not any better than anybody else right now at coming in and doing that.” There has to be some scaffolding of information, of a knowledge base, of what actually decreases risk in the system, and of how we know that risk comes from a valid source in order for there to be a rational insurance market. Speaker 1

I think people would just argue, though—sorry.

Oh, sorry, let me just add on that. To emphasize: With insurance, you need to price risk, so you need lots of structure that is determining risk. When insurance companies are insuring against liability risk for automobile accidents or construction-site accidents—or let's go back to pharma—they're doing that against a backdrop of a ton of structure that defines the risk. We've got lots of history of automobile litigation and liability. We have tons of codes that govern how you run a construction site. We've got all this regulation around drugs. So insurance has got a lot of structure to price on the basis of, and we don't have that right now.

I think there's this idea that insurance companies, because they do risk, will come in and magically solve the problem of where the risk is and what should have been done. But notice that we've just reinvented the problem, which is: What do the companies need to do? We don't know what that is right now, and I do not think insurance companies are going to become our AI regulators. Again, we would not have any oversight of that. That would be insurance companies saying how much risk we're going to allow for biorisk or whatever. They have to build on something.

My institute that I ran at the University of Toronto up until last year just released a report. We had lots of discussion about insurance. Insurance is a nice complement to building a regulatory market and building regulatory technologies. It's actually another one of the carrots that you can use: If you implement this regulatory method, then you can get insurance. But it's got to be that kind of partnership, and those are the kinds of products that we're starting to see emerge.

Armilla is a company that creates this regulatory technology and has been one of my go-to examples of a startup in this domain. Through a partnership, they said, “If you use our approaches, we now have an arrangement through Lloyd's of London, I think, or other insurers as well, and you can get insurance.” But the insurance companies aren't going to become our AI regulators, and it wouldn't be appropriate for them to be that. So I think a basic “just mandate insurance and it'll all work its way out” is not a realistic view of the way insurance markets work, regulation works, or democracy works.

The democracy point, I think, is strong. At the end there, though, you were sort of getting to what I think the advocates for the insurance idea would say, which is, first of all, this market is going to be massive. So this isn't a niche corner of the insurance market that insurers would say, “Forget it. It's not worth our time to figure that out.” It seems like if you're an insurance company and there's AI happening and it's touching everything and the risk is massive, you would presumably want—

What's the risk of what, right? What do you insure against? To insure against liability risk means you think that courts are going to be able to impose requirements and standards. Or you insure against compliance risk—regulatory compliance risk—but then that requires government to have regulatory structure. We can have bad things happening, but that doesn't mean risk for the companies unless they have liability attached to that, whether tort liability or regulatory compliance liability. So it's not a massive market unless there are requirements that they're going to be held to, either through the courts or through government.

I think the notion is mostly liability risk, and also that it would be a lot of these same organizations that the insurance companies would turn to to try to help them get a handle on it. So it would be, you know, who? I guess I'm interested to know, for the future of Fathom, does Fathom envision itself being one of these private regulator entities? What other organizations are you looking at in the world today and saying, “These guys seem like they could step up into this role”?

In the insurance context, it would be those same candidates that the insurance companies would go to and say, “Hey, we need—we'll pay you to help us figure this out, and we might insist on companies going through your audit process or whatever if they want to buy the insurance.”

I think the hope is actually to end up in a pretty similar spot, where experts are defining standards and also conducting audits, but it's less concentrated through one agency or one commission and more concentrated in the sort of global insurance market, which, in theory at least, has a lot of skin in the game.

I would love to answer the Fathom question first. Fathom is interested in showing proofs of concept here. We think this is enough of a novel idea that the way in which the marketplace starts will be important for success, and so we'd love to see some proofs of concept out and then help those that are very interested in doing this work be very successful at doing that work.

We remain a nonprofit. Any interest we have is nonprofit-related and philanthropically funded, and there's no endgame here for there to be an equity play or any of that. But I do think it's one of those things that if we don't show what good looks like, people are just going to constantly be like, “I don't have time to listen to a 2-hour podcast right now on this.”

I don't know who has time to listen to us. Honestly, it's great.

I listen to it. Yeah. But we do need to start showing also that it's complicated, and we do want to work out the kinks by having real-world examples of it. I think in the near future you'll see us trying to show proofs of concept, working with partners who are actually the technical people in this space, right?

I don't know if I have permission to share the technical people that we are giving grants to to see them work on this, but I think in the near future it'll be clear that we think there are some great technical minds working on this right now, that we are giving small grants to to try to see them do this work.

Well, I will just again come back to this: Insurance is making a bet in this world at the end of the day, and if they're making a bet based on no different risks, you could easily imagine a world where they go, “Okay, well, that's such a big catastrophic risk that if it happens, there's no one to sue on the back end, and so we'll insure against that risk because the chances that we actually have to be there at the end of the day to pay it out are very low. Society will have fallen apart, for example.”

There just isn't—I don't think there's a magic to the governance of how insurance works that they're going to be able to bring in the best third-party validators of risk and have them do the best work. Their focus is going to be, “Okay, what's the greatest tangible next risk that's coming up that we could actually be on the hook for here?” To the extent they can't do it and they're taking a guess, they're taking the same guess as everybody else is taking here.

I get it: It feels like once you start mandating it, the scaffolding will fall into place. But I actually think thinking through how that scaffolding is governed so that we are creating the best third-party certifiers is the most important question, and making sure those people are actually accountable to society rather than to any perverse economic motive on the back end. Speaker 1

Do you want to offer any closing thoughts?

We need to get moving. We need to get innovative. We want as many people in the conversation as possible, poking at the model and coming up with new ideas. That's, again, a reason you like markets: You need lots of different minds, lots of different perspectives, lots of different knowledge. You need that conversation happening, and you need this.

But we need to get going. I think we can't really stand back and say, “Well, let's design this perfect structure.” It's like, we need the MVP of new approaches on regulation, and let's get started. I think SB 813, whatever form it ends up in—and it will continue to evolve, because of course it's like, “Oh, here, let's throw this out here.” “What? Oh, wait a second. We need to change this. We need to fix that.”

The key thing is that we need to get down this pathway. So I think that's the key message I would take away.

Markets can help us in addressing this. They need to be overseen by governments. Governments should be deciding what's acceptable risk, and we need to get going.

Perfect. Dr. Gillian Hadfield and Andrew Freedman, thank you both for being part of The Cognitive Revolution.

It was such a pleasure. Thank you for having us.

That was terrific. Thanks.