为超级智能承保：AIUC 的保险、标准与审计如何加速 AI 采用

Hello and welcome back to the Cognitive Revolution. Today I'm speaking with Rune Kvist and Rajiv Dattani, co-founders of the AI underwriting company, who aim to unlock enterprise AI adoption by certifying and insuring AI agents. Their core insight is that security and progress are mutually reinforcing. Just as good brakes, seatbelts, and airbags are required if you want to drive 70 miles an hour, rigorous standards for AI agent behavior and reliability are critical for society's effort to realize the potential of increasingly powerful and autonomous AI systems. Their approach, which has already won support from an impressive list of industry leaders, including Cognition, Ada, Intercom, and many more, combines three elements. Frequently updated technical standards that codify the latest best practices, periodic audits to verify adherence on an ongoing basis, and insurance to align incentives and provide financial protection when things do still go wrong. As always in this conversation, we get into the many details of making this work, including the fact that today's insurance policies generally don't explicitly address AI risks, creating ambiguity about coverage for AI incidents. The AIUC1 standard that they've developed in partnership with technology and security leaders across a wide range of industries, which covers data and privacy, security, safety, reliability, accountability, and societal risks, their approach to auditing client companies, which combines analytical evaluation of technical safeguards with systematic red teaming, how the data generated by red teaming helps insurers price risk in domains where historical loss data is either limited or nonexistent, how they plan to structure financial incentives so as to avoid a race to the bottom, such as that which affected the credit rating agencies in the run-up to the 2008 financial crisis, and how, even though the market may not be able to effectively insure against the larger-scale existential risks from AI, the government, as the de facto insurer of last resort, still benefits tremendously from the governance that this model creates. Overall, I have to say I really like this approach. So much so that I did participate in the company's seed round alongside leading investors including Nat Friedman, Emergence, and Terrain. Certainly, there is a place for government regulation and also for unencumbered experimentation. But I think a private sector approach to AI reliability powered by enterprise customers desire to move with all responsible speed and designed to align all parties financial incentives with consumer and public safety seems much more likely to get the important details right. Not just once, but over and over again as both the technology and the market mature. For now, I hope you enjoy this conversation about how the AI underwriting company seeks to build AI confidence infrastructure and ensure the intelligence age with co-founders Rune Kvist and Rajiv Dattani.

Rune Kvist and Rajiv Dattani, co-founders of the AI Underwriting Company, welcome to The Cognitive Revolution.

Thanks for having us.

I'm excited for this conversation. You guys are doing some really interesting innovation at the intersection of trying to make sure that everybody can have all their AI goodies, which I am very excited about getting, but also making sure that we keep things on the rails, which I think is obviously extremely important as we head into a brave new AI world. So I'm excited to dig into everything that you're doing, the standard that you've put out, and the future that you envision.

Maybe, for starters, because I think you've used a pretty provocative analogy in some of your writing and in the way that you've introduced the company, tell us what the downside scenario would be for AI if we don't do a good job of this. Obviously, there's existential extinction, but you've got sort of the nuclear outcome, which is a scenario where we don't get the goodies, but we still maybe get some of the bad stuff. Then maybe you could compare it to an industry where things have gone well that you take inspiration from.

Totally. And maybe before we get to the nuclear outcome, a lot of the concerns that we're seeing in the world today relate to risks that are much more mundane in some ways: data risk, safety and security risk, brand disasters, and data leakage. Some of these are age-old concerns that become much more severe in a new environment where there are many more vulnerabilities and, frankly, people understand the vulnerabilities much less.

But as you play that into the future, we'll move to more and more gigantic systems that are going to be way smarter, and that opens up new kinds of nuclear scenarios. One that's been talked about a bunch relates to terrorists that want to inflict a huge amount of harm. If they have access to brilliant scientists who can help them create, for example, a new virus, they'll be able to do that way, way cheaper. So you can imagine new kinds of COVIDs coming out, and they'll be way easier for adversaries to create.

You can also start to imagine what a misaligned kind of system looks like, where today that might look like slight deception or sycophancy. But as they get smarter and smarter, they'll do some of the things that humans can do. One of the things that I sometimes get questions about is, as AIs get smarter, will they still fail? I think there are a lot of analogies to humans, where a really smart CEO will no longer mess up basic logic, but their IQ enables them to run kind of N+1 levels of fraud. That's the kind of thing where, as AIs get smarter, you get different kinds of scenarios, and we think that really opens up nuclear scenarios.

A different category of scenarios altogether is geopolitics. Some of these may not have to happen directly because of AI, but if you really get national security leaders believing that AI will give their nation a decisive strategic advantage, that might lead them to preemptively attack their opponents in ways that may look like regular military escalation but are really prompted by AI.

And, Nathan, you asked about other industries. I think there's not one—we don't look to one industry and say that industry is perfect, nor is there a perfect analogy for AI. But the thesis of what we're building here, and I think we'll get into this more later, is: How can we bring together learnings from lots of other industries?

The 3 components to what we're building—the insurance, the standards, and the audits—have come out of cars. When, post–World War II, there were a lot of car crashes on highways, insurers really developed crash-testing solutions that led to airbags and then created the incentives for that to happen. Go further back to Benjamin Franklin's Philadelphia, and fire insurers led to building codes. No one was saying, “Stop building,” at that time. It was, “How do we actually keep building, but build in a way that's safer?” That's really the essence of what we're trying to draw on here for our solution.

Yeah, it's funny you went to even some more tail risks, or surprising possibilities, under the umbrella of the nuclear outcome that I had in mind. When I say the nuclear outcome, for me that's shorthand for: We don't get the super-cheap, abundant energy that we should have from nuclear technology, but we do have thousands of nuclear weapons that are, collectively, a sword of Damocles hanging over all humanity.

And I do worry about a similar thing with AI, where it gets weaponized or there's this AI Cold War, but maybe we also freeze it out from a lot of the practical upside—the day-to-day quality-of-life improvements that I really want. I often think of airplanes because it's sort of like nuclear: There is the potential for dramatic explosions, and obviously if one happens near you or you're on the plane that goes down, that's really bad. But yet we do get the upside from planes. We fly cheaply and all the time.

As I always think back to Dumb and Dumber, it's actually more likely you'll get killed on the way to the airport than on the plane itself. So it is remarkable in my mind how contingent the end state for some of these potentially transformative technologies can be based on even just a few incidents that happen early on in their development, which can put us on one course or another.

And I think, Nathan, we're already seeing that in some parts of AI today. If you take autonomous vehicles, you still cannot drive an autonomous vehicle outside of San Francisco very far before you get stopped. Yet drone warfare is increasingly autonomous, and it looks like there are no barriers to its being deployed. So, in some ways, we're already starting to see that when it comes to autonomous drones and vehicles.

Yeah. Okay, well, let's get into the positive vision: insurance, standards, and audits. This is like the holy trinity of a virtuous cycle that you guys are trying to kick-start. So, paint the picture.

I can start. At a high level, the core idea is that security and progress are mutually reinforcing. A very basic analogy is that if you're a race driver, the reason you wear a helmet and a seat belt is so that you can go faster around the corners. The same is true for AI. The better we're able to steer and secure our systems, the faster we can deploy them, and the more we can lean into some of these agentic capabilities.

So, in the debate between the doomers and the safety integrationists, we think this is a core point: They are, in fact, mutually reinforcing.

The core of this, Nathan, is really as you were describing: if we don't get security under control, we don't think we'll get progress. We won't get the benefits of the airplanes.

The 3 components are insurance, audits, and standards. Let me explain what each of them is, and then I'll talk about how they link. Insurance is the financial protection for when something goes wrong: who's actually paying for it, and can you actually get the financial coverage behind it?

A standard is: What is the best practice, and can we actually codify the set of requirements that you need to meet to follow those best practices? Then there's the audit, which asks, "Have you met the standards?"

The reason we think we need all 3 of these is that insurers will struggle to underwrite the risk and know how to differentiate a good risk from a bad risk unless there is a clear set of requirements. They need to be able to say, "Yes, I understand these are the best practices. Are you meeting them? Which of them aren't you meeting?" That would affect your premium, just like it would with your car insurance if you're driving recklessly.

The auditing is essential because a standard without auditing doesn't really help. It's very easy for companies to say, "Yes, I'm doing these things," and self-attest. But if there isn't oversight to say, "Actually, yes, somebody has rigorously checked this and made sure that the standard is being met," there's a trust gap there.

This is how we think about the flywheel. Critically, in the past, insurers have funded the standards and funded the auditors to become more robust and create that oversight. They've created incentives for the companies who are creating the risk to comply with the standards because they want the insurance, and they want cheaper insurance.

Creating a pool of capital over here on the insurance balance sheets, funding the research, and having actors who are creating the risk actually wanting to comply and wanting to have the insurance in place—we think that aligns the incentives in the right way.

Maybe one other thing I'll say on this is that, at the moment, this debate has become polarized between "Let the AI developers cook" and just let them commit to voluntary commitments and that'll be fine, all the way through to the other extreme: "We need top-down regulation. We needed it yesterday, and it's got to be really strict."

We actually think this is a neat middle ground where you're aligning the incentives because the insurers want you to grow. That's how they grow their premiums as well. They don't want to have overly onerous requirements, and they don't want to have requirements that don't actually track risk, because then no one will buy insurance from them.

On the other hand, they don't want it to be too lax, and they don't want you to be able to break your commitments because they're the ones paying. This is a neat market mechanism that aligns the incentives between progress and security.

We can ground ourselves in some of the historical examples and work through them. The very first place we saw the incentive flywheel in practice was back in 1752 in Philadelphia, when Benjamin Franklin set up the first fire insurance company in the United States.

The basic problem was that Philadelphia was growing, with the population increasing 10x over that decade. People were packing houses closer and closer together, which created a lot of fire risks, and the fire risk would spread. No individual actor was carrying the entire risk because the entire city might burn down.

The fire insurance company would obviously have to pay whenever a house burned down, so they were interested in reducing the frequency of fires. They created building codes, which are standards: How far apart should houses be? Can they be under large trees that might be hit by lightning or not?

Then they created the first fire inspection. Inspectors went into homes to see whether they were meeting these building codes as a way of enforcing them. It's that very basic logic that we've seen pop up again and again.

For example, in the 1900s, when electricity came out, houses started burning down again. Underwriters Laboratories, funded by the insurers, popped up to define the UL certificate, which you'll see on your light bulbs today. It's a product security standard that they will then go and audit against.

That's the kind of mechanism that we're trying to bring to AI.

Where do we stand today? I was doing a little deep research on the topic, and I guess my takeaway from what ChatGPT told me is that, mostly, it seems like the harms that people anticipate from AI systems are covered under existing insurance policies, at least when it comes to the taxonomy of harms—the kinds of harms that are insured.

People are reasonably comfortable with the kinds of things they expect. But there's some ambiguity around whether the fact that an AI might cause them, as opposed to some other cause, could put something outside of scope. It seems like a lot of people right now are just living with that ambiguity.

I understand this is one of the things that you recommend the industry clear up: what's covered and what's not covered. How do you see that today, and what are the big points where we need clarification?

The state of play today, when it comes to AI insurance, is that it's ambiguous what's covered in current insurance policies. For example, AI can help you create new kinds of cyber vulnerabilities, and of course, people already have cyber insurance today. But it's unclear whether AI-induced cyber vulnerabilities would be covered by the current policies because AI isn't mentioned anywhere.

What we think is going to happen, at a high level, is very similar to what happened when computers came out back in the early 2000s. Leaking confidential data or having downtime wasn't an entirely new set of harms, but when computers came out, those harms happened at a very different frequency and with a very different severity.

There was the same question: Is this just age-old insurance, or is there something new here? What happened was that, over time, as lawsuits came in to clarify whether things were covered, cyber insurance was split out into a separate set of products because the risks needed to be priced completely differently, and the products needed to have a different structure.

We suspect that the same thing is true with AI. One thing we know for sure is that current insurance policies have not priced in AI risk because insurers don't know how to price AI risk. This doesn't serve anyone.

The companies that are insured and their customers are saying, "I'm not sure if I'm covered." In the 2000s, when it was cyber, this took years to work through the courts, with insurers literally suing customers and vice versa.

From the insurance perspective, they haven't priced in the risk. They're taking on risk that they're not being paid for, and we expect that's going to show up in loss data sooner rather than later.

That's a really—I had a very brief foray into the insurance industry some years ago, and it strikes me that this is a really hard challenge for them because they like to do this sort of thing based on historical data.

We can look back at how many car crashes there are. We can look at how many homes burn down from fires. With a couple of important assumptions, like the idea that things will be relatively uncorrelated, you can be pretty confident that you can write a bunch of policies, not shoot yourself in the foot, and you'll probably be okay.

But the same challenge is popping up with AI everywhere. I was just speaking to a group of educators last week and saying, "I know you guys have been trained that the central organizing principle of the field is that you want to be evidence-based, but you can't be evidence-based because by the time you see a study, you've got 2 more generations of AI, and the kids are using something that didn't exist at the time of the study."

Here we are, basically with the same challenge in insurance, right? Do we have any sense right now for what the frequency of these things is going to be, what the distribution of the magnitudes of harms is, and, if people are—yeah, maybe we have some of it—but I'm interested in how much we do have, and then how that translates into any sort of decision on pricing?

Yeah, there are a couple of issues here. One is what you said: AI is just moving too quickly. It's moving faster than anything insurers have ever known, and so the slow rollout—letting the data trickle in, picking it up in a few other categories, and then developing a product off the back of it—just won't work. If you were building an insurance product today based on historical AI data, you'd be building for the pre-LLM era and not even thinking about agentic risk.

That's one issue. I think the other issue here, though, is that they don't really have a mechanism. They're not set up to even collect that data. These losses are happening in the small places, which is typically what we're seeing at the moment. There was an EY study on this recently: enterprises are having losses of $1 million due to AI, while smaller companies are having losses in the thousands or tens of thousands of dollars. Insurance might not even be seeing this.

If they can't see it, they don't have the loss data from which to price. What we expect is that there's a long tail of small losses coming in, and then we're hearing and seeing the big ones. They're the ones that make the press, and the rest of them are being ignored both by the insurers and by the companies, which are just writing them off. If you don't think about how you're trying to price that, it's just really hard for insurers to apply that model.

There are a couple of approaches we're taking here. The nice thing about AI is that you can actually do what's known as red teaming; you can run evals. I expect your listeners are very familiar with what that is in the AI context, but if you translate that to insurance, it actually creates a massive pool of synthetic data for how this product is going to behave in the real world. Insurers can literally plug and play this into their pricing models.

They can say, “I see that this failed the red teaming that I did in these particular ways. If that happened in the real world, I can estimate what the loss would have been,” because they know how to estimate losses from a data leak or a court case that would have happened, right? That's their bread and butter. They can take the data from the red teaming on both frequency and severity and plug that straight into a pricing model.

The other thing that's nice here is that you can create pretty clear parametric pricing triggers, as they're known in the insurance industry. You can say, “Actually, I can know definitively: did the AI behave as it was supposed to, or was this an out-of-distribution event?” By using that, you can then very quickly say, “Right, I'm going to pay out $1 million because this happened, and I'm just going to price it. I'm going to pre-agree the pricing, basically, and pay out,” rather than waiting for a long court case to come in and settle on that.

One other big consideration is that there is always someone underwriting the risk, even when there's no insurance. Today, when, say, a hospital or bank deploys an AI system, implicitly they are pricing the risk to decide whether they want to go ahead or not. Although it's an extremely narrow problem to really pin down the price, the most important fact is that this is happening every day, all the time: people are trying, in more or less sophisticated ways.

We think, by and large, the rigor of the insurance industry is going to make it more likely to get that pricing right than nonexperts who are distributed across every enterprise trying to make these decisions for themselves.

Yeah, we're all self-insured whether we know it or not at the moment.

That's right.

You know, I guess one of the things that makes this sound really hard to me, if I try to imagine myself being an insurance company, is that it feels like, again, with car accidents or whatever, I would guess there's some sort of normal distribution of the damages, right? There could be some outliers, but those would probably be relatively infrequent, and I can put most of the incidents in a pack. Whereas there are other things in the world that are characterized by more of a power-law distribution—venture capital returns, for one.

I would assume, although I don't know for sure, that biological incidents probably fall into this category, where I'm sure there have been many viruses leaked from many labs that just didn't go anywhere, but then you get one COVID and it turns into a global event. My gut says AI risk might look more like that, in that it could be a few incidents that drive the bulk of the total weight of the losses. Is that right? Do we even know how to think about that? And how much of a problem does that pose for insurance?

And then, I guess, is there any way—assuming my mental model is not way off—to start to attack that problem?

The first answer is no one knows what the tail-risk distribution looks like because we have not yet seen it. We can learn from nuclear energy insurance—specifically, nuclear power plant insurance. The same intuition applies here: probably nuclear failures are going to have a power-law distribution, where the really big failures will carry most of the losses. And, in fact, that's true.

In the U.S., there is an insurance scheme for nuclear power plants. This is required by law. The way they deal with these tail risks is that the government basically created a scheme in which it said, “We are going to require every nuclear power plant operator to have insurance, and we're going to require you to take on liability, no matter where in the supply chain things fall. No matter whether it's one of your vendors who provided you with a faulty thing, you are responsible for all of this. You're going to take out insurance. But, on the other hand, we're going to cap the liability. If there's damage greater than $15 billion, this is going to come out of the government's balance sheet.” It's basically a government backstop.

The reason they're interested in this is that you're exactly right: the market may not be able to carry these very, very big tail risks. Yet the government would really like there to be an insurance industry because it provides an important governance function. It means there's now a market-based way to have independent third-party auditors with direct financial incentives to price the risk accurately and enforce best practices in the industry.

Basically, it doesn't require the government to do this work, which the government may not think it has the capabilities to do or may not be responsive enough to the market. There are some lessons from nuclear for how to get the best of private-market governance while allowing the government to cover risk where the market may not be able to really respond to it. I think this idea of capping risks is going to be critical when it comes to AI, which is going to allow you to cover a broad set of risks up to some amount.

Let's go to the standards and the audits. You guys have put out this AIUC-1 standard, the first standard that I'm aware of anywhere, at least in the Western world, for what to do when you're building and deploying an AI agent. Tell me about it. What does it contain? What does it require companies to do? And how did you go about assembling all of these ideas?

Yeah, I can speak to the high level here.

The core idea with AIUC-1 is to take all of the concerns related to AI agents that slow down adoption and put them into one framework that outlines, for each of the risks, what you're supposed to do in enough detail that you can have independent third-party auditors come and verify whether a particular company is doing those things.

The design principle here was to go out and spend a bunch of time. We met with more than 500 people across the enterprises that are deploying these systems today. These were security leaders, general counsels, and so on, across banking, health care, and other industries, and we tried to figure out what was keeping them up at night.

The answers will not surprise you a ton. It's going to be things like data leakage at a new scale. Are we going to provide incorrect advice to our customers? Are we going to have jailbreaks and prompt injections? Are we going to discriminate against our applicants? And so on.

The goal is that when a company meets the standard, the enterprises that are deciding whether to adopt it or not will get the data foundation they need to understand whether they can trust that particular agent.

The way it's used in practice, Nathan, is that often you'll have AI agent builders selling to large enterprises. Think of the large banks and the large technology companies. They want a third party to have vetted the tool they're looking to buy, and they ask them for their AIUC-1 report.

So, a large company is looking at an AI chatbot developer or an AI customer-support developer, and they'll say, “Share with me your AIUC-1 report. It should have all of the detail on what data and privacy controls you have, what security controls and implementations you have, and what you're doing to protect customer safety.”

What output filtering do you have, for example? How does it perform against bias? Is it tracking its role correctly and not responding with anger or things like that? Or is it responding appropriately to anger if a customer really wants a refund and is saying, “You must give me this”? Is it going to respond correctly?

What are they doing with reliability? Do they have a robustness filter in place? Are they appropriately mitigating hallucinations? What are their human-in-the-loop practices, and what's the human oversight? What are the societal risks that they're tracking and covering?

In one report, the enterprise can look at it and say, “It's been third-party tested. I trust it, and I can see all the detail that I might need and preempt all the questions I have.”

Yeah, got you. So, 500 people is a lot to get on board with anything. How easy or difficult did you find it to boil all this down into a sort of consensus standard?

I'm sure there must have been some things that were contentious one way or the other in terms of, “Including that is overkill,” or, “Not including that leaves this risk unaddressed.” How did you know where the line was? What's the frontier of debate in terms of what's solid enough to be in the standard and what's still the next set of considerations?

We found it to be remarkably uncontroversial for most of the topics. There's a lot of interest in just creating one shared language. A lot of the concerns they had are actually just about churning through the issues and providing clarity on what should be included, rather than debating where the line is.

In part, that's because the standard doesn't exactly specify what should be done; it's more about disclosure. That way, any particular enterprise can decide whether this is okay with them, given that this may differ radically depending on whether you're a hospital or a retailer. They just have different risk preferences.

If you get some of those design principles in place, it's actually relatively easy to get people on the same page. The frontier of the controversies is probably related to some of the more societal harms.

Should we check whether a particular product enables people to create cyberattacks at a much grander scale than before? That may not really be top of mind for any particular enterprise that's applying the standard. It might seem a little bit like overkill.

That's where we've had to make some judgment calls about what will best serve adoption—not just for any particular individual enterprise, but for the industry as a whole. As you were mentioning, when it comes to, say, nuclear, one incident can chill a whole industry. Those are some of the effects that we've started to include in the standard.

We still focus on how to do that in a pragmatic way, where you can actually assess it and get it done without having to spend the dollars to check whether any random bug-breeding tool is going to bring down a whole industry. But we do think it's important to keep tabs on.

So, I guess, how does this look in practice? I was just talking to Andrew Lee a couple of days ago. He's launching this new product called Tasklet, and it's very much a bet-on-the-models agent platform.

His idea is, “Forget all these step-by-step flowcharts and all this ‘this field maps to that field’ that you would see in a traditional automation software experience. We're just going to bet on the models to figure it out. We're going to give them all the tools they need. We're going to give them the memory, and they're going to keep getting smarter and go figure stuff out.”

I asked him, “Would you like to buy insurance?” And he said, “Yeah, we've actually had a really great record so far. I thought people were going to be really freaked out, or that we might see weird rogue behavior from the agents from time to time. Fortunately, we haven't.”

But still, he was like, “Yes, I would like to buy insurance, if only because I can take that insurance to my prospective enterprise customers and show them that we've got this. That should definitely ease our go-to-market process and speed it up.”

So, it's perfectly consistent with your thesis there. But if we dig into it a little bit more, who do you think is primarily pushing this? Is it coming from an enterprise-demand side, or from the vendor side, where the vendor wants to prove its qualifications?

And how much are we getting? Is there any way that we can quantify what we're getting? You do these various things, but what was the risk before? What is it after?

Do we have any way of saying to an enterprise buyer, “Okay, you're going to buy this vibe-coding tool, and we're going to bound the risk in some way where we can make some level-of-confidence assertion that you're not going to have more than this big of a problem”?

What do those final statements that people are buying into look like?

We see demand for what we think of as a confidence infrastructure, both on the developer side and on the enterprise side. With or without insurance and standards, there are still risks. The enterprise is still nervous. They are still asking tons of questions of the developers today and sometimes just not buying because they can't get to confidence because there's too much ambiguity.

The enterprises definitely would like to have clarity: one language, one standard, and one report that they can trust. But the flip side of this, as I was saying, is that the onus is currently on the developers to create clarity individually to earn trust and confidence.

They have all these tools available. They can write public blog posts on their security posture, and that's great, but of course they've graded their own homework. Everyone thinks that their security posture is great.

They will sometimes reach out and make financial guarantees off their own balance sheets. That can be—frankly, I think that's a great way to do it—but again, their balance sheets may not be able to carry it.

We actually see that the most forward-looking developers are as interested in creating this kind of confidence infrastructure as the enterprises, even though, of course, the enterprises are the ultimate beneficiary of it.

To your point around the impact of this on risk, probably the best way to look at this is that when we go and certify a particular AI developer, we run a set of evals that finds a set of vulnerabilities. We'll often do that in multiple rounds.

In the first round, we tend to find quite a lot of things that they themselves are very interested in solving. We often find that sometimes they can get up to a 25% failure rate on certain kinds of attacks against the company, which is obviously not what they want, nor what the enterprises want.

We'll then, with the help of the standard, help them implement safeguards, whether that be a groundedness filter, some kind of monitoring solution, or a content-moderation filter. We can often see those same failure rates drop by 90%.

The other side of this is that there are no panaceas. AI security as a field, and the safeguards in it, are a work in progress, and it's an open research problem. But there are some pretty basic steps they can take, and we can see that pretty quantitatively from our round of red-teaming to the second or third round of red-teaming.

That is the way that we get a little bit more confidence that, through this process, the AI companies that are deeply hungry for improvement actually will improve in a quantifiable way that they can convey and communicate to their customers to earn confidence. That is ultimately what allows them to invest in this stuff when they have a million things that they could be investing in.

I wanted to double-click on the kinds of companies you’re working with. It seems like a tacit assumption of most of the conversation so far has been that they’re focused on application developers who are presumably, in most cases, not training their own foundation models. They may be using a commercial API, or they may be taking an open-source model and doing something with it, but they’re providing the wrapping around that. I don’t mean that in a derogatory way, but obviously there’s a lot that goes around models to make them useful for enterprise purposes.

But there are other layers of the stack that one could think about wanting insurance for, or wanting to do similar audits on. Are you guys doing stuff at the foundation-model layer, or even deeper, at the physical data-center infrastructure layer as well? Or are we limited for now to the application layer?

Okay. Yeah, we think this model applies at all 3 layers. If you think about this kind of alignment of incentives, research into security, and how that unlocks progress, it’s clearly most acute at the application layer at the moment because these are the companies that don’t have large enough balance sheets to make large promises on their own. It’s also, to some extent, where the risk is headed in terms of the fact that they’re the ones building and deploying agents today, even more so than the other foundation-model developers, for example.

But if you think about data centers, we spoke recently to the chair of the audit committee and risk committee at one of the top tech companies. They were saying, “We’re investing an astounding amount of money into data centers right now, and our public shareholders don’t want to be bearing the risk of what happens if the data center goes down.” That could be anything from just wanting uptime to an attack on the data center, or an issue with its security and it gets breached.

They were saying, “Can we buy insurance for that, and can we have assurance from a third party that these risks have been well managed?” That wasn’t historically the case with the data centers they had built and invested in the past.

Now, that’s obviously its own standard, right? AIUC-1 doesn’t currently cover those risks. What we expect we’ll do is run the same process that we’ve run with AIUC-1, where, as you’re going to describe, we talked to 500 executives and understood what the risks and mitigations are, and we talked to our experts. We’ll do the same thing on data centers, where we bring in the right folks who are really leading the thinking on this, both the buyers and the builders of them.

We’ll codify the best practices. We’ll figure out the ways in which you actually confirm that the security is in place, and then we can insure off the back of that. You can imagine exactly the same applying for foundation models.

One way to think about it is as a game of earning insurance confidence over time. Start with the risks that might cost millions of dollars or tens of millions of dollars, move on to insuring risks that might cost you hundreds of millions of dollars or low billions of dollars, and then move on to the tens of billions of dollars eventually. Think of it as a ladder up.

The adjacent places where you have a lot of relatively small incidents give you more data, allow you to build confidence that you have good models, and then let you slowly earn the confidence to move up the ladder.

Yeah, that makes a lot of sense to me, also just from the standpoint of what I see people doing—who seems to be doing a good job and who doesn’t at this stage of the game. I wouldn’t say I know too much about the data-center layer. My sense is that there are a lot of new concerns here for them, for sure, but at a minimum, they’re well aware that they’re putting tens of billions of dollars into the ground. They have very obvious incentives because they’re well aware that they’re self-insuring.

The foundation-model providers strike me as being, with some notable exceptions, remarkably good about this stuff. Again, we could question whether the whole enterprise is a bad idea at a big-picture level, but they’re doing a lot of work to taxonomize risks and to build defense-in-depth approaches and all that kind of stuff.

At the application layer, I guess it’s starting to get better, but certainly for most of the last 2 years, what I’ve seen is weekend hackathons turning into businesses. People never really gave this any thought before they built something and launched it, and they just have zero protections in place whatsoever in many cases.

So I do think, from the standpoint of just where there’s low-hanging fruit, where there’s a lot of needless risk that enterprise buyers legitimately should beware of and should be asking these kinds of questions, the application layer absolutely has a lot of low-hanging fruit. There’s a need to be able to identify and separate who has done a good job and who has made a best effort—even if you can’t take the risk to zero—to get to a place where they can responsibly be deployed in an enterprise or high-stakes setting, and who hasn’t.

There are a lot that haven’t. Would you agree, disagree, or add any color to my sketch there?

I partly agree with the assessment. One piece of color to add is that, at the agentic application layer, the promises being made to enterprises are much thicker in some ways.

If you buy Claude for your own purposes and go to Claude.ai, the promise of what Claude will help you do isn’t very specific. They’re not claiming that they will give you correct medical advice 90% of the time. They’re just saying, “Here’s a helpful tool. Use it for whatever. It will hallucinate.”

If you move into a space like customer support, the kind of public claims these companies are making are, “We will resolve X% of your tickets,” and implying in those sales processes, “We will not create massive brand disasters for you. You can trust us to interact directly with your customers.”

So, the thicker the promises, the thicker the assurances need to be, and the thicker the insurance also needs to be. That’s a little bit of color for why the agentic application layer is the place to start.

One other piece of color is that the distinction between foundation models and agentic applications is blurring. Take an example like Claude Code. It is clearly not a foundation model. It is a system of models, at the very least, with a bunch of different things going on there, a bunch of integrations, and a bunch of access.

This line is really blurring. We think agents are the most helpful frame to think about it, and that’s where some of the application-layer companies got there first. But the companies you think of as foundation-model companies are quickly moving into that space.

So let’s talk about the nature of the audits that you guys are doing. You mentioned red teaming. If I’m an application developer—and I am—I’ve got Waymark. We are in a blessedly low-risk environment because we help media companies make marketing assets, generally for small and local businesses.

My standard line for our team has always been: If somebody breaks into our system and watches all these TV commercials that we’re making for local businesses, nothing bad will happen. That’s a great comfort that we have available.

What if I want to do this? You can go poke at our thing from outside and prompt it or ask it to make hateful or whatever commercials, and you probably could find some gaps, if I had to guess. What more do you do? Would you get under the hood? Would you read the source code? What are all the different angles of attack that you bring to bear on figuring out the many ways an application could go wrong?

Totally. The foundation is to start by grounding ourselves in the real-world incidents that have happened, so we can figure out what it is we’re looking for. We have a database of real-world incidents that have happened that inform the things we should be checking for.

Some of those are things that you might think of as very concrete headlines. BBC says, “Wow, Air Canada’s chatbot hallucinated a refund policy.” Great. We should clearly check for hallucinations that have financial implications.

Some of them might be slightly more speculative, like a research paper that shows that, under these experimental circumstances, AI chatbots will try to blackmail the user. We also think of that as an incident; it’s just slightly more speculative.

That forms the foundation for the audit. We then try to abstract that into a taxonomy of both the harms we’re checking for and the kinds of attack vectors that are relevant: the latest research into jailbreaks, and the latest examples of people playing with prompts on Twitter to find different ways these models break. That becomes the checklist of things that we want to check for.

The way we check it is, broadly, for each of these harms and risks, we specify the technical, legal, and operational safeguards you must have in place.

And what are the evals you need to run? The technical safeguards could be, “Oh, you must have a groundedness filter that reduces the chance of hallucinations.” Or it could be that you must have a plan for what happens when it hits the fan. What is your AI incident plan?

Those get checked either by looking into the codebase, reading manual policies, and so on. Secondly, there’s the technical testing element—the red teaming—which really grounds everything. It’s not enough to have an AI groundedness filter in there if it doesn’t work. So, that’s where we take all these scenarios and try to break the chatbot to see how effective it is.

The output of this is, you can imagine, a report that says, for each of these technical, legal, and operational safeguards, what do they have in place today? And for the evals, it displays the results of what we found in a way that both helps enterprises make sense of it, but also leaves room for them to make the final judgment call, with some level of confidence, about whether they should go or no-go.

So, one way to think about this, Nathan, is that there’s a trust-and-verify approach that we take, right? What are the policies you have? Have you implemented the safeguards? And can we verify that they work?

Again, you could imagine exactly how this would then also translate to foundation-model developers, where they have these policies on what safety testing they’ll do and how they’ll implement various safeguards. How do we actually verify that’s happening?

Yeah, you mentioned that there are multiple rounds of this process, and I think it’s probably worth talking about why that’s important. I was just reading a paper from Nicholas Carlini, Sander Schulhoff, and others. I’m sure you’ve seen it, “The Attacker Moves Second.”

The basic notion of that paper was that if you take the attacks as static and say, “Okay, I’m going to tweak my defenses to mitigate those attacks,” you can do that, and you will mitigate the statically defined attacks that you were trying to mitigate against. However, when you then deploy those defenses into the real world, with potentially adversarial attackers coming at you, they find ways around those defenses at a remarkably quick rate.

What they found in their study was that human red teamers were basically 100% undefeated. There were no defenses that were totally impermeable to human red teamers. So, how do you go through an iterative process to not just have this kind of security theater of, “Well, we had this one, and we had one shoe attacker one time, and now everybody takes their shoes off at the airport, and then we tell ourselves we’re good,” as if there were no other way that you could get something onto a plane?

How do you think about that iterative process, and how do you know when to stop? This could obviously go on forever, too, right? How do you even decide when you’re getting to a point of diminishing returns, or a point where you’ve hit some sort of best, good point on a Pareto curve?

There isn’t a single neat answer. Ultimately, this is a judgment call. The important thing to know is that we do a few rounds every time we do the auditing, and then, to get certified, the companies commit to doing quarterly audits.

They will change their product. The world’s incident landscape will evolve. There will be new research papers into new ways these systems will fail, which we then continuously incorporate into this body of tests that we run each quarter.

In practice, it becomes a question of cost and effort versus confidence. These are not new considerations at all. Cybersecurity has the exact same thing: with enough effort, every system is reachable. I’ll say that’s pragmatic, and over time, the cost of running good audits goes down as intelligence becomes cheaper, but the set of attack vectors probably also grows.

So, there isn’t, in fact, a very neat answer for where you stop. Ultimately, this is guided by the market. What level of assurance do you want, and what are people willing to pay for to get that assurance? It’s a messy answer, but it’s the actual pragmatic answer for how most of this stuff gets settled in every domain today.

Yeah. What does this look like in terms of your business model? Who pays you? I suppose the developers pay you. How much do they pay you? How does that depend on what sort of product surface they have?

What does that quarterly update process look like? Is it the same thing as the original, or is it more streamlined as you go forward? What’s the overall customer story for the AI auditing company?

Yeah, totally. The company that’s building their system pays. That can either be what you think of as the disruptors in the space, these AI-native companies, but it can also be enterprises that are developing something in-house.

You’ll see a bunch of enterprises building their own customer-support agents because they think that’s a real moat, and they still want to have confidence before they deploy that. So, it’s the companies that develop the systems that fundamentally pay. They pay on a yearly basis, and the certificate lasts for 1 year. With that, you’re required to have these quarterly technical tests.

The pricing depends, as you’re saying, exactly on the risk surface and the product surface. This also ties to value. Bigger companies tend to be able to generate more revenue out of this, but they also tend to have a bigger risk surface, so it scales a lot.

Other parts of the business model that are interesting are that the same companies that are interested in getting certified overlap a lot with the companies that want to have insurance. For the same purposes, they want to take risk off the table for their customers and deploy faster. For the enterprise, they want to reduce the risk so they can deploy faster.

That insurance business model also works on—you typically buy an insurance policy on a yearly basis for a yearly fee. As you can imagine, these layer on top of each other.

We’re experimenting a little bit now with an AI-native company that wants not just to get certified for the product in general, but also wants to certify specific deployments to specific customers. They want that extra layer of guarantee.

They may want to know, “I’m a bank deploying this AI voice customer-support chatbot. I want to know that, with my specific customer-support policies, my specific customer set, and my specific policies, this will also work.” That’s another way to get more precise in the value.

That’s where these disruptors are really willing to pay, because it’s going to unlock a contract. But it’s also very clear that this is the maximum level of assurance an enterprise can get when it’s done literally in their environment.

Can you tell us a little bit more about what the requirements are from the customer side? I’d be interested to know if you’re willing to share tangible price ranges. I assume this sounds like it starts in the maybe five figures and probably goes into the six figures, depending on how big the product surface is.

Do companies need to make team members available to you? What, beyond the desire and willingness to pay, do they have to bring to the table?

First, before we sign a contract with anyone, we do a gap analysis. We take the standard and, from the outside in, look at what evidence is publicly available that gives us a sense of where they stand against the standard.

We then do an in-depth review, row by row: where do you actually stand today? That gives us some kind of effort estimate. What’s the amount of engineering effort you need? What’s the amount of non-engineering effort you need? And from our side, what is the total scope here?

That’s where we come together and say, “What’s the price going to be here?” The range I mentioned is broadly correct, and that gives everyone a sense of the costs involved on both sides. When they commit, they commit for a year. At the end of that year, they can decide whether they want to recertify.

This is very much in line with how they already think about certifications in the cybersecurity space, things like SOC 2 and ISO standards. A gap analysis is the core thing that tells everyone what effort is involved and what the cost is going to be. We can do that, frankly, within 24 hours.

What we find, Nathan, is that anywhere from our largest customer, which has thousands of employees and has been running for years—for them, it’s a small handful of hours of actual engineering or security-team lift, actually, in that case, and just evidence collection for us—to our smallest customer, which had 2 or 3 employees when we started working with them.

We needed to work with them in a much more hands-on way to implement the security measures required in the standards and remediate the issues we found through the testing. That was obviously a much more involved process, so it’s kind of hard to give a stock answer without doing that analysis.

Yeah, well, it seems like something everybody’s going to need, right? In a similar way, whether it’s me as an individual driver of my minivan or a nationwide trucking company, everybody’s going to have to have some sort of insurance to offer it, presumably before too long.

Do you think that’s true? Is there a class of AI app that, for whatever reason, just never gets into this, or does this ultimately become something that really everybody has to do, in the same way that drivers do?

I might separate the insurance from the third-party testing for this, right? I think for almost every kind of valuable use case you can imagine, there is a need to have somebody who is not just marking their own homework actually test it and say it is secure to ingest customer data or a company’s private, sensitive IP and data, and make sure that’s all being protected, et cetera, et cetera. And then I think on the insurance side, I think that’s where you could imagine more of a bifurcation.

Traditional SaaS has been sold on kind of a “have a nice day” basis, right? Often these aren’t insured. You might get some service credits if something goes wrong, but you’re typically not going to get more than that. If you compare that to cars and houses, obviously everyone carries insurance, and in lots of cases this is mandated. You could imagine one version of this is a similar kind of bifurcation where the highest-risk deployments and the places that are creating externalities also really should be insured because that’s where you want an extra layer of protection from a strong, established balance sheet to provide the financial coverage. Whereas if I take your example of producing media copy, you could say the risk here is manageable and it doesn’t need insurance.

Yeah, makes sense. I was briefly in the financial services industry, as I alluded to, and I happened to have another one of these Forrest Gump extra roles in an important scene in recent history: the problem of the credit-rating agencies. Basically, what happened in the late 2000s is that these companies were, in a sense, captured by their customers. They had the incentive to say things looked good. They didn’t have the incentive to turn over too many stones, and so we ended up with all of these very highly rated bonds going bad.

How do you think about preventing that outcome? I guess there are probably multiple dimensions to it. One is just, how do you make sure that your audits stay weird enough? It almost seems like you need heterogeneous cognitive profiles on your team to make sure that you are bringing weird and off-model ideas to the table as often as possible. But then there’s this incentive question, too. At some point, as this gets normalized, there’s going to be some pull, or there might even be some competition.

That’s where I think the rating-agency thing went bad. It was like, “Well, if you don’t rate us high, we can always go to the other rater, and they’ll rate us high.” So that’s like, “Well, geez, what do we do? We’re going to lose all our business by giving our own customers bad ratings.” That’s a tough line to hold. You’re trying to create a race to the top, but there is at least some precedent for this sort of thing ending up in a race to the bottom. How do you think about just making sure we stay on the right side of history here?

The core thing that’s going to create the right incentives here is the insurance. That gives us the skin in the game not to just lower the standards. When you take the ratings example, Moody’s does not have financial skin in the game outside of trust. So when the financial crisis hit, there wasn’t a direct financial mechanism that forced them to internalize the risks that were being created there.

When you look back at, for example, the UL certification that’s on all of your electrical products, which I think of as broadly a very successful instance of standards, that was created by the insurers. It started from the insurers having data on what was going wrong and having the direct incentive to make sure the standard covered that. So we think the insurance piece creates the skin in the game to prevent this race to the bottom. There’s a lot to learn from history there.

In terms of how we keep up with it and keep up with the frontier, Nathan, this quarterly process is actually just really involved from our side. We have a mix of the actual people we’re certifying and the agent builders, because they’re on the ground saying, “Here’s a new safeguard that’s really working well for us.” Great, let’s feed that in. We have a range of academic researchers and other people building safeguards commercially, giving us input—people from Stanford, the University of Illinois, Haize Labs, Gray Swan, and Venture AI.

With enterprises, we’ve just launched our enterprise consortium, which is a mix of security and risk leaders from Fortune 500 enterprises who are also on the front line of the deployments and saying, “Here are the risks I’m worried about and the risks I’m seeing.” All of these people come together to give us input on what’s missing, where the frontier’s headed, where our standard isn’t keeping up with that, and how we write the precise requirement in a way that will capture the spirit of what should be in here.

So really, it’s not just about our team here, which of course has lots of these capabilities and everything you’re describing. It’s really about how we mobilize the whole ecosystem around this and bring that together.

Do I understand correctly, just to rewind to the insurance piece and the incentives: Are you planning to actually provide insurance? It sounds like you’re both going to, if I’m understanding correctly, have skin in the game at the insurance payout level. Is that right?

Yeah, we’ll have skin in the game, Nathan. The way this works is we’ll become a managing general agent. That’s the kind of insurance term for it. In practice, what that means is the insurance will pay us, and our payouts will be tied to the actual underwriting result—the profit that they get on the insurance product. So if there are large losses, we don’t get paid, or we get paid much less. If, in fact, our risk assessment was good and there are fewer losses, we also benefit from that financially.

Interesting. And if there are large losses, the insurance company will not want to work with us.

We’ll go out of business.

Yeah, gotcha. Is there a big difference between that and writing your own insurance? Is it just a matter of the size of the balance sheet that causes you to structure it this way, or do you imagine at some point building a giant balance sheet and underwriting your own policies and doing your own payouts? Or is there a reason to let traditional insurance companies handle that indefinitely?

In insurance—and actually, reinsurance is often where the risk sits—it’s really a cost-of-capital game. It’s kind of a risk-aggregation game, as we were talking about earlier. There’s a lot of benefit there in being diverse across different risk lines, across selling through brokers who are selling multiple products, et cetera.

Really, we want to focus here. We’re focused on the AI risk, and so for us this gives us the benefit and flexibility to go out and write and structure insurance products, knowing we’ve got insurance partners—some of the most established insurers in the world who’ve never failed to pay out a claim. They’ve been around for hundreds of years. This is their business, and we’re really excited to write with them rather than having our own balance sheet.

I’ve been following this SB 813 proposal for a while. The idea here, as I’m sure you guys are very well aware, is to create some sort of private governance constellation of options where companies could opt in to abide by a certain standard. Maybe you could even be one of these multistakeholder regulatory organizations, I think they’re called. In exchange, the companies get some liability insurance. There are also a bunch of other proposals for liability reform or changes in all sorts of different ways.

Do you have ideas on policy? What policies do you think would work best? What are most compatible with the business that you’re trying to build? Are there any that are in direct conflict with it? What do you think is the future of that?

From our perspective, anything that promotes the creation of a market here and market-based solutions is the key, for the reasons we talked about earlier. That’s the thing that aligns the incentives, because if you have a free market of oversight, you’re going to actually incentivize robust risk evaluation without hampering growth, and that’s really important. We don’t want this to become too onerous and crowd out the startups and crowd out innovation.

There are plenty of policy ideas, like you mentioned, that do allow for this and do allow for and promote the creation of best practices and third-party oversight. The other thing that’s important, though, is what we think SB 813 didn’t do: actually force the companies who are creating the risk to own it, right? We don’t want to create safe harbors where you have too many free passes, where you can create risk and then you’re not punished for that and you’re not held to account for that.

And so we think that's a fine line, and we think insurance actually allows you to internalize these externalities and take on the risk you're owning while still creating the market around it.

Last question: who all was involved? I understand there are quite a few well-known companies that are starting to buy into this. So maybe give us a little overview or name-check of who's hopping on the train.

Yeah, on the startup side, we've had an awesome group of founding technical contributors from various different spaces. We work closely with Ada and Intercom in the customer support space. We work with Cognition on code, and we've worked with a company called Recraft in image generation. So we're trying to really look across the spectrum of the different kinds of modalities and deployments. Maybe Rajiv, you can speak a little bit more to the kinds of companies that are in the enterprise consortium.

Yeah, so we've got risk and security leaders from financial services, like JPMorgan Chase, for example, who are really thinking about how do we adopt AI across the bank globally, not just in specific verticals. In tech, the chief information security officer of Confluent, for example, is helping to shape AIUC-1. The deputy CISO at Anthropic is deeply involved, and we've got others from healthcare, retail, et cetera. So it's really trying to represent across industries and actually also across geographies. We've got folks from Asia, Australia, and Europe, and we're really trying to bring this together into one framework and make sure it's as strong as it can be.

Cool. Well, we're out of time. This has been great. Any last thoughts you want to leave people with, or anything that we didn't touch on that you want to make sure to mention?

I think this is pretty comprehensive. Love it.

Cool. Well, Rune Kvist and Rajiv Dattani, co-founders of the AI underwriting company, thank you for being part of The Cognitive Revolution.

Pleasure. Thanks for having us.