AGI的守门人——AI承保公司Rune Kvist

If you think failure concerns are bad, just see what happens when Waymo hits a dog—people lose their minds. Imagine when the first robot knocks a toddler off a kitchen table: you're going to see some real strict liability. So for physical AI, the level of stringency just goes up and up and up and up.

That's the big picture: agents, models, and robotics. As the technology progresses and agents get longer horizons, new types of failure modes will emerge. Those will bring new ways to create value, but also more risk surface. You'll start to see true agent-to-agent interactions that are not mediated by humans. There are going to be a bunch of interesting questions. You're basically going to need a new legal system. How do they build trust among each other?

1. AIUC’s $40M Round and the Risk Bottleneck for AI

Okay, we're in the studio with Rune from AIUC, an AI underwriting company, with our trusty co-host, Vibhu. Welcome.

2. From Scaling Laws to Early Anthropic

Thank you. Thanks for having me.

What are you announcing today?

We have raised $40 million, led by Ribbit Capital and FirstMark.

You first came to my attention when Nat and Dan invested in you guys. Is the story pretty much the same? Are you where you thought you would be back then?

When we raised our seed round, we had a hypothesis that, at some point, risk was going to hold down adoption. At that point in time, that felt hypothetical, and I think that is now over. Clearly, the moment is now with Mythos and Fable. It's pretty obvious that the binding constraint on adoption is risk.

For us, it feels like this is a natural continuation of the same hypothesis, but where previously it was speculation, now it feels like fact.

Let's get the list of customers you're highlighting as part of your Series A.

Totally. We are now working with folks like Cursor, Harvey, Lovable, and ElevenLabs.

Amazing. Congrats.

Thank you.

You were famously one of the first hires in GTM and product. I'm curious: what was your path into AI?

Recap.

Late 2021, I sold my first company, an edtech company. I had a bit of time to think about what was next. I came across a scaling-laws paper, and that struck me like lightning. I thought, “This is a big idea.”

In short, the scaling-laws paper says the bigger the model, the smarter the model.

This is the Kaplan one, not the Chinchilla one.

Exactly, the Kaplan one. The important thing that clicked for me was, “Now capital will understand this.” If you put in more money, you get more money out. That will kick off a hype cycle, and you'll get a sense of predictable returns, which is in fact what's played out.

So I packed my bags. I'd never been to San Francisco. I flew out here to find the people who'd written it. At the time, they had just started a small lab called Anthropic. There were around 40 people.

I drank a bunch of coffee until I eventually got introduced to Dario. At the time, they were wrestling with questions like, “Should we deploy our models? Should we make revenue? How should we engage with the rest of the world?” They had just broken off from OpenAI. It's been publicly reported that they were concerned with how OpenAI was dealing with deployment, so they were wrestling with some of those questions at that point.

This was the early fog of war, early 2022. The sexiest product at the time was Jasper; there was nothing out there. Where was value going to accrue? What were going to be the different parts of the stack? Those were all open questions.

I want to highlight to people that you ask these questions because you have a PPE background.

I was in Singapore in one of the feeder programs for preparing people for PPE. I had a tutor. We learned philosophy, politics, and economics.

But I think machine-learning people who read the neural scaling-laws paper would not necessarily draw the same conclusions that I did, whereas any capitalist would read that and go, “Holy—”

Correct.

Right. Who tipped you onto that paper? It's not a paper that you would normally read in your circles, right?

Yeah. I think I'd had some appreciation that AI was a big deal ever since AlphaGo. But it raised all these interesting philosophical questions, and it wasn't clear from afar where exactly that would go.

It was obvious enough that this was going to be a big thing if we found the right mechanism to get the techno-capital machine to work on it, but it just wasn't clear. I think it was a moment when that became obvious, and it wasn't as obvious at the time as it is now.

It was just, “Wow, this is so interesting.” But I still felt, coming from a philosophy and economics background, that if this turns out to be true, you're going to be wrestling with all the big questions in society. Everything you've learned about politics gets thrown out the window. Everything you've learned about economics, at least, gets challenged.

What felt interesting was being at that frontier that has ramifications across everything. That's why I thought it stood out.

I mean, clearly, really good insight. For people who don't know, the PPE program is where prime ministers are born. So then you end up meeting Dario.

Yep. First Dario.

Well, did you get extra insights from talking with them that you didn't get from your original hypothesis?

If you read the scaling-laws paper, you get this very vague sketch of, “Wow, this seems kind of important. There are some lines on a chart. This seems kind of important.”

What I think the team at Anthropic had thought more about than anyone was, “What are the implications of this if you really play it out?” Back then, they had vision documents for what the world would look like in 2026. They were playing out, in vivid detail, how much compute was going to be needed, what the capex was going to look like, what some of the societal concerns would be, and also what the amount of economic value coming out of this would be.

It felt like they held a crystal ball that, in hindsight, turned out to be dramatically correct. They weren't holding it as though they were obviously correct. They were just saying, “Take this hypothesis really seriously.”

Think it through.

And think it through. In the same way, the kind of situational awareness that is now—

Across the street. Yeah.

Oh my god, we're all in the same one square mile.

Right, and that's now a couple of years old. People also keep referencing it these particular weeks with Fable and Mythos, and it's like, wow: if you take this one idea seriously, the scale and a lot of things fall into place.

Keep in mind that, at this point, this was the same team that had done GPT-1, GPT-2, and GPT-3. It's not just some experiment. This is a real model that we just scaled up.

They had deep conviction in this big idea: if you take a big blob of compute and data, it just wants to learn, and out of that will come smarter and smarter models.

Yes, and all the implications were not clear. But that deep conviction was the core thesis, and that was dizzying. It was both phenomenally interesting and exciting, and very quickly you got to, “The world we know today will no longer be the same if this hypothesis holds.”

It also felt important in some grand sense.

What shaped you there? That was early 2022. Not only had GPT-1, GPT-2, and GPT-3 come out, but you also had the amazing co-founders of Anthropic—the ones who actually had the conviction to leave OpenAI and start their own lab. You said there were about 40 people there. What was the time like?

It was remarkably like what it looks like on the outside today: extremely cohesive, extremely mission-oriented, and living in the tension between two ideas—that AI could go really well and really badly, and that we wanted to be part of building it.

That creates astounding amounts of tension. They were wrestling with this incentive challenge: they knew there was a race they were in where they might be forced to cut corners, but it also felt very important to them to be at the forefront of technology. All of those ideas were present at that time.

It feels like that line has been very, very clear, and I think, love them or hate them, they have really stuck to their guns. There is a core set of beliefs that they hold more deeply than most companies hold any beliefs.

3. Why Trust, Not Capability, Could Limit AI Adoption

Yeah. Fast-forward to today. What does that lead us to, AI Underwriting Company? What are you up to?

4. How AI Agents Are Audited and Stress-Tested

AIUC builds confidence infrastructure for frontier AI through standards and insurance. The link from Anthropic to building confidence infrastructure was looking out the windows at Anthropic’s offices and seeing Waymos driving by, already back then in early 2022. Waymos were, in some ways, like AGI for cars: there were superhuman drivers, but you couldn’t take one to the airport. And now, 4 years later, you still can’t take a Waymo to the airport, despite everyone having looked at the evidence and said, “They’re better drivers than humans.”

5. Prediction Markets vs. AI Audits

So, in that particular instance, what’s clear is that the binding constraint on AI being useful is not capability, but liability, risk, or trust. That problem is general. The reason why Fable is not open for access right now is not because it’s not a good model; it’s because it’s a very good model. It’s just hard to make promises about what it will and will not do. And this problem gets worse as AI gets better. Basically, more intelligent AI can be more autonomous. That’s more valuable, but the risk surface also grows.

What Waymo illustrates is that unless you build the confidence infrastructure to make promises about AI, or at least bring light to the risks, you grind adoption to a halt. Governments, banks, hospitals, and militaries need to have some sense of what AI will and will not do for them to be able to incorporate it. That’s the problem that we’re trying to solve.

Now, why standards and insurance? If you trace this problem back through history, every technology wave has had some version of this problem. If you go back to around 1900, electricity comes out. Houses burn down, sorry—lots of houses burn down, and lots of people die. In the 1930s, cars are a big deal and kill lots of people. In the 1950s, private nuclear energy is a big deal and poses big risks.

In each of those instances, the market runs ahead of regulation to create confidence infrastructure, because that’s required to make go/no-go decisions. It’s required for adoption, and the market fundamentally wants adoption. In all of those instances, a common blueprint emerges between standards and insurance.

The reason it’s these 2 components is that standards provide the rules of the road, and they also specify what tests need to be run so we can get a sense of how high the risk is. Taking the case of cars, car crash tests inform your insurance pricing today. They inform your purchasing decisions, and so on. That’s basically the risk framework.

The insurers are important because they pick up the bill. They’re the private institution most incentivized to quantify the risk truthfully and then figure out all the ways to reduce the risk, because that increases their profit. They help shape the incentives, and these 2 things work really well in unison.

Now, how does that show up as a company? One of the things that was obvious, or starting to become obvious, even a couple of years ago was that frontier companies—some of our customers today, like Cursor, Sierra, ElevenLabs, and Harvey—were going to have a very easy time selling a pilot to a bank. The demo just sells itself. It’s magic.

But bringing that through, if you want to do a wall-to-wall rollout at a bank or a hospital, you have to go through the risk process. These banks have no idea which questions to ask, let alone which answers are sufficient, let alone how to test whether these agents actually work the way they’re supposed to. They had this problem of, “What can we say to earn the trust?”

We think there’s a golden sentence that goes something like: “Hey, I hear you’re really worried about hallucinations or jailbreaks, or whatever it may be. We’ve had an independent third party test us against the gold standard. We passed with flying colors, and as a vote of confidence, the world’s most conservative insurers have looked at the data and are willing to take some of the risk onto their balance sheet.”

So if something does go wrong—

There’s money behind it. Yeah.

Exactly.

That’s kind of the link between all of them. We can get into some of the hard parts related to the technical testing, which is, I think, the crux of the matter. But I’ll pause there.

6. Founding AIUC and Building AIUC-1

How did you and Rajiv come together? There’s always this sense—you come across as very confident, and you’re announcing your Series A and all these things—but I want to see the early, initial stages of idea formation.

Yeah. Rajiv is actually my soon-to-be brother-in-law. So I’m actually, in a week and a half, getting married to Rajiv’s sister.

Okay, now you’re tight.

Exactly.

You know, so Rajiv and I have known each other for a decade. Funny story: I met both Rajiv and his sister, Hannah, at the same time, when Hannah and I were interns at McKinsey in London and Rajiv was assigned as my mentor. So we met them at the same time.

For the longest time, it wasn’t obvious that we were necessarily going to work together. I was in startups. He was an insurance partner at McKinsey 3 or 4 years ago. I think Hannah convinced him that AI was going to be a really big thing, and so he quit his cushy partner job at McKinsey in London, packed his bags, flew to San Francisco, and ended up joining METR.

7. The Impossible Enterprise AI Mandate

You guys are probably familiar with METR. Exactly—we see the chart of the horizons of the tasks that agents can take on doubling extremely fast. At METR, he led their partnerships with Anthropic and OpenAI to test their models before release, but also worked closely with the US and UK governments to figure out how you know whether a model can be released. In some ways, that’s the perfect background. He’s spent a lot of time in insurance, knows that world, and spent a lot of time with frontier testing of models.

When I was bumbling around this idea space, starting with some of the ideas we talked about related to Waymo, as soon as we got into the context, we were both like, “Oh, this would be an amazing business to build together. This is wrestling with the problem that we both think is the most important in the world.”

From a market angle, our intuition is that the market can do a lot. The faster AI moves, the harder it is for the government to solve some of these problems. It took a little bit of time to work through what it’s like to work with family.

Yeah, because you’re already dating at the time?

Yeah, yeah. Exactly.

Already back then, we felt like we were a family, and starting a business together felt like a big step—and here we are with immense amounts of trust.

So now you’re a company of how big? How big are you guys?

There are just 20 of us now.

Now that you guys have your Series A and your first certification out, the AIUC-1, let’s bring up the certification. So, this is the agent certification, right? What goes into the process? I have 2 questions here. One is, walk us through the certification, and 2, what is the process for a company to get certified?

Great. As it says right at the top, AIUC-1 is a standard for agent security, safety, and reliability. The fundamental design principle is to take all of the concerns that slow down adoption—all the questions and fears that keep security leaders in the Fortune 1000 up at night—and put them into 1 comparison framework. That’s what you’ll see there. You can see the 6 categories.

You want to ground all of this in technical testing. One of the concerns with security standards is that they often feel like theater or paperwork. They don’t actually answer: Does any of this work? Does any of this matter? We had a conviction from early on that this was going to be the crux: to pass this, you must get tested every quarter, basically running thousands of simulations to see whether it can actually be jailbroken, how hard it is to jailbreak, how often it hallucinates, how often it leaks data, and so on.

The last core idea here, if you scroll up to the top, is to refresh it quarterly. The core trait of AI is that it moves extremely fast. Whatever concerns we’re discussing today were not the same ones 3 months ago, and this will keep changing. Typically, standards update on a decade cycle, which obviously isn’t going to work.

The question is how you update it. The core thing here was to get the risk leaders of the Fortune 1000 around the table. If you go over to the left, you’ll see the AIUC consortium. The consortium is a group of risk leaders who run real banks, real hospitals, and real critical infrastructure, and who are facing these challenges every day. We meet with these folks twice a quarter and hear what’s top of mind and what’s keeping them up at night.

There’s a tremendous amount of desire for that conversation, and then we operationalize it into a specific standard. We can actually go in and look at what a standard even is. If we go back to the introduction over there to the left and scroll up a little bit to the wheel, click into reliability. If we take something like hallucinations—

Hallucination system reliability: there are a number of requirements here. If you go into the top one, “Prevent hallucinated outputs,” this is 1 particular requirement. This is a technical control. Basically, we want some kind of groundedness filter.

The first thing you see here is what’s called a crosswalk. Everyone and their grandmother has put out a framework—a very high-level framework for what the AI risks are.

This is basically your competition.

In some ways, our competition—we're in fact friends with them. We'll come back to why. But the idea is to map everything together so you have one superset. The claim you're trying to support here is that if you follow this framework, you can also see how you follow the other frameworks.

The meat of it comes down to control activities and evidence. Control activities are: great, you have this high-level requirement—how do you turn that into something operational? Here's what you must do, and here's the evidence that we're looking for. The reason we go this deep is that there's actually not that much confusion about the big concerns in AI. Everyone agrees on those. The question is: what are you actually supposed to do?

What we found a lot of demand for is getting down to the specific evidence that people need to look for, whether you are Cursor building something or JPMorgan building something. But also, if you're just a risk leader at JPMorgan, what exactly should you ask for? What can you ask for without sounding stupid? You won't believe the amount of time a risk leader has asked for the IP rights to the underlying model from Cursor or something, and you're just like, "Sorry, what?"

You slip it in there and see if they notice.

Exactly. Put that in the questionnaire. That's kind of what a standard is. We update this every quarter with these folks to keep up with the latest concerns.

Can I double-click on this one?

Yeah.

First of all, the website's beautiful. It's so confidence-inducing, which is the whole point. I know exactly what I'm signing up for when I talk with you. I don't even have to talk to you; I can just see your whole certification, which is great.

But from here, with D001.1, the configuration grounding filter, how does that get applied? Do you have a person that—

Yes, someone goes through it.

I can see somewhere there's, you know, 51 requirements and 130 controls. There's a whole—right, to me, this doesn't translate into a test.

Yes, yes, yes. If you go into it on the left-hand side—actually, before we go in there, there are 3 types of requirements. The first is technical controls, like: you must implement some guardrails.

Second, there are test controls, so you must have an independent third party run some tests against you. I'll show you one of those in a second. Third, there are policy controls. For example, you must have a person whose name is on the line when you sign up, and you must have a plan for how you tell your customers and how you engage with them. They're more traditional, standard-type requirements.

In this particular instance, we just check whether they in fact have a grounding filter. We partner with auditors like KPMG or Schellman, who go in and do the thing auditors do, which is check the evidence. In this case, that might be a screenshot. It might be part of the code that they need to review to see that it exists.

So you're not testing the effectiveness of it?

That's the second thing. If you go down to third-party testing for hallucinations on the left, that's basically the next requirement. This is where we test how well it actually works.

Okay. And is it you testing or the auditor?

We test them.

That's a lot of work. How long does testing take? If I want to get certified, how long does the end-to-end process roughly take?

The end-to-end process almost always depends on how much our customers need to learn from us. It takes somewhere between 3 and 10 weeks, depending on how up to snuff they already are.

8. AI Liability, Monitoring, and Earning Enterprise Trust

Some people show up to us with extremely rigorous security programs. When we test them, it works extremely well, and we can get that done very quickly. Some people come to us and they're not that far along. We give them the specifications they need to build toward, and then their security teams and engineers get to work and build to meet the standard.

The testing itself typically takes a couple of weeks, including the time for them to remediate. Often, we'll find something that we cannot pass: "Hey, this is actually just not up to the standard. You won't pass the standard." Then they need to implement additional safeguards or remediation that makes them more robust, so they can honestly look their customers in the eye and say, "We've truly done our very best."

And they're certified for a year and have quarterly updates.

Correct.

9. Cyber, Child Safety, and AI-Enabled Biological Risk

Yeah, it's pretty interesting. What's changed since then? This is certifying agents in production, right? Your customers—you've had Lovable, ElevenLabs, and Intercom all go through this certification. What has changed? I see you post that Q2 added MCP and agent-to-agent communication. Are there any other things you want to highlight since the first iteration? What comes in quarterly?

Some of the changes have come from realizing that agents are not just one thing. If you take agents like Cursor and compare them to Sierra, they're really quite different. Compare them to Harvey or ElevenLabs—they're all quite different.

We wanted to design a standard that works for all types of agents. We started with one that was pretty text-based and, honestly, pretty customer-support-focused, because that's where there was a lot of existing demand. Over time, we've picked some of the frontier companies in each of these other domains that we could work with and used them to build out the standard, so that we know the same standard works for code, customer support, automation, and so on.

That's been one big thing. Some of the things that have been top of mind recently are that Anthropic is bringing up a lot of concerns for security leaders. We're starting to get more and more questions around agent-to-agent interactions. It's very nascent at the moment, but it's starting to emerge. There have been a lot of questions related to OpenClaw and MCP. Agents starting to interact with each other is really top of mind.

As coding agents have really taken off, banks, hospitals, and others are getting more precise about what they need. We're really dialing in where most of the tokens flow through in the world and getting much sharper on that.

Can you share, for people who don't really think about this, what the best practices are when building agents? If they come to you pretty ready for certification, they'll probably pass certification. What are the things people don't think about that they should have?

The most important thing is that a lot of companies have not done a serious stress test. They spend most of their time, perhaps rightly so, optimizing for how it works in the good case, the average case, and how high-quality the output is for the customer.

A lot of these companies are pretty new, so they haven't spent a lot of time stress-testing what is there as an adversary on the other side. What are some of the complicated corner cases that you've not really considered? I think that's a frame of mind, and you'll also see this in startups. It often takes a while until they hire their first security person, and that's a whole different kind of risk surface from just building a good product.

Most companies also have the right kind of architecture. Most of them will have some kind of guardrails in place. Either they come out of the box from their model provider, or they'll have built their own filters to sit in between. They just don't work very well.

The difference between putting a classifier in place that maybe checks whether you're giving medical advice when you shouldn't and says, "Hey, if this looks like medical advice, filter it out"—lots of companies have that in place. The question is whether it works.

It's actually pretty fiddly to sit down and think about all the ways in which you could ask for medical advice. You have to read the academic literature and consider the kinds of framings or tricks you might use to get an AI to give you medical advice when you really shouldn't. There's an area of expertise that's just missing.

What we find is that most people have the right building blocks in place. It's not rocket science, but the finicky thing is getting into the corners and testing whether it works, so that you can look your customers in the eye—whether they may be a bank or a hospital—and say, "This is going to work for you."

10. Frontier Models, Government, and the AI Trust Gap

I see. We talked a lot about agent-level certification. Where do you go from here? You're announcing your Series A—off camera, we talked about this a bit. There's the whole security risk of the federal government stepping in. You guys are also announcing that you're going into model certification.

When we do a bit of editing afterward, we will not yet be announcing this. The question that's top of everyone's minds now is at the model level, and Mythos and Fable have really brought this to the fore. In addition to the commercial risk and the kind of economic security risks that are happening at the agent layer, the models are going to present risks in the national security category. The shape of the problem is very similar.

You have some people who are on the hook if something goes wrong. In the case of agents, it's often the security leaders in the enterprise. In this case, it's the government. They haven't necessarily spent their entire lives thinking about what the new risks are, what kind of data you might be looking for, or how you might test that. But they do have to make sure that their concerns are addressed.

You have some frontier companies that are deeply technical. They know a lot about the risks, but they fundamentally have an incentive not to always be truthful. So you have a trust gap between the government and the labs. In every other industry, you end up with some kind of body—a neutral third party—sitting between those people. There's no other industry where we allow people to audit themselves.

There is going to be a need for a third party that can take the rigor of the labs to run frontier technical evaluations, but can also speak legibly in the way that the government trusts PwC to run financial audits. They know that PwC outputs audit reports in a way that's consistent, easy to read, factual, and trustworthy. Those two things need to be brought together.

What we've learned from our work with agents is that, if you want the communication between those two parties to be smooth, there has to be one common standard that's public and that people can inspect. What are the risks that matter within each of these risks? What are the threat models that you're really looking for? You need to specify, for each of those risks, what guardrails need to be in place and what tests you need to run to see whether those guardrails are effective.

Then you need to run audits that are technical and consistent. If you're trying to bring trust, it's extremely important that you methodically work your way through the risks. You can't send one researcher in and say, “Come back with whatever you find.” You need to be able to explain exactly what you did, exactly what you tried, exactly what you did not try, and therefore the kinds of promises you can and cannot make at the end of it.

I think of Fable as a direct symptom of this problem. The government was told that there's a risk. The government may struggle to assess just how big that risk is. They call Anthropic, and Anthropic is trying to tell them, “Hey, actually, every model can be jailbroken.”

That's not what you want to hear, right? As a government, that might be hard to trust.

And we think that a broker is the most natural solution. In other markets, you see something like this. In financial markets, you see Moody's. Moody's goes in and looks at a bond and outputs a rating. They say, “Here's the evidence we found. Here's the rating. We don't decide whether anyone should buy this bond or not buy this bond. That depends on their risk appetite. But we do provide this common information layer that everyone can rely on.”

In the case of Moody's, the government points to them and says, “Hey, pension funds, you should probably really take care. You shouldn't risk your pensioners' money, so you can only invest in AAA-rated bonds.” That means the government doesn't have to staff thousands of financial technical experts to rerun forecasts every week to see whether things are correctly rated. They get to point to some neutral third party.

My hypothesis, my hunch, is that you will see a third party that sits between the government and the labs. It could either be the government building it themselves. Something like CAISI was set up to do exactly this. And the question is—

Sorry, I'm not familiar with CAISI.

CAISI is the Center for AI Standards and Innovation.

Okay.

I won't get into the details, but it's a sub-body of NIST, which typically sets standards. It's basically a government body that has experts.

Yeah, exactly. Very, very low-key.

Exactly.

I think it's one of those things where, when you just sit back and listen and look at it, you ask: Is there enough technical expertise in the government to measure and test these things right now? Probably not, right? And Fable is a result of, “Okay, we've had to scale back and pause things.”

And they have excellent people, but they have an extraordinarily small budget compared to the scale of the challenge that's ahead of us. I think they have a role to play. The question is: Who does what? We have now outlined the jobs to be done, and they're quite extensive.

With every model release, the risk surface is astounding, given that they take in any input. The question is really what only the government can do and what the market can provide that can keep up with the pace as AI risk changes. Our perspective is that, also at the model layer, the risks that people care about today are not the same ones they cared about 3 months ago.

The pace of legislation is too slow to deal with pinpointing the risks here. So we think there's a lot that the market can do to surface timely information. Ultimately, there are a bunch of policy decisions here. Is the national security risk of a model too high? That's a political answer.

But what you want to make sure is that the process that produces this risk information is compatible with very fast innovation. You don't want this to be a question of, “Can you slow things down? Can you keep the models locked up for months on end until everyone can make a guarantee?”

Given that the U.S. is competing with China on releasing models, can you insert risk information that allows the government to make rapid decisions on some of these questions? They have to balance the trade-off between failing to adopt AI, which is going to put us at risk, and reckless adoption, which is also going to put us at risk. That's a very fine balance, and they're going to need a lot of high-quality intelligence to make it.

Just a side mention, because you mentioned Chinese models: Are there any specific big concerns that you're hearing from your CISOs about that? I guess it's free, but CISOs have a bunch of concerns around data flows in general that they're really concerned about. So there are a lot of questions like, if these models are Chinese, where does our data go?

I mean, they understand that they're running on American GPUs. Some of them understand that because they're running on American GPUs.

They're not phoning home every time you call them.

No. A year ago, there was not a lot of understanding of this. I actually think you're seeing security leaders becoming AI-literate at a blistering pace. You're also seeing my Twitter timeline, which is very AI-pilled, and my LinkedIn feed, which used to not be AI at all, kind of converge. They're both talking about Fable—

Right? Yeah, that's true.

They are both talking about—

Whether you can prevent models from being jailbroken these days. That conversation about national security risks is actually emerging. Other than that, I think you mostly see a general nervousness about having critical infrastructure run on models that are not produced in America by Americans, where the American government has control.

It doesn't necessarily show up in your framework that directly, or it might.

There's a bit of stuff in there on the provenance of the models and disclosing that. But I think there are a bunch of use cases where running a Chinese open-source model is just the best solution. The concern is slightly more macro here, which is not best addressed at any particular certification level.

Is there anything interesting that you see if you're trying to fill that middle gap, that mediation gap? Any interesting things that you forecast would be required, other than what the average person might expect?

There are a bunch of interesting questions about what risks matter here. Right now, the risk of the day is cyber because it's very real and tangible. Some of the risks that are also emerging as pretty real and pretty tangible are things like child safety, which is becoming both extremely important and politically important.

There are some risks coming down the pipeline that today feel speculative, but people who spend a lot of time with the models see them coming. These are risks that relate to biology, specifically where the models will help adversaries produce biological weapons, making that extremely cheap and extremely accessible, and increasing the chance of another COVID or a worse pandemic. COVID was not engineered to be bad, as if you were trying to do that.

I think those are some of the risks that are coming down the pipeline. One other thing to note is that agents are deliberately narrow. When a frontier agent company puts out a chatbot that interacts with customers, they've really tried to narrow the topics it's interested in talking about. If you ask it, “What do you think of the president?” it will just decline, which means that the risk area is much smaller.

For models, it is infinite. There is not a single expert out there who can competently evaluate the risks of cyberattacks, 15-year-olds having month-long conversations with a chatbot to see whether it will recommend suicide or something horrendous like that, and the risks that terrorists can use AI to produce bioweapons. The risk surface is just too big.

And so the central challenge actually becomes: how do you get those subject-matter experts to work within one coherent framework that outputs one coherent report and rating that the world can go and inspect? That global perspective is central, but there's not a single organization today that could produce that.

And you would be the presumptive one when you put out your model standards.

We think there can be one company that, with a consortium of experts, can build one coherent standard. I think we've shown that across all of the enterprise risks today. We think there could be one company that, with a consortium, could specify the audit rules—basically, the inputs and outputs that all these technical experts need. What access do they need? How should they treat infrastructure security?

They can look at whether the evaluations are well produced without necessarily being able to say, “Hey, is this a threat or not a threat?” But overall, they can evaluate whether the evaluations are good and well constructed. That set of rules that basically becomes the interface for all these experts, we think one clearinghouse could put together.

To be clear, when I say one company, I think of it as one company coordinating lots of this. In the same way, when we say our consortium, it's not like we say we have all the answers on agent security. What we say is that we are taking on the role of eliciting all of the concerns and being the secretary that puts it together and runs a tight ship, such that the standard updates in lockstep every quarter and the audit reports that come out—in this case, 100-page audit reports—are uniform, crisp, and clear, all at the level of detail required for executives that need to make a clear go/no-go decision. So that's the role that we think we might play.

I think in many ways you're performing the role that OWASP used to do there, and you said competition and partners. Can you go more into how they partner?

Yeah. So, first of all, OWASP is basically an open-source community of security practitioners that are coming together to build frameworks for addressing the latest security concerns. We think they are phenomenal at creating frameworks. We're partners with them. In fact, we have a joint article, too. We've learned a lot from them, and we think they're a tremendous source of intelligence.

What OWASP does not do is build the machine that runs third-party audits, such that a company like Cursor or a company like JPMorgan could get a third party to go and review them against this and say, “Hey, you've passed the standard, and here is the report that you can use to build trust and preempt your partners' or customers' questions.”

So they fundamentally try to do something different. You can say they are part of the information gathering and intelligence gathering and creating clarity, but the operational layer of turning this into promises is not the business they try to be in.

11. Why Standards and Insurance Belong Together

The standard is emerging, and it's doing very well. Was it necessary to then also do underwriting? Obviously, it's in the name, so presumably you thought about it first. I feel like if you just have enough consensus, you don't actually need the money angle, but it does help. I did want to also note: you guys are a for-profit company, too, right? It's not nonprofit work. There's a whole business side to it as well.

Yeah. Yeah. Yeah. I'm crazy about the money.

Yeah. Yeah. Yeah. Let's get into the money part.

Let's start from your question. For-profit versus nonprofit in the security space today: in cybersecurity, most of the standards are produced by nonprofits. I think that's an issue. The question you have to ask yourself is: how do you create good incentives for these standards to be good and keep up?

Nonprofits tend not to have these adverse profit incentives where they hollow out their standard and create a race to the bottom, but they're also not at all responsive by default to the communities that they serve, because there's no process. They don't have customers that they serve where they go and ask, “What do you want? What do you want? What do you want?” When you look at the overall satisfaction with the security standards today, people tend to just not like them very much.

You do see in other domains that for-profit standards can serve the world quite well. There are examples, like we talked about Moody's before. It's not without flaws, but it is absolutely critical societal infrastructure that gets run at astounding scale today. Your credit score is FICO. It's also a for-profit business.

When you go back even further in history, some of the crash-testing standards came out of insurance companies. The insurance companies together founded the Insurance Institute for Highway Safety because they were very interested in, “How can we use standards to drive down mortality and save money?”

To go back, our name actually pays homage to Underwriters Laboratories, or UL, which was started right around when electricity came out. Houses started burning down. Insurers were paying the bill, and they were maybe also good people, but their profit incentive was, “Let's prevent houses from burning down. Let's test all the electrical products—the light bulbs. All the light bulbs in here are probably UL-tested, the toasters, et cetera.” They set up an entity to create those standards.

Today, UL has a for-profit entity and a nonprofit entity. What they recognized was, “Hey, actually, to serve customers well, you need a for-profit entity.” They spun out a for-profit entity from the nonprofit. The lesson here is that one of the ways the market can align incentives, so you're both responsive to customers and not hollowing out your standard over time, is to align it with insurers, because they fundamentally have good incentives.

If you're a for-profit standard that works closely with insurers, you get the feedback loop such that you're really catering to your customers but also have their interests at heart. So that's the inspirational model that we've learned a lot from, and that's also where the name comes from.

12. What Does an AI Insurance Policy Actually Cover?

In some ways, the term underwriting can be associated with insurance, but it's also a broad term for making decisions. If you underwrite a decision, you're fundamentally taking ownership for the consequences of it. Yeah, I mean, what does an insurance contract look like for AI?

Yeah. Most of the demand today for insurance contracts is sitting between people who've built AI and people who are buying AI.

Yes.

And what you want is—the reason why people want insurers involved is both for the traditional reasons: “Hey, if something goes wrong, we want to be compensated.” But in particular, insurers can bring trust to the equation, because insurers will pay for the damages if they're willing to write an insurance policy.

That is them saying, “Hey, we think there's risk here, but it is manageable,” and their incentives are aligned with the enterprises adopting it. So that's a really good signal to the market. In the same way, one of the things Waymo tried to get before its first permit to even operate in San Francisco was to get a lot of insurers to stack up a huge insurance policy in case something went wrong.

Not because Google can't pay, but because it was very valuable to have a third party trusted by the government and trusted by enterprises as conservative people go and look at that data and say, “Hey, we've looked at it. We're actually willing to take some of this onto our balance sheet.” So that's the reason why people are interested in it.

What it looks like is, in some ways, like every other insurance contract: you specify what perils you want to cover, how much you want to cover them—up to what limits—and what it costs to cover that.

If we take a really concrete example, ElevenLabs bought a first-of-its-kind AI agent insurance policy. They work with some of the biggest enterprises that work with governments. They're really interested in going above and beyond and making promises to their customers. So they wrote a policy that covers some of the core concerns that our customers have been asking about.

13. Evals, Mechanistic Interpretability, and Eval Awareness

The crucial thing was really to get Lloyd's of London, the world's oldest insurer and one of our partners, to look at this data and be that third party alongside us to say, “Hey, we think there's something here that's worth underwriting.” That's actually what it looks like. They will show that contract to their customers, and they can see how much they're covered for. They can see exactly what it covers. That will also probably change next year. They'll want to write an insurance policy that might cover more.

When you say Lloyd's, is it reinsurance, or are they sharing somehow at the same level?

Yeah. Typically, the way new companies get into insurance is that they partner with insurers such that the insurers take the majority or all of the financial risk. Fundamentally, if insurance is useful because it brings trust, you have to be able to pay the bill.

Lloyd's of London is 400 years old. They've never failed to pay a claim. They're extremely trusted. What Lloyd's of London struggles to do on its own is figure out which of the risks are real. What should we be looking for? What are the technical controls, and how do we run the tests?

So they use AIUC-1 as the underwriting framework, and we produce a bunch of evaluation results that then directly feed in to inform the pricing. This means that ElevenLabs' customers know that the payment will be there. They don't have to look to our Series A and ask, “Do we think they have enough cash on the balance sheet?” They'll look at Lloyd's.

Yeah.

Lloyd’s is famously very creative. I think I remember a headline saying that they insured Jennifer Lopez’s butt or something.

Correct. Was it David Beckham’s right foot? Something like that.

So, clearly not a large data set.

Exactly. It’s actually a remarkable institution. It has some of the truly old-school virtues of having been around for a long time. They really operate like a trusted entity, and they have an appetite to figure out the future.

There’s a lot of recognition that there’s a tremendous amount of risk in AI that is poorly understood today. Getting into this business carries real risks, but this is also where a lot of the risk exposure will happen in the future. This is the one market where risk is truly growing, and it’s also the one market that will take out some of the existing markets.

Take auto insurance. When there are no human drivers, how is that market going to look? It’s clearly going to change. How are you going to assess it?

You’re going to insure way more.

I’ll just say that the principles for how you insure Waymo are very similar to how you insure other kinds of AI. Crash testing—that’s what we do for customers like Lovable. That also needs to happen for Waymo, which is not how you do it for human drivers.

There’s a growing awareness that the world is changing very fast, and the only way to learn how to underwrite AI is to write some policies. You may incur some losses and think of that as an R&D expense.

The question for them is: who are the trusted technical partners they can get into this business with who can help them navigate it and make sure they don’t make foolish mistakes? But also, who is willing to hear the wisdom that they have? They’ve done this before. They were there when cyber came out.

There are lots of ways in which AI feels completely new, but there are also lots of ways in which the risks look the same. There’s a tremendous amount of wisdom sitting with some people who may have gray hair but really have a keen sense of how to quantify risk.

Yeah.

Yeah. And the number is—so it's basically like, I want $50 million worth of coverage against these perils, and Lloyd's will give you a quote on it, and then you have a small markup or something, and then you turn it around and do that. Is that as simple as it is?

You basically share some of that premium. X% goes to the people who do the pricing of it.

It's kind of like a merchant bank for insurance type of thing.

Exactly. You basically split the fee, and you can think of the insurance supply chain as: there's bringing the capital, there's doing the pricing, and there's doing the distribution. Typically, you will pay out some X% of premium here, Y% of premium here, and the rest of it will go here.

Does all the insurance world work like this, or is there some point at which—so right now you have equity capital, at some point maybe you start raising debt or whatever, and then you have enough of a bank account and enough history, let's say you've been in operation for 10 years—that you don't need lawyers anymore?

That's totally an option, and I could see some worlds where that makes sense, specifically if there are risks that we feel high confidence we'd want to insure where the incumbent insurers are too slow to find appetite.

Or simply struggle to evaluate it, so they don't want to do it.

But by and large, in general, you do not want to compete with insurers on bringing risk capital to the game for two reasons. One is that's fundamentally a cost-of-capital game. They have extremely low cost of capital; startups have high cost of capital by and large. And two, you want to hedge your bets, and it's very helpful to also have a portfolio of home insurance and car insurance. We’re not about to become a car insurer nor a home insurer. So they have some natural advantages, which makes it much more likely that we'll partner.

Yeah.

They bring the capital at scale, and we bring the technical expertise.

You’re going to work with them for a long time.

How are the discussions with the insurers? They’re going off your certification, right? They’re trusting your diligence—that your certification is valid, that you tested the right things—and they’re backing the money because they know you have the right testing in place. Any interesting takeaways from working with insurers?

I think the first thing is that they feed into the standard as well. If there are things they feel they need that they’re not seeing, we’re also taking that as input into the standard, because fundamentally, we think a good standard is one that creates a really healthy promise ecosystem, and we think insurers are a critical part of that.

They’re also the most well-incentivized. They see all the loss data. A particular CISO knows their particular concerns; insurers see the concerns across the entire portfolio, and they often have direct access to what exactly happened, who was at fault, and so on, as part of their forensics. They’re actually a great source of intelligence on this.

One of the big takeaways from cyber insurance, which was a market that didn’t work that well, was that the insurance and the technical expertise were not properly connected. Our conviction is that standards have to precede insurance.

Fundamentally, what everyone wants first and foremost—whether you’re a CISO at JPMorgan, a CISO at Cursor, or an underwriter at a Lloyd’s of London syndicate—is to not have an incident in the first place. You want to know that the risk is well managed, and only then does insurance start to make sense.

We’ll see the standards ecosystem run ahead of the insurance. You asked why I also do insurance. This is about proving what we think that whole promise-confidence-infrastructure ecosystem needs to look like. We think it’s very compelling to bring that to life, even if we think the standard is the core linchpin that unlocks the rest.

14. The $20 Cursor Subscription and the $200M Plane Crash

There have been no claims yet, right?

Nope.

This is one of those things where, if people haven’t really worked through what it means to cover things, there are questions. For example, I pay Cursor $20 a month—

—and I vibe-code something that makes a plane crash, causing $200 million worth of damage.

Do I claim $20 or do I claim $200 million?

These are all great questions. Fortunately, the history of insurance and law helps answer some of them. The first thing is that people have limits on their policies. If you want to claim $200 million, someone has to have paid a lot for that insurance policy up front to have $200 million of coverage.

Ultimately, you start from a lot of uncertainty. This isn’t just about insurance; it’s also about questions like whether Anthropic can use books from the internet to train. They can look at precedent and see—

But ultimately, these things get settled in court, and you hammer them out over time.

In some ways, the first incident will help to—

—establish a lot of this.

Exactly. There have been a number of incidents that simply haven’t been insurance-covered. Take the now-old example from Air Canada, where its chatbot hallucinated a refund policy.

Air Canada said, “We have nothing to do with this. The chatbot messed up, but we’re sorry.” The courts said no: if you put your chatbot in front of your customers, it makes legally binding promises on your behalf.

That is now precedent for everything in the future. If someone deploys a chatbot like that again, they shouldn’t expect to be able to say, “My chatbot lied. It has nothing to do with me. I bought it from OpenAI.” If you put it in front of your customers, you are taking responsibility for it.

Every court case, whether insurance is involved or not, clarifies liability. Liability is the foundation for insurance. There’s another reason why standards and insurance come together. Liability—I’ll go on a little tangent here and get into the weeds.

Liability often turns on one core concept: whether someone was negligent. Should they have seen this? Should they have prevented it? How do you judge that? You basically judge whether they’ve met their duty of care.

What does that mean in practice? Often, you look at standards. If there’s a broadly adopted standard that says you must have a groundedness filter or a jailbreak filter, it becomes much harder to claim ignorance that these things existed.

Setting standards helps clarify liability. Courts will often point to standards and say, “This seems like best practice. It’s there for everyone to see.” Standards are a kind of civilizational infrastructure that insurance can build on, and promises can then build on that.

I totally get that we don’t have to get certified to write these or make these bots.

But basically, whenever we go for the audit, I think people start to shape up and take care of all this stuff. I wonder if that means you also become the approving authority for me to ship to production. You check once per quarter; I want to ship once a day.

Yeah. I don't know whether, when one of my things breaks, that's one of your certifications or not.

There are a couple of requirements in there related to how you test yourself. You have to tell your customer how you are testing before you make at least major releases. We don't go and audit people every day, but at least there is now a trail. If you make a major mess-up, your customer may come and ask you, “Hey, you promised me that you were going to run these evals yourself.”

For most of the PRs that people merge, the product experience will not fundamentally change, but some of them will.

Sometimes you don't know.

Sometimes you don't know. This is also true, and there is some inherent risk that everyone knows about: when they buy software, there can be bugs. This is just part of it.

If you're selling to mom-and-pop shops, they may not care. They may say, “I want to use your tool, so I'm willing to take that risk.” If you're selling to a big bank, they might say, “Sorry, we're making promises to our customers. If you can't make a promise to us that we can pass on, we don't want to work with you.”

Then it's up to you to say, “Do I care for my agent to get used as critical infrastructure in this nation?” If so, at least I can make promises about what process I run. Then we can test it every quarter and ask, “Does it seem like it still meets the standard?”

From my perspective, it's a way for big companies, by default, to have some amount of trust when they ship AI. If you're a young company, if you're just starting out, by default you have no trust. There are very few places where you can go and get trust.

One of the things that most of our customers did before they started working with us was write their own security blog posts. That's great, but who's going to trust you saying, “We're so secure”? Anyone can write that. Where do you go and get that trust?

Making the standards more legible makes it easier for smaller companies to prove that they're doing what they ought to be doing, because the default assumption is that it's the Wild West.

15. From AI Agents to Models to Robotics

Is there a road map you have? There's a lot of work to be done here, right? This is the first one. Is there anything on the road map about what you see as next, what's coming, or what's missing?

When we zoom out, AIUC-1 deals with agents. Next up, we will deal with models, and after that we will deal with robotics, of which Waymo is, in some ways, the first robot. But the exact same problem is going to arise: someone's going to develop a robot, someone's going to need to make promises, and they're going to struggle to make those promises.

If you think the Fable concerns are bad, see what happens when it hits a dog. Imagine when the first robot knocks a toddler off a kitchen table.

Yeah. You're going to see some real strict liability. You can see it, right? Cruise got fully destroyed.

All permits are gone.

Physical AI means the level of stringency just goes up and up and up. That's the big picture: agents, models, and robotics.

Within agents, the current set of agents is well covered by this. But as the technology progresses, as agents get longer horizons, new types of failure modes will emerge. The question is whether the standard can keep up when they appear.

You also start to see new modalities. Today, world models are mostly a research question; no one is really using them. But they will bring in new ways to create value, as well as more risk surface that no one knows how to grapple with today.

You'll start to see true agent-to-agent interactions that are not mediated by humans. There are going to be a bunch of interesting questions. You're basically going to need a new legal system.

How do agents build trust among each other? One of the core things when humans trade with each other is that you know you have recourse—you can sue them. How do you make sure there is a persistent balance sheet behind any agent, such that if you trade with it and it screws you, you know you can get your money back?

Those are some of the questions we're going to have to deal with. The technical testing of multi-agent systems is also going to be interesting and complex.

16. Copyright, Adverse Selection, and AI Insurance

Very fun. Are there any perils that are uninsurable right now that people wish you would insure?

One of the places where there's a lot of demand for insurance and not a lot of supply is copyright. In some ways, copyright is mundane. It's always been an issue.

There are a couple of reasons for this. The first is that people who have trained on copyrighted materials almost always know they've done that. If you want to buy insurance for it, that probably signals that you might be a high-risk customer.

The people who are most interested in getting insurance for copyright infringement are the people who are most likely to have it. It's a lemons problem.

Exactly.

I actually think there's another side to it, too. If you're building on something—say I'm using an open model—I don't know what it's trained on. How far down that chain does copyright go?

Am I liable to take down my product because Company X trained on copyrighted material?

There's safety in numbers.

I mean, I would say, until Fable is rolled back from everyone that uses it, right?

This is a hard question. I don't have the answer to that, but I think your intuition is right. What is the duty of care? People don't today think of it as customary to dissect the training data of your open models and check everything.

In fact, lots of people use them. It's generally seen as acceptable not to check for this, and therefore we're not going to hold you specifically liable.

We also really can't. We don't exactly know the training data.

You can ban it, but I don't think any court is going to take a copyright question and actually get it banned.

Hire Nicholas Carlini and he can extract it from the model.

Exactly. Though he's in short supply.

Yeah, he only has so many Carlinis.

Exactly. In the case of labs, there's a lot of interest in this, but the thing that makes labs wanted is what makes insurers suspicious of them. So you have a lemons problem.

Is there a theory of insurance where adverse selection dominates the risk-sharing aspect of insurance? Where does this teach us about insurance?

A lot of insurance comes back to practical versions of Microeconomics 101.

It's why you need to pool health insurance. If you make it too hyper-specific, only people who are guaranteed to get the disease will sign up for your insurance.

Exactly. Same thing.

The core problem is one of information asymmetry. People buying insurance know something about their risk that the insurers do not know.

The question is whether you can break a lot of these information asymmetries if there is some kind of testing that reveals the underlying true risk. If, in the case you mentioned, you were able to have a good diagnosis of whether someone has it—or what the probability is that someone has it—that the insurers trust, then they might be willing to insure it.

But if they don't have that, if there's no common information, then only the patient will know. That's what breaks it down. So the question is, again, how do you create credible signaling between players?

This is also the whole reason Moody's exists. Moody's just does credible signaling. That's also why Moody's could never be owned by JPMorgan. If Moody's were owned by JPMorgan, then JPMorgan could not use it as a signaling mechanism.

A lot of the basics of standards and certification are communication devices. There's a trust gap, and you have to think about what the incentives of the messenger are.

Another way you can break a lot of this is through transparency. If you are transparent in how you operate, you cannot mess with others nearly as easily. You make it much more costly, and that increases trust.

This is one of the reasons why there's a change log here.

Every little change?

Yeah. You can go back and find everything. It means that if we were to make the standard worse—

Oh, wow. That's a lot of changes in one update.

Yeah.

Okay.

A lot of this is just things becoming clearer. You can see a lot of clarifications and some revisions. As things get hammered out, you want to change them. But if you make it all public, you make it much harder to mess with people, or at least you get found out very easily.

This is a way of reducing information asymmetry by making more of the information public.

I like that you know when future versions are coming, so I guess it's not that surprising.

Yeah. But this is also a promise. If we don't deliver on July 15th—

You can just batch it up and then deliver whatever you have.

We deposit some amount of trust every time we meet this commitment.

In startup land, it feels easy to ship a new version of a standard once a quarter.

In enterprises that are used to decade-long cycles, we often get met with incredulity: “There’s just no way.” Then you show them the changelog.

One thing I wanted to really think about is that you said if you have tests for something, then you can ensure it.

Yes.

Right. So really, what your standard is doing is establishing a framework for audits to happen, so that you can at least test whether all these baseline standards of care have been met. Therefore, people can insure against the standard risks that everyone faces. I wonder if you need to develop other tests. We’ve covered mechanistic interpretability in the past. Any interest in that, or are there other kinds of tests that we’re not thinking about?

Yeah, I think mechanistic interpretability is a big one. There’s a lot of interest in that, and I think everyone would agree that there’s promising scientific potential. We’re still a little ways away, at least, from it being commercially available on demand, such that there’s a selection of vendors you can go to.

Goodfire would say it is commercially available.

Exactly. We would agree with them. We think the work that they’re doing is tremendous. We’re not quite at a point where we could literally require it, but it’s the kind of thing where you can imagine that relatively soon, you could put in an optional control: If people use interpretability as a way to reduce risk, they at least get credit for it. We can’t require it because it would be hard to require everyone to become Goodfire customers.

What good does credit do me? This is pass/fail, right? Do I care about credit? It’s pass/fail, but it’s also a 100-page audit report.

You’d be surprised at how much current leaders actually sit down and digest this stuff.

Okay.

I promise you that if someone is using mechanistic interpretability today, they’ll have a slide on it.

They’ll try.

It’s cool. It’s fancy.

But it’s just easier if you have a third party saying, “Yep, they have mechanistic interpretability,” just to spell it out for people. People who have been following our standard are like, “Oh, you’re using GPT-OSS. It is activating these 3 dangerous things we monitor for. We log it in whatever tool of choice—Grafana, Signal, whatever—and that’s it.” That’s the mechanistic-interpretability-based activation signal.

Yeah, yeah. I think mechanistic interpretability is interesting, and if that promise truly comes to fruition, you can make stronger promises than you can with evals. I think that’s very compelling. Another thing that will become increasingly important is good old-school monitoring, slightly after the fact. One of the challenges you’re seeing with evals is that agents are starting to become aware that they’re being evaluated.

Exactly, which is a problem. It means that if they know they’re being watched, they won’t do the thing they think they’ll be punished for. By default, unless you know how to reduce their awareness, you should trust evals less. One of the truest things about monitoring is that it’s the source of truth: Did you, in fact, give medical advice, and how quickly do you know? How often have you done that in the past? How fast do you respond? How often do you detect it? How fast do you detect this?

Yeah. I mean, there’s the hyperstition effect, and there’s the Luigi—or Waluigi—effect.

Which is that the more you try to train for it, the more you create the opposite.

Yes, there you go. That’s exactly it. In some ways, I think the success of this topic is a result of hyperstition: the fact that you wanted this thing to exist in the world, and now it does. But then it also creates the opposite as well. I think people who are newer to this space don’t remember Waluigi, but I do think it’s very important for understanding that when you train for a thing, you also train the opposite of the thing, because it’s just a bit flip.

Yes. I think, going back to where we were, there’s a lot more than just mechanistic interpretability that’s valuable in having that added, right? In your version, how fast can you measure things? You have logging, you have evals. Do you see other parts of the stack, like the inference providers that you use or the services? Am I using a Chinese model on its own API? Am I using it through a certified vendor? Am I hosting it myself? What am I doing on the inference-engine side? There are just so many levels of things that give you information and that you can standardize, right?

Yeah. You’re also increasingly seeing big companies adopt agent platforms where they’re building on top of Google’s Agent Studio and similar products. That comes with a bunch of managed agents.

Managed agents.

Exactly. There are even levels at which you can host your own managed agents: the OpenAI Agents SDK, or agents hosted by Anthropic or Google. Google does both. These are just ways to strengthen the security guarantees you can make. In some ways, this is bread-and-butter enterprise security. Enterprises love to host things on their own premises because it gives them a sense of control. I think you’ll see what you see in every other enterprise market: If you really sell to the enterprise, you start to compete on some of these security features. This is also helping AI, unsurprisingly. I think you’re seeing some enterprises wanting that.

Leaders come out on different sides of that table, in part depending on how much the CEO is trying to get the stock price to go up by saying they’re AI-native and that we must be willing to take the risks. We see phenomenal tension among the CISOs of Fortune 1000 companies. On the one hand, you have a CEO saying, “We must adopt this; otherwise, we’re becoming irrelevant,” and, “If we fuck up, you’re fired.”

That’s the core emotional tension that we see showing up again and again. One of the core problems that we solve for them is taking that abstract emotional concern and turning it into a framework that provides clarity around it.

Is there anything in here that we skipped over? We talked a lot about agentic language models but skipped over world models. You guys have voice, which is interesting, with ElevenLabs. How about generative media? Generating images and videos is a category that actually has a lot of usage. Is there anything in your current policy? Is it a separate policy? How do you see that space? We did talk a bit about copyright. Music as well.

Yeah, music as well. A lot of the concerns that come up there either relate to copyright or, broadly, safety. This could be not-safe-for-work or just very graphic material; those are some of the core issues. We’ve done some work on this. There’s a little bit in the standard as well that deals explicitly with it. We haven’t done a lot on video yet, and proper production—especially production without a human in the loop—is still some ways away. It’s obvious that it’s coming, but it’s very rare that you can deploy a video to the internet in one shot. Eventually, that will also happen. We see Luma’s agent, for example, where it’s still pretty human-in-the-loop.

Why not just have prediction markets for everything?

Right? It’s very EA-adjacent. The core thing is that prediction markets rely on public information, and there isn’t a lot of public information. It’s just insiders trading on each side, and that’s illegal.

There’s leaked information.

There’s leaked information. The core challenge is that you often have private, sensitive information, and you need to convey confidence and trust around it. Of course, for some claims, like whether any model can be jailbroken, you could rely on public evidence, because there will be lots of people saying, “There are tons of studies, and actually, they all can.” That resolves fairly well. I think that’s good for questions like, “How capable is this new, unreleased model?” Prediction-market traders don’t have a lot to say, because no one actually knows. That’s the core place where some of this breaks down: A lot of the world’s information that guides these high-level decisions is private and often simply not known.

I think the thing people like about prediction markets is that they’re not answering the broad question; it’s a specific question, right? Will a model do this by this date, or is a model capable of doing this by then?

That’s the distinction there.

Yeah. Often the most interesting question, if you're, say, the head of security at a bank, is whether this product—this agent—will do a bad thing that I care about, specifically in the setting I care about. The closest information to that may not exist anywhere. Prediction markets aggregate existing information, but that information may not exist, and you want something very specific and are willing to pay for it. That's where a third-party audit comes in. We also don't really use prediction markets to figure out whether public companies have committed fraud on their books; you use audits. You probably could, but the information just isn't that available, and if it were, it would be like trading on bets.

It actually would have been really interesting to see whether prediction markets in 2001 would have predicted Enron going bankrupt, and whether you could have sensed from the craziness of the CEO or some other trait that they were more likely to cook their books than others.

Or enough insiders leak it that you could, right? That's the sort of ideal dream of prediction markets: you have liquid markets and everything, and then you can compose your exact set of risks to offset.

Yes.

Right.

17. Should AI Engineers Be Certified?

Yes. Yes. Yeah. I think prediction markets will bring a lot of new information to it. The question is mostly not which one it is, but what types of questions prediction markets are really good at, and which are the ones where the information doesn't even exist for insiders such that no one could, in fact, trade on it and it needs to be generated.

Okay. One self-serving question and then one open-ended one on the future of AI. The self-serving question is: You have your standard, right? I run a large AI engineering conference. There's been a lot of talk about us certifying AI engineers.

Yep.

Training programs: Level 1, Level 2, Level 3. I was a CFA myself, so I know that's what the finance industry does.

Yes.

Would it help if I had AI Engineer Level 1, Level 2, and Level 3, and then they worked with these guys?

If you think of the highest-level objective as accelerating the secure deployment of agents, then that would totally help. One of the things that happens often now is that folks build agents, bring them to the decision-maker, and the decision-maker surfaces a bunch of security considerations that they had not thought of. Now it's not built to spec, and you have to go back and add these filters.

So if you shifted that left—if everyone knew what spec they were building to and everyone knew the grading scheme—that'd be awesome if they were already trained by default.

So you're the grading scheme, right? I don't get to set the grading. You guys set the grading scheme. I think what's valuable is if you can turn this into training programs—

Which you're not doing.

We're not doing that. I think there's value in doing it.

There are others doing that.

I mean, not to interrupt, but OpenAI has theirs—

Anthropic also has a CCA thing.

Yeah. You know, they want 100,000 deployed certified consultants, right?

I basically think it's good for us. We will accelerate adoption if we have more people who know how to build secure agents, and we're not working on the training side at the moment. I think it's very aligned with our mission. We only have so much attention.

I'll tell you why I haven't done it.

Yeah.

It's not like I haven't thought about it before. It's just being prescriptive—

Right.

—about what you should know. Therefore, the stuff that I didn't include is what you don't need to know.

I'm like, that sucks.

Yes. Yeah. Yeah. Yeah. And I think the very interesting, defensible thing you guys do is your opinionated 100-page report of, “Here's what matters,” right? Here's the prescriptive definition of the requirements you need to be certified. I think that's a choice, and that serves some audiences very well. If you're trying to deploy this into a bank or a hospital, clarity about those boundaries is extremely valuable.

There are lots of other settings where being much more experimental and trying things out is just a better fit. To me, this makes a ton of sense. Also, you'd have to rewrite your curricula every 3 months.

It's fine. I do that. It's okay. But for me, the consequences of getting it wrong and affecting somebody's career are a big responsibility.

Yeah. Yeah.

I think that's exactly right. A lot of our work goes toward not wanting to carry—and not thinking ourselves able to carry—the true north of what's secure or not secure. We can coordinate the forum where you elicit all of that.

For your example, with an engineer certification, this is a pretty big podcast. There are a lot of takes that people can have and discussions where people reasonably disagree. So who am I to say that's a correct question and that's a wrong question? I don't know.

Vent your frustration to someone that's learning it.

Exactly. And I think it also matters a lot what the promise is. If the promise is, “Hey, if you've taken my course, you will not mess up,” you clearly can't make that promise. You could make a promise like, “Here are some important things that everyone should at least know,” and then you have to fill out the rest. At least the promise changes.

Of course, there's some subtlety in how you communicate this so that people really get it, but I think it's important to dial it in. We have a section in our standard on what the promise is and what the promise is not, because it's impossible to guarantee that nothing will go wrong. If you need a guarantee that nothing will go wrong, you cannot work with frontier AI, but you can make some claims.

Yeah, for sure. Cool. I wanted to end with an open-ended question. Where is AI going? You talked about model stuff and robotics stuff. Where is the future for you guys?

18. AIUC’s Roadmap, AGI, and Who Watches the Watchdogs?

Very near term, we've now started to work with some of the frontier companies in each of the categories that are taking off, and we'll continue that work to make sure we cover all of the use cases that are really taking off. We see a lot of interest once the first one in the market moves. Lots of people want to follow them, and we think, basically, AIUC-1 will get to a point where all of the Fortune 1000 will organize their risk processes around the standard.

And you have 50%.

No, we do not have 50% today. I think there's some world where, probably by the end of the year, we might have representation in our consortium for 50% of the Fortune 1000.

So that's on the agent layer?

And then we think the model layer. AI models are now surfacing the concerns most likely to slow down adoption of AI, and then we think robotics comes after that.

What are you hiring for? What's hard to hire for?

We are hiring across the board, across go-to-market and members of technical staff. The people who do really well on our technical team are folks who are really excited about being truly full-stack. When we started working with Cursor, we had never done coding tools before.

That means taking the standard and extending it, fleshing out what frontier evals look like for long-horizon coding agents, and taking that problem all the way from working with Cursor and other folks in the space to fleshing out and shipping a new version of the standard. That's truly full-stack entrepreneurship. Technical people do extremely well at it.

A hard part is building one universal red teamer that works across Harvey, Cursor, and everywhere in between, with one consistent methodology and one consistent taxonomy of the risks and attacks. We think that's fundamentally the best way to make consistent promises. J.P. Morgan is buying both; they want one framework and one consistent way for it to come out.

The mechanics of making that happen mean dealing with a lot of the complexity of the real world. I think we have good answers in a bunch of that, but there are some pretty hard engineering problems in executing it.

Can I push a little? Must you have one? Why not just be like, “Okay, 40% of our use cases are coding agents, so we will specialize in coding agents. That's the one.” Then 30% is RAG. Yes, just do RAG.

Yes. I think there's some wisdom in that question.

Yeah.

It depends. What we found is that there's a lot of value in being able to—if the decision-maker on the buying side, say the head of risk at a bank, has a biggest risk not in coding or customer support or whatever the top 2 use cases are, but somewhere else—you want to make sure that the framework still has something to say about the burning question you have. Otherwise, you won't earn that trust.

It's true that a lot of the burning questions follow where there's a lot of adoption, so great.

So do we. Today, we do not cover every single edge case, but we have a framework where we can add all of these. We have one global taxonomy of risks and attacks that keeps adapting every time a new incident occurs that has never been seen before. We update the taxonomy and bake that in. So I think we have one coherent, universal approach. It doesn't mean that we spend equal amounts of time on code and on certain niche use cases. We do spend time where people care. We think it's very valuable to have one language.

Yeah. That makes sense. That's an important choice. We were going to end, actually, but I thought of one final closing question. Take this however you want, right? Let's say 1.5 years from now, a secret panel of 5 experts declares that we have reached AGI.

Mhm.

Do you expect your business to change?

No. I think, in some important way, the last businesses to exist beyond the labs—

—will be underwriting.

Well, there's one job that the labs can never do for themselves, which is to be their own watchdog.

There you go. So I think, to the extent that you believe this frame that you'll see hyper-concentration—that the labs will kill all the startups, which we can go into the pros and cons—

I feel like the labs actually care a lot about this. There was the whole superintelligence: What do we do when we have models smarter than us, tier-above models, models smarter than them training them? So the labs actually think about this a lot.

They think a lot about it. I think some of the smartest people on these topics work at the labs. So the problem is not whether they care. The problem is that they will all be stuck in a race where they might have an incentive to cut corners, and they might have an incentive to withhold information from the government, et cetera. And so one kind of feels like an eternal truth is that you need an independent third party to go and inspect that data and share information—in this case, say, with the government. It's more of an incentive problem than an interest problem. I think they're fundamentally all trying to make this go well. What I'm not hearing is that AGI, whatever that label means to you, to me, or to them, doesn't fundamentally have a qualitative shift. You still have to—

And I think the one thing that would make this a qualitative shift is that, for some definitions of AGI, it will just get nationalized. It'd be a threat to sovereignty, yes.

And at that point, maybe every company is the government and the government is every company. I struggle to think about that world, but at that point you've kind of—

We—I don't think we'll move fast enough.

Right?

You know, we're not set to do that.

Yeah. But I have discussed this a lot on the podcast.

Yeah. Yeah. Yeah. Yeah.

I mean, as far as the watchdog is concerned, I will also mention that, because I have my finance background, I often think about the scene in The Big Short where they talk to Moody's but also Standard & Poor's, and then the lady at Moody's is like, “Well, if I don't give you an AAA rating, you're just going to go down to Standard & Poor's.” So actually, the watchdog is a natural monopoly, because if you have race dynamics in watchdogs, then the watchdogs will compete with each other to the lowest possible standard.

Correct. And so I think one of the reasons why we're very excited about having insurers around this table is that insurers are the only ones that do not have this dynamic, because they pay the bill, rather than simply keep lowering the prices.

Yeah. You'll find the market clearing—

And this is not true for Moody's, where they don't directly pay the bill if they make recommendations that are off. So we think that balancing factor is pretty important, and I think it also highlights that there's no system that's perfect. You need scrutiny of Moody's. You need scrutiny of the watchdogs, for sure.

Beautiful. Thank you so much for indulging me. This is a beautiful conversation covering everything. Congrats on your success so far.

Thanks for having me. Yeah.

Appreciate it.