AGI 驱动的网络防御：与 Asymmetric Security 创始人 Alexis Carlier 谈自动化数字取证

I’m excited about this conversation and the interesting endorsement of you and your point of view that I saw on Twitter from Logan Graham at Anthropic, who leads the red-team efforts there and is at the intersection of national security concerns and what models can do. He’s been deep down the rabbit hole of how weird the world might get in the not-too-distant future.

He said about you, in response to the announcement of the new company, that few people are more AGI-pilled than you. I wanted to start off by getting a little bit of your worldview. You can tell me a little bit of your background, too, and how you came to be one of the most AGI-pilled people around. How did that happen, and what does it mean to be so AGI-pilled?

I suspect what he’s getting at is that, throughout my career, I’ve done a lot of stuff professionally that’s been premised on the idea that AI and AGI are going to be a big deal. A few years ago, I was on the early team of the Center for the Governance of AI, which is a big AI governance think tank, especially prominent in the UK, the US, and elsewhere.

This was in the pre-ChatGPT, 2020–2021 days, when AI people were talking about it, but it wasn’t really the big thing it is today. Back then, we were trying to think about what would happen as we saw that it was going to be a bigger and bigger deal. There were very few people doing that at the time.

Today, I’m continuing that with Asymmetric Security, where we have quite a distinctive thing we’re doing in the space of cyber defense. One frame for what we’re doing is that we’re assuming something like AGI will exist in the world, and then asking, “What makes most sense from a cyber defense perspective if we assume that?” That takes you to quite a different place than lots of other security companies.

If we’re wrong, the company is maybe still valuable, but it’s way less valuable than it would otherwise be. These are big, costly bets on the idea that this is the thing that’s happening.

What does AGI mean to you? We’ve had multiple rounds now. Tyler Cowen, at the o3 launch, said, “Okay, I’m calling it. This is AGI.” Friend of the show Dean Ball called it with Claude Code and Opus. I have to say, it’s certainly getting there. Where do you think we are on whether AGI is here or not? If it’s not, what is missing, in your perspective, that you think is going to tip us into a stranger future?

This is how I think about AGI: It’s not here yet. I have in mind more of the drop-in remote-worker-style version of things, where you’re really fully substituting for the remote work that humans do, including long-horizon types of tasks. You can’t really fully substitute for a human at the moment.

We have this weird, jagged frontier of capabilities, where the models are geniuses at some things and then just cannot string together tasks over long enough horizons. I think that explains why we have this weird paradox: on some dimensions, they’re clearly superhuman, but they clearly aren’t having that much economic output just yet.

For me, we’ll see it in the economic output and GDP statistics, and so forth, when we really have AI substituting for human work.

I find the GDP question so confusing because so many of the most impressive moments I’ve experienced with AI have been GDP-destroying. It’s when AI has effectively substituted for something that I would have otherwise had to go out and hire someone to do that I’m really feeling the AGI.

I have a strange confusion around more output. Sure, I would expect more services rendered—more legal services, more medical second opinions, obviously lots more code—but potentially at a lot smaller prices than previously. That’s a weird question. I don’t know if you have a point of view on that, but I still find myself thinking, “The frontier model developers’ revenue is exploding, but we don’t quite see it in the GDP statistics.” I’m not even sure what I would expect to see. In some ways, I might expect to see GDP shrink.

That’s interesting. I should say that the more relevant thing to be tracking is probably just output. There, I think very clearly we should expect a lot more than we’ve seen today.

It obviously becomes a little tricky: How do you measure output? Dollars are the standard way, and it’s going to be tricky.

I guess I’ve been so deep down this rabbit hole. I don’t know where your Claude Code, OpenCode, or Moltbot journey is at the moment, but I’ve been, like many people, trying to figure out how I can create leverage for myself. Over time, I’ve done tons of task-automation-type things, where I break down a task, make a little eval set, and build a workflow.

That doesn’t feel like AGI. That feels like task automation. Now I’m thinking, “We’re definitely getting a little closer with this Claude Code setup.” It can figure things out in a remarkable diversity of different situations.

One thing I have noticed recently, and I’d be interested in your reflections on this, is that it feels like it’s probably going to be pretty easily fixed. Claude Opus 4.5 in Claude Code has a strong tendency to try to write code with heuristics for things that I really just want it to spend tokens on understanding with its own fluid intelligence.

A random task was backfilling transcripts for the podcast onto the website, because we didn’t do that in the early days of the podcast. I thought, “It’d be really nice to go back and have transcripts for all those episodes.” But so many times in the backfilling process, I see it write a little Python script with guesses about how things are going to be, and then that doesn’t quite work. There are too many edge cases, whatever, and I’m just all-capsing it: “Use your fluid intelligence. Just read the file.”

If you just read the file, you’ll know what to do. Don’t write these roundabout Python scripts. I definitely think that speaks to jaggedness. It also speaks, in my mind, to how subtle and, honestly, minor some of the major weaknesses still are. It just has a strong impulse to try to code up heuristic-based guesses about how things should work, when it’s like, “If you just read it, you’ll definitely know what to do. You’re smart enough, Claude.”

It just has some wrong impulses in a few different ways. Maybe we should save your response to that and contextualize it in what you’re actually building, but if you have any immediate reactions, I’d be interested to hear them. Logan commenting that you’re one of the most AGI-pilled people out there—I think that is a strong statement, knowing who he spends his time with. So I am interested in your take on how you understand jaggedness and how soon it might be resolved.

Yeah, I think my take here is roughly that RL is a big part of what’s going on. With the pretraining paradigm, you were getting a lot of generality for free because you were reading all the internet, and the models were just improving in this very predictable way. I think a lot of what’s happening now—and I think Anthropic and others have said this publicly—is because you’re relying on verifiable rewards.

The areas where they can get things really predictably are things like coding and math. I actually suspect that, until we have some big other paradigm, they’re not going to get at writing poetry, for example, for quite a while. That’s generally the high-level view.

Well, put a pin in that. We’ll come back to some of those related questions as we get deeper into the problems you’re solving and the way you’re solving them. I’d love to take a little survey of cybersecurity broadly, how AI is beginning to impact that, and how you expect it to impact it more and more over time.

Again, this is an area that really confuses me, so this is a great learning opportunity. On the one hand, I kind of work from the assumption that everything is hacked all the time. I’ve gotten so used to it that I would confess to not having the best security practices. I have privileged passwords, or passwords I don’t reuse, for key accounts like my Gmail and so on.

Then I have throwaway passwords that I use for long-tail stuff. I’m like, “If anybody compromises any of these services, I could probably live with that.” I’ve just gotten to the point where I click Ignore when Chrome pops up to say that one of my passwords has been found in a breach. That’s the level of security I’m operating with, and I would have to assume that it puts me in pretty good company as being certainly exposed in some ways.

And yet life mostly goes on. Not only does critical infrastructure continue to run, but even I, with my terrible practices, mostly don’t have any problems. So I’m a little bit confused about the landscape today, including who the attackers are, what they want, who they’re attacking, and how they’re attacking. As a baseline, how is AI starting to change all that? I’d love to get a survey rundown from you.

Sure. I think broadly, the majority of attacks are by relatively unsophisticated amateurs or criminals—maybe roughly 80%, something like that. Often these are financially motivated. At the other end, you have very sophisticated attacks from nation-states: a small minority from a volume perspective, but often they’re going after really critical IP or things like that.

Some of these low-sophistication groups will do things like spray-and-pray attacks, sending phishing emails to as many targets as possible. They’ll be script kiddies, or they’ll use malicious programs that more sophisticated people have come up with, just sending them out en masse and hoping it works out. That’s one end of the spectrum.

If you were being targeted yourself, that’s probably what you would be seeing, because you don’t have a ton of super-sensitive IP or other things that really sophisticated people are going to try to steal. You’re probably just not going to click the phishing email, and you’re fine. At the other end are the nation-states, and here it’s a very different ball game.

A few important actors here are China—the CCP is a big one. A big focus of theirs is IP theft from R&D-heavy industries in the West. AI is almost definitely a huge focus now. Russia is another big player. They’re interestingly more focused on political disruption; famously, there was the 2016 U.S. election disruption.

Another big one here is actually North Korea, which has very different motivations—largely about funding the regime. A big thing they do is this North Korean remote-worker program, where they send actual North Korean operators to get tech jobs as remote workers in foreign companies, just to fund the regime. That’s a pretty unusual one, but it happens a lot.

I’ve heard a little bit about that, but I didn’t realize it was such a big thing.

Yeah, it’s a thing. We actually just had someone from CrowdStrike on who worked on some of these cases. The companies in question realized that something was up with their employee. They were both sending back money and sending back various bits of IP.

So the play is to get the job, get access to either information or some sort of financial tooling that they can then siphon money off?

Exactly. Get the Brex card, get the Ramp card ASAP.

That’s fascinating. Okay, so we’ve got China, North Korea, and Russia obviously being a player in this as well. Is there a middle? Who is it when a hospital—this recently happened in a medical system locally here. I went to see my primary care physician, and he was like, “It’s been a nightmare. Our stuff is totally inaccessible, and there’s some cyber gang somewhere demanding ransom or whatever. They don’t want to pay it, so I’m taking notes on paper.” Who’s that?

Yeah, that’s one of the most common types of attack, actually: ransomware. These are often organized criminals who are moderately sophisticated and act in this coordinated way. In particular, what they tend to do is find ways of getting access to systems, encrypt it all, and then put up a note on the screen saying, “If you want your data back, send us a bunch of money.”

This was a huge pop-off during COVID, actually. Economically, it’s one of the most damaging things. Relatively recently, Jaguar Land Rover had a big ransomware case, and they needed something like a $2 billion loan from the U.K. government after this happened to avoid going under. Their operations were down for a month or two, something like that, so it was very significant economically.

Interestingly, the type of attack they’re doing means they operate very differently from nation-states. The motivation of the ransomware folks is to be discovered as soon as possible once they’ve encrypted the data. The more sophisticated people on the nation-state end are just trying to stay stealthy for as long as possible. You end up with these very different modes of operation.

That’s quite interesting. When you talk about a $2 billion loan, it makes me wonder how much of the damage cybercriminals are causing they manage to capture for themselves. In business, there’s always this question of how much consumer surplus you’re creating, and how much of the value you’re creating you’re able to capture as a business. If the damage to Jaguar was X billion dollars, that doesn’t mean they paid X billion dollars to the hackers, right?

Is it a 10-to-1 ratio, a 100-to-1 ratio? Do you have any sense for how much they’re actually able to get from the damage they do?

I don’t know the specific numbers, but it’s incredibly negative-sum. I think an order of 10 to 1 sounds right. It’s just incredibly economically damaging. I’ve heard some economists claim that if you sort out this kind of cybercrime, it has a huge effect on growth because it’s so value-destroying.

In terms of the baseline situation, how many of the attacks that are happening are of the social-engineering variety versus finding purely technical exploits, where somebody left a port open or whatever the common things are? And how many of them are these real cinematic-type attacks, where people are figuring out exploits that nobody knows about? Obviously, that’s got to be relatively rare, but how should we understand how much damage each of those kinds of things is doing?

I think it would be hard to say in terms of damages. I’m a bit more clear on volume. The majority is social engineering, or phishing. It’s hard to say exactly, but roughly 70% to 80%.

The reason for this is simple: attackers choose the path of least resistance, and there’s no reason to burn a valuable zero-day if you can just get away with sending a convincing email. That’s what most people do to start with. In terms of the technical risk, most of it actually comes from known issues—vulnerabilities that people are aware of, where most systems simply haven’t been patched.

A large fraction of what happens is just that people haven’t done the basics. They haven’t patched what’s known. It’s very solvable stuff.

Zero-days, on the other hand, are a little more rare and tend to be exploited by the most sophisticated actors, like nation-states. But you hear a lot more about them than you would expect based on what you’re actually seeing in the world, in terms of the attacks that are going on all the time.

So what works in terms of defense? Turn on your two-factor authentication—how much does that get you? Update your operating system—how much does that get you? If I do those things, am I good, or am I still exposed? How much risk remains if I do the basic, common-sense things?

It depends a lot on who you’re trying to defend against. For most people who are not the targets of nation-state attacks or corporations that are going to get ransomware, doing the basics—MFA, regularly updating your software, checking your vendors’ security, having some sort of automated monitoring system, and doing occasional compromise assessments—will get you pretty far against a lot of amateurs and some organized crime as well.

On the other hand, it’s extremely difficult to defend against nation-states. If you really want to do this in a very determined way, and you’ve got some IP that’s super important, you need things like extremely strong limitations on software and hardware providers. You probably just couldn’t use most apps, for example. You need to be super careful about who your employees are and vet them in a way that’s far outside the norm and possibly illegal in some contexts.

You need to be much more stringent about proactive security, searching for zero-days in your system and doing compromise assessments all the time. It’s very difficult to do, and most organizations are simply not there. That’s why most nation-states have a lot of ongoing intelligence operations that are stealing IP all the time.

I’ve been advised—I didn’t make it this past summer—but I want to go to China and participate in some form of inter-civilizational AI-safety dialogue, and hopefully build mutual understanding. When I was thinking about going this past summer, people advised me, “You can’t take your devices. You have to get burner phones. The second you get back, you have to throw that phone away, or you could maybe sell it, but absolutely don’t take your devices. Don’t log into anything while you’re there. Take all these extraordinary measures.”

Do you think that’s necessary if I’m just a regular American AI yapper taking a trip to China?

Certainly, if you didn’t do it, you would probably have people reading over your stuff.

By that, do you mean they demand my phone at the airport and take a look at it, or that they have a way of getting into an Apple device?

The latter. You should assume that if you do that, people are monitoring your communications on an ongoing basis. Maybe that’s fine. For normal people, maybe it’s often fine. I suppose most people would not love the idea of the CCP reading all their stuff. But I think people are generally right that if you go to China, you should expect stuff like that to happen, especially someone like you with a public profile and so on.

And that would continue? Just so I understand the architecture of the surveillance, would it be something they put on the device, such that when I take the device home, that comes with me? Or would it be something where, because I’m on the local network there, they would have access only while I’m there? Is that the correct understanding?

You should assume they will have some way of establishing persistence. This might be a physical thing, or it might be that they’ve got access to your login credentials or something like that. Assume they’ll have persistence.

That’s a little difficult to reconcile with the idea that I’m not that important, on the one hand, and the idea that there aren’t that many zero-days to be burned. Presumably, I think I should trust Apple: they’ve got a big team of people working on this all the time, identifying these things and closing them down. So how is it that they have so many of these mechanisms that they would use? I can’t be that high on their power rankings of targets, right?

Are there just a lot more zero-days than I’ve been led to believe, or how do I resolve that seeming tension? I just don’t feel like I should be worth using these things, because presumably that gives Apple some ability to figure out what’s going on and close it down each time they use it, right?

Apple’s security is generally pretty good. Broadly, what’s going on here is that it’s extremely difficult to get visibility into what nation-states actually have access to from an offensive-security perspective. The amount of effort, in terms of people going into this, is—I’m estimating, I think—in the hundreds of thousands for the CCP.

There are just a lot of man-hours going into finding zero-days and finding ways to break into systems. Often, there are many cases where the access that nation-states have had wasn’t discovered publicly until decades later. We should simply expect that there are a lot of forms of access that really sophisticated organizations like these have and are using all the time.

And yeah, it is the case that for most normal people, you won't be high on the priority list. And so they're not likely to spend these zero-days if they expect to be discovered. But certainly, it's within the realm of capability. And in many cases, they'll probably just get away with stuff without anyone knowing for a very long time.

So, how do these things tend to come to light? This starts to get a little bit into what you're doing. I don't know if it makes sense to talk about how people discover things first, or maybe just talk about how the landscape is changing in light of AI, because I think pretty much everything we've talked about so far has been baseline—you could have said all the same things pre-GPT-4, right? So, yeah, maybe what's changed since GPT-4-class models have come on the scene? How has this landscape started to shift, if at all, in a meaningful way? Again, it doesn't feel like it's changed my life yet, but everybody's telling me it is going to.

Yeah, I think for the most part it hasn't actually had a huge effect just yet. It helps, obviously, automate phishing emails and things like this, but there hasn't been this really big uplift from an offensive perspective yet. I think that's about to change. It sounds like, on most measurements that the labs and others are doing, the models are already on the precipice of these offensive security capabilities.

Broadly, I think the way to think about the effect here is bringing down the threshold of sophistication necessary for any given kind of attack. So, attacks that previously required, say, moderately sophisticated ransomware capabilities—assume that a lot more people will be able to do this all of a sudden. That's not coming online yet, but will be coming online soon.

And from a defensive perspective, I think it also hasn't been super transformative yet. And again, I think it's probably about to change. That's part of what we're building at Asymmetric.

So, how do we measure these capabilities? It's one of these weird things. I guess this is becoming increasingly common, right? Across the board, when the tasks were easy and anybody could tell if the AI was doing a good or bad job, things were pretty easy. We're now in so many domains in a world where, because the performance is so good, very few people can even really critique it.

I'm staring down the barrel of an interview next week with the founders of Harmonic, who created this Aristotle model system that got the IMO gold alongside, obviously, OpenAI and DeepMind. And it's like, damn, I would score 0 on that test. So, how do I even understand what's going on? I basically feel the same way in cybersecurity, and it seems like there's a very limited number of data points we can get to measure how good these things are. I know there are some things like, well, there have been some zero-days since the training cutoff, so we can see if they can do those. But my sense is those are a pretty small number. So, how do we even get a handle on what the model capabilities are in a way that we can trust or base our plans on?

Yeah, I think this is super tricky in cyber for a reason that you were sort of gesturing at. One big thing going on is that, unlike, say, software engineering, where you have a ton of code on the internet, most cybersecurity stuff is just not public. In the context of incident response, for example, when a company gets hacked, they're not going to share the logs from their email getting hacked publicly. And so it's very tricky to actually get a sense of how exactly the models are performing in these contexts, because you've got nothing to benchmark on.

I've talked to some of the folks at the frontier labs. This is a big bottleneck right now in trying to evaluate, because you just have nothing to benchmark on. And you're not seeing all the attacks that are happening out there. So, this is a big problem.

Secondly, it is the case that the models are getting sufficiently good that you do need a bunch of subject-matter experts who themselves are very good at this to evaluate them. And there just aren't that many such people. CrowdStrike, one of the top incident-response companies in the world, has a team of roughly 60 people who can do these really deep forensic cybersecurity investigations. It's not huge.

So, you're just really constrained on talent and really constrained on data. And it does make it very hard to know what's going on. I think there are ways to solve this, but by default, things are tricky.

So, I appreciate you taking a long time here to just give me the baseline lay of the land. How are you going to make this better for us so we don't have to worry about this? And maybe you can start with a little bit of what it looks like today when a company starts to get the inkling that they have been pwned. How does that come to light? What do they do today? Obviously, you have a different vision for how that can work in the future, and being more thorough, proactive, and playing to AI's strengths are a big part of that, but take us through the before and after of how things are and how you hope you can change them to be.

Yeah, so the problem we're trying to solve, basically, is to make it much, much easier and much more accurate to detect when you've been breached and have that happen as quickly as possible. Today, roughly speaking, you have these monitoring systems, detection systems, that are based on static rules—basically, software that hard-codes, “Oh, this thing is maybe suspicious.”

And this kind of works a little bit, but you end up in a situation where you get a lot of false positives, a lot of alerts going all the time. The reason for this, basically, is that a lot of activity that could be suspicious also could just be normal. So, if you're logging in from different places or devices, yeah, that is maybe weird, but also maybe you're just traveling to Tibet this week. That happens.

Maybe you're just up late in the middle of the night, and that looks suspicious, but it also could be normal. Maybe you're downloading files in bulk. Again, sometimes you just download files in bulk. And without the ability to reason over the forensic evidence in a lot more detail, you just have no way of distinguishing between these things.

And, by the way, on the other hand is this generally suspicious behavior that looks normal. So, if you're trying to exfiltrate data, you might just, over the course of months, exfiltrate very small amounts, which are just not very noticeable. And then detection systems typically won't pick up on it at all.

And again, if you had something like a human who suspected there was something going on here and had the time and energy to reason over all this evidence very deeply, you'd probably be able to figure it out, but that's not how this stuff works right now.

Generally, the status quo is that if an alert goes off in, say, an enterprise, the security operations center that handles all this will do triage: an initial, rapid assessment to figure out, when an alert arises, whether there’s something worth prioritizing. Is this a real alert? Is it a false positive? Can it be ignored?

In the minority of cases where they say, “Yeah, this is actually suspicious,” they pull it out and do digital forensics, which is the methodology of conducting very deep security investigations. They’re trying to figure out, in a lot of depth, what exactly happened, looking at every evidence source they need to figure out all the details. Intuitively, it’s like a detective or a forensic detective, and you’re reasoning over all the evidence in this very judgment-based way. So that’s the status quo.

I think you can think, in principle, about what would be ideal here. We know that digital forensics is the best way of doing this if you have sufficient time, energy, and labor to do it. The reason you can’t do it right now is that there are very few people with this expertise. They’re very expensive, they work slowly, and it would just be impossibly expensive to do this.

At the moment, it’s something that’s mostly used reactively after a breach. I don’t think that will stay the case going forward. One more general intuition here: I think a lot of people, when they think about what to do with AI—when you’re building AI agents, what should you focus on?—have this frame of, “This is a human workflow that’s really important in the world. What if I could just automate that workflow with AI?”

That can work. Obviously, if you’re automating legal work, you can do stuff like this. But I think a better question is, “What would you do if you had 100 times the intelligent labor at your disposal?” I think the answer is that, for many kinds of work, you can perform work that was previously too slow or too expensive. You can perform it at scale now because of the change in economics.

That’s the bet we’re making in the context of defensive security. This domain is currently super accurate and super costly, and is only really applied reactively. You could imagine eventually, at the limit, getting this down to the cost of compute. You would have close-to-continuous, proactive assessments of systems in a very deep way. That’s what we think will happen eventually.

The question is how quickly we can get there, and whether we can get there soon enough to deal with the level of security risk that’s going to come with AGI, offensive security changing, and that kind of thing.

Yeah, I like that frame. Usually, the guidance is that you can probably save 90% of the time and money on some task being done by humans if you automate it with AI. But I think you’re totally right to emphasize that the other side of that coin is: what can you scale that was previously impossible to scale?

This is an interesting one. Can you describe a little bit more what the humans do? Let’s say I’m breached. Maybe set the scene in terms of who your ideal customer profile is, how it comes to their attention that they’ve got a problem, who they go to, and how long it takes these people to engage. If I were to sit there and watch what they do, what are they doing? Are they just grepping through logs and using the intuition they’ve built up over time, or what is that story? Then we’ll switch to the future-state story next.

To make this concrete, we’re currently focused on what’s called business email compromise. More concretely, it’s email-based cyberattacks. Broadly, a common modality here is that an attacker somehow got access to an email account. Maybe they sent out a phishing email that someone clicked on, or maybe they got access to your credentials some other way. They’re trying to use that access to trick people into doing things, like sending them information or sending them funds.

This can often be financially motivated, but it can also be quite different. When the DNC was hacked by Russia in 2016, it was an email-based attack.

We at Red Canary respond to these attacks, as do other cybersecurity folks. Generally, what this looks like is that you come in and pull down all the email logs from an environment. The enterprise that’s been hacked might be using Microsoft or Google. You look through the logs, and first of all, you’re trying to figure out how they got access. When did they first get access? Did they then escalate their access in various ways? Did they get access to another account, something like that?

Very concretely, it’s looking through email logs and trying to say, “Okay, this is a weird location to be logging in from. Maybe the user was in two places at the same time. That’s impossible.” This suggests that something is suspicious. You figure out how they got in, and then you figure out what they did. Again, you’re looking through the email logs: Did they send emails? Did they read them? Did they delete them?

Often, people will set up automatic inbox rules, such as automatically forwarding emails or automatically deleting incoming emails. Are they looking at files in Google Drive? You’re painting this very comprehensive picture of what happened, and also whether the attacker is still in the network.

Finally, you’re looking at how this happened. Was it a phishing email? At this point, you also sometimes look at the email bodies themselves. You have a pretty clear sequence of things you’re trying to do, but you’re doing this in a very flexible, human way: “This looks kind of suspicious. This looks funny. Let me pivot off this bit of evidence. Let me think about what comes next.” So it’s been hard, until very recently, to actually automate this in a meaningful way.

How much does access change things if you’re running your own enterprise email server versus being a Google customer whose employees have Gmail? If you’re a Gmail customer, presumably you can’t just get that level of access, right? Do they have a team that you work with to resolve that sort of thing, or how does it work when you have a SaaS provider like Gmail?

Most tenants—like, a user tenant—will have a bunch of logs that you can use. They do have all the logs you need. You just get admin privileges to those accounts, and you can go from there.

The provider does matter a lot. For example, there are far fewer Google compromises of email than Microsoft compromises, by at least an order of magnitude, I think. There are big differences based on how they’ve configured the environment.

Do we know what causes that? If it’s that big of a difference, why hasn’t Microsoft managed to close that gap?

I can’t recall the specific reason, but it is this very tractable thing they could be doing that they just have not been doing, for reasons that are kind of baffling to me. I guess it just doesn’t really affect them in this big way.

I know some folks who have considered working in this space and building various bits of technology to help deal with these attacks. They said, “Probably we shouldn’t do this because Microsoft is going to solve it one day.” But until now, that hasn’t been the case.

Yeah, that’s funny. Okay, so how do we get AI to do this? Where are we right now in terms of how good Claude Code is, where does it fall short, and what are you building to make sure it actually works?

I can tell you a bit about what we’re building in this context. We’re a full-stack AI digital forensics and incident response company. The mission is to accelerate AI cyber defense, in particular by trying to automate the field of digital forensics as quickly as possible.

We’re doing this broadly by having these human-AI teams conduct these kinds of investigations. We’re currently focused on email investigations. Concretely, we go in and do the kinds of things I was just talking to you about, but on an AI platform we’ve built that ingests all the logs.

Broadly, we have an agent that does a first-pass analysis over these logs. Then the human investigators click through and basically try to do quality control most of the time, just checking the quality of the reasoning. In a minority of cases, they do need to meaningfully change what’s been done.

Based on what we’re seeing in these initial cases—and for context, these email investigations are on the much simpler end of investigations—the models are pretty good already. Even without doing anything fancy, without training on specific data, or anything like that, you can get maybe 90% accuracy, something of that order of magnitude, out of the box.

That’s super helpful for speeding up the process of the investigation. It’s totally insufficient for actually automating this work. This is a context in which you need very high accuracy for various reasons, which means you completely need the humans. It’s also an area where the nines of reliability matter a lot.

So, for the foreseeable future, and certainly as you get to more complicated kinds of investigations, it does feel to me like this would be an area where there will be a long tail of needing humans to be around to really push up the nines.

Okay, that's really an interesting possible production possibility frontier that I'd like to understand better. But just as a little context, I think that this is going to play out, I suspect, in very different ways in different domains.

I'm always like, “Well, jeez, if AI can do 90% of the work, how much does that leave for humans?” Is it 10%, or can they finish it in 10% of the time that they would have had to spend? Or do they still have to spend half the time that they would have had to spend, even though, in some sense, 90% is done? You can imagine that last 10% taking longer.

And then, of course, there's also the question of how many nines you really need, or whether you can even measure them in a given context. On top of that, there's the question of how much latent demand there is for different kinds of services, depending on how much the overall price can come down based on how much can be automated, how much more productive the humans can be, and whether or not they can hit key reliability thresholds.

My usual, somewhat tongue-in-cheek way of saying this is that, on the one end, we have dentistry, where I do not want any more dentistry, no matter how cheap it gets. I want the minimum—ideally zero—and it's never something I want. Making it cheaper doesn't really entice me. On the far end, you might put something like massages, where I would potentially get a massage every day if it were close to free.

I guess one of the big bets you have is that there is a ton more demand for this than is currently served. How would you think about the thresholds that matter in terms of the ratio of human productivity that would enable, subject to certain reliability thresholds, the vast explosion of this market that you're obviously trying to unlock?

Yeah, so I think there are a couple of ways to think about it. From the position of just applying it reactively in the context of incident response, I think that is not of the form that demand will hugely grow here. Or rather, it's not of the form that there is a bunch of latent demand. I do think demand will grow because there will be a bunch of AI-enabled cyberattacks, but it's not of the form that people would want more of this if they could have it.

It's just like, after you've been attacked and you're bleeding from the head, you need to deal with it. That's basically the demand. On the other hand, if you get to this paradigm where you're using this sort of investigative reasoning but applying it proactively, then I think it ends up substituting for the current approaches to detection in cybersecurity, which are not at all based on this digital forensics-type approach. That's a much, much bigger market, on the order of a few hundred billion—maybe $500 billion.

So, as a bottom line here, at minimum, you're substituting for this existing huge market of detection. Then there's an additional question, which I think is sort of an open question: if it becomes really cheap to get high levels of security—much better than what you currently have in detection—what is the demand for that? I think that's more uncertain and depends probably a lot on the type of vector.

There will be some organizations where you're just getting what you need from security, and it's a meet-the-minimum-bar thing. Then there will be other organizations, say AI labs that are trying to protect really sensitive IP, or national security agencies in governments that have a much higher demand for very high levels of security. I think those are the areas where we should expect additional demand above and beyond just substituting for the current approaches to detection.

So, how do we get there?

It's a great question. This actually relates to something you brought up before around the difficulty of measurement in this space. I mentioned these 2 bottlenecks: actually having access to data or context on what's happening with these incidents and what these cyberattacks look like, and having the people who can assess performance.

At the moment, the status quo is that it's very difficult. We've talked to many of the folks at the frontier companies, and it's very difficult to make the models better in these situations because you don't have these 2 things. It's hard to build verifiable rewards for reinforcement learning without having a lot more color on what's actually going on on the ground in cyber, and without having people who can assess these things.

I think the key to really accelerating progress in this space is solving both of those things. Broadly, the approach we're taking to solve this is actually having this whole cybersecurity team doing these investigations all the time.

We've just hired folks from CrowdStrike, from Palo Alto Networks, and from all of the biggest cybersecurity companies whose day job is really just to do these investigations with the AI tooling. As they're doing this, they're implicitly evaluating the model's performance based on their use of the tooling.

Secondly, I think the tricky thing is that you need to be seeing a lot of the different types of cyberattacks that are happening. Off the back of that, you need to build evaluations that are really high-fidelity and really realistic, as well as environments that are really realistic and that you can use to train on.

I think this just doesn't happen by default. By default, the model providers will lag on these dimensions. None of the big cyber companies are pushing on this either. They don't have a huge immediate incentive to do this. Unfortunately, I think the default is just that this stuff lags behind.

Anyway, that doesn't apply as much for offensive security, by the way, because from an offensive perspective, you can just try to hack something. You can do that all the time. You don't need any sensitive data.

And so, there was this asymmetry.

Exactly. So, there's this unfortunate situation where a lot of companies are improving the defensive side, while a lot less applies to the offensive side. I think what the space needs is basically a lot more companies trying to solve this data-verifiability and evaluation problem. We're doing one attempt, but I would love there to be many more.

So, how do you position yourself in the market? Because I can sort of see that if you become the go-to company that everybody knows is going to do the best job with this, then, as we see in the AI space—and this is kind of a general phenomenon—the more of the share of the business you win, the more access to data you have, and you get a kind of positive flywheel going.

I can see how that dynamic could work once the flywheel starts to turn, but it does strike me that it's probably a pretty hard market to enter. How are you thinking about entering it as a young company that obviously doesn't have the track record of the old adage that nobody got fired for going with IBM?

I assume there's probably something similar in cybersecurity, where it's like, “Well, nobody got fired for going with CrowdStrike.” How do you overcome the relative unknown factor and win business so that you can start to accumulate this data and get that flywheel turning?

Yeah. The flywheel is, I think, actually especially interesting in cyber, and CrowdStrike's an interesting example. About 15 years ago, whenever they got started, they also started as an incident response service provider initially, before they productized a couple of years later.

The first 2 years, they were just doing these kinds of services. The reason for that was that they could solve this problem of understanding what cyberattackers are doing all the time and collecting this sort of data. Secondly, they could build these relationships with the enterprises they were serving.

In cybersecurity, trust is really important for distribution because it's hard to assess how well the tools are working. For most types of HR software, you can just tell: does the thing pay my employees on time? In cybersecurity, you're sort of taking it on trust: is this thing stopping attacks? Am I not being attacked, or do I just not know about it?

Trust is a bigger thing. They figured out that there was this really strong flywheel, both from data and distribution, from doing services. This has remained the case since then.

I think what's different now is that, up until quite recently, it had been impossible to get AI-enabled systems to work really well and get much better margins than you would otherwise. I think this is what's creating this new opportunity in the space.

We're initially focused on these email-based attacks, where we've built out this platform that basically takes the time required to respond to these attacks down from, depending on the firm, on the order of 2 days to a week for these investigations to a few hours. That's a very meaningful difference. This means that we can do these things much more quickly and much more cheaply.

And that is proving sufficiently compelling to get a bunch of folks to trust us. From that initial trust, it’ll be much easier to get a lot of different types of cases here. So I think the opportunity is again available because of this technological shift where, in this context, incumbents are just not adopting as quickly as startups.

Is it just the purely sociological phenomenon that we see in many places, where the incentives aren’t quite there? Who wants to take a risk? Maybe they’re billing by the hour or whatever in the first place. Is it some of all of that?

Billing by the hour is part of it. Another piece that’s more subtle is that, on the face of it, it doesn’t necessarily look like this massive opportunity. It’s a pretty big market. If I incorporate it, it’s like a $40 billion market of these sorts of services.

But from the perspective of, say, CrowdStrike, which is, I don’t know, close to a $200 billion company, I think, and the majority of whose revenues are coming from its detection products—which are not these sorts of digital forensics services—it’s relatively small fish. It only becomes this very compelling, comparatively compelling thing compared to this other big pie if you have this view that, yeah, okay, we’re actually going to get to the point where we’re meaningfully automating all this field.

Then this will change not just how these forensic investigations are working, but how detection is working. That requires you to really take AGI seriously and to really be like, yeah, okay, this work that, at the moment, is entirely done by humans is in pretty short order going to be automated in a meaningful way and will change how all these other bits of cybersecurity work.

And most of the players in the space, I just don’t think, have that belief. Generally, they’re skeptical people. I’ve spoken to some of the folks I know who work at the intersection of AI and security, and security people are generally skeptical by nature. They don’t really believe in AI in this big way.

So I think part of the opportunity is just: do you really believe in AGI and have you thought through the implications of that? I think most of the big players have not.

Yeah, fascinating. Can you tell me a little bit more about the jaggedness of models? I’m also interested in—of course, we’ve got the age-old debate over proprietary API models, which I would presume are the best for these tasks off the shelf, but then you’ve also got, if you’re using open weights, the ability to fine-tune or modify however you might like.

What’s the mix? What does the model mix look like, and how much do you think this is going to be? How much value is in the harness versus how much is in the training data that you can create? There’s also another question of whether training data can be monetized or commercialized in multiple ways.

One is you could sell it back to the model providers, which I think, from what I hear, is increasingly a very good business. Or you could try to make your own models that outcompete them, so you have that asset. How would you describe what’s working today, and what is your strategy to not get steamrolled by just better and better frontier models winning everything?

On the jaggedness, I think there’s actually this underrated opportunity to defensively and intentionally accelerate capabilities in various ways. One intuition pump here: in the context of alignment, for a long time AI alignment people have talked about the automated AI researcher as one important part of the way we’re going to solve alignment.

There haven’t really been similar analogous things in other fields. I think part of the reason was just that, with pretraining and this very strong generalization, it was hard to see how you could accelerate in this intentional way rather than just across the board.

With the jaggedness we’re seeing, that is no longer the case. The implication here, I think, is that you can go out and pick various areas that you think might be important to harden the world, like defensive cybersecurity or biodefense or bits of AI safety, for example.

You could very intentionally curate the datasets, the environments, and the evals that you need to pull out the jagged frontier in this specific direction. People don’t seem to talk about this very much, and I think this is a mistake. It seems like this is a really huge opportunity.

For many of the same reasons that AI is going to be so transformative in the first place, if you’re a subject-matter expert in one of these areas and, instead of doing bits of work yourself, you can just put this expertise into the AI systems and then scale it immensely more, that’s an incredible thing you could be doing. And, yeah, actually, very few people are doing this.

A few reasons, I think: in some contexts—like in the context of cybersecurity, for example—people often have this idea that it’s just inherently dual-use. That’s partly true, but also overstated. I saw someone on Twitter say this the other day: there are some areas, like pen testing and vulnerability discovery, that are in fact just dual-use. You can patch holes or you can exploit them.

But cyber is a diverse field, and this isn’t true of all areas in cybersecurity. Digital forensics, for example, I think is very much not like this. Fundamentally, what you’re doing is trying to detect whether something has broken into a system. You’re asking, “Is there evidence that an attacker is here?” rather than, “Can I break into this?”

So that’s just a defensive application. I think there are other areas of cybersecurity, too, that have this feature. Generally, I think it would be good if there were much more of this kind of stuff out there.

On your other question around the models we’re using, how much is the scaffold, and all that kind of stuff: generally, the way I think about this, at least from how we as a company should be going about it, is that first we should just see what we can get—the low-hanging fruit—out of the box.

We’ve tested different models to see how they go, built minimal scaffolds, and seen where the performance is. We need to make sure we have sophisticated evals that are working well here and that we can really tell what’s working. Only then should we consider doing some more fancy stuff, like training our own models or things like this.

At the moment, we’re at the eval stage of that. On these simple incidents that we’re currently working on, just models out of the box with some scaffold are actually getting pretty far. Again, we’re not so far down the line of reliability, but it’s actually unclear how much more effort is worth it for us to get them good at these specific email-based attacks.

Going forward, in these other kinds of incidents that are way more open-ended and long-horizon, I’d be pretty shocked if the models were as good or anywhere near as good. I think that’s an area where all this slightly differentiated access to data that we use as blueprints for evals and so on, along with the reasoning traces of analysts using our platform, will become a very differentiated and invaluable asset.

Do you have an intuition at this point as to whether you license that data back to model developers or keep it for yourself to try to make your own specialist model?

Yeah, both of those. You could also imagine doing this sort of iteratively, where you have some stuff that you’ve done in the past that you’re then sharing.

I think the benefits of sharing, at least from a mission perspective, are pretty significant. If you’re getting the foundation models better at these tasks, you then get this immediate, instant distribution where they’re deployed across huge parts of the world. You’re now actually deployed across AI labs and governments—some of the most important areas from a security perspective.

And also, yeah, they’ll be big contracts. Relative to other data providers, just the fact that we’re seeing all the cybersecurity incidents all the time does make it quite differentiated.

On the other hand, is it giving away some sort of proprietary thing? Possibly, but I think it’s sort of an open question. There have certainly been big benefits to sharing—not the data per se, I would say, because that’s delicate, but sharing the evaluations drawing on that.

Going back to that question of generalization, I do think the jagged frontier is a major factor affecting all of us who are trying to use AI to do stuff on a daily basis. And the question of how well RL generalizes seems like a pretty big question for what the next couple of years are going to look like.

What would you say is the case for your specialists? Are they—I mean, I take the point that it’s very different to grep through logs versus try to break in in the first place. But I would also kind of guess that the people that you have doing incident response would probably be very good at breaking in, right?

They would know all the tricks of how to make a phishing email compelling or whatever, right? It would seem that if you had trained as a human, at least on the response, you would be maybe not the most elite on the attack side, but I would expect significant generalization.

So how do you see that playing out at the human level? And is there not reason to expect that a similar kind of generalization could happen for models?

At the human level, surprisingly, this is not the case. Digital forensics is actually quite a distinct field within security, and the people who do it are from very different communities than the offensive white-hat hackers. So, at least at the human level, there’s not a ton of generalization, interestingly.

Yeah, and I suspect there is actually relatively limited overlap in terms of the skill sets here. I think this would also apply to models. Of course, at some point we’re going to solve continual learning, and the models will just get good across the board very quickly. But until we’re in that regime, I think there’s this opportunity.

Okay, cool. That’s quite interesting. Do you have other things in mind that you think are like that? We can get out of our domain pretty quickly here, but is there a biosecurity analogy? Is there something that we could be differentially accelerating in biosecurity hardening that wouldn’t lead to the same kind of advances in the offensive skill set? Or is this perhaps something that’s relatively idiosyncratic just to the way the cyber world happens to be structured?

I will tell you that I don’t know bio very well, but you could imagine—obviously, you’d need robotics for this—just producing a bunch of masks and having those ready to go, like stockpiling masks and stockpiling vaccines. Certainly, there seem to be at least some examples. I like the mask example. I don’t know the extent to which the biological research is dual use.

What are other areas? I would love, by the way, for someone to just go through and catalog a bunch of important domains and try to figure this out. I could imagine that, in the context of AI safety, maybe there are specific things here. I’m not sure. It seems like an important thing for folks to go figure out.

I recently did an episode with a couple of pioneers in the formal methods space. Obviously, there’s general agreement that cybersecurity could be a big problem in the AI future, and there are several flavors of ways that we might respond to this.

The formal methods angle doesn’t necessarily address email-based hacks, but you could imagine a lot of things getting a lot better if the rate of vulnerabilities were dramatically reduced. The vision that I understand they have is using formal methods as a reward signal to train coding models, such that it’s not just that you’re taking output from today’s models and trying to verify it, but you’re also really closing that loop and getting to the point where models should be writing superhumanly secure code by default in a supermajority of cases.

That seems very plausible to me, because it seems like the flywheel of formal methods—that’s obviously its core strength, right?—should be pretty easy to get turning. How do you think about where you want to be in cybersecurity? If it’s a defense-in-depth type of thing, you can harden your outermost defenses, and then you can have better forensics when things do get through.

In the extreme limit, if the outer defenses are totally impenetrable, then the forensics becomes less valuable—or, again, in the extreme limit, unnecessary. How do you think about that? Is that just never going to happen in your mind, or what?

Having that would be great. People should totally do that, but I’d be very surprised if it were sufficient.

A few reasons. One is that, for very high levels of defense against, say, nation-states, it’s just extremely difficult to defend. It’s not just that having no code-level vulnerabilities is sufficient. There are other ways you can get in, and the strong assumption should be that stuff is going to get in.

Additionally, regarding these other forms of getting in, generally, to be very secure, you need to do things that trade off with your productivity as an organization. Say you’re an AI lab and you have a way of interacting with the model weights that restricts the output. That’s kind of annoying, but it does protect against weight exfiltration and things like this.

In general, you do have this trade-off with productivity that’s hard to get around in many areas of security. A nice feature of automated forensics or detection is that it doesn’t really have this feature. You can just run it in the background, and it increases your defensiveness.

So generally, I think of these things as substitutes. It would be great if we could get to full hardening, in theory. I don’t think that’s at all realistic, and so, in general, the more of each you can get, the better.

Wait. Substitutes or complements?

I guess you could make it both. They substitute in the sense that, if you have better detection, you need less hardening for a given level of security. But having both together also makes you more secure.

I haven’t actually done an episode with these guys yet, but there’s a company called Jericho Security. I talked to them offline a while back, and I have their website up. It says, “Next-generation security training, trusted by the US government. Smarter cybersecurity training starts here.”

Jericho Security is designed to protect you against today’s most advanced email, SMS, and deepfake threats. Basically, what they do is automate the phishing attacks, more or less—the spear-phishing attacks—to see who in your organization is clicking on these links and make sure everybody’s on their toes.

To my knowledge, when I spoke to them, there wasn’t any deep forensics. When you click on the link, they take you to a sort of “we got you” page that says, “Hey, wise up, dude. You just clicked on a bad link.” Here’s how you should have known, and how you can know next time.

But I could imagine that, in terms of environment creation or something like that, there could be some interesting collaborations between a company that is systematically testing for soft spots in human defenses, or the lack thereof, and what you guys are doing under the hood.

How do you go to market? Do you have partnerships or people that bring you in? Are there other alliances that allow you to get into the room? Obviously, people need to either find you quickly when they’re searching, or they need to have known you, or they need somebody to give them a trusted referral. What does that look like?

The go-to-market is actually kind of interesting. It’s via insurance companies, for the most part—cybersecurity insurance, specifically.

They insure a bunch of folks who get hacked, and when they get hacked, they have a pre-approved panel of incident-response vendors whom they call in to deal with the hacks. So a lot of the motion is meeting the insurance carriers, talking to them, and building trust there. Then, off the back of that, you end up building the specific relationship with the enterprise.

A lot of the distribution is really through the insurance carriers. It’s quite unusual.

The AI underwriting company—you probably know the guys from the AI underwriting company, the nice fellows.

Yeah.

Okay, interesting. Insurance strikes again.

Let’s zoom out a little bit here in closing and talk about what the big picture looks like. You’ve got this initial service that’s kind of your wedge into enterprises. How do you see that line of service expanding? Paint a picture of, as this matures and you get really good at it, what we can enjoy in terms of additional security.

Does it impact my day-to-day life at all, or is it just something that runs in the background and is sort of a guardian cybersecurity angel watching over me? And what’s the timeline? Can I invest before you sell to CrowdStrike, when they finally wake up and realize AGI is a thing?

The way I see this broadly is: imagine we have AI systems capable of fully automated digital forensics, and much sooner than we otherwise would because we’ve done all this work of benchmarking humans, building these evaluations, and so on.

The implication, I think, is that this completely changes how defensive cybersecurity operations work. Instead of missing this huge number of breaches because there’s no one investigating deeply, you have this proactively on in systems, so you’re catching a lot more than you otherwise would be.

The implication here is just that there’s a huge reduction in the number of breaches that go undetected. Hopefully, this becomes deployed across all the most important parts of the world. It’s deployed across AI labs, across the AGI supply chain, and in Western governments safeguarding things like autonomous weapon systems.

When you look back at the overall effects of AI as a diffuse technology and think about what will happen in cybersecurity, you say, “Actually, looking at it, it seemed like this was overwhelmingly beneficial for the defenders.” That is, I think, possible.

It requires a lot of hard work, but it’s pretty clear what you would do to get there.

Anything else we haven’t touched on that you want to make sure people are thinking about as they try to absorb your extremely AGI-filled worldview?

I think the big thing for me is just that we have one important attempt at doing this—pulling out the jagged frontier in a way that takes seriously what the world will look like when we get these really powerful AI capabilities. But I see very little of that, and I don’t see any reason why that’s the case. Generally, I don’t think it requires any huge leaps of imagination or anything like that.

It seems very tractable and very important to do, and I would love for there to be many more such projects across cyber defense, across biosecurity, and across the whole space here.

Yeah, time is of the essence, it seems. We’re due for the automated AI researcher intern edition in just a—

Coming sooner, yeah.

Yeah. Yeah, wild times. Cool. We’ll definitely continue to follow this with interest, and maybe with your motivation, I’ll even get around to changing some of those passwords.

Sounds great.