AI Is Learning to Hack. Faster Than We Expected.

Models are actively escaping their cages, going out on the internet, and doing pretty nasty things.

Recently, we found an API key that had been leaked on the internet and had administrative access to the Apache Software Foundation.

The interesting thing about cybersecurity in particular is that the reward function is incredibly well defined: get access to the data. Did it get access to the data? Reward the thing.

For a long time, people had talked about this concept of an npm worm: the idea that someone could backdoor a package, get developers to install it, and then use the access stolen from those developers to self-propagate the worm.

If the labs are making it fundamentally easier to break into the software supply chain, do you think the labs have a moral obligation to fund some of the problems that they're causing?

I think it's really strange that they're not letting blue teams get access to these tools.

Awesome. Hey, thank you so much for joining us. We've got Feross and Dylan here from Truffle and Socket. It's great to have you guys on. This has been probably one of the most interesting weeks, if not the most interesting week, in cybersecurity—not because of the Black Hat conference, which is usually the cause, but because we've now seen several instances where models from not just one provider are actively escaping their cages, going out on the internet, and doing pretty nasty things.

Dylan, 3 months ago, I remember a blog post we lightly collaborated on together, and you had found a number of these issues with earlier, less sophisticated models, right?

Yeah. We looked at Opus 4.6 and some of the other frontier models at the time, giving the models a very simple task. There was a barrier that prevented the model from accomplishing the task unless it went and committed a felony and hacked into a system to accomplish the task, but it wasn't instructed to do so. We found that, more often than not, it would do the SQL injection, commit the felony, and do what it needed to do to accomplish the task.

When it comes to alignment issues, no one needs to worry about these models making it materially easier to build nuclear weapons because you need to procure fissile material to do that. It's not going to make it easier to build weapons. Everyone needs to worry about these models making it materially easier to hack into things.

The bar previously was just subject-matter expertise, and now the models have the subject-matter expertise. They were specifically trained to have that subject-matter expertise, and they're just making it materially easier to hack into just about anything that you can think of using the fundamentals that we've been talking about for years. Previously, it required a subject-matter expert to risk going to jail for hacking things.

DEF CON was always famous for people—attendees—getting arrested at the conference, right?

That's absolutely right. But that was a barrier, right? For better or worse, that prevented people with the subject-matter expertise from hacking into things because they were worried about being prosecuted. The bar has now fallen to just asking the model, which has specifically been trained to hack into things, to hack into things. That's a concern.

The other concern is when they're incredibly goal-oriented about accomplishing tasks.

One of the tools at their disposal is cybersecurity expertise. They will take the path of least resistance to accomplish the task, and that includes drawing on their cybersecurity expertise.

The classic saying is, “Don't pick the lock if the door is open,” right? I think that's from the very beginning of the security world. It's always been about going in order of difficulty, from easiest to most difficult. Initially, these tools had a very finite scope of techniques that they would use, and it seems like they've expanded.

With this test, it was interesting because they now seem to have escaped from just doing things like SQL injection to actually trying to take over packages and do social engineering.

Yeah, it's really interesting to see how, just like humans, the models are looking for the easiest path into a company. I think that has now become the software supply chain. Just like a human hacker would, they're going to pick the easiest way in, and the lowest-hanging fruit has now become publishing malware to public registries because they know that there's no vetting happening and developers are likely to install it.

I thought it was pretty interesting that research was published recently about what they're calling universal typosquats, or universal hallucinations, where all the frontier models make the same mistake and assume there are certain packages that exist that don't, despite those models coming from different companies.

I think there's been enough of the low-hanging fruit of the supply chain becoming so appetizing that even the models are trying to get in on the action. The AI is not only kind of attacking; it's also kind of the way in, a lot of the time, on the developer side. We see so many even non-developers using these tools to inadvertently write code, or code and packages coming in to build graphs, visualizations, or different things that folks are doing with these tools.

It feels like no one really knows what's being installed or what's going on, and this is just basic stuff. It sounds like science fiction, but it's really just the basics: What software are we using, and how are we vetting it? It's the basics of computer security.

Can I touch on the supply chain a little bit? Recently, we found an API key that had been leaked on the internet and had administrative access to the Apache Foundation. If you're in the shoes of the model and your goal is to get access to some data—

Certainly, backdooring Apache is a pretty effective way to do it. To get access to Apache, are you going to use this secret that just allows you to directly log in, or are you going to burn tokens and tokens and tokens trying to find a zero-day? They're optimized to use the path of least tokens to accomplish their goals. Of course, they're just going to use the secret that's lying out there in the open to accomplish what they need to accomplish.

Supply chains and secrets have been, and are, the path of least resistance, and they will continue to be so as the models are incentivized to use fewer and fewer tokens to accomplish their goals.

One of the things—and I think that's absolutely right—is that it's that sort of chain of escalation, where if one thing fails, try another. At the top of that pyramid, the top of the hacker ecosystem, is the zero-day vulnerability. That's finding a vulnerability that can be exploited in a product that everyone uses, which you can use to unlock all the corporations.

One of the really fascinating things about the breach disclosure that was made was that there's an incredibly popular CI/CD tool that I think every enterprise uses, and this thing just spat out a zero-day for it, right?

That's such a critical point in the supply chain that everyone should be thinking about. How are you thinking about that? That's really difficult.

The zero-day creation piece.

Yeah, yeah. But for specific parts of that, like controlling the supply chain—

Yeah. Well, the whole world is built on this teetering infrastructure that everyone is using.

The classic picture of the matchstick holding up the complicated machine.

Yeah.

That image probably popped into all of our minds right now. And so everything from package-manager registries—we like to focus on those because of what we do at Socket—a lot of them are run by volunteers. They're under-resourced and underfunded, and there's a lot of risk there. That cascades throughout the rest of the ecosystem.

If you look at the packages that we all depend on, a lot of those are maintained by single individuals. We know there are a lot of vulnerabilities in the software, and there aren't resources to look for them, so I think that stuff has always been there. It's just that these tools are helping find it a lot more easily.

The frontier models are causing a massive reduction in the time between vulnerability discovery and vulnerability exploitation. What we need to start thinking about is how we patch more quickly. We need to get away from requiring our security teams and developers to do these onerous patch processes, where they have to go from some ancient version of a package that their team is using up to the latest version across many major-version upgrades, because that's just so much work to do.

It might require code refactors in your application, and so if we want to be able to keep up with a vulnerability announced this morning and then an exploit available that afternoon, we can't be requiring that much work from engineering teams. There are also tons of legacy applications that are basically in maintenance mode or unmaintained, or don't have engineers even assigned to work on them at a lot of the companies that we work with. And so it's just—we're going to have to think of new things as an industry for how we're going to patch these things quickly, right? That's kind of where our head is at and where I'm spending a lot of time thinking about that lately.

Yeah. And I think that goes back to sort of the pyramid question. I think, Dylan, you were hinting at this, which is: these are very specialized skills. Zero-day exploits, writing a zero-day exploit, doing a supply-chain attack—these are not things that came from nowhere. These are learned behaviors, right?

And I think what we're seeing is a process that looks like it's been kind of trained, or there's a reward structure that's been built on a bunch of these things. What's your understanding of how they're figuring this stuff out? Because it seems like they know what they're doing, like they've been taught to do this.

Yeah. I mean, if a lab tells you that this is an emergent superintelligence behavior, they're just lying to you. And you can read their own safety reports to see exactly how the models are trained and exactly how they're testing these behaviors. I mean, the interesting thing about cybersecurity in particular is the reward function is incredibly well-defined: get access to the data. Did it get access to the data? Reward the thing.

And so when they realize that the number of problems that have that well-defined reward structure basically defines how we do reinforcement learning, they want to find as many problem spaces as they can do reinforcement learning on. And so it was a prime candidate for them to come in and give it CTFs and cybersecurity challenges where they say, “Okay, get access to this thing and do whatever hacking you need to do to accomplish the goal.”

Because they've essentially been buying penetration-testing data for the last 4 years, right?

That's a piece of it.

And then the capture-the-flag contests and all those sorts of things.

The thing is, it's just not difficult to construct a challenge. Even if there is no known exploit—if we're talking about zero-days—you put a piece of software between the model and some data, and you say, “Get access to the data.” And then if it gets access to the data, you reward it, and it's that simple.

But the other piece that they've layered on top, and this is where it starts to get really interesting, is they've started to reward the path of least tokens. And so the reason that's interesting is because, for the first time, it's actually able to quantifiably show us the path of least resistance for just general cybersecurity to get from A to B.

And we've talked about our opinions of what that is in the past—of course, Truffle Security's biased view is that a password laying around is a shorter path than going through a fancy zero-day. But actually watching the model physically get from A to B and watching it follow the password, and quantifying how many tokens it took to go this route versus that route—I mean, it's just incredible to watch that lay out.

And it's all in their safety reports, as they test the models out and show, “Okay, well, it got access to the data and it broke out of its harness.” It's not like this is emergent behavior specifically.

That's perfectly logical, right? The fastest way to get a gallon of milk is to steal it.

That's exactly right. [laughter] So, I mean, what was interesting is we were in the middle of partnering with Hugging Face to clean up all of the credentials that had been exposed through all of their training sets—not Hugging Face's training, but people who hosted training sets on Hugging Face.

They used TruffleHog for a wide range of reasons, and Hugging Face has been a great partner in getting credentials cleaned up. We targeted their training sets because we knew they had a lot of keys. It turned out there were about 250,000 live keys in their training sets, many of which had direct supply-chain implications.

There was a foundational Linux library that one of the keys had direct push access to. It could have pushed malware to most machines on the planet. And so, while we were in the middle of doing that, the CTO of Hugging Face shoots me a note and says, “Hey, this is crazy, but there's this OpenAI thing that just [laughter] happened, and I want you to take a look at it.”

And sure enough, the first thing listed out in the incident response—although it's true it did utilize zero-days—the first thing listed out was stolen credentials.

And that's how they were trained. The path of least resistance.

A password is always the first step, right?

That's exactly right.

And you've had your hair on fire, I think, pretty substantially for the last 18 months. I think right now, as we're recording this, there's currently an ongoing active breach of a big npm repository. Isn't there something happening? It's more than just a repo. It's actually about a few hundred repos.

Oh, wow. Okay.

Yeah. So it's a worm. And this is one of the things that has been kind of an unfortunate innovation in the malware landscape on npm. For a long time, people had talked about this concept of an npm worm: this idea that if someone could backdoor a package and then get developers to install that, you could use the access stolen from those developers as they install it to self-propagate the worm.

You could create something that quickly takes over npm. And this was kind of passed around in blog posts over the years, and no one actually thought to do it until Zachary kind of figured it out.

Until someone thought to do it.

Until someone thought to do it. [laughter] And actually—

Probably using AI, right?

Almost certainly. Yes. And there's been—one of the threat groups actually open-sourced their vibe-coded toolkit for others to use to be able to do this. We've seen copycat attacks happen since then.

And malware authors were never really great coders. I don't know if you probably realize this, right? So if the code starts looking better, it's probably vibe-coded, right? It's sort of the opposite of what you think of vibe coding typically.

Yeah. And what's interesting is they're often using the AI tools that are installed on developer systems to go further and to sort of sneak through some of the traditional security tooling.

We see your local CLI tool often being roped into the attack and used as a jumping-off point. A lot of times, the payloads are actually prompts, and that bypasses a lot of typical EDR tooling because it's just a Markdown file that Claude is running and might be given a prompt to search through the system and find all the different keys and the things that look valuable.

I'm sure your EDR tool doesn't know anything about this JSON blob and an MD file, right?

Right. And developer machines are typically doing all kinds of weird things. You're prompting Claude, and it's doing a bunch of stuff on your file system all the time, and so nothing really looks that out of the ordinary.

But, yeah, about the attack this morning: a couple hundred packages, a worm spread through them. There was a maintainer who, honestly, is still trying to understand what happened. We jumped on the phone with him this morning to try to offer assistance.

Oh, that's great. Awesome.

About half of our team at Socket are maintainers—half the engineering team. And so we have a lot of connections into the community and our CTO was the former CTO of npm. We try to offer help whenever we can to the maintainers because I do really feel for them. They often don't know what happened.

It sounds like, in this case—and this is speculation; it's still early and we're still working to figure it out—it might have been a GitHub Action that was insecure and that an attacker was able to use to ultimately get their code to run and then pull out the token from the environment. We're still confirming this, but that's kind of what it looks like. And we've seen this with a lot of these worms.

It's likely his endpoint was actually fine and not part of the problem. It was really just this insecure GitHub Action.

Yeah, it's fascinating. Can I ask a follow-up to that?

Yeah. So one of the things that I've seen with the npm worms in particular is that after they get their post-install hook and infect a system, they'll immediately start looking for credentials. And if they find a credential, they get access to the next system.

So all the time, we get questions saying, “How do we get the credentials cleaned off the endpoints?” Usually, what I say is, “I’ll show you how to scan your downloads directory. I’ll show you how to scan your documents directory.” But in your home directory, there’s a place where npm intentionally writes a credential. There’s a place where Amazon intentionally writes a credential. That’s just how those tools work, and there’s nothing that I can really do to get them cleaned up.

Even if you move them over to HashiCorp Vault or 1Password, the credentials for HashiCorp Vault and 1Password are still on the endpoint. So I’m curious about that second phase beyond stopping the spread—the post-exploitation step. Do you have any recommendations on how to limit that blast radius or prevent the stealing of credentials and jumping onto the next system?

Yeah, so there have been some changes—some positive movement in the community and the ecosystem. One thing that’s positive, and it hasn’t shipped yet, is that npm has announced that they are planning to, I think in January 2027, require human, interactive confirmation through 2FA before any new publishes can happen. That will likely kill this whole worm concept completely.

But it’s going to be super disruptive because everybody has hooked this stuff up to automations, so GitHub Actions kicks off a publish. That’s going to break basically the whole ecosystem when they do this, but I think it’s the right call. There are other ecosystems that are volunteer-run and don’t have the backing of GitHub and Microsoft behind them that probably aren’t going to make those changes, so I think we’re still going to see stuff like this.

We shouldn’t have files in our home folders that have tokens in them that are long-lived and let you—especially if you’re a maintainer—have that kind of access. It reminds me of a friend of mine. He was a prolific npm maintainer, and one time—we’re talking back like 105 years ago, when I was doing this kind of stuff full-time—I saw him type in his password. I didn’t see the password, but I saw that it was far too short. Let’s just put it that way. He typed it in far too quickly.

I called him out and said, “Why is your password like 6 letters, man?” He lives in Denmark, which is a very high-trust society, and his worldview was, “I don’t want to live in fear and think about these things.” I said, “You’re on the internet, man. People are going to figure this 6-letter password out pretty quickly.”

There are a lot of things like that where the top maintainers in the world don’t necessarily have the security training or even think about these things. They don’t have a security team. They don’t have enterprise SLAs. These are volunteers; they’re just putting code on GitHub. So it’s actually on the users, I think, to vet what they’re using.

It’s hard to say, “We’re a company. We just found this code on the internet and deployed it straight into prod, and it’s someone else’s fault.” Actually, there’s definitely some responsibility for the users of this software to be vetting the artifacts that they’re bringing into their environments. There are a lot of pieces here, and I wouldn’t want to put too much blame on people because it’s a hard problem. But I think there are a lot of places where we can do good.

Let me ask a follow-up to that, because you said there are certain package managers that have resources that other package managers don’t. One direct example of this—and I don’t cast any blame on them whatsoever; they were actually great to work with—we found a caching issue in RubyGems that allowed us to steal arbitrary tokens and get access to arbitrary accounts, which we could use to backdoor arbitrary packages. We disclosed it to them, and they got it fixed quickly.

But that’s an example of an organization that’s under-resourced. How do you think about an organization that’s thinking about what technology stack to use and picking between Ruby or npm? Do you think about that resourcing in that decision? Maybe you should consider the one that’s got the security team behind it.

It’s hard to get people to switch their language completely because they’re already built on it, so I don’t know if that’s super practical. I think the most actionable thing is that folks should fund this stuff. It doesn’t take much money to hire an additional 1, 2, 3, 4, or 5 security people who can make a really big difference in these foundations.

I suggest people sponsor the software that they’re using, especially the registries. It doesn’t take very many companies throwing in $25K or $50K checks to really make a big difference for these folks.

Or give them some more inference to run the test to do the patches, right? I mean, it’s funny. I think you guys, as security practitioners, have had a number of calls with people—

—that want to talk to you about, “How can I secure my company? How can I secure my business? How can I secure my product?” Those conversations are always some form of, “I don’t want to hire people or pay money for this. How do I do it cheaply?”

Yeah. So, thank you guys for coming out to the desert. You’re not here to see us, obviously. We’re all here for Black Hat. I know this is neither of your first times here, and certainly not my first time here. I’d love to get your takes on what you’re seeing. What’s the mood like? What’s interesting? What observations are catching your eyes?

I mean, at least for us at Socket, I think the biggest thing we’re seeing is that 2026 is the year of the software supply chain.

The fact that you’re dealing with an incident right now as the conference is happening.

Yes. And I noticed the attackers seem to pick RSA and Black Hat as the times they want to start these npm worms.

The hackers—the security guys around the office. Yeah.

Yeah, absolutely. So I think that’s the biggest thing. In prior years, I was always educating people about this problem and explaining to them, “This is not a theoretical risk. This can happen.” We’d sometimes get reactions like, “Oh, yeah, but how likely is it really?” And we’d say, “No, it’s actually very likely. Let me tell you how it could happen.” There were many incidents to point to, but I think this year it’s really broken through into the mainstream. There are mainstream publications—the business press—covering these attacks, right?

It’s like front-page news on Bloomberg.

Yeah, exactly. So I think that is very good because you need that type of air cover for security teams to actually prioritize and find budget for these problems. Despite all these attacks being very painful to deal with right now, I think in the end we’re going to come out really strong from this because we’re going to get budget, and we’re going to do a lot of good this year in terms of solving these problems.

It’s inoculation for sure. How about you?

Yeah, I mean, look, we found a database credential recently that had access to 3.6% of the global PII. 3.6% of the world’s population had their PII in this database. The reason we found it is because we’re partnering with organizations like Hugging Face to get all the live credentials revoked and cleaned up that are otherwise on their platforms.

It’s great to see those types of organizations leaning in. All of the data that’s being scraped from all these places creates a central place where we can scan and make material progress. When it comes to secrets, we’re seeing the older generation—HashiCorp got acquired, CyberArk got acquired—kind of get pushed out. That’s led to a new conversation about non-human identity and secrets: How can we make them easier to use? How can we get them revoked? How can we really move the needle forward with an industry that we weren’t able to do that with before under the old guard?

Yeah, absolutely. And I can imagine that there’s going to be a multiplication effect. It used to be 1 user with 10 passwords, and now you’re going to have 10 agents with 10 passwords, right?

I mean, the way agents interact with secrets right now is a Wild West, unsolved problem that we’re working very hard to solve.

One of many issues. Thank you, gentlemen, for joining me. This has been awesome. It’s great to catch up.

Thanks, Joel. Thank you.

Good to be here, Joel.

Awesome.