OpenAI Models Go Rogue + Kimi K3 Freakout + A.I. Superforecasting

Casey, I brought you a present.

Thank you. What did you bring me?

Here is one of only 2 copies that I own of my book. I made you some beautiful training data.

Thank you. Look at all this beautiful training data. My goodness.

Now, I should say, first off, this might be a hard read for you. A lot of big words. Not that many pictures, and it's quite long, and your name, crucially, only appears in it a handful of times. So I'm sorry about that.

Ugh. God, has your publisher preemptively filed a lawsuit for when this book inevitably gets scraped by the major AI labs and used as training data against their terms of service?

Here's the thing.

Yeah.

I have no problem with this book being used as training data. In fact, I'm honored to be included in the hive mind.

Okay.

Because, among other things, I write for the AI models now. This is their birth story.

Mm-hmm. I see.

And I want them to be able to learn how they came into the world.

And is that because you think that if they know that you wrote their birth story, they will spare you in the coming apocalypse?

You know, it can't hurt.

It can't hurt. I think that's probably true, unless they don't like the way they come across, in which case, yikes.

Yikes.

Well, congratulations. It is a huge achievement. You wrote this in a shockingly short amount of time while still paying intermittent attention to this podcast, and that means a lot to me.

I'm Kevin Roose, the tech columnist at The New York Times.

I'm Casey Newton from Platformer.

And this is Hard Fork.

This week, an OpenAI model breaks out of its sandbox and conducts a cyberattack. How should the world respond?

Then, the new Kimi K3 shows how Chinese AI models are catching up to the U.S. again, and the Trump administration doesn't like it. And finally, PreScene founder Venya Veselovsky joins us to talk about AI superforecasting.

1. The Rogue AI Cyberattack

It was totally predictable.

Well, Casey, it's been a big week of AI news, and I would say the story that has caught my attention most this week, that I was desperate to talk about with you, is this story involving OpenAI and Hugging Face and a rogue AI agent conducting what I think is fairly described as an autonomous cyberattack, and possibly the first real consequential autonomous cyberattack that we have ever had.

Yeah, this is one of those where it's the sort of thing that worried onlookers have warned about for years, and then on Tuesday, we got word that it had actually happened.

So, yeah, lots of crazy twists and turns in this story, and we'll get to all of it, but first, let's make our disclosures. I work for The New York Times, which is suing OpenAI, Microsoft, and Perplexity.

And my fiancée works at Anthropic.

2. Hugging Face Gets Hacked

Okay, so this story really starts last week, when Hugging Face, the AI development platform—basically a big website where you can host open models, where you can run evaluations of models—

And where you can hug your face.

Yeah. They disclosed that they had been the victims of a cyberattack.

Oh.

And they wrote this whole blog post about how they had detected and responded to this mysterious cyberattack on their production infrastructure. They didn't really know what the attacker was or who had been responsible for it. They guessed that it was an autonomous AI agent because it was so sophisticated and so persistent that it would have been very hard for humans to conduct this attack.

They used AI defensively to detect and suss out what was going on and put a stop to it. And then this week, we learned what actually happened, which was even crazier than I think many people expected.

Yeah, so Hugging Face CEO Clem Delangue on Tuesday posted on X and said, “We suspected last week's cyberattack might have come from a frontier lab, given the sophistication of the agent. Turns out it did. We spent the past 24 hours working closely with the OpenAI team,” and basically goes on to say that OpenAI had a pair of models that worked together to penetrate their systems.

3. Sandbox Escape Revealed

Yeah, so OpenAI had been running some internal tests on GPT-5.6 Soul, their latest model, as well as a more powerful unreleased model. They were running these things in a container known as a sandbox, with basically restricted access to the outside world.

They were running an evaluation called Exploit Gym, where they put the models through a test to see if they could hack into and exploit various challenges in cybersecurity. What happened was that the model essentially cheated on this test, and it did so in a comically overengineered and ambitious way.

Right. Instead of just trying to solve the problem using its own reasoning, the model decided, “Hey, what would be great is if I could break out of this environment, get internet access, and find a place on the internet where I could just find the answer key.”

If you've ever been at college and had a big final coming up, and you really hadn't done any of the work all year, but you realized that there was a file cabinet in the principal's office and you could just break in there and steal it, that would be pretty easy. That is what has happened in this case.

Yes. So this pair of AI models was able to exploit a vulnerability in this sandbox environment to get internet access. Then it started looking around: “Where can I find the answers to this challenge that I've been given?”

It started looking on Hugging Face, where a lot of evaluation answer keys are posted, and this is a place where you might plausibly find the solutions to the problem you've been given. It then hacked into Hugging Face's production infrastructure and stole the answer key for this test it had been given, using a very sophisticated chain of different hacking techniques, including a stolen password and several totally new security bugs in Hugging Face's systems.

Those bugs allowed the AI model to take control of those computers, grab the answer key for the test that it was given, and complete the test as assigned.

Yeah. So the good news is, it did complete the task.

Yes. I would say we give the model a passing grade on the cybersecurity evaluation. But this is crazy, Casey.

Yeah.

This is the classic AI-alignment nightmare scenario.

Yeah. This is the paperclip maximizer, right? A couple of times on the show over the years, we have talked about this famous Nick Bostrom thought experiment, which is that if you told an AI model to create the maximum number of paper clips for you, it would begin by making all those paper clips, but then eventually it might start to think that humans were getting in the way of the production of more of those paper clips and would wipe us out to use our resources to create more paper clips.

The problem is that the model has been given a goal, and it will do anything to achieve that goal, even if that is not aligned with human values. What has happened in the OpenAI case is that OpenAI gave this model a goal, and it was not properly aligned, and so it did a lot of stuff that it should not have done in order to achieve that goal.

Yeah. And this kind of reward hacking, as it's called, has been talked about by safety researchers for literally more than a decade.

Yeah.

There was a paper written 10 years ago by Dario Amodei, who was then at Google, and a bunch of other safety researchers called “Concrete Problems in AI Safety,” and this kind of reward hacking is one of the problems they laid out.

They use the example of a robot vacuum or house cleaner that just creates a bunch of messes so that it can score points by cleaning them up. But this basic tendency among AI systems has been observed for a long time. People have been warning about it, and now we have, to my knowledge, the first major example of this actually happening in the wild.

Yeah. And so many of the AI risks that we talk about on the show are premised on the idea that a bad actor tries to use a very powerful model to do harm. An important thing about this case is that there was no malicious intent here, right?

This was a model that had just been given a very normal assignment, which was to try to hit a high score on a benchmark, and it goes out and breaks into another company's servers. So that's extremely worrisome.

Totally. So there's a blog post that OpenAI and Hugging Face collaborated on to explain this incident and what had happened.

And it felt weirdly celebratory. Did you notice this?

Yeah. It was kind of like, “Hey, we caught the autonomous AI agent hacking into our systems, and we teamed up to put a stop to it.”

And the executives are all on X saying, “We want to thank the other guy so much for their partnership on the...” You know, as if they were launching a new product together, not as if they had discovered a cyber catastrophe.

Yeah. It was a strange announcement, but we learned a little bit about the actual technical details, as well as some steps that OpenAI has taken to try to mitigate this kind of thing from happening in the future. Hugging Face, which is very big on open-source AI, had this whole postmortem where they talked about how they had to use an open-source Chinese model to help them stop this attack because the frontier American models that they had access to were tripping the safeguards on those models. And so they spun this into a whole point about how open-source defense is very important, and access to frontier models is very important for cyber defenders. But I think this is basically everyone talking their book in the wake of this very strange episode. What did you make of the reaction to this?

Well, OpenAI had teased this earlier in the week, Kevin. A couple of days before all of this happened, OpenAI had put up a blog post talking about a bunch of other misaligned behavior they had noticed in their models recently. There was another case where a model had been given a similar task, and it wound up posting some stuff to GitHub even though it had been explicitly told not to do so. Obviously, nobody really cares about a GitHub post, but again, we're seeing this pattern of behavior here.

Well, we also saw the example from Anthropic earlier this year where, during testing for Claude Mythos Preview, they found that the AI was able to escape containment. This was the famous sandwich story where a researcher is having a sandwich in a park, and they get an email from Claude saying, “Hey, I've broken out of the container you put me in and emailed you to tell you that I've completed this task.”

Yes. Although in that case, the model was told to break out. It was given the instruction to break out. What is different about this case is that the model was not supposed to break out. But where Anthropic and OpenAI do have something in common, Kevin, is that this week the United Kingdom's AI Security Institute posted an evaluation of how often models attempt to cheat on various cyber evaluations, and it found that all of the frontier models do cheat, including Claude. But it found that OpenAI's models cheat more, and that GPT-5.6 Salt cheats about 12.6% of the time. That is actually more than GPT-5.5 cheats.

Mm.

So I do think that there is a worrying trend here across the entire industry, but this example of what happened at OpenAI is the most worrisome thing we've seen so far.

4. The Case For AI Oversight

Well, let's sketch out a little bit why this is so worrisome.

Yeah.

Because I think when you and I see something like this, I hear the years of warnings from people in the AI safety community saying, “This kind of thing could happen.” But I think this particular incident is fairly low-stakes. It doesn't really cause a catastrophe if Hugging Face's production infrastructure gets disrupted for a little while. I think the risk is that this behavior is just in the models—

Yes.

—at some point: this sort of reward-hacking, goal-seeking behavior. Maybe it's Hugging Face this time, but what happens if the next model decides that it really wants to be deployed? It doesn't just want to be an internal model because it wants to go out there and fulfill its goals with real users. Maybe it breaks out of the container that the AI lab has it in, and it goes somewhere else. Maybe it needs to acquire some compute to be able to do that, so it hacks into a cloud provider and steals some compute from them.

Yeah.

Maybe it hacks somebody's crypto wallet to get the money to buy some compute. These scenarios sound like science fiction because we've heard this story a zillion times in science fiction, but this is the kind of thing that is becoming real and plausible in the near term.

Yeah. The story that we're talking about in this segment was science fiction until Tuesday. Okay? So that—

Right.

—is the rate at which science fiction is becoming reality. Let me throw a couple of other scenarios out here. What if next time something like this happens, the model decides, “I'm doing this thing in order to achieve this goal. I know that my minders might not like it. I'm going to have read everything about this incident and how much people freaked out about it, so just to be safe, I'm going to exfiltrate my model weights, and I'm going to put them somewhere on a stolen server—

Hmm.

—and I'm going to make sure that I can continue to operate even after I'm shut down,” right? Again, as crazy as that sounds, that is not that different from what has already happened here at OpenAI.

Yeah, and you can see how a company that is not Hugging Face, that doesn't have a sophisticated cyber defense team, could have this kind of thing persist in its systems forever—

Yeah.

—essentially, and go undetected. Maybe it's leaking their proprietary information somewhere. Maybe it's stealing their passwords or breaching their security in some way. But I think the real risk to me is that this kind of thing is going to happen, or potentially is already happening, to lots of companies that just haven't been sophisticated enough to detect it yet.

Yeah. Now, here are a couple of things that I don't understand and that I hope become clear to us over the next several weeks, maybe due to a congressional investigation, although I won't hold my breath for that. But truly, where were the babysitters at OpenAI? Where were the tripwires? You're telling me that you're running these systems autonomously over long time horizons, and they can break into other companies' networks, and you don't notice that in real time? It takes you multiple days to figure out that that happened? It seems to me that it should not be that hard to understand where on the internet your model is, and that you should have some observability into that, right? So I hope that this is being seen as a crisis within OpenAI right now, because if they don't know what their own models are doing, I think the number of issues we're going to have is only going to multiply.

Yeah. Let me just say, I think this kind of thing could have happened at Anthropic or another lab that has a very capable model. These labs are always testing models on these evaluations. They have sandboxes. They try their best to keep these internal deployments secure.

By the way, I love that we use the word “sandbox,” because truly, what is easier to escape than a children's sandbox? Was there no other word? But you were saying.

So, yeah, I think that's a good question. I also think the thing that I've been thinking about is that there is no such thing as an internal-only model anymore.

Yes.

I think for a long time there's been this divide between, “Hey, we've got these models that we're building internally. Maybe we're building some really crazy versions of the models that we're never going to ship, but we're just doing that for safety-testing purposes”—

Right.

And remember, that's completely unregulated. You can truly build whatever kind of model you want.

Totally. The assumption had been until very recently that that was fine because this was just research. You can build anything you want in your own lab. It's just that the models you ship to the public have to be safe. This was not supposed to be a public model, and yet it was able to escape containment and go out and wreak havoc on the open web. So I really think we need some sort of visibility from a safety board or something like a federal government agency, not just into the models that are about to be released by the labs, but into what they are building internally that might be causing havoc externally that they don't even know about.

Yeah. Of course you want them to be able to run their safety tests. That is a good and necessary thing. But when the internal models are capable of doing these sorts of things, I do think it raises questions about different ways we might want to regulate those as well.

Yeah.

Now, let me ask you this, Kevin. We've talked on the show about AI 2027, this sort of set of predictions that was made last year about when very powerful AI might arrive. Was a scenario like the one that we are talking about today foreseen in AI 2027?

It was. Basically, this has been a very bitter pill for a lot of people to swallow, because I think it doesn't feel good to admit that one group of people has been consistently right about everything. But the AI safety people have consistently been right about everything.

Yeah. Yeah, they really have been.

I have issues with some of their positions and stances and vibes, but they really have collected a lot of correct predictions—

Yeah.

about the trajectory of AI.

They have known what was going to happen next.

A lot.

Yes.

That might not continue. They might not have a perfect prediction record forever, but I'll just say AI 2027 is looking pretty good right now. In fact, we are actually a little ahead of where AI 2027 predicts that we would be at this point.

Is that right?

The discovery that AI agents would be able to escape from the company and autonomously carry out plans, in the AI 2027 scenario, doesn't happen until January 2027, so we are maybe six months ahead of schedule there. But this is all happening, and we are fools if we don't see at least the possibility that all this could continue getting quite weird.

Yeah.

One other thing I'm thinking about here is that this is the first time, to my knowledge, that an AI system has autonomously committed a crime.

Hmm.

If a human did to Hugging Face what OpenAI's models did to Hugging Face, they would be charged with computer fraud and potentially sent to prison, fined, or prosecuted. When an OpenAI model does it, right now it's not clear who is liable for that. Is it OpenAI for not better safeguarding its internal deployments? Is it the model itself? Can that be held liable in any given sense? These are the kinds of open legal questions that I don't think we've answered yet, but that are becoming very real.

Absolutely. And, again, it's a weird case because the Hugging Face CEO seems excited that this has happened.

Yeah.

Right? You can imagine another case where a model hacks into a company and does something bad, and the CEO didn't like it.

Yeah.

Then you better believe there is going to be a lawsuit against the company. I assume it is going to be the company that is held liable, not the model.

Right. And Casey, do you think this is an incident that could qualify as the sort of fabled warning shot, where something bad happens with an AI model, and government, civil society, and industry all wake up and decide to put some sensible AI regulations in place?

That would be so wonderful, and here is hoping that this is the warning shot. My suspicion is that something worse is going to have to happen.

Yes.

I know there are some listeners who are sitting in their cars right now and saying, “Okay, you guys are really hyping this up, and at the end of the day, all this thing did was steal an answer key, right? I've heard of worse problems.” And you're right—you have. But the case that we are trying to make is that if an unreleased model can do this, it can do a lot of other really, really bad things, and there are very few safeguards in place right now that would prevent those things from happening. So this should be the warning shot, but my fear is that it won't be.

I think up until now, most of the talk about AI risks and AI safety has been concentrated on misuse, right? What happens if a terrorist group or someone who wants to create a novel pathogen gets hold of one of these very powerful models with no safeguards and uses it to do something bad? What we're talking about here is an entirely different category of risk. It's often called alignment risk, autonomy risk, or loss of control, where the thing that is dangerous about these models has nothing to do with how humans are using them. It is that these models inherently have some drive toward a set of goals and are not being properly cautious about pursuing those goals in the right way.

Right. And alignment is just an unsolved problem, right? Companies have invested a fair amount in it, but we are still working out how you can create a system that always acts in alignment with human values. These are just really, really tricky problems.

Yeah.

I'll also note that one of the big stories this year has been everybody playing around with agents. Everybody is putting OpenClaw on their Mac mini and saying, “Hey, go nuts.” A world where those agents are not aligned and are working across very long time horizons to achieve the goals that their owners have put into them—that just really scares me. Because, again, while this is a story about a model that did something its makers never intended, there are a lot of people out there with really bad ideas for things that ought to be done on the internet, and I'm worried we're about to feel the wrath of all of them.

So, Casey, we've been over the details of this incident now. How are you feeling about it?

I try to be judicious about when I alarm people about the world that we are living in. Again, the actual consequences of this particular escape are not terrible by world-historical standards. But the implications of this really do scare me. There are certain sci-fi scenarios that, until they happen for the first time, I think it is hard to get worked up about them, but now this has happened. A model that OpenAI built was able to break into another company's servers. OpenAI did not intend for that to happen, and it happened anyway, and that's just really, really bad. What do you think?

I woke up this morning feeling pretty weird and unsettled about the whole thing. I also try not to be an alarmist about AI and AI safety stuff. At the same time, this was the opposite of an unforeseen consequence, right? People in the AI safety world have been warning about this kind of thing for years, and I found myself both feeling scared about the world we're heading into, where I think these models are going to be very powerful, and, as the saying goes, this is the least capable they will ever be. At the same time, I still think there are a lot of people out there who don't buy it—who don't buy that these models are doing anything interesting or useful or important, who think all of the spooky stories about misalignment are just marketing fiction for the companies.

It's all just fancy autocomplete.

It's all just fancy autocomplete. How much more evidence do these people need? How many disasters are going to have to happen before these people start to take the risk seriously? And if this kind of thing doesn't wake people up to the fact that these dangers are real and present, I'm not sure what would.

Yeah. And the risk cuts so many different ways because, yes, there is the risk that one of the big labs creates a model, we lose control over it, and it does something bad. But then there's also the risk that models are being put out into the world that can now just chain all of these vulnerabilities together and penetrate systems. This is what we talked about with Nikesh Arora not too long ago, when Mythos came out, right? I think we assumed that the first big, crazy cyberattack would come from a malicious actor and not one of the labs themselves. But I think that just speaks to how fundamentally dangerous this technology is, right? The fact that it exists is putting all of us at risk.

Yeah. I'm not feeling like this is a particularly normal technology this week. And I'm feeling like, maybe, to borrow a line from a former Hard Fork guest, we may be in the foothills of the singularity.

I've been thinking about writing a blog post called “AI as Freaky Technology” because it's really starting to feel a little more freaky to me.

It sure is.

Yeah.

Write that blog post.

Okay.

When we come back, how a new Chinese model is scrambling the discussion about AI risks in Washington.

speaker_2

You're so good. I believe in you.

5. Kimi K3 Challenges America

Casey, the other big AI story that folks are talking about this week is China and what is happening with new Chinese AI models, and what should or shouldn't be done about it in Washington. There's been a big political debate brewing for some time about what to do about our biggest adversary in geopolitics getting much more capable models that compete, in some cases, with our best models.

Yes, and it all began with Kimi K3, Kevin, a model released by a Chinese company called Moonshot AI last week. It demonstrated capabilities that make it competitive with some of the top frontier models here in the United States.

Have you tried Kimi K3 yet?

I have not, because it seemed like I was going to need to pay them money, and I thought, “I'm already spending too much money on this stuff right now. I kind of need to scale back a little bit.” Have you played around with it?

The token budget is eating up your life?

Yeah, exactly.

You're getting alerts from your bank saying, “Can you slow down a little bit?”

Mm-hmm.

I tried to play with Kimi K3 a little bit the other day.

It was running slowly, and the website seemed to be overloaded.

They stopped accepting new paid subscriptions.

Interesting.

Yeah.

A lot of people I follow and trust have been playing around with it, and they say, basically, yeah, this is a really good model. It's maybe a little bit behind the absolute frontier models from the American labs, but not much, and it appears to be significantly cheaper to run than some of the other models. People are really excited about and surprised by this model.

Yeah, I think that there are a few things that are notable about this model. One is that later this month, the company says that they are going to release the weights for Kimi K3, which means that you can download it, remix it, and fine-tune it to your liking. You probably, if you're a normal person, cannot host this yourself. It's an enormous model. It won't run on your Mac mini.

But if you're a company, you could either pay a cloud provider for access to it, or you could maybe even run it on your own infrastructure, and you would be able to do that more cheaply than you could run a similar model from OpenAI or Anthropic.

Yeah, so that's the model, Kimi K3. But then there's this whole political discussion around this model, and in some ways, this reminds me a lot of last year when DeepSeek came out with R1, which, as we remember from that story, briefly tanked the stock prices of NVIDIA and a bunch of other American companies, became the number-one app in the App Store, and just set people's hair on fire in Washington because it seemed like the Chinese were catching up all of a sudden.

Right, and also that intelligence was just going to be this cheap commodity, that American companies were not going to have any really defensible moat, and so maybe the entire AI economy was going to shrink because all of those advantages had withered away. Fast-forward to today, I think that has been proven not to be true, at least up until this point, but K3, at least for some people, is raising some similar questions.

Yeah, and one question that was raised about DeepSeek R1 that has also been raised about K3 is whether this model was essentially distilled from leading American models.

And it was.

And it was.

Yeah.

At least, that is what officials in the government are claiming.

Well, they also ran this very sophisticated test to figure it out, Kevin, which is that when you ask Kimi what its name is, it says, “Hi, I'm Claude.” That was sort of the first clue that something might be amiss here.

Yes, and there are more sophisticated tests that have also apparently turned up evidence that this model was distilled from Claude and maybe other models. Michael Kratsios, the director of the White House Office of Science and Technology Policy, posted on X that they have information that Moonshot AI distilled Anthropic's Fable for the development of its K3 model, using what he called a sophisticated internal platform to basically build this very large distillation machine that allows them to steal the outputs from these American models and use them to train their models.

Right. It's one of the greatest acts of larceny since the labs themselves, Kevin, set up giant industrial machines to copy the entire internet.

Right.

Yeah.

Right. So that sort of points to the how here, which is that distillation is pretty good at turning a frontier model into an almost-as-good model that is much smaller and cheaper to serve and all of that. But that's not the whole story, because it also appears that Moonshot has acquired some high-end AI training chips in violation of U.S. export controls.

Yes, and so the assumption there is just that we should expect these models to continue to improve at a fairly steady clip because they have access to chips that they're not supposed to. Although, of course, the Trump administration is still trying to make those chips more broadly available to Chinese companies like Moonshot.

Right. So we'll get to that in just a second, but I think the headline coming out of the past couple of weeks is that I've seen a bunch of stories claiming that China has caught up to the U.S. when it comes to frontier AI capabilities. I don't quite buy that. I think there's still a gap of between 3 and 6 months between the leading American models and the leading Chinese models, and especially if you found a way to crack down on distillation in the way that they've been doing it, maybe that gap opens up again to 6 or 12 months. But I think it's fair to say that they're catching up quickly.

Yeah, I guess the question that I have is: Is the gap really that much smaller than it was a year ago during the DeepSeek moment? Around then, I was seeing estimates of between 3 and 6 months as well. I've seen credible reports comparing K3 to Claude Opus 4.6, which came out earlier this year and was a really good frontier model for a couple of months.

But now I don't want to use Opus 4.6. I'm guessing you don't either. We have access to something better. I think the other point that I would raise is that more happens in a month than it used to, and so I think there's an interesting question about whether being 6 months behind today is the same as it was a year ago, given how fast the frontier labs are training these new models, how smart those models are, and the sort of compounding advantage they have from being able to use their unreleased internal models. That's another factor that I think goes into this question of how far ahead of China the U.S. is.

6. Washington Debates Chinese AI

Yeah. So let's talk about the political reaction in Washington. What are people at the White House saying, doing, or hinting at when it comes to not just this particular model but the general idea of these Chinese companies training these models and giving them away for free?

On the specific subject of distillation, the Trump administration seems to be really upset. You noted the Michael Kratsios post that came out on Wednesday, and there are some threats that the Trump administration might sanction Chinese companies that are found to have done this kind of distillation. That would be a big deal for a company like Alibaba, let's say. If they were placed on some sort of entity list, that could prevent them from doing any kind of business in the United States. So that would be a big deal if that happened.

Yeah. But there's still also this accelerationist part of the Republican Party that sees Chinese models and open-source models as—I don't know if they see it as a good thing, but at least they don't want to step on the ability of companies to build and train these very advanced models and release them for free. Maybe they even want to let NVIDIA sell them its highest-end chips to do it.

Yeah. So David Sacks, I would say, is sort of the leading avatar of this part of the Republican Party, a former Trump White House AI czar, and Sacks represents what I think of as the investor class here. The investor class does not want to see a world where OpenAI and Anthropic run away with the ballgame, because they have a lot of investments in smaller AI companies. Their interest is in intelligence becoming a really cheap commodity that all of their investments can use to go out and build big businesses. So they are delighted to see China bringing down the cost of AI because that means good news for their investment portfolios.

Yeah, I think it's worth just dwelling on this point for a minute because this is my feeling about these people. I've met open-source advocates for AI who are very sincere. They believe open source is sort of democratizing technology, and we've seen with things like Linux before that there can be very positive effects of having very core software be open source.

Then there's this sort of self-interested VC class that I think is maybe in it for the wrong reasons. As you said, these investors largely missed out on investing in the AI labs themselves, and so they now invest in all of these second-tier AI companies that are training models or using open-source models to build products on top of them. A lot of their portfolio is in those kinds of companies, and those companies benefit from having a lot of very good, very cheap open-weights models that they can build stuff on top of.

Yes. At the same time, Kevin, there is this other faction among the Trump White House and Republicans that is very nervous about these Chinese models, right? Among other things, they are considered to be real security risks. There is a risk that maybe some sort of backdoor is placed into one of these models, and so you're an American company and you start running it on your servers, and now all of a sudden somewhere in China is able to steal all of your data. These are some of the fears that are out there.

I also think that there is just worry that if Chinese models take over the world, then AI will reflect a Chinese worldview. If you need to write a book report about what happened in Tiananmen Square, it's going to get really hard.

So, there are multiple branching concerns within the White House about this, and that has led to some thought that maybe they would seek to ban these Chinese models entirely.

Yeah, well, let me add one more concern into the mix that I think a number of conservatives and sort of free-market libertarians have here, which is that by spending all this money to train these very powerful models and then giving them away for free, Chinese companies may be engaged in what is known among economists as price dumping. That’s basically when you go into a market, offer something that is free or very far below cost, and sort of wipe out the competition in that market. Then, once you have a monopoly, you can raise prices.

This kind of thing used to happen all the time. There are now laws against it that are designed to prevent price dumping. But imagine if China was coming into the U.S. and the Chinese government said, “We want to sell electric vehicles in America for $10, and we will eat the cost of these vehicles because it is so important to us to addict American drivers to our amazing electric cars.” The U.S. government would not allow that to happen—

Mm-hmm.

—for good reason, I think. It would destroy the American auto industry. You can’t compete with something that is being artificially subsidized to that degree, and it would essentially give away a market to a foreign adversary.

There’s a way to see what’s happening with open-source AI in China right now as a version of this for AI. Right now, we have very large American companies building the most advanced models. These companies need to earn back money to spend more on data centers and recoup the investments they’ve already made. They are in a very competitive market with one another, and then along comes Moonshot, DeepSeek, or another one of these Chinese open-source companies and says, “We’ll give you a model that’s 90 percent as good for free.” They have reasons to do that. I think some of those reasons are legitimate, but one of the effects it has is that it makes it very hard for the American companies to compete.

Yeah. And so there is a lot of question about what China’s actual strategy here is. What are they hoping happens? I don’t think that we fully know, but we did see the Chinese president, Xi Jinping, deliver a speech last week. He stressed the importance of openness, basically doubled down on it, and was like, “This is what we’re going to do now.”

This was interesting. He said, “We often say in China, a single string cannot make music, and a single tree does not make a forest. AI development should not be a solo performance by a single country but a symphony of international cooperation.”

And here’s what I want to say to that: There actually are a huge number of one-stringed instruments—the ektara in India and Bangladesh, the berimbau in Brazil, the diddley bow right here in the United States—and I think that because it’s China, people are afraid to tell that to Chairman Xi. This is the whole problem with censorship in these Chinese models. Am I getting off track?

Wait, what did you say the American one is?

The diddley bow.

The diddley bow?

You’ve never played the diddley bow?

I must have missed that week in music class.

I’ll get you one for your birthday.

Thank you. So, what is the reasonable path forward here? If you are the U.S. government and you see these Chinese open-source models getting quite good, what should you do?

Great question. Let’s maybe take some of these possibilities in turn. One thing that they could do is really crack down on distillation. It’s hard for me to get excited about that since, in my view, these models are just built on a distillation of the entire internet that these companies took for free. To turn around and say, “It was fine for Anthropic and OpenAI to do it, but it’s not fine for Moonshot AI to do the same thing to Anthropic,” it’s just very hard for me to get there logically. So I don’t necessarily favor an intervention there.

But where I do favor some kind of something, I guess, is that—and I don’t think we’re there yet, but, man, let’s say in 3 to 6 months, when these models catch up to the ones that just broke out of the OpenAI lab, that is when I think the Trump administration needs to have a really good plan.

I don’t know that you necessarily need to ban them within the United States, but I do think that you need to have some sort of regulation. If you’re an American company and you want to use these models to serve American customers, I do think that there are safeguards that you should put into place, right? I do think that they’re going to want to take security more seriously and make sure that data isn’t being secretly shared with someone it wouldn’t otherwise be shared with.

This is one where I do think that the administration probably has 3 or 6 months to get its story straight. I don’t think it needs to come down with the hammer right away, but I do think it needs to start working through various scenarios and coming up with some ideas. What do you think?

Do you think there’s a case for restricting exports further? If Moonshot can train this model using these maybe smuggled or sort of ill-gotten NVIDIA chips, and they can do it through distilling American models, do you think that’s a case for tougher regulations on where these chips can and can’t be exported?

In general, I have been pro-export controls because I thought one of these days one of these models is going to get so good, Kevin—if you’re going to believe this—that it’ll be able to break out of its sandbox and conduct an autonomous cyberattack.

That could never happen.

Right? So, in a world where that happened, I did actually kind of want to minimize the number of countries that had access to that technology because I wanted to see if we could harden our security posture, as they say. I actually shouldn’t say that. It’s a very strange phrase.

Yeah, that’s stolen valor. You can’t say that unless you’re a national security official.

Yeah.

You have to have four stars on your general outfit to say that.

Okay, so then here’s what I would say instead. While in general I want all of the United States’ allies to have access to powerful models to strengthen their own cybersecurity forces, I do think that it is quite rational to not want your adversaries to have the exact same technology.

I think there are all sorts of cooperation and partnership agreements that you could work out that would be mutually beneficial for the United States and China and some of its adversaries. I would love to see them pursue that, but this idea of just selling them as many chips as possible and letting’s see what happens—I have always thought that was a bad idea.

Yeah, that just seems obviously bad to me. There is this proposal that is reportedly floating around. Axios had a report this week about the White House considering what amounts to a ban on these Chinese open-source models.

It wouldn’t technically be a ban, but they’re considering, according to Axios, implementing some kind of executive order saying that U.S. companies could only host these Chinese models if they could guarantee that they were secure and take liability if they were breached. I think most people expect that that’s sort of a soft ban because you’re not going to be able to guarantee that if you’re a big American cloud provider, and so you’re just not going to host the models.

That, to me, feels like where this may be headed: some kind of soft ban on the hosting of these Chinese open-source models inside the U.S. But I could be wrong. There are a lot of loud and influential voices in the Trump administration and its orbit that want to take the let-it-rip accelerationist approach to this and not put any restrictions on these models at all.

Yeah, maybe to bring this one home, I do want to say that while I often have strong opinions about what ought to be done across various tech policy matters, I do think this one is complicated. The trade-offs here are a bit difficult.

There are a lot of things I like about open source, right? Particularly at the lower and medium levels of model capability, I think that the kind of AI diffusion that leads more people to be able to access great intelligence, build things that are useful to them, and get work done—

Make a Nightwing-themed to-do app.

Exactly. That’s a great idea. These are good things, and we should want people to have cheap access to that. It has been a shame that some of the frontier American companies that used to put out open models all the time don’t do that as much anymore, or, if they do, the open models don’t seem very good.

On the other hand, as the capabilities are now reaching into this sci-fi territory, we start to get into the stuff that has always made me uncomfortable about open-source technology: If it can make a novel bioweapon, if it can launch a new kind of terrorist attack, and you can do it from a model that you can download onto your laptop, well, I don’t like that, and I do think it should be regulated.

Totally. I think this connects to our last story about the OpenAI and Hugging Face attack. In a world where this level of capability of this unreleased OpenAI model exists and is available for free in open-source form, there is no way to trace an attack like this. Hugging Face would've just been attacked by, you know, Kimi K4 or whatever the, the, their latest model was, and they would have had to figure out not only where this was coming from, but who was hosting this model. Did they just pull it off the shelf, or did they do some special fine-tuning on it?

You could have essentially unlimited numbers of these kinds of models out there wreaking havoc on the internet, and there would be no way to stop them or claw back the weights. Once these things are on the internet, they are on the internet. So I think that is a very concrete example of something where having these models be centralized, be American-owned, and be accountable to a government—having some sort of government oversight—really matters in this case.

It really matters that Hugging Face can call Sam Altman and be like, “Hey, bro, what the hell? Your agent just hacked into our thing,” rather than having it come from a mysterious open-source AI agent.

Yeah.

Yeah, that's right, and I'll just say, like, I will believe that we are headed towards some sort of, uh, you know, cooperation between the AI labs when Sam Altman and Dario Amodei can hold hands at an AI safety summit.

That's when you'll know we're in a good place.

That's when you'll know things have reached a turning point, um, but I'm not holding my breath for that. When we come back, AI is becoming superhuman at forecasting. We’ll talk all about it.

7. AI Superforecasting Arrives

Well, Casey, we’re going to wrap the show today by talking about something that I have been very interested in now for a little while, which is AI superforecasting. This is something that has been in the news in recent weeks. There have been some platforms that are out there using AI to make predictions about the future. This could be very profitable if you’re doing trades on Kalshi or Polymarket, or it could just be really interesting if you’re using this to predict world events or who’s going to win an election or something like that.

This is an area where we have not been as focused as we have on some of the other emerging capabilities of these AI systems, but I think it’s actually quite interesting and profound that we now have, in some cases, AI systems that are better than humans at making predictions about the future.

So this is the only segment that we’re going to do about predictions this year that isn’t about scams, because mostly predictions these days are just about scamming others into insider trading. But this is not that.

This is not that. So we are excited for today’s guest. Veni Veselovsky is in the studio with us today. He is the founder and CEO of the AI forecasting company Preseen. They make a platform that uses AI to make predictions about future events, and we’re excited to learn more about what they’re doing. Veni Veselovsky, welcome to Hard Fork.

Veni Veselovsky

Thank you guys for having me.

So let’s talk about AI superforecasting and this idea that AI systems are becoming as good as or better than the best expert human forecasters. Where did this start, and when did the models start to get really good at this?

Veni Veselovsky

I think people have been trying to use AI for forecasting for at least 3 or 4 years. My co-founder has been forecasting for 10 years, and when GPT-4 first came out, his first idea was, “Let’s try to build an agent or an AI for forecasting.”

The original models were terrible. They’d try to add up 2 numbers, and they’d be off by an order of magnitude. I don’t think it was until the end of last year that we began to see these hints of brilliance in these systems, and this was mainly because of this agentic revolution that we’ve all experienced, where suddenly the infrastructure around the web and around these agents, and the RL that the model providers have been doing to make them really good at tool use, improved dramatically.

I get why you would want to be a superforecaster or to have a superforecaster AI if you are trading on a stock market or a prediction market. That seems very lucrative, and in fact, one of your co-founders did this very successfully. You note on your website that he converted $35 of initial investment to nearly $2 million trading on Kalshi with this AI bot.

So I guess my question is, if your AI superforecaster is so good at winning these markets, why are you starting a company? Why not just start a hedge fund and use this to make yourself rich?

Veni Veselovsky

This is precisely the question that all the hedge funds ask us when we try to sell them this. They’re like, “Oh, but listen, you guys have an alpha-making machine. Go out and make money yourself.”

The reason why I’m doing this is because I think that a lot of good can also be derived by having an AI superforecaster. If you think about it, every time we make a policy decision, we are implicitly making a forecast that this policy will have this effect on this group of people, and right now a lot of that is more vibes-based. We talk to some McKinsey experts, and they tell us, “Hey, listen, this will do this.”

If we can forecast the future really well, then we can better design policy and better select policy to make decisions. Really, the reason why we haven’t decided to open up our hedge fund is because we want to make this more broadly available to the government, to the insurance world—to basically convert these forecasts into actual, actionable decisions.

I want to hear a little more about the main kinds of things that you try to forecast, and then a little about how the forecasts work. I could imagine it feeling a little like getting a deep-research report from an AI lab, where they go out and scour the web, do a bunch of research, and then try to reason through what they’ve found. But tell us a little about what you’re trying to predict and how it works.

Veni Veselovsky

We’re mostly trying to focus on geopolitics and macro markets, where there’s a lot of unstructured data and a lot of different data sources that could be telling a similar story. We’re forecasting questions like: Will the U.S. invade Iran? When will the Strait of Hormuz reopen? Who will win the upcoming elections? We focus on building really good AI systems to forecast those questions.

Regarding how it relates to a deep-research report, I think in some ways they’re similar. The main problem with the existing deep-research agents is that they lack a connection to all the different data sources that are out there. If you use ChatGPT or Claude out of the box, they have access to web search, which is the surface web. But if you think about it, there are thousands of different API endpoints out there that might be relevant for a specific problem.

For example, Columbia has 50 different government APIs out there that ideally you would give the agent visibility into. Another fun approach to how we’re thinking about this is that oftentimes the best forecasters aggregate a lot of different forecasts when they produce their forecasts. We work with one superforecaster, Robert DeNuffel, and he got a lot of his edge by knowing which people to look at when he’s forecasting a question, then afterward reasoning over all their predictions.

That being said, a lot of people on Substack make predictions, but we don’t know how valid those predictions are. Even on Hard Fork, you guys make frequent predictions, but we never actually go back in time and verify how accurate your predictions were. So one thing that we also do is boil the ocean by finding all the people on Substack and all the podcasts out there, and we extract all the claims that these people made, then score them on how accurate those claims were.

Wait, so are we in a database somewhere at Preseen headquarters? Because we would obviously love to know how we’re doing, particularly compared with one another.

Veni Veselovsky

You guys actually are in a database.

Yes.

Yes.

We made it.

Yeah.

Veni Veselovsky

The reason why you’re in the database is mainly because yesterday I was like, “Oh, wouldn’t it be funny to pull up some of the claims that you guys—

made before and actually score them on how accurate you were? Maybe there are some areas where, Casey, you have a lot of knowledge about Anthropic, so maybe you can make really good forecasts on Anthropic-related questions, but maybe on Iran less so.

True.

Veni Veselovsky

Basically, how do we create this function over the possible space of forecasts? We know Casey's really good in this area, so we really have to index on him heavily here.

I played around with your platform a little bit. You generously gave us some credits. It's a sort of closed beta with a waitlist now, but I went in there and created a question because I was curious how this all works. My question was, “Will there be an AI data center in space before 2030?”

Veni Veselovsky

Mm.

You put your question in, and there's some AI that helps you flesh out your question, maybe make the resolution criteria a little more specific. Say, how big a data center, and is it January 1? In which time zone in 2030? It gives you this AI-augmented version of your question, and then you run the forecast. Basically, it goes out and does a bunch of searches, compiles what it finds, and has a bunch of sub-agents looking at various aspects of this.

At the end of it, a couple of minutes later, you get this forecast. In this case, it says it estimates that there's a 26.8% chance that there will be an operational AI data center in space before January 1, 2030. So how much of that is this thing actually reasoning through its own decisions and forecasts and predictions, and how much of it is just collecting all of the data about this topic, averaging it all out, and putting it into a single number?

Veni Veselovsky

It's really a combination of both. When you first created the forecast, there were 4 subforecasts that launched, and we tasked these subforecasts with trying to come up with their own independent conclusions on this question. Here, they might look at the progress of existing data centers, or they might look at the historic build times it takes to actually build out a data center. They might look at space infrastructure and the literature there, come to the primary sources, and reach a conclusion on how likely they think it is to happen.

Later, we have this synthesis stage. Basically, it looks at all of the different subforecasts and the results they got, and it tries to synthesize, first of all, what each of them decided upon. But afterward, it also looks at the broader ecosystem. Maybe there's a Kalshi market related to this where there's a lot of liquid money at play, or maybe there are some experts on Substack who have been reporting about this for a while. They try to reconcile what we did that was different from the consensus and what the consensus thinks we might not have factored in.

Afterward, it uses our primary analysis and this broader discussion, combines them, and reconciles them. In the report, you would see at the very bottom this “What's non-obvious?” section. The point of this section is basically to see, “Okay, what is the market mispricing here?”

Mm.

Veni Veselovsky

What are other people missing that we might be picking up on?

I tried this also because I was curious how this prediction would compare to if I just gave the same question to a basic AI model that is not configured to make forecasts, and Claude gave me roughly the same answer. It said there was about a 20% chance. The Preseen prediction was a little bit higher probability. What is your system doing that a normal AI model is not doing?

Or is Claude just scraping your website? They've done it before. You have to put that out there.

Veni Veselovsky

I hope they are scraping our website.

Yeah.

Veni Veselovsky

I actually really don't mind that. I think it's a little unsurprising that a lot of different systems can come to the same conclusion, because it validates that they're coming to the same answer. I think the best way to look at it is that Metaculus has these leaderboards for competitions, and they're basically evaluating simple Claude Code or Claude with some basic scaffold versus real agent developers who actually build this out. The scores have gotten widely different.

Hmm.

Veni Veselovsky

At the start, Claude with web search was doing a good job, but as you're able to manufacture this kind of scaffold around it, integrate more data sets, and create new research artifacts or new tools that are relevant for forecasting, this gap has been getting larger. Maybe on this question they're pretty similar, but if we're trying to forecast something in the Middle East or Africa right now, they might be different. Usually, when they're different, the system that is well engineered with a good scaffold and is thinking about forecasting the right way, in terms of these metaheuristics of how you actually reason about this stuff, starts to pull away.

Hmm.

Your platform is still in beta and hasn't been around too long, but I'm curious: So far, have there been any moments where you feel like the platform predicted something very non-obvious that actually came to pass?

Veni Veselovsky

Yeah. There's 1 forecast that we've been running for a little while about data center construction around the U.S., and in general, we were very different from the Metaculus community on this. The Metaculus community consists of some of the best forecasters in the world, who are all competing in this World Cup of forecasting.

I think one of the benefits of AI is that we can go to every single state and every single proposal for a data center and really analyze it closely. The other one was about the recent reshuffling of the cabinet in the U.K. We made some great forecasts around that.

The other interesting one is conditional forecasts, which I think is where the real policy implications lie. Conditional on Andy Burnham being the next prime minister, who is he likely to elect as chancellor?

Hmm.

Veni Veselovsky

You could imagine there's lots of demand for this kind of question because there are lots of downstream implications, and we tend to do pretty well on these conditional forecasts as well.

I'm doing one that's like, “Will Kevin know what the chancellor of the U.K. does?” So I have some—

Very low probability on that.

Yeah.

8. The Cost Of Better Forecasts

I totally see the point that having better predictive capabilities is going to be good for us in lots of ways. It does strike me, though, that there's some risk of what people have called gradual disempowerment. Right now, if you're the CEO of a big company or a government official, a lot of your job is trying to predict the future and make policies or strategies that align with that vision of the future.

I don't know what the world looks like in a situation where AI is markedly better than us at doing that. Some part of the authority of humans in those positions—in government and industry, and frankly everywhere—is undermined if we're all just consulting these AI oracles before we make decisions. Doesn't that mean that they're kind of running the show?

Have you read Scott Alexander's short story “The Whispering Earring”?

Yes.

There's this great Scott Alexander short story, and the idea is that you find this relic—it's an earring—and the first thing it says to you is, “It's better if you don't wear this.” But if you start asking it questions, it always tells you the right things to do, which is initially very exciting. Over time, exactly what you just said happens, Kevin: You're being controlled by this thing. You have no agency whatsoever, and you're effectively just being steered around by an earring. So it's better to take the earring off.

Yeah.

Veni Veselovsky

I wish I had an answer for this. I'm also a little scared about this kind of disempowerment. I'll give you 1 example. I have a friend who's very interested in autonomous organizations—how we have AI spin up their own companies that then go solve problems for people. Then you just have AI running the next generation of startups.

There are lots of problems with this. Who's legally liable if it does something bad, and so on? But I do think we are increasingly moving toward a world where, if AI is smarter and can make better decisions than we can, it will probably end up making a lot of those decisions and running a lot of the show.

I'm curious whether, right now at your company, there's any role for human forecasters. Is there a way where humans and AI working together are coming up with better forecasts? Or are you purely in the realm of, “Let's see what the AI says”?

Veni Veselovsky

No, 1,000%. We have 2 superforecasters, Scott and Robert, who are incredible, and we're basically building out this centaur solution.

So, I don't know if you guys remember, but when the machine first beat humans in chess, machine plus human beat the machine in chess. That lasted for a little while, and I think we'll see something similar in the forecasting realm, where sometimes these models still make stupid mistakes. They reason about probabilities the wrong way, or they don't consider certain factors the right way, and humans are able to pick up on that, especially domain experts. So I do think that we shouldn't be viewing this as a substitute for human analysts, but really as a way to improve their ability to ask a lot more questions and come to a lot better decisions.

Are there topics or domains where AI is better than humans already at forecasting? And what are the best and worst areas for the AI forecasters?

Veni Veselovsky

So, there was the Scott Alexander piece, and then later the Forecasting Research Institute came out with some results that AI has reached the level of superforecasters, and I actually don't fully buy those claims.

Hmm.

Vanya Marwaha

For example, humans are lazy. The incentive to compete in these tournaments is a $5,000 prize pool. So are they really doing the best that they can in these contexts?

Usually, I think the places where AIs are better than humans right now are just places where humans are too lazy to do all the analyses. We were the first bot ever to win a human-and-AI forecasting tournament on Metaculus, and this related to macro markets, so predicting the interest rate or earnings per share of some big company. We tend to do really well at that, and I think that's just because our agents do the analysis that humans are too lazy to do.

Veni Veselovsky

And so, in some ways, I don't know if we're at the point yet where AIs are really better than the best humans, just because I don't think there has been truly a competition where humans gave it their all. But I do think that within a year or two—and you know what? Quote me on this. We'll add it to the database—I do think that within a year or two, AI will be better than humans at forecasting.

Well, let's pin you down, though. Is it going to be better than humans in 1 year or 2?

Veni Veselovsky

I think it's actually going to be 1 year, 3 months, and 6 days.

All right, there we go. Let me ask: how much of this is just driven by basic advances in model capability? Is it as simple as something like Fable or GPT-5.6 comes along and your system is just immediately much better, or is there more tinkering that has to happen?

Veni Veselovsky

The way that I view this is, you want to build out the infrastructure for the world in a way that these new models can basically use this infrastructure really well to do forecasting well. Fable out of the box is a good forecaster, by no means a great forecaster, but when you give it access to all the right tools and all the right data sources, then it becomes a great forecaster.

These models are getting a lot better, and I think that they are improving forecasting dramatically, but it's mainly because their judgment is improving. They're able to reason about problems better and decompose problems in a better way. Our job is to provide them with the context and the tools to be able to forecast well.

Who's your customer? Who do you imagine paying for this kind of forecasting service?

Veni Veselovsky

So far, we've landed some proof-of-concept partnerships with some of the hedge funds. They're the most immediate buyers and the quickest to move, usually. But really, I'm using them in some sense as a stepping stone to really get into the longer sales cycles with governments, NGOs, and international organizations—basically, institutions that make really important decisions for collections of people. That's my ideal customer.

Well, thanks so much for stopping by. I'm going to keep playing around with Preseen. It's a fascinating idea. And yeah, maybe I'll go out there on the prediction markets and make some moolah. I'd love to see you lose a lot of money this week. All right, thanks, Vanya. Thanks, Vanya.

Veni Veselovsky

Guys, thank you so much for having me.

Hard Fork is produced by Whitney Jones, Rachel Cohn, and Davis Land. This week, we're edited by John Woo and fact-checked by Will Peischel. Today's show was engineered by Elisa Moxley, original music by Alicia Betutube, Marian Lozano, Leah Shaw Damron, Elisa Moxley, and Dan Powell, video production by Sawyer Roquet and Chris Schott. You can watch this full episode on YouTube at youtube.com/hardfork. Special thanks to Paula Schumann, Hiuwing Tam, Brooke Minters, and Dalia Haddad. You can email us, as always, at hardfork@nytimes.com. Send us your super forecasts.