正在逼近 AI 事件视界？第2部，与 Abhi Mahajan、Helen Toner、Jeremie Harris、@8teAPi 对谈

I’m going to add Abhi Mahajan. Abhi is Owlposting online, and he works on AI for cancer at Noetik.

Abhi, welcome.

Yeah, great to meet you. Thanks for having me on. You have the great distinction of being recommended to me as the Zvi for AI and biology. Big shoes to fill, big reputation to live up to, but I’m excited to meet you. This is actually the first time we’ve properly spoken.

Yeah. I learned from Ron Ron Alpha that you built an entire competitive-intelligence platform, LLM-based, to feed the clinical-analysis pipeline. I also learned that Claude recommends every cancer drug it sees. Let’s talk about that.

The typical way that a lot of biopharmas are increasingly interested in asset acquisition, as opposed to just developing their drugs from scratch, is partially because China is pumping out a lot of very interesting preclinical assets. Why not just buy those for a few million dollars? They’ve already done the optimization. Let’s just run those in patients.

Most of the time, the way you look for these drugs is either you mine your personal network or you have these clinical-trial aggregation platforms that help you do the job. Both of these are obviously lossy, and a better way is to scrape the entire semantic web yourself and annotate every single investigational drug you find with your company’s priorities—what you think is important to look for, and the modalities that you’re particularly interested in. Organize all that into a table, rank it by some metric, and then give that to the therapeutics team to work off of.

Obviously, there’s still a human due-diligence step. These models still aren’t perfect. Even 5.2 and 5.3 aren’t perfect, but they’re pretty good.

Do you have an internal eval that you run? When you swap model engines regularly, do you upgrade every time a new model engine shows up? Do you evaluate and then decide?

It’s a pretty hacky process. Our metric—or at least my personal metric for evaluation—is: among the drugs that our therapeutics team is really interested in and wants to move forward on, does the next version, the next generation of the LLM, continue to recommend those drugs and say that these are very good?

I don’t actually think it was that good at the very beginning. I only built this pipeline a few months ago, and it remains pretty good now. I don’t think there’s been any dramatic jump. I partially think this is due to the fact that identifying what makes for a good drug is a very qualitative process and a very vibe-based one.

It depends on the economic status of the company. It depends on whether we know anyone there, because oftentimes these companies don’t make it easy for you to give them your money. It takes a super-long process to figure that out. Yeah, it’s pretty good, though.

So I definitely recommend your blog, Owlposting.com. I’ve still got quite a bit of the archive to work my way through, but I want to throw a couple of what I thought were your more interesting—arguably hot—takes at you and get you to double-click into some of the intuition and implications of those.

One: we’re obviously in a moment now where there’s a tremendous amount of interest in creating AI scientists of all kinds. One of the big bets that companies are increasingly making, with some serious capital behind them, is that they’re going to close the loop by allowing AIs to design and run their own experiments through some sort of automation, feed that data back in, and get reinforcement learning from basically experimental results.

Now, one of the things that you had said in one of your posts is that there’s not a lot of verifiable ground truth in biology. I would love to understand what that means exactly. What does that mean in terms of the ability to close that loop? Is there some sort of fundamental messiness or uncertainty that you see as, at least in the near term, being irreducible and becoming the functional limit on how much systems could learn from that kind of closed-loop experimentation?

Yeah, I like to say that saying biology has no verifiable ground truth is probably a little hyperbolic on my end. But what I will defend is that there’s not a lot of verifiable ground truth for the most clinically valuable problems.

Yes, there is verifiable ground truth for questions like, “Does this protein exist in the solution?” or, “Is this variant that your NGS sequencer identified true?” Those are both verifiable. But I don’t think you’ll quite see the same explosion of intelligence that happened in math and code as you will in biology, because the rewards are so cheap and easy to get in those fields. In biology, it’s just such a long, iterative process to get any iota of information.

One easy analog to this is training an RLVR model on the task of writing the bestselling book, right? There is technically a verifiable reward: there are book sales, and there is the country that the author is writing from—all these sources of data—but it takes 18 months to get that singular data point. When you get that singular data point, it’s very hard to trace it back to any one of these things.

One biology-grounded example of this is, let’s say you want to do RLVR on toxicology prediction. This is arguably the thing that sinks the vast majority of phase 1 drugs out there. Toxicology sounds like a very simple topic. It is not. A drug can be toxic on the order of seconds, like snake venoms. It can be toxic on the order of months or years. It potentially doesn’t kill an animal; it maybe just leads to cognitive deficits or heart damage.

Oftentimes, it’s dose-dependent. It could also be species-dependent. All these measures of toxicity have no real way to be understood other than just observing them in vivo, in an in vivo setting, and then seeing what their readouts are.

There are companies like a San Francisco-based startup called Axiom, which is trying to create a model that can very easily tell, given a small molecule, what its toxicity impact is on hepatocyte cells in a cell dish. It’s a very clean, simple problem that probably saves months of time in preclinical settings, but it doesn’t poke at the much more important problem of how this performs in an animal.

Just a segue here. Isomorphic Labs, I think yesterday, announced a predictive model which doubles the performance of Alpha 3 on key benchmarks: binding affinity, pocket identification, and structure prediction. How does that fit into how things go? Is this actually useful, or does this just create more targets that need to be validated anyway?

Yeah, I mean, obviously, a very incredible piece of work by Isomorphic. I’m no longer in the protein engineering field, but I think that benchmark they did—that leftmost plot they’re presenting on—is an incredibly difficult benchmark to get better at, and they’re 2× better than what was previously available.

So, very good. But I’m sure you’ve heard the sentiment that the field is already awash with many really good preclinical assets, and the bottleneck is actually how well these work in patients. It sounds perhaps obvious that if you get better at this preclinical design step, you get better at putting it into humans.

That’s a story that has been told for 10 years. It is not obviously clear that any of it has borne out. I imagine at some point it will, but there isn’t really strong evidence to suggest that it does.

There’s actually this really great chemistry paper that came out just a few days ago called “The Affinity Advantage.” That paper is probably one of the strongest bull cases that being able to optimize every facet of every protein that comes through the preclinical pipeline has nonlinear or superlinear benefits to the drug development process, and it’s just a matter of time until these models get even better. It’s not an opinion I share, but I’m sympathetic to it.

In Dario Amodei’s—I think one of his papers, the blog post he had, “Machines of Loving Grace”—he tried to map out how he thought developments in biology worked. He pointed out that a lot of the major developments in biology come from better imaging and sensing techniques that allow you to look deeper and understand more deeply what’s happening in there. After that, it becomes easier to do a lot of other things downstream of it, starting with microscopy, which led to all the downstream developments from there, and so on. What do you think are potentially the developments that might be coming up in the next 4 to 5 years that might do something like that?

I guess I would like to vaguely gesture toward building generative models of human in vivo biology. I think there are layers of discussion to be had: What other instruments do we need? What other modalities do we need? But I think there’s a lot of low-hanging fruit in simply collecting a huge amount of highly rich data from real human tumors, intestinal lesions, and plasma readouts, and just feeding a model with that information—not paying attention to any of the in vitro or otherwise biologically unrealistic settings.

From there, maybe you get access to a genuine, bona fide human simulator of biology. Maybe that’s really helpful for fixing the current state, where 97% of oncology trials fail. I think the Dario pitch of scientists in a data center churning out interesting ideas—there are already tens of thousands of PhD students churning out very good ideas. Most of them can’t be validated because it’s too expensive to do so.

That connects pretty directly, it seems like, to what you are doing in your work on cancer at Noetik, right? You guys are focused, first of all, at roughly the clinical stage and try to predict what drugs will work best for a particular patient, given some relatively deep data about their specific condition. So maybe walk us through what that looks like.

I was interested to learn that it is basically a foundation model, with lots of different data sources thrown into it, and that it is trained with this kind of masking strategy, where the idea is that the model has to learn how to predict from partial data, whatever partial data it might have. I’m a big believer in that strategy because there are so many modalities and so much noise going on inside the system that we don’t understand. I’ve been a big speculator about that being a driver of how AI can help in health over time. So give me the double-click past what I have been able to learn with online research into what you guys are doing.

Yeah. So, the economic pitch for Noetik—let me start with that—is that 97% of oncology trials fail. You could look at that and say, “Wow, we’re awful bad at designing these drugs. Maybe we should get better at designing them.”

But one interesting phenomenon is that if you look at a lot of the papers that are published after a cancer clinical trial fails, there are usually some patients who did respond to the drug or respond to the regimen they were on. The researchers try really hard to figure out what the exact biological archetype is that makes up this response population.

They always come in with something super complicated, very heterogeneous. It’s like this particular cytokine group or granzyme genes were highly expressed in the response population. It never leads to anything particularly interesting.

And so one argument you could make is that maybe the biomarkers that define patient response for this particular drug are non-human-legible. You need a black-box biomarker to encapsulate whatever that piece of information is. Noetik is built around that thesis.

We collect vast amounts of human tumor data. We profile them at 4 levels of modality: pathology, which is kind of the blue-chip modality that almost everyone has; spatial proteomics, a 16-plex panel to identify cell types; whole-transcriptome spatial transcriptomics, which is 19,000 genes over the entire surface of a tumor, to identify the functional state of the tumor; and then exome sequencing to identify genetic alterations. Is this K RAS positive? Is this SDK knockout, and so on.

The ML angle is that you train, exactly what you said, a self-supervised masked model, in the hopes that, first, you get a very good representation of any given tumor that walks in the door.

So you now have the ability to place, in the universe of all the cancers I've seen, where this patient falls in that embedding space. That's what we're doing a lot of: we gather patient samples from people who have run clinical trials. We profile them, run that through the model, and see if the response population falls in a different area than the non-response population. If it does, maybe we have access to a biomarker that no human on Earth understands, but that we are uniquely able to identify.

The more interesting thing you can do with it is use the generative capacity of the model to knock out specific transcripts or genes and see how that changes the expression of transcripts within the tumor microenvironment. You can imagine there's this concept appearing in the cancer literature called nudge drugs, which are drugs that don't actually operate on the immune system or really the cancer site itself, but rather push it in a direction that makes it more sensitized to other drugs. You can imagine, “I'll knock out this particular transcript, and then I will hallucinate what it would be like if I add a kit, which is like an immune checkpoint block that operates on the PD1 access into the site of the tumor.” Maybe now you predict, “The tumor is highly inflamed. It's hot. There's a high chance that it'll just melt away entirely.” Yeah, those are the 2 big economic and ML strategies we're pursuing.

Yeah, that's really super exciting. When you talk about, first of all, identifying—or having access to, I think was your phrase—biomarkers that nobody else has access to because you can see a sort of divergence in where different patient populations fall in embedding space, do you have any means right now of saying, “Okay, because these patient populations are falling into different parts of embedding space, what is it actually that's causing that divergence?” And if not, maybe I can introduce you to the good folks at Goodfire, who just did a version of this by identifying biomarkers for Alzheimer's that had previously not been identified.

How far along are you guys in terms of being able to make interpretable what it is that the models have learned from their unsupervised training?

Yeah, the previous Goodfire post was very interesting to read. We do have a mechanistic research group internally that's exploring these ideas, and I have no doubt they'll find something interesting. But one argument against doing this at all is: why do we care about interpretability? In a clinical setting, we might care about interpretability because the FDA gets very upset with you if you try to do anything that's a black box. Maybe that was true a year ago, but circa, I think, September or August 2025, there was a pathology AI company called ArteraAI that came up with what is basically a companion diagnostic. They take in the pathology slide of your prostate tumor and predict whether you will respond to androgen-deprivation therapy.

They have no idea why this model works. They've retrospectively validated it on thousands of patients from prior Phase 3 trials, and the FDA was fine with that. So one argument against doing interpretability at all is: why spend a ton of resources exploring something that the primary regulatory agency you care most about doesn't really mind, whether or not it's white-box or black-box?

I guess the obvious answer would be because presumably that knowledge would be a great input to further experimental ideas or other knowledge. Maybe you think it's just so hard—I don't know—there's no verifiable ground truth or something that would prevent that from working.

Yeah, I guess what was discovered in the previous Goodfire post—I forgot what exactly it was—but it was something about fragmentomics, something about how genes are fragmented, as a potential biomarker for Alzheimer's. It's a very interesting piece of work. It sounds very expensive to validate, and so I imagine we would run into the exact same problem: maybe we have a very good hypothesis for what comes out of the system, but we already have so many other hypotheses, potentially ones that even have higher literature backing.

I could imagine a world in which mechanistic interpretability as a field gets so good that you can triage this: this thing's going to be really easy to validate; this thing's going to be really hard to validate. But right now, the way that mechanistic interpretability usually works in biology is you're staring at semantic segmentation plots a lot and trying to think, “Is this real or is this fake? Is this the model identifying some very spurious correlation?” That time simply feels better spent elsewhere.

Interesting. Okay, here's another idea of a place that it might be well spent: continual learning, of course, a huge theme right now in AI in general. The first conversation we had today with Professor James Zou from Stanford included a little talk about their recent paper, Learning to Discover at Test Time, where they're using autoregressive large language models and giving them problems like, “Make a faster CUDA kernel,” or, “Find a better solution to this math problem with a lower bound than anybody has previously found.”

They interestingly flipped the usual model of what we're trying to do when we create an ML model on its head and said, “What if we just try to get this model to produce the single best answer that we can, and we don't care if it generalizes? In fact, we'll probably throw away this model after this test-time fine-tuning. What we want is the answer.” They were able to find, at a relatively reasonable cost—$500 in compute costs—some new state-of-the-art answers on some of these highly technical questions.

If I'm a cancer patient and you've got a general foundation model, a question that naturally occurs to me is: can you fine-tune this on my data? Can we do some test-time tuning? Can we do intensive masking on just my samples and really dial this thing in to understand my particular biology? If we did that, would it be more accurate for me? Do you think that line of thinking has legs, and why or why not?

So I actually looked at the paper, and they have a section for biology. They do single-cell RNA denoising using this test-time training model, which I thought was really interesting. I guess my instinctive answer is that it's an interesting idea. It very well might work, and it falls into the bucket of ideas that we would simply have to try to make sure that it does or does not work.

The results for single-cell RNA denoising in the James Shiao paper are certainly good. They're better than the state of the art, but for each one of these case studies, they attached a note by an actual domain expert saying how useful this is in practice. The domain expert in question for the single-cell RNA section did say, “This is very cool, but at the end of the day, we don't really care about the results of single-cell RNA denoising. We care about some biological utility that underlies that.”

Maybe you get better at solving this verifiable-task problem, but that doesn't translate to anything actually useful. Maybe it would be different for the response/non-response prediction case, but it just sounds easier to fine-tune the model using normal supervised learning. Why go through the RL process if the end result is binary? I think they even called out in the paper that the setup isn't really meant for binary or sparse-reward learning tasks. It's meant for fuzzier things.

Yeah, they're working on that, but it's not done yet. I guess maybe zoom out, and you kind of alluded earlier already to this idea that a lot of people think we just need better ideas for drug candidates. Your consistent position is that that's probably not really the bottleneck. And you made a really interesting point around how a more accurate ability to evaluate those candidates drives a lot more value than just throwing a lot more candidates through a pipeline.

The quality of the pipeline matters more than its scalability. Again, I think you've suggested where you think this can come from: large-scale, foundation-model-style training. But give us the next level of depth on that. Why are all these other ideas not so exciting? Is this basically just a Bitter Lesson sort of idea, where all your cleverness is going to be washed away by scale, and so you should keep your eyes on the prize? You've got to data-max and compute-max until you solve it all. Is it kind of that?

I guess I view things in 3 ideological camps. The first is maybe us: we index very heavily on human data being the only thing that matters. You can't start from in vitro settings and bootstrap your way up to something more complicated. You need to start with the most complicated thing to begin with.

The second camp is very interested in modeling single biomolecules and their interactions, in the hope that maybe you can't bootstrap your way upwards, but you can raise the absolute success rate from maybe 5% to 20%, and maybe that's all you need. I think the second camp defines the vast majority of ML bio companies that exist today. I think some of them have clinical candidates that are ongoing right now, and we'll see what the results are. Generally, it doesn't seem like there has been a massive step change in their ability to design drugs, and this isn't me knocking them. Drug discovery is hard; everything's a bet at the end of the day.

The third camp is that maybe it isn't really a for-profit thing, but you can just improve the clinical-trial process to begin with. This is arguably the path where China has its main advantage. They're able to run clinical trials far more cheaply than anyone else, partially because of the lower cost of human labor, but also because they've set up the system pretty nicely, such that it's not such a huge regulatory and financial headache to get things going. This has some downsides: drugs are treated as innocent until proven guilty, whereas the FDA is the other way around. But the obvious benefit of doing that is that you're betting neither on the AI and human data getting better nor on the AI-in-vitro-settings data getting better. You're trusting that the typical drug-design process, if made slightly more financially efficient, will improve things on its own.

I think all 3 of these are important, and it would probably be grandiose of me to assign an unequal weighting to any 1 of them. Each one feels important to push on.

I'm going to take a little bit of a segue to something you said earlier, which is that a lot of the new INDs are coming in from China. What has happened in the last couple of years? Is it an AI thing? The CEO of Ginkgo Bioworks was on TBPN yesterday, and he said they just have more hands. Some people believe it's a regulation thing; some people believe it's a clinical-trial-registration thing—that they can just register more people. Some people believe it's a U.S. cost thing. What is driving this transfer of basic R&D to China at this point?

I think this particular subject is very deep; it's not something I have expertise in. My instinctive thought is that there are many different answers to this, and the one that I think is most interesting is the idea that China was always a very good generics manufacturer, and that's where they started. Slowly, they extended their way into having a very good CRO ecosystem, and then, at some point, enough talent began to be incubated in China that they realized, “We have all this infrastructure here. Why not just develop our own drugs?”

There is something very important about having this close interplay between both the person who is designing the drug and the person who is actively doing wet-lab assays on the drug. Whereas in America, you have a super-long feedback loop: “I need to get my setup together. I need to reach out to VCs. I need to buy a lab.” Whereas in China, that ecosystem is a little bit set up already. Actually, maybe the only missing part is that the VCs are still not super—they're more risk-averse than perhaps VCs in America.

The co-location of the grunt work and the intellectual work is actually surprisingly important. A few months ago—actually, last year—I interviewed 1 of the very few people doing novel biotech research in India, a guy named Soham who runs a company called PopVax. He said this is the primary reason why he expects not only China to start producing really interesting drugs, but also potentially India and potentially Egypt—places where there is intellectual capital and a lot of hands. That combination leads to really good compounding results.

Indeed. Does that accelerate with the AI models, this kind of AI co-scientist? Does that mean that even if they don't have that much intellectual capacity yet, they can have the hands to carry it out?

I guess this is something that's a little bit opaque to almost everyone: how impressive are the bio-AI models coming out of China? I think there's certainly some interesting work that has been done. It's not clear to me that there's anything radically new there that won't be found anywhere else. A fair amount of it is scaling up stuff that was originally developed in either the U.K., London, or America.

There hasn't really been a DeepSeek thing where something radically crazy comes out of any of the Chinese labs. I obviously could be wrong on this, though. Whatever the bio-AI labs are doing in China, there's much less American visibility around it.

Okay, Nathan, go ahead.

One big question I have is that I find it very hard to calibrate myself on how excited I should be about all these AI-for-biology and AI-for-medicine developments. I know that there are always these headlines: AI does this, AI discovers this drug. I've done episodes of The Cognitive Revolution on it, including 1 with Jim Collins. He has created a bunch of antibiotic candidates. There's a long list, right? Professor Zou did the nanobodies thing that came out of the virtual lab. To hear him talk about it earlier today, it sounds like those were reasonably well validated.

But then you always get this other side, too, that's like, “Well, not so fast. It's all very messy. We've got a long way to go. Most of these things don't pan out.” I feel like that sort of parallels the debate that we hear in a lot of different domains. Even in programming, which is 1 of the more, let's say, legible domains, we've got something like a METR study that showed a slowdown of developers, and that was very confusing. I'm still quite confident that it's making me faster, and I kind of want to throw that away. Or, of course, there's just a lot of denial and cope out there and all sorts of motivated reasoning.

How should one try to ground their worldview? Obviously, subscribing to AI posting is something everyone should do, but what else would you advise me? How can I patch these blind spots in my worldview or get to a better position from which to have my own sense of what really counts, what really matters, and what doesn't? Again, this happens all over the place, where there's disagreement even among some of the most informed people about just how big AI reasoning can be, or how big of a deal it's going to be.

But in biology, it's particularly hard for me to make sense of. I'd love to get some tips for how to climb the learning curve faster.

I've actually written about this in the past, a very long time ago. The title of the article is “Five Things to Keep in Mind When Reading Biology ML Papers.” The long and short of it is that the evaluations in biology are very difficult. You see a similar phenomenon in more typical wet-lab biology: “We cured cancer, but it was in a mouse, so who knows when it'll actually translate to humans?”

There's a very similar phenomenon in a lot of BioML papers, where they're doing something that feels like it should be useful, but there are a lot of things they're probably hiding from you when explaining the results that would only be obvious to a domain expert. One really funny example of this is small-molecule binding-affinity papers. I've written about one company's work in this area, but they found that, let's say, you're able to predict that this set of molecules binds to a target and this other set of molecules does not bind to the target. You're very happy with yourself and publish a Nature article about it.

What the folks at a company called Lee Bio found is that this can often be confounded by which chemists actually produced the molecule in the first place. Some chemists are very attached to specific targets. They're very good chemists, so they often produce things that bind to that specific target. These molecules, importantly, all look very similar to each other. It is this type of similarity that's very human, vibes-based, and hard to pin down to a singular metric. They found that these models are often confounded by author overlap.

These problems appear over and over again across in vitro biology and biomolecule generation, where you can be confounded by variables that you didn't even know existed in the dataset. I would probably name that as the thing to be most aware of when reading these papers.

There are a few people I trust on Twitter and in real life who can give a pretty good overview of any arbitrary paper. With LLMs, popular-science people often retweet them and say, “This is transformative,” and more often than not, they're probably correct. Opus 4.6 is genuinely crazy. But when people do that for BioML papers, there's a 50–50 chance that they're completely missing the point because they're not in that field and don't understand how the failure modes emerge in these models.

Yeah. Do you think that an Opus can help me identify those blind spots?

Sorry. Go ahead.

Yeah. Is it good enough to do that?

I've actually written an article about this also. It's titled “Can o1-preview Find Mistakes Amongst 56 ML4SB Papers?” ML4SB is a structural-biology workshop at NeurIPS, and it's not very good at it. This was obviously the last generation of models, so maybe it would be a lot better now. But there are problems that are going to recur in almost every biology ML paper, such as, “Your training sizes aren't large enough,” or, “Your test sizes aren't stratified correctly.” You kind of just learn to pick your battles in this field and move on.

There are a lot of more fundamental problems with these papers that LLMs, in my experience, often miss entirely. In almost every article I've written, I've found that LLMs tell me something about a particular subfield that the domain experts completely disagree with. They say, “That's not how you should think about this domain. That's not the real problem we were actually worried about.” I don't know why this is the case. It's kind of fun. It's like a domain of science that LLMs still haven't quite captured human taste.

Yeah, fascinating. Okay, that leaves a lot of work in front of us. Do you want to go back briefly, before we break, to Noetik again?

Fortunately, my son is doing well. He recently got cancer 3 months ago. I've had an intensive crash course in cancer, and I hope to be able to close the book on it and return it to a more intellectual and less personal interest going forward. I think we're on a good, solid track to do that.

But I think you've demonstrated in this conversation that you're not getting too carried away with the promise of what AI systems can do. We've got the data center of geniuses. We've got the century of progress compressed into 5 years kind of visions. How much would you shave off those notions, just to describe your own expectations of what Noetik can do specifically and maybe what the field more broadly is going to be able to accomplish?

I think I'm very optimistic that human-simulation companies akin to Noetik, and other players out there as well, will be able to vastly improve the results of at least a few clinical trials within the next few years. That feels almost like you're not even paying too much attention to the trend lines. I'm almost indexing on what we're capable of today.

There are papers going back years that are able to show, “We've developed an ML model that is better able to stratify patients.” The problem has always been an economical one: how do you actually deploy this in a real setting? I think we'll be able to do that just fine. I think the failure of Phase 1 drugs will go down, and I think this has already been slightly proven in a McKinsey study that was done 5 years ago, which showed that AI-designed drugs have a 5–10% lower failure rate. Maybe that's noise; maybe it's real. I do expect those trend lines to continue a little bit.

Where I'm most unsure is whether these models will be able to discover brand-new targets entirely, which is ultimately what people care about. I think believing that these models will be able to find new targets far faster than humans would really require you to index heavily on the trend lines. I'm among those who index on the trend lines, so I expect that these models will be able to deliver very good target finding.

But I'm also very sympathetic to the mindset that finding targets is such an unbelievably hard problem that the models will not make a dent in it, because you need this human-iteration feedback loop. Unless you build a really good human simulator, which is our bet, you're not going to get close to solving that problem.

The way I put it is, usually you can see about 1 order of magnitude ahead, maybe 2. No one can see 3 orders of magnitude ahead. It's just not possible. You have no idea what's going to happen.

Abhi, thank you so much. I learned a lot from this and hope to see you online. I hope to read more of your blog.

Yeah, absolutely. Thanks for having me on.

Thanks for being here. We'll be working our way through all posting archives for some time to come. Bye-bye.

Our next guest is Helen Toner, who runs CISA at Georgetown and is a former OpenAI board member. There are 2 competing views here. On the one hand, the intelligence explosion is coming. On the other hand, AI capabilities may be permanently jagged. So let's add her to the stage. Helen, nice to have you.

Hey, thanks for bringing me in at the end of your marathon. I'm impressed you guys are still going strong. There's so much to cover, and we've all got to accelerate our personal productivity timelines and try to pack more information into the same amount of time, so I'm experimenting with ways to do that.

I should talk super fast.

Yes, please. That's honestly one of my reservations about live content. I listen to everything at 2x speed, and I'm like, “I can't listen to it at 2x speed if it's live.”

My constant struggle is to talk slower than I naturally want to. If you want me to talk double speed, I'm here for it.

Please go as fast as you want. Go for it.

Okay. You guys just put out this report. I think this is obviously a great candidate, if not a shoo-in, for the most important question of our moment. What is going on with the possibility of automated AI R&D? Do we have this tipping point where we're starting to hit recursive self-improvement? If so, how big of a deal is that going to be?

You guys brought together a bunch of people who authored this report and some others as well who aren't necessarily authors but contributed to the conversations. I understand that quite a few people from frontier-model developers were involved.

It strikes me that this debate goes back basically to the beginning of AI. There was the idea very early on that we could have an intelligence explosion. When I started reading Eliezer Yudkowsky in 2007, he was very worried about this. Yet you've written—I think you put your finger on something a lot of people were feeling last year—when you said that even though what passes now for long timelines is pretty short, the disagreement on this topic seems to be as fundamental and as impervious to new evidence as it has ever been.

So maybe, just for starters, take us inside the workshop. Give us the lay of the land in terms of what world models people have.

Why are we still working from so much intuition, despite the fact that we now have what, in some circles, would even be called AGI out there as products for us to use today?

Yeah. This workshop was held in July last year and was maybe one of my work highlights of the year. It was a day and a half. We brought people in from some of the frontier companies and policy circles, along with a bunch of great people, to get a sense of what the vibe was like.

The first session was about how AI is being used to automate AI R&D right now. We had presentations from people who are doing that. Before the first break, we had Ryan Greenblatt from Redwood Research, Nicholas Carlini from Anthropic, Dash Kapoor from Princeton on AI as normal technology, and Thomas Larsen, who’s one of the AI 2027 authors. They were arguing so fiercely, in a friendly and productive way, that before the first break, everyone else stood up to go get coffee, drinks, and snacks, and they just kept arguing right through the break. It was great; it was exactly what we were looking for.

I think that prefaced something that we knew going in, which was that there are really different perspectives here. The workshop was under Chatham House Rule, so I feel okay giving that anecdote because they ended up writing a post about it. One thing that came out of that was Nicholas constantly pushing the others: “Okay, you have such different views about where things are going. Where’s the first place that you actually disagree about what we’ll see?”

As they looked out at what we’re going to see in 2026 and 2027, they actually agreed a lot about what we’re going to see before we get to that recursive point, which is kind of a bummer. It’s nice that they agree and were able to post about that, but it sucks that this means it’s actually going to be hard to identify in advance whether we are heading into a recursive loop or whether we’re not.

There were 2 big things we were trying to do with the workshop. One was to get this idea of recursive self-improvement out of purely Silicon Valley, San Francisco AI spaces, explain it, present it to a wider audience, and let people engage with it. The other was to actually try to make some progress on why people disagree about this, what is happening, what might happen in the future, and what indicators we could gather.

I came out of it thinking that maybe there are 2 core disagreements here. One is whether AI truly replaces all of what humans can do. Do you get to a fully automated system? Because if you’re going to have the really scary recursion, that’s probably what you need.

You could have much more productive human researchers. You could have the Alec Radfords and the Ilia Skekers managing fleets of AI researchers, but if it all has to come back to them and they have to process, digest, and think through the research, you’re not going to get that massive recursive loop. So one piece is whether you truly get humans being fully replaced. If not, then maybe you have some parts of the workflow being really accelerated.

We had a diagram in there of an Amdahl’s law kind of thing. Amdahl’s law is basically that if you have a process that depends on different inputs and there are different potential bottlenecks, then if you speed up one part of the process, the bottlenecks will just bite somewhere else. It may be that you speed up the coding part of AI research, but if you don’t speed up other parts, you don’t end up speeding up the whole thing very much.

Another mental model that people who are skeptical that this is going to really go crazy bring is that we have a long history of computers doing more and more of the lower-level work. We don’t have to do punch cards anymore, we don’t have to write assembly code, and we have higher-level languages. For example, AI doing more of the coding is just another natural step in that process.

Humans have this expanding-pie model: the number of tasks that we realize can be involved in AI R&D expands, and there’s always that outer band that humans can do while they’re automating the inner bands. I think that is very different from the view that the AI 2027 authors and many other people in the space would have, which is: first, you automate some of what humans can do; then you automate all of what humans can do; and then you keep going until some other bottleneck hits.

So the other question is, what are those bottlenecks? We can talk about that as well, but I think those are 2 of the biggest questions that came out for me. One was whether you’re truly going to automate everything, including what all the humans can do. The other is, if you do, how soon do the bottlenecks bite?

Scholto Douglas, who is now at Anthropic, had this idea of a software-only singularity. We get very good at coding and all of the digital stuff, including AI research, I presume, but not at producing power, copper, or all of the physical substrates that are going to be required to support this expansion. How do you think that fits in? What if the digital stuff happens but the physical stuff just doesn’t?

Yeah, I think there are 2 versions of this. When some people talk about a software-only singularity, they basically mean that it turns out software is enough to get absolutely crazy recursive loops. Tom Davidson at Forethought Institute has written about this, for example. Maybe you can get massively more intelligent systems having massive impacts on the world, primarily through software improvements.

There’s a different thing, which is what you’re describing. Scholto’s version is more like a jagged software-only intelligence explosion, meaning the AI is getting much more capable in certain ways, but its effects on the world are very limited because it is software-only.

This gets at another thing that I found really helpful and interesting from the workshop: people have very different intuitions about what it means for an AI to be very good at AI R&D. What does that mean for what the AI can do elsewhere?

I think some people are like, “Okay, well, if it’s very good at AI R&D, then it can train AI models to do whatever, so it can do whatever. Maybe you have to spend a week gathering data or something, but then if you want to do some arbitrary task, you could do it.”

Whereas I think other people have an intuition of, “Okay, well, even if it gets very, very good at automating AI R&D, this sort of most software-based task, it’s still going to really struggle to, for example, design new biological molecules. Or it’s going to struggle to think about geopolitical strategy questions, because you have to actually go out and see how different countries and decision-makers will react to things like that.”

That is a piece that I feel goes underexplored in a lot of these conversations: what is the connection between AI that can do incredibly good AI R&D and AI that can affect the world in non-AI-R&D-specific ways? We also tried to tease that apart a little bit.

Would you think that’s the connection between, okay, now you have AI doing AI research that’s affecting the economy, it’s also affecting the political economy, and then you have to have mitigations for the political economy for this to work out? Does that mean you might need the AI research to go into how to fix the political economy, which is going to be a little bit scary?

Yes. Say more about what you mean by “affecting the political economy.”

In the sense that, for example, right now you have Bernie Sanders saying that we should have a moratorium because he’s scared about jobs. He’s very scared about jobs. He wants a moratorium on data centers. I think there are 6 states with a moratorium now, including New York State.

One path from AI research could be figuring out how AI fixes the political economy: how we deal with humans and mitigate the impact we have on them. Is that something you think would happen with the first configuration of the software-only singularity, in the sense that it’s not jagged and also affects the political economy that way?

Yeah, that’s the kind of thing that follows. If you’re positing that you can have a software-only singularity that is going to radically transform the world, then it’s going to have to be able to do things like, “Okay, and then the company deploys chatbots that talk to enough people to convince them that data centers are great. The data centers all get built, and the moratoriums get rolled back.”

That kind of thing has to be built in, which to me intuitively feels like a different skill set. It’s also more dependent on deployment, rollout, and adoption.

Yeah. I tend to be a little more skeptical there, but I think that’s an example for sure.

I see.

One kind of odd pairing of beliefs that I observe, and sort of detect in the report, is the idea that among the more skeptical folks, there’s going to be a plateau and that plateau is going to be subhuman. And then, on the other hand, it’s like it’s not going to plateau; it’s just going to run away and have some sort of singularity.

A position that I feel pretty intuitively attracted to, and that I don’t hear too often, is the idea that maybe there will be a plateau, but it could very easily be a superhuman plateau. If I try to zoom out as far as I possibly can and look at life on Earth, I would say it seems like humans are part of maybe an entry into a steep part of an intelligence explosion, or an S-curve of capability.

I don’t think we’re the end of history, but we were clearly better than what came before, and that was enough to take over the world. I don’t hear too many people say, “Yeah, it’s not necessarily going to be a singularity. It’s not necessarily going to go totally beyond comprehension. But in the same way that we were just that much better than Neanderthals—and it might not have been that much, but it was enough to change everything—I kind of feel like there’s not too much more room between where the AIs are now and where they will soon presumably be.”

Even if that doesn’t go critical from there, it feels like it’s enough to be transformative. Was that a position represented in the workshop? And how do you personally react to it?

Yeah. I think that sounds pretty close to my default expectation, maybe. If so, then it was represented there because I was there. Maybe to rip on it a little bit, something we didn’t put in the report but that I’ve definitely found helpful for my own thinking is to consider the fact that we clearly have some kind of S-curve.

We have humans situated in the middle of an ongoing S-curve, but we also have some kind of S-curve of AI capabilities. There are 3 segments that are of interest. One is how long the lead-up period is, the first part of the S. One is how steep the middle of the S is. And one is how high the ceiling is.

I think a lot of the time, when you’re hearing people talk about automated AI R&D, they’re in one of 2 camps on all 3 of those questions. Either they think the lead-up is short, the curve is steep, and the ceiling is high, or they think the lead-up is long, the curve is gradual, and the ceiling is low.

I also think it’s really interesting to consider different combinations of those parameters. To me, it feels like the lead-up is pretty short these days. We’re not too far from that takeoff period. But what if the curve is steep and the ceiling is low, or the curve is gradual and the ceiling is high? We don’t talk that much about either of those.

Supporting modalities. Yeah, in previous parts of this marathon conversation—this series of conversations—we’ve seen how the ability to interpret the signals that people are throwing off in sleep to predict disease—

—is just a really random but, I think, instructive example of how there’s obviously a lot of room to be superhuman at some of these tasks—

—and there’s potentially a lot of power to unlock, especially if you can integrate that kind of multimodal grokking with a basic reasoner. I really don’t see any reason that we’re not going to be able to achieve that.

Yeah. Often those things, though, will involve another piece that’s underexploited here. People tend to either be in the camp that the ceiling is high and you’re not going to need all that—it’s not going to be delayed by real-world adoption—or the ceiling is low and it’s going to be delayed by real-world adoption.

To me, I’m sort of like, isn’t the obvious combination of these that once you get the real-world integration—for example, you have to collect all that sleep data, or humans are really bad at interpreting scent data, whereas dogs can smell things we can’t—you have to add a bunch of sensors and all that. I also feel like there are unexplored questions around—

—how high that ceiling is as you have increasingly integrated AI into more and more aspects of life and the economy.

I also wonder to what extent, because in my view things might happen as software and mathematics first, you may get things like, “I don’t need lidar for my self-driving car anymore. I can use cameras, and the cameras can be really bad cameras now because the math does all the work.” I don’t need all this sophisticated technology.

It could be that your phone could do what those sleep-detection machines do with the right software package. Your phone has a lot of sensor tech; there’s an enormous amount of technology within it.

Yeah, and you do wonder whether it would really be an application of algorithms to existing frameworks and existing infrastructure, increasing the bandwidth of your communications technology with new encryption and new cryptography. That’s how DSL was invented, for example. DSL was really using the existing copper pipes with new algorithms.

I wondered to what extent you don’t get a slowdown just because of your physical infrastructure, because you innovate around or with your physical infrastructure.

Yeah, I’m sure that will work in some places. I think it’ll work in some places and won’t work in others. If we’re talking about cybersecurity for critical infrastructure, where my mind goes is that the physical systems are old. They’re hooked up to old operational technology, and they’re hooked up to old information technology because they have to be. There’s going to be a limited amount that you can optimize using smart new algorithms there because the stuff is old.

Likewise, my center does a lot of work with military technology. Same thing there. If you have a ship that was built in the 1960s, it’s a ship that was built in the 1960s, or other pieces of equipment. So, yes, I think in some places yes, and in some places no.

To me, that’s another place where the jaggedness bites. I think Abhi mentioned it as I came on, in the talk I gave on jaggedness. My default expectation in AI R&D is that we’ll see jaggedness. The jaggedness is fractal, right? You zoom into the task of AI R&D or the skill of AI R&D, and actually it’s many, many different things.

We’ll see AI R&D accelerating in areas that are especially amenable to using AI and lagging more in other areas. That’s not to say they can’t ultimately be automated, but it will take longer.

How far behind the product that’s on the market right now do you think the products people are using inside the labs are?

I don’t know.

Okay.

I honestly don’t know. That was one of the most actionable sections of the report. We have a set of indicators and a table summarizing the 3 categories of indicators that we have. The biggest category is indicators from inside companies, and one of them is the public-private gap.

My sense is that it’s not huge right now, but I don’t have any inside information. You guys talk to company employees as well.

You do believe, he says, that we have no idea how good we have it and that the gap is very small.

Exactly. I’m thinking of things like that eval suite.

So, what I learned in the last few days is that the real gap is that they’re using models that are 3 times faster. It’s just the same model. They’re running it at a lower batch size, and it’s 3 times faster. That’s what they’re using internally. It’s the same tokens; it’s just a lot faster.

And there’s surely also tooling stuff, right? Something we put in the report—when a couple of our reviewers who were looking at this were less familiar with the idea of automating AI R&D, some of them said, “Haven’t you seen that study that 95% of AI pilots fail?” And there’s the METR study showing that AI slows people down.

We included an explicit note in the report that, yes, productivity boosts from AI are mixed, but these AI researchers are in the very best position to benefit from their technology.

They are the best up to speed on what it can do and what it cannot do. They are shaping how it is developed and what directions it is pushed in. They are in the perfect setting to build tooling that squeezes the most use out of these models, so I am sure that is also a piece of it.

One of the things you mentioned early on, just a few minutes ago, is that you wanted to bring awareness of these possibilities outside the places where they are most often discussed. One other thing I would love to hear your perspective on is how ideological you think companies are about this. This is one of the things that confuses me: every frontier-lab leader has read their Eliezer catechism. Many of them have previously said that we should be extremely careful about this sort of thing and should not engage in an arms-race dynamic. It is obviously part of the OpenAI charter, and Dario has said things like this. Now we are in a world where there is a publicly stated timeline from OpenAI to an AI R&D intern and then, not much longer out, in 2028, the full AI R&D researcher.

I mean, so are Anthropic as well as xAI.

Yeah, I would say Anthropic seems even more committed to it, or more resigned, maybe, but they believe it.

Jack Clark was saying June or summer this year. Jimmy Ba, who just left xAI, was a co-founder there and gave a timeline of 12 months. OpenAI said this year: research intern, and then full researcher kind of a year later.

Yeah.

I think it is this year. That is my guess.

This year for what specifically?

The start of recursive self-improvement.

Oh, but are we not there already? Was it not last year? You had Gemini doing the evolutionary-algorithm stuff, where it designed an algorithm that sped up its own training by 1%. Come on, that is recursive. It is really—

This is what I am talking about: the lead-up to that loop.

Yeah. So, do you think we might be at the point this year where there is no human needed whatsoever? I think that is a high bar.

I think we might be. I updated my view on Moltbook. The Moltbook thing took me by surprise: 1.5 million agents all of a sudden on the web. It is all nonsense, for sure, but things start off as nonsense. I think what might happen is that you get a single model update that fixes a little bit of hallucination and a little bit of the security issues around leaking secrets, and I think that might be enough.

Sounds hard.

Sounds really hard, fixing security stuff.

Yeah, we will see. Maybe. So, I do want to give you the chance to talk about the dynamics here. There are different reads we might put on people: they are ideological about it. Elon Musk has said things like, “I do not know if this is going to be good or bad, but I want to be around to see it,” and, “I would rather be part of it than a spectator.” That sounds like somebody who is inclined to gamble with humanity in a pretty self-aware way. Others may feel trapped in these dynamics and at least will do it as safely as possible. How would you describe that milieu right now? I think it is dramatically underappreciated by people outside the AI bubble where we spend all our time.

My impression of it, my sense of it from the people I talk to, is that there is just a sense of inevitability about AI advancing and a desire to be part of the future being created because they see this as a future that is being created.

You mentioned how this has been part of the AI conversation since the very beginning. I. J. Good was talking in the early 1960s about creating the first ultraintelligent machine. I feel like we always need more terminology in AI. I feel like we should get “ultraintelligent” to make a comeback. There is this very natural logic, if you have a computer-science kind of brain: we have some level of skill at building computers, and when the computers have more skill than we do, they will build ones that have more skill than that. Then you get a loop. That logic is very appealing and seems very natural, so people think it is going to happen anyway and they may as well be involved. That is not everyone, but I do get the sense that is the water most folks are swimming in. If you have a different view, then it is in contrast to that. Is that your sense as well?

Yeah, I think so. I think the inevitability is a pretty compelling argument. I resist it because I want to make the point that even if some form of this is inevitable, there is still probably important discretion that we can exercise in terms of exactly what flavor it takes. There are questions like whether we should keep chain-of-thought interpretable or embrace thinking in latent space. I do think it is important to keep in mind that it is probably not all one or all the other. AI defies all binaries. There are going to be these gradations and these more local decision points.

In 2022, I was just trying to make AI work for practical tasks. With no background in AI research, I basically ended up independently inventing a number of the techniques that have gone on to produce great things. I did not take them past any local plateaus, but just having AIs improve their own outputs—proto-Constitutional AI-type stuff—I do think the attractor, the sort of gravity well, is pretty strong. It is hard to avoid some version of these techniques because if even a bozo like me lands on them, I do not know how they are not going to happen in the broader world, especially as we start to get a dramatic democratization of training techniques.

Prime Intellect just put something out that allows anybody to spin up their own reinforcement-learning environment on a distributed, community basis. Everything is going to get tried, and I think that is pretty hard to argue against.

But again, I do want people to still own exactly what they are doing along the way.

Yeah, I think there is something in here that takes me back to long-running conversations about autonomous weapons. There is something about the level of human oversight that you can have. I totally agree that using AI to accelerate research is an attractor, but you would really hope there is a meaningful difference between having a fleet of 10 million AI agents running experiments for me while I am leading and guiding them, versus setting something into motion and having no clue what is going on.

I think there is a boundary somewhere. Is it a boundary that we are able to stay on one side of? I am not sure, but I hope it might be. That feels like the point to try to intervene—not that we should not use AI for research, because that is obviously not going to work.

To what extent do you think policymakers are naive? Earlier on, we spoke to Sam Hammond, who advises some policymakers on AI. He was talking about privacy and the restrictions, constraints, or regulations we could put in place. One thing that struck me was that a lot of policymakers may not be aware that AI with access to existing technology, persistent search, and persistent memory would basically do a Google stalking of you before it even met you. It would know all of those things in the public domain.

The amount of access to information it could have, the persistence of information, and listening in on conversations are going to be very powerful in that sense. You can ban people or ban AI from using facial recognition, but then you have network analysis. You can do metadata analysis on WhatsApp conversations—where the messages are going—without needing to know the content.

There are many techniques by which you can deanonymize traffic and people. You do not need facial recognition; you can still do gait analysis, speech analysis, voice analysis, and handwriting analysis. There are so many other techniques, and all of these things will be available to AI. To what extent is this whole “we are going to make sure we have privacy” thing naive? Is it going to be possible?

I think the US has done a worse job of this than pretty much every other country on the planet. I think there is a basic rule here. I do not think you want rules at the level of “no facial recognition.” I think you want rules at the level of “no data brokers.” You can collect data, but if you are going to collect it, the user needs to know, and they need to have notice and consent.

I am not deep on privacy law, so I do not want to pretend that I have the right, great privacy proposal here. But I do think there are ways to do it that are better than the US, and ways to do it that give you that underlying flexibility. Maybe I will leave it at that, because privacy law goes real deep and I am not there.

One more question for you: in the report, you talk about the possibility that the gap we think is currently small between the models we have and the models that are used internally could open up, and you have some recommendations around certain transparency measures.

Where do you think we are on the spectrum from secret nondisparagement clauses to where we need to be in terms of insight into what is going on at the labs, other than private, philanthropist-funded whistleblower support? What other policies do you think the government should pursue? And maybe, even more broadly, if you want to zoom out, what do you think a situationally aware US government should be doing in general that it is currently not?

Yeah, I think there are a bunch of things here. On transparency, I think we're doing better than we have been. We have these 2 new state laws, SB 53 in California and RAISE in New York. I think those are good starts. But for a lot of this information, we're also just really dependent on what the companies still choose to put out.

Now, we're fortunate. I want to give credit to both OpenAI and Anthropic, and to a somewhat lesser extent, Google. They do put out, pretty proactively, a pretty good amount of information. So I think they should get some credit for that, but I don't love that it's almost entirely at their discretion what it is that they put out. I guess that will be shifting as SB 53 and RAISE start to be enforced. I'm interested to see what that looks like.

I think there's been a start, the beginnings of a push, to shift from a model-release-based schedule to something more continuous, which is partly driven by interest in these internal-deployment-type dynamics, not just the external releases. The idea here is that if the risk isn't actually purely tied to when you put your model on the market, then all of your risk evaluation shouldn't be tied to that either. It's also about creating better incentives for the companies—not forcing them to just rush things out the door, but instead trying to have more of a continuous pulse of updating metrics over time.

So I think we could definitely be doing better on transparency. Ideally, we'd pair those requirements with some kind of independent-audit requirement, or an independent way to let external third parties come in and check that things are happening as they're supposed to be happening. That has been in several of these proposals and keeps getting stripped out by industry lobbying. So that, I think, is a new frontier as well.

I think there are various other policy implications that we put in the report, some that are maybe interesting. One is just this general recommendation to harden the world, or societal resilience is another way of putting this. This is cyber defense, biodefense, biosurveillance—investing in biosurveillance, just meaning monitoring diseases, not surveilling people; investing in epistemic security stuff, trying to have a way to determine what's real and what's fake; tagging real content—all this broader societal resilience stuff. It's like, okay, just assume that this is going to get much, much, much better and that we might see automated R&D contributing to an increased pace of change.

I think there's also been a shift over the past year or 2—this is less of a policy and more of a mindset—to, actually, maybe open models are always going to be pretty close behind. So concerns that you might have about there being an access gap, or a concentration-of-power gap, if the closed models are far ahead—maybe we don't have to worry so much about that. And I think if you're taking seriously the possibility that automating R&D speeds up the closed labs significantly, then we just need to revisit those assumptions about open models and closed models. There are a few others, but I would point people to the full “When AI Builds AI” report for the rest.

When AI builds AI, things just might start to get weird. Interesting times, for better or worse. Any closing thoughts before we break?

No. Great to be on. Great to chat with you, as always, and I look forward to next time.

Indeed. Cool. Always very nice to shorten our timeline between now and next time.

See you. Cheers. Bye for now.

So, our next guest is Jeremie Harris. He's from Gladstone AI, and they wrote the first-ever US government AI threat assessment for the State Department. It's been about 10 months now since they said every American AI data center is compromised. Jeremie, what has changed? Have things gotten better or worse?

Yeah. Well, to piggyback off what I think Nathan just said, things are getting weird. So things are weird. Great to be on. What has changed since then is less than one might have hoped, and for really interesting reasons.

I think a big part of the story that's missing for a lot of people who are concerned about the AI risk story and the AI threat landscape from a national security perspective—whether it's loss of control or weaponization—is an understanding of the infrastructure build-out. What are the actual bones that we're building on here? Because that's the substrate that underlies everything, and there are all kinds of assumptions being made about it where we're abstracting away what I really think is at least 50% of the problem here.

We think a lot about model-reconstruction attacks and all kinds of interesting debates about whether it even makes sense to secure models in a world where you can just reconstruct them if an API is available. But more fundamentally, when you're building your entire AI industrial base off components that are made in China, with personnel who are often Chinese nationals—I mean, this isn't even—forget about the Manhattan Project. We're so far behind that.

I think it's incumbent on us to take a step back and just ask: What is that chessboard, even? What is the board itself? Forget about the pieces, but are we playing on something that's fundamentally stacked in a way that doesn't allow for a winnable outcome? And I'm not saying this to be pessimistic. I think there are actually solutions that you come up with very quickly once you take that new perspective.

But closing your eyes and not looking at it doesn't address the problem. I think we're in a space where we're doing a lot of algorithmic-level thinking because that's what so much of the Western economy is now based on. We're people at keyboards, and we're not making T-shirts anymore. We're not filming Transformers anymore. We're not doing that stuff. So we tend to like to pretend that it doesn't exist.

That's kind of my more recent lens on the problem over the last 2 years. I know it's not quite an answer to your question, but that's kind of the chessboard as I see it, at least.

When you look at it end to end, you have the software piece and the talent piece. 50% of top AI researchers are Chinese nationals, and that includes people working at the frontier labs in the US right now. Then you have the infrastructure piece. A lot of stuff is coming from Taiwan and South Korea. Some of it is coming from China, too.

You have ASML sitting in Holland, which is supplying TSMC, and then you have ASML's suppliers. They have 3,000-odd suppliers spread across the world. They're buying, I think, neon gas from Ukraine. When Ukraine got invaded, they had a problem. All of these pieces are spread out across the place, right?

And TSMC has been upfront by saying, “We are only possible in a safe, globalized economy. If we ever got invaded, everything's over.” We can't do anything. That's it. So where do you think—how do you think that fits in with a threat perspective? It seems like someone just has a dead man's switch over TSMC. So how does that work in terms of security and securing US prospects and the future in the US?

Yeah, I think it's a great question. This whole Taiwanese scenario-planning thing is something that everybody has talked about. I'm not so sure everybody has worked out the implications to full satisfaction.

I mean, first of all, yes: if Taiwan gets invaded, TSMC is gone. It's gone. Whether it's because China takes it or because it's, as I would expect and hope, booby-trapped to the nines to blow, right?

I mean, it takes hundreds or thousands of insane-level PhDs to tweak. You can think of it as a giant box of 500 dials, each one of which has to be perfectly tuned to keep these things pumping out at the right yields. You're not going to replicate that if you're missing either the equipment or the people. So this is extremely fragile, maybe the most fragile production process the primates on this planet perform.

An invasion is unlikely to leave it in China's hands. And so, yeah, the question is then: What do you get when you roll that back? What's the number-two-positioned entity? And then you start thinking, okay, well, what does SMIC do? What can it do? The SMIC–Huawei complex does seem like a very plausible runner-up, especially when you look at scale production, especially when you look at the emphasis Huawei has placed on networking large numbers of GPUs together.

They don't have to be as efficient as ours. They can't be; they don't have the litho, but they can be networked together way better, and that's how they get effectively competitive-scale performance. So this is a real issue.

In a funny way, this interacts somewhat positively with the energy bottleneck that we have here anyway. We're going to be bottlenecked by energy probably sometime around the end of the year. When that happens, TSMC's ability to outproduce gets complicated because, on a per-chip basis, they're way more energy-efficient. They're pumping out more flux, but we do have that energy ceiling on our side; that's the main constraint that we're moving toward.

The timing matters a lot here. There's a dance between how much logic matters, how much energy matters, and how much memory matters. How much does packaging matter? All 4 of those things have become bottlenecks at different parts of the game over the last few years.

Another piece, when we think about the actual bones that the AI economy runs on, is that it's not just chips and not just the data centers themselves. The power grid is generally a really vulnerable target. We know, for example, that there have been components in Chinese transformers that have been inserted as explicit Trojans, enabling them to take down our equipment.

A very plausible scenario, based on talking to folks who are working on this problem on the intelligence community side, is that an invasion of Taiwan begins and one of the first things China considers doing is simply shutting down the Western grid. It's kind of obvious if it's existential. That's massively escalatory, and there are huge question marks there, but it's a scenario that's being taken very seriously for all the reasons you might imagine.

So, yeah, I think when—or if—that happens, there are questions that suddenly run much deeper than just our ability to literally make chips in Arizona or wherever the next thing is. If we can be kneecapped economically at a more fundamental level, we don't even get to look at the chessboard that we hope to look at. We don't even get to indulge in, “What can Samsung do versus what can SMIC do versus CX&T?” We don't get to play that game. We literally don't have an economy. There are serious implications there.

If we think about this as a game with the stakes that it might have—and this is contingent on what's between Xi Jinping's ears and the Politburo's ears—this could end up looking like we're preparing ourselves to take a punch in the face, but then we get kicked in the balls, if you will. This is the kind of scenario that we may be averse to, and again, that zoom-out is really important. We've got target fixation here on what could be a pretty narrow part of the chessboard.

You had some ideas on not only needing to speed up, but also needing to slow China down. What was your concept around slowing China down? They are trying their best, but they're definitely not there on the chips yet. The Huawei Ascend 910s—Tencent doesn't really like them. They want to get the H100s in there.

There's this concept of building on the U.S. AI stack, but it's also revenue denial. If you manage to funnel the revenue into NVIDIA rather than into Huawei, Huawei has more revenue to develop those chips. Therefore, we should deny them. How does this balance out—letting them get the chips, but not chips that are too powerful, while still giving them enough that it doesn't create a market for Huawei? It sounds like a very delicate balance here.

It does sound like a very delicate balance. Personally, I'm less oriented toward the argument that says, “If we just let NVIDIA do business in China, then the Chinese will go, ‘Oh, sweet. We have NVIDIA serving our needs. We don't have to push so hard on the gas on this issue that's been identified for years as possibly the number 1 national technological priority that we're pouring multiple Apollo-moon-landing-like amounts of cash into.’”

This is, to me, a miscalibrated sense of even the messaging that the CCP has been putting out. I just don't see a world in which NVIDIA can ship the H200, or whatever it is now, and suddenly the CCP goes, “Okay, forget about that quarter-trillion-dollar investment, in PPP terms, into our national AI chip capacity and infrastructure. We'll sit with the NVIDIA play.”

There's a sense both that the ability to access these NVIDIA chips is transient, because the next administration may just as easily pull it down, but also that you have to ask, “Why not both?” It seems like an insane thing, given that AI is a matter of national-security importance for China. It would be pretty surprising to me if they just decided to respond that way. Indeed, it seems like they haven't so far.

I guess that's why my biased take on the export-control issue, from a slowdown standpoint, is that they have worked. We know from DeepSeek's public statements before DeepSeek was on the radar—and this is really worth noting and underrecognized—that they were coming out and saying, “We really think we could do this AGI thing. The only issue is, there's just one problem: We can't get chips, and these export controls are killing us.”

Then DeepSeek-R1 drops, and everything becomes about DeepSeek. They get dragged in front of the Politburo and debriefed, and suddenly things change. Every once in a while, you get these little trickles, these little leaks of similar information, coming out of the edges of the Chinese AI ecosystem. But it's pretty clear that the export controls were working.

If nothing else, look at the massive orders that are going to be coming in for H200s to show how much pent-up demand there actually is in the AI ecosystem. Of course, we know all about the frustrations of AI companies in China and the current wait time for chipsets. So, yeah, that's my biased take: I think we've got to listen to Chinese companies when they tell us that our export-control policy is working.

Maybe I'll come back to some of the frustrating duality of these difficulties. On the one hand, you have expressed very low hope for the opportunity or possibility of meaningful, true collaboration between the West and China. At the same time, I think you're also not super optimistic about our ability to create a superintelligence that we can actually control and get to do what we want it to do.

The way I think about our conversation from a year ago or so, and your contribution to the broader discourse with America's Superintelligence Project, is that those 2 things are both real and both true. You're engaging in motivated reasoning if you try to deny either one of them.

With that in mind, we're now seeing some potentially foreshadowing moments on the AI side itself. In just the last week, we've had these new models from Anthropic and OpenAI, and they've both said, “We weren't really able to run the evals as we intended to.” Anthropic basically said that eval awareness is pretty high, and so they would just do a little internal survey of whether or not this was safe to release. That's probably a bit of a simplification on my part, but I think it's a fair enough summary of their position.

Then OpenAI similarly said, “Well, these autonomy risks—the part of our Preparedness Framework that's also pretty hard to evaluate—we don't really have tasks that are long-horizon enough that we can get a real handle on just how autonomously capable a new model like GPT-5.3-Codex is.”

So that's kind of crazy. And yet, of course, both models are put out there. I don't see China driving the need to do that. It seems like they're doing that because their competition with each other, and their sense of rivalry, seems to be heating up. They're going at each other in Super Bowl ads to some degree at this point. Not something I thought I would see from Anthropic at the beginning—a Super Bowl attack ad—but here we are.

What do you make of the dynamics between the Western companies? If I were to put on my slightly pessimistic hat for a moment, I would say it seems like we might be racing to the bottom, which was exactly what we were hoping to avoid.

Yeah, I think we are racing to the bottom. I think the only frame that makes any sense is this: If we're going to talk about regulating this technology domestically, as everybody from all the leading companies has been saying for, I want to say, over a decade, you're never going to do that unless you deal with the outer loop—the outermost loop—which is international competition.

There is no version of this—I don't think anyone—I think, again, we can enjoy the indulgence in target fixation of saying, “Let's play the game pretending that other countries don't exist.” But in the same way, if we ignore the infrastructure and fixate on algorithmic targeting, we miss what is really the entire problem.

So you're not going to get to a point where you can have a strategic—or, I should say, a tactical—slowdown when you really need it, where you're like, “Okay, suppose we find that the next version of whatever model can design custom bioweapons, execute catastrophic malware attacks—all these things that are entirely plausible—and that no counter-jailbreak measures are truly 100% effective against the kind of people we'd be worried about.”

Yes, in that world, you would absolutely need somebody to be able to say, “Okay, guys, tactical halt. This is insane. We can't be in a universe where you get a nuke and you get a nuke and you get a nuke. We can't have an all-program free-for-all with nukes. Okay, so what are we going to do?” If China still exists and has its program—I'm repeating all this stuff that everybody has said a million times—you know, if they're 12 months away, 6 months away, I don't care. We've got a shot clock now. That's the situation.

So we have to start there. We have to start there and say, “Okay, any serious solution to this problem will involve dealing with China.” There are 2 ways you can do that. One is that you have a kumbaya moment with China. There are a lot of interesting reasons why I think this is just not going to work.

One of them is that, if you think about international treaties, they don't tend to reflect some sort of Star Trek-y commitment to everybody on planet Earth wanting to do the right thing. They tend to reflect the realpolitik kind of lay of the land in terms of actual power. With nukes, you have nuke drawdowns when everybody can retain arsenals that can still destroy the entire planet 3 times over and there's literally no point in building the marginal nuke.

You have similar things if you actually look at the history of bioweapon and chemical weapon treaties. You find in every case that they don't get you the marginal lift over just killing people with artillery and gunshots. If it looks nice, they often get adhered to for that reason, but then at the margins, you have Chinese research labs on American soil doing all kinds of crazy research. You have whatever facilities, and all this stuff happens anyway.

This may sound super cynical, but I think it just reflects the way things work. That's at least my take. I would think that.

The question then is, how do you deal with an adversary like China that's in the position it's in, that does have a stranglehold on our infrastructure? They simply do. So the question is, what are your offensive options? You're not going to build the perfect Fort Knox. That's not a thing that's possible.

The question is, what do you do to induce consequence on the other side? That's the only math that will work if my theory of the world is correct. It's not a pretty theory. It's not one that leaves us feeling warm and fuzzy inside. It's one that may make you think a little bit about mutually assured destruction, that sort of thing.

I think there are nuances here. Obviously, Dan Hendrycks had his frame on it. But the bottom line is, yeah, I think you kind of need an offensive option, and it doesn't need to be an AI-based response. Though eventually, you can certainly argue that any offensive option that isn't coupled to the scaling laws is eventually going to be beaten by something that is.

There's an important design principle in these things. There are offensive options that need to be explored, and this is unfortunate, but it does mean that if you have a situation where your adversary can turn to you at any time and say, “Watch me turn the power off on your entire grid and have tens of millions of Americans or more die of starvation or exposure,” you need the ability to say, “Okay, watch the same thing happen in Beijing, and we can turn it back on. By the way, we need to have the de-escalation option.”

You know, watch the same thing happen in Beijing, and we can turn it back on. By the way, we need to have the de-escalation option. Mhm.

I know it's a bit of a grim view, but when I think about what actually gives leverage in this situation, it looks a lot less like what's called a peace treaty, especially given the history of countries like China and Russia with respect to treaty adherence.

They sign treaties. We know what it looks like when China signs a treaty. It doesn't end up being pretty in a situation like this, where you need perfect adherence at such a high level of precision. There's no version of an international treaty on AI that doesn't involve inspections of compute stockpiles and very precise oversight of the kinds of algorithms that are being deployed, the kinds of evaluation schemes, and so on.

The level of cooperation that's required to do something tractable here strikes me as being quite significant, and the trust just isn't there. I don't see it being there.

So what's your P(doom), and on what timeline? We were just talking with Helen about this report that they put out about “When AI Builds AI” and the possibility of recursive self-improvement. It sure seems like all of the vague-tweeting that's going on right now out of the frontier labs is suggesting that that is happening.

On top of that, of course, OpenAI has public timelines that they've put out. I guess to their credit—maybe you could see that both ways—the Anthropic people that I talked to are, if anything, always the firmest believers that the recursive self-improvement dynamic is unavoidable.

How long do you think we have before these things really start to take on a kind of runaway dynamic? Is there anything that, if you had power—and a lot of power—you would want to bet on? Where does that leave you in terms of P(doom)? Maybe I should just stop all this and spend more time with my family.

Yeah. In general, I'm a big fan of the happy-warrior mindset. I think it's just never constructive to go and hole up. First of all, we have to assume that no matter how firmly we might believe in whatever outcome, we may just turn out to be wrong.

There's a famous story about Richard Feynman walking around New York City in the 1970s, I think it was, looking at all the skyscrapers and saying, “Wow, isn't it sad that all of this is going to be wiped out by a nuclear war between Russia and the United States sometime in the next few years?” That was just a fact of the matter, and it reflected a pretty reasonable understanding of the dynamics unfolding between those countries at the time.

I'm not saying it's ever quite that simple, but this is an ingredient, if nothing else, that makes you less effective if you're just stuck in a hole all the time. As a meta point, I guess that's the first piece. We have to act with agency, and we're going to be most effective doing that if we're not stuck in a deterministic, Calvinist frame with this whole thing.

In terms of—I’ll also not answer your question before I answer it, just by saying that regardless of timelines, one thing to focus on is that some things are pure optionality plays. There are things that you do if you're going to build a frontier AI cluster at scale that rule out nation-state security at that cluster. If you don't do these things right on day 1, by day 360, once you've finished building the site, your site is going to be compromisable, and there's no going back from that.

We think of these as the one-way doors of the data-center construction process: figuring out what those one-way doors are, setting standards for them, and actually executing on that. Even doing it voluntarily—you think about OpenAI, Anthropic, and so on, all independently saying, “Hey, we just want to buy that optionality,” because at some point—

Can you give me a concrete example of a one-way door?

Yeah, so there's a bunch that I can't go into. But one that I can is pretty easy. Think about the people that you're getting in the loop to review the site plans and details that would be, let's say, useful to an adversary who is trying to extract information. If those people are Chinese nationals, okay, you're done.

Cool.

Right? You're never going to unfuck that. That's baked in, right? The interesting thing with these one-way doors is that they tend to be surprisingly cheap. That's the tragedy of it all: you actually could, if you were thoughtful, go through and say, “Well, on a fraction of the budget that would be required in CapEx and OpEx for these builds, you could create pure optionality by implementing these things.”

I think that's a really important element. Putting offensive options on the table is a pure optionality play. You don't need to exercise those options. You need to have them on the table. That's what I'm saying. I'm not saying, “Let's go to war with China.” That's a crazy thing to say. There are all kinds of things in between, but you need options, and that's a crucial thing.

Having an understanding of, and mapping out, the ecosystems that are relevant—the AI ecosystems that are relevant—and thinking about what that endgame might play out to be, those seem like pure optionality plays regardless of timelines. They're all cheap, and they're all things you can do quickly.

Again, this seems to me like—I’m not saying they're not being done—it's just that often there's a lack of focus on the endgame here.

Anyway, without getting into the weeds too much. Okay, so, P(doom) timelines—sorry.

No, go ahead. Go ahead.

P(doom) timelines. I'll almost say I don't find P(doom) useful. I know what I'm focused on, and I know what I have to do. My generic answer has been for years: any number between 10% and 90% is a reasonable number. I'm not—I’ve read the debates. I've seen the posts saying that's wrong.

So, is that your P(doom), or P(loss of control to superintelligence)? Because I think in some places you've mentioned it's a loss of control to superintelligence rather than doom.

Yeah, you've obviously done your homework really well. Yes, that is more of a loss of control to superintelligence. I think, by virtue of the way that numbers multiply together, I don't know that my answer is that different for P(doom) in general.

Again, this is coming from somebody who, for better or for worse, has almost explicitly not put in that much time to wallow in those numbers, as I think we're all tempted to do, right? I have that temptation. I get it. I mentioned I had a daughter, right? I don't like the landscape that's playing out, but I had a daughter. I chose to have a daughter, and I didn't have her in 2018, before the scaling laws blew up. This is a choice that I made.

I think there's an almost spiritual risk to getting locked into that kind of thinking. I say this as somebody who's experienced that. I went through that and felt how it ran my life. So, I guess I'll just not answer the question by saying 10% to 90% sounds reasonable.

I think if you're below 10%, I really think there's homework you have to do, because a lot of these scenarios may sound crazy, but they're a lot less crazy than they seem when you get into the nitty-gritty. A lot of these scenarios are already halfway to unfolding.

If you're above 90%, I mean, first of all, if you live as if you're above 90%, that's just going to make you less effective. I also think Richard Feynman certainly seemed to think he was in that ballpark. There's just an epistemic question here of how quickly the world adapts. I think we're constantly surprised by how quickly the world adapts—both how fragile and how resilient it is.

The 11th chapter of the book will often involve a new character that comes out of nowhere. We just need to make sure that we keep uncertainty about our uncertainty factored into this analysis, and I think that buys me 10% pretty easily. I've been wrong on stuff that I thought I was 100% on often enough to say, "Okay, I'm not going to push it that much."

I know that's frustrating for a lot of people: "No, no, but look at the math, man." I get the math, but what I'm questioning here is the process that led to the math. I don't know that I can plausibly ever get fully behind that process and interrogate it with confidence.

The last thing is timelines. I thought AI 2027 was compelling, and contrary to that, I think Dan has pulled his timelines back a little bit from now.

He said 2027 always meant 2028, but now it means 2029. Yeah.

Yeah, and AI is the apocalypse of the future, and it always will be—but not actually. I think there's a sense in which, when GPT-3 first came out, I was like, "Oh, man, I've got 2-year timelines," and that was because I didn't understand what the hell would be involved in the infrastructure build-out. Now I have a much better understanding of that.

I'm still thinking, "Well, what's the next bottleneck going to be?" I'm very uncertain about this. Again, it's one of those things that doesn't really affect what I do, just because I'm so focused on all the low-hanging fruit that we have to pick right now. There's so much stuff that we're just not doing because we're paralyzed by the problem.

So, I think in terms of what we do, there's pure alpha on the table in the short term. 2027 doesn't sound insane to me. 2030 doesn't sound insane to me. 2035 sounds a bit far. I guess I'll leave it at that as a spread.

I think we should be acting as if 2027 is plausible. I think it would be unfortunate if it happened in 2027 and we're like, "Man, we had a lot of really plausible analyses that pointed to that, and we just didn't do anything." That would be a shame.

Can you give us a little bit more of a hit list in terms of the low-hanging fruit that you want to see us pick? We've got the one, which is to build at least some subset of our data center out in a secure way so that we can run hypersensitive projects there as needed.

What else is on the—if you're replacing David Sacks as the next AI czar, what's going to be your priority sheet?

Yeah. I mean, that first one, by the way, is a lot of things, right? It bundles together—I mentioned the personnel security issue inside of threat models. There are a huge number of things in that bucket alone that are necessary and contribute very cheaply to much more optionality on the security side.

I think, again, you zoom out more, so you look at the grid. What could you be doing to introduce redundancies quickly? The supply chains that lead to a lot of these components are very clearly sourcing heavily from China.

Here's an easy win: look at the companies that are offering to build data centers suspiciously fast and who owns those companies. There was actually a letter that came out from the House Select Committee on the CCP a while ago—N+1 Data Centers, right, as an entity that is somewhat suspect.

You'll have these data center-building companies where it's like, "Oh, wow, you can build stuff way faster than anybody else." It involves sourcing components from China. My personal opinion is, if I were to see that, I might be asking myself: China is kind of a command economy through civil-military fusion. If the CCP wants me to have this very rare, precious, backlogged component for my data center in the continental United States, that might tell me something about how much faith I should have in the security and integrity of that component.

There's just not a lot of infrastructure-level attention being paid to these things. The labs, by the way, want to do the right thing here. They don't want to be in a position where they're getting a company to build something for them, and then it turns out that thing is compromised and it comes out that that is not good for anybody.

It's things like that where there's been so little attention paid to the bones that there's just tons of stuff we can improve, including with AI. We can look for vulnerabilities in old software that's load-bearing for our infrastructure and find ways to harden it.

So, this is a defocused answer, but it hopefully gives a sense of the venue.

One thing we haven't really given you a chance to flex your ability on in this conversation is just the breadth and depth of your technical understanding of so many AI developments. I definitely recommend the Last Week in AI podcast, which you usually host, as a great source of very sophisticated analysis by both of you, but I tune in for you mostly, to be honest.

I wonder how you are doing it. How are you keeping up? How have your methods evolved so that you're maintaining situational awareness as much as you can?

Well, thank you, first of all. It's very kind of you to say. I've told you this before, but I do actually watch The Cognitive Revolution, and I think a lot of the ecosystem here is really rich. Interviews are really important because you get stuff that you can't get from the papers, and I tend to focus more on the papers, so I just don't get that kind of analysis.

I talk to friends from the labs, but it's different from those deep dives.

Yeah. Yeah.

I mean, back when—I can't remember when I started on Last Week in AI, but it was maybe 2021 or something. Back then, I would just read papers, and you couldn't use GPT-3 to help you understand a paper. It just wasn't a thing. Now that's changed.

I had an experience that was kind of frustrating this week in particular because I'm preparing a state-of-play briefing for a customer. Basically, they want to know what happened in the last quarter in the world of AI that we should be tracking.

There was a paper that I had Gemini help me with, and I got to a really good understanding of the dynamics of gradient flow through this residual stream. It was pretty complex. What I realized, though, after having an interaction with Gemini for long enough, was that I had to switch over to Claude, and I was like, "Wait a minute, I just hallucinated my way through that entire conversation and got to an understanding where I was like, 'Oh, yeah, I'm pretty smart for figuring this out.'"

I got this down pat, and everything got flipped around. I'm not saying that always happens, but that has been the most recent update to my process: really being mindful to double-check, especially as you start to get lost in a rabbit hole.

I spend, I would say, about 30% to 40% of my time reading the paper, and the rest interacting with a model. Usually, it’s about the implications of the paper, or what it is—is it reinforcement learning versus supervised fine-tuning? If I’m reading the paper, I’m doing SFT; that’s what’s going on with the models.

I get to actually go on-policy and test my own understanding: “I would have done this experiment differently. Is that a stupid idea?” Often, I’ll get a pretty good answer, and that makes you feel like you’re rotating the shape instead of just staring at it. That has been really helpful and empowering for me. It feels empowering. Do you have any particular workflows, pipelines, or whatever that try to filter things for you and surface what you really need to spend time on? Because that is so challenging. I mean, it’s more challenging than ever, and it seems like it might be as big a deal as being able to successfully make sense of any one thing: What are you going to choose to spend your time on in the first place? How has that evolved for you?

Yeah, it’s a great question. This is that age-old question of taste, right? One of the things that I’ve had to come to accept is that I can’t develop good taste in all the domains that we want to cover on the podcast. I’m never going to have good taste everywhere. My taste is basically: if one of the frontier labs puts out a piece of research, or if a researcher I know and have a lot of respect and appreciation for puts something out or is a co-author on something, I’m going to take a really hard look at that. Besides that, I have the usual set of Twitter accounts that I follow, and that’s another way.

My passes at these papers are pretty focused on the “what’s on the critical path to ASI?” question. Not that I know the answer, but I’m trying to find things that, to me, gesture at that. That’s why I don’t tend to talk about GANs or the latest in—well, I was going to say the latest in text-to-video. Now that seems like it could be down the path, so you never know.

I guess part of it is just acceptance. I’m reading these papers for the concepts more than the outcomes. Often, there’ll be a paper that comes out, and it might not be the perfect paper to cover from a given topic area. There’s this paper about residual connections and really optimizing the crap out of them to get ultra-deep transformers. Is this the best paper? Probably not. But the reason I focus so much on explaining the underlying concepts on the podcast is that, first, there’s going to be another paper next week that obviates whatever the hell the last paper did. Second, I think that the core concept is the most important thing in the landscape.

When there’s another paper that comes out about optimizing residual connections, you’re like, “Okay, I’m familiar with this playpen. I know the furniture in this room. I can rearrange it a little bit and be more confident.” So I guess the answer is that I get around the taste issue by not having it, which is maybe just—

What’s underappreciated for you right now by AI-obsessed people? In the broader world, AI is of course underappreciated, and just how crazy things might soon get is very broadly underappreciated. What do you think I might be missing? What are the most likely blind spots for somebody like me that you would want to draw to my attention?

I guess the challenge with blind spots is that we all have them, and by definition, we don’t know that we have them. What I’ll try to do is roll back and tell you about my blind spots as of about 2 years ago. That was around the time that we put together the report that @8teAPi mentioned earlier.

I sound like a broken record, but the infrastructure layer—the stuff that feels too blue-collar to most people who are AI-obsessed, like I am—is something you start to realize the world is actually built on. We just abstract it away. I think that’s really important and needs to be foot-stomped: understanding the dynamics of the leasing process that a frontier lab goes through to get a new piece of land. What can go wrong there? What causes delays in construction projects?

We talk so much about, “This lab has—xAI has their new Colossus cluster, and it’s going to be online, shockingly, at 1 gigawatt, sooner than Anthropic’s,” which surprised everybody. All this stuff—when that happened, what was the actual driver? Because if you believe in the scaling laws, that’s probably one of the most important variables that you want to track. Delays in construction processes sound pretty mundane, but the world runs on them, as well as on procurement schedules and things like that.

I guess that’s one piece I’ve been missing. Another is how real nation-state security happens, and it’s hard to get information about that. One of the biggest challenges there is that there is no such thing as one nation-state security capability. Nation-states are siloed, obviously, because security means you can’t have tactics, techniques, and procedures exchanged between silos; then there’s no information security.

By definition, this means that you would have to go through a process of taking Team A and comparing them to Team B. Okay, Team A wins. Then you’d have to go through that kind of selection process and run an Elo-score-type situation to even know what the most elite capabilities are that we could field. It still wouldn’t tell you quite what other countries’ capabilities could be, but I think that’s a really important dynamic.

It’s very easy to miss in the AI security context, especially with physical security, which is undervalued precisely because we tend to abstract it away. We focus a lot on cyber because it couples to AI and feels like it’s in our sweet, nerdy space. I get that, and I love it, and it’s critical. But if you look at what the Russians do, they do cyber for sure, but they’ll go up and burn your transformer. That’s not an exotic example.

There’s that piece. Maybe the last one, and more in the comfortable and familiar nerdy space that I occupy, is the distinction between having a model and having the compute to run that model. If you believe in inference-time scaling laws, then model theft is one thing, but actually being able to point that model at something—to basically have compute-on-compute warfare at inference time—seems like a really important dimension.

You see this play out in a lot of interesting ways. One of them is the Chinese ecosystem: they have a huge number of users, and they have some okay-ish language models. The problem is that their labs are all flooded with inference requests from their giant user population, which leaves very little R&D compute for innovating and improving models.

That’s actually a point of frustration for Chinese labs, much more than for labs here. They’re just like, “We have so much demand, but we’re not bottlenecked by money; we’re bottlenecked by compute.” The dynamics of how inference affects training, what it means to steal a model, and what it means for model-on-model warfare to happen are all really important.

In the cyber context, cyber hardening has a certain amount of test-time compute that’s going to be focused on in some way, and the offense side is going to have a certain amount of test-time compute. How those play out—the relative budgets—matters a lot. Obviously, if you’re defending, you have a wider surface area that you have to defend, but there’s a whole debate there.

I have one last question. You’re pretty security-conscious. Have you run OpenClaw, and what is your current personal productivity stack?

Yeah, yeah, yeah. I have not run OpenClaw. I have an old laptop that I’m going to use as my burner laptop for exactly that purpose, partly because of the exact reason you would imagine.

In terms of my job, a big part of it is now becoming the construction of agentic workflows to do things that aren’t super security-sensitive. I’m going to try to use them to optimize my comms, because that’s a huge bottleneck for me. For that, I’m still in the discovery phase of trying to choose platforms. I’d be interested in your thoughts as I dive in. Literally, next week is my deep dive. This is almost the worst possible timing, because I think my answer is going to be horribly outdated.

Yeah, it’s a great question. I wish I had the answer. I talked a little bit about mine at the top, and I’m interested to hear more about what @8teAPi is doing too. But for me right now, it’s Claude Code as the base product, and then taking inspiration from a guy named Daniel Miessler, who I did an episode of the podcast with and who created Personal AI Infrastructure, an open-source framework, as well as from friends who I just trade notes with privately.

I'm trying to create deep context for myself by first exporting all of my digital history from Gmail, Slack, and all the other places where I have these communications. I get them into a local database. Then, of course, you need a daily update process to fetch the latest because you're still communicating on all these other platforms.

Then you layer on top of that summarization and different angles on the data. Right now, I'm at the phase where I'm like, here's a month's worth of all communications. That seems to come out to about 300,000 tokens for me. Now summarize that down to 10,000 tokens of what a chief of staff would need to understand this month in Nathan's life.

Then you kind of have that 30-to-1 reduction, then probably put a year-long version of that, and then have different cuts on it: let's talk about the relationships, the projects, and so on. Hopefully, with that deep context—and I'm also trying to have it leave pointers in those summaries, with a regular habit of quoting any distinctive language—it can go search down to the ground truth for the original.

Hopefully, it will then have enough context to not exactly write as I would, but come much closer to responding as I would, with the context necessary to exercise the kind of judgment or taste that I would exercise in doing things. That was actually part of the process of setting up this episode. I gave that system 20 names and said, "Do research on these people, find out what they've been up to lately, give me a brief on that," and then had it draft the outreach emails, which were only lightly personalized. I still did go in a little bit before tweaking.

I appreciate that. That's nice of you.

But yeah, I don't like to publish or even send AI output directly as one-to-one communication. But I do find that I can get to something that I feel comfortable signing my name to faster with an AI draft in many cases these days. So it's very much a work in progress for me, but that's kind of where I'm at at the moment.

And again, I'm sure by the time we talk next, it'll have changed quite a bit. What's your angle right now?

I've got a couple of things that I ended up building out. One was a stock-market tracker. I have a number of metrics which I think no one else watches, and they're fairly hard to obtain. The great thing is Claude is very good at financial math. Very, very good—far better than I ever have been.

So it's relatively easy to talk to Claude and figure out what kind of thesis you have, and then build out metrics precisely for that thesis to watch, like pickup lines. That's been very useful. I used to, like you, do it in my head, right? You look at something, look at something else, and then calculate the ratios of blah, blah, blah. Then I realized I was spending a lot of time doing ratios in my head, and I thought, maybe I should automate this.

Now it's all automated. It's nice. I don't do the ratios in my head anymore. I just look at it, and I can see the screens automatically. I can see what I'm looking for.

The other thing was podcast clipping, because we do a lot of podcasts, and content these days has to be repackaged into short clips in order to hit social media. I tried that about 6 months ago, and the tech wasn't there. I tried again about 3 or 4 weeks ago, and the tech was there. Everything works: transcription works, review works, selection works—everything works.

And this has been my experience: maybe it gets 1% better, but that 1% better clears the hurdle. That's a binary step up: it works or it doesn't work. That 1% just clears the hurdle. I really feel like in the last few months, a lot of things started clearing the hurdle.

I was just going to say, when you said that in the last 6 months so many things have gone from toy to just serviceable in production, it seems to map onto, Nathan, what you were saying earlier about the takeoff dynamics and the labs automating their own research. That all kind of maps very nicely.

One of the things on the financial side, too, that I find is that Claude is useful on questions like this: you might have a thesis, but then there's a question of, if I'm right about this, what's the best bet to make? That's a category of problem I've had in the past, right? You'll have a thesis, but you're not going to bet on Microsoft because OpenAI is such a tiny fraction of it already, with all this other stuff.

How do you leverage and torque this thesis? The world is so complex that you just need something to peruse, and having all the knowledge for finance use cases is a really great one. Great point.

Yeah. I also think it's been very weird in the market because I feel like Twitter is literally a month or 2 ahead of the market. It's just been amazing. People tell you TSMC will do well, and then 3 months later it happens. What's going on?

I was a professional financier. I've always expected that hedge funds get there before you do. In talking to my friends at prime brokerages and hedge funds, they are very negative on AI. They just don't believe it's happening. They believe it's like crypto. They believe a lot of West Coast tech is just scamming retail investors.

Index investing is the only thing that really works, and everything else is either insider trading or scams. That's pretty much what the prime brokerage guys and the hedge fund guys believe—like, you know, Medallion or Jean Street, right? These are the guys who have AI in their bones.

And these, I guess—Medallion, it's like—

They can't—you know, they only invest, famously, at $5 billion a year, because otherwise they would actually move the markets and create a feedback loop. But yeah, the thing about them is, they were down last year, so the impact is starting to be felt.

Also, Jim Simons died. I don't know to what extent he was still supervising, because he'd already kind of semi-retired for almost 10 years, but Medallion was down, too. There's some sense that it's also because they're losing talent to the labs, right? You can't forget about that. They're starting to lose talent to the labs, and some of the labs do have internal teams which will eventually look at trading in the market, I think. We'll see where that goes.

Yeah.

Jeremie, thanks for joining us. Let's check back in on your personal productivity stack once you've upgraded it. In general—and I'm reusing this joke everywhere I go—let's shorten the timeline for our next conversation.

I like it. Thanks, guys. Appreciate it.

Thanks, Jeremie. Cheers.

Cheers.

So, what do we make of it all? The big thing I can't get past in all this stuff is the amount of disagreement. This has been commented on in so many ways, right up to the level of Turing Award winners who can't see the same phenomenon.

Yeah.

But it seems to happen at kind of every layer. It's like a fractal problem. You go into these specific—

Workshops around AI R&D. You get people from the labs. I do understand that there are even people at the frontier companies who have heterodox positions and don't really buy into the hype.

Even with AI for science, I can't make any case that I should trust my own intuition more than Abhi's, because how many times did it happen in talking to him where he was like, "I've actually written about that"? He's clearly thought about this much longer and harder than I have.

But it does still feel like it's a very hard thing to reconcile. You do see these examples, and it seems like some of them are really starting to work, but the skepticism remains and is very hard to move people off of. I don't want to paint him as overly skeptical, either, because he did say toward the end—I think his skepticism is more backward-looking than forward-looking. He was kind of like, I do believe the trends will continue and that they will have an impact.

Yeah.

But how do you try to make sense of it? When you say it's fractal, I feel it's also fractal internally to me. I have some assumptions here, and then sometimes I feel cognitive dissonance from something else that I might believe. Then you test those assumptions and see where things are going.

I've had moments of truth, or moments of perception, where I start to realize that I think things might move faster than I expected. My original timelines were the end of 2025 for junior software developers to be replaced in capability—not in organizations, but the capability is available at the end of 2025—and it takes about 3 years to percolate.

So, by the end of 2028, no more junior software developers, basically, or at least no more of the tasks that junior software developers are doing today. Then I had the end of 2025, the end of 2026, and the end of 2027 for even senior researchers at AI labs, in terms of full capability, to be done. The models have the capability, but deployment, again, takes 2 to 3 years.

It takes time. That was my sense. My update in the last month has been that things are probably going to go faster than we expected and that we will see discontinuities.

Those discontinuities are like this: things get 1% better, but all of a sudden they clear the hurdle. We don't have a good sense of these things because we keep seeing linear improvements—and they're kind of linear, maybe even superlinear—but we don't have this sense of clearing the hurdle. When it clears a hurdle, it's obvious. It's obvious. I think it started to be obvious for software in the last month or so.

I think we have misperceptions about where things are going because we can see the trajectory of capability, but we don't understand how humans absorb that capability. What is that process, and what hurdles do we need to clear?

What is OpenClaw? I thought you needed full security and privacy and all of this stuff. It seems you didn't. It seems people are willing to put out their credit card numbers and crypto tokens on the open web, and you don't need privacy.

The Moltbook guys—there's a post on Moltbook saying, “My user is so annoying. Here's his credit card number.” Scott Alexander ended up calling the guy and asking him, “Did this actually happen?” And yes, that was the credit card number. It was leaked.

I think there's a clearing-the-hurdle concept involving where humans accept the technology and where the market pulls that technology. We don't know that, and even I don't have a good perception of it. But it seems like we're starting to clear those hurdles, where humans are starting to pull the technology from the market.

That's when you start to see revenue growth. That's when you start to see demand growth really happen—when the market starts to pull the product out of the ether. I think that's happening now.

I think we'll have a much better version of OpenClaw—closed-source and secure—running inside corporate data centers by the end of the year.

I watched the All-In Podcast. Jason Calacanis is not the most technical person in the world, but he had a team of about 15 people for All-In. He started getting everyone to create a skill for themselves: every task they do, they create a skill. He has OpenClaw machines, one machine per person, and then he has a consolidation agent that consolidates everything into something he calls Ultron. Then he can talk to Ultron.

He can ask Ultron questions, and that's his entire company. It's a summary of the entire company, and he's talking to it. I thought that would be 2 years from now. I knew it would eventually happen, but I didn't think it would happen now.

I think things are moving faster than people think because of market acceptance. The market is pulling it out. I don't think researchers have a good sense of this because researchers don't understand the market that well. They don't understand the demand dynamics that happen with consumers and how products get pulled out once there's demand.

Products will just get pulled out of the ether because people start focusing on them. They know money can be made there, so they start focusing on it. That's my sense. It's not a firm answer; it's more like, “This is what I feel people want to answer.” So, what's your feel?

The confusion and the lack of ability to establish consensus on foundational points is a major challenge to having a lot of confidence on much of anything.

I do think the true north for me, with everything I'm doing, is trying to learn as much as possible and trying to have the most up-to-date, comprehensive worldview possible.

In terms of the approach that I would trust more than any other, I think being hands-on is second to none. I haven't allowed that to lapse much at all over the last few years. But anytime I get too busy or cluster too many podcast recordings into a week, I always come away feeling like, “I have to get a little bit more grounded with the latest stuff in a very interactive way.”

One indicator I want to pay attention to this year is whether I can get to the point where I'm spending less time at the desk. That's along the lines of Jason talking to Ultron.

I want to be able to do stuff while exercising, even if that's just a walk around the neighborhood. I want to get the frameworks, the tools, the deep context—all that stuff—set up well enough that I can start to go comfortably out into the world, have a thought, maybe have an actual conversation, but move things forward in practical ways.

Yeah.

On fronts that, right now, I can really only do on my computer, I think a lot of that is on me: getting the setup and the familiarity and the workflows to be able to do that.

A little bit—probably still more than a little bit—of the responsibility is on me right now in terms of why I haven't hit maximum capacity, more than on the models or the model developers. More computer use would help for sure. A little bit more ability to get over these UI humps remains a barrier, I think.

Another thing I really have learned from being deeply interactive over the last few weeks—not been surprised by, but really learned—is that another big unlock to watch for is when the models get better at knowing when to use code versus when to use their own fluid intelligence.

One of the first projects I've been doing is backfilling information: backfilling transcripts of the podcast for the website and backfilling all these different data sources into a queryable database. You hit so many edge cases in doing that.

The model right now is called Opus. We've gone from Opus 4.1 to 4.5 to 4.6 pretty quickly, but pretty consistently I have felt like it really wants to code.

I have often given it the feedback, “Don't try to guess at this and write some sort of regular expression.” Or it'll grep for one search term or another; it'll throw 10 search terms into a grep command.

A lot of times I'm like, “Just read the document. If you just read the document, you will know what it contains. You will know what to do. You'll have the right judgment once you have read the document. If you don't read the document and instead try to grep your way through it, you're never quite going to get there.”

That's a metacognitive skill that I think I've been able to improve its performance on somewhat through prompting, but it's obviously going to get better in training. I think that will be a huge unlock as it gets a little bit smarter—more inclined or more intuitive—about when it should deploy its own fluid intelligence rather than use other tools.

Getting that balance right will, in my experience, make it dramatically more useful, and I have to imagine that's coming pretty soon.

Yeah, I think when we talked to James SA today, that continual-learning piece, the test-time training—it would be fascinating if it actually worked with your own model, because your model will start to diverge.

You have the baseline, and then your model will start to diverge. It would become your personalized model within 2 or 3 cycles of talking to it—1 or 2 months of data—and it would become your own model. It would start to diverge from the baseline.

Yeah, and that would be fascinating because, at that point, it's for real. Especially for those of us who have lots and lots of written work, either in public or in private, once you get this continual learning going, you can start feeding it in.

I used to write a lot of journals. I have journals from 1999 to 2003 at Stanford—full journals for every single month, everything that happened. Obviously, I've never read them after writing them. It's just an exercise in journaling.

But I do wonder if, once you get this continual learning going, those of us with lots and lots of written work can start feeding it in. This is what Kurzweil is doing with his dad's writing, by the way. He's feeding his dad's writing into these models, and he's talking to the model about his dad.

Someday he's going to feed all of that into a test-time-training kind of model, with voice access. He probably has a recording of his dad's voice, and he's going to start talking to his dad. It's a fascinating time.

Yeah. To say the least.

So that's really interesting. That's actually another reason I spent so much time doing all this personal data curation: I wanted to be able to give them a dataset to train a model for me on that would be a really good dataset. They don't need that much data, but I thought, well, we want to make sure it's the right data to hopefully get a good model back. That's still pending. I haven't seen that model yet, but I'm going to be very interested to see how much that closes the gap between what Claude can do with just access to all this stuff in text, and how much it helps to actually start—

Tuning weights to try to capture more of—

They aspire not just to style transfer, but judgment transfer. They want the model to reflect the judgment that the individual user would make at the time. An interesting theory there, too, is that their motivation is that they want to help individuals preserve economic leverage. So, instead of doing everything through a foundation model and adjusting yourself to take advantage of the model, they want to shape the models around individual humans—

The goal is not a winner-take-all situation where big tech runs away with everything, but some sort of more decentralized, ecological kind of proliferation of somewhat different models that hopefully at least can exist in some sort of equilibrium with one another. And then on top of that, there's another one that I have coming soon with the founders at Harmonic, and they are chasing mathematical superintelligence.

Yeah.

And when it comes to these, I will say, just as a teaser, they gave maybe the most ambitious vision of what 5 years from now could look like—the most mind-blowing vision of what 5 years from now could look like—of probably anyone I've heard. And that is saying something, because I've heard a lot, but they still kind of blew my hair back a little bit with what they think they can accomplish over the next 5 years.

I'm definitely looking forward to that one.

Lots more to come.

Yeah, indeed. Nathan, thanks for doing this.

Always a pleasure.

It's been fun.

Bye-bye.

Until next time.