AI 编程的危险幻觉？——Jeremy Howard

It literally disgusts me. I literally think it’s inhumane. My mission remains the same as it has been for about 20 years, which is to stop people working like this.

Jeremy Howard, a deep learning pioneer, a Kaggle grandmaster. He is a huge advocate for actually understanding what we are building through an interactive loop, a notebook, a ripple, the act of poking at a problem until it pushes back. He argues this is where the real insight happens. And the funny thing is they're both right.

LLMs cosplay understanding things. They pretend to understand things. No one’s actually creating 50 times more high-quality software than they were before. We’ve actually just done a study of this, and there’s a tiny uptick in what people are actually shipping.

The thing about AI-based coding is that it’s like a slot machine in that you have an illusion of control. You get to craft your prompt and your list of MCPs and your skills and whatever, but in the end, you pull the lever, right? Here’s a piece of code that no one understands.

Yeah.

Am I going to bet my company’s product on it? The answer is, I don’t know, because I don’t know what to do now. No one’s been in this situation. They’re really bad at software engineering, and I think that’s possibly always going to be true.

The idea is that a human can do a lot more with a computer when the human can manipulate the objects inside that computer in real time, study them, move them around, and combine them together. Whoever you listen to, whether it be Feynman or whatever, you always hear from the great scientists how they build deeper intuition by building mental models, which they get over time by interacting with the things that they’re learning about.

A machine could build an effective hierarchy of abstractions about what the world is and how it works entirely through looking at the statistical correlations of a huge corpus of text using a deep learning model. That was my premise.

This video is brought to you by Nvidia GTC. It's running March the 16th until the 19th in San Jose and streaming free online. The key topics this year are agentic AI and reasoning, high performance inference and training, open models, and physical AI and robotics. I'm so excited about the DJX Spark. I've been on the waiting list for over a year now. It's a personal superco computer that is about the size of a Mac Mini. It's the perfect adornment to a MacBook Pro, by the way. And you can fine-tune a 70 billion parameter language model with one of these things. And I'm giving one away for free. All you have to do is sign up to the conference and attend one of the sessions using the link in the description. As for the sessions, I'm interested in attending Ammon Sang's talk. So, he's the CTO of Cursor and his session is code with context. Build an agentic IDE that truly understands your codebase. Now, obviously, Jensen's keynote is on March the 16th. He said he's going to unveil a new chip that will surprise the world. Their next generation architecture, Vera Rubin, is already in full production. And there's speculation we might even get an early glimpse of their new Fineman architecture. So don't forget folks, the link is in the description. If you're attending virtually, it's completely free. Don't miss it.

Jeremy Howard, welcome to MLST.

Welcome to my home. Thanks for coming.

Yeah. Well, where are we now?

We are in beautiful Moreton Bay in southeast Queensland. We are by the sea, in my backyard.

The weather didn’t disappoint.

It certainly didn’t. It doesn’t often, but if you were here yesterday, it would have been very different.

Well, I don’t know where to start. I’ve been a huge fan, probably since about 2017 or 2018. Of course, you had the famous ULMFiT paper. When I was at Microsoft, I remember doing a presentation about that because now we take it for granted that we fine-tune language models on a corpus of text and then continue to train them and specialize them. But apparently, this was not received wisdom.

No, this was the first time it happened—kind of the first or second. Quoc Le and Andrew Dai had done something a few years ago, but they had missed the key point, which is that the thing you pre-train on has to be a general-purpose corpus.

No one quite realized this key thing. Maybe I had a bit of good fortune here, in that my background was in philosophy and cognitive science, so I’d spent some decades thinking about this.

The technical architecture of ULMFiT—just sketch that out.

I’m a huge fan of regularization. I’m a huge fan of taking a model that’s incredibly flexible and then making it more constrained, not by decreasing the size of the architecture, but by adding regularization. Even that, at the time, was extremely controversial.

That was by no means a unique insight of ours. What Stephen Merity had done was take the extreme flexibility of an AWD-LSTM—a kind of very classical stateful recurrent neural network, towards which things are gradually heading back nowadays—and add 5 different types of regularization. He added every type of regularization you can imagine.

That was my starting point: I now had a massively flexible deep learning model that could be as powerful as I wanted it to be, and it could also be as constrained as I needed it to be. Then I needed a really big corpus of text. Funnily enough, this was also Stephen. He had been at Common Crawl, and I think he helped make the Wikipedia dataset.

Then I realized that the Wikipedia dataset made lots of assumptions. It had all these UNK tokens for unknown words because it assumed classic NLP approaches. So I redid the whole thing, created a new Wikipedia dataset, and that was my general corpus. Then I used an AWD-LSTM and trained it.

It was actually overnight, for 8 hours on a gaming GPU. I was at the University of San Francisco, so we didn’t have heaps of resources—probably a 2080 Ti or something, I suspect.

The next morning when I woke up, I had the same 3-stage architecture that we use today: pre-training, mid-training, and post-training. Once I’d trained something to predict the next word of Wikipedia, I figured it must know a lot about the world. If I then fine-tuned it on a corpus-specific dataset—what we could now call a supervised fine-tuning dataset—which in this case was a dataset of movie reviews, it would become especially good at predicting the next word of those. So it would learn a lot about movies.

I did that for about an hour and then spent a few minutes fine-tuning the downstream classifier on a classic academic dataset, considered the hardest one at the time. It involved taking 5,000 movie reviews and saying whether each one had positive or negative sentiment, which today is considered easy. At that time, the only models that did it quite well were highly specialized models that people had written their whole PhDs on. I beat all of their results 5 minutes later when I fine-tuned that model. It was amazing.

And the other interesting thing is this kind of methodology around how you do the fine-tuning.

The way we did the fine-tuning was something we had developed at fast.ai. This was year 1 of fast.ai, so we were still in our very early days. One of the extremely controversial things we did was focus on fine-tuning existing models because we thought fine-tuning was important.

Some other folks were doing work contemporaneously with that. Jason Yosinski did some really great research, I think during his PhD, on how to fine-tune models and how good they can be, as did some other folks in the computer vision world. We were among the first—there was a bunch of us really investing in fine-tuning.

We felt that using a single learning rate to fine-tune the whole thing all at once made no sense because the different layers have different behaviors. This is one of the things Jason Yosinski’s research also showed. We developed the idea that it was also way faster if you just trained the last layer, because it only had to backpropagate through the last layer. Then, once that was pretty good, you backpropagated through the last 2, then the last 3.

Then we used something called discriminative learning rates, so different layers would be given different learning rates. Another critical insight that no one realized for years, even though we had told everybody, was that you actually have to fine-tune every batch norm. All the normalization layers do actually have to be fine-tuned because that’s moving the whole thing up and down or changing its scale.

When you do that, you can often just fine-tune the last layer or two. We found that with ULMFiT, although we did end up unfreezing all the layers, only the last 2 were really needed to get close to a state-of-the-art result. It took seconds.

Yeah. The discriminative learning rate thing is interesting because I think the received wisdom at the time was that when you fine-tune a model, if the learning rate is too high, you blow out the representations. So I guess the wisdom was that if you don’t have a really low learning rate, you’ll just destroy the representations.

There was no received wisdom because nobody talked about it.

No one cared. It was just that nearly no one cared. Transfer learning was not something anybody thought about. Rachel and I felt like it mattered more than anything, because only 1 person has to train a really big model once, and then the rest of us can all fine-tune it. So we thought we should learn how to do that really well.

We spent a lot of time trying lots of things, but in the end, the intuition was pretty straightforward, and what intuitively seemed like it ought to work basically always did work. That's another big difference between how people still today tend to do machine learning research: they think it's all about ablations, and you can't make any assumptions or guesses. That's not at all true. I find nearly everything that I expect to work almost always works the first time, because I spend a lot of time building up those intuitions—that kind of understanding of how gradients behave.

I think there's a dichotomy, though, between continual learning, which is when we want to keep training the thing but maintain generality, versus fine-tuning a thing to do something specific. There's always been this idea that, yes, you can make a model specific, you can bend it to your will, but you lose generality and you kind of degrade the representation. So tell me about that.

Yeah, there's some truth in that, although not as much as you might think. On the whole, the big problem is that people don't actually look at their activations and don't actually look at their gradients. So something we do in our software, in our fastai software, is we have built into it this ability to see at a glance what your entire network looks like. Once you've done it a few times, it just takes a couple of hours to learn, and you can immediately see, “Oh, I see. This is overtrained or undertrained,” or, “At this layer, something went wrong.” It's not a mystery.

So basically what happens is, for example, you end up with dead neurons that go to a point where they've got zero gradient regardless of what you do with them. That often happens if they get headed off toward infinity. You can always fix that, so it's not as bad as people think by any means.

Something that trains well for continuous learning, when done properly, can also be trained well for a particular task if you're careful. In a sense, you do want the neurons to die, and I'll explain what I mean by this: We want to bend the behavior of models to introduce implicit constraints, because without constraints there is no creativity, there is no reasoning, and so on and so forth. In a sense, you actually want it to say, “Don't do that. You want it to do something else.”

I don't think of it that way. To me, it's more like—I find thinking about humans extremely helpful when it comes to thinking about AI. I find they behave more similarly than differently, and my intuition about each tends to work quite well. With a human, when you learn something new, it's not about unlearning something else. Something I always found is that when I got models to try to learn to do 2 somewhat similar tasks, they almost always got better at both of them than one that only learned one of them.

I was reminded a little bit of the Dino paper from Lun. This whole regime of self-supervised learning—I mean, that was a vision model, but the idea was, okay, we're doing pretraining, and we want to maintain as much diversity and fidelity as possible, so that when we do the downstream task, we've got more things that we can latch on to.

Yeah. Semi-supervised and self-supervised learning was such an unappreciated area, and Yann LeCun was absolutely one of the guys who was also working on it.

I actually did a post because I was so annoyed at how few people cared about semi-supervised learning. I did a whole post about it years ago. Yann LeCun looked at it for me as well and suggested a few other pieces of work that I had missed. I was kind of surprised at how incredibly useful it is to basically come up with a pretext task.

In vision, we did this before ULMFiT. It was in medical imaging: take a histology slide, mask out a few squares, and predict what used to be there. I had some of my students at USF doing stuff with that. It was basically entirely taking stuff that we and others had already done in vision.

Yeah.

This idea of masking out squares—we didn't invent it. Masking out words was the obvious thing, and this idea of gradually unfreezing layers we had done before in computer vision. The whole idea of starting with a pretrained model that was general-purpose had been in computer vision. There was a really classic paper, actually, in computer vision—it might have been around 2015—that was entirely an empirical paper saying, “Look what happens when we take a pretrained ImageNet model,” predicting what sculptor created this sculpture or predicting what architecture style this is, and in every task it got the state-of-the-art result.

It really surprised me. People didn't look at that and think, “I bet that ought to work in every other area as well, whether it be genome sequences or language or whatever.” But people have a bit of a lack of imagination. I find they tend to assume things only work in one particular field. That's really true.

Yeah. I guess there's 2 things there. First of all, we were kind of hinting at this notion of almost Goodhart's law, or the shortcut rule, that you get exactly what you optimize for at the cost of everything else. But that doesn't seem to be the case, because we can optimize for perplexity in the case of language models, and, as you say, what seems to happen is—we're getting into the distributional hypothesis here a little bit. You know the word by the company it keeps. When we have an incredible amount of associative data, it might be masked autoencoding or any of these things like that, the model seems to build something that we might call an understanding, like—

Or I have always thought of it as a hierarchy of abstractions. It needs—if it's going to predict, if the document is, “Here was the opening that Bobby Fischer used,” and it has chess notation, to predict the next thing, it needs to know something about chess notation or at least openings.

If it's like, “And this was vetoed by the 1956 US president,” you need to know—you don't just need to know who the president was, but the idea that there are presidents, and therefore that there are leaders, and therefore the idea that there are groups of people who have hierarchies, and therefore that there are people, and therefore that there are objects. You can't predict the next word of a sentence well without knowing all of these things.

So my hypothesis for why I created ULMFiT is that, in order to compress that as well as possible to get that knowledge, it would have to create these abstractions, these hierarchies of abstractions, somewhere deep inside its model. Otherwise, how could it possibly do a good job of predicting the next word?

Because deep learning models are universal learning machines, and we had a universal way to train them, I figured that if we get the data right and if the hardware is good enough, then, in theory, we ought to be able to build that next-word-predicting machine, which ought to implicitly build a hierarchical structural understanding of the things that are being described by the text that it is learning to predict.

I think that they can know in quite a superficial way. There are a myriad of surface statistical relationships, and they generalize extraordinarily well. It's miraculous.

It is.

But the thing is, I want to contrast this with other comments you've made about creativity. I think knowledge is about constraints, and I think creativity is the evolution of knowledge, respecting those constraints. Therefore, AI is not creative. You've said the same thing. You've said AI isn't creative. So, on the one hand, how can you say that they know and not think that they can be creative?

I don't think I've used that exact expression. I remember chatting with Peter Norvig on camera, and both of us said, “Well, actually, they kind of are creative.” We just got to be a bit careful about our choice of words, I guess.

Piotr Woźniak, who's a guy I really, really respect, kind of rediscovered spaced-repetition learning, built the SuperMemo system, and is the modern-day guru of memory. The entire reason he's based his life around remembering things is because he believes that creativity comes from having a lot of stuff remembered, which is to say, putting together stuff you've remembered in interesting ways is a great way to be creative.

LLMs are actually quite good at that, but there's a kind of creativity they're not at all good at, which is moving outside the distribution. I think that's where you're heading with your question. I'm just framing it this way to say you have to be so nuanced about this stuff, because if you say they're not creative, it can give you the wrong idea. They can do very creative-seeming things. But if the question is, can they really extrapolate outside the training distribution, the answer is no, they can't.

But the training distribution is so big, and the number of ways to interpolate between them is so vast, we don't really know yet what the limitations of that are. But I see it every day, because my work is R&D. I'm constantly on the edge of and outside the training data.

I’m doing things that haven’t been done before. And there’s this weird thing—I don’t know if you’ve ever seen it before. I see it multiple times every day, where the LM goes from being incredibly clever to worse than stupid, not understanding the most basic, fundamental premises about how the world works.

Yeah.

And it’s like, “Oh, whoops. I fell outside the training-data distribution. It’s gone dumb.” And then there’s no point having that discussion any further.

Yes.

You’ve lost it at that point.

Yes. I love Margaret Boden. She had this kind of hierarchy of creativity: combinatorial, exploratory, and transformational. The models can certainly do combinatorial creativity, but for me, it’s all about constraints. This is what Boden said, and even Leonardo da Vinci said that creativity is all about constraints.

You’ve spoken about this—we’ll talk about dialogue engineering—but when we talk with language models, it’s a specification-acquisition problem. We go back and forth, and actually, when we think about the process of intelligence, it’s about building this imaginary Lego block in our mind and respecting various constraints. When you respect those constraints and continue to evolve, then those things are said to be creative.

So, when you add constraints to language models—via supervision, critics, or verifiers—then they are creative. With AlphaEvolve, we’ve seen many examples of this. But the illusion is that, on their own, sans constraints, they don’t have hard constraints. Obviously, they have the behavioral-shaping stuff that we’re talking about, but that’s why they can’t go outside their distribution.

I think they can’t go outside their distribution because it’s just something that that type of mathematical model can’t do. It can do it, but it won’t do it well. When you look at the 2D case of fitting a curve to data, once you go outside the area that the data covers, the curves disappear off into space in wild directions. That’s all we’re doing, but we’re doing it in multiple dimensions.

I think Boden might be pretty shocked at how far compositional creativity can go when you can compose the entirety of the human knowledge corpus. I think this is where people often get confused. For example, I was talking to Chris Lattner yesterday about how Anthropic had gotten Claude to write a C compiler. They were like, “Oh, this is a clean-room C compiler. You can tell it’s clean-room because it was created in Rust.”

Chris created Clang, probably the most widely used C/C++ compiler nowadays, built on top of LLVM, which is the most widely used foundation for compilers. They were like, “Oh, well, Chris didn’t use Rust, and we didn’t give it access to any compiler source code, so it’s a clean-room implementation.” But that misunderstands how LLMs work, right?

All of Chris’s work was in the training data many, many times. LLVM is used widely, and lots and lots of things are built on it, including lots of C and C++ compilers. Converting it to Rust is an interpolation between parts of the training data. It’s a style-transfer problem, so it’s definitely compositional creativity at most, if you can call it creative at all.

You actually see it when you look at the repository that it created. It copied parts of the LLVM code, which Chris says today, “Oh, I made a mistake. I shouldn’t have done it that way. Nobody else does it that way.” You think, “Oh, wow, look—they’re the only other ones that did it that way.” That doesn’t happen accidentally. That happens because you’re not actually being creative.

You’re just finding the nonlinear average point in your training data between Rust things and building-compiler things.

All of that is true. First of all, I think we shouldn’t underestimate how big this combinatorial creativity is. The code is on the internet, but they also had a whole bunch of scaffolded tests, which meant that every time some code was committed, they could run the tests. They basically had a critic, and they could then do this autonomous feedback loop.

In a sense, it’s very similar to the recent research by OpenAI and Gemini, where you’re trying to solve a problem in math and you already have an evaluation function. The same is true of the ARC Prize, right? You have an evaluation function, and what people discount is that even knowledge of what the evaluation function is is partial knowledge of the problem. So, you can brute-force search, use statistical pattern matching, use the verifier as a constraint, and you can actually—

They don’t even need to do that, right? You literally already know how to pass those tests because there’s lots of software that already does it.

So, it just uses that and translates it to Rust. That’s all it did, which is impressive.

Yeah.

I’m much less familiar with math than I am with computer science, but from talking to mathematicians, they tell me that’s also what’s happening with Erdős problems and stuff. Some of them are newly solved.

Yeah. But they’re not sparks of insight. They’re solving problems by meshing together very closely related things that humans have already figured out.

So, on the subject of Claude Code, I know you’ve spoken extensively about vibe coding. Rachel had some interesting work out. She quoted the METR study, which showed that productivity actually went down when people were vibe coding, but I think—

John Carmack

They thought that it went up, which is the most interesting thing.

And then there was also the Anthropic study. Maybe we should rewind a little bit. Dario had this essay out the other day—I think it was called “The Adolescence of Technology” or something like that—and he was basically saying, “Look, we have all of these amazing software engineers at Anthropic, and they are just so productive.” He was extrapolating to the average software engineer, so there’s going to be mass unemployment because soon we’re going to be able to automate all of this with AI.

It doesn’t make any sense. Elon Musk said something a bit similar a few days ago, saying, “LLMs will just spit out the machine code directly. We won’t need libraries or programming languages.”

Yeah.

The thing is, none of these guys have been software engineers recently. I’m not sure Dario’s ever been a software engineer at all. Software engineering is an unusual discipline, and a lot of people mistake it for being the same as typing code into an IDE.

Coding is another one of these style-transfer problems. You take a specification of the problem to solve, and you can use your compositional creativity to find the parts of the training data which, interpolated between them, solve that problem. You interpolate that with the syntax of the target language, and you get code.

There’s a very famous essay by Fred Brooks written many decades ago, “No Silver Bullet,” and it almost sounded like he was talking about today. He was specifically responding to something very similar. In those days, it was all, “What about all these new fourth-generation languages and stuff? We’re not going to need any coders or software engineers anymore because software is now so easy to write. Anybody can write it.”

He guessed that you could get, at maximum, a 30% improvement. He specifically said a 30% improvement in the next decade, but I don’t think he needed to limit it that much, because the vast majority of work in software engineering isn’t typing in the code.

Yeah.

John Carmack

So, in some sense, parts of what Dario said were right. For quite a few people now, most of their code is being typed by a language model. That’s true for me—maybe 90%. But it hasn’t made me that much more productive, because that was never the slow bit.

It’s also helped me a lot with the research and with figuring out which files are going to be touched. But any time I’ve made an attempt at getting an LLM to design a solution to something that hasn’t been designed lots of times before, it’s horrible. What it actually gives me every time is the design of something that looks, on its surface, a bit similar.

Often, that’s going to be an absolute disaster, because things that look, on their surface, a bit similar—and I’m literally trying to create something new to get away from the similar thing—are very misleading.

First of all, I’m exasperated by what I see as the tech-bro predilection to misunderstand cognitive science, philosophy, and whatnot. We’ve spoken to so many really interesting people on MLST. For example, César Hidalgo—he wrote the book “The Laws of Knowledge”—and even M. Chirimuuta, who’s a philosopher of neuroscience, were talking all about how knowledge is, basically, protein.

I think that knowledge is perspectival. I don’t think knowledge can be this abstract, perspective-free thing that can exist on Wikipedia. I also think that knowledge is embodied and alive. It’s something that exists in us, and the purpose of an organization is to preserve and evolve knowledge.

So, when you start delegating cognitive tasks to language models, you actually have this weird, paradoxical effect: you erode the knowledge inside the organization.

Well, that's true, and that's terrifying. There are often these arguments online between people who are like, “LMs don't understand anything. They're just pretending to understand.” And then other people are like, “Don't be ridiculous. Look what this LLM just did for me.” The funny thing is, they're both right.

LLMs cosplay understanding. They pretend to understand things. This was the interesting thing about the early cognitive science work with Daniel Dennett. That's basically what the Chinese room experiment is, right? You've got a guy in a room who can't speak Chinese at all, but he sure looks like he does because you can feed in questions and he gives you back answers. All he's actually doing is looking things up in a huge array of books or machines or whatever.

The difference between pretending to be intelligent and actually being intelligent is entirely unimportant as long as you're in the region in which the pretense is actually effective. It's fine for a great many tasks that LLMs only pretend to be intelligent because, for all intents and purposes, it just doesn't matter until you get to the point where it can't pretend anymore. Then you realize, “Oh my God, this thing's so stupid.”

I'm a fan of Searle, by the way. He said that understanding is causally reducible but ontologically irreducible, and he was saying there was a phenomenal component to understanding, but you don't even need to go there.

The interesting thing about knowledge being protein is this idea that it's basically this canonical idea: the world is a complex place. None of us understand it. It's like the blind men and the elephant. We all have different perspectives. It's a very complex thing, and so we all do this kind of modeling.

The interesting thing is that language models sometimes seem to understand, and they understand because the supervisor places them in a frame. Inside that frame, when you have that perspective of the elephant, they're actually surprisingly coherent, but we discount the supervisor placing the models in that frame.

Yeah. Searle versus Dennett was what everybody was talking about when I was doing my undergraduate degree in philosophy. Consciousness Explained came out around then, probably a little bit before the Chinese room. It's interesting because the discussions were the same discussions we're having now, but they've gone from being abstract discussions to being real discussions.

It's helpful if people go back to the abstract discussions because it helps you step back. It's very distracting at the moment to look at something that's cosplaying intelligence so well and go back to the fundamental question. Anyway, I just wanted to mention that it's this interesting situation we're now in where it's very easy to get the wrong idea about what AI can do, particularly when you don't understand the difference between coding and software engineering.

Yeah, which then takes me to your point—or your question—about the implications of that for organizations. A lot of organizations are basically betting their futures on a speculative premise: that AI is going to be able to do everything better than humans, or at least everything in coding better than humans. I worry about this a lot, both for the organizations and for the humans.

For the humans, when you're not actively using your design, engineering, and coding muscles, you don't grow. You might even wither, but you at least don't grow. Speaking as the CEO of an R&D startup, if my staff aren't growing, then we're going to fail. We can't let that happen.

Getting better at the particular prompting skills, whatever the details of the current generation of AI CLI frameworks, isn't growing. That's as helpful as learning about the details of some AWS API when you don't actually understand how the internet works. It's not reusable knowledge; it's ephemeral knowledge.

If you wanted to, you could actually use it as a learning superpower, but it can also do the opposite. The natural thing it's going to do is remove your confidence over time.

I agree that's the natural thing. This is especially pertinent for you because your career has basically been about educating people in technology and AI literacy. The default behavior is very similar to a self-driving car: there's this tipping point where, at some point, you're not engaged anymore, you're not paying attention, and you get this delegation of competence and understanding debt. That's the default thing.

This study from Anthropic a couple of weeks ago contradicted Dario completely because it said that there were a few people in the study asking conceptual questions, actually keeping on top of things, and they had a gradient of learning, but most people didn't.

My hypothesis about that is that the ideal situation for generative AI coding is that, like us, we've been writing software for decades. We already have this abstract understanding. We're using it in domains that we know well, and we can specify and remove loads of ambiguity. We can track, go back and forth, and stay in touch with the process.

But the default attractor is for people to go into autopilot mode. They have no idea what's happening, and it's actually making them dumber.

I created the first deep-learning-for-medicine company called Enlitic back in 2014. Our initial focus was on radiology, and a lot of people were worried that this would cause radiologists to become less effective at radiology.

I strongly felt the opposite, and I did quite a bit of research into this, looking at what happens when there's fly-by-wire in airplanes or antilock brakes in cars or whatever. If you can successfully automate parts of a task that really are automatable, you can allow the expert to focus on the things they need to focus on.

We saw this happen. In radiology, we found that if we could automate identifying possible nodules in a lung CT scan, we were actually good at it—which we were. The radiologist could then focus on looking at the nodules and trying to decide whether they're malignant or what to do about them.

Again, it's one of these subtle things. If there are things you can fully automate effectively, in a way that removes that cognitive burden from a human so they can focus on things they need to focus on, that can be good.

I don't know where we sit in software development because I've been coding for 40-ish years. I've written a lot of code, and I can glance at a screen of code and, unless it's something quite weird or sophisticated, I can immediately tell you what it does, whether it works, and whatever. I can intuitively see things that could be improved and possible things to be careful of. I'm not sure I could have got to that point if I hadn't written a lot of code.

The people I'm finding who can really benefit from AI right now are either really junior people who can't code at all, who can now write some apps they have in their heads. As long as the apps work reasonably quickly with the current AI capabilities, they're happy. Or they're really experienced people like me or Chris Lattner, because we can basically have it do some of our typing and some of our research for us.

People in the middle, which is most people most of the time, really worry me, because how do you get from point A to point B? Without typing code, it might be possible, but we have no experience of that. We don't know: is it possible? How would you do it?

Is it kind of like going back to school, where in primary school we don't let kids use calculators so that they develop their number muscle? Do we need to do that for the first 5 years as a developer? You have to write all the code yourself. I don't know.

If I were a developer with between 2 and 20 years of experience, I would be asking that question of myself a lot, because otherwise you might be in the process of making yourself obsolete.

Yeah. Well, this is another thing about knowledge that this César Hidalgo guy said. He said that knowledge is non-fungible, which means it can't be exchanged. What he means by that is that the process of learning is, in some important sense, not reducible.

You have to have the experience, and the experience has to have friction. When we build models of the world, we learn; there's this phrase, “reality pushes back.” We make lots of mistakes, we update our models, and we're just placing these coherence constraints in our model. That's how we come to learn.

So you use Claude Code, and there's so little friction in the process. That's exactly what this study from Anthropic said: there was so little friction that they didn't learn anything.

Right. Yeah. No, exactly. Desirable difficulty is the concept that comes up in education. But even going back to the work of Ebbinghaus, who was the original spaced-repetition learning guy in the 19th century, and Piotr Wozniak more recently, we find the same thing: memories don't get formed unless it's hard work to form them.

That's where you get this somewhat surprising result that revising too often is a bad idea because it comes to mind too quickly.

And so, with repetitive spaced learning, with stuff like Anki and SuperMemo, the algorithm tries to schedule the flashcards just before the moment you’re about to forget. So then it’s hard work. I studied Chinese for 10 years in order to try to learn about learning myself. I really noticed this: I used Anki, and because it was always scheduling my cards just before I was about to forget them, it was always incredibly hard work.

It was always incredibly hard work to do reviews because almost all the cards were ones I was on the verge of forgetting. It was absolutely exhausting. But my God, it worked well. Here I am. I haven’t really done any study for 15-plus years, and I still remember my Chinese.

Yeah.

It was always incredibly hard work to do reviews because almost all the cards were ones I was on the verge of forgetting. It was absolutely exhausting. But my God, it worked well. Here I am. I haven’t really done any study for 15-plus years, and I still remember my Chinese.

Well, I know. Coming back to your radiology example, one example people give is call centers. We have this notion that, in an organization, we have high-intelligence roles and low-intelligence roles. For me, intelligence is just the adaptive acquisition and synthesis of knowledge.

We assume that the low-intelligence roles doing the call-center stuff don’t adapt, which means there are certain things an organization does that don’t change, so we could automate them and we don’t need to update our knowledge. I think that discounts the fact that, with the radiology example, having this holistic knowledge matters. In a call center, so many weird edge cases come in and so many weird things happen, and that information filters up through the organization and we adapt over time.

When you start to automate things and actually lose the competence to create the process that created the thing in the first place, you lose the evolvability of that knowledge in the organization. You’re actually kind of cutting your legs off.

Yeah, absolutely. In my company, I tell our staff all the time: almost the only thing I care about is how much your personal human capabilities are growing. I don’t actually care how many PRs you’re doing or how many features you’re doing.

There’s that nice John Ousterhout, the Tcl guy, who recently released some of his Stanford Friday Takeaway lectures. He has this nice one called “A Little Bit of Slope Makes Up for a Lot of Intercept.” Basically, the idea is that in your life, if you can focus on doing things that cause you to grow faster, it’s way better than focusing on the things that you’re already good at. That has a high intercept.

Yeah. So the only thing I really care about, and I think is the only thing that matters for my company, is that my team—I’m focusing on their slope.

Yeah. If you focus on just driving out results at the limit of whatever AI can do right now, you’re only caring about the intercept. I think it’s basically a path to obsolescence through a company and the people who are in it.

Yeah. And I’m really surprised how many executives of big companies are pushing this now because it feels like, if they’re wrong—which they probably are—and they have no way to tell if they are because this is an area they’re not at all familiar with, if they never learned it in their MBAs, they’re basically setting up their companies to be destroyed.

I’m really surprised that shareholders would let them do that, setting up such an incredibly speculative action. Here we are. It feels like a lot of companies are going to fail as a result of the amassed tech debt that causes them not to be able to maintain or build their products anymore.

There are loads of folks out there like François Chollet. He really gets it. He understands this, and he’s always said that it’s about this kind of memetic sharing of cognitive models about the domain and how we refine them together.

This is another big scaling problem with generative-AI coding, right? The ideal case is that I’ve done this, I know a domain really well, I can specify it with exquisite detail, and I tell Claude Code, “Go and do this thing.” The model in my mind doesn’t matter.

Then you go into an organization, and now I need to share my knowledge with all of the other people. I’m sure you have this in your company as well: this knowledge-acquisition bottleneck is a serious problem in organizations. When it’s just me, I think I’m probably about 50 times more productive using Claude Code. It’s absolutely magic, and I can see why people are so excited about it.

But people don’t seem to understand the bottleneck and how that doesn’t really translate to many real-world organizations.

No one’s actually creating 50 times more high-quality software than they were before. We’ve actually just done a study of this, and there’s a tiny uptick—tiny uptick—in what people are actually shipping. That’s the fact.

Obviously, I’m an enthusiast of AI and what it can do, but my wife, Rachel, recently pointed out in an article that all of the pieces that make gambling addictive are present in vibe coding.

Yeah. dark flow. I was going to bring that up. You have to tell us about

Coding.

Yeah,

It’s this really awkward situation where almost everybody I know who got very enthusiastic about AI-powered coding in recent months has totally changed their mind when they finally went back and looked at how much of the stuff they built during those days of great enthusiasm they’re using today, whether their customers are using it today, and whether they’re making money from it today. Almost all the money is being made by influencers or by the companies that produce the tokens.

The thing about AI-based coding is that it’s like a slot machine in that you have an illusion of control. You can craft your prompt and your list of MCPs and your skills and whatever, but in the end, you pull the lever, right? You put in the prompt, something comes back, and it’s like, “Cherry, cherry.” It’s like, “Oh, next time I’ll change my prompt a bit. I’ll add a bit more context.”

Pull the lever again. Pull the lever again. It’s stochastic. You get the occasional win that’s like, “Oh, I won. I got a feature.” So it’s got all these hallmarks of loss disguised as a win, a somewhat stochastic feeling of control, and all the stuff that gaming companies try to engineer into their gaming rooms.

None of that means that AI isn’t useful, but gosh, it’s hard to tell.

I know. And Rachel, just to be clear, she also said that one of the hallmarks of gambling is that you kind of delude yourself into thinking you have some awareness of what’s going on, but actually you don’t.

But let’s do the bull case a little bit, though. I do think in restricted cases it is very useful, and these are cases where we understand and can place constraints and specifications. But even in those cases, you could argue, on the one hand, that we’re not going to be unemployed anytime soon because you just do more work.

On the addiction thing, I’ve noticed that I’ve had 14-hour Claude Code marathon sessions, and I actually feel addicted to it. It’s like a slot machine. It really is.

Been there, too. Absolutely.

Yeah, I know. I’ve never felt more drained writing code. I actually need to take a rest afterward, like a few days’ rest, because it completely—

Was crap, you know. Yeah, definitely. I’ve had some successes, right? In fact, we’ve spent the last couple of years building a whole product based around where we know the successes are going to be, which is when you’re working on reasonably small pieces that you can fully understand, that you can design, and that you can build up your own layers of abstraction to create things that are bigger than the parts that you’re building out of.

I had a very interesting situation recently where it was kind of an experiment, basically. We rely very heavily on something called the IPython kernel, which is the thing that powers Jupyter notebooks. There had been a major version release of the IPython kernel from 6 to 7, and it stopped working in both of the products that we were trying to use it with.

One was called Jupyter Notebook Classic, which is the original Jupyter notebook, and the other is our own product called SolveIt. It would just randomly crash. IPython is over 5,000 lines of code. It’s very complex code, with multiple threads, event loops, interfaces with IPython and Python, ZMQ, all kinds of different pieces, and debugpy.

I couldn’t get my head around it, and I couldn’t see why it was crashing. The tests were all passing. I wondered if AI could solve this. I’m always interested in the question of how big a chunk AI can handle on its own right now.

The answer turned out to be yes. I think it can. I spent a couple of weeks—I didn’t develop a lot of understanding about how the IPython kernel really worked in the process, but I did spend quite a bit of time pulling out separate components.

The answer was that, in 2 hours, Codex 5.2—I think it was 5.2 at that time, or maybe 5.3 had just come out—couldn’t do it. But if I got the $200-a-month GPT-5.3 Pro to fix the problems, it could.

By rolling back between those 2 pieces of software, those 2 models, I could get things working over a couple-of-weeks period. Like you say, it wasn’t at all fun. It was very tiring, and it felt stressful because I wasn’t really in control.

But the interesting thing is that I now am in a situation where I have the only implementation of a Python Jupyter kernel that actually works correctly, as far as I can tell, with these new Version 7 protocol improvements.

And now I’m like, well, this is fascinating because we don’t have a kind of software engineering theory of what to do now. Here’s a piece of code that no one understands. Am I going to bet my company’s product on it? The answer is, I don’t know, because I don’t know what to do now. No one has been in this situation.

Does it have memory leaks? Will it still work in a year’s time if there’s some minor change to the protocol? Is there some weird edge case that’s going to destroy everything? No one knows because no one understands this code. It’s a really curious situation.

I mean, first of all, we should acknowledge the pernicious erosion of control. At the very beginning, you have 10% AI-generated code, and then you can just see how it creeps up and up. At some point, 6 months down the line, a PR comes in, and now 60% of the code is AI-generated. Do you see what happens? You slowly become disconnected.

But the bull case for this is, in AI, there’s this idea called functionalism: We don’t care what the intelligent thing is made out of. As long as it does all of the right things, then we would say it’s AI. It’s the same thing with software.

So the bull case is, I understand the domain. I don’t need to know how to write the quicksort algorithm. I just need to understand it, right? I just need to have all of these tests, and it needs to go into deployment, and these things need to happen. At that point, what I don’t actually care.

I quite like that framing, but what that actually does is say, wow, software engineering sure is important, because software engineering is all about finding what those pieces are and how they should behave, then how you can put them together to create a bigger piece, and then how you can put them together to create a bigger piece. If we do that well, then in 10 years’ time, we could have software that is far more capable than anything we could even imagine today.

But you’re already going to get that with really great software engineering. Yeah, you want to be careful.

I think, in the end, the IPython kernel I’m finding, for example, is just too big a piece, right? Because, in the end, the team that made the original IPython kernel were not able to create a set of tests that correctly exercised it. Therefore, real-world downstream projects, including the original nbclassic, which is what the IPython kernel was extracted from, didn’t work anymore.

So this is where our focus is now on the development side at Answer.AI: finding the right-sized pieces and making sure they’re the right pieces. Knowing how to recognize what those pieces are, how to design them, and how to put them together is actually something that normally requires some decades of experience before you’re really good at it. Certainly, it’s true for me. I reckon I got pretty good at it after maybe 20 years of experience.

Yeah, it’s a big question: How do you build these software engineering chops, which are now even more important than they’ve ever been before? They’re the difference between somebody who’s good at writing computer software and somebody who’s not. That feels like a challenging question.

I know. And there’s also this notion that there are so many different ways to abstract and represent something. The world is a very complex place. Maybe the way we’ve been abstracting and representing software is mostly a reflection of our own cognitive limitations, right?

Even in the sciences, in physics, you tend to have a lot of quite reductive methods of modeling the world. Then you’ve got complexity science, which is just embracing the constructive, dissipative, gnarly nature of things.

I think a lot of software today, we don’t understand. For example, there are many globally distributed software applications that use the actor pattern, and this is basically like a complex system. The only way we can understand it is by doing simulations and tests, because no one actually knows how all of these things fit together.

So you could argue, I guess, as a bull case, that maybe we already are doing this at the top of software engineering, and that is what we want to do eventually anyway.

Yeah, I’d say probably not.

You see companies like Instagram and WhatsApp dominate their sectors while having 10 staff and beating companies like Google and Microsoft in the process. I would argue this way of building software in very large companies is actually failing. I think we’re seeing a lot of these very large companies becoming increasingly desperate.

For example, the quality of Microsoft Windows and macOS has very obviously deteriorated greatly in the last 5 to 10 years. Back when Dave Cutler was looking at every line of the NT kernel and making sure it was beautiful, it was an elegant and marvelous piece of software. I don’t think there’s anybody in the world who’s going to say that Windows 11 is an elegant and marvelous piece of software.

So I actually think we do need to find these smaller components that we do fully understand and build them up. Here’s the problem: AI is no good at that. I say that empirically. They’re really bad at software engineering.

I think that’s possibly always going to be true because we’re asking them to often move outside of their training data. If we’re trying to build something that literally hasn’t been built before and do it in a better way than has been done before, we’re saying, “Don’t just copy what was in the training data.”

Again, this is a confusing point for a lot of people because they see AI being very good at coding, and then they think, “Oh, that’s software engineering. It must be good at software engineering.” But they’re different tasks. There’s not a huge amount of overlap between them, and there’s no current empirical data to suggest that LLMs are gaining any competency at software engineering.

Every time you look at a piece of software engineering they’ve done, like the browser, for example, which Cursor created, or the C compiler, which Anthropic’s Claude created, I’ve read the source code of those things quite a bit. Chris Lattner is much more familiar with the compiler example than me, but they’re very, very obvious copies of things that already exist.

So that’s the challenge: If you want to build something that’s not just a copy, then you can’t outsource that to an LLM. There’s no theoretical reason to believe that you’ll ever be able to, and there’s no empirical data to suggest that you’ll ever be able to.

Yes. I think the punchline of this conversation is—and I’m sure you would agree with this—that we need to have the combination of AI and humans working together, right? Humans provide the understanding and all of the stuff we were saying about knowledge, but we can still use AI as a tool.

We need to design operating models, or ways of working, that make sure we don’t diminish our competence and understanding, right?

So it’s a very fine line.

That’s been our focus, and we both focus on that for teaching and for our own internal development.

The stuff I’ve been working on for 20 years has turned out to be the thing that makes this all work. We should give credit for this to the guy who created the notebook interface. Although lots of ideas go back to Smalltalk, Lisp, and APL.

Basically, the idea is that a human can do a lot more with a computer when the human can manipulate the objects inside that computer in real time, study them, move them around, and combine them together. That’s what Smalltalk was all about, with objects, and APL was the same with arrays.

Mathematica basically is a superpowered Lisp, which then also added this very elegant notebook interface that allowed you to construct a kind of living document out of all this.

I built this thing called nbdev a few years ago, which is a way of creating production software inside these notebook interfaces, inside these rich, dynamic environments, and I found that made me dramatically more productive as a programmer. Today, even though I’ve never been a full-time programmer as my job, when you look at my GitHub repo output, I think GitHub produced some statistics about it, and I was just about the most productive programmer in Australia. It’s working, and a lot of the stuff I build has lots and lots of people using it because it’s such a rich, powerful way to build things.

So it turns out we’ve now discovered that if you put AI in the same environment as the human, again in a rich, interactive environment, AI is much better as well, which perhaps isn’t shocking to hear.

If you use Claude Code, which I know you do, and it’s a very good piece of software, the environment we give Claude Code is very similar to the environment that people had 40 years ago. It’s a line-based terminal interface. It can use MCP or whatever. Most of the time, it nowadays just uses Bash tools, which, again, are very powerful. I love Bash tools. I use CLI tools all the time, but it’s still just using text files as its interface to the world. It’s really meager.

So we put the human and the AI inside a Python interpreter, and now suddenly you’ve got the full power of a very elegant, expressive programming language that the human can use to talk to the AI.

The AI can talk to the computer. The human can talk to the computer. The computer can talk to the AI. You have this really rich thing. Then we let the human and the AI, in real time, build tools that each other can use.

That's what it's about to me. It's about creating an environment where humans can grow, engage, and share. For me, when I use SolveIt, it's the opposite of the experience you described with Claude Code. After a couple of hours, I feel energized, happy, and fulfilled.

I'll give you my take. I think the thing you're pointing to here is that there's something magic about having an interactive, stateful environment that gives you feedback.

And that is because our brains can do a certain unit of work. We think through refining and testing with reality. During my PhD, I used Mathematica and MATLAB, and I agree: we've got this REPL environment—here's the matrix, do an image plot, change this, and this is what it looks like now—and it's a wonderful way to refine my mental model about something.

But Claude Code does a lot of this stuff. I think it's mostly a skill issue. I think the people who use Claude Code effectively do this. I've written a content management system.

It's possible. Yeah. So, you know, I've written a content management system called ReScript. When I'm putting together a documentary video, it can pull transcripts, and then I can verify the claims.

Part of AI literacy is just understanding the asymmetry of language models, right? When you give them a discriminative task, they're actually quite good. If I tell it, in a subagent, to go and verify every individual claim, it's much more accurate than if I were in generation mode and generating a bunch of claims.

And the stateful feedback thing, again, I can have some kind of schematized XML dump, and I can have an application on the side that's visualizing it. It's a feedback loop, and for me, this is an AI literacy thing. The good people in AI are already doing this.

Yeah. So I don't fully agree with you. I agree that you can do it in Claude Code, and I agree it is an AI literacy thing as to whether you can do it, but also Claude Code was not designed to do this. It's not very good at it, and it doesn't make it the natural way of working with it.

I don't want to say it's an AI literacy problem because that's like saying, “Oh, it's a you problem.” To me, if a tool is not making it natural for a human to become more knowledgeable, happier, more connected, with a deeper understanding and a deeper connection to what they're working on, that's a tool problem. That should be how tools are designed to work.

Many models and tools are expressly being evaluated on, “Can I give it a complete piece of work and have it go away and do the whole thing?” That feels like a huge mistake to me, versus evaluating whether a human comes out the other end with a deep understanding of a topic, so that they can really easily build things in the future.

I agree with all of that, but then there's the other interesting angle: there was a famous talk by Joel Grus—and we'll talk about this—and he said that notebooks are terrible. They're really bad from a software engineering point of view, and at the time, and maybe still now to a certain extent, I agree with him.

I've done ML DevOps. I've worked in large organizations, trying to figure out how we bridge data science and software engineering. Claude Code is already more toward the software engineering side, and what that means is it creates idempotent, stateless, repeatable artifacts. So, as you say, from a pedagogical point of view, it's really good having this stateful feedback because I can understand what's going on, but then I need to translate that into something deployable.

Can you tell us the story of how you responded to Joel Grus? It was a bit of a fiasco, wasn't it? Just tell us about that story.

He did a really good video called “I Don't Like Notebooks.” It was hilarious and really well done. I was totally wrong. All the things he said notebooks couldn't do, they can, and all the things he said you can't do with notebooks, I do with notebooks all the time. So it was a very good, very amusing, incorrect talk.

Then I did a kind of parody of it called “I Like Notebooks,” in which I basically copied, with credit, most of his slides and showed how every one of them was totally incorrect.

But I actually think your comment about it comes down to the heart of it, which is this difference between how software engineering is normally done versus how scientific research and similar things are normally done. I think—and I agree—there is a dichotomy there, and I think that dichotomy is a real shame because I think software development is being done wrong.

It's being done in this way that's all about reproducibility and these dead pieces. It's all dead code, dead files. I will never be able to express this one-millionth as clearly as Bret Victor has in his work, so I'd encourage people who haven't watched Bret Victor to watch him. He shows again and again how a direct, visceral connection with the thing you're doing is all that matters. That's his mission: to make sure people have that connection, and that's basically my mission as well.

So for me, traditional software engineering is as far from that as it is possible to get. I think it's gross. I find it disgusting, and I find it sad that people are being forced to work like that. I think it's inhumane, and I just don't think it works very well. Empirically, it doesn't work very well. It's much less good for AI as well as much less good for humans.

It hasn't always been that way. With Alan Kay and Smalltalk, Iverson and APL, Lisp, and Stephen Wolfram with Mathematica, to me these were the golden days, when people were focused on the question of how we get the human into the computer to work as closely with it as possible.

That's where the mouse came from, for example: to click and drag and visualize entities in your computer as things you can move around. So I feel like we've lost that. I think it's really sad.

With Claude Code and stuff, the default way of working with them is to go super deep into it. There's a whole folder full of files; you never even look at them. Your entire interaction with it is through a prompt.

Yeah.

It literally disgusts me. I literally think it's inhumane, and my mission remains the same as it has been for about 20 years, which is to stop people working like this.

I know. But casting my mind back, I used to work with data scientists. They were using Jupyter notebooks, and what I found was that, typically, back then, if you checked them into Git, it wouldn't look very good. Most of these data scientists didn't know how to use Git. They would run the cells out of order, which meant it wouldn't be reproducible. There were all sorts of things like that.

But the thing is, I agree with you that you can use them in this workflow. It comes back to what I was saying before about the call center and it being a low-intelligence job. The data scientists—the reason why they're doing intelligent work is that they're actually creating something that doesn't exist. They're figuring out the contours of a problem. They're actually working in a domain that is poorly understood.

You could argue now that the bull case is that when data scientists can succinctly describe the contours of the problem, maybe we could go to Claude Code and implement it properly. But how do we bridge between those two worlds?

I think that'd be a terrible, terrible idea. You don't want to remove people from their exploratory environment. Research and science are developed by people building insight. Whoever you listen to, whether it be Feynman or whoever, you always hear from the great scientists how they build deeper intuition by building mental models, which they get over time by interacting with the things they're learning about.

In Feynman's case, because it was theoretical physics, he couldn't actually pick up a spinning quark, but he did literally study spinning plates. You have to find ways to deeply interact with what you're working with.

So many times I've seen data science teams—because you're right, data science teams aren't very familiar with Git and aren't very familiar with things that they do need to understand—and so often I've seen a software engineer become their manager. Their fix to this will be to tell them all to stop using Jupyter notebooks. Now they have to use all these reproducible blah-blah virtual environments, blah-blah. They destroy these teams over and over again. I've seen this keep happening, because the solution is not to create more discipline and bureaucracy; it's to solve the actual problem.

For example, we built a thing called an nbdev merge driver. A lot of people don't realize this, but actually notebooks are extremely Git-friendly. It's just that Git doesn't ship with a merge driver for them. Git only ships with a merge driver for line-based text files, but it's fully pluggable.

And so, you can easily plug in one for JSON files instead. So we wrote one. Now when you diff—when you get a Git diff with our merge driver—you see cell-level diffs. If you get a merge conflict, you get cell-level merge conflicts. The notebook is always openable in Jupyter.

nbdime did the same thing, so there are 2 independent implementations of this. There were problems to solve, but the solution was not to throw away Brett Victor's ideas and move people further away from their exploratory tools; it was to fix the exploratory tools. I think all software developers should be using exploratory-based programming to deepen their understanding of what they're working with, so that they end up with a really strong mental model of the system they're building and working with. Then they can come up with better solutions, more incrementally and better tested.

I basically never have to use a debugger because I basically never have bugs. It's not because I'm a particularly good programmer; it's because I build things up in small steps, and each step works. I can see it working and interact with it, so there's no room for bugs.

I'm so torn on this because I agree with you, and I'm also skeptical of people who say that organizations converge onto ways of doing things and no longer need to evolve. They no longer need to adapt. Innovation is adaptivity, right? We should increase the surface area of adaptivity as much as we possibly can. So we need people who are constantly testing new ideas and finding these constraints. But by the same token, we need to use the cloud. We need to use CI/CD. We need to get this stuff into production.

Yeah. So, absolutely do both. nbdev ships with out-of-the-box CI integration, and the tests are literally there. Because the source is a notebook, the entire exploration of how the API works, what it looks like when you call it, the implementation of the functions, the examples of them, the documentation of them, and the tests of them are all in one place. It's much easier to be a good software engineer in this environment. So, yeah, do both.

Do you remember there was that statement that existential risk should be an urgent priority, and it was signed by folks like Hinton and Demis? You responded basically with a rebuttal. That was with Arvind, you know, the snake-oil guy. Tell me about that. Do you think we should be worried about AI existential risk?

I mean, that was a certain time, wasn't it? I feel like things have changed a bit. Thank God. I feel like we—not just me and Arvind, but broadly speaking, the community of which we're a part—probably won that. Now we have other problems to worry about.

But basically, at that point, the prevailing narrative was, “AI is about to become autonomous. It could become autonomous at any moment and could destroy the world.” That very much comes from Eliezer Yudkowsky's

Work, which—

I think clearly has been shown to be wrong at many levels to this point.

They would refute that, obviously. Of course they would.

Yeah.

It's one of those things that they can always refute, just like any doomsday cult, unless you give it a date and the date passes.

Well, even I've updated a little bit. I now think I would say that these models can be said to be intelligent in restricted domains. The ARC Challenge showed that. So, if you place constraints into the problem, you can go faster toward a known goal. Even with agency, you can put a planner on there, and if you know where you're going, you can get there faster.

But that doesn't help you. You can have all the intelligence and agency in the world, but if you don't have the knowledge and the constraints, then you're going in the wrong direction faster. I think they don't seem to appreciate that these models don't actually know the world.

None of that was even relevant to Arvind's and my point, which was and is that it's a misunderstanding of where the actual danger is. The danger is that when you have a dramatically more powerful technology entering the world that can make some people dramatically more powerful, people who are in love with power will seek to monopolize that technology. The more powerful it is, the stronger that urge from those power-hungry people will be.

So, here's the problem: if you're like, “I don't care about any of that. All I care about is autonomous AI taking off—singularity, paperclip, nanogoo, whatever”—the obvious solution to that is, “Let's centralize power.” This is what we kept seeing, particularly at that time: let's give either very rich technology companies or the government, or both, all of this power and make sure nobody else has it. In my threat model, that's the worst possible thing you can do, because you've centralized the ability to control in one place, and therefore these people who are desperate for power just have to take over that thing.

Could we distinguish, though, what you mean by power? Because we've just spent some of this conversation talking about how it's not actually as powerful as people think it is.

But I'm not even—mine is an even-if thing, right? I'm just saying, even if it turns out to be incredibly powerful, I don't even want to argue about whether it's going to be powerful, because that's speculative. Even if it's going to be incredibly powerful, you still shouldn't centralize all of that power in the hands of 1 company or the government.

Yeah. Because if you do, all of that power is going to be monopolized by power-hungry people and used to destroy civilization. Basically, you'll end up with a case where all of that wealth and power will be centralized with the kinds of people who want it centralized.

Society has faced this again and again for hundreds of years. When writing used to be something that only the most exclusive people had access to, the same arguments were made: if you let everybody write, they're going to use it to write things that we don't want them to write, and it's going to be really bad. Ditto with printing, ditto with the vote. Again and again, society has to fight against this natural predilection of the people that have the status quo power to say, “No, this is a threat.”

So, when we're saying, “Okay, what if AI turned out to be incredibly powerful?” would it be better for society for that to be kept in the hands of a few, or spread out across society?

My argument was the latter. Now, there's also an argument which is, “Don't worry about it. It's not going to be that powerful anyway.” I just didn't want to go there because it's not an argument that's easy to win: you can't really say what's going to happen. We're all just guessing.

But I can very clearly say, well, if it happens, would it be a really good idea to only let Elon Musk have it, or would it be a good idea to only let Donald Trump have it?

Dan Hendrycks spoke about this offense–defense asymmetry. It's actually very important for us to have countervailing defenses. But let's just take that as a given for a minute, because obviously when we look at something like Meta and Facebook, it's quite clear what the power imbalance is. They control all of our data. They know what we're doing with something like OpenAI and Claude.

It's not as good as we thought it was because humans still need to be involved. But, for example, they have all of our data, right? You might be working on some new innovative technology, and you're using Claude and sending all of your information up there, and they can now copy you. What kind of risks are you talking about, to be more concrete?

Jeremy Kahn

Yeah. No, I was not talking about any of those things, right? At the time, I was talking about this speculative question of what if AI gets incredibly powerful? Now, for example, they say that this is the new means of production, and that seems completely hyperbolic to me.

In your best estimation now, if there are risks, what are they?

If there are risks with the current state of technology, I think some of them are the ones we've discussed, which is people enfeebling themselves by basically losing their ability to become more competent over time. That's the big risk I worry about the most.

The privacy risk is there, but I'm not sure it's much more significant than it was for Google and Microsoft before. You used to work at Microsoft, so you know how much data they have about the average Outlook, Office, and so on user. Ditto for Google—the average Google Workspace or Gmail user. Those privacy issues are real, although I think there are bigger privacy issues around these companies, which the government can outsource data collection to.

Back in the day, it used to be companies like ChoicePoint and Acxiom. Nowadays, it's probably more companies like Palantir. The U.S. government is actually prohibited from building large databases about U.S. citizens, for example. But it's not prohibited from contracting companies to do so, and companies are not prohibited from doing so. So, I mean, that's a huge worry, but I don't think it's one that AI is uniquely creating.

You're in the U.K., as you know. In the U.K., surveillance has been universal for quite a while now. It certainly makes it easier to use that surveillance, but a sufficiently well-resourced organization could just throw 1,000 bodies at the problem.

I'm not sure these AI privacy problems are maybe more common than they used to be.

Yeah, Jeremy, I've just noticed the time. I need to get to the airport.

All right.

This has been amazing. Thank you, sir. Thank you for coming.

Jeremy Kahn

Yeah.

Hope you had a nice trip. Thank you so much.