《存在性希望》播客：我对 AI 未来的积极愿景

Beatatric Urkers

I’m very happy today to be joined by Nathan Labenz, who runs the Cognitive Revolution podcast. We were just talking briefly before starting this recording about how many episodes you’ve done. It’s so many, so it’s a bit intimidating to interview such an experienced podcast host. Feel free to direct me if you have any prompts.

Not at all. Thank you. I’m excited to be here and looking forward to this conversation. For my part, I’m basically just obsessed with AI and trying to understand it as well as I can. It’s such a horizontal technology, touching all aspects of life and society, that there’s a never-ending number of angles to approach it from.

Eight episodes a month honestly isn’t enough to get after all the angles that I would like, but it’s probably as many as anybody could reasonably produce. I certainly don’t expect anybody to listen to all of them, but it’s been a really fun learning journey for me. I honestly don’t consider myself very charismatic, so I mostly just pinch myself that anybody wants to listen to it at all. I’m looking forward to this conversation with you today.

Beatatric Urkers

Today’s angle is going to be the theme of this podcast: existential hope. I know you’ve said that the scarcest resource is a positive vision for the future, so I think that’s what we’re going to try to dig into today, especially in relation to AI. I think that’s the most interesting question right now.

If you woke up 10 or 20 years from now and we had a really good, positive future with AI, what would you see around you?

It’s a hard question, and I say that all the time. The scarcest resource is a positive vision for the future. I really do mean that, and I don’t think that I’m particularly advantaged in terms of having a crystal ball. Another one of my common jokes is that my crystal ball gets really foggy more than a few months out, so I’m in uncomfortable territory trying to see farther into the future than that. But here we go.

I think we are going to see just about everything change. The reason I called my podcast the Cognitive Revolution is by obvious analogy to the Industrial Revolution and the Agricultural Revolution. You go back into these earlier periods for these previous revolutions, and what people were doing before versus after is just totally different.

At some point, we were small bands of hunter-gatherers, always on the move and always searching for food, kind of living literally hand-to-mouth. Then we figured out how to grow food, and that created a whole different thing at a much bigger scale at which people could come together and live together. Those economies of scale created the beginning of the technology exponential, which looked really flat for a long time but seemingly was already on an exponential even then, when people didn’t know it.

The same thing happened again with the Industrial Revolution. Mechanizing farming took us from a scenario where 80% or 90% of people were growing food to where, today, I think only about 2% of people in the developed world are needed to grow food because machines can do a lot of that work.

I think the same thing happens for cognitive labor. If AI were to stop progressing today, I think we already have powerful enough AI to automate the majority of cognitive work. We don’t have everything wired up in the right way, and we don’t have all the data structured in the right way for AI to consume it. There’s a lot of plumbing and implementation work that would need to be done to realize that dream of automation, and that would take 5 to 10 years—and probably longer, because I always tend to underestimate the timeline to implementation.

But I think we do have sufficiently powerful AI that we could automate a majority of cognitive work already. Then the question is: What are we going to do if everybody has moved from roving around to settling down and farming, then from farming to factory jobs, and then from factory jobs to white-collar cognitive jobs, or at least a significant part of the economy? What do people do if AIs can handle the majority of that?

I don’t know. One candidate answer is the caring economy, broadly. That could be the next big thing. I think that’s a little bit challenging even there, because you do see these studies and survey results that often show people prefer talking to AIs for a lot of things.

AI doctors, for example, tend to get higher ratings on bedside manner than human doctors because they have some unfair advantages. They can be infinitely patient. They aren’t time-bound in the way that human doctors are, so they’ll answer all your questions, and they don’t really have any constraints on how much time they can spend with you.

I wouldn’t say that’s exactly a fair head-to-head comparison, but there are some real advantages there. I’m not sure how that shakes out.

Teaching is another area where I think that, for as long as the world is at all recognizable, we will want humans to be role models for the next generation of humans. But we already see schools where AIs are starting to be responsible for all the content, and humans are moving into more of a guide, coach, mentor, and motivator role.

The AI gives you the lessons and grades your homework, and the humans are there for these softer skills. Is there enough demand for that that people will need to do it for jobs, and will those jobs absorb all the people who are probably going to get displaced from the jobs they’re currently doing? I don’t know. That seems like a stretch, and it’s kind of a rocky transition, but that’s at least one answer.

I think another answer is that we might actually have the life of leisure that people have dreamed about for a long time. Famously, Keynes said 100 years ago that by the time we got to today, we should be working 15-hour weeks. We’re obviously not, but maybe that’s another way that things could go.

Zuckerberg has had some really interesting ideas. As with many things in AI, I have very mixed and ambivalent feelings about what Zuckerberg is up to, but in his post and message where he introduced the personal superintelligence concept, one of the things he pointed out that I thought was really interesting is the macro trend that people are spending less time working and more time socializing, creating, and consuming media.

Maybe consuming media is crowding out connecting and socializing a little bit, which isn’t necessarily all to the good. But the big trend of the shift from work to leisure may accelerate. Maybe people start to do a lot of things in VR and AR, and maybe Neuralink becomes a very broad technology.

These experiences, especially if they’re literally connected into your brain, could start to be extremely compelling.

Maybe we're spending a lot of time in VR is one answer for the future 10 years from now. Beyond that, it's really hard to say, right? Classically, people are like, well, we didn't know what the cell phone was going to bring us. Nobody had Uber in mind when we introduced the iPhone, so what are the apps that are going to be built on the AI technology foundation? I think it's very early and very hard to say.

But yeah, I think one other thing that could be really interesting is a sort of radical egalitarian mode of access to frontier technology. I mean, I often use the example of doctors. Again, it's a scarce resource today. Obviously, not everybody can become a doctor. It takes a long time, a lot of training, and it's very expensive, yada yada. Not everybody can access a good doctor.

With the AI technology, that should change, and people should be able to get quality medical advice regardless of their means. So that's exciting. I was reminded, in thinking about this, of the Andy Warhol quote where he goes on about how the great thing about American consumer culture—this was back in, I think, the '60s—is that everybody can get the same stuff. The president drinks Coke, movie stars drink Coke, and you can get your own can of Coke and know that it's the same as the one they're drinking. It's all the same. Even if you were richer, you couldn't get a better Coke.

That's been true of the iPhone. I don't think that'll be true of AI in exactly the same way, because I do think there will probably be frontier, very high-powered systems that, frankly, not everybody needs on a daily basis, but there are still uses for them. But if you think about that VR world, one of the big differences right now between the haves and have-nots is just the sort of experiences that they can access.

That could perhaps become really collapsed, where the exciting life of adventure that is currently only available to the select few could perhaps be made scalable through some combination of Neuralink-type connections and VR, all delivered in a low-energy, low-resource way that could scale to potentially everyone in the way that Coca-Cola did years ago.

So I'm probably going to get a lot more wrong there than right, but those are at least some musings about just how different the future could be. And it's coming at us fast. This technology—the Industrial Revolution took like 200 years, depending on how you want to count. The electrification of the United States took 60 years, from 1880, when electricity was invented, to 1940, when basically everybody finally had electricity.

The huge difference there was they had to actually build out the wires to everybody's house. Now we already have the wires that deliver the AI to the point of consumption, and so you can have these centralized upgrades where, from one version to the next, the capability leap and what everybody at scale can access can flip much faster than in previous generations.

So I think it's going to be a wild ride, exciting and also a little bit scary.

It's both a very exciting time to be a human and probably challenging coming up, at least. There's a lot of threads to pull on. Thank you for being so concrete about these things. Is there anything that you're personally just so excited for?

Well, I've dreamed of self-driving cars since I was a kid. I used to sit in the back seat with my mom or dad driving, and so often we'd be sitting at a red light and nobody's going the other way. That bothered me so much, even as a kid. Even then, it felt like if we had a little more will, that was probably solvable. Even without AI, you don't need AI to change the light when it's clear that nobody's coming, right?

That may be a theme as we get into this: what is it going to take to be successful? A little more societal will—collective will—to demand better, higher expectations from the public, I think, is one thing that could be really critical to realizing the good future and making sure we don't get bogged down, as we have at times.

But now we've got self-driving and, again, it works. Waymo is amazing. I don't know if you've used one, but it's been a while, actually, since my last Waymo ride. Fully autonomous. You summon it with the app. It shows up. Nobody's in there. You get in, and it drives you where you want to go.

What was really striking to me was how quickly I got bored with it. I had been thinking about this for literally decades, but I found myself checking my phone 5 minutes into the ride and needed to intentionally remind myself, “Hey, you've been looking forward to this for a long time. Put your phone away and actually try to observe this moment and savor the first experience of real self-driving.”

But it was so good that it just felt like a background reality literally within minutes. The safety data seems to suggest that it is already a lot safer than human drivers. The price point at which it is selling in the San Francisco Bay Area, at least, is quite a bit higher than Uber. So it seems that people are willing to pay more for the self-driving experience.

Whether that's safety or because they don't want to talk to the driver, I'm not sure exactly what's driving that difference, but the difference in price seems to be pretty well established. It has been sustained for a while at this point.

But yeah, that's a huge one, right? I dream of going to visit my grandmother, who lives 4 hours away, and being able to either work or ideally sleep on the way there—just doing overnight in the car. You start to imagine the different form factors too, right? If I truly don't have to pay attention, then you can have a very wide range of car types.

You could have sleeper cars that you just get into and go to sleep, then get up and get out of bed at your destination. That's like a mobile hotel room. That alone would be an incredible improvement and should come with 30,000 fewer road deaths in the United States. I think there's like a million road deaths annually across the world.

So it's obviously going to take a while for that to be built out, but yeah, I've been waiting for that one for a long time.

Beatatric Urkers

Yeah, I agree. The first time I went in a Waymo, I was also just like, “Wow.” Quickly, you get used to it. But I think it's really one of those things that also just feels like it makes sense. I feel like when I went in one for the first time, it just felt like, “Oh, why aren't we already doing this?”

I actually did a special episode recently on autonomous vehicles and what we need to do to get them coming as soon as possible.

Imagine going to bed in your car on Friday night and waking up Saturday morning out in nature. That would be amazing as well.

If we zoom out a bit, one thing I think is interesting about existential hope is having really big visions for the future. If we think big about how good the future could be—in the previous prompt, I gave you 10 to 20 years—I'll give you 100 years if you want, or even longer. Of course, your crystal ball gets fuzzy—or foggy, maybe is the term. But if you get to dream, what do you think would be a best-case scenario, especially in relation to AI?

Yeah, that's a tough one for sure. I find the fog of AI—I don't like the term “war” to describe what's going on in AI, because I want to make sure AI developments are nothing like war for as much and for as long as possible. But the fog of war around what's going on with AI right now is a really hard thing to penetrate.

Even among people who are obviously extremely informed and very knowledgeable—even titans of the field—there are these fundamental disagreements around what currently exists and what's going to happen in the immediate term. The farther you go out into the future, the more radically difficult it gets.

I'm on board with the people who hope that we would cure all the diseases—things that were totally fantastical until quite recently. It was like, “Well, nice of you to dream that, but okay.” That stuff just seemed to be limited to the realm of dreams. Now, with AI's ability to grok what's going on at many different levels of biology, the potential for us to actually hit something like a Kurzweilian escape velocity—where every year, your life expectancy increases by more than a year—no longer seems totally far-fetched.

Even in preparing for this, I was looking at a recent paper about the creation of novel antibiotics. We haven't had many antibiotics created in a long time, but a single group at MIT just discovered multiple new antibiotics with new mechanisms of action that are effective against antibiotic-resistant microbes.

I think it's another interesting sign of the times. I swear, that sort of thing would have been all anybody could talk about if it had happened when I was a kid. Now, something as big as that can happen, and I find that most people, even those who are relatively plugged in, just haven't heard of it because there are so many other things going on. I increasingly have these blind spots too, even though I've created the job for myself of trying to keep up with all this stuff.

Curing all the diseases still sounds a bit hubristic, perhaps, but it does seem to be increasingly not totally fantastical. I would certainly like to live longer and healthier than my current life expectancy would suggest, which is obviously critical. I think it's sort of a straw man, but people who aren't exposed to this kind of thinking much will say, “Well, that'll suck. You'll be old or decrepit for all those later years.” Obviously, that's not the real hope. So that's a big one.

Becoming a multiplanetary species is also a great aspiration for humans. I definitely think Elon has become hard to defend in some ways, for sure, and I won't defend all of his actions by any means. But the general idea that we should aspire to get off planet Earth and out into space makes a ton of sense.

It's really interesting, and I don't think we're going to have good answers on this for a while. This might be one of the last questions that we have any traction on: Do we think that noncarbon forms, or something that's truly very different from us in terms of a substrate, could carry on our values, our consciousness, our intent, our volition—whatever—into space for us?

I really don't know. Another way to come at that is: Do AIs have any moral value? Are they moral patients? Do they experience anything? Does their experience matter? I'm radically uncertain on those questions.

It does seem like if you were to ask what the best way is for us to project ourselves into space, getting away from the current form of our bodies would probably be a natural part of a lot of design plans for that to happen. But I'm really unsure whether we should be confident that we could create something out of a totally different substrate and feel that it matters in the same way that we're confident we matter.

I don't know. Anyway, there's a long time to figure out some of those details, and we'll see what comes. But I think the goal of getting off Earth and getting out into space is very worthy, and we should definitely dream those kinds of big dreams.

Can AI help us get there, or does AI take the baton and actually go out and do that? There's this idea of the worthy successor, which I think is, on the one hand, a dangerous idea that we should not lean into in the immediate term without having a lot of these difficult questions answered—far more than we have them answered today.

But if we did have those answers, and I really felt like I understood where consciousness comes from and believed that these things had it and were having positive experiences, then I could imagine a sort of worthy successor that would genuinely be worthy. It might be a lot more suited to travel through space over great distances and great lengths of time.

But yeah, I don't know. That's all pretty fuzzy stuff, I suppose.

Beatatric Urkers

No, yeah. Well, I mean, fuzzy, but I think they're concrete ideas, and I think they're all very interesting. I agree that it's hard to be confident about them.

On the consciousness part, I feel like it's one of those things that, even if we obviously cannot be certain of it, it's such a big “if true” that it's worth thinking about already, to some extent, just because of that.

To scale it back a little bit and zoom back to the here and now, is there anything you think is underestimated in terms of near-term AI applications? Something that's maybe boring but transformative? Is there anything like that that you've come across recently?

Yeah. I think the inference-time scaling paradigm—I think folks like Dario Amodei and Sam Altman have been talking about this for a while—but it's hard for people to make the leap with them. Even for me, I'm always trying to keep up with what the true frontier visionaries are thinking and envisioning.

In terms of something boring but transformative, this just came up in a couple of different threads: the idea of the spreadsheet. People used to sit there with big pieces of paper and a pencil, do the calculations, and have to erase and fill things in again or whatever. Then you had the spreadsheet, where you could just make a change and have that change propagate through all the calculations in an auto-updating way.

This idea of auto-updating, or things that are running in the background for us, could give us a ton of value in a world where life is still mostly recognizable. For one thing, imagine a 2nd opinion for everything.

I kind of live this way myself today. If I'm going to send important correspondence, if I'm working on some sort of deal, or if I get a contract from someone that I have to sign, I'll take that contract, run it through 3 or 4 AIs, and say, “I'm one party to this. Here's the previous communication, and here's the contract I just got. What should I be concerned about?”

The AIs' outputs are now so much better and so fast that sometimes I'll take the 3 or 4 outputs, put them into another window, and say, “Okay, give me a single comprehensive summary of all 4 of these.” That helps deduplicate the points and make sure I get all of them.

Once I work through that and come to some idea of whether I'm ready to accept this—okay, great—but maybe I have some points I want to discuss. I'll bring that back to the AI as well and get its input.

It's not about having the AI tell me what to do, although I do think that more and more autonomous agents will be coming, and we'll certainly have more and more decision-making delegated to AIs. For the moment, it's about having that 2nd, 3rd, or 4th check on all the things that I do. That gives me the ability to move a lot faster, with a lot more confidence and accuracy in what I'm doing.

I think you could also see that in all sorts of matchmaking, whether that's economic or romantic, or even just getting together with friends. Why don't we hang out with friends every night? One reason is that coordinating that stuff takes a lot of time. By the time you're done with your workday, you're like, “Oh, I don't know who's available, whether I have time for this, or whether we can even figure it out.”

But the AIs can definitely do that sort of thing if we set them up to run in the background across a pretty wide—and certainly growing—number of different matchmaking problems. So I think that will be really interesting too.

Beatatric Urkers

It's just greasing the wheels of commerce, greasing the wheels of dating markets. All these things are relatively high friction. This is another example where exactly how you get to the good equilibrium is going to be an interesting challenge. Right? We are seeing this now in hiring.

One of the examples I always give to business owners for things they should be doing with AI that they're probably not yet is that they should have an agent going out and searching for candidates they might want to proactively reach out to all the time. Every CEO of a startup or midsize company—if you said, "Hey, should you be doing more proactive recruiting?"—they basically all say, "Yeah, we should ideally, right?" But who has time for that?

Similarly, with cold outbound and sales prospecting, all else equal, if you could just add on 10 good, targeted emails to possible new customers every day, would you do that? Yeah, you probably should do that. No doubt. But again, who has the time?

The AIs do have the time. There is the question now of how we deal with all that volume on the receiving end. Companies are starting to report that it is getting harder to separate the real résumés, so to speak, from the AI résumés. There are interesting ideas about maybe having to pay $1 to apply to a job to limit the spray-and-pray approach.

Something like that. I think these are the kinds of new mechanisms that are going to take some time to develop. We're going to have to encounter some of these problems, live with them for a minute, and then figure out solutions. But I firmly believe that there's just a lot of value to be unlocked in matches that are not made and deals that are not struck just because people don't have the time.

If you could, there's another big principle: some things are hard to do but easy to verify. I think a lot of deals fall into this category. It's hard to find the next customer. It's hard to find the engineer you want to hire. It may be hard to find the person you would be interested in going on a date with, whatever. But when it's presented to you, a lot of times you can recognize it pretty quickly when you see it.

If we can get the AIs to propose good ideas to us and then we can quickly verify them, I think there's a huge amount of value to be created by automating away a lot of that friction.

Yeah, you could find the best friends or partners ever if you were able to have almost everyone in the world scanned by an AI. I'm doing this a little bit in my family. My wife and I have 3 kids, and there's always the question of, "What are we going to do on the weekend? Can we get these kids out of the house?" They're 3 boys, and they go wild if they don't get out of the house.

So it has become a priority for me to find something to take them to do on ideally every weekend day. But the search for that is another one of these things that's tough. ChatGPT is pretty good at answering, "Hey, what's happening in Detroit this weekend?"

ChatGPT now even remembers my family, and it kind of knows what I've been interested in in the past. So it's really good at doing a much more comprehensive search than I would do for all the little weekend festivals and this and that. We are actually getting out more as a result of the search costs for finding something interesting to do having dropped significantly.

I would say that's a 2025 thing, and a little bit of a 2024 thing, but now it's getting good to the point where I can get the kids to bed on Friday night and be like, "Okay, AI, what should we do tomorrow so these kids are not going off the walls by midafternoon?" More often than not, I get a pretty good answer.

Beatatric Urkers

That's really interesting and a really concrete use case. Yeah, I definitely trust it more with travel planning and stuff like that these days as well, just for advice, comparing options, and things like that. Is there a specific sector that you think is maybe a bit more ripe to have AI fully integrated and shape its trajectory? I mean, do you think that science, healthcare, education, or governance are especially ripe? I'm asking both in terms of the potential for AI to transform them and which sectors you think would actually be able to deal with some change right now?

Yeah, I think it's all of the above, really. I think it's just a question of timing, both on the development side and probably on the adoption side. The canonical first answer is software engineering, and that seems to be driven by the fact that the AI developers themselves are software engineers. They're interested in solving their own problems. It's also driven by the fact that it's really, at least comparatively, easy to validate the outputs of the models in software work because it's just text. The text gets compiled, it gets run, and you get an error message really quickly.

I just saw in the last 24 hours that Replit, which is a platform that I love for building software, introduced its Agent V. One of the big things that's different about its third-generation agent from the previous version is that it now not only writes the code to build your application, but then it will spin up a browser and become your QA agent.

The thing goes through this full loop: create the application, actually try to use it yourself, find the issues—not just issues like whether it compiled or whether there was some immediate runtime error, but from a user standpoint, actually take on the role of the user, use the browser, and click in the same way that a human would use the software. Then it finds issues that way, comes back, and tries to fix them.

The tightness of that is, I think, one good heuristic for how quickly things will come to different industries. It's just how closed that loop is and how fast that feedback process can spin. Increasingly, the pattern of use that you'd have as an end user of a system like that looks a lot like the training paradigm.

I wouldn't actually sign on to the idea that pure scaling of pretraining is over. My sense is that narrative has been a little bit overblown. But clearly, there has been a big shift from just training on all the internet with pure next-token prediction to recognizing that we also need to sculpt the behavior of these things and teach them to be good at particular tasks of interest.

The way that's happening, especially in software, looks almost indistinguishable from—well, that's a little too strong. It's distinguishable, but it looks a lot like what you're doing as an end user. They give the AI a task. It takes a number of attempts to try to solve that task. If it can get it right at least 1 time, then that's enough of a signal for it to get a reward and learn to steer more toward the right solutions than the wrong solutions.

That can all happen in a pretty tight feedback loop. If you were to compare and contrast that with, say, medicine, I alluded to antibiotics earlier. There, you've got at least some part of the feedback loop that's just a lot slower. You can't run many experiments and get an answer immediately.

Maybe this is the loop we'll get close to, because a big part of how they're developing the antibiotics in the first place is with these in silico experimental setups. It's like, "Okay, we have some idea of a target in a bacterium that, if we could disable it, would kill the bacteria." From general biology knowledge, we know that if we could disable that target, it would kill the bacteria. But how could we do it?

Now you can generate huge numbers of candidate small molecules or larger molecules. You have a wide range of space to explore here. Then you can run a simulation to see whether they bind and whether they seem like they would work. Some other tools allow you to ask whether they would bind to other random things and perhaps cause collateral damage, or whether they're hyper-specific to this particular target.

You can get pretty far. I should look this up, but I think the ratio in the paper of things that they put forward as candidates to the ones that actually worked was not super high. In other words, they were able to get a pretty high hit rate out of the in silico experiments.

Still, for now, it has to go through a clinical trial process, and the ultimate feedback—that fewer people are dying from bacterial diseases—is going to take a while. A lot of things will be rate-limited, I think, by those kinds of bottlenecks. If there's any bottleneck in the system, it'll slow down the iteration relative to pure software engineering.

But the other thing that will start to happen is that you'll have these in silico experiments or simulation environments. That's also happening in self-driving a lot. They don't just train on actual data; they also augment that data in a ton of different ways and create all sorts of scenarios that they may never have encountered in the wild but that you would definitely want to be able to handle.

I've seen examples where they'll show a helicopter landing on the highway in front of the car. That may never have happened in any training data they had, but they still wanted to make sure that you wouldn't drive directly into the helicopter that just landed in front of you. So it's not to say that these offline bottlenecks have no workaround, but they're certainly a lot harder.

I guess the way I think about what's really ripe is where there are the fewest of those things. They could be social, too. In education, I'd say that already there's never been a better time to be a motivated learner.

If you turn on ChatGPT's teach and learn mode, go into voice mode, and give it access to your screen, that's by far the best way for me to learn about biology because there's so much background knowledge and so many terms. I'm trying to understand what's going on at the intersection of AI and biology, and the biology part is just—there's so much. But when the AI can look over your shoulder and you can casually say, “Hey, what's this?” or “Why does this even matter? Why are they even talking about this?” there's truly never been a better time to be a motivated learner.

Then the bottleneck maybe becomes: do we have a system that's designed to create and encourage motivated learners? What exactly is the purpose of the school system? What exactly are the incentives, and how do people in those systems understand their own interests? Those may prove to be important bottlenecks as well. But if you want to learn something, AI can help you dramatically accelerate that already today.

Beatatric Urkers

Yeah, for sure. I didn't even know that there was a teach and learn mode on ChatGPT.

It's relatively new. Yeah, I'm hoping to get the product manager from that onto the podcast to talk about it more. But there are other examples, too. Khan Academy was an early pioneer of this, and I did an episode not too long ago with the founder of a school system called Alpha School, which has become kind of famous recently.

They do academics in 2 hours in the morning, and then the afternoon is entirely enrichment: projects, field trips, group work, whatever. Kids get to explore their passions and interests. As I alluded to earlier, the AI is entirely responsible for delivering the content.

They're not soft on academics at this school. You still have to learn all the same stuff. In the U.S., we have the core curriculum, which is—I don't even know a lot about it—some sort of officially sanctioned list of the things you're supposed to learn and what public school kids are measured on. The founder of this school system is saying, “I want to show that what we're doing works on that level. We're not going off and creating our own curriculum. Our kids are scoring super high on the same exact tests that all the other kids are taking, but we're doing it in 2 hours.”

AI is doing all the content, and AI is doing all the evaluation. The adults at the school are now playing these other roles: mentor, coach, guide, et cetera. Two hours a day seems to be enough for traditional classroom delivery. The “sage on the stage” model—there are good little monikers for this in education—is just not that efficient.

So, yeah, it's cool. I think my kids are a little young for this now, but I think they will have a radically different educational experience than I did. That's for sure.

Beatatric Urkers

Yeah, yeah. I heard about Alpha School. That seems amazing, and I'm very glad that someone is doing it already, so your kids can have it soon, hopefully, as well.

I also wanted to comment on the antibiotics thing. I forgot to comment on that before. That was something that I had completely missed, for example, and I'm very interested in this space, so that's amazing.

It also reminds me of something. This podcast is the Existential Hope Podcast. It's part of the Foresight Institute, and we were co-founded by Eric Drexler and Christine Peterson. Eric Drexler, in his old book Nanosystems: Molecular Machinery, Manufacturing, and Computation, from the early '90s, wrote about the design space. He was foremost thinking about molecular machines and what we could be doing, but I think that, in silico, these things are just really, really promising and interesting—the bridge between the world of bits and the world of atoms. Hopefully, we'll see a lot more of that soon.

I wanted to move toward the Drexler idea because I know that another thing I've heard you mention a few times on your podcast is this idea he had of comprehensive AI services. It's a web of more specialized AIs rather than general agents. Do you have any takes on comprehensive AI services? I'd also be curious to hear if you have thoughts on how that compares to something like a tool AI approach, or the AI Scientist, if you've heard that one recently. How do you think about these approaches—how they differ, and which ones you think are most promising?

Yeah, it's a great question. I love the comprehensive AI services vision. I guess one thing I've observed in a couple of different realms of life is that anything in pure form is dangerous. That could be sugar purified out of naturally occurring sugar-rich food, or it could be cocaine purified out of coca leaves. You go to the Andes and people chew coca leaves their whole lives, and it's not a problem, but you purify it to cocaine and you're immediately dealing with something that's pretty dangerous. Heroin from poppy seeds is another example. There are a lot of examples of this.

I generally think that what seems to be stable in nature is some sort of ecology, some sort of equilibrium, some sort of buffered system. The way we maintain homeostasis in our bodies is through a lot of buffers, such that any insult that comes into the system runs through multiple layers of defenses. Hopefully, we have enough of those layers, and they can each push back in their own way to neutralize and ultimately be resilient to that threat.

I think the idea of the singleton and all this stuff is contentious. There are pros and cons to everything. But the idea of a singleton, or some sort of superintelligence that can do everything and is way more powerful than everything else, doesn't feel stable to me. That's not to say that it couldn't ever be achieved, but I don't like the idea of a superintelligence that's better than all humanity at every task because I have no idea how we would control such a thing.

I have to imagine that it would probably spin out of control. It might achieve its goals, to the degree that it has goals, but I have a hard time imagining that we could be in a stable equilibrium with such a thing for a long time. So I tend to prefer the idea of a more competitive, interactive, buffered system.

I think that, at least for me, is kind of at the core of this comprehensive AI services idea. It's safety through narrowness. It's not to say that the AIs aren't really good at what they do. They could be superhuman at what they do, in the same way that we have superhuman chess players that can only play chess and superhuman protein-folding AIs that can only fold proteins.

You really don't have to worry because you know what kind of inputs they can accept and what kind of outputs they can generate. You don't really have to worry that they're going to do something surprising to you. It could be surprising locally within the domain—“Oh my God, I didn't expect that this protein would ever work this way,” or whatever—but it's going to be in its lane.

I think that would be a really good design decision, if we could manage it, to try to have AIs that are potentially superhuman in their domain but are, in a pretty fundamental way, limited to their domain. That way, they don't run off and do an end run around whatever guardrails we've tried to put in place and surprise us in really negative ways.

I think the gradual disempowerment crowd might say, “Okay, even that—but what's the role for the humans in that picture?” There are definitely still some hard questions to answer there.

We're starting to see these—just in the last few months, there have been 3 new memes, new phrases coined. Gradual disempowerment is one, the intelligence curse is another, and the abundance trap is a third that I've recently come across. They all seem to be getting at this idea: if the AIs are doing everything, what are we going to do? Is there going to be an incentive for the entities—whether they be governments, maybe AIs, or who knows, corporations—to invest in people if we're not really needed to do the economically required work in the same way that we used to be?

I'm not saying that the comprehensive AI services idea solves all problems. I think people still rightly have some good questions. There's also the idea that you could have multiple delivery models for that. You could have comprehensive AI services that are decentralized in their creation and decentralized in their ownership.

The episode of the podcast coming out in 2 days is about user-owned AI through a crypto scheme that is designed—and “scheme” makes it sound, I think, less than it is.

Let's say crypto. It is kind of a scheme, but that sounds negative. Protocol, I guess, is maybe the right word: a protocol that would allow people to contribute to training and have sort of a claim on the future revenue from inference from models that they contribute to in a decentralized way. That's fascinating stuff as well.

It does create risk if people, of course, worry about weapon creation, right? That's kind of the canonical one: if everybody has an AI that can, if asked, create a bioweapon and help you distribute it, then we're going to have probably some people who are going to do that. And then what are we going to do? Can we be ready for that?

One way to answer it would be with the superhuman biodefense agent as part of the array of the comprehensive AI services. Part of being comprehensive would be having great biodefense, I suppose.

But yeah, I guess one other take on the whole comprehensive AI services thing is that it makes me quite uncomfortable that the plan among frontier AI developers seems to be basically to try to get the AIs to be able to do the AI research as soon and as well as possible. Use that to accelerate the AI research, which is already going super fast. They're like, "Well, we've got 500 really good researchers here at DeepMind or at OpenAI or whatever, but if we had AIs that could perform at that level, we could have 5 million. That would be amazing."

I'm kind of like, "Oh God, that seems like a recipe for potentially creating something super powerful, but also kind of losing control over what it is we're creating," and creating something that maybe could be this incredibly refined and powerful form of intelligence that kind of pierces through all the buffers that we currently have.

So time and speed is a concern. This comprehensive AI services thing sounds like a slower-developing plan. I think that would probably be good, but I'm not sure how we get from the trajectory we're on, where we have multiple people credibly approaching that tipping point where the AIs are going to start to do the AI research.

OpenAI reported with o3 that o3 was able to do 40% of the pull requests of real OpenAI work that's actually put into their codebase. Exactly what that measures is always subject to caveats and debates: how meaningful is that, and what should we really understand that to mean? But it was 0% to 5% in the previous generation of models, so it clearly represents some meaningful scale-up.

I don't know. How do we get to comprehensive AI services that could come online in the timeframe before some of these Manhattan Project-style things fly a little too close to the sun? That's maybe the toughest question for me on the vision, and that's maybe where we do need some sort of regulation.

People who dream of abundance are often quite allergic to the notion of regulation. Certainly, regulation has denied us a lot of abundance that I think we rightfully should have at this point, like cheap nuclear energy, for one, and warp speed for these new antibiotics would be another. But boy, the risk of getting AI to do all the AI research before we even really know what we're trying to create is one thing that I do think, without wanting to be too much of a party pooper, the government might have a little role in constraining.

It could try to change the incentive, because right now they're all racing each other, and it's hard to imagine how they get off that track and do something more buffered, more stable. Right now, it doesn't even seem like they have that much of a plan for that, to be honest with you.

They're also kind of like, "And by the way, we're also seeing that the AIs are scheming more, and they're starting to be deceptive. By the way, they're also becoming more situationally aware when we do these evals. We're starting to see more and more that they're actually recognizing that they're being evaluated," which means we have a harder time trusting that what they're doing in the evaluation process is even representative of what they're going to do outside of it.

They're just kind of like, "Hopefully we'll solve that along the way. Maybe the AIs will be able to help. Don't worry about it," is the general vibe. I'm definitely not comfortable with all that.

Yeah.

How do we get from there to the comprehensive AI services vision? That's a tough one, but I would love to see something more like that become kind of the default vision.

Beatatric Urkers

Yeah, I think that we recently did a little world-building exercise of what it would look like to actually have a tool-AI future, which I think is kind of the same as comprehensive AI services, depending on exactly how you define them.

Basically, we were thinking about what it would look like to have AI that's mainly focused on being a tool. So it's limited, maybe, in some of its agenticness and some of its generality, but very useful to us. I think the main thing that we thought could potentially put us on that trajectory, because I agree, it's not the trajectory that we're on right now, would be some sort of legal- or insurance-driven way to get there.

Insurance companies probably don't want to cover systems that are too opaque or that no one is liable for in any way. So that's what we thought was the most probable route, if anything.

I like that. I'm going to do a podcast before too long. I actually just made a very small personal investment in the AI underwriting company, and they are trying to realize that vision. It would maybe be extra nice if there were a mandated insurance requirement, because one thing the companies could do today is just not buy any insurance and not have to deal with it.

If we required insurance and brought in that whole mechanism of trying to model out the risk and price it, some things might be uninsurable. If they're uninsurable, maybe they can't happen. I think that could be really good.

I'm personally not really an investor for financial returns. I mostly just throw very small amounts of money into things that I believe in, that I want to see exist, and that I want to be on the team for. In that sense, I am personally invested in that notion, so I do really like that.

Beatatric Urkers

Well, great to hear that someone is already sort of working on it. One thing that I also wanted to talk to you about is that you feel like you're one of not that many people who are trying to balance taking seriously both the transformative positive opportunities of AI and the risks.

I'm curious to hear what you think it's like to balance that tension, and how maybe you keep yourself from sliding too much into one or the other while trying to stay real about it.

It honestly comes very naturally to me, and I kind of just feel like the updates that we get on a regular basis require that. I don't really know how I could have any other worldview than this classically ambivalent one: super excited by the upside, and hoping my fear is a healthy fear of the downside.

There is some actual fear there, for sure, but you just see these eureka moments. I have one presentation that I call “Eureka Moments, Bad Behavior,” and it's like, you see these antibiotic things, and you see—

There was one where a Stanford group under Professor James Zou created what they called the Virtual Lab. A human gives a problem to an AI, and the AI gets to spin up its own other agents. They can give it various tools, and the AIs are getting quite good at using tools.

Something like 1% or 2% of the overall tokens in this process were from humans; the rest were all AI. They ended up designing new treatments for novel strains of the COVID virus as well.

I contrast that with the antibiotic thing, where the human scientists were using these very specific, purpose-built AI models in this virtual-lab setting. The AIs were using these very purpose-built, specific AI models but doing a similar job to what the human scientists were doing. Ultimately, in both cases, they were creating new treatments for diseases that had evolved to evade our previous treatments for them.

So that's amazing, right? How can you not be super excited about that? But then the next post, as I scroll through Twitter—and I do find Twitter, honestly, still to be the best place to stay up to date, for better or worse—will be, “Deception is on the rise in the latest models,” and we're starting to see these scheming behaviors.

How do you look at something that has that power but also reflects back to us some of the worst tendencies that we have and not feel these dual-track feelings? To me, that just seems like the only place to be.

I just did an episode of the podcast with a guy who is an executive coach to a bunch of people at various AI companies, including Sam Altman. His name is Joe Hudson, and I asked him a similar question. He said that in his experience, everybody at the frontier companies has this mindset.

He said that from the outside, and from what you see on Twitter, that's a little bit misleading, because you've got booster accounts and doomer accounts. I don't rule out that either of those could be right. Especially, I do think we should not dismiss the doomers.

But his view was that everybody he's ever met at these frontier companies has this mindset. He's got a personal testimonial from Sam Altman on his executive coaching website.

Everybody he’s ever met at these places, he says, has that dual-track mindset. They are all seriously grappling with the ramifications of their work. They’re all asking themselves, “Are we doing the right thing?” So I think that’s encouraging relative to maybe how it’s commonly understood. How do we reconcile that with the racing is another interesting question.

I also asked him if he ever expected that we’d see an AI developer stand down because they feel like we can’t go any further in a responsible way. His answer was no. He said they’re problem solvers, and they’ll just look at that as another problem to solve. They will not stand down; they’ll just say, “We can solve this one,” just like we solved the last thousand problems that we came across.

So there do seem to be some contradictions in that overall report, from the mindset and cultural perspective of what’s going on at the companies. But at least at that first level, I was glad to hear that there’s serious engagement with both sides. For me, that’s just—I don’t know, maybe not a great answer, but I feel kind of compelled toward that worldview by the developments that I see on a regular basis.

Beatatric Urkers

I agree with you, actually. You put it very simply like that—it comes naturally to you—and when I think about it, I think it comes naturally to me as well. So maybe that’s a good answer, simply put.

However, as you said, on Twitter, that’s not really the impression, necessarily, depending on what bubble or hole you end up in. To some extent, at least what we’re trying to offer in this podcast is thinking about the positive trajectories, because the negative ones are very easy to agree on and envision. There are definitely concrete ideas about how things could go poorly in relation to AI, and always in relation to new tech.

Do you have any thoughts on why you think that is, and what might be missing from the discourse that could help us aim better toward more positive futures?

It is hard. Eliezer Yudkowsky had this famous, at least in my mind, idea that you can’t predict what something will do. This also, I think, goes back to maybe Vernor Vinge. I’m not as deeply read in science fiction as I should be, but the idea is that if you could predict what the thing would do, then you would be as smart as the thing. If it’s genuinely smarter than you, then one of the ways in which that presents is that you can’t predict what it’s going to do.

Eliezer then adds to that: You can predict that you will lose to it in a competition. If you go back to the superhuman chess player, you can’t predict the moves that the superhuman chess player will make. If you could, you would be a superhuman chess player yourself. But you can predict that you will lose to the superhuman chess player because it is, indeed, a superhuman chess player.

I think it’s just hard to envision. We also touched a little bit on the “nobody envisioned Uber” trope with the iPhone. I think that’s very real, too. The collective process of invention, innovation, and remixing everything is superhuman relative to any individual. So I think both sides of this are pretty hard to envision, and we probably should expect to be surprised.

We should probably expect the future to be quite weird, quite alien, and quite surprising. I don’t know what we can do about it. I do think higher standards in politics would help. Of course, we have all these culture-war preoccupations. I’ve often wondered why nobody is running on just a super-pragmatic idea: “Here are 5 ways that we’re going to make daily life better for everybody.”

On my list, self-driving cars would be one of those things. We’re going to get those things out to everybody. Everybody hates their commute, right? As far as I understand, commute length is highly correlated with unwellness. The length of your commute and your grumpiness level seem to be strongly correlated. If that’s true, and everybody kind of knows that, I don’t think many people love their commute. You hear it occasionally, but most people don’t.

Why is nobody prioritizing these daily things? Similarly with energy, why has nobody said, “Your electricity bill could be 10% of what it is if we just built some power plants to make it so”? We wouldn’t even necessarily have to make the environment worse to do that. This is amazing stuff. Why is nobody pushing that?

I don’t feel like I have a great answer for that. But Tyler Cowen famously said that one of the most high-impact things you can do is help individuals raise their personal level of ambition. Maybe there’s a society equivalent of that. Can we help society raise its level of expectations? Where is my flying car? But for real—really, where is it? Why is nobody even talking about it?

Why has our leadership lost all connection to making material life better? Everything we hear is about how we’re going to split up the pie a little differently. You almost never hear about how you’re going to make a more prosperous future that can give more to everyone.

Even though, in some theories, that’s basically central to a democratic country working at all: the idea that because there’s a little more every year through economic growth, that provides the grist to make all these deals and make living together possible, because it’s not a zero-sum game. Everybody can feel like they’re winning. But we’ve kind of lost track of that. I don’t know why, and I’m not sure how to fix it.

But if people were encouraged en masse to demand better, that seems like it could really help. If we don’t screw it up and can avoid all the downside things, I think the upside things are pretty well on track to happen. So I don’t know. It’s a great question. I don’t have a great answer to it.

Beatatric Urkers

Well, that’s good news that you think they’re on track to happen. I also think there were a few very interesting points. The point you had about raising ambitions feels to me a little bit like what the U.S., and especially the Bay Area, has done with tech: raising the ambition and expectations of what startups can do.

I also like this idea of, instead of just thinking about how to split up the pie differently, thinking about how you can grow the pie. That’s another really nice and useful Drexler idea: Paretopia, where everyone gets a little bit better off, or at least no one gets worse off. If we could grow the pie, that would be a great way to go about it.

In relation to that, if you could rewrite the science-fiction canon around AI, what kinds of stories or narratives do you think we should do more of? Is there anything you’d like to see more of in general?

I do confess I’m not as well-read as many thinkers in this area. Other people should have better answers to that question. I think more positive visions obviously go without saying. Maybe more branching scenarios would be interesting.

The AI 2027 scenario that recently made waves was notable for multiple reasons, but one big one was that it had multiple endings. Maybe one twist on the idea of helping society demand more—raising our expectations and ambition—is to make it somehow clearer to people that this is up to us, that we get to decide what we’re going to do.

Even if it’s not an individual decision, France built a bunch of nuclear power plants. For all the things that aren’t going super well in France right now, they have relatively abundant nuclear energy that’s not contributing to the carbon problem. We could have had that, and we don’t.

These are hinge points in history where certain things could go one way or another. Showing just how different the future ends up being based on whether you do or don’t make certain decisions, or deploy certain technologies, might help people adopt a more possibility-oriented mindset.

Maybe everybody sort of sees everything as entertainment now. I don’t want to get too attached to this idea because I’m just coming up with it, but I do think there’s a way in which you see these moments where people are, even in the midst of history, relating to it as a viewer—as a sort of passive consumer of content.

Is there any way to create content that could snap people back out of that mindset? If we’ve gotten so used to just consuming video content that we start to treat even real life as something that’s just unfolding for us, in a way that we don’t influence, in the way that video content does, could we change that?

An AI could really enable this, too, right? The ability to create these forking scenarios—why doesn’t it exist today, at least with high-production-value stuff? Probably because it’s really expensive to create even a single mainline story.

So, branching stories that people are only going to explore a few branches of may not be economical, but AI could perhaps make that economical. What if you had to sit there in your Netflix app and make decisions, and then get a world that was meaningfully influenced by the decisions that you made?

Could we teach people that there are real consequences to decisions and that their agency really matters? Could we incept that idea through an entertainment medium? I don't know. But the biggest idea that comes to mind is trying to bring some sense of the contingency of history and the agency that people have, if only collectively, to decide what the future is going to look like.

Bringing that to the fore seems like something that I would hope would happen.

Beatatric Urkers

Yeah, that's actually a really interesting point—the branching scenarios. Didn't Black Mirror do something like that a few years ago? But yeah, I agree. It'll probably be a calling.

A couple of experiments like that, certainly.

Beatatric Urkers

Yeah, but none that maybe has made a huge impact. Also, just because it's very on topic, one thing that we did—we've done a bunch of world-building experiments with the Existential Hope program at Foresight. One thing that we did recently wasn't necessarily a branching scenario, but it was more like 2 options for potential AI futures.

One was the Tool AI, the one that I mentioned recently, and another was just a more DAC approach [?]. I think it's also interesting because it shows that there are different paths we could take, and it is, to some extent, up to us. So, yeah, I agree, and I know that FLI also did something recently called Tomorrow's AI, where I think they explored different options.

There are some things like that coming. I want to shift gears a little bit and pick your brain on podcasting. After all the episodes that you've done, what are the main lessons that you've learned? Do you have any recommendations for someone hosting a podcast as to what you think would be a good thing to do?

Not really. To be honest, I sometimes call myself the Forrest Gump of AI. What I mean by that is that I'm just stumbling my way through and often find myself in interesting places, usually as an extra in notable events. But I haven't been that strategic about it.

I think the main thing that I try to do is, with apologies to Tyler Cowen, have the conversation I want to have. When I started this, I wasn't a content person, honestly, really at all. I had never created much content before, and I don't think I would be doing it now if it weren't for my AI obsession.

The way that the podcast started for me was that my friend Erik was starting a podcast network, and he said, “Hey, all you do is talk my ear off about AI. Why don't we record a couple of these and see if it becomes a podcast?” I was like, “I don't know. I don't know how to do that.”

If you watch my feed on YouTube, you see that the production value remains relatively low. But he was like, “We'll take care of everything for you. All you have to do is talk. If it works, it works. If it doesn't, it doesn't.” So I was like, “Okay, I'll try it.”

The mindset that I went into it with, which I think has served me reasonably well—although I certainly can't say it will generalize from my situation to others—was basically that I just wanted to learn as much as possible. If I could get people to teach me things or have interesting conversations, and I live in Detroit, so I'm not at the epicenter of AI, which is obviously in the Bay Area, I could be more plugged in and have conversations I wouldn't otherwise get to have this way.

If I could do that and nobody listened to it, but I got value from it, then that would be great. That alone could be a win. I went into it with an attitude of, if I'm having conversations that I want to have and I'm learning from them, that's enough for me to be happy with the way I'm spending my time. Anything else was basically just gravy.

The audience isn't that big. Metrics are tough. I feel like it's not clear, honestly, how many people are listening sometimes. I know when we put out an episode with Zvi and his audio quality is bad, because I get a bunch of messages telling me that we need to get Zvi a microphone. So there are at least some people listening.

I'm kind of haphazard in that respect and almost strategically unstrategic—or strategically focused on my own personal growth—and then I let the chips fall where they may. I also came into it with the luxury of having a number of different things going on. I wasn't trying to make it my full-time job, and it's still not my full-time job.

I'm not in a position where I'm forced to think too much about which episodes do well, which ones don't, and what the numbers look like. I can't help but do a little bit of that, but I mostly try to stay true to the original idea: I want to learn as much as possible. These conversations can be a good regular cadence for meaningful learning and patching my blind spots. The rest has all just been letting the chips fall where they may, to be honest.

Beatatric Urkers

You know, I think to some extent that feels like great advice, because it's encouraging to hear that you can just do it for the joy of it and think about things you're curious about.

I think Joe Rogan would actually describe himself pretty similarly, from what I've heard. For years, he was just shooting the shit with his comedian buddies, and then it was some jiu-jitsu buddies or whatever. Then it kind of blew up, but I think he did a lot of episodes before he became huge, and mostly he was just getting high and having fun, I think.

Beatatric Urkers

That's true. Same with Tim Ferriss getting high before podcast recordings. But I am having fun.

Yeah, yeah, it's true. It's a wide range, and with you, I guess it's a wide range within a narrow topic. But like you say, it's a very broad technology that obviously touches on everything.

Beatatric Urkers

But yeah, I think that's all we have time for. Nathan, thank you so much for coming. It was really nice to chat with you about all of this. Thank you.

My pleasure. Thanks so much for the invitation. This has been really fun.