AMA：AGI时代的职业建议、我的研究方法（Sholto & Trenton）

Today, this is going to be an Ask Me Anything episode. I'm joined by my friends, Trenton Bricken and Sholto Douglas. You guys do some AI stuff, right?

Yeah. We dabble.

They're researchers at Anthropic. Other news: I have a book launching today. It's called The Scaling Era. I hope one of the questions ends up being why you should buy this book, but we can kill 2 birds with 1 stone.

Okay, let's just get at it. What's the first question that we gotta answer? Take us away.

I want to ask the softball question that I heard before: why should ordinary people care about this book? Why should my mom buy and read the book?

First, let me tell you about the book, what it is. These last few years, I've been interviewing AI lab CEOs, researchers, people like you guys, but also scholars from all kinds of different fields—economists, philosophers. They've been addressing what are basically the gnarliest, most interesting, most important questions we've ever had to ask ourselves.

What is the fundamental nature of intelligence? What will happen when we have billions of extra workers? How do we model out the economics of that? How do we think about an intelligence that is greater than the rest of humanity combined? Is it even a coherent concept?

What I'm delighted with is that, with Stripe Press, we made this book where we compiled and curated the best, most insightful snippets across all these interviews. You can read Dario Amodei addressing why scaling works. Then, on the next page, Demis Hassabis is explaining DeepMind's plans for whether they're going to go with the RL route and how much of the AlphaZero stuff will play into the next generation of LLMs. On the next page, of course, is you guys going through the technical details of how these models work.

There are so many different fields that are implicated. I feel like AI is one of the most multidisciplinary fields that one can imagine, because there's no field, no domain of human knowledge that is not relevant to understanding what a future society of different kinds of beings will look like.

You're going to have Carl Shulman talking about how the scaling hypothesis shows up in primate brain scaling, from chimpanzees to humans. On the next page, there might be an economist trying to argue—like Tyler Cowen, explaining why he doesn't expect explosive economic growth and why the bottlenecks will eat all that up.

That's why your mom should buy this book. It's the distillation of all these different fields of human knowledge applied to the most important questions that humanity is facing right now.

I do like how the book is sliced up by different topics and across interviews. It does seem like a nice way to listen to all of the interviews in one digestible way.

There are 2 interviews I've done that haven't been released publicly before that are in the book. One was Jared Kaplan, who's one of your co-founders. This is another example: he's a physicist, and he's explaining scaling from this very mathematical perspective about data manifolds.

Then, on the next page, you have a totally different perspective. It's Goren Gordon talking about why general intelligence actually evolved in the first place. What is the actual evolutionary purpose of it?

It's page by page, right? You can just get a digest. Even for me, the person who's been on the other end of these conversations, it was really cool to read it and just be like, “Oh, actually, now I realize how these insights connect to each other.”

Yeah, the only other thing that stood out to me was the introduction section.

The only thing that stood out to you?

Yeah, that was really the only thing that was noteworthy. I just mean, what stood out in accessibility was the introduction section and the diagrams for all the different inputs that enable you to train a machine-learning model. Stripe Press books are also just beautiful. They have these nice side captions explaining what parameters are, what a model is, and these sorts of things.

Actually, when we did our episode together, a bunch of people—I don't know if you saw this—independently made these blog posts and Anki cards and shit where they were explaining the concepts, because we just passed over some things.

Hopefully, we've given a similar treatment to every single interview I've done, where you can read a very technical interview with a lab CEO or an engineer or a researcher, and the side is like, “Here's more context, here are more definitions, here's more commentary.” I feel like it elevated the conversations.

So, in other words, my parents will finally understand what I do for a job.

They're gonna get it very well. Maybe my parents will, because I got a book.

All mine need to know is that my name's in a book.

You're a co-author.

All right. Should we get into the AMA questions?

Let's do it.

1. AI models not making connections across fields

All right. Brian Krav asks, “The issue you raised with Dario and occasionally tweet about, relating to models not making connections across different topics—some sort of combinatorial attention challenge—what are your thoughts on that now? Do you solve it with scale, thinking models, or something else?”

The issue is, one of the questions I asked Dario Amodei is, look, these models have all of human knowledge memorized, and you would think if a human had this much stuff memorized and they were moderately intelligent, they could be making all these connections between different fields.

There are examples of humans doing this, by the way. There's Donald Swann or something like this. This guy noticed that what happens to a brain after magnesium deficiency is exactly the structure you see during a migraine. So then he's like, “You take magnesium supplements and we're gonna cure a bunch of migraines.” And it worked.

There are many other examples of things like this where you just notice 2 different connections between pieces of knowledge. Why, if these LLMs are intelligent, are they not able to use this unique advantage they have to make these kinds of discoveries?

I feel a little shy giving answers on AI stuff with you guys here. But Scott Alexander addressed this question in one of his AMA threads, and he's like, “Look, humans also don't have this kind of logical omniscience.” He used the example of, in language, if you really thought about why 2 words are connected. It's like, I understand why “rhyme” has the same etymology as this other word. But you just don't think about it, right? There's this combinatorial explosion.

I don't know if that addresses the fact that we know humans can do this, right? Humans have in fact done this, and I don't know of a single example of LLMs ever having done it. Actually, yeah, what is your answer to this?

I think my answer at the moment is that the pretraining objective doesn't necessarily imbue models with the skill of making novel connections or doing research. It imbues them with this nice, flexible general knowledge about the world, but not necessarily the kinds of things that people are trained to do through PhD programs and through the process of exploring and interacting with the world.

At a minimum, I think you need significant RL in at least similar things to be able to approach making novel discoveries. I would like to see some early evidence of this as we start to build models that are interacting with the world and trying to make scientific discoveries, and modeling the behaviors that we expect of people in these positions.

I don't actually think we've done that in a meaningful or scaled way as a field, so to speak.

Riffing off that with respect to RL, I wonder if models currently just aren't good at knowing what memories they should be storing. Most of their training is just predicting the next word on the internet and remembering very specific facts from that.

But if you were to teach me something new right now, I'm very aware of my own memory limitations, and so I would try to construct some summary that would stick. Models currently don't have the opportunity to do that. Memory scaffolding in general is just very primitive right now. I mean—

Right, like Claude Plays Pokémon.

Exactly, yeah. Someone worked on it, it was awesome, it got far, but another excited Anthropic employee then iterated on the memory scaffold and was able to very quickly improve on it.

So that's one. I do also just wonder if models are idiot savants. The best analogy might be Kim Peek. Kim Peek was born without a corpus callosum, if I recall correctly. Each hemisphere of his brain operated quite independently. He'd open a book, there'd be 2 pages visible, and each eye would read 1 of the pages.

He had a perfect encyclopedic memory of everything he'd ever read. But at the same time, he had other debilitations—functioning socially, these sorts of things. It's just kind of amazing how good LLMs are at very niche topics, but can totally fail at other ones still. I really want to double-click on this thing of why there's this trade-off when it comes to memorization.

Apparently, it's connected to this debilitation, but WikiText is like 5 megabytes of information. The human brain can store much more, so why does the human brain just not want us to memorize these kinds of things and is actively pruning? I don't know. But we don't have to do it right now. We'll do a separate episode.

[Speaker?]

Yeah, just one thing I'll say on that is, there is another case study of someone with a perfect memory, so they never forgot anything. But their memory was too debilitating. It'd be like your context window for the transformer is trillions of tokens. And then you spend all your time attending to past things, and are too trapped in the details to extract any meaningful generalizable insights from it.

[Speaker?]

2. Growing the podcast

Yeah. Terrence Deacon, whose book you recommended, had this interesting insight about how we learn best when we're children, but we forget literally everything that happened to us when we were children, right? We have total amnesia. And adults have this in-between where we don't remember exact details, but we can still learn in a pretty decent way. And then LLMs are on the opposite end of this gradient where they'll get the exact phrasing of Wiki text down, but they won't be able to generalize in these very obvious ways. A little bit like Gwern's optimizer theory, no?

Yeah, I think I probably got it from that. Gwern has definitely had a big influence on all this for me as well.

I feel like what's underappreciated on the podcast is that we have this group chat, and we also just meet up a lot in person. All the output from the podcast comes from you and a couple of other people feeding me ideas and nudges, and then I can use that as an intuition pump during the conversation.

Yeah, you're not the only one.

What do you mean?

I benefit immensely from just hearing what everyone else has to say. It's all regurgitation in one way or another.

3. Career advice given AGI

Another question? Yes. Maybe Rabid Monkey asks, “Imagine you have a 17-year-old brother or nephew just starting college. What would you recommend he study, given your AGI timelines?”

That's so tough, right? I don't know—become a podcaster? I feel like that job's still going to be around.

It's funny, because I studied computer science, and in retrospect—at the time, you could've become a software engineer or something. Instead, you became a podcaster. It's kind of an irresponsible career move, but in retrospect, it's like—it kind of worked out. Just as these guys are getting automated.

I get asked this question all the time, and one answer that I like to give is that you should think about the next couple of years as increasing your individual leverage by a huge factor every year. Already, software engineers will come up and say, “You know, I'm 2 times faster,” or, “In new languages, I'm 5 times faster than I was last year.” I expect that trend line to continue, basically, as you go from this model of, “Well, I'm working with some model that's assisting me on my computer, and it's basically a pairing session,” to, “I'm managing a small team,” through to, “I'm managing a division or a company.”

Basically, that is targeting a task. I think that deep technical knowledge in fields will still matter in 4 years. It absolutely will, because you will be in the position of managing dozens—or your individual management bandwidth will be maxed out by trying to manage teams of AIs. Maybe we end up in a true singularity world where you have AIs managing AIs and this kind of stuff.

But I think in a very wide part of the possibility spectrum, you are managing enormous, vastly more resources than an individual could command today, and you should be able to solve so many more things with that.

That's right, and I think I would emphasize that this is not just cope. It genuinely is the case that these models lack the kind of long-term coherence that's absolutely necessary for making a successful company. Just getting a fucking office is kind of complicated, right? So you can imagine that for sector after sector—the economy is really big, right? And really complex.

Exactly, and so I don't know the details, but I assume if it's a data-sparse thing where you've got to know the context of what's happening in the sector or something, I feel like you'd be in a good position. Maybe the other thought I have is that it's really hard to plan your career in general.

I don't know what advice that implies, because I remember being super frustrated. I was in college, and the reason I was doing the podcast was to figure out what it was I wanted to do. It wasn't the podcast itself. I would go on 80,000 Hours or whatever for career advice, and in retrospect it was all mostly useless, and just try doing things.

Especially with AI, it's so hard to forecast what kind of transformations there will be, so try things, do things. It's such banal, vague advice, but I am quite skeptical of career advice in general.

Well, the piece of career advice that I'm not skeptical of is: put yourself close to the frontier, because you have a much better vantage point from there. You can study deep technical things, whether it's computer science or biology, and get to the point where you can see what the issues are, because it's actually remarkably obvious at the frontier what the problems are. It's very difficult to see—

Actually, do you think there is an opportunity? One of the things people bring up is that maybe the people who are advanced in their careers and have all this tacit knowledge will be in a position to be accelerated by AI. But you guys, 4 years ago or 2 years ago, when you were getting discovered or something—that kind of thing where you have a GitHub open issue and you try to solve it—is that just done, and so the onboarding is much harder?

That's still what we look for in hiring. So, you know?

Yeah, I'm in favor of “learn fundamentals, gain useful mental models,” but it feels like everything should be done in an AI-native way, or top-down instead of your bottom-up learning. First of all, learn things more efficiently by using the AI models, and then just know where their capabilities are and aren't.

I would be worried and skeptical about any subject that prioritizes rote memorization of lots of facts or information instead of ways of thinking. But if you're always using the AI tools to help you, then you'll naturally have a good sense for the things that it is and isn't good at.

Okay, next one. What is your strategy, method, or criteria for choosing guests?

4. Guest selection criteria

The most important thing is: do I want to spend 1 to 2 weeks reading every single thing you've ever written, every single interview you've ever recorded, and talking to a bunch of other people about your research? I get asked by people who are quite influential to come on the podcast. They'll often say, “Would you have me on your podcast?” More often than not, I say no, for 2 reasons.

One is just, okay, you're influential; it's not fundamentally that interesting as an interview prospect. I don't think about the 1 hour that I'll spend with you. I think about the 2 weeks, because this is my life, right? The research is my life, and I want to have fun while doing it. So is this going to be an interesting 2 weeks to spend? Is it going to help me with my future research or something?

The other is that big guests don't really matter that much if you just look at what the most popular episodes are or what, in the long run, helps a podcast grow. By far my most popular guest is Sarah Paine, and before I interviewed her, she was just a scholar who was not publicly well-known at all. I just found her books quite interesting.

I have electric chemistry with her. And by the way, on a viewer-minute-adjusted basis, I host the Sarah Paine Podcast, where I occasionally talk about AI.

That’s funny.

And then it’s David Reich, who is a geneticist of ancient DNA. He’s somewhat well known, and he had a best-selling book, but he’s not Satya Nadella or Mark Zuckerberg, who are the next people on the list. And then again, I think that pretty soon it’s you guys or Leopold or something, and then you get to the lab CEOs or something. So big names just don’t matter that much for what I’m actually trying to do.

It’s also really hard to predict who’s going to be the David Reich or the Sarah Paine, so just have fun. Talk to whoever you want to spend time researching, and it’s a pretty good proxy for what will actually be popular.

5. Choosing to pursue the podcast long-term

What was the specific moment, if there was one, that you realized that producing your podcast was a viable long-term strategy?

I think when I was shopping around ad spots for a Mark Zuckerberg episode. And now when I look back on it, it's not in retrospect that mind-blowing, but at the time I'm like, “Oh, I could actually hire an editor full-time, or maybe more editors than one, and from there turning into a real business.” Because before people would tell me, “Oh, these other podcasts are making whatever amount of money.” And I'd be like, “How?” You know?

So I have this running joke with one of my friends. I don't know if you've seen me do this, but every time I encounter a young person who's like, “What should I do with my life?” I'm like, “You gotta start a blog. You gotta be the Matt Levine of AI.” You can do this. It's a totally empty niche. And I have this running joke with them where they're like, “You're like a country bumpkin who's won the lottery. And you go up to everything and everyone and just like, ‘Guys, a scratch pad. Get the scratch pad.’”

I do want to press on that a bit more because your immediate answer to the 17-year-old was to start a podcast. So what niches are there? What sort of things would you be excited to see in new blogs, podcasts?

I wonder if you guys think this too, but I think this “Matt Levine of AI” is a totally open niche, as far as I can tell, and I apologize to those who are trying to fill it in. The other thing I’d really emphasize is that it’s really hard to do this based on other people’s advice, or to say, “At least I’m trying not to fill a specific niche.” If you think about any sort of successful new-media thing out there, it has 2 things that are true: It’s often not just geared toward 1 particular topic or interest, and 2, the most important thing is that it’s propelled by a single person’s vision. It’s not a collective or whatever.

The thing I really want to emphasize is that it can be done. 2, you can make a lot of money at it, which is probably not the most important thing for the kind of person who would succeed at it, but it’s still worth knowing that it’s a viable career.

3, basically, you’re going to feel like shit in the beginning, where all your early stuff is going to kind of suck. Maybe some of it will get appreciated, but it seems like bad advice to say, “Still stick through it in case you actually are terrible,” because some people are terrible. But in case you are not, just do it, right? What is 3 months of blogging on the side really going to cost you?

People just don’t actually seriously do the thing for long enough to get evidence or get the sort of real-life feedback on, “Oh, this is how you do it. This is how you frame an argument. This is how you make a compelling thing that people will want to read or watch.” Blogging is definitely underrated.

So you both had blogs that were relevant.

Not that. They were somewhat relevant. But I think, more so, that we have all read almost all the blogs that do in-depth treatises on AI. If you write something that is high quality, it is almost invariably going to be shared around Twitter and read.

Oh, this is so underappreciated. Two pieces of evidence: I was talking to a very famous blogger you would know, and I was asking him, “How often do you discover a new, undiscovered blogger?” And he was like, “It happens very rarely, maybe once a year.” And then I asked him, “How long after you discover him or her does the rest of the world discover them?” And he’s like, “Maybe a week.” And what that suggests is that it’s actually really efficient.

I have some more takes. Let’s hear them. This is the AMA. I believe that slow, compounding growth in media is kind of fake. Take Leopold’s “Situational Awareness: The Decade Ahead.” It’s not like he was building up an audience for a long time, for years or something. It was really good. Disagree or agree with it, if it’s good enough, literally everybody who matters—and I mean that literally—will read it. And it’s hard to zero-shot something like that.

But the fundamental thing to emphasize is that the compounding growth, at least for me, has been that I feel like I’ve gotten better. It’s not so much that somehow the 3 years of having 1,000 followers were somehow compounding. I don’t think it was that. I think it was just that it took a while to get better.

Yeah, certainly. When Leopold posted that, the next day it was almost like you could picture it being stapled to the wall, so to speak, on Twitter. You could go to any event for the following week, and every single person in the entire city was talking about that essay. It was like Renaissance Florence or whatever.

That’s right. Yeah. The world is small.

The world is small. What would you say is your first big success? I’m trying to think back to when I first found your podcast. I distinctly remember you had a blog post on the Annus Mirabilis, and Jeff Bezos retweeted it, I think. I’m trying to remember if it was before that or not, but, yeah. I’m curious about your answer.

I feel like that was it. And it wasn’t something where it was some big insight that deserved to blow up like that. It was just taking some shots on goal. They were all insight-porn-y, and then one of them, I guess, caught the right guy’s attention, and, yeah.

Yeah, that’s something else that’s underappreciated: a piece of writing doesn’t need to have a fundamentally new insight so much as give people a way to express cleanly a set of ideas that they are already aware of in a broader way. If it’s really crisp and well articulated, then even still, that’s very valuable.

The one thing I should emphasize, which I think is maybe the most important thing to the feedback loop, is that it’s not the compounding growth of the audience. I don’t even think it’s me getting more shots on goal in terms of doing the podcast. I actually don’t think you improve that much by just doing the same thing again and again. If there’s no reward signal, you’ll keep doing whatever you were doing before.

I genuinely think the most important thing has been that the podcast is good enough that it merits me getting to meet people like you guys. Then I become friends with people like you. You guys teach me stuff. I produce more good podcasts, so hopefully slightly better. That helps me meet people in other fields. They teach me more things.

With the China thing recently, I wrote this blog post about a couple of stories about things that happened in China. That alone has netted me an amazing China network in the matter of 1 blog post. Hopefully, if I do an episode on China, I will be better as a result. Hopefully that happens across field after field. Just getting to meet people like you is actually the main flywheel.

Interesting. So, move to San Francisco?

Yes. If you’re trying to do AI, yeah.

Oh, very important question from Jacked Pajeet: How much can you bench? You can’t lie because we both know the answer.

At one point I did bench 225 for 4. Now I think I’m probably 20 pounds lighter than that or something.

The reason you guys are asking me this is because I’ve gone lifting with both of you. I remember Trent and I were doing pull-ups and bench, and it’d be like, “Bench,” and he’d throw on another plate or something. Then instead of pull-ups, he’d be cranking out these muscle-ups.

It’s all technique. Let’s make sure.

So they both bench more than me. But I’m trying my best. Ask again in 6 months.

Yeah. What’s your favorite history book? There’s a wall of them behind you.

6. Reading habits

Oh, obviously, the Robert Caro LBJ biographies. The main thing I took away from those books is that LBJ had this quote that he would tell his debate students. In his early 20s, he taught debate to these poor Mexican students in Texas, and he used to tell them, “If you do everything, you’ll win.”

I think it’s an underrated quote. That’s the main thing I took away. You see it throughout his entire career, where there’s a reasonable amount of effort that goes by 80/20. You do the 20 to get 80% of the effect. And then, if you go beyond that and think, “Oh, no. I’m not just gonna do 20%; I’m gonna do the whole thing,” there’s a level even beyond that, which is an unreasonable use of time. This is going to have no ultimate impact, and you still try doing that.

You’ve shared on Twitter that you’re using Anki, or even a Claude integration. Do you do book clubs? Do you use Goodreads? And what are you reading right now?

I don’t have book clubs. But spaced repetition has genuinely been a huge uplift in my ability to learn. Mostly because it’s not even the long-term impact over years, though I think that is part of it. I do regret all the episodes I did without using spaced-repetition cards, because all the insights have just sort of faded away.

The main thing is, if you’re studying a complicated subject, at least for me, it’s been super helpful to consolidate. If you don’t do it, you feel like a general who’s like, “I’m gonna wage a campaign against this country.” And then you climb one hill, and the next day you’re in retreat, and then you climb the same hill. There might be a more kosher analogy.

And then the other question was, what am I reading right now? My friend Álvaro de Menard, author of Fantastic Anachronisms—can I just hold it up? Actually, it’s right here. I hope he’s okay with me sharing this, but he made 100 copies of this translation he did of his favorite Greek poet, Cavafy. Hopefully I didn’t mispronounce it. Sorry, that one has a good inscription for Gwern, because that’s his copy, but it’s super delightful, and that’s what I’ve been reading recently.

Any insights from it so far?

Poets will hate this framing. I feel like poetry is like TikTok, where you get this quick vibe of a certain thing, and then you swipe. Then you get the next vibe, swipe. Álvaro, I’m sorry.

No, that’s interesting. How do you go about learning new things or preparing for an episode? You mentioned the 1- to 2-week period where you’re deep-diving on the person. What does that actually look like?

It’s very much the obvious thing: you read their books, you read the papers. If they have colleagues, you try to talk to them to better understand the field. I will also mention that all I have to do is ask them questions, and I do think it’s much harder to learn a field as a practitioner than just learn enough to ask interesting questions. But for that, it’s very much the obvious thing you’d expect.

“Based Carl Sagan” asks, “What are your long-term goals and ambitions?” AGI kind of just makes the prospect of a long term harder to articulate, right? You know the Peter Thiel quote about, what is your 10-year plan, and why can’t you do it in 6 months? It’s especially salient, given the timelines.

For the foreseeable future, grow the podcast and do more episodes, maybe more writing. So we’ll see what happens after 10 years or something. The world might be different enough. So basically, podcast for now.

Something you’ve spoken to me about, particularly when you were trying to hire people for the podcast, was what you wanted to achieve with the podcast. In what way do you want the podcast to shape the world, so to speak? Do you have any thoughts on that? I remember you telling me, “I really want people to actually understand AI and how this might change their lives,” or, “What we could be doing now to shape the world such that it ends up better.”

I don’t know. I have contradictory views on this. On the one hand, I do know that important decisions are being made right now in AI. And I do think, riffing on what we were saying about situational awareness, if you do something really good, it has a very high probability of one-shotting the relevant person, and people are generally reasonable. You make a good argument, it’ll go places.

On the other hand, I just think it’s very hard to know what should be done. You gotta have the very correct world model, and then you gotta know how, in that world model, the action you’re taking is gonna have the effect you anticipate. And even in the last week, I’ve changed my mind on some pretty fundamental things about what I think about the possibility of an intelligence explosion or transformative AI as a result of talking to the Epoch folks.

Basically, the TL;DR is, I want the podcast to just be an epistemic tool for now, because I think it’s just very easy to be wrong. And so just having a background level of understanding of the relevant arguments is the highest priority. Makes sense. What’s your sense? What should I be doing?

I mean, I think the podcast is awesome, and a lot more people should listen to it. There are a lot more guests I’d be excited for you to interview.

Gotta give me your recs. So it seems like a pretty good answer for now.

Yeah. I think making sure that there is a great debate of ideas—not just on AI, but on other fields—and everything is incredibly high-leverage in value.

How do you groom your beard? It’s majestic.

7. Beard deepdive

I don’t know what to say—just genetics. I do trim it, but—

No beard oil?

Sometimes I do beard oil.

How often?

Once every couple of days.

That’s not sometimes! That’s pretty often! But do you have different shampoo for your head and your beard?

No.

What kind of shampoo do you use?

Anti-dandruff.

Do you condition it?

Yeah.

How often do you shave it?

Who put you up to this? We’re giving people the answers that they want.

Big shampoo. Big beard oil.

Yeah, you can sell some ad slots to different shampoo companies, and we can edit it. Maybe we sold an ad slot. Who knows?

Sorry, you had this idea of merch. Do you want to explain this T-shirt idea?

Yeah. People should react to this. Someone should make it happen. Dwarkesh wants merch, but he doesn’t want to admit that he wants it. Or he doesn’t want to make it himself because that seems tacky.

So I really want a plain white tee with just Dwarkesh’s beard in the center of it. That’s it. Nothing else.

But you were saying it should have a different texture than the rest of the shirt.

Oh, so when I was riffing off it, maybe a limited-edition set can have some of your beard hair actually sewn into the shirt.

Oh my God. That’d be pretty cool. I would pay. I would pay for that.

How much?

I’ve got patches all over my beard. It depends on how much hair: if it’s like one is in there somewhere versus the whole thing. “Do I have to dry-clean it? Can I wash it on the delicate setting?”

But really, I think you should get merch. If you want to grow the podcast, which apparently you do, then this is one way to do that.

You think beard and beard hair in the future is necessary?

Oh, yeah. Oh, yeah.

8. Who is best suited for running an AI lab?

Which historical figure would be best suited to run a frontier AI lab?

This is definitely a question for you guys.

Oh. No, I mean, I’m curious what your take is first. You’ve spoken to more of the heads of AI labs than I have.

Yeah. I was gonna say LBJ. Is the question who would be best at running an AI lab, or who would be best for the world?

Yeah, what outcome do you want?

Because I imagine what the best AI lab CEO succeeds at is raising money, building up hype, and setting a coherent vision. I don’t know how much it matters for the CEO themselves to have good research taste or something, but it seems like their role is more as a sort of emissary to the rest of the world.

And I feel like LBJ would be pretty good at this: just getting the right concessions, making projects move along, coordinating among different groups to maybe—

Oh, Robert Moses.

Again, not necessarily best for the world, but just in terms of making shit happen.

Yeah. I mean, I think best for the world is a pretty important precondition.

Oh, right. Who’d be best for the world? There’s a Lord Acton quote: “Great people are very rarely good people.” So it’s hard to think of a great person in history who I feel would really move the ball forward and whose moral judgment I also trust.

Yeah. We’re lucky in many senses with the set today, right?

That’s right.

The set of people today are both trying to care a lot about the moral side and trying to drive the labs forward. This is also why I'm skeptical of big grand schemes like nationalization, some public-private partnership, or just generally shaking up the landscape too much, because I do think we're in one of the better counterfactual universes.

I mean, whether it's alignment or some kind of deployment safety risk, that level of difficulty is just the nature of the universe. But the human factors in a lot of the counterfactual universes—I feel like we don't end up with people who even pay lip service to the idea that anybody could have an ASI takeover. I think we live in a pretty good counterfactual universe, all things considered. We have a good set of game players on board.

That's right. That's right. How are you preparing for fast timelines?

9. Preparing for fast AGI timelines

If there are fast timelines, then there will be this 6-month period in which the most important decisions in human history are being made. I feel like having an AI podcast during that time might be useful. That's basically the plan.

Have you made any shorter-term decisions with regard to spending, health, or anything else?

After I interviewed Zuckerberg, my business bank balance was negative 23 cents. When the ad money hit, I immediately reinvested it in NVIDIA. Sorry, but were you asking from a sort of altruistic perspective?

No, no, just in general. Have you changed the way you live at all because of your AGI timelines?

I never looked into getting a Roth IRA.

He brought us Fiji Water before.

Which was in plastic bottles, so—

Dwarkesh has changed.

Well, have you guys changed your lifestyle as a result?

Not really, no. I just work all the time.

But you would be doing that anyway, or would you not?

I would probably be going very intensely at whatever thing I'd picked to devote myself to.

Yeah, yeah. How about you?

I canceled my 401(k) contribution.

Oh, really?

Yeah, that felt like a more serious one. It's hard for me to imagine a world in which I have all this money that's just sitting in this account and waiting until I'm 60, and things look so different then.

I mean, you could be a trillionaire with your marginal 401(k) contributions.

I guess, but you also can't invest it in specific things. I might change my mind in the future and restart it, and I've been contributing for a few years now.

On a more serious note, one thing I have been thinking about is how you could use this money to an altruistic end. If there's somebody who's up-and-coming in the field that I know, who is making content, could I use money to support them? And I'm of two minds on this.

One, there are people who did this for me, and it was actually kind of responsible for me continuing to do the podcast when it just did not make sense, as there were a couple hundred people listening or something. I want to shout out Anil Varanasi for doing this, and also Leopold, actually, for the foundation that he was previously running.

On the other hand, the thing about what that blogger was saying is that the good ones you actually do notice. It's hard to find a hidden talent. Maybe I'm totally wrong about this, but I'd feel like if I put up a grant application saying, “I'll give you money if you're trying to make a blog,” I'm actually not sure about how well that would work.

There are different things you could do, though. You could say, “I'll give you money to move to San Francisco for 2 months,” and sort of meet people and get more context, taste, and feedback on what you're doing. It's not so much about the money or time; it's putting them in an environment where they can more rapidly grow. That's something that one could do.

I think you do that quite proactively, in terms of deliberately introducing people that you think will be interesting to each other and this kind of stuff.

Yeah. No, I mean, that's very fair. Obviously, I've benefited a ton from moving to San Francisco, and it's unlikely that I would be doing the podcast—at least on AI—to the degree I am if I wasn't here. So maybe it's a mistake to judge people based on the quality of their content as it exists now and just throw money at them—not throw money, but give them enough money to move to SF, get caught up in this intellectual milieu, and then maybe do something interesting as a result.

The thing that most readily comes to mind is the MATS program for AI research. It seems like it's just been incredibly successful at giving people the time, the funding, and the social-status justification to do AI-safety-relevant research with mentors.

Oh, and you have a similar program. I know you're probably selecting for a slightly different thing, but I assume it's going to be power-law dominated. Have you noticed a pattern among the MATS fellows or your fellows of who is just like, “This made the whole thing worth it”?

We have the Anthropic Fellows Program.

That's right, yeah.

There have been multiple people whom Anthropic and other labs have hired out of this program, so I think the return on investment for it has been massive. And, yeah, apparently the fellows—I think there are 20 of them—are really good.

But what is the trick to making it work well or finding that one person?

I think it's gotten much better with time. The early fellows, some of them did good work and got good jobs, so now, with later fellows, the quality bar has just risen and risen and risen. There are even better mentors now than before, so it's this really cool flywheel effect.

Originally, it was just people who didn't have the funding or time to make a name for themselves or do ambitious work. It was kind of like giving them that niche to do it.

Seems really key.

You can do other things that don't have to be money. You could put out ideas for things you'd be really interested in reading or promoting.

Yeah, yeah, yeah. There's something coming there.

Any other questions we want to ask? The thing I have takes on, which I rarely get asked about, is distribution.

Distribution of AI?

No, sorry—MrBeast-style distribution, where people, I think rightly, focus on the content, and if that's not up to snuff, I think you won't succeed. But to the extent that somebody's trying to do similar things, the thing they consistently underrate is putting the time into getting distribution right.

I just take random takes about it. For example, the most successful thing for my podcast in terms of growth has been YouTube Shorts. It's a thing you would never have predicted beforehand, and they're responsible for at least half the growth of the podcast or something.

I mean, I'd buy that. Why wouldn't you predict it? I guess there's the contrast between the long-form, deep content and YouTube Shorts and stuff, but I definitely think they're good hooks. Good content.

I have takes on how to write tweets and stuff. The main intuition is to write like you're writing to a group chat—

To a group chat of your friends, rather than this formal whatever.

What else comes to mind here?

Well, maybe it's interesting—the difference between TikTok and YouTube Shorts.

Oh, yeah. We've never cracked TikTok.

Why not? You've tried?

Yeah, I mean—

Tried? Have you done everything?

No, I have not done everything.

Have you read these poems? Maybe you're in a bubble bath with some beard shampoo on.

Reading poems? That'd be incredible if you got any of that to go viral.

You have to do that now!

Manifest. Reading a poem, uncross your legs.

Last episode it was the interpretability challenge. Now it's Dwarkesh in a bubble bath.

I gotta sell the book somehow, you know?

But you literally do it like Margot Robbie—

Yeah, exactly. Explaining the scaling mechanism.

Yeah, yeah, yeah. So what is scaling?

And that's how you crack distribution.

And that's how you crack distribution. Oh, but, yeah, when we did our episode, it launched and you were sharing interesting tidbits about how it was doing, and the thumbnail you wanted to use and the title. I think I even asked you to share more details because it seemed interesting and cool—subtle things.

But it seemed like you also kind of just hated it, playing this game of really having to optimize all these knobs.

What I realized is that talent is everything, so I'm really lucky to have 3–4 editors who I'm incredibly proud to work with. I don't know how to hire more of them. They're just so good and self-directed. Honestly, I don't have tips on how to correct that. I hired those guys.

One of them was a farmer in Argentina, one of them was a freshman maths student in Sri Lanka, one of them was a former editor for one of MrBeast's channels. The other is a director in Czechia who makes these AI animations that you've seen in Notes on China, and he's working on more essays like that. I don't know how to replicate that catch again.

God, that's a pretty widely cast net, I've got to be honest. Damn. But they're all—goddamn, they're so good. And this was just through your challenges and tweeting about it?

That's right. I had a competition to make clips for my podcast. I rounded up a couple of them this way. It's hard to replicate because I've tried afterward.

Why do you think this worked so well with the video editors? You tried a similar approach with your chief of staff.

The difference is, with the video editors, I think there is this arbitrage opportunity where there are people who are willing to work hard and obsess about getting better over time. All of them go above and beyond on that, but you can just find people in other countries, and it's not even about the wages. I've 10Xed their salaries or something like that. It's just about getting somebody who is really data-oriented, and there is this global arbitrage there.

Whereas, with the general manager, the person I ended up hiring—and who I'm super excited to work with—is your childhood best friend, Max Hearn.

Max is so great.

He would have plenty of other opportunities. There's not this weird arbitrage where you find some farmer in Argentina.

Yeah. But yeah, it is striking that you were looking for a while and then just kind of mentioned offhand that Max was looking for something new.

This is going to be like a total 12-year-old-learns-about-the-world kind of question, but I genuinely don't know how big companies hire. I was trying to find this person for a year, and I'm really glad about the person I ended up hiring. But if I needed to hire 100 people for a company, let alone 1,000 people, I just do not know how to find people like this at scale.

Yeah, I mean, I think this is the number-one issue that startup CEOs have: hiring. It's just relentlessly the number one.

And the thing I was stunned by is how it didn't seem like my platform helped that much. I got close to 1,000 applications across the different rounds of publicizing it that I did. A lot of really cool people applied, I think. But the person I ended up hiring was somebody who was just a reference, a mutual-friend kind of thing. A couple of other top contenders were also this way. So it's weird. The best people in the world don't want to apply, at least to things like this, and you just have to seek them out, even if you think you have a public platform or something.

Yeah, the job might just be so out of distribution from anything else that people would do.

So Aditya Ray asks, “How do you make it on Substack as a newbie writer?”

I think if you're starting from scratch, there are 2 useful hacks. One is podcasting, because you don't need to have some super-original new take. You can just interview people who do, and you can leverage their platform.

And 2 is writing book reviews, again because you have something to react to rather than having to come up with a unique worldview of your own. There's probably other things, and it's really hard to give advice in advance. Just try things. But those, I think, are good cold starts.

The book reviews are a good suggestion. I actually use Gwern's book reviews as a way to recommend books to people. By the way, this is a totally under-supplied thing. Jason Furman is this economist who has about 1,000 Goodreads reviews, and I've probably visited his Goodreads on 100 independent visits. Same with the Gwern book reviews or something, right? So book reviews are a very under-supplied thing if you're looking to get started making some kind of content.

I like that.

Yeah. Cool. Thank you guys so much for doing this. This was fun. We'll turn the tables on you again pretty soon. How does it feel being in the hot seat?

It's nice. Nobody ever asks me questions. Nobody ever asks, “How is Dwarkesh?”

Cool. Yeah, I'm super excited for the book launch. Thank you.

The website's awesome, by the way.

Appreciate it. Yeah. Cool. Thanks, guys. See you later.

Thanks.