你最大的杠杆：设计你的 AI 职业生涯，实现最大影响力——对话 80,000 Hours 创始人 Ben Todd

Hello and welcome back to The Cognitive Revolution. Today my guest is Ben Todd, co-founder of the nonprofit career strategy organization 80,000 Hours and author of the book by the same name, which, 10 years after its first edition, is being re-released today, May 26, fully rewritten for the modern AI moment.

Looking back on the last decade, I think you'd have a hard time finding a source of information that would have better prepared you for the present-day situation than 80,000 Hours. Their emphasis on pandemic preparedness predated COVID by years and was tragically validated by the COVID experience. They were well ahead of the curve in recognizing the importance of AI and encouraging people to work on AI safety projects. Their podcast and blog have been a consistently excellent source of fresh perspectives on cutting-edge ideas, and their free one-on-one career advisory service was useful to me personally as I made the jump from entrepreneur to full-time student of AI roughly 4 years ago.

With that in mind, today's conversation is an overview of Ben's latest thinking on how you—yes, you—can apply your skills to improve the chances that AI really does end up benefiting all humanity. We begin with Ben's thoughts on AI timelines, a question that he cleverly reframes, encouraging people to ask not when exactly AGI or superintelligence will arrive, but when, under varying assumptions, your own personal impact will peak. Ben argues that under all but the most extreme short-timeline views, there is still time to invest in positioning yourself for maximum impact. From there, we turn to the top problems that Ben and the 80,000 Hours team see in the world today: that we might lose control of AI systems, that AI could concentrate power in unprecedented and deeply problematic ways, and that we're still not well prepared for the next pandemic.

We get Ben's perspective on arguments for and against AI safety-focused people going to work at frontier AI companies, including his thoughts on the critical discipline of continually questioning one's own motives and the importance of pure effects. We discuss different lines of work, including technical research, policymaking and advising, communications, and organization-building, all of which Ben believes have an important role to play. We assess the funding environment, with the encouraging conclusion that there is currently plenty of funding available to support ambitious projects. We analyze whether it makes more sense to join an existing organization that's working to scale or start a new one from scratch. And we get Ben's take on what new ideas, including concerns around AI welfare, gradual disempowerment, and space governance, are potentially undervalued today.

To be clear, this conversation is not a substitute for the book, which contains a lot more practical advice than we were able to cover today. But I hope it serves to inspire you to think bigger and more prosocially about your own career, and to alert you to the outsized impact that your personal contribution can still have on the shape of the AI future. With that, I hope you enjoy my conversation with Ben Todd, co-founder and author of 80,000 Hours.

Ben Todd, founder of 80,000 Hours and author of the new book by the same name, 80,000 Hours, coming out in the United States on May 26. Welcome to The Cognitive Revolution.

Thanks so much for having me. I'm a longtime listener.

That's very kind. I've certainly followed your work for a long time as well. The premise of today's conversation is that AI is going to be a big deal. The decisions that get made around exactly how to shape it, how to deploy it, and how to govern it over the next few years could have a really outsized impact on the future for a long time to come.

I want to get into that and get your take on how people who maybe don't work directly in the space yet, or who do a certain kind of work but aren't sure if they're making the biggest impact they can, might think about pivoting their careers to try to make the most positive contribution they can. But maybe before we get into that, I'd love to set the stage with the backstory behind 80,000 Hours, the story of the name, and the principles that you use as you think about how to advise people and how to think about their careers. With that table set, then we'll dive into the AI moment that we're in.

The name 80,000 Hours is taken from the typical length of a career: 2,000 hours a year for 40 years. The idea is that it's the biggest decision you'll ever make from the perspective of your personal life, but even more so from the perspective of your impact on the world. That's the biggest lever you have to pull.

Maybe it sounds obvious, but so much discourse about what it means to have an impact is focused on things like buying fair trade, turning off the lights, or cycling to work. It's all these “every little helps”–type decisions. But because your career is, for most adults, about half of their working life, it's more than everything else you do put together. Even a really small improvement in your career, especially in terms of its impact, is going to dwarf the impact of all these other small changes you could make. So it's really worth thinking about very carefully.

At the same time, so much career advice is basically a bunch of stories about successful people, or a bunch of slogans like “follow your passion.” Given the huge stakes of this decision, it's not treated with all the seriousness and research that you should actually bring to bear on such an important question.

The organization came from the fact that we were students at Oxford, asking ourselves, “What should we do with our own lives?” We wanted to find something that would be enjoyable, would pay the bills, and would make a contribution to society. We thought, “Which paths would be best? There's more than one path we could go down.” It was really hard to get any good advice on which to pursue, so we started doing our own research on this.

That turned into a talk we gave at Oxford. There were 24 people in the audience at the first-ever talk, and about 6 of them eventually totally changed their lives and came to talk about the ideas. Several of them came to us and said, “You should really start an organization about this. These are clearly powerful ideas that no one else is talking about.” That was the start. In 2010, we gave the first-ever talk, and then I started working full-time on it straight after I graduated. That's what I did for the first 10 years of my career: I ran an organization trying to figure out which careers were best.

A very small version of some similar advice that I've given to people over time, when somebody is so perhaps unwise as to ask for my take on their career, is simply not to be in a huge rush to jump into the next thing. A very simple bit of math I always tell people is: How long do you think you'll be at your next job? Typically, the answer is at least 2 years. Then I say, “Okay, in that case, if you could spend an extra 2 months finding the next job and, even considering only your narrow self-interest, it paid you 10% more, then that would be totally worth it.”

And yet people are often so anxious and feel such urgency that they can't be idle or not doing something for even 2 months. They'll, in my opinion, rush to do the next thing and really under-optimize what their next phase of their career could have been. That means taking the same idea, making it a lot more prosocial, and zooming out to the scale of really thinking about your entire career.

One thing that definitely jumps out about 80,000 Hours—and this has been true of the podcast and many of the different things that you guys have put out over time—is that you've been super early on AI and advising people to think about it, even back when there really wasn't all that much AI, at least of the sort that we now know to be concerned with. So how did you do that? What was the sort of habit of mind, principles, or framework that you were using even 10-plus years ago that allowed you to focus on AI being so important?

Obviously, that take has aged well. If you have any other takes that maybe haven't aged so well, you could take a little detour and share those as well, but I think that will be helpful for people to understand. Given your track record of forecasting what would be a good place to spend your career, I think that'll help build confidence in the rest of the advice that we'll get from you today.

Part of the context is that one of our most important pieces of advice is to focus on problems that are big, neglected, and potentially solvable. So then there's this question of what those problems are. We had AI as a potential nexus of issues on our radar very early on, partly because our paths happened to cross at Oxford with Nick Bostrom, who wrote Superintelligence, which was published in 2015, and a lot of those ideas originated earlier.

I think some of that was, at that point, almost just reasoning based on this very big-picture, Ray Kurzweil–style argument that computing power is growing and eventually there will be enough to get to human-level AI. That could happen at some point in the next couple of decades, or even if it's 50 years, it's a huge deal.

But I think where things really heated up was maybe around 2016. I wrote a small call-to-arms piece in 2016 that pointed to a number of things, saying, “Maybe now is the time to really focus a lot more on AI.” One big thing, I think, was AlphaGo beating Lee Sedol, which was big news at the time. But I think what it really illustrated was that deep learning is a really successful paradigm, and it seemed like we should just expect more breakthroughs to come if you drew that trend line forward.

We didn’t predict that LLMs would be as good as they were, but the very broad trend—that more important things would come out of this paradigm—was borne out. Around then, we also had OpenAI being founded, so there was a definite move toward people really betting on this approach. You could see that the potential stakes here seemed enormous. It’s very easy to forget how weird some of this stuff seemed back in, say, 2014.

There’s that famous Andrew Ng quote where he says, “Worrying about AI taking over is like worrying about overpopulation on Mars.” That was the attitude of a lot of people in the field at the time. But that meant these things were really neglected. We thought, we don’t know exactly when it’s going to come, but there’s a trend, and if it does come, it’s a huge deal, and it’s still being really neglected. That was the basic case for it at that point.

So, fast-forward to today, and I wonder: How AGI-pilled are you now? I think this is also a really important table-setting part of the conversation because so many AI conversations go sideways based on differing expectations of what’s going to happen, how fast, and how big of a deal it’s going to be. For the purposes of what to do in the next phase of your career, I think there are legitimately some people these days who are like, “I may only have one more phase of my career,” and so it’s time to go all in on something or liquidate whatever various forms of capital they might have accumulated. Obviously, others are planning for longer timelines.

I don’t know that you want to put yourself into a box as having a very particular view on that, but how are you thinking about it, and how should that color the rest of the conversation?

Yeah, I would say I’m pretty AGI-pilled on the scale of people. I really think this is a spectrum: over time, as more and more developments have happened, I’ve become more and more AGI-pilled. It takes time to really move your guts on this. I like to think in terms of—this is a big simplification—but I plan in terms of 3 main scenarios.

The first is that something like AI 2027, or what people at Anthropic or OpenAI say they’re planning to do and expect to happen, comes true: They manage to automate AI R&D in a couple of years from now, like 1 to 4 years from now. That can cause an algorithmic feedback loop, which then means you add maybe 5 years of AI progress in 6 months, 1 year, or 3 months—something like that. Then we have pretty powerful, pretty general-purpose autonomous AI, say, sometime around then. Maybe even 2028 in the more aggressive versions of that.

That’s what seems to me like the realistic shorter-timeline, faster-takeoff-type scenario. That one’s quite interesting because you have this powerful AI before most jobs are automated and before you have robotics. Then there’s this crazy deployment process that happens next.

But I do still think it’s quite possible that those timelines are a bit too optimistic and AI that can do AI R&D will take a bit longer. Maybe it’s in the early 2030s. Compute scaling will start to slow down, probably in the late 2020s, because at some point you use up basically all that fab capacity. It’s all used up, and then you have to start building new fabs, which takes a bit longer compared to just switching over capacity.

It’s still very possible that an algorithmic feedback loop isn’t possible and you can’t get a big acceleration. Then I think a lot of people think, “Oh, an intelligence explosion won’t happen.” That’s wrong. It will still happen, but it will just have to be driven by building way more chips. Then maybe it takes 3 to 10 years instead of 6 months. So that’s getting to really powerful stuff by the end of the 2030s. We can think of it as a medium-timeline scenario.

Then the third one is just that maybe—this seems increasingly unlikely—the current paradigm runs out of steam, compute scaling slows down and becomes too expensive, the revenue isn’t high enough, or whatever, because they can’t keep improving it. Then you might have a longer plateau until there’s a new paradigm, or you just wait for economic growth to build up enough computing power to push on to the next level. I like to think in terms of wanting to plan across these 3 scenarios.

Nice. I don’t put too much weight on that last one, for what it’s worth. It seems to me like the path is pretty clear, and I do have a little bit of a tendency to—

That’s interesting. I wouldn’t even say I’ve necessarily underestimated the timelines for capability advances in AI. If anything, I’ve probably slightly overestimated how long it would take for models to get as good as they have already become.

On the flip side, I’ve tended to expect impact to come a little sooner than it has, and that disconnect is definitely an interesting thing to try to constantly recalibrate myself on. But I can’t find the wall that would lead us to the plateau scenario.

Well, I would agree, but I do think the compute issue could be that wall. One way of seeing this is that we have maybe 4 years of current scaling, roughly until 2028, and then a bit slower until 2032. If at that point you haven’t automated AI R&D, then you might plateau around there. It might just be very hard to clear the final bottleneck. That’s the kind of way I’d make the case.

Again, it’s also interesting to see the AI 2027 forecasts. If you look at their confidence interval, I think the 80% confidence interval is something like 2027 to 2050. Even Daniel Kokotajlo is saying there’s a 10% chance that it’s beyond 2050. So, yeah, I think there’s some chance of it, but it was not the main thing I would bet on myself.

I agree.

So anyway, given that context, people can choose for themselves where exactly they want to sit there. You have a 5-part framework for how people should think about their careers. Maybe introduce that, and then you can comment a little bit, too, on how you would reweight those different pillars of building a career depending on where you sit on that timeline question.

Yeah, in the book, there are many different frameworks, but we have this one, which I call the career framework. If you’re comparing a list of options and you want to do a side-by-side on them, what things should you look for? In brief, that’s the impact of the option; the career capital you’ll get from it, which is overall career development, skills, connections, credentials, and character; your personal fit, so how good you will be at it compared to other people, and all the other things that matter for job satisfaction and your personal goals; and then you could have exploration value at the end.

What will taking this job tell you about which jobs are best for you in general? What will you learn about your career from taking this? Many jobs are experiments as well.

With the effect of AI, the key thing is that the longer your time horizon, the more important gaining career capital and exploring are, because you have more time to make use of those skills you’ve gained and that information you’ve gained about what fits you best. The shorter your time horizon, the more you should just focus on doing whatever is best in the next couple of years. So, as AI timelines have come down, career capital and exploration value are a bit less weighted. But you wouldn’t ignore them entirely.

One way to see that is, suppose you have a very short timeline and you think, “Okay.” And also, to be clear here, when I say timeline, what I actually mean is not when we’ll have AGI. It’s not an AGI timeline. What matters is which years will be most impactful. If you want to have an impact, at which point will you be able to have the biggest impact? The time between now and then is basically your time horizon for acting. It’s when the best opportunities are.

If you’re very focused on AI, that will probably be the years immediately before and after AGI is developed. Though, in a slower takeoff, it could be 10 or 15 years after. There might still be all kinds of crazy stuff happening that you could help with. So, probably your timeline shouldn’t be 3 years. In expectation, there will be things to do after 3 years that are really impactful as well, so you should probably be planning over 10 years.

Ajeya Cotra had this recent post about whether you should get married, and she said the expected length of your marriage now is 10 years. That was her averaging over different timelines in terms of when the world will just become so crazy that it’s not worth planning beyond that.

And maybe 10 years feels a little bit long for a career-planning timeline, but something like 5 or 10 years, I think everyone should be thinking about at least that long a period. Should you ignore career capital and skill development? No, because if you could spend 1 year and make yourself 20% more productive, you would recoup that after roughly 4 or 5 years. So even if you only had a 5-year time horizon, you should still make those investments that will pay off over that period. There are often a lot of things people can do to increase their impact by more than 20%.

For example, retraining in machine learning—lots of people have done that—or entering a policy career. You can often transfer into one in a year and then maybe have much more impact after that. So I still think it’s really worth thinking about what you can do to maximize your impact over that whole time horizon, but it’s just a bit shorter than in the conventional world, where maybe you have 40 years to play with and you can really invest for 10 or 20 years and then pay it all off in your 50s.

Let’s also talk for a minute about what you think people should be trying to move the needle on. AI is one of the premises of this show, and one of the reasons I enjoy making it so much is that it’s obviously a general-purpose technology, a horizontal layer, something that intersects with everything. Increasingly, work in AI is almost saying nothing, because in any sector or any business, there’s going to be some sort of AI touchpoint soon, if not already.

What do you think are the big levers that really matter most? Maybe give a little argument for each of those big focus areas.

I’d start with what problems to focus on, and then you could talk about the best interventions or solutions to those problems. Again, here we would use the framework of which problems are most important, most neglected, and most solvable. With importance, you can partly think: How likely is this problem, and how big would the scope be if it happened?

I also quite like to think about urgency, because some problems come before the others, so you need to address them first and then deal with the other ones afterward. Or they might help you deal with other problems, so you want to deal with them earlier.

We have a list on the website that we’re always updating, depending on how our assessment of these factors is changing. Right now, we have loss of control of autonomous AI as the top, just because the stakes would be so big if we lost control of human-level-or-beyond AI. That might be irreversible and could eventually result in total human disempowerment.

It is much less neglected than in the past, but we’re still probably only talking about 1,000 or 2,000 full-time people working on these risks. By comparison, exactly where you draw the circle around the AI industry is unclear, but there could easily be 100,000 or 1 million people essentially working on AI capabilities and accelerating it even faster. So that still seems like a big, neglected, important risk.

Over time, we’ve broadened out the other risks that we’re focusing on. One that is a bit newer to the list is concentration of power. There are a bunch of ways AI could be very concentrating or very centralizing.

One is if there is an intelligence explosion in a feedback loop. In exponential growth, the gap between the first project and the second project stays the same. If it’s a 2-month gap now and they both grow exponentially, it’s still a 2-month gap in a year’s time or 2 years’ time. But if you go into hyper-exponential, or super-exponential, growth, then the gap actually widens.

You could have one company drawing well ahead of the others, gaining a digital workforce that is equivalent to a whole nation’s worth of people today. That would be more power than any single company has ever had before.

There are a bunch of other ways AI could be centralizing. We saw the Department of War was pissed off that they weren’t being allowed to use Claude to spy on every American. That’s something that wasn’t technically possible in the past, because they just didn’t have enough staff to analyze all the data and figure out what everyone was doing. But with enough AI, you could literally trawl through every bit of public data about people and probably build up quite a good picture of what they’re doing.

That makes surveillance much more effective than it’s ever been in history, which would be great news if you’re a wannabe dictator of any kind. A lot of these elements of this concentration-of-power issue are even more neglected than alignment. There’s very little thought right now given to things like exactly what instructions more powerful AI will obey in different circumstances. Is it possible for the CEO of a single company to just tell the AI what to do, or should there be more safeguards than that?

The third—and maybe I should have started with this—is engineered pandemics. They seem like they’ll be possible eventually even without AI, but AI could make them possible sooner. It definitely seems possible to make a pandemic that’s much worse than any naturally occurring ones.

Countries like North Korea might do this for deterrence: “If you invade us, we’ll release this virus.” Mutually assured destruction, but with something that’s much easier to build than 1,000 nuclear weapons, which is what that kind of thing used to require in the past. And then that could also just get out because of a lab leak. There are lab leaks all the time, so if these things ever get built, they’ll probably end up getting out at some point.

Those are the top 3, and then there are also some even weirder and more neglected issues that we could talk about later. They’re interesting for some people to work on, even if they’re not our mainline suggestions.

These AI companies and their marketing campaigns are getting weirder and weirder. First it was, “AI might kill us all; we’d better be really careful about it.” Now, I was just at an event in San Francisco this past weekend, and there was definitely a lot of talk from people at the frontier companies about concentration of power.

That’s another really weird marketing campaign for them to be running: “We’d better be careful about this technology, because we might end up having a crazy amount of power that nobody else is going to be happy with.” It’s a very strange way to—I mean, this technology-as-marketing thing just doesn’t really ring true to me at all.

I just think these people actually believe AI is a huge deal and that there are these risks. That’s the simplest explanation of what’s going on, not that it’s some kind of plan to make themselves seem really important and therefore raise more funding.

Yeah, that is quite the double bank shot, if it is. Let’s do maybe just a minute of marketing for the AI companies—or at least I will.

Obviously, all those top cause areas that you laid out are downside risks, and I think 80,000 Hours, fair to say, has been associated with downside prevention or downside mitigation over time. Is it fair to say that’s in part because, speaking for myself, my gut says that if we can avoid all those problems, we’re going to have it pretty good? I don’t know what life is going to look like; it’s definitely a fuzzy picture.

One of my own personal cause areas that I recommend all the time is trying to develop more concrete visions of what a positive future and a rewarding life in the context of AGI abundance could look like. Are you coming at it from the same perspective, where your belief is that there’s a lot of upside as long as we make it to the other side of some transition? How do you think about upside and making sure we realize the upside? Do we need to, or does it just take care of itself?

In general, I just want to focus on whatever has the most impact, which could be increasing upside or avoiding downsides. The question is just what the best opportunities are right now.

Maybe this is going to get too theoretical, but I think it’s interesting. There are roughly 2 ways to improve the future. There’s speeding it up, so bringing a better future earlier in time. And then there’s making sure it happens at all, or changing where it ends up in the final reckoning.

There’s this interesting point that if you think the future is going to get way better and you can speed that up by 1 year, it doesn’t only help people now; it actually means you have 1 extra year of that great future. Beth Bezos actually makes this point in the AI doc. He’s like, “Well, we need to develop AI ASAP, because we’re bringing forward the glorious future of everyone being in space, with crazy technology and all these things.”

This is the argument that Nick Bostrom makes in Astronomical Waste and that is the astronomical waste he’s talking about. But then there’s a second half of the paper where he points out that if we’re going to get to this great future, speeding it up only gets you a little bit extra of that.

If you can avoid an extinction risk, that means the whole future never happens at all. That’s way higher value if you actually believe in this better future. I was joking that Jeff Bezos is like someone who’s read the first half of “Astronomical Waste,” but then kind of got bored and didn’t really read the second half, where it says this is not what to focus on.

Toby Ord has a paper where he works this through in more detail. So that’s the very high-level perspective I come at this with: AI development is already going extremely fast.

By working on just bringing that a little bit earlier, that's not having that big an effect on the world. But if there are risks that might jeopardize the entire future, and that future might be a lot better because of the technology, then there's a huge value in reducing those risks. At the same time, that's way more neglected because, as we just said, there are already a million or so people working on accelerating AI capabilities, but the number of people trying to avoid these big downside risks is in the thousands. And so, that's how we see the basic case for focusing on those right now.

Okay, cool. Let's get even more into the nitty-gritty of it. I think obviously a lot of this stuff is going to be highly individualized, right? I want to preface by saying nothing can really be one-size-fits-all because everybody has their own skill set, background, and so on.

But we can—I don't know if you have a better way to frame it—I think, as we get more and more specific, frame the conversation in terms of, “If this is you, here's how to think about it,” with the understanding that individuals will probably only see themselves in certain sections, not all of them. To some degree, there's only so much that somebody can change about themselves, and they probably shouldn't try to go too far in terms of making themselves into something that they're not, or that would be too unnatural for them to try to become.

With that caveat—and you can elaborate on that caveat—how would you break down the roles that people could pursue? We could do roles and skills. What are the kinds of jobs that exist? Which ones are in most demand? Who are they looking for?

On the really broad approach, a very simple approach I like is just to come up with a short list of plausibly impactful things that could tackle these problems and then choose between those based on personal fit. So, if you have 5 ideas, try to figure out which one you're best at. That's often the best approach if you really simplify it down.

In terms of what roles to focus on, a couple of the most important types would be technical research and engineering. There's a lot that needs to be done, with loads of subprojects in that area. One that's on my mind recently is Metaculus, which does some of the best AI evaluations, so that we can actually know what is happening and how close we are to automating AI R&D. That might be the most important thing going on in the world right now, but we have very poor measurements of it. They're pretty much the leading group doing this.

They were saying recently that they have 20 really useful projects they'd love to do, but they only have the capacity to do 2 or 3 of them because you need a bunch of engineers to implement all this stuff. Other examples include AI control research and AI interpretability. One interesting thing that's happened in the last few years is that a lot of these areas used to be more of a high-level conceptual research bottleneck, but now it's much more about actual engineering.

There's a lot of concrete work to do: Can we use this AI to monitor that AI? Can we red-team in this way? Can we detect deceptive behavior quickly? It's about figuring out all the systems needed to do that type of thing.

Second would be government and policy. A lot of this stuff will have to involve government in some way. There's still a big lack of expertise among people who straddle the AI technical world and the government world. So, there's a lot to be done in building up that community and working on some of the priorities there, which we could go into.

Third, I might say, is communications. The level of understanding of AI is still really low, and very few people are working on the risks. Just getting the ideas out there, improving understanding of them, and mobilizing people to work on these things—there are huge amounts to be done there.

A fourth big category would be organization building. We need people to run the organizations that do all these other things. That's management, accounting, legal, HR, recruiting—just loads of getting-stuff-done-type skill sets.

There's also lots of need for all kinds of specialists in other areas: lawyers, economists, and engineers to work on biosecurity. Even historians have useful things they could potentially do. So, as the very broad categories that apply to a lot of people, the ones I just mentioned would be technical research, government, communications, and organization building.

Fun fact: in the 347 episodes we've done, I've had 1 historian on the podcast, Mark Humphries. He was doing some pretty interesting stuff in terms of using AI models to transcribe and make sense of all these old documents in the Canadian archives that nobody had really ever looked at.

His pipeline has changed a lot, of course, but he also had a really interesting and quite viral post on Gemini 3 just before it came out. He was showing how zeroing in on a particular ability it had unlocked—to reason about what it was seeing in visual documents—in ways that prior models simply hadn't been able to do.

I think that's a really interesting example of how something that seems so far afield can contribute back into the mainline discourse. Because AI itself is getting so far afield, even in such an unexpected niche there's an opportunity to make interesting discoveries that really advance people's understanding. So, I would again just encourage people to think pretty broadly about how many different opportunities there might be.

I can also say, again, from this event this past weekend in San Francisco, Meter in particular is desperately trying to find people. It's all a sign of the times. This is maybe a little bit of an exaggeration, but the attitude was, “Yeah, all of our measurements are pretty much saturated, and it's getting tough to increase the scale beyond what we already have.”

There's a lot of work to be done as the models are racing past our ability to measure them. They're looking to staff up and bring as much talent to bear as they can on keeping a handle on exactly what the current capability frontier really looks like.

Many of the things that you outlined there notably happen both in the frontier companies that are developing the AIs and in a variety of other organizations that orbit them—in some ways check them, and in some ways maybe even oppose them. This has been debated for a long time, and I think people have very different intuitions. Sometimes the discourse has gone around in circles.

How do you think right now about whether people should try to go to the frontier companies and contribute there, versus holding the person and the kind of contribution they're going to make constant and doing that from some other outside angle?

The short answer is that it's complicated. If you want to do technical alignment and control research, the frontier companies are some of the best places in the world to do that research. They have some of the strongest research teams, and they're also in a great position to actually implement the research, which I think is a big part of it.

You can come up with some idea, but if someone doesn't actually see through all the details of implementing it in the product, it doesn't help that much. On the other hand, a lot of people have done great research outside of the labs. I would name Redwood Research as another group here that helped pioneer the AI control agenda. They've done a bunch of other really useful pieces of work, including deceptive-alignment research with Anthropic. So, it's definitely possible to do work outside of the labs.

Another big factor is that the big argument against it is maybe you're speeding up AI development, which is hastening the risks.

I think that is worth thinking hard about. Each individual has to make a judgment on that themselves and how they want to relate to that. I think another big factor is how aligned you are with the people who work in the labs. In general, I try to avoid adversarial strategies, but if you have the attitude, “AI is happening, and I prefer to have a more socially minded or safety-conscious company win, so I’m going to try and help them win,” I don’t think that should be entirely ruled out as a strategy.

I think what drives a lot of the disagreement here is that people who think, well, we’re very likely to have an existential risk and alignment research doesn’t really have any hope of working, basically think the only option is to have an indefinite pause. By working at the labs, you’re not really helping with that or actively making it worse. Whereas people who are more of the attitude that, A, this is happening, we can’t really stop it, and B, we’ll probably get through, but it’s more a question of making the chances as high as possible, tend to be much more keen on working at the labs. So, I think that’s another really big source of disagreement over this.

Yeah, there are so many different angles on it. I find them all compelling, but I also find myself going back and forth on it at different times. Maybe just to name a couple of arguments that people might want to check out. One, which I think came from Redwood Research, is the “10 people on the inside” argument: unless there is some international treaty or whatever—which I certainly don’t rule out, but we’re not that close to it at this moment—this is going to happen. If it’s going to happen, then a small number of people working within the companies who really care about the right things and can take important actions at critical moments could be one of the most important places to be, one of the highest-leverage places to be, certainly.

Then another argument, especially if you want to do alignment or interpretability research, is that you can do a lot of that research outside of the companies. But one of the big things we’re worried about is that models are going to change in important ways, potentially in a pretty short period of time, potentially in surprising ways even to people within the companies, due to emergent properties that arise, right?

We see the general pattern of AI capability advances as the loss function dropping smoothly, but the specific tasks that the model can do often seem to have these more discrete jumps from one generation to the next. We have seen, obviously, many of those on the capability side, but also some of them on the problematic-behavior side, where it seems like we’ve gone from models that didn’t really do any sort of deception to models that now sometimes do some deception. Obviously, that’s not something the companies train for, but with a certain scale of RL or what have you, all of a sudden that seems to pop up from one generation to the next.

So again, being on the inside of the companies and having access to the frontier models might be pretty important to being able to discover that next bad behavior in time to make a difference about it, or to just do interpretability research to understand the internal workings of models well enough, or soon enough, or at the frontier that matters the most, to really make the biggest difference. I’d be interested in your reaction to all of those, if you have any.

I guess I would say, for me, the alignment thing resonates more than the interp side. I feel like with alignment, you do see these qualitative shifts in behavior, and I think trying to align whatever is the best open-source model at this moment is probably quite a different activity from doing alignment work on mythos, for example. Whereas our interpretability understanding is still sufficiently nascent that you could probably still make quite a bit of headway on something like GPT-OSS or any of the Chinese models. There’s still plenty there that is not understood.

Any other analysis you would give on those early, well-known, but still important arguments?

Yeah, I think you’re exactly right. It depends on the specific project. Maybe to throw another argument in there: I do think it’s very easy to be biased toward wanting to work at a famous, rapidly growing, highly paid company that everyone is talking about in the world. It’s pretty cool to be like, “Oh, the most impactful job for me is to work at this company.” That’s a very convenient place to be in, right? So I think people should also question that—question their motives—when they’ve concluded that’s the best option.

In general, if someone’s going to do it, I would want them to have a pretty concrete strategy that they’ve actually thought about: why it needs to be executed in this way rather than something that could be executed in a different way, like you’re suggesting with the open-source models in interp.

With the 10 people on the inside, I think part of the idea is that there are diminishing returns. There’s low-hanging fruit, so being able to just take the first couple of opportunities might increase safety a lot, even if we’re very far from what would optimally happen. You might also be able to do things like raise the alarm about stuff that’s gotten really out of hand.

But it does seem like, for most people, it’s not sustainable to work at a company where they just disagree with the mission of the company, and they tend to only last a couple of years before quitting, as we saw with pretty much everyone who quit OpenAI who was especially safety-concerned over the years. So, in general, most people should probably work with a company where they’re aligned with the mission and they don’t have that source of friction.

But if you are that very unusual type of person who could work somewhere where you really disagree and you think you might be able to make things a bit better at the margin, I do think it is worth considering, because basically all the options seem bad in different ways. The idea that no one who’s worried about safety works at any of the top labs or top companies is also a pretty scary world to me. It seems like, ideally, if it was possible, you’d have some people at all of them. But whether there are actually people who can carry through with that and make it worthwhile is another question.

Let’s look at that in a couple of different ways. I really appreciate you saying people should question their motives. I think this is of profound importance for individuals at AI companies and for AI companies as organizations. There’s just a lot of, “Somebody’s going to do this, and we’re better than the other guys, so we better do it, and therefore we need to win.” Yikes. This sort of leads us to a competitive situation, which I think is quite problematic.

I always say it’s the smartest of times, it’s the stupidest of times. This image that I have of Sam Altman and Dario from the AI event in India, where they couldn’t bring themselves to join hands as everybody else on the stage was joining hands, will be a real time capsule and a real shame if we end up messing this up.

If there are some alien historians or whatever one day excavating the planet and they come across this image, they’d be like, “Oh, they were so close. They almost created a world of abundance for themselves, but in a scene fit for a Greek tragedy, here you have the 2 leaders who are basically building the exact same technology at very similar companies with very similar strategies, hating each other as individuals so much that they couldn’t manage to figure out any way to work together or couldn’t see the virtue in what the other one was doing.”

It’s a real mess. That is a real mess, and I think that’s something that everybody needs to think about for themselves: how do I play a positive role in making that less of a mess and not contribute to just everybody escalating the same dynamics that currently exist?

So, with all that said about counterfactual analysis of this sort—somebody’s got to do it, so I guess it should be me—and how do you advise people to really take an honest look at their own character and assess, “Am I actually going to be able to go into such an environment and do what I, in the abstract now, think I will do or want to do?”

It’s a hard question, and I think it’s hard to get that kind of objectivity on oneself, but it’s a pretty important crux for whether you should do it or not, right? Can you hold up under that pressure and be true to your values when the stakes are high and when it really might cost you something? How does one get a good sense for whether or not they’re up to that challenge?

I think this is a huge topic. One starting point would be, like you’re kind of gesturing at, that a lot of the reasoning in there is pretty shallow. It’s just, well, the company has good vibes, so I’m going to encourage you to try and be more objective than that.

Think of these companies like political actors. All the leaders have many incentives. They have many different goals, not all of which are noble. Then try and get down to the level of specific decisions and the track record that they have. Did they actually make the hard decision in that case or not?

Try and be grounded in objective decisions as much as possible. When they made this commitment, did they follow through on it? What do people say about this person’s character who know them well? Actually try and take a view on it rather than just go with vibes.

That's about assessing the company. Assessing yourself is even harder, because you're going to be super biased there. This is just one small technique, but I do think cultivating some close friends who are willing to call you out on stuff is really helpful for this. Most people won't really call you out. They'll just go along because they want to have a friendly relationship. So if there are any people in your life who will do that, I think that's really valuable.

Yeah, potentially making commitments to them as well—almost like a written-down commitment: If this type of thing happens, I will do this. I think it's very easy for your standards to slide gradually over time, right? So finding ways to have some kind of pre-commitment in place where you're going to reassess, I think, is really helpful as well.

I guess a fourth thing that comes to mind is that you will become more similar to the people you work with. So, again, that just means really taking the context seriously. And like we said with the 10 people on the inside, most people can't—if they think there's a company that's being really reckless—actually hack going in there and trying to make it better. That's actually a very difficult thing to do.

So I think you should be skeptical of your ability to change the prevailing culture and the people around you. Just focus on picking the right culture in the first place.

What is the argument that people have made a lot who I think are especially skeptical of AI companies, but I guess they may just have a high P(doom), as you framed it earlier, that safety work is ultimately still counterproductive? Sometimes that's framed as saying safety ultimately is just an accelerant of the capability frontier. For example, we had things like RLHF that were supposed to be a safety measure, and in some sense they do make any given individual model safer to use, but then they also have had the property of making the AIs much more useful and just accelerating the overall pace of development in the space, bringing lots more resources in as it became clear that these things can, in fact, be harnessed and become super useful. I wonder how you feel about that now.

The argument that those people usually make is just to try to shut the whole thing down. This is not my point of view, to be clear, because I do really value the upside that we're getting, but there's some truth to that argument. I can't quite dismiss it. How would you assess that argument, and how moved should people be by the “even AI safety, even technical AI safety work, is still bad” argument?

Technical AI safety definitely does seem to sometimes improve capabilities and therefore accelerate AI overall. The question is just: Are you getting enough benefit out of it to make up for that cost? Worlds where we don't do any safety research—that seems pretty scary, right? My sense is it's best to have a thriving safety ecosystem that's really tracking the biggest risks and trying to do something about them, even if it will also accelerate things a bit more compared to a counterfactual where things are still going very fast, but we don't have a safety research ecosystem.

It does depend on whether you think—if you just think alignment or AI control is ultimately impossible, then, yeah, that doesn't make sense. I think a bigger point I might make about this is that I come at this from a perspective of just—we're very uncertain about all these different priorities and what's going to happen, and that, at least for me, makes me intuitively want to see a bit of a portfolio of efforts across different scenarios.

So I'm pretty into the idea of there being some people who are really trying to build support for some type of international pause. I think a 1-year pause at the point at which an algorithmic feedback loop becomes possible, or some type of temporary pause, seems like it could be extremely helpful. At the very minimum, that's a thing I think many people could agree upon. But I also want there to be a lot of effort going into the scenarios where this is happening—it's not really pausing—and just betting 100% on the pause seems like that's leaving a lot of low-hanging fruit on the other scenarios on the table.

Yeah, a portfolio approach, I think, is kind of always an answer to this. It does seem like there's room for—I mean, it depends on your probabilities on the different scenarios and what your portfolio would be—but at least for me, they're also pretty spread out.

Okay, let's keep moving through a couple of other areas. I don't know if you have anything more to say about working at frontier model companies. If there's not, we can move on, but if you have any other kind of specific advice, I think one of the things people are going to ask is—there was a pretty good post on this just yesterday. I forget who it's from, but it was, I think, a pretraining lead at Google. It got some plus-ones from others around the space, including Sholto from Anthropic. It was very much leaning into just grinding hard on technical skill.

I don't know if you would second that or highlight anything else that you think is the most important way to position oneself for work at those companies. Obviously, it's super competitive. They're hiring a lot, but it's still sufficiently competitive that I don't think it's an easy place to find your way into. So, any other thoughts on that before we move on to other venues for possible impact?

You mean advice on how to get jobs at these places? This is just very general-purpose advice, but there's nothing to substitute for talking to people at the company and just saying, “Okay, if I was going to do a 3-month crash course to be in the best position possible to get this job, what should I do?” That answer will depend on different people, but often learning technical skills is really useful. They're also hiring a lot of other roles, so it would depend on what role you're going for. I wouldn't have a ton to add on top of what people in those companies would say.

Got you. Then let's change gears to policy. This is another, I think, very black-box sort of thing to many people, right? If you're not inclined to do hardcore technical work or research, and especially if you have a higher P(doom), then you're like, “How can I get into the policy arena?”

I'd be interested to hear your thoughts on, first of all, what policies do you think are actually robustly good and worth advocating for? And then how should we think about how we can position ourselves to actually move the needle on the likelihood that those things do, in fact, come to pass?

Yeah, we touched on the strategic pause earlier. But, like, I agree with the current administration, and just the vibe is that it's very unlikely to happen in the short term. We need to think out a couple of years into the future, when people will be freaking out even more. We might be on the cusp of an obvious feedback loop. There might be a new administration. That could be a pretty different scenario, and I think at least a strategic pause could be something that a lot of people would sign up to. They'll just see that it's better for everyone.

The common response is, “Well, what about China?” But China would obviously have to be part of the deal, and you would say, “We're going to pause if you pause. That's the deal.” They actually have more incentives to take that deal than we do because they're behind. They also care about the risks a bunch. They've had high-level talks about AI risk among the very top leadership of the party. So I do think laying the groundwork for that is useful.

One thing in particular is that you can't really do that unless you have some way of enforcing it, which means you need at least some minimum level of tracking the compute and knowing where major AI training is taking place. If that's not set up in advance, you just can't actually really do it. So I think things around compute tracking are really useful.

This almost sounds too simple, but getting the capacity to have some type of off switch seems useful, because right now there's not an easy way to just turn off a lot of compute quickly if there were an autonomous AI replicating in a bunch of data centers. That's kind of a relatively low-bar thing you might want to have prepared.

Then there's a bunch of maybe more boring-sounding things, but I do think more transparency is good, like making sure the government can actually track what the capabilities are over time. Right now, it would be possible for a company to start an intelligence explosion and keep it secret potentially for a couple of months at least, which could be important. So measures to really make sure we would know if that was happening seem really good.

There's a bunch of other ideas. I do think the idea of having red lines and emergency response plans also seems pretty hard to object to. Dangerous behavior that would trigger further investigation or certain responses by the companies, and having that widely agreed upon across the industry.

But this is more on the alignment stuff. Obviously, there's a whole set of other things for pandemic prevention, avoiding concentration of power, and a bunch of the other risks. In general, I just feel like there's a lot that could be done. Even if it can't be passed in the current administration, there's a lot of groundwork that can be laid to make it much easier to implement these things in the future.

How would you handicap where we are in terms of the development of policy proposals versus advocacy or other efforts to bring them into the Overton window and eventually make them happen? I guess one way to say that is: do we have most of the good ideas we need, or do you think there's a lot of blue-sky policy room still out there to explore and develop new ideas? For the ideas that we do have, are they well developed, or do they still need a lot of work to really flesh them out?

And then the third part is that these things are not immediately about to happen. So for that one, I guess it's more like: where do you think people should go? What's the best margin to work on to actually move the needle on the probability of it happening? But yeah, I gave you a lot there, so take your time.

I'm going to slightly cop out with a portfolio approach again. I think all of these things are pretty needed. A lot of the policy proposals aren't super fleshed out; there's a lot more that could be done with fleshing them out. I think we also need people developing new broad approaches. New approaches have only really been developed in the last couple of years, so I'm sure there's a lot more that could be discovered.

And then I think there is also a big political-will gap to actually get any of these things implemented. My sense is that understanding of AI is still very low. It's much better than in the past, but the number of people in Washington who really know what the METR time horizon chart is seems actually still pretty small, from what I can gauge. That seems like a very basic level of understanding of the situation.

I think more training—getting people to understand the issues and care more about them—would help. I think more popular support helps as well, because politicians won't do things unless there's some type of electoral incentive, ultimately. They will sometimes, but it's much more touch and go. There's still very little of that.

There's a lot of just general negative sentiment about AI, but it's being directed into banning data centers, which doesn't really seem to help because those companies will just build those data centers somewhere else, potentially the UAE, which seems significantly worse. If we're lucky, they'll build them in Australia or the UK or somewhere.

It's that classic cradle of American values, the UAE, which has also proven to be quite a dangerous place to have a data center as of late. I don't know how those plans are going now, but I think some of them definitely have to at least be rethought on a couple of levels, I'm sure.

Let's hear one more beat on pandemics, too. To the degree you can focus this on the AI angle and pandemics, the audience is pretty diverse—actually, I've learned that over time. But if there's one thing that unites us, it's a general obsession with AI. Obviously, some of the things people can do on pandemics are not going to be AI-related, but increasingly it seems like some are, and I'm interested in your thoughts on pandemic prevention, especially as it intersects with AI.

Pandemic prevention is a really exciting area because there's just so much to do that would really help reduce the risks. With all these alignment issues, everything is much more debated. We don't really know what's helping or hurting, but with pandemics, people agree there's a lot of things we can do that would very likely help.

Then, in terms of using AI to help reduce these risks, I think there could be some things in this area. We could be using AI to do better disease surveillance, monitor all these gene-synthesis companies, and check that they're not being used to manufacture super-deadly viruses.

But the main thing that comes to mind is that the other things we need to do for pandemics are these very entrepreneurial, engineering, build-build-type projects. I would say if you're working on AI now and you have this kind of “I just know how to build stuff with AI, I know how to make systems with AI and get stuff done” skill set, then you could just apply that to these bio-risk projects. You could take that skill—that kind of entrepreneurial, doing-stuff skill set—and apply it to this cause.

A couple of the projects that need to happen are wastewater monitoring, so that we can detect new pandemics really early. The cool thing about this is you can just synthesize all the DNA, and then you can look for anything that's growing exponentially, which is a signature of a pandemic. That lets you pick up pandemics that have been completely unobserved in the past. You can potentially detect them earlier.

Another project is red-teaming all the gene-synthesis companies to make it so that it's not easy to use them to start developing dangerous segments of viral DNA. That's preventing it from happening at all. The wastewater stuff is making sure we know quickly when it is actually starting.

And then the third is just lots of measures to prevent spreading once we've noticed something. Again, we could design much better PPE that's cheaper, more effective, and more comfortable. Then we could have huge stockpiles of it ready to go for when the next pandemic happens, and give it to all the essential workers who, during COVID, did keep working.

But that's because COVID was not actually very dangerous for young people. If you had a pandemic that killed 10% of people—or even 80% of people who got it—I don't think you'd see a lot of people packing shelves at supermarkets. So we would need PPE to keep society functioning. That is the kind of thing that costs billions of dollars, but that's not expensive on the scale of the risks here, relative to budgets and so on.

And then there are things like air filtration in homes. If you have positive air pressure and filter the air that comes in, then it's basically preventing air from flowing into your house, and it can keep it pretty clean. UV lights can sterilize the air, and we could just have those installed as standard. These would also make the common cold way less common, which would just be a really nice side effect, even without all the pandemic concerns.

The final pillar would be really rapid vaccine development. The type of person who could run a startup is the type of person who could do one of these projects. I think it's maybe been a little bit neglected compared to AI at the minute, especially these projects really aimed at cutting off the worst-case scenarios. There's just not that many people working on them. We're talking about a couple of key organizations really focused fully on them.

You mentioned people coming from entrepreneurial roles, and that reminds me of a conversation I recently had where somebody said they were trying to help staff certain roles in the government and were looking for people with an entrepreneurial background. I was like, “Hmm, that's an interesting juxtaposition.” I'm not sure how many entrepreneurs are going to want to go join the government, actually, because I think that personality type is often one that started a company in the first place because they didn't want to work at a big company.

In some ways, the government is the even bigger, more bureaucratic, more frustrating environment to work in for people like that. Do you have a taxonomy of backgrounds or personality types that you think map particularly well onto the problems that are most important or the different organization types? I'm interested in who should go into government. Maybe it should be entrepreneurs, but if it's not going to be entrepreneurs, who should we be looking for to take on these governmental roles?

I've also heard that entrepreneurial thing. I think the entrepreneurial skill set—making difficult things happen, building coalitions of people, and setting up systems—is just super valuable everywhere. The key filter would just be: could you actually hack this, or would you find it too frustrating? But if in doubt, I would say try it.

People have a lot of misconceptions about careers; they see them in a very picture-book vision. Maybe they picture the DMV or something, where they think that's what government is like. But there are so many different organizations and teams, and maybe you find some that you're really aligned with. It's a very different experience working with them.

Also in government, there's a lot of need for more analysis and technical-expert-type roles, which can fit people who are a bit more nerdy or researchy but want to do something more applied, where they can actually see the effects of their actions compared to academia. That can also be a really good profile for that type of person. And then the ability to get things done operationally as well—I think, as far as I can tell, that's really needed. All the other organizations need that skill set as well.

How do you think about joining versus starting? Obviously, this is somewhat of a timeline question, but I've heard both points of view recently. One is, “We still have a lot of gaps where we need somebody to step up and start a new organization, and so if that could be you, you should do it.” On the other hand, I've also heard there's a lot of organizations that are trying to hire, and there's not enough talent to go around.

I would say my subjective sense right now of the AI nonprofit space is—and obviously there would be exceptions to this—

I don't mean to paint with too broad of a brush, and you can also feel free to disagree. But it seems to me like there's a lot of funding available for organizations that can demonstrate that they're executing reasonably well and have some capacity to scale. What are your thoughts? How would you talk people through the starting-versus-joining decision?

Yeah, there definitely is funding available. It's a difficult thing to think about, because more funding is still helpful. There's always more you can do. But I think it is true that if you show up with a really good organization, you can raise a lot quite quickly.

In the past, Coefficient Giving was responsible for a lot of funding, but that's actually broadened out in the last couple of years. There's a lot of new funders entering the space. And there might be a lot more funders if some of the people who made money from Anthropic start donating, which they have pledged to do. The founders of Anthropic have all pledged 80% of their equity, I think it is, to donate eventually.

Equity is doing quite well at the moment. We're probably just talking about tens of billions of dollars from that alone. Now, obviously, it won't all come immediately, because they wouldn't want to sell that whole stake immediately.

So there's a lot of funding. Again, I kind of think both of these things are true. There is scope to start new organizations, and there are a lot of important gaps. But there are also organizations doing great work, and you could just join them and try to multiply them.

I think entrepreneurial people are often a little bit biased against the second option, because if you start an organization yourself, it feels much more satisfying. You get to be like, “Well, there was nothing, and now I made this thing.” Whereas just making another organization 5% more effective is harder to feel tangibly. But if that other organization is having a huge impact, then it can often be better to just throw yourself behind the thing that is working and double down on it.

I would consider both paths. Founding is obviously difficult, and it's maybe not the thing for most people. So if you are that type of person who can do it, then I think that should be taken seriously.

Yeah, there's an interesting thing about founding. How possible is top-down founding, where you're just like, “Well, there's this idea I think is really important. I'm just going to go and build that thing”? It seems not to work that well in the for-profit sector. And I remember Y Combinator really encouraging us that the best companies develop out of problems that you were solving for yourself, or almost like side projects that you didn't think were a company, and then you realized they actually were.

I think that maybe applies more to for-profits because the space is so much more efficient and competitive. It's harder to spot an idea. Whereas I think in the nonprofit world, it's more possible to just be like, “Okay, there's this funder who wants to fund this thing. There's this important gap. I'm just going to fill this gap.” And there have definitely been cases of people doing that. If you're really motivated by the impact, you can actually put that into action.

There are a couple of different organizations helping people who want to do this kind of thing now. There's one called 6 Impact, which is an accelerator in this space. Maybe also worth mentioning for mid-career people, Successive, which helps transition mid-career people into careers working on AI risk in some way. And there are also a lot of fellowships, like the Horizon Fellowship, which transitions people into policy careers in 1 or 2 years. So if you're a technical person now but you're interested in policy, you could apply there, and they can help you switch really quickly.

Yeah, cool. Shout-out also to Halcyon as another network that I've seen with a very focused and not super-scaled but high-success approach. They've also been pretty good, especially, I think, for mid-career people making moves into the AI space.

Do you have a requests-for-startups list for the founders among us? Are there top-priority gaps that you see?

Yeah. So Catalyze Impact, which I just mentioned, has a meta-list of all the other lists. You could just go and check that out and browse all of the 20 or so lists of project ideas they have there. And, yeah, there are a lot of ideas at the minute.

In terms of what stands out most, you asked about what someone with an AI application skill set could work on that's valuable. One broad area is that if you think we're going to be facing these big risks soon, caused by much more powerful AI, then one strategy you might want to take is to think about how you can essentially use AI to address these problems. Then your tools are getting better as the problems are getting more severe; they're growing in tandem.

One category there is using AI to improve epistemics, improve our understanding of what's happening, and improve decision-making. There could be a lot of scope to build tools here. Could we have automated fact-checking? Could we have automated analysis of people's prediction track records, like different politicians and pundits, and say, “Well, you said all these things in the past, but this fraction was actually correct”? This could improve our discourse.

Another thing would be government. People in governments are going to be relying more and more on AI to make decisions rapidly. They're going to have AI advice, especially if the world starts speeding up the pace of change. It might be hard to stay on top of things with just our normal human brains. It would be nice if those people had AI advisers that weren't necessarily just developed by the companies that they're supposed to be monitoring.

Ideally, there would be more impartial AI advice that we could check is designed from the ground up to do exactly what this government wants it to do. So there's this kind of AI chief of staff or AI decision-adviser application, which is the kind of thing that the market might not build. You might be able to build a profit model around it, but the key application would be more of a nonprofit application for these key government decision-makers.

Forethought's researchers have written about this idea. If you want to see more of the project ideas in this space, they have a couple of articles about it.

As one kind of calibration point for just how helpful AI can be on some of these things, I'll say we'll put that in the show notes. The show notes these days are now fully AI-generated, generally not even really reviewed by me, but they do tend to come out with a couple dozen links.

All of the organizations that we've mentioned, and the list of lists that you mentioned, I think Claude will be able to find with some help from the Brave Search API, I should say. All that stuff will be in there. If you go check that out, know that I probably didn't look at it between now and the time that it hits the website, which means that there could be some errors in there.

I wouldn't say you should trust it to be fully comprehensive or fully error-free. But for that kind of scale of project, I'm confident enough now just letting the AI run with it. It's clear that it's doing a pretty good job and adding value, even to the point where I'm usually precious about publishing AI stuff without a review. But at the helpful-links level, it's reliable to the point where I let it fly.

We've focused very squarely on these big-picture risks from AI and what to do about them. I know that you've also, over time, put a lot of thought into other ways to help people and the world. There have been 80,000 Hours career guides for how to help the global poor, improve public health, improve animal welfare, and all those sorts of things.

If somebody is really interested in those topics but is also developing an AI skill set, how much room do you think there is to apply AI in organizations that are already executing in those different domains? My guess would be that they probably haven't caught up to the AI frontier, or anywhere close, in a lot of cases, although I could be wrong.

What would you say is the state of just mundane opportunity to apply AI to mundane but ideally well-chosen causes and make a difference that way?

Yeah, I do think the skill set of managing teams of AI agents to do real work is going to be one of the most valuable skill sets of the next few years. You can apply that skill set to any problem, and I think you're right that most industries and fields still aren't really applying AI anywhere near where they could be.

You could work on global poverty but just be an expert in doing it as effectively as possible, using the best tools. Those tools will also continue to get better, so your leverage will increase as AI improves.

In general, I think there is a bit of a disconnect here, because the people who want to work on these other issues tend to—there's a reason why they don't want to bet on AI being the key thing. Often, they want to do something that's more grounded, evidence-backed, or common sense.

That same thing would also prevent them from wanting to make a big bet on AI being real, because if you did believe that, then it's harder to see the case for these issues in some ways. If we avoid concentration of power, loss of control of autonomous AI, a massive bio attack, and all these things, then we're probably going to end up in a world that is far wealthier than today, because we'll have this massive population of digital workers who can hopefully give us some pretty good advice about all these problems.

Many economic models show that, if you really work that through, you could easily have GDP that's 100, if not even 1,000, times more than what it is today eventually, potentially even higher. Part of the way that happens is just that loads of goods and services become basically free, because the models become rapidly commoditized and it's a very small cost of compute to get basically world-class services. This is eventually where it goes on any topic.

Then you have robots that produce goods far more cheaply than now, because a robot should eventually cost like $1 an hour, whereas human workers are like $10 or $20 an hour. That just makes it a lot easier to reduce poverty, because the world is much richer and everything is much cheaper. That will make a huge difference to reducing it by itself.

The main kind of lever that I would focus on is just making sure that the global poor don't get cut out of the AI windfall. That kind of gets us back to the concentration-of-power stuff, because I do think there's a good chance that America really draws ahead of other countries, and it's not particularly guaranteed that they share the benefits with the rest of the world. But making sure that everyone gets some of the spillover, because we would get the cheaper services and things like that, would help.

Sometimes I talk about this as another policy idea: eventually, we want to work toward there being a grand bargain on AI, where all the major countries agree not to race and struggle over it in exchange for a sharing of the benefits among everyone. If we could come to that type of agreement, it could reduce the risks a lot, avoid concentration of power, and be a fairer outcome for people. I think working on something like that would be more what comes to mind for me when I think about what could be a really high-leverage, neglected thing to help with global poverty these days.

But it comes back to a more AI-focused approach, rather than just going and using AI to distribute malaria nets another 10% more efficiently. That's still a good thing to do, but it's not what comes to mind to me as the highest-leverage thing you could do if you're really taking AGI seriously.

Yeah, got it. Speaking of taking things seriously, we teased earlier a couple of other, more speculative or esoteric topics that could potentially be really important. One that's on my mind, and I want to hear what's top of mind for you, is the possibility of AI consciousness, subjective experience, and moral standing. I'm still very uncertain as to whether that's something that I need to worry about, but the evidence does seem to be mounting quickly that I should at least be taking it somewhat seriously. So that's kind of on my radar now. What else is on your radar, even if it hasn't quite cracked the inner circle of the most important things yet?

The way I like to think about this is that, if you're trying to have the biggest impact, you want to work on neglected problems. This means that there's always this dynamic where you start working on something, a bunch of people join you, it becomes less neglected, and then you want to move on to the next, even weirder thing.

It kind of means that people always think you're crazy at every point, but you want to be—well, not always, but you want to be seriously thinking about being one step beyond what's already accepted, because otherwise it's not going to be as neglected. Now there's this question of alignment, catastrophic bio risks, and, as you're saying, at this conference, even people are taking concentration of power seriously. That's quite a big success, because only a year or two ago, no one was really talking about this, and just a couple of groups, like Forethought Research that I mentioned earlier, have got this as part of the discourse over the last couple of years, which is impressive.

But then, as those efforts are successful, what's the next thing? I do think what to do about digital minds is potentially a really big one. I think eventually it will become a huge topic, because we'll just be talking to AIs all the time who seem basically exactly the same as humans, and at that point many people will think they have rights as well. So it's going to become a huge debate.

The question now is whether any groundwork can be laid so that, when that debate happens, it is in a better position to come to a reasonable answer. I don't think we're ever going to solve the problems of philosophy of mind—we probably just need to plan on never solving them—because they haven't been solved for thousands of years, and it's unlikely that we do it in the next couple of years. But we then need to think, given that we don't know what policies would make sense anyway, about a balance of the different views—basically, the different views about what to do about this.

A few others: another one that I was tweeting about recently is space governance. When I was writing these into the book, my editor was like, “These ones I think you should cut, because this is getting too out there.” And I was like, “Okay, that probably means these are the ones I should keep, right? Because if you already knew all this, then it would be too obvious.”

If AI accelerates technological progress, it could become technically possible to settle, to spread out into space, much sooner than it seems from a common-sense point of view. In particular, we're not talking about humans going out; we're talking about sending small AI probes out that can then replicate themselves and build solar panels when they arrive somewhere.

The way this would likely work by default now is a Kalan Grab type situation, where whoever launches their stuff first claims all the stars. That's where most of all the energy and matter is. I think 99.9%, with 29 9s after it, of all accessible matter and energy is not on Earth. So, from a really big-picture perspective, ultimately what matters is what happens with space.

There's basically no one talking about this or thinking about whether there's anything in current laws that could set stronger precedents about how this would happen and make sure it's not just a land grab. Maybe we should think about even just modeling the dynamics of this. It seems like there might be a kind of whoever-gets-there-first-can't-be-dislodged effect, because if you arrive first, you've already built an industrial base out in space, and then it's hard for people following you to displace you later.

We don't actually know that's how it would work, but it seems like it's at least good to understand whether there's a massive first-mover advantage or not. So this would be an area where people could do more research. I think it could be quite cool to have an institute for space governance. This is another startup idea: a specialist think tank that's just like, “We've got all the experts on this topic; we're thinking about it.” The idea would be that when people were like, “Oh, this might actually happen soon,” they would turn to that group as the experts.

I don't know if you've had an episode about gradual disempowerment, but that's another one on this list. Even if we had aligned AI and avoided concentration of power, things could just work in a way that humans get competed out and the economy becomes pretty unfriendly toward us. That seems quite likely to me, and no one really seems to have a proposal for how to prevent that.

I guess basically the plan right now is to use the aligned AI to advise us on how to avoid that outcome, which might work. But there are very few people thinking about this. It gets into your points earlier about visions for what a good outcome would look like. There's very little material on that, really. People are just like, “Well, The Culture series.” That's pretty much what we've got, which is leaving a lot to be desired.

This is a bit of a—I don't want to shortcut your list if there's more you want to add—but one idea that I've been coming back to over and over again is writing utopian fiction, just to try to make the future, the goal state, a little more concrete. I think we've seen a little bit of evidence these last couple of weeks that it could even have a positive effect as it flows through the training data and gives AIs a conception of themselves based on our imagination of what they maybe should be like in a mature state.

Do you have any feedback on that idea at large? If somebody was actually going to try to steer the future through utopian fiction, what advice would you give them? I've heard people speculate about ideas like, “Can you flood the internet with positive training data and therefore influence the values of AI?” I presume there are a lot of reasons why that wouldn't work.

With utopian fiction in particular, one thing I would say is that the track record is very bad. Most people who've tried to write a utopia, when you look back on it, it just seems like a dystopia to us today. It's very hard not to be trapped in the values of your times.

So I'm quite into the idea of what we should be shooting for being this idea from Will MacAskill, which he calls a viatopia. It doesn't specify a particular end state; it's trying to just get civilization in a better position to navigate to the best end state later.

For example, one thing that would help is not having an irreversible existential risk, because then you've cut down a lot of options. But also, it seems that not having an authoritarian government would help to preserve debate and mean you have more options to navigate the future.

Having more information about what’s happening—these are the kinds of things that seem robustly good for navigating things, and a lot of people could agree on them even if we don’t know the particular end state. The analogy Will uses here is: imagine you’re lost in the wilderness and you don’t know which way to go to escape. You might still be able to say, “Well, I need to get water. I need to get to higher ground so I can look around.” These would still be robustly positive goals to work toward. Hopefully, from there, later you’d have a chance to figure out which direction you should actually walk in.

Yeah, I like that. Viatopia. You’ve been generous with your time. I know it’s getting late for you across the pond. Maybe just in closing, are there any other organizations you’d want to shout out that can magically show up in the show notes? Any other topics or tidbits that we didn’t touch on that you’d want to make sure you leave people with? And maybe just tease a little bit what else is in the book that we didn’t talk about, so people aren’t left with the impression that this conversation fully substitutes for the book and are still motivated to go pick it up?

Yeah, I mentioned a lot of organizations except for 80,000 Hours itself. So maybe just briefly mentioning: it’s kind of amazing, in a way, that there are actually jobs that can help with maybe the most important—or one of the most important, and maybe the most important—moments and transitions in history. You can actually switch into these and have a really big impact and do something incredibly interesting with your time as well.

80,000 Hours has a job board that lists around 1,000 open jobs across these issues, as well as lists of funding opportunities and fellowships that can help you transition into them. And then we have free one-on-one advice, so you can talk to someone, and they can introduce you to people in the issues that you’re interested in and help you transition into the field. If you’re interested in switching, those are probably the 2 most useful things to know about.

The book itself really aims to cover all the big questions of career advice, starting from what actually makes for a satisfying job in the first place: what should you look for in an enjoyable job and a meaningful job, and why simple answers like “follow your passion” don’t make sense. Then it goes through which skills are most valuable given AI, and it has a list of concrete skills you can learn. It has advice on the fastest ways to learn those skills, as well as how to choose a problem to work on and which problems are most pressing.

And then there’s a lot of practical advice: if you have a bunch of options, how do you decide between them? What are the most common decision-making mistakes? How do you get a job? What’s the most efficient way to do job hunting? All of these questions. So I’ve really tried to make it the most research-backed career guide ever, covering all of the most important questions, with how to wrestle with AI woven through it.

And, yeah, I think the thing I would say is: when you look back on this time, maybe one of the biggest challenges in history we’re facing, how do you want to look back on it? Do you want to be like, “Well, I was trying to escape the permanent underclass, so I made a bunch of money,” or, “I was accelerating AI”? Or do you want to think, “No, I did the best I could to help this go well”? We don’t know how it’s going to turn out, but I did my part.

Yeah. Yeah, you only have 1 career, so it’s really worth thinking about what you can do with it that’s best. Hopefully, 1 day we’ll all have grandkids, and when they ask, “What did you do in the great AI transition, Grandpa?” it’ll be satisfying to have at least a decent answer.

Yeah, recommend checking out the book 80,000 Hours. It comes out May 26th. You can pre-order it before that as well. I am also an alum, so to speak, of the free career advice offering that 80,000 Hours makes available. I did that really just as I was getting serious about AI, and got into it on a full-time student occupation, but I got a number of helpful introductions out of that meeting, so I can definitely endorse that service.

AI is going to become a bigger and bigger deal, and we’ve got a lot of need for a lot of different profiles, a lot of different kinds of people to help. So, with that—

Mhm.

Ben Todd from 80,000 Hours, thank you for being part of The Cognitive Revolution.

Thank you so much.