直面“智能诅咒”：与 Workshop Labs 的 Luke Drago 对谈，来自 FLI Podcast

Today I’m sharing a special cross-post episode from the Future of Life Institute podcast, hosted by Gus Docker and featuring Luke Drago, co-author of The Intelligence Curse and co-founder of Workshop Labs. I wanted to bring this conversation to your feed because it highlights a critical question that I think society should be grappling with much more than we currently are: Is it wise to design AI systems to compete directly with and potentially replace humans as economic actors?

Personally, I’m relatively optimistic about humanity’s ability to adapt to the social and economic changes that will come with AI, and I tend to worry much more about catastrophic scenarios where we lose control of AI systems entirely. But this conversation did force me to confront the possibility that things might still go seriously wrong even if we do manage to solve the alignment problem.

Luke focuses on a particular failure mode that he calls the intelligence curse. This concept echoes the resource curse phenomenon that we see in some resource-rich but underdeveloped countries today, where an extractive elite manages to maintain power without democratic legitimacy or even much in the way of cultivating the productivity of its own population, simply because they control key resources.

By analogy, in a future where AI systems power the economy and human labor is no longer much of a bargaining chip, whoever controls the AI could have a dangerous level of power. I have to say, as hopeful as I am that the AI revolution can finally free people from doing work they don’t enjoy, this dystopian vision is a pretty natural extrapolation from what happens in today’s world when human workers are rendered economically uncompetitive for whatever reason. And as we’ve seen even in many parts of the United States, the results are not pretty, nor are they without consequences for the rest of the country and the world.

Luke, to his credit, does have some very interesting ideas about what we can and should do to solve this problem. At the societal level, he recommends investments in open-source AI to commoditize the intelligence layer and prevent excessive economic and political rents from flowing to model owners. For companies, he emphasizes the need to design AI systems that empower individual users while allowing them to retain control over their economically valuable data.

For individuals, he suggests guarding your valuable know-how carefully, developing N-of-1 career paths, and chasing moonshot projects sooner rather than later.

My name is Gus Docker, and I’m here with Luke Drago. Luke, welcome to the podcast.

It’s great to be here. Thanks for having me.

You have this essay series on The Intelligence Curse. Maybe we should just start at the very core of that and ask: What is The Intelligence Curse?

I’d summarize The Intelligence Curse pretty simply. The idea is that if you have nonhuman factors of production and they become your dominant source of production, your incentives aren’t to invest in your people.

This sounds very abstract. What does it mean to have a nonhuman factor of production? What does it mean that we can build things that actually replace us? And why doesn’t this just result in an AGI utopia?

I think we have some concrete examples, and one of the ones that we point to in the essay—and what we actually named the effect after—is the resource curse. There are states that rely primarily, or have a significant amount of their income come through oil revenues, as opposed to investment in their people.

What you end up seeing is that, because investments in oil produce a greater return than investments in their people, those states oftentimes funnel money toward the oil investments as opposed to their people. The result of this is a worse quality of life for their people, who have much less power, because at the core, your ability to produce value is a core part of your bargaining chip in society.

The worry here is that, as we get more and more advanced AI systems, governments and companies will be incentivized to invest more in building out even more advanced AI systems, as opposed to empowering workers and citizens.

Exactly. I guess one objection here that I hear from economists is that, if we look at previous technologies, we see that they basically increase wages and increase living standards, unevenly and with setbacks, but over time we see increased wages and living standards. Why isn’t the same just going to happen with advanced AI?

I think this is a category distinction in what we’re trying to do. The last thousand years of technology has been technology that can do everything that any human can do, better, faster, and cheaper. Of course, the question then is: If this allows capital to convert directly into results without removing the need for other people in the middle, why wouldn’t companies just invest more and more money into this?

I don’t think it’s Machiavellian. I don’t think it’s an evil plot by them. What I think instead is that, if you have the opportunity to save 50% on your wage bill while also getting better, faster, more reliable results, most people are going to take that option.

My concern here is that, as we continue to build technology that is designed to replace rather than to augment, we move closer and closer toward a world where people just don’t matter. Then, of course, you’re reliant on other forces—you’re reliant on the government—to make sure that you still have a high quality of life when you can’t produce it for yourself. I think it’s a very precarious situation to be in.

If we think about pensioners today, for example, they don’t produce much for society. In fact, they are, in a sense, a draw on society’s resources, but they’re still protected. Why couldn’t we imagine an expansion of that system?

This is the obvious solution that comes to mind for people. We will have universal basic income, and we will have protection of individual rights, and so we will maintain agency and relevance in an age of advanced AI.

I end up arguing something like this: The core proposition is that your economic value is an important part of your political value.

We’ve seen in the history of democracies that oftentimes they start at the moment when there are diffuse actors who have varying amounts of capital, who need to find ways to settle disputes without violence. The emergence, for example, of British democracy and the Magna Carta came because there were lords who had power that wasn’t equivalent to a king necessarily, but who sure had a lot of influence, and that came from the material possessions that they controlled.

This necessitated free courts and some sort of a way to solve disputes in Parliament. The evolution kept moving backward and backward, and we continue to see that economic liberalization is oftentimes a precondition for the democracies we really care about.

Now, there are nondemocracies that are fine places to live, that don’t wildly trample on human rights. But of course, we know that there’s an extremely strong correlation between governments that respect your rights and enable you to be prosperous, and governments that are democratic. These things aren’t one-to-one, but they’re pretty damn close.

The concern that I have here is that, as we level the underlying economic structure that creates these bargaining chips that put us in power, we end up reducing those bargaining chips.

Pensioners are a fantastic example here because, of course, a pensioner isn’t someone who appears and never works for the rest of their life. Pensioners have 40 years of working extremely hard, paying into a system, and then being active members of society who have a bargaining chip. So, in the last 10, 20, or 30 years of their life, they get this exemption.

It’s because of the system that we have built that this is stable. I would also add that, in the history of the United States, for example, we treat our retired folks way better today than we did before things like the New Deal, which involved massive amounts of unrest and workers trying to use their bargaining chip.

I’m very concerned about a world in which we’re all pensioners forever, with no way to actually bargain and at the mercy of the next election for what happens in our subsequent years.

Which economic metrics should we be looking at if we want to try to confirm whether the intelligence curse is actually happening, or disconfirm the hypothesis?

There are a couple of things that I take a look at. Income inequality seems quite important.

We talk about a sudden takeoff in AI, where there’s suddenly a boom and all of a sudden AIs are way, way smarter than us. I think you might want to also look for this in economics. Is there a sudden moment in which capital immediately begins compounding?

Every dollar you put into a system produces some sort of an outbound return. If you see this kind of rapid accumulation, where you remove talent from the equation and suddenly capital begets more capital, then the actors who already have lots of capital can really rapidly accumulate.

Now, it's already the case that having capital makes it easier to get more capital, but there are a bunch of boundaries and a bunch of restrictions, and outsized players can still win. So, outside of mass income inequality, I'd also take a look at things like economic mobility. Is it the case that people who aren't rich can move upwards in society? The United States, of course, is a very famous society for having this as a marker of its success: that you can come from anywhere, start from nothing, and win.

That doesn't mean you're guaranteed to win, but there's always a pathway. And I think if those pathways start to close, that would be a very alarming signal here. Now, we'll talk about—I presume we're getting to the pyramid replacement here—and I think there are some things that we really want to look at as well, including rising unemployment rates, especially among your earliest age brackets, those that are just entering the workforce. Those are a couple of the metrics that I'm taking a look at here and that I've advised others to look through.

Yeah, actually, explain that concept for us, if you would. Pyramid replacement—what does that look like?

At the beginning of the series of essays, we say that it's pretty likely that if the technological trend continues, you're going to lose your job. And we try to tell a story of how we think that's going to happen, and we start with the example of the multinational white-collar firm. These are very large companies that often do a whole lot of work. Every year, they hire a new class of analysts or a new class of entry-level employees whose goal is to work their way up the pyramid.

They hire a lot of them. They spend a whole lot of time recruiting from the top universities. They show up on campus. And their goal is to create this pipeline of talent because the company has a lot of people at the bottom and a few people at the top. But as people at the top leave because they retire or because they find other opportunities, you need a funnel of leadership.

Our claim is that AI first makes it very easy to replace the people at the bottom. Now, there's actually a paper that came out, I believe, yesterday, starting to show some empirical evidence for this. In some fields, AI is augmenting, but in others, it's just replacing. We've seen that here in these targeted fields. I can't recall each one off the top of my head, but obviously software engineering is one of them. We've seen a shrinking in the number of job postings, the number of job offers, and overall employment among the 22-to-25-year-old bracket in these fields.

That's exactly what you would expect if it is easiest to automate the entry-level work first. Our claim then is that AI is going to move up the pyramid as it gets better, more and more agentic, and capable of doing more tasks with long-horizon planning. As companies are able to capture more and more of that knowledge for themselves, what they're able to do is move up the pyramid, replacing people bottom-up as opposed to through a kind of middle-out or top-down replacement. One day, you wake up to find that all of your colleagues are AI, and the next knock at the door is booting you out, too. We think this can happen at every level of a white-collar firm.

Now, there are a bunch of exceptions here. Obviously, it'll work differently in some industries. Some sectors within a company are going to be easier to automate than others. And I think this is not exactly how it works in blue-collar work. Speculatively, I think blue-collar work might look more zero-to-one: there aren't the robots required to do lots of blue-collar work, and then there are. I'm less familiar with, and I've spent less time in the literature on, the structure of blue-collar companies, but my understanding is that there are a lot more people who do a similar job. It's a bit less pyramid-shaped. It's a bit flatter, with a small pyramid at the top. That's a pretty disastrous situation if robotics is able to rapidly automate those jobs.

Yeah, you might even imagine that the managers of a bunch of physical workers, or blue-collar workers, might be replaced before the workers themselves. You could imagine systems that can automate invoicing and scheduling and so on being easier to do with, or being replaced by, AI before we have fully functional robotics to actually do the blue-collar labor.

And I do wonder if we're talking about the trend already happening. I mean, this is a quite complex question, but how do we know that it's happening because of AI? Say there are fewer job postings related to programming. Could that be because of a general market trend, interest rates, or something different than AI?

I'll flag that the paper I'm talking about is one that I've looked at. I've not spent a ton of time with it yet, so I don't want to speak as an expert on that paper. I'll spend some time on that myself. But that particular paper, if I understand it correctly, works to isolate that and try to understand what the mechanism was here.

My best guess is that you want to look at a couple of different factors. One, you're going to want to see what industries are being affected. We have a pretty good sense as to what tasks are automatable right now and what tasks aren't. We know, for example, that software engineering is extremely automatable at its base level. And so you would expect to see, if it's AI, that the tasks that we know were easier to automate are the ones that are falling, while other ones are being augmented or are much less affected.

My understanding—again, I haven't read the entirety of the paper; I've just skimmed the initial findings—is that that's roughly what you're seeing. And if that's not the case, that is what I'd be looking for here: based on existing and projected AI capabilities, which sectors are seeing changes in employment, and does that match our expectations?

Yeah. Actually, let's dig into that a bit more and think about which sectors or which jobs or tasks would be protected from automation.

Mhm.

And I've suggested some mechanisms of protection that we can talk about. For example, if you're a lawyer, there might be legal restrictions on replacing you. I don't think we're going to see an AI judge employed by the government very soon. Or at least, I don't think we're going to see that until that's probably the last job to be automated. So, how do you think about legal restrictions on automation, and could those become more important as we face this increased market pressure to automate?

Derek Chang, who's at the Windfall Trust now but was at Convergence Analysis or Convergence Research—one of those—I think has a really good piece on what jobs are likely to be more and less resilient to automation.

There are some of the ones that you would expect. Obviously, things like physical laborers are more resistant right now. I think there was a story for 50 years that automation hits physical labor first and mental labor second. Actually, we're seeing the exact opposite, given the way we're making progress in capabilities.

I think your judge point is quite interesting to me, and I think it's correct. The jobs that have strong legal protections are going to be harder to automate. Now, of course, that doesn't mean the people who are in those jobs aren't going to automate their own work. This is both an example of opportunity here and also an example of some sort of gradualist empowerment, where you just automate away to a generic model that makes decisions on your behalf.

I think it could be a bad world if every judge was using the same AI model to make the same decisions. Great, there's a human judge, but it's the same prompt and the same outputs. At the very least, you'd want more diversity that represents the actual beliefs, feelings, and understandings of the judge involved.

Other roles that I think make sense to talk about here are lawyers, kind of. I think the lawyers who are at the partner level are going to be very easy not to automate. Paralegals are a different story. Entry-level law work is an interesting one here because, of course, the job of first-year lawyers who've just been hired is mostly grunt work.

If a firm can hire half as many of them, it might be the case that, on paper, it's hard to automate lawyers. But the law firms that have lawyers working there automate their own work to such a degree that either A, you get an abundance of new law firms arising, or B, larger ones continue to accumulate capital without hiring new people.

I think an important question for what happens next is, at that moment of initial automation, where a whole lot of entry-level jobs get cut and we start to see reductions, what happens next? Is it A, that large firms continue to grow and monopolize the industries, or B, that we get an abundance of smaller firms that allow for more diverse economic output? Rudolph and I are much more excited about that second world than that first one—the one where this creates a bunch of opportunity—but I don't think it's by default. I think we have a lot of work to do to get there.

It's actually an interesting point that you could see a job such as being a judge staying and not being automated, but in practice being automated because the judge is using an AI model to make educated guesses about cases. That would be a way for society to maintain the formal structures we have today without actually thinking about which functions in society we're interested in automating.

I think that would be quite a bad situation to end up in because then we haven't actually grappled with the question of whether we want to outsource the profession of being a legal judge to AI.

Exactly. One of my real concerns there is, again, that same model. If everyone's using GPT-7 and calling that thing in to do all of their judge work, then whatever flaw exists in GPT-7, that's now your judge. My concern isn't just whether we've automated the task, but with what information we're automating it.

We also have, perhaps, another barrier to automation: judgment in a broader sense and taste. For example, you can have hundreds of AI models generate whatever you want—whatever piece of writing or imagery you want—but judging what is actually interesting to people is perhaps more difficult to automate.

Do you think we might remain employed because we have human judgment and because we have taste, or do you think that's ultimately also automatable?

It really depends on the pace and progress of capabilities and exactly what we aim for. I am much more excited about a world where that is a strong, durable human advantage: diversity of taste.

One example here: are you familiar with Nomads and Vagabonds? He's an artist on Twitter. He actually did the art for The Intelligence Curse and did the art for Workshop Labs.

My understanding, after working with him a bunch, is that he takes a Stable Diffusion model and fine-tunes it on his own work and the kind of work that he's aiming for. He's gotten very, very good at prompting it, and he produces these absolutely brilliant results.

I just cannot get that kind of result out of a model. I don't have the taste for it. I don't know what kind of data should be going in in the first place. I don't know how to write my prompts like he does.

I've worked with him before because, obviously, we worked on The Intelligence Curse art. I know he gets hundreds of outputs, and yet he releases a very select few. I think that's a fantastic example of someone who could use AI to be an exceptional tastemaker. I think his judgment is really exceptional there.

It's still his work going in and his work going out. Because of this new medium that he's using, it's been one of the best examples I've seen of an artist fully embracing new technology while still maintaining their own distinct style and taste. I don't think anyone could look at the art that he's outputting and say it's anyone but his own.

That's one of the things that I'm really excited about moving the technology toward. But I don't think that's the goal of the major companies. Again, this definition that OpenAI uses of AGI is predicated on doing most economically valuable human work.

That is a very different game from, “We're going to do some economically valuable work, but it's all going to be tools in your hand that are going to allow you to change and shape the world.” That's a different ball game: to do all of it versus to do some of it. The target right now is total automation. It's a very, very different outcome.

One barrier to automation that you mentioned in the essay series is local and tacit knowledge. This would be knowledge that's spread out, that's difficult to formalize in the way that you can train models on it, and knowledge that's perhaps shifting constantly. It intersects with taste and judgment in a sense.

Is this local and tacit knowledge a way for us to remain relevant?

This is part of our belief at Workshop Labs. If I summarized our thesis in 2 sentences, it's that we believe the bottleneck to long-term AI progress runs through high-quality data, specifically data on tacit knowledge and local information.

That's the skills that you accrue throughout doing the things that you do. That's really hard to digitize, not because it's impossible to digitize, but because it's hard to know where to get it, because you have it. Second, there's local information: the kinds of things that you see around you, the opportunities that you can spot because you are an embodied person with access to real-time information about everything in your sphere.

Right now, the labs really want this data. It's why there's a rush to integrate with your browser. It's why there's a rush to build these bespoke RL environments where an expert gets involved in helping to create a model that's really good at this one task.

But you have a distinct advantage, which is that right now, you have that data. The kind of data that's valuable to AI progress is in your pocket and on your laptop. It's in your day-to-day life.

Our proposition is: why don't we take that data and put it to use for you, entirely privately, so that you don't have to trust us? We just can't train a model and sell the data to your boss. We can't train a larger model to automate you.

We can take an existing model and dramatically tune it toward your work, lock it down so that only you can use it, and let you put it to work. I think you should have control over the tools that augment you, and you should reap the benefits of the data that already exists in your world. That's what we're aiming to do here at Workshop.

I actually think you could see a future in which there is this form of tension between leadership at a company and the workers at a company, where the workers are unwilling to give up their tacit and local knowledge to a model for that model to train on. Company leadership might be quite interested in gathering that data and training on it so that they can reduce labor costs.

Is that perhaps some new tension in the economy?

I think that's one of the tensions. But I also think one thing that people oftentimes forget is that 50% of Americans work at small and medium-sized businesses. These are not the kinds of companies that have hundreds of people from which they can mine surface-level data. These are the kinds of companies where most people on the team are doing something that actually matters. If they didn't show up for work, something wouldn't work.

Because of that, they have lots of specific information about their processes that's really important. I think the outcome I'm excited by is one where AI shifts the direction away from extremely large companies because, candidly, a lot of those tasks are automatable today.

But humans retain this advantage, are able to put their existing advantages to use with that embodied experience, and are able to train models that can help them compete much faster and better—creating an explosion of small companies and small enterprises that really understand what's going on locally.

Ultimately, that could help break the efficiency gap we usually see, where large companies are more efficient because of their scale, because we can put so much intelligence to work for the average person.

But I think this really means that those important things that make you competitive just shouldn't be given away. I'm a strong believer that data is the new Social Security number, and I wrote a piece about this a while back where the thing that you got for caring about privacy in 2015, candidly, was worse ads. There are some exceptions, right? Dissidents obviously need to care about privacy. People in authoritarian countries who are talking badly about the government need to care about this.

But for the vast majority of people in the vast majority of cases, you got worse ads. I think in the next 10 years, if you aren't careful with that proprietary information, if you say, "All right, Lab A, I'm going to give you everything in my life to get moderately better ChatGPT results," and they don't lock this down for you and don't take extreme care to make sure they're not going to train on it, you are 1 button push away from having someone hoover up that data, sell it to the highest bidder, and use it to automate you out of the economy. That is a much different situation for the value of your data, and I think people would do a whole lot better if they'd start caring about that soon.

I don't think we're there quite yet, but it's part of the reason that we care so much about privacy at Workshop: We are aiming at creating a solution that is able to guarantee these things so that we can't use that data to automate you. On a societal level, what you might get from handing over your tacit knowledge is a slightly better AI model. But on a personal level, if you're a maths PhD student on a low salary, you might get offered hundreds of dollars per proof that you provide with a step-by-step solution to train a model on. That is quite an economic incentive.

Do you think we as a society will be able to overcome this incentive to give up our data just when the individual incentive is so strong?

This is part of the arms race, and it's why we are laser-focused on delivering models that aren't just okay and private, but are better at your existing work than an off-the-shelf model because of the data that they have. And because of this, your work improves. I don't think it's the case that you can win this game by walking in and saying, "Look, we have worse tools and we can't pay you, but don't worry, it's private." People don't make decisions like that.

The answer has got to be that the default tool that you want to use cares about what's going on here. And I think Apple is a fantastic situation here, where Apple at its bones is what I would call a privacy-second company. For very few people, the selling point for Apple is, "Oh, this thing is entirely private." But Apple understands that, especially in the United States, it is the infrastructure with which almost all modern communication happens. And so they understand they have a responsibility to protect user privacy.

Unlike many other companies, they have locked everything down to ensure that your messages are private, your phone calls are private, your interactions are private, and that your device doesn't get a virus. They've gone through painstaking efforts so that you know that device is always reliable and always works for you. Anthony Aguirre at FLI has a paper on Loyal AI Assistants, and I know he talks about it as well in Keep the Future Human. But you have got to know that the model that is helping organize and orchestrate your life works for you, not for someone else.

That means it has to be good at working for you, and it has to be verifiably working for you. I think that's how we plan on overcoming some of these incentives. I don't think the labs are going to pay every single human on Earth a couple hundred dollars to gather up all their data. And I think that might be the scale of what they need to do to actually beat this with that kind of incentive. So I think by delivering an actually better experience for users, and then secondly layering on extraordinary protections here, we can both serve customers well and fulfill our impact.

How would we guarantee that the data that I'm providing remains private? Is there a way to do that without just trusting Workshop Labs?

I have more to preview on this soon, once we launch in September and October, with a couple of blog posts that I think will walk through what we're working on here. What I can say for now is that, as an industry, there are now increasingly more ways to do this. You can do things like encrypting all information in transit, decrypting it within what we call a trusted execution environment, where I am using NVIDIA secure enclaves, and then attesting to the code that is running so that you can see that nothing is being extracted from that. And you can store the weights of a model, for example, also encrypted.

If we move back to the intelligence curse for a bit here, we talked about—or you mentioned—social mobility as an indicator of the intelligence curse happening, perhaps decreasing social mobility as an indicator. Could you sketch out what a bad scenario looks like here? What does it look like if we have a more static society with lower social mobility, where capital is the main driver of progress, but that progress is not made by a set of diverse actors? It's made by companies that are larger and larger. What does that kind of society look like?

So I think there are a couple of examples here, but I'll just tell the story through the perspective of 1 guy. Let's say I'm a college graduate in 2030. I've graduated from college, and I'm struggling to get a job. I for some reason studied CS. I'm not sure why I did that in the 2020s, but in 2026 it wasn't obvious what was going to happen.

So I've woken up in 2030, and I cannot find an entry-level job. I also couldn't find internships. Maybe 1 or 2 companies here and there, but on the whole, it's just way cheaper not to get me involved. So I can't get a job. I'm relying on unemployment, which is increasingly strained because I'm not the only undergraduate who can't get a job.

A whole lot of undergraduates can't get a job. Meanwhile, Microsoft has published record earnings because they've been able to halve their expenditure on employees and double their output. This is exciting for a lot of reasons, but remember that in the U.S., corporate taxes are a very small amount of the federal budget. 50% of federal tax revenue comes from income tax.

So we have a smaller and shrinking income tax base because fewer people are making that income, while companies are posting record profits. And, of course, they have the kind of money to work to evade those taxes as well. So our social safety nets are increasingly strained. Unrest is increasingly popular. People are very upset.

They have a lot of time on their hands. The thing they do is they protest or they get very upset.

And the result of this is that our social safety nets just stop working. They’re not able to keep up with the strain. We have to reduce payments and make fiscal cuts. It’s in the name of tightening our belts and pulling ourselves up by our bootstraps.

In 2040, a whole lot of people just aren’t employed. There was a battle, a political debate over what we would do, and we passed some sort of UBI for a while. But that UBI wasn’t sufficient for the kind of standard of life that you would expect, and it’s increasingly unstable.

Of course, now we have a couple of companies who are really, really powerful. Those couple of companies are increasingly realizing that they’d be better off if governments weren’t getting in the way all the time, asking for things. If you look at the Tom Davidson coup paper about how an AI, or an individual armed with AIs, could take power, you’ve got increasing social unrest and instability in institutions.

This is a ripe environment for someone to come in and disrupt an existing order. Maybe that happens democratically; maybe it happens nondemocratically. But the result is that suddenly, not only are you less economically safe, but you’re also in a situation where the routes you took for granted to restore your economic stability are now out of grasp. They’re harder for you to get.

Mhm. Yeah, that doesn’t sound so great. Isn’t it the case that companies—say, Microsoft, Google, NVIDIA, and perhaps OpenAI and so on—will be in fierce competition to provide products for consumers at the very top? So, even if you have the main drivers of the economy being capital deployed by massive companies, you would see innovation from competition, and you would see better products and services.

Yeah, potentially. One of the ways that you can break the intelligence curse—or one of the necessary components—is commodifying the intelligence layer. If it is the case that 1, 2, or 3 players have a monopoly on intelligence, then they can continue to raise the rents.

I saw a tweet recently that said something like, “If you are a rentier around a commodity, you’re a landlord. And if you are a rentier around a monopoly, you are a renter.” You are totally at the mercy of the monopoly to continue to set your rates here.

A world in which there’s prolific, cheap intelligence, and then your job is to specialize into the thing that you do, is a better world to be in. But I think the goal of the labs is to get this recursive self-improvement and just take off. In that kind of scenario, it’s a very different game. That’s one player that’s won, or a couple of players that have won.

I don’t think commodifying fixes the problem entirely, but I do think it’s a necessary precondition to breaking this intelligence curse.

You mentioned Microsoft posting record profits and so on. Perhaps a naive question here is to ask who they’re selling to in this world. If the college graduate doesn’t have a job, who are they actually selling to? Which services and products are they providing?

I feel bad that I’m picking on poor Microsoft here. I don’t know if they’re the right people to pick on. I don’t mean it, Microsoft; it’s not you specifically. I just picked the first tech company that came to mind.

But let’s go a bit broader. Who are the companies selling to? I think we talk about this in the piece, but the core thing here is probably to each other. The B2B environment is quite large, and it is not necessarily true that there has to be what we now call the consumer level in a technology space. A whole lot of companies get by just fine selling to each other.

I think you can expect that to continue to occur across a variety of areas, especially as the core fundamentals become more important. These are primarily land, compute, energy, and intelligence. The more important those get, the more important the businesses that can provide them become.

Of course, governments are other possible clients, but it is not the case that you have to have this vibrant, consumer-style economy that we have today. I think this world has way fewer Starbucks—sorry to pick on them. I think it’s got way fewer cafés and way fewer phone cases, but it’s probably got a whole lot more data centers.

You can see labs trading with each other, AIs trading with each other, and providers trading with each other in this increasingly closed loop.

The intelligence curse is a kind of riff on the resource curse. Are there any lessons we can take from how countries have dealt with the resource curse in trying to deal with the intelligence curse?

Yeah. The resource curse is not guaranteed doom. It’s a curse, but it’s breakable. There are, of course, great examples of countries that did break it. The obvious one here is Norway.

Norway is a state that has a sovereign wealth fund fueled by oil revenues. It does have a real economy on top of that. One of the things to be careful about in this comparison is that, of course, oil is not a one-to-one replacement for all human labor. It’s a very tempting investment target if you already have a lot of it. You still need humans somewhere in the chain, and you can get a more diverse economy. More diverse economies tend to win out over these oil states in direct comparisons, but it’s a very tempting curse.

But what happens in Norway? Norway is, by many, many metrics, one of the best countries in the world to live in: excellent education, excellent social services, really stable government, and really democratic government.

How does this happen? We use some of the quotes from officials of the time, and we looked at some of the case studies in the paper. But a core thing here is that Norway had extremely resilient institutions before the resource curse. Before they discovered oil, they had an excellent civil service that was really good at understanding what to do when this happened, and a very low-corruption society.

The question for me is, do we think we currently live in a world with excellent institutions and exceptionally low corruption? I don’t think so. I think basically every American that has looked at our government has said, “Something here is fundamentally broken,” and it’s been that way for decades. It seems like every time we think we get a reformer in, what we get is increasing brokenness.

I don’t think we’re currently in a situation where we have selfless members of Congress and extremely resilient institutions. I think what it’s going to take to withstand the pressures if you actually get total automation is stronger institutions and more resilience than you would need to withstand the kind of oil pressures here.

Of course, another thing going for Norway is that there is still room for a dynamic human economy on top of that, and so you can reinvest that money. Saudi Arabia is a great example of this. As Saudi Arabian officials have become increasingly concerned that we are near peak oil and that renewable energy is increasingly going to be the way of the future, they are trying to invest their petrodollars into creating a more sustainable—not uppercase-S environmentally sustainable, just a more dynamic—economy that attracts large businesses. Dubai has done this as well.

Now, of course, an important question here is that while the economics are now starting to move toward democratizing, you’ll notice that these states I’m mentioning here, which are sometimes cited for providing a high quality of life for some people—Saudi Arabia and the UAE—have a high quality of life for certain kinds of people, for people who are economically important to the state.

But, of course, they also rely on an underclass. In Saudi Arabia’s case, I wouldn’t say it’s the beacon of gender equality in the world. For half the population, I wouldn’t say those freedoms are well afforded.

As Saudi Arabia has moved toward this more diverse economy, it has also concurrently started liberalizing its gender relations. Under MBS, I’m not going to call it heaven or anything, but there’s been a real effort to somewhat liberalize this relationship in an otherwise pretty conservative society. It is not an accident that these things are happening concurrently.

I think one of the things you should be wary of is arguments that say, “Well, we’re going to centralize all power in the hands of a couple of actors, we’re going to automate the entire economy, but the incentives are going to exist for the state to really care about you.” The example that we have of a state where this is true is Norway.

In other states, if you’re not economically useful, it’s a bit harder of a sell. It’s not always true; there are exceptions. We talked about this case study in Oman, where there was a credible threat of revolution, and this helps force the state to dole out its rents.

The argument is that the rentiers would like to have all of the rents, but they also really want to remain in power and continue to get some rent. If it’s cheaper for them to capitulate than to lose, then that’s an easy out for them.

But, of course, when we’re talking about AI that can automate every job, we’re also talking about the automation of repression and increasing surveillance. As we make things more legible, it’s easier for governments to trend toward this despotic realm, where they can also put down dissent and prevent these kinds of forces that would otherwise force states to capitulate.

So, increasingly, by increasing the state’s ability to surveil to such a dramatic degree, you have this moment where states are very weak, and then, once they’re able to automate repression, they’re suddenly very strong.

In both outcomes, you risk losing the ability for democratic processes to work. Do you think we'll be able to shape the future economy using our culture, using our values, or do you think that what matters most in the end is the underlying features of AI as a technology and the economic incentives that it causes?

Yeah, incentives are a powerful thing, but they are not predetermined. One, they're not predetermined, and two, they're not ironclad. We have so many examples in history of great people defying incentives. I can just rattle them off. Washington deciding to step down, becoming the great Cincinnatus, and not making himself king is one obvious example here: a leader looked at the incentives, looked at the ability for him to gain power, and said, “No.”

Oftentimes, I think one of the ways to reconcile structural views of history and great-man views of history is that these structural forces set up the incentives, but individuals can then defy or alter those incentives and make different choices. Incentives aren't law, but they are really powerful. You want to align your incentives so that you're not hoping that, every time a bad thing could happen, you are totally reliant on the character of the person in power such that they ignore every incentive in front of them.

We talk about this in the paper. We said that economic forces are a predominant force here and a very powerful force, and that societies are extremely exposed to these incentives. But there are other things that shape their values as well. Cultural forces are very powerful, and oftentimes countries make decisions in favor of their culture, or societies do, that are culturally good for them even if they're economically bad.

The existing power dynamics that we have also enable this. One example here is Brexit, which is an obvious example of a country's population choosing something that is probably against their economic interest for a different value set. I'm not commenting on the merits of that debate. I'm simply saying that there is a strong economic argument on one side and an argument on sovereignty on the other. That sovereignty argument won the public, even if it failed to persuade their elites.

I'm not saying that every outcome should be like Brexit, but I'm saying that this is the kind of thing where you actually can make different trade-offs here. But of course, there's that very famous quote about—I think it's Charlie Munger—that says, “Show me the incentives and I'll show you the outcome.” If you have the opportunity to move those incentives in a positive direction for humanity, you really should.

One way to do this is to think about which technologies we want to develop first and which technologies we want our most talented people to work on. We can talk about differential technological development. If you look at the landscape as it is now, which technologies are currently undervalued? Where should we be pushing so that we can change the incentives that the technologies create?

So I'm biased, but my company seems to be doing a pretty good thing here. Obviously, we're not in stealth. We've announced that we exist, and we've got a one-pager of what we're doing, but no one's seen the thing we're working on yet. This fall, we're very excited to roll that out and really show people what we're working on here.

I think there are a couple of categories. We walked through 3 in the piece. One—and this is kind of counterintuitive—we talk a lot about these defensive acceleration technologies. The idea is that you actually have to mitigate AI's catastrophic risks in order to get over this barrier.

The reason for that is that AI's catastrophic risks provide a very good reason to centralize it in the hands of a couple of people. It is true that by default AI could be extremely dangerous. It could be extremely powerful and extremely dangerous. It could make it easier for actors to develop bioweapons. It could make it easier for random people to do bad things.

Governments and companies are going to use those as credible arguments—real arguments—to centralize this intelligence and decommoditize it, to have a couple of actors who have dominant control over it. Of course, the downside of that is we know that the more we centralize this into the hands of a couple of people, the more it looks like a monopoly instead of a commodity, the worse off regular people are likely to be in the long run.

What we want to do instead here is derisk the technology fundamentally. If we're going to build it—and I'm not saying that we do—but if we're going to build it, we should make sure that it's safe. I think there's been this long-running argument in the AI safety space that doing this is not possible or a waste of time. We're increasingly seeing interesting results here that indicate maybe there's actually something to be done.

Kyle O'Brien had a paper with A.C. a couple of days ago talking about how, if you just remove biological materials information from the training data when you do pretraining, you end up with models that are somewhat tamper-resistant even when you try to reintroduce that later in fine-tuning. That is the kind of research you want to be seeing a whole lot more of right now. You want to find the kind of research that means that, if we develop it, it doesn't have to be in the hands of one actor forever—that one guy is not declared the total controller over intelligence.

Then, of course, you really want to work on technology that helps democratize this tech with humans still in control. Again, part of what we're working on here is trying to find use for these last-mile automation tasks, taking advantage of an individual's data, and finding ways to make that even more competitive for them even as there are larger models.

That sometimes looks like modifying existing models. It might look like doing something entirely different. But finding ways to put existing human data to use means that the tools that you control are the ones that are helping you do better and that they don't disempower you.

You also want to work on the kinds of tech that can help strengthen democracies. I think Audrey Tang's vision here is quite inspiring. And so I think those are the 3 buckets I talk about: tech that actually makes it possible so that, if we build it, it's going to be diffuse as opposed to a monopoly; tech that keeps humans firmly in charge; and technology that is able to help strengthen our democracies such that, if we can't prevent them from being a monopoly, we have fallback options.

One of the ways I think about this, to close this loop here, is on social media. I think there are 2 problems in social media, or 2 approaches, and I think you should take them both concurrently.

One approach is to say—the common one—that social media is super addictive, and so the government should regulate it in some way. The government should restrict certain kinds of features that are in it, or age-gate it or something like this. I think an approach that is oftentimes less appreciated and is absolutely necessary, because you can only regulate things so much, is to also introduce technological alternatives.

There's been a massive rise of screen-time apps, for example. Opal's one of them. You download a thing and it helps you reclaim your focus, because a whole lot of algorithms are pointed at you and now you need something pointed outwards.

We're trying to build the thing that's pointed outwards, because so many people are trying to take your job or take you out of the economy, and we think we can build tools to keep you in it. I think if we're right, that could be one of the largest markets in history, because if you are building the tools that help keep people involved, people are going to want to be involved. They're going to want to stay involved in the future. And I think that's a pretty powerful tool to be building, both from an impact perspective and from a market perspective.

We're facing this tension between trying to control the downsides of AI by centralizing it and then spreading the upside by giving as many people as possible access to the models. One answer to this tension is just to say that we need to open-source AI fully. What do you think about that vision, and how does it interface with what you're talking about?

So I am probably more pro-open-source than I think the average person on the podcast. I think part of this is because of this real fear of monopolization. I think it is the case that, if open-weight models are not a core part of the future, you can increasingly charge these wild rents for them.

I think there are a couple of people who have strong incentives to build them. So I don't think it's the case that they're going to fall behind in some near future. I also think there's this very pervasive argument, especially within the AI safety community, that open-weight models are always going to be behind.

It is absolutely true that, in a hard-takeoff scenario where you just FOOM and go straight to superintelligence, that's going to be the case. Someone's going to win that race. That's game over. In basically every other scenario, what we have seen is the exact opposite.

I remember hearing a couple of years ago that there's no way that open-weight models could catch up. They're too far behind, and especially there's no way that China could catch up. It's just impossible. Chinese models right now—Chinese open-weight models—are like 6 months behind the frontier, and some of them, I think, maybe are even more ahead.

Kimikatu, for example, is a really excellent English-writing model. I would wager it’s probably the state of the art at that. This does not look like we’re seeing open-weight models slowing down. The gap continues to close, even on providers that have less access to high-quality compute.

There’s something going on in both the way in which we train them and the data that we’re using that still provides advantages, such that compute isn’t everything. And so I think, if the argument that I oftentimes hear is, “Open-weight models can’t catch up; it’s not a core part of the story,” I just don’t think this is true. I think if you’re taking AI safety seriously, you’re going to have to focus on making open-weight models safe, because open-weight models are going to be a reality, and they’re going to be quite powerful.

How do we do that, though? I guess that’s the main worry with open-weight models. It’s just that we can’t—if we put something out there that’s open weights, we can’t then take it back. And so we don’t have this feedback loop of trying to test something, then pulling back, and then perhaps putting a more limited version of that model out there.

So how do we deal with a technology where, if we release it, that capability suite is now out there indefinitely?

Yeah, this is where, again, I’ll cite Kyle O’Brien’s work here. It’s quite important. The kinds of work that you want to do here are to create tamper-resistant open-weight models, such that reintroducing the information by trying to tune them in a certain way breaks them or doesn’t work.

I don’t have a lot to say. I know I’ve talked with Kyle a bunch, and so I know some of his work is forthcoming. I don’t want to jump the gun on anything here. But as a separate note, the kind of holy grail here is a model that, when you try to reintroduce this, just stops working or breaks because of something they’ve done.

I don’t want to preempt any announcements. I know there are people who are working on this in a broad variety of sectors, but those are the kinds of safety innovations that I think are extremely important and that move our option space.

If you’re someone who thinks doom is really likely, the best thing to do is not continue to evaluate the models to see if we’re getting closer, because if we’re getting closer, we’re going to actually have to do something about it. And I think, from a technical safety perspective, right now you’re either betting on this catastrophic warning shot that I’m not convinced actually slows anything down.

I think we talked—we have a 7-paragraph footnote in The Intelligence Curse. We couldn’t fit it in the main thing, so I footnoted it, talking about how, in a whole lot of scenarios, a warning shot actually just increases the speed at which AI progress happens because somebody gets spooked by it, and the response is, “We need better defenses faster.”

So I think if you’re counting on, “We’re going to keep evaluating the thing, and then we’re going to see that it’s dangerous, and we’re going to stop building it,” best of luck. I don’t think that is an extremely tractable approach. I think more investment is better spent by a whole lot of extremely talented technical experts on actually building out the capabilities that are required to make even open-weight models tamper-resistant and safe.

I think this is genuinely achievable. I don’t think this is an intractable agenda. We have seen more progress on it than I expected to see, and I think as people have chipped at it, as papers have made it clear that this could be possible, more and more people are starting to get excited about this. I think that’s more of the direction I want to go here.

If we don’t have the option of controlling AI using a central authority, it seems to me that we are somewhat at the mercy of how the technology just turns out to be. So if it is the case that we can limit what models can output and perhaps have the models stop if you try to use them to create a biological threat, say, well, that’s great. But what about the next possible danger, and the next possible danger?

If we don’t have a way to control AI as at least a backup option, are we just kind of at the mercy of how the technology turns out to work?

Yeah, this is one of the concerns. We are at the mercy of how fast we can rush our defenses. But that means that rushing our defenses is perhaps one of the most important things that we could be doing.

On other fronts, we recognize this. In pandemic preparedness, for example, we can’t ban pandemics. It’s not possible. Pandemics are always a background risk throughout the world. And yet this means that our response can’t be to do nothing.

Our response has to be, “We know this is a possibility. This is on our threat map. What’s everything we can do to build the kind of Swiss cheese model of defense for pandemics?” I think that approach is extremely relevant with AI dangers.

One other thing that I’d say here is the kinds of proposals that I’m talking about—the ones that I’m explicitly proposing here—are those that try to do this controlled superintelligence explosion. The kinds where we say, “All right, 12 people running after AI? Too much. One guy’s going to do it. We’re going to monitor him every step of the way.”

And what that policy results in is one person—or one body, one entity—having a unilateral advantage over everyone else forever if they actually achieve this kind of hard takeoff. And then you are just at the mercy of the people who control the weights. Aligned superintelligence in the hands of one person makes that person a de facto dictator unless they choose not to be. And that is not a good outcome.

Now, there’s a separate category of policies which I’m not necessarily supporting. This is not me endorsing these, but I don’t think they unlock the kind of Intelligence Curse-style risks. And that’s if we just don’t build it.

So if it’s the case that you can very consistently say, “The Intelligence Curse is real, and therefore I’m going to advocate for never building systems that can replace humans,” I don’t know how tractable that policy is. I’m not sure that’s the right approach, but I don’t think “no one gets it” unlocks the risk.

The concern that I have is that a whole lot of well-meaning people are going after “one guy gets it.” And I think the much more likely outcome is not between 0 and 1 on extremely powerful AI; it’s between 1 and many. And if those are my 2 options, man, I’m definitely on the side of the latter rather than the former. I think the latter is a world that you can move toward.

Spreading AI capabilities—that seems to me, when I read the founding essays of OpenAI, to be the vision that they had. They wanted to make sure that Google didn’t have a monopoly on AI technology, and they wanted to empower everyone with AI models. And that vision seems to have degraded over time.

How do you make sure that doesn’t happen to the vision you have for Workshop Labs?

It is one of the things I think about the most, because the road to hell is paved with good intentions. It is paved with people who are working on things that ultimately end up working against their cause.

There are a couple of things here. There’s the basic legal stuff, like we’re a public-benefit corporation with a fiduciary mission not to automate people. It’s in lawyer-speak for enhancing economic opportunity, but that is explicitly our goal.

This is instead of doing the generic thing of saying, “To make sure AI benefits people.” And it’s like, okay, but what does that mean? Does that mean we’re going to put it in charge, and then we think it’s going to benefit people? Or does that mean we are going to try to do a certain thing?

In our case, this is the economic empowerment argument. It is our mission to make sure that AI actually meaningfully increases your power in the economy rather than decreasing it.

I think also I’m a believer that personnel is policy. And so the kinds of people that you bring onto the team will push you in certain directions. Our hiring process is laser-focused on mission alignment.

It helps that we have been incredibly public. We kind of stumbled on this company by accident. We had worked on a bunch of research in the area quite publicly, and then realized that we had proposed a technical agenda and wanted to go after parts of it ourselves.

But of course, there’s also the broader question of what you do technically. This is why we are so committed to launching on day 1 with extremely strong privacy guarantees, because you shouldn’t trust me that if you hand all of your data to me, then I’m going to be a good steward of it. What you should instead know is that there’s literally nothing I can do to use it in a nefarious way. That’s a much more powerful guarantee.

It’s not this “trust but verify” thing. It’s, “I can demonstrate to you we have taken every measure humanly possible to prevent ourselves from training a larger model on your data.” And so every piece of data that we get from you is used to your benefit, and we can’t use it against you or use it to sell it to your boss.

I think that’s different from a promise. We’re trying to give an actual guarantee here, such that we can’t use the data in this way. That presents lots of novel challenges for our team, but I think it also presents some novel opportunities, both as to how we position ourselves and the kinds of things that we can do to help make your experience better as opposed to worse.

We want these models to genuinely be aligned to you and loyal to you alone. And we’re going to keep that vision centered as we continue to work on this.

It is really 1 of the big technical, perhaps even political, questions of our time. We have AI models that are aligned to certain interests. That's a whole separate question of whether we can even align them to certain interests, and that, in my opinion, is an unsolved problem, but they happen to have certain goals and preferences. Those preferences are a kind of mix of what the companies are interested in, what governments are interested in, and what end users are interested in.

The balance between which preferences should be strongest in the model is a very interesting question, and something that we—yeah, there's a lot of work to be done there.

For example, I expect us, in not that long, to have personal agents that can do our email and our calendar for us. Is that agent working on my behalf when I ask it to book a hotel for me? Or is there perhaps a kind of corporate preference to book a certain hotel that OpenAI might have an agreement with, something like that? You could quite easily see the incentives, or the preferences, of the model becoming muddled between what the end user wants and what the companies are interested in. Do you see a principled way to solve this, or is this just like any other product where the company selling the product is interested in something, and the consumer is interested in somewhat the same thing, but the preference sets do not perfectly overlap?

I think if you talk to a model and ask it for something, it should do 1 of 2 things. It should either answer in your interest or tell you when it's not. If we're going to go down the rabbit hole of LLM monetization via advertisements, it should be exceptionally clear what is an advertisement and what isn't.

I think it started this way in search, and it's less so now. But even still, if you're searching for something on Google and you type in something, you can see which things are ads. This should be really obvious.

Because, of course, what we're building is based on the belief that these things should be loyal to your interests. OpenAI, Anthropic, or us shouldn't sign some sort of a deal and then disguise or nefariously let you know, “Hey, by the way, here's a hotel you should be looking at.”

That's a really bad situation to be in if your model doesn't work for you. I think this is just true as a consumer. You want to know that when you are asking something for advice, you are getting the kind of advice, the kind of information, the kind of truth that you would give to a friend because you genuinely care about them. That's what makes these tools useful: they work on your behalf.

Imagine if there is a Black Mirror episode that really stuck with me. It was in the new season, where a woman has a brain transplant, and they upload half of her brain to the cloud. This is great because she's still alive, but every couple of hours she turns off and gives an advertising pitch about something. She has no recollection of giving the advertising pitch, and then she wakes back up and doesn't even know what's happened.

She only finds this out because other people tell her, “Hey, why did you just bring up this travel site in the middle of your lecture?” It's great that the technology has enabled her to do this really cool thing—she's still alive and able to live her life—except, of course, if she suddenly needs to give a sponsored ad for something or she goes out of the coverage area.

Because they have this monopoly control over her, because you don't have competing vendors for your brain upload—you've got half your brain here and half of the processing power in the cloud, and only 1 guy has that chip—what ends up happening is that they start her on a very cheap plan. It's only a few hundred bucks a month: “You're so good to be alive.” And then they say, “Oh, we have this deluxe plan now, and you can go outside the coverage area if you buy the deluxe plan.” Then it's, “Oh, you're now on our premium tier, and if you just upgrade a little bit more, you can get rid of the advertisements.”

Suddenly, the thing that made your life so much better is now a massive hindrance to your quality of life because 1 guy has total control and gets to jack up the rents as they see fit. That is the kind of scenario that we're trying to avoid. Part of this comes through democratization of technology, and part comes through ensuring that they're actually loyal to you. My expectation here is that in the future, if we get to the good future, everyone has an agent that's aligned to them, that advocates for their interests, that they know is working for them.

1 thing I'll add here to close it up, to close the loop here, is that 1 of the places I really agree with Sam Altman is on this concept of AI privilege. The idea is that, actually, if you're giving this much information to a system, it probably shouldn't be used against you. This is different from other technologies. So I'm probably someone who'd advocate for more privileged technologies rather than less, even on the status quo ones.

But if you are constantly interacting with this thing and it's helping organize your life, that's a powerful tool in the hands of someone who wants to be nefarious to you, who wants to understand your life, who wants to interrogate it instead of you. And because it's a chatbot, it's not going to know when it should reserve its right. Maybe it could, but maybe it doesn't know when it should use its 5th Amendment right. It's not clear. It doesn't have a 5th Amendment right right now. It probably doesn't have a right against incriminating you. And if it has that much access to your life, it probably should.

That's 1 of the more, I think, really value-aligned things OpenAI has called for recently: some sort of concept like that. And I endorse that wholeheartedly.

Yep, both on clearly stating when there's advertising happening in model outputs and on the privacy, or AI privilege. I do fear that consumer preferences are just not set up for these things.

If we look at social media, if we look at digital services in general, it seems to me that consumers are interested in free products that are ad-supported, and companies are interested in hiding, to the maximal extent, what is an ad and what is not an ad. It's more effective if you can't tell the difference between an ad and generic information. It's more effective when an influencer personally endorses a product, but that is happening because they're getting paid, not because they actually like the product—sponsored content and things like that.

So you have those 2 things that we see now. Doesn't this point in the direction of the default AI future being ad-supported, and being a future in which it's difficult to tell what is an advertisement and what is not?

Yeah, no, I think that is the default future. It's why we exist. If I thought the market was, on its own, through the forces of nature, going to correct itself here and didn't require an insurgent actor who was going to work on this, I wouldn't exist. If we didn't think it was required for someone to build the technology to make the future better, we would do something else.

But I think part of this is aligning your incentives with your customers. I could not talk enough about Apple. I think this is a fantastic case study in aligning your incentives so that you're serving the right people. Where does Apple make the money? In the device they sell to you. You, as a consumer, had a very strong preference for that device working. And 1 of the places where we haven't seen this trend of injected advertisements really work is in actual personal devices—the 1 device you have that's your gateway to everything.

Sure, lots of content on that device has this injected information, but you know your device works for you. Actors tried this with Amazon's Kindle, where I had—I think it might still have ads on the front black-and-white e-ink page. I'm not sure if that has ever worked for anyone. It certainly hasn't worked for me, at the very least.

But even with strong incentives, the vast majority of mobile devices don't serve you ads natively. The apps on top of them do. And I think this speaks to a very important point: sometimes you need the thing to work for you. You need to know that it works for you.

This, I think, is again a really massive market opportunity. And I think it's especially true when you're building things that have a lot of data on the user, that the user proactively hands over, that helps them do their job. That kind of thing, I think users, at least in our initial conversations, are more skeptical of handing over all this data unless they know it works for them.

I think being the provider of the thing that people know works for them, that also delivers value to them, is a really powerful position to be in. I think a lot about companies like Apple.

Yeah, yeah. As a final topic here, we can talk about a great essay you had on how to respond to the special time we're living in. It's a time in which AI progress is moving incredibly fast, and you called for moonshots, starting a startup, say. What is it that especially young people should be looking at in these times?

The default paths are closing. And this is true no matter what. I wouldn't bet the house on any 1 intervention, right? But my company could win everything.

We could do everything we set out to do, and the consulting jobs are still going away. I have no interest in changing parts of this pattern. I think it's not our job. Our job is to ensure that the next iteration of the economy works for you: when this change is said and done, you're in a better position than ever before to achieve, as opposed to a worse one.

But the economy is still going to change. Even technologies that create new jobs—if that's the way we can move the pendulum, instead of being a job replacer, to a new job creator—change the nature of the economy. I think that's going to happen basically no matter what, and you're already starting to see it.

The Fortune 500 company that your parents told you you've got to join when you graduate from this prestigious college—because, come on, man, we didn't pay for all that tutoring for you to do a startup, join a think tank, or go to the small company no one's ever heard of—those are now the least risky options because they are still opportunities for you to win in. They require you to think on your feet, be bright, do well, and really understand the environment around you.

Those safer jobs are the first target for automation because companies with 500,000 people on their payroll are going to want to cut some of that payroll. If you're an n = 1 person at a company, if you do an important job that nobody can replace by virtue of being there, you're much safer than if you do a job that 1,000 other people at your company also do, because you are extremely automatable in that role.

I think that's what we're going to see. The automation of rote tasks has the opportunity to do one of 2 things. It can be the start of a total pyramid replacement, where we as a society decide that our value is to replace all work and hope the next thing works out. Or it can be an opportunity for us to build an economy that is more local, more individual, and allows you, as an outsider, to have more opportunity than ever to move in and become somebody.

But that's not going to happen if you don't change your path now. I think this is especially true for the classic prestige paths: people who got all straight A's, nailed their SATs, went to the right college, and have only ever done the right thing according to the status quo. No matter what happens in the next 10 years, I think now is the time for these moonshots because we know the window is still open. It's become easier than ever, and everything else looks more risky.

So, if you're someone who's hesitated on doing the risky thing, and Jane Street has knocked on your door and McKinsey has come calling—"Look, here's this massive paycheck. Come do this for a year or 2"—know that you are going to be on the last chopper out of Saigon. If you manage to get yourself through that, you are the last breed of consultants. That industry is dying. You are the last breed of entry-level whatever.

We are moving towards, if we can win, a more specialized economy. And I think, no matter what happens, if that's the winning play, I think you should take it. So, I strongly urge people to take more risks during this time. I think it's more important now than ever.

Luke, thanks for chatting with me. It's been really interesting.

Yeah, Gus, this has been great.