[BidClub_]
Hard Fork · · 72 min

Bad Apple + The Rise of the AI Empire + Italian Brain Rot

Kevin RooseCasey NewtonKaren Hao

Podcast
TL;DR
  • Apple’s U.S. App Store tollbooth has been judicially dismantled for now; Apple is going to appeal. Judge Yvonne Gonzalez Rogers found Apple willfully violated her 2021 injunction by charging 27% on external purchases, tracking users who bought up to a week after clicking out, and deploying “scare screens” to suppress conversion. She referred Apple and finance VP Alex Roman for potential criminal prosecution, concluding: “As always, the cover-up made it worse. For this Court, there is no second bite at the apple.”

  • The immediate beneficiaries are subscription, media, and creator businesses that can now communicate directly with iPhone customers and retain more revenue. Kindle has added a Get Book button, while Spotify and Patreon are implementing external-payment links; Roose’s hoped-for endgame is price competition that forces Apple to improve its own system and reduce fees. The remaining uncertainty is whether fraud rises without Apple’s controls, though Casey Newton called its “pristine, vigilant control” mostly fantasy, citing fake ChatGPT apps and a $10 Blue Prince copy.

  • Apple’s strategic vulnerability is growing even as its financial results remain formidable. Newton connected extractive developer economics to the abandoned car project, generative-AI stumbles, and weak Vision Pro ecosystem; Roose countered that Apple still reported $95.4 billion of quarterly revenue, up 5% year over year. The investable tension is between emerging “cracks in its armor” and a cash engine that has not yet paid a visible price for missing new platforms.

  • OpenAI’s announced restructuring preserves nonprofit control while removing the limits that made its economics genuinely unusual. Its LLC is slated to become a public benefit corporation controlled by the nonprofit, but investor and employee profit caps will disappear and the nonprofit will receive an equity stake. Alongside Fidji Simo’s appointment as CEO of applications, the restructuring supports Karen Hao’s contention that the labs are increasingly building products “that they can charge lots of money for,” even as Sam Altman says OpenAI “is not a normal company and never will be.”

  • Hao’s “empire of AI” thesis treats scale as a resource-extraction model, not a neutral technical inevitability. She traces frontier models to scraped work, low-paid annotation, labor automation, and permanent data centers consuming enormous power and water in communities promised mostly temporary construction jobs. In drought-stricken Uruguay and parts of Chile, she argues, the “benefit all of humanity” mission collides with people bearing extraction’s costs while receiving few of its purported gains.

  • The sharpest disagreement concerned whether general-purpose scaling is worth its externalities or even necessary for useful AI. Roose cited GPT-4 tutoring gains in Nigeria and AlphaFold; Hao rejected the premise that communities must surrender lithium or water to obtain those benefits, pointed—cautiously—to DeepSeek as evidence that constrained development can yield generality, and favored task-specific systems like AlphaFold. She could believe OpenAI might build systems appearing capable of automating most labor within two years, but called sweeping promises to cure cancer or climate change an elastic justification for capital and regulatory deference.

  • Italian brain rot could be an AI-native, crowdsourced entertainment ecosystem, with open remix economics and obvious commercialization risk. A 24-year-old Romanian creator’s Ballerina Cappuccina drew more than 45 million TikTok views and 3.8 million likes, while creators recombined absurd characters into affairs, pregnancies, airstrikes, and Ice Bucket Challenges. Newton saw the possible birth of crowdsourced entertainment—“if they can think it, they can visualize it”—while Roose immediately imagined Disney licensing the IP and turning it into seven movies.

Digest · the substance, structured for research

1. Apple’s up-to-30% toll finally lost its shield against direct links

  • Newton’s consumer case begins with Kindle’s missing buy button: Apple’s commission made in-app book sales uneconomic, forcing readers into a browser, an Amazon login, and a purchase flow that should have been trivial.

  • Beyond charging up to 30%, Apple prohibited developers from telling users that Spotify, Netflix, games, or other services might cost less on the web because the seller could avoid Apple’s fee.

  • In 2021, Judge Yvonne Gonzalez Rogers found Apple had violated California unfair-competition law and ordered it to permit links to direct payment. Apple changed its rules, but Newton characterized the result as “malicious compliance”—the absolute minimum, delivered while “dragging and kicking and screaming.”

2. Apple’s compliance design became evidence against it

  • Apple imposed a 27% commission on externally completed transactions, only three points below its standard cut; once payment-processing costs were added, leaving Apple’s system could cost more than staying inside it.

  • The reach extended beyond an immediate checkout: if someone clicked from an iOS app and purchased on the web a week later, Apple could attribute the transaction and charge the developer its commission.

  • Apple also inserted a warning screen whose message Newton translated as: “Hey, loser, looks like you’re trying to do something stupid. You’re probably gonna die.” Disclosed emails showed senior executives, including Tim Cook, debating language designed to make the exit feel riskier and suppress clicks.

  • Rogers’s conclusion was categorical: “Apple willfully chose not to comply with this Court’s injunction,” intending to preserve an anticompetitive revenue stream. Believing the court would tolerate that “was a gross miscalculation,” she wrote; “the cover-up made it worse.”

3. Alleged courtroom deception raised the stakes beyond antitrust

  • Finance vice president Alex Roman testified that Apple did not know before January 16, 2024 what external-purchase fee it would charge and decided on 27% that day. Business records showed the plan’s main components had been determined in July 2023.

  • The judge referred Roman and Apple for potential criminal prosecution for perjury; the episode does not treat that referral as a conviction, but as the most extraordinary consequence of the company’s conduct under oath.

  • Roose was struck by internal project names such as Project Michigan and Project Wisconsin, which executives used while designing the fee so they could discuss the plan without making obvious that they were doing some sort of price fixing. His broader reaction: dominant platforms “did not end up that way by accident”; they worked deliberately to prevent competition and preserve market power.

4. Developers can now test whether direct commerce expands the market

  • Effective immediately under the order, Apple had to stop charging commissions on external links and updated its U.S. guidelines. Kindle added Get Book, while Spotify and Patreon began implementing the links they had long wanted.

  • The Kindle flow still opens a browser rather than completing the purchase natively, but Newton emphasized the larger change: businesses can again tell customers about their sites, prices, and offers without Apple restricting that communication.

  • Roose kept Apple’s strongest defense alive as an open question: removing controls might expose users to scams, malware, or users being taken advantage of. The alternative outcome is simpler—more choice and lower prices because developers no longer surrender 30% of revenue.

  • Casey doubted Apple’s protection story, recalling more than a dozen misleading ChatGPT apps when no official app yet existed and a $10 copy of Blue Prince uploaded on launch day. “They are not paying the attention to the App Store that they are telling you.”

5. Apple’s cash machine masks a weakening innovation bargain

  • Newton argued Apple treated developers like grateful tenants even though apps are a core reason people buy iPhones. Remove Amazon, Spotify, Patreon, and similar services, and customers would begin considering alternative devices.

  • He connected that imbalance to Apple’s broader misses: billions spent on an abandoned car, embarrassing AI walk-backs, and Vision Pro’s weak adoption, partly because developers see little opportunity to get rich building for it. These were “little cracks in its armor,” not a prediction of imminent collapse.

  • Roose supplied the counterweight: Apple’s latest quarter produced $95.4 billion in revenue, up 5% year over year. Newton replied that the ruling was never meant to impoverish Apple, only to shift a small portion of wealth toward developers and prove the disputed tactics were unnecessary.

  • Their preferred outcome is competition, not zero commission: external sellers charge materially less than 27%, Apple improves its payments, and developers regain leverage. Newton contrasted Microsoft, which never demanded 30% of every Windows software sale and thereby left room for its ecosystem to thrive.

6. OpenAI kept nonprofit control but normalized the upside

  • OpenAI abandoned a plan that would have separated its nonprofit from control of the commercial enterprise after objections from former employees, Elon Musk, others in AI, and attorneys general. Under the revision, the nonprofit remains in control.

  • The operating LLC is slated to become a public benefit corporation, formally responsible to the public as well as shareholders including Microsoft and SoftBank. The nonprofit receives a stake, while previous profit caps disappear and investors and employees gain unlimited upside.

  • Altman opened his employee letter with “OpenAI is not a normal company and never will be.” Yet the economics become more conventional, while board member Fidji Simo is leaving Instacart to serve as OpenAI’s CEO of applications over product and business divisions.

  • Hao said constant news can distract from enduring fundamentals. During her 2019 three-day embed—the first journalist profile after OpenAI became a company—she already saw idealism, openness, and freedom from commercialization yielding to secrecy and the competitive drive to reach AGI first.

7. Hao reframes AI ethics as accountability for concentrated power

  • OpenAI expected extensive access to produce a sympathetic narrative, Hao said, but it could not clearly articulate its vision, plan, or definition of AGI. Her critical profile was followed by three years in which the company stopped speaking to her.

  • The pivotal exchange concerned scaling’s environmental cost. Ilya acknowledged the concern but told her, “when we get to AGI, climate change will be solved”—a claim she saw as a “cop-out card” in which an undefined future achievement erases harms created along the way.

  • Hao prefers “AI accountability” to “AI ethics” because it names the power relationship: developers accumulate extraordinary resources by claiming they need them for AGI. Addressing present harms, she argues, is also how society prevents future ones rather than treating the two agendas as rivals.

8. The “empire” operates through data, labor, resources, and mission

  • Hao’s analogy to older empires is structural, while explicitly conceding that today’s companies are not comparably overt in their violence: both claim resources, design rules legitimizing the claim, exploit labor, and concentrate the resulting benefits at the center.

  • The claimed resources include artists’ and writers’ work, scraped online data, and contract labor used to annotate data and clean models. OpenAI’s stated goal of outperforming humans at most economically valuable work also makes AGI, in her formulation, “a labor automation machine.”

  • The legitimizing story is a modern civilizing mission: development supposedly benefits all humanity. Hao’s reporting instead found rural, poor, marginalized, and Global South communities absorbing extraction’s costs without receiving the promised share of “progress.”

9. Data-center promises conceal permanent demands and temporary jobs

  • Companies may enter communities through shell companies or sell projects as hundreds of millions of dollars of investment plus employment. Hao said the headline jobs are commonly construction roles; far fewer people remain to operate a completed data center.

  • The infrastructure then consumes enormous power and cooling water around the clock and is effectively permanent. In Chile, activists fought projects they feared would consume drinking water; Google chose to build in Uruguay during a drought when poorer residents faced contaminated water.

  • Governments accept the bargain hoping infrastructure will attract later investment—perhaps a Microsoft office and software-engineering jobs. Hao situated Chile’s response within centuries of extraction and its recent lithium role: opening natural resources to multinationals is repeatedly presented as a route to broad growth that often fails to arrive.

  • Roose’s pushback was concrete: a World Bank randomized trial reportedly found GPT-4-assisted tutoring significantly improved Nigerian students’ scores, with especially large gains for girls who had fallen behind. Could data-center or lithium access be exchanged for benefits such as free ChatGPT Pro?

10. Scale’s benefits do not settle whether scale is necessary

  • Hao rejected the premise that a country must surrender lithium or water to obtain educational gains. The current path—massive data, compute, and resources—is a choice, she argued, and constraining harms could force “true innovation” toward alternatives.

  • DeepSeek was her qualified specimen: she sees “a lot of” problems with it, yet says it demonstrated that resource-constrained development can still produce models with greater generality. The alternative path is not fully known because the dominant industry has barely explored it.

  • Personally, Hao regards today’s general-purpose models as “fruit from a poisoned tree” and rarely uses ChatGPT. She did use Google Reverse Image Search to identify OpenAI’s apparent $10,000 Brazilian designer chairs, illustrating her distinction between predictive tools and generative systems fed her unpublished research.

  • Roose, by contrast, uses generative AI to research topics such as AGI’s intellectual history and changing laboratory definitions—work he says fell from weeks to minutes. Hao declined to tell him to stop, calling AI itself a “perfect use case” because labs constantly stress-test their systems on that subject.

11. AGI rhetoric is broader than the products now being built

  • Hao believes many safety researchers are sincere; some described humanity’s possible demise with voices quivering. Her critique is that Silicon Valley’s wealth enables a narrow focus on distant catastrophe while a disproportionate share of resources bypasses harms already affecting people.

  • She could believe OpenAI might, within two years, produce systems that appear able to automate most labor, prompting executives to hire AI instead of workers. Other AGI forecasts require case-by-case scrutiny because the definition changes with the claim.

  • Newton’s practical AGI is a subscription replacing his assistant’s customer service, scheduling, and sales work. Hao accepted that definition but distinguished it from the sweeping system invoked to raise capital, rally support, resist regulation, “solve climate change,” and “cure cancer.”

  • Roose offered AlphaFold as evidence that AI investment can unlock science; Hao agreed it was remarkable precisely because it was task-specific, trained on curated data for a scoped problem. She called that path “perpendicular” to general scaling and argued engagement optimization, flattering chatbots, and the need to justify a $40 billion raise are pulling labs toward ordinary products.

12. Democratic AI governance has to cover the whole supply chain

  • Hao divided governance into data, compute, models, and applications, arguing that affected people—not only companies—need agency at every stage. Individuals should be able to opt in or out of datasets, with public consortiums debating what accessible data belongs in them.

  • Content moderation cannot remain an improvised internal choice, as when OpenAI employees debated whether pornographic images should stay in a dataset. Hao wants those value judgments subjected to open public discussion.

  • Communities should know data centers are coming, receive credible long-term water, power, and employment projections, and contest projects through local government. Contract workers should receive conditions consistent with international human-rights norms; simply being an American company does not make OpenAI’s development “democratic.”

13. Italian brain rot previews an open, AI-native entertainment market

  • Emerging in January, the format combines AI-generated creatures, faux-Italian names, and theatrical narration. Ballerina Cappuccina—created by a 24-year-old Romanian, not an Italian—surpassed 45 million TikTok views and 3.8 million likes.

  • Its compressed storytelling is the feature: one roughly 15-second clip moves from honeymoon to affair, pregnancy, discovery, pursuit, and airstrike. Another has Chimpanzini Bananini, a banana-chimp hybrid, nominate fellow characters for an Ice Bucket Challenge.

  • Roose saw a shift from universally visible memes toward siloed feeds where millions can follow a phenomenon he never encounters. Newton’s explanation joined novelty—“Fuck that. Give me Ballerina Cappuccina”—with young people’s perennial desire for a language older audiences cannot understand.

  • Newton’s bullish case is permissionless remixing: creators without animation skills can visualize an idea, reuse apparently untrademarked characters, and avoid the Bored Apes model of a “homeowner’s association for creating entertainment.” Roose’s bear case was immediate institutional capture: Disney licensing the characters and stretching Chimpanzini Bananini across seven movies.

Kevin Roose

Casey, have you heard the exciting news this week?

Casey Newton

Which news, Kevin?

Kevin Roose

The Golden Globes are adding a podcast category.

Casey Newton

I did not hear that.

Kevin Roose

Yeah, that just came out. So yet another award we're not gonna win.

Casey Newton

Well, I don't know about that, 'cause if I know one thing about the Golden Globes, it's that until very recently, it seemed like you could just bribe them directly to win. I don't know if that's still true, but we should look into it.

Kevin Roose

Yeah, what does it cost to win a Golden Globe these days?

Casey Newton

I don't know, a few hundred dollars.

Kevin Roose

Check's in the mail.

Casey Newton

Wait, unless there are tariffs. Topical.

Kevin Roose

Okay, now we're definitely not winning.

Casey Newton

Now we're not winning 'cause I accused them of corruption?

Kevin Roose

Yes.

Casey Newton

Oh, listen, we speak the truth on this podcast, okay?

Kevin Roose

We do.

Casey Newton

I don't care what it costs me.

Kevin Roose

I'm Kevin Roose, a tech columnist at The New York Times.

Casey Newton

I'm Casey Newton from Platformer.

Kevin Roose

And this is Hard Fork.

Casey Newton

This week, the scathing court ruling that forced Apple to give up some control over its App Store and could send an executive to jail. Then, author Karen Hao joins us to discuss her new book on the history of OpenAI and the hidden costs of reaching massive scale. And finalini, it's time for me to teach Kevin about the joys of Italian brain rot.

Kevin Roose

Mamma mia. Casey, have you noticed a smell in the air over San Francisco this week?

Casey Newton

Many smells in the air, Kevin.

Kevin Roose

Well, the smell I'm talking about, Casey, is the smell of freedom.

Casey Newton

Mm.

1. The App Store Ruling

Kevin Roose

Because in the last week, Apple has lost its iron grip on the iOS App Store, thanks to a ruling by a judge.

Casey Newton

Commerce is legal in America again, Kevin.

Kevin Roose

Yes. So we're gonna talk about this today. Apple has been forced to make some big changes to its App Store by a lawsuit that was brought by Epic Games, the maker of Fortnite. A judge ruled last week that Apple had not complied with an earlier injunction, and we will get into all of that. But first, I just want you to make the case that this matters to normal people. Why should the average person with an iPhone care about what Apple's rules for its App Store are?

Casey Newton

Well, to me, it actually starts with the Kindle app, Kevin. Lots of people love to read on their phones and tablets, and I think most people I know in my life have had the experience of opening up the Kindle app or the Amazon app thinking, "I wanna buy that book," and then there's just kind of a big blank spot where you're expecting to see the buy button. And Apple is the reason for that blank spot. They charge such a high commission on e-books that Amazon and other companies cannot profitably sell them. And so, since the dawn of the App Store, buying a book on your phone—something that should be very easy—has required you to open up a browser, log in to an Amazon account, and navigate that whole system. Amazon is not alone. Many, many developers have had to go through similar contortions just to be able to sell their products and still make any kind of profit.

Kevin Roose

Yeah, this is the so-called Apple tax of up to 30% that developers have to pay when they want to charge for apps or purchases within their apps. And for many years, Apple has not only levied this tax, but has also made it impossible for those developers to direct users off Apple's platforms and say, "Hey, if you want a better deal on this Spotify subscription, or this Netflix subscription, or this purchase of an iPhone game, you can actually go on the web and get a better deal there, because there, we don't have to pay Apple's 30% fee."

Casey Newton

Yeah.

Kevin Roose

That has not been allowed. And so Epic Games, which makes Fortnite, brought a lawsuit years ago to try to get those policies changed. And in 2021, a judge in California named Yvonne Gonzalez Rogers ruled that Apple had violated the law in California against unfair competition. She ordered Apple to allow apps to provide users with links to pay developers directly for their services, and that way they could avoid paying Apple's 30% commission. And after that ruling, Apple did go and make some changes, but apparently, they didn't do a good enough job.

2. Apple Resists Compliance

Casey Newton

No. And I would say this has been apparent to most people who've been following this. I think we've talked about this on the show. Apple did what is often called malicious compliance, doing the absolute least while dragging and kicking and screaming the whole time.

Kevin Roose

Yeah, so we're gonna talk about some of that malicious compliance. But let's just say straight up, this was a scathing opinion.

Casey Newton

Yeah.

Kevin Roose

I have rarely read a judge who is so obviously angry at a tech company for doing what they did.

Casey Newton

No, this was the kind of speech that you typically only see on a Bravo reality show.

Kevin Roose

Yes. So Judge Gonzalez Rogers not only accused Apple of doing this kind of malicious compliance, but she also accused them of outright lying to the court under oath. She referred both Apple and its vice president of finance, Alex Roman, for potential criminal prosecution for perjury. And we should just read the last paragraph of the order from Judge Gonzalez Rogers, which is truly the mic-drop moment. She writes, quote, "Apple willfully chose not to comply with this Court's injunction. It did so with the express intent to create new anticompetitive barriers, which would, by design and effect, maintain a valued revenue stream, a revenue stream previously found to be anticompetitive. That it thought this Court would tolerate such insubordination was a gross miscalculation. As always, the cover-up made it worse. For this Court, there is no second bite at the apple."

Casey Newton

Period. But you know what? It kind of was a second bite at the apple, 'cause she bit them the first time, and then they didn't do it, so she had to bite them again.

Kevin Roose

Yes. So let's just talk for a second about some of the details that were revealed in this judge's opinion that have come out about how Apple tried to skirt compliance with this earlier 2021 injunction.

Casey Newton

Yeah. Well, this was well-known to all of the developers: if you wanted to use an external sales system in the App Store, you still had to pay Apple a commission, and that commission was 27%, or 3% less than what Apple was charging. And of course, these companies have to pay the payment provider. So basically, Apple created a system where you were actively disadvantaged in multiple ways from trying to operate outside of the App Store.

Kevin Roose

Yes. So I knew that Apple was charging a commission for apps that would send people—like, if you're Spotify and you want people to be able to subscribe to your app on the internet and pay a lower price directly to you rather than going through Apple, you could do that under Apple's revised rules, but Apple would actually charge you a 27% commission, which by the time you added credit card fees on top of that, would probably be more than the 30% that they would charge you. So this was clearly a case of Apple trying to say, "Well, go ahead and use this other system, but it's not actually gonna save you any money."

Casey Newton

No.

Kevin Roose

And what I did not realize until I read Judge Gonzalez Rogers' opinion here was that Apple would not just collect those commissions if you went directly from an iOS app to the web to buy a subscription or a service. If you went a week later, they would be able to track that you had gone to the web from the iOS app, and they would still charge the developer that commission.

Casey Newton

Yeah. It was absolutely outrageous.

Kevin Roose

It was insane, and it was also not the only thing that Apple did to try to dissuade iOS users from going to external links to buy goods and services outside of its payment system. Casey, what is a scare screen, and how did Apple use this?

Casey Newton

The scare screen was a popup that you would see when a user did actually try to click out of the App Store to make a purchase using an external system. And while these were not the exact words, Kevin, here was the vibe: "Hey, loser, looks like you're trying to do something stupid. You're probably gonna die. Do you wanna try it anyway?" And believe it or not, Kevin, when people saw a message that had that vibe, most of them just chose not to click it.

Kevin Roose

Yeah. And what was so amazing about this was that Apple had tried to protect some of its private company communications from being seen by the judge in this case by claiming some sort of attorney-client privilege, but the judge said, "No, no, no. Out with it. Let's see those emails." And so we have, in this opinion, lots of emails between Apple executives, including Tim Cook, the CEO, talking about the very specific language to put on this scare screen and how to make it even scarier so that users would be less inclined to go outside of Apple's ecosystem and make a purchase.

Casey Newton

Yes, and these internal documents showed that the company would lose minimal revenue or no revenue at all from this, right? That they built a system that was maximally designed to protect their revenue, which was contrary to the judge's order, which she wrote in the spirit of increasing competition and other companies' revenue.

Kevin Roose

Yeah. So, to put it mildly, Judge Gonzalez Rogers did not find any of this charming in the least, and she also directly accused at least one Apple executive of lying outright under oath about what it had done. Casey, explain the perjury charge here.

Casey Newton

Yeah. This perjury charge was leveled against Alex Roman, the vice president of finance at Apple. Among other things, she focuses on this moment where he testifies that until January 16, 2024, which is when Apple's revised system went into effect, Apple had no idea what fee it would impose on purchases that linked out of the App Store. He testified that the decision to impose a 27% fee was made that day, which is so obviously untrue.

Of course, during the legal proceedings, business documents revealed that the main components of the plan were determined in July 2023. So basically, this guy got caught red-handed, and the judge is going to punish him for it.

Kevin Roose

Yeah. And so, effective immediately, according to Judge Gonzalez Rogers's order, Apple has to drop these commissions—these 27% fees—on these external links. As of last week, Apple had officially updated its App Store guidelines to allow those links out of the app in the U.S. But Casey, what are the implications of this, and how are other developers that put stuff on iPhones reacting?

3. Developers Bypass Apple

Casey Newton

Developers are reacting by implementing the links that they've always wanted to have. So in the Kindle app, for example, now you will see a “Get Book” button. You'll tap it, and it'll kick you out immediately into a browser where you can complete a purchase. Spotify and Patreon are also doing something like this.

This is not a perfect solution. You can't actually just buy a book in the Kindle app yet, for reasons that actually aren't entirely clear to me. Maybe we'll get there. But on the whole, we are essentially removing the restrictions that prevent outside businesses from communicating with their customers, telling them about deals, and telling them about their websites. These sorts of onerous restrictions on the speech of other companies have been wiped out.

Kevin Roose

Yes. And I think that gets to why these arcane and somewhat small-seeming changes to the rules governing Apple's App Store really are important. Apple has been, for many years, this sort of godlike gatekeeper for any company that wants to make things for the billion-plus iPhones out there. They have made extremely strict and specific rules about how developers can and can't build their apps and sell products and services to customers.

They have effectively been a landlord over the entire digital services economy. And I think, judging from this opinion, they have really abused that power, and now they are getting slapped on the wrist for it.

Casey Newton

Yeah, and I think it has been to their own detriment, Kevin. Apple's view is that these developers should feel lucky that they get to sell in the App Store at all, when in reality, a big reason that we buy iPhones is because of the apps that are there. If you took the Amazon app, the Spotify app, the Patreon app, and all these other apps off the iPhone, people would start considering alternatives, right?

And so I think that the balance between the developers and Apple had just gotten completely skewed, and Apple has not been recognizing the value of what those developers are bringing to iOS.

Kevin Roose

Yeah. So you think this ruling is a good thing?

Casey Newton

I think it is absolutely a good thing. I think it has been long overdue, and I hope it is upheld after Apple appeals, which it is going to do. But what do you think?

Kevin Roose

Yeah. I think it's an open question. Apple's defense of these App Store rules has always been some version of, “We're protecting our customers,” right? “If we let people sideload apps onto the iPhone in a way other than through the App Store, people will put all kinds of dangerous malware and stuff on the iPhone, and you'll be sorry. If we let people pay for things on external websites, then people will run all kinds of scams and people will be taken advantage of. And so by implementing these rules, we're really protecting our customers. It's for your own benefit, essentially.”

And I think it'll be really interesting to see if, when these restrictions are gone, people actually do say, “We wish that Apple were taking a more active role here. We want some of these restrictions back,” or if the net result is just going to be that people have more choice and they pay a little less for stuff because the developers making that stuff are not having to pay 30% of their revenue to Apple.

Casey Newton

Well, I think that's going to be the case. This whole argument that Apple maintains this pristine, vigilant control over the App Store has always been mostly a fantasy. Think about the early days of ChatGPT, before there was an app. You would go onto the App Store and search for ChatGPT. You would see a dozen-plus apps that were all clearly misrepresenting themselves as OpenAI, and some of them were among the most revenue-generating apps in the entire App Store. Apple could have stepped in to prevent that. They didn't.

I'll give you a more recent example. One of the best video games of the year is called Blue Prince, spelled P-R-I-N-C-E. All of the gaming bloggers love it. I've been playing it, and I've been loving it myself. The day it came out, somebody just ripped it off and uploaded it onto the App Store and was selling it for 10 bucks or something. Why didn't Apple know that? They are not paying the attention to the App Store that they are telling you they are paying.

Kevin Roose

Yeah. To me, the most interesting part of this, as with a lot of these antitrust trials that are going on right now, was just seeing the internal communications at these companies. In this ruling, there are all these fascinating excerpts from emails and messages between Apple executives talking about the various plans they had to circumvent this injunction and charge this 27% fee. They had all these code names, like Project Michigan or Project Wisconsin, so that they could talk about this stuff in a way that would not make it obvious that they were doing some sort of price fixing.

And it just makes you realize that these giant tech monopolies did not end up that way by accident, right? They have had to work very hard for a very long time to prevent competition, to keep their market power and their dominance. And I don't know, man, there's just something really depressing about that.

These are companies that used to succeed by making good things that people loved. In some respects, they still do that. But they also spend a ton of time with their top executives in meetings talking about whether the fee should be 27% or some other number. And it just makes you realize that they have really lost the plot here.

Casey Newton

Absolutely. Well, let me try to cheer you up a little bit then, Kevin, because I think there actually is a negative consequence for these folks of growing their profits so large on the basis of this extremely easy money, where they just make every developer pay this very high rent to them. Apple has been missing the boat on next-generation technologies.

We know that they invested billions of dollars into a car project that they could never figure out and have had to abandon, right? We know that they are struggling to figure out how to do anything with AI and have had to walk back a bunch of claims recently in a really embarrassing way. We know that the Vision Pro, their most recent hardware initiative, is not taking off, in part because developers do not want to make apps for it because they have not been able to get rich making apps for it, right?

So all of this stuff is just adding up in a way where Apple's decisions really are coming back to haunt it. And while it remains a giant, and I'm sure will for a very long time, we are starting to see some little cracks in its armor.

Kevin Roose

Yes, and yet Apple just reported its earnings for the last quarter. It made $95.4 billion in revenue, up 5% year over year. So despite the fact that it is missing all of these new innovations and trends, that it is late on generative AI, and that it hasn't succeeded with the Vision Pro in the way that it had hoped, it is still doing quite well as a company. So I don't know that this is actually coming back to bite them in the way that we might hope it would.

Casey Newton

Well, let's see what happens. The idea behind these rules was never to make Apple a tiny company that was struggling to get by. It was just to get them to share a very small portion of the wealth with a large number of developers. Apple has done a ton of incredible, innovative things. They deserve to be rewarded for that. They deserve to take some sort of commission from the apps in the App Store, right?

But this has been about trying to create a more level playing field for other developers out there. And if the end result of this is that Apple is still pretty rich and profitable, I think that will actually make the point that the judge is making, which is that there is no need for Apple to engage in the sort of shenanigans it's been up to.

Kevin Roose

Yeah, I think the best outcome possible here is that all the big developers that can afford to develop their own payment systems for their apps or send people to external websites to buy things do that, and they start charging way, way less than 27% for that. Apple is ultimately forced to improve its own payment system and maybe reduce its fees to, in other words, compete.

That is what all of this is about: forcing Apple, a company that has not had to compete for the affections of iOS developers in a long time, to finally step up and do something different.

Casey Newton

Keep in mind, even Microsoft, which was sued for anticompetitive behavior back in the early 2000s, never said, “We want to take a 30% cut of every software program sold on Windows.” They actually left a lot of money on the table, and it helped that ecosystem to thrive, right? I would like to believe something similar could happen here.

Kevin Roose

When we come back, we’ll talk to author Karen Hao about her new book on OpenAI and the costs of building such big models.

4. OpenAI Changes Its Structure

Kevin Roose

Well, Casey, it’s a day ending in Y, so there’s some OpenAI drama making the rounds this week.

Casey Newton

Yeah, although I don’t know if this is so much drama as the company is trying to retreat from drama, Kevin.

Kevin Roose

OpenAI announced on Monday of this week that it was no longer trying to get out from under the control of its nonprofit board. That was something that a lot of people, including Elon Musk, had objected to. A lot of former OpenAI employees and others in the AI field had said, “Wait, hey, wait a minute. You can’t do that. You’ve still got to have this nonprofit board controlling you.” And OpenAI, after hearing from some attorneys general that they were not happy about this plan, has retreated. So what is the new plan, Casey, and how is it different from the old plan?

Casey Newton

The old plan was basically that the nonprofit was going to no longer have any control over the for-profit enterprise. It was going to become a separate thing and invest in various AI-related causes and philanthropies. Under the new plan, the nonprofit is going to retain control over the for-profit, so basically the status quo is going to be in effect, Kevin, except for a couple of key changes.

One is that what is now a limited liability corporation is going to become what they call a public benefit corporation. And a PBC, as they are called, has responsibility not just to think about shareholders like Microsoft and SoftBank and everybody else who owns a chunk of OpenAI, but also to think about the general public, right? So that’s one important idea that’s there.

The other big idea is that the nonprofit is currently set to get an unlimited amount of profits if OpenAI does eventually become a trillion-dollar company. That’s not going to be the case anymore. Under this new model, the for-profit is going to give some stake to the nonprofit, but after that, it’s going to be a very normal tech company. Everybody who owns shares, all of the employees, they can get unlimited upside, and the more money that OpenAI makes, the more money that they can make, too.

Kevin Roose

Right. So these profit caps that OpenAI had previously put in place, where investors like Microsoft were limited to earning some multiple of the amount that they put in and no more, those caps are now going away.

Casey Newton

Yeah. They put on their thinking caps and they said, “We’re getting rid of the profit caps.”

Kevin Roose

Well, it just goes to your point that you’ve been making on this show for years now, which is that OpenAI is a very weird company.

Casey Newton

Yes, and I have to say, when Sam Altman wrote a letter to employees this week, the first sentence of the letter was, quote, “OpenAI is not a normal company and never will be,” and I felt so seen. I felt so seen.

Kevin Roose

Somebody’s been listening to Hard Fork.

Casey Newton

And in other OpenAI corporate news, the company announced late Wednesday that its board member Fidji Simo would leave her job as CEO of Instacart to become the company’s new CEO of Applications, overseeing its business and product divisions.

Kevin Roose

So we are not going to do a whole segment about the OpenAI corporate conversion story this week.

Casey Newton

Because we love you too much.

We love our listeners too much.

Kevin Roose

We would not subject you to that. But we are going to talk about it and many other things related to OpenAI with Karen Hao. Karen Hao is a reporter who’s been covering OpenAI and the AI industry for years now, and she has a book that’s coming out later this month called Empire of AI, where she writes about Sam Altman and OpenAI and what she calls the dreams and nightmares of this very strange company.

Casey Newton

Absolutely. And before we do that, Kevin, do we have anything we want to disclose?

Kevin Roose

Well, let me make mine first. My boyfriend works at Anthropic.

Casey Newton

Kevin, you’re coming out? I’m so happy for you.

Kevin Roose

No, I work at The New York Times Company, which is suing OpenAI and Microsoft for alleged copyright violation.

Casey Newton

Interesting, and my boyfriend works at Anthropic.

Kevin Roose

Yours too?

Casey Newton

Yes. Anyways, let’s bring in Karen. Karen Hao, welcome to Hard Fork.

Karen Hao

Thanks so much for having me.

Casey Newton

So, I imagine your book is sitting there behind you on the shelf. It’s all printed up. It’s ready to go, and then this very week, OpenAI puts out a story. It says, “Hey, maybe we’re going to change our structure around again. Why the heck not?” So what’s it like trying to write a self-contained book about a company that just never stops making news?

Karen Hao

Tiring. But honestly, people have been asking me this question a lot: How do you even write a book at book scale? Because usually it’s months on end before it goes to publish. And I think sometimes the news is actually a little bit distracting in that, yes, there are a lot of changes happening. Yes, things are evolving really fast, but there are some fundamentals that are ever-present. And so I try to keep the book focused on the things that don’t change so much.

Casey Newton

Yeah. And among other things, this book is a history of OpenAI, so maybe let’s go back all the way to the beginning. What was this company like when you started writing about it?

5. OpenAI's Idealism Fades

Karen Hao

So I started writing about OpenAI in 2019, and I went to the office to embed with them for 3 days as the first journalist to profile what had just become a newly minted company. Right before I started covering it, it was still a nonprofit, and it had this explicit goal that it should be a counterbalance to for-profit companies. And it became clear to me during my time at the company then that the idea that this was a bastion of idealism and transparency, and was going to be totally open and share all of its technologies with the world, and not at all be beholden to any kind of commercialization, was already going away.

There were a lot of early signs of that that I picked up on while I was there. There was a lot of secrecy for a company that purported to be incredibly transparent. And there was a lot of competitiveness, which to me suggested that if you’re going to be competitive and you want to specifically reach AGI first, you are going to have some really hard trade-offs with this transparency mission and this “open up everything to the public” mission.

Casey Newton

So, I’ve talked to some people at OpenAI who have said that they felt quite burned by some of your early coverage of them, like they were expecting something different than they got. And you write in the book that after you published your story on them, they stopped talking to you for 3 years. I’m just curious: What do you think surprised them about your coverage, or whether they should have been surprised given some of the questions you were asking?

Karen Hao

I think they were surprised because they gave me a lot of access, and they thought that I would adopt a lot of the narrative that they were giving me. To be honest, I came in without a lot of expectations. It was actually my first-ever company profile, and I was going in with an open mind: This company presents itself as this ethical lighthouse, so let’s try to understand how they organize themselves and how they try to achieve the goals they’ve set out to do.

I found that they couldn’t quite articulate what their vision was, what their plan was, or what AGI was. The prioritization of the problems that they said they were focusing on just didn’t quite feel right to me. I pointed out to them that there were environmental issues that were starting to become more and more of a concern as AI models were scaling larger and larger. Ilya said to me, “Yes, of course that’s a concern, but when we get to AGI, climate change will be solved.” And I thought, “That’s kind of a cop-out card—to say, ‘When we get to the thing that we don’t know how to define, all the problems that we might have created along the way will just magically disappear.’” That’s when I started thinking that we needed to scrutinize this company more and be more cautious about taking everything they said at face value.

Kevin Roose

Right. It sounds like a microcosm of the arguments that have taken place for the last few years among the AI safety crowd and the AI ethics crowd. The AI safety people are worried about existential risk and bioweapons and malicious use of these systems, while the AI ethics crowd is much more worried about issues like bias, environmental concerns, and things like that. I want to make sure I’m characterizing it fairly: You yourself are coming more from the perspective of the AI ethics crowd, and you think we should be paying more attention to the immediate harms of these models rather than trying to avert some future harms.

Karen Hao

I would call it the AI accountability crowd. The reason why I use the term “accountability” instead of “ethics” is because I think accountability acknowledges that there’s a huge power dynamic happening here, where the developers of these technologies have an extraordinary amount of power that they’ve accrued and amassed and are continuing to accrue and amass based on this narrative that they need all of these resources to build so-called AGI.

I definitely come from that perspective, and I think that if we take seriously the present-day harms of what is happening now, that will help us not get to future harms because we will be more thoughtful about how we develop AI systems today so that they don’t end up having wild, detrimental effects in the future. I think this idea that we don’t really know how bad AGI might be or what the catastrophic scenarios are isn’t quite right, because we already have so much evidence of how AI is affecting people in society. AI is harming people literally right now. We need to address that, document that, and change that.

Kevin Roose

One of the central arguments of your book is that OpenAI and the AI industry in general have become an empire. It’s the title of your book, “Empire of AI.”

Karen Hao

Yeah.

Kevin Roose

And that it’s done so by exploiting people and resources around the world for its own benefit. Sketch that argument for us.

6. The Empire of AI

Karen Hao

If we think about empires of old and the long, centuries-long history of European colonialism, they effectively went around the world and laid claim to resources that were not their own, but they designed rules that suggested they suddenly were. They exploited a lot of labor. They didn’t pay workers, or they paid them extremely little, and that labor ultimately helped to fortify the empire. All of that resource extraction and labor exploitation accrued benefits to the empire.

They did this under a justification of a civilizing mission. They were ultimately doing this to bring progress and modernity to the rest of the world, and we’re literally seeing empires of AI effectively do the same thing. What I say in the book is that they’re not as overtly violent as empires of old. We’ve had 150 years of social morals and progress, so there isn’t that kind of overt violence today. But they are doing the same thing of laying claim to resources that are not their own.

That includes the labor of a lot of artists and writers. That includes all the data that people have put online, which they’ve just scraped into these internet datasets. That includes exploiting the labor of the people they contract to help clean their models and annotate the data that goes into their models. It also includes labor exploitation in the sense that they’re building technologies that are ultimately designed to outperform most humans at economically valuable work.

OpenAI literally says that its definition of AGI is to create AI systems that will be able to outperform most humans at economically valuable work. That is a labor-automation machine, so they’re also exploiting labor in the sense that they’re creating these AI systems that will make it dramatically more difficult for workers to demand rights. They’re doing it under this civilizing mission where they’re saying, “Ultimately, this is for the benefit of all of humanity,” but what we’re seeing is that’s not true. When you go far and away from Silicon Valley, to places like the Global South, rural communities, impoverished communities, and marginalized communities, they really feel the brunt of this AI development, this extraction, and this exploitation. They’re not at all receiving any of the supposed benefits of this accelerating AI “progress.”

Kevin Roose

Let’s talk about some of that extraction of natural resources. This is one of the things that your book gets into that I don’t think gets discussed a lot in the context of AI. Tell us about some of your reporting and what you saw.

Karen Hao

I ended up spending a lot of time in Latin America and also in Arizona to understand the sheer amount of computational infrastructure that is now being built to support the generative AI paradigm and the quest for AGI. These are massive data centers and supercomputers being plopped into communities that initially accept this kind of infrastructure either because they don’t know about it, because companies enter these communities through shell companies and aren’t transparent about putting this infrastructure there, or because they’re persuaded into it by what seems to be a really positive economic case.

A company comes in and says, “We’re going to give you hundreds of millions of dollars to build this data center here, and it’s going to create a bunch of jobs.” What they don’t say is that the jobs aren’t permanent. They’re talking about construction jobs, and once the construction jobs are over, there actually aren’t that many jobs for running the data center.

These data centers consume an enormous amount of power, and they consume an enormous amount of water because they need to be cooled when they’re training these models 24/7. This infrastructure is permanent, so once it gets put there, even if a city doesn’t have that kind of energy anymore or the water to provide to these data centers, they can’t really roll it back. In Chile, I was with activists who had been fighting tooth and nail to try to stop these data centers from literally taking all of their drinking water.

They were also entering communities in Uruguay, where I was spending time during a drought. People were literally drinking bottled water if they could afford it, or contaminated water if they couldn’t, because there wasn’t enough fresh drinking water to go around. That was when Google decided to build a data center there. When I say that the current AI development paradigm is creating a lot of harms at a mass scale, that’s the kind of thing I’m referring to.

Kevin Roose

Yeah. Part of empire-building is about exerting political power, right? I’m curious why the governments in Chile and Uruguay are okay with this. What is the mechanism through which they’re deciding to grant all of this power to these AI companies?

Karen Hao

A lot of governments learn that they have to serve the Global North if they want to get more investment, jobs, and opportunity into their country. In the AI case, it ends up not being a good bargain, but a lot of them don’t know that upfront. They think that if they can open up their land, water, and energy to these companies, somehow they’ll get more investment and more high-quality, white-collar jobs in the future.

I was talking with politicians who said they hoped that if they allowed a data center, Microsoft would eventually bring in an office with software-engineering jobs nearby. That’s the reason they end up doing this. Chile has a really interesting history in particular, in that it has dealt with centuries of extraction.

Most recently, they've become a huge provider of lithium for the lithium boom. They've developed this mentality over time that this is what they do. They open up their natural resources to these multinationals, believing that somehow this will convert into economic growth—broad-based economic growth for people. But unfortunately, it doesn't really.

Kevin Roose

Well, I want to push back on that a little bit because I think if I'm being sympathetic to the people, the politicians, and the communities that are accepting this stuff, there's a case to be made that it is actually helping them—maybe not in terms of direct GDP or economic growth, but the World Bank recently did a randomized controlled trial with students in Nigeria who were given access to GPT-4 for AI-assisted tutoring and found that it boosted their test scores significantly. The gains were especially big among girls who were behind in their classes.

As I'm hearing you talk about the exploitation taking place, I'm thinking, “Well, maybe there is something that they're getting in return. Maybe there is something worth it to them. Maybe this technology can, in some instances, help level the playing field between poorer countries in the Global South and places like America.” Maybe there's a deal to be had where it's like, “Okay, you want to extract our lithium, you want to build a data center in our country? Sure, but you have to give all of our students free access to ChatGPT Pro or something like that.” Is there any sort of fair exchange that you can imagine that would help these people?

Karen Hao

I think this question is premised on the idea that we have to make these trade-offs in order to get that kind of gain. We have to give you our lithium in order to have some kind of educational boost from ChatGPT, and that's a premise that I just don't agree with. I think there are ways to develop AI that give you the gains without this kind of extraction. The reason why I call it Empire of AI in the book is, in part, to point out that this is not the only pathway to AI development.

These companies have chosen a very particular pathway of AI development that is predicated on absolutely massive amounts of scale, massive amounts of resources, mass amounts of data.

Kevin Roose

Well, that's how you get the models to be general and good and to be able to work in all kinds of different languages. You're suggesting there's another path. What is the path other than—

Karen Hao

Yes.

Kevin Roose

—through scale?

Karen Hao

We don't necessarily know what it is yet, but it isn't being explored at all, and there are already signs that there can be other ways to get to these more general capabilities without that scale. DeepSeek is a really interesting example of this. I think there are also a lot of problems with DeepSeek, but DeepSeek demonstrated that, even in a resource-constrained environment, you can actually develop models that have more generality.

This is what science is: You have to discover the frontiers of what we don't know yet. The industry has fallen into this very specific scaling paradigm that they know works, but that comes with so many externalities that it's ultimately not actually achieving what OpenAI says its mission is: “benefit all of humanity.”

If we constrain the problem to ask how we can get more positives out of this technology without having all of that negative harm, I think there would actually be more innovation that would come out—true innovation—that would be more beneficial.

Kevin Roose

Karen, one thing that is very clear in your book is that you are not a fan of the big, general-purpose AI models. You call them “monstrosities built from consuming previously unfathomable amounts of data, labor, computing power, and natural resources.” Is there any way for people to engage ethically with these models, in your view, or is it all fruit from a poisoned tree?

Karen Hao

I think the way that they're being developed right now—personally, I do think that it's fruit from a poisoned tree.

Kevin Roose

Do you use ChatGPT at all?

Karen Hao

Not really, no.

Kevin Roose

Have you ever?

Karen Hao

Yes, I have.

Kevin Roose

Did you? I'm just curious because writing a book is—I’m doing it now, and I'm finding a lot of uses for AI. I'm just curious: This is a very thoroughly researched book. Was it helpful? Were any AI tools used in the creation of this book?

Karen Hao

No generative AI tools, but I did use predictive AI tools. I used Google Reverse Image Search to try to figure out the price of OpenAI's furniture because they had some really nice chairs. I was trying to explain the level of upgrade that happened when they went from a nonprofit in one office to this new, Microsoft-backed capped-profit entity in another office. When I ran the reverse image search, it came up with Brazilian designer chairs that were $10,000 each.

I do use predictive AI, but I did not use generative AI for this book other than to understand how the tool works and test out its new features. I never used it for getting research or organizing thoughts or anything like that because, at the end of the day, I'm writing a book about OpenAI, and I'm not going to willingly hand a bunch of my data about what I'm thinking about and what I'm researching to OpenAI in the process.

Casey Newton

And that's where you and Kevin are different. I want you guys to interact about this a little bit because, Karen, let me tell you: If Kevin can use generative AI to do something, he's doing it, okay? There's going to be a lot of generative AI going into the making of this book you're writing, right?

Kevin Roose

Well, in the research phase—

Casey Newton

Yeah.

Kevin Roose

Because I've found that it's not that good at composing. But it is super, super useful for: “Give me a history of the term AGI, where it originated, who the first people to use it were, how it evolved over the years, and how every lab has defined it in all of its various publications.” That kind of thing would have taken me weeks before, and now it's minutes.

Casey Newton

Right. So, Karen, make your case—

Karen Hao

Yeah.

Casey Newton

—that Kevin should stop doing that.

Karen Hao

I'm not going to make that case, but what I'm going to say is that this is the perfect use case for these tools. These companies are constantly testing their tools on AI topics. That is the thing that they stress-test their tools on. If there were any topic in the world that these chatbots would be particularly good at talking about, it would be AI and AGI. So, Kevin, move forward. Fire away.

Kevin Roose

Here's another thing that I wanted to ask you, Karen, because I think this is another place—

Karen Hao

Yeah.

Kevin Roose

—where we sort of disagree.

Karen Hao

Yeah.

Kevin Roose

You are very skeptical about the claims that the AI labs are making about AI safety or the concept of AGI. I guess I'm trying to understand that argument. My view on these folks is that they are sincere, that they are sincere when they worry about AI posing risks to humanity. I think that's why they're investing tons of money into AI safety and trying to work on things like interpretability—figuring out how these language models work.

Is your view that they are sincere but just wrong about AI possibly being an existential threat, or that they don't believe it at all and are just using AI safety as a smokescreen or an excuse for raising money and continuing to build their models?

Karen Hao

I think it totally depends on who you're talking about. In general, I think a lot of people are incredibly sincere about believing in these problems. I don't have any doubt about that. I talked with a lot of them for my book. I talked to people whose voices were quivering while they were telling me about being really, really scared about the demise of humanity. That's a sincere belief and a sincere reaction.

I think there are other people who pretend that they believe in this as a smokescreen. But I think, by and large, a lot of these people do truly believe, and their heart is where their mouth is. They are trying to do good by the world. My critique is that this particular worldview is just really narrow. It's just really, really narrow and a product of being in Silicon Valley, which is one of the wealthiest epicenters of one of the wealthiest countries in the world. Of course, you're going to have the luxury to think about these really far-off problems that don't have to do with things that are literally harming and affecting people all around the world today.

It's not that I don't think we should focus on any research into these problems. That's not what I'm saying. But I think the sheer amount of resources that are going toward prioritizing these problems over present-day problems is a super...

It's just not at all proportional to what the problem landscape literally is in reality.

Kevin Roose

Yeah, so when people like Sam Altman or Dario Amodei or Demis Hassabis say that we are a couple of years away from something like AGI or even superintelligence, your view is that that just has no reflection on reality, or that we should cross that bridge when we come to it and pay attention to the stuff that we can actually observe in the world now?

Karen Hao

So I think it also depends on how they define AGI. When OpenAI says that they are 2 years away from potentially automating away most labor, I could believe that they're on a path to systems that would appear to do so in 2 years and then lead to a lot of company executives deciding to hire the AI instead of hiring workers. If we're talking about AGI under another definition, then it would have to be a case-by-case question: How are they defining AGI, and what is their timescale? But do I think that OpenAI has high convictions to try and create a labor-automating machine and that they have the resources to start making a dent in labor opportunities for people? Yes, I do.

Casey Newton

Well, maybe let's have the kind of “How do you define AGI?” conversation. It's come up a few times during this conversation, and I know there are a lot of folks who regularly remark that the definition of AGI seems really amorphous and slippery to them.

Karen Hao

Yeah.

Casey Newton

It doesn't feel that amorphous to me. I work—

Karen Hao

Mm-hmm.

Casey Newton

—with an assistant. My assistant does customer service stuff, scheduling stuff, a little bit of sales. If there was a tool that I could use and pay a subscription to that did those things on my behalf, I think I would say, “Yeah, I think that feels like AGI.” So that's how I conceive of it in my mind, but I know there are so many folks out there who say, “No, no, no, no, no, the definition is always changing and slippery,” and this is a really big problem. So Karen, how do you feel about it?

Karen Hao

What you are describing—yeah, if you wanted to define that as AGI, that's totally fine, but I don't think that's how the companies are necessarily defining AGI, right? They are not defining it well, but when they need to raise capital, when they need to rally public support, when they need to get in front of Congress and try and ward off regulation, the things that they say are, “One day AGI will solve climate change. One day it will cure cancer.” I think the AGI system that you're describing is not exactly the AGI system that they are sketching out in that broad, sweeping vision that they're trying to use as justification to continue doing what they're doing.

Casey Newton

Right. There's a lot of hand-waving that goes on when somebody says that some future AI technology is going to cure cancer. It's leaving out many steps.

Kevin Roose

Well, but—

Karen Hao

Yeah.

Kevin Roose

In partial defense of the labs here, I think we have seen things like AlphaFold, which was Google DeepMind's system that solved the protein-folding problem, essentially, and that was—

Karen Hao

Yeah.

Kevin Roose

—not something that they thought was going to be the end of their progress toward scientific cures for disease. That was sort of the beginning stages, and actually, if you talk to biomedical researchers, they say that was a huge deal and really did make it possible—

Karen Hao

Yeah.

Kevin Roose

—to do all kinds of new drug discoveries. And I guess that part feels a little separate to me from the AGI discussion. But it does feel like the quest for AGI, the scaling up of these models, the attempt to make them more general—there have just been good things that fall out of that process, and also some externalities that you mentioned, Karen. But I'm just curious if you see any positive applications of the scaling hypothesis and the sort of dominant paradigm.

Karen Hao

I don't think I have come across a positive application that I think justifies the amount of cost going into it, and I think, to return to DeepMind's AlphaFold, that was not a general intelligence system. That was a task-specific system, right? That's what I advocate for. I think we need more task-specific AI systems, where we give them a well-scoped problem, we curate the data, we then train the model, and then it does remarkable things. I totally agree that AlphaFold was a remarkable achievement, and I don't think that has much correlation with what AGI labs are now doing with the scaling paradigm. Those are 2 perpendicular tracks to me.

Casey Newton

Yeah, I think it's clear that the hype is far ahead of the results right now. We have heard a lot more about AGI curing cancer than we've actually seen progress toward curing cancer at the moment of this recording. Now, some people believe that's going to change very soon, but I can understand why, if you read a lot of headlines and you don't see cancer being cured yet, you'd have some questions.

Karen Hao

Yeah, and I think the other thing here is these companies are continuing to say that they're AGI labs, they're pursuing AGI, but they've dramatically shifted, and now they're really just focused on building products and services that they can charge lots of money for. And all of the maneuvering that they've tried to do to make it seem like that is on exactly the same path as what they're saying is AGI—come on. That's probably not what's happening here. In the last episode, you guys were talking about AI flattery and the debacle around that, and how they're turning to maximizing for engagement because this is the thing that they've realized gets them a lot of users, gets them more cash flow. That is ultimately what they're now building. So I think what they're saying they're building and what they're building is also starting to diverge in the new era, I guess, where they need to be able to justify a $40 billion raise.

Casey Newton

Yeah.

Well, let's sort of bring it home here by talking about one thing that I think all 3 of us agree on. You write that the most urgent question of our generation is: How do we govern artificial intelligence? I agree with you on that front, Karen. So let me ask: How do we govern artificial intelligence? Please help us.

Karen Hao

Democratically?

Casey Newton

Yeah. So what does a more democratic way of governing AI look like?

7. AI Needs Democratic Control

Karen Hao

So to me, it's like you consider the supply chain of AI development. You have data, you have compute, you have models, you have applications. I think at every single stage of that supply chain, there should be input from people, not just the companies. When companies decide that they're going to curate a data set, there should be people who can opt in and opt out of that data set. There should be people who can weigh in not just on their own data; maybe there should be consortiums debating what kind of publicly accessible data should or should not go into these tools. There should be debates about content moderation of the data because, as I write in the book, there were a lot of moments in OpenAI's history where they just debated internally, “Should we keep pornographic images in the data set or not?” And then they just decided it on the fly. That, to me, is not democratic governance. We should be having open public discourse about those types of decisions.

When it comes to compute, there should be an ability for communities to even know that data centers are coming into their communities. And they should then be able to go to a city council meeting and actually talk with their city council, talk with the companies about whether or not they want the data center, and have good, solid information about what the long-term trajectory of hosting a data center would actually look like.

And when it comes to the labor—the contract workers who are working for AI—they should follow international human rights norms, because a lot of the conditions in which these workers are working do not follow international human rights norms. So I think that's the way that I think about all of these different stages: All need to be democratic. And when OpenAI says, “We're going to develop democratic AI simply because we're an American company,” that's not how it works. Everyone actually has to participate, have agency, and have a say to shape and change what is and isn't developed and how.

Kevin Roose

Well, Karen, this has been a fascinating conversation. I really appreciate your time, and thanks.

Karen Hao

Thank you so much for having me.

Casey Newton

When we come back, turn your brain off. It's time to talk about Italian brain rot.

Kevin Roose

Ooh, sounds fancy.

8. Italian Brain Rot Takes Over

Casey Newton

Kevin, if I were to start referring to you as Kevinini Russolini, what would that mean to you?

Kevin Roose

I would think it was some sort of mockery of my Italian heritage.

Casey Newton

I would never. I would never. What about Tralalero Tralala? You know him?

Kevin Roose

No, I think you're having a stroke.

Casey Newton

What about Bombardino Crocodilo?

Kevin Roose

Okay, now this is just getting ridiculous.

Casey Newton

Ballerina Cappuccina?

Kevin Roose

Nope.

Casey Newton

All right. Listen, if you or someone you love recognizes any of these terms, Kevin—you may be suffering from a case of Italian brain rot.

Kevin Roose

I'm almost afraid to ask. I have not been following this story, although I know you were very excited to tell me about it today. What is going on with Italian brain rot?

Casey Newton

Do not be afraid of Italian brain rot, Kevin. If you have been on TikTok, Instagram, or YouTube over the past many weeks, you may have encountered this unique form of AI-enabled insanity.

Kevin Roose

Typically, I know that brain rot refers to this feeling of cognitive decline related to excessive use of social media or something like that. People on TikTok are always complaining about their brain rot, but what is Italian brain rot?

Casey Newton

If you want to catch up on this, I highly recommend a story in The Times by Alisha Haridasani Gupta, who catches you up. This stuff started to emerge in January, and it really is an AI phenomenon. Recently, Kevin, we've seen advances in some of these text-to-video generators. So you might be able to, for example, create a short clip of a little coffee cup that is also a ballerina. Well, congratulations, you just invented Ballerina Cappuccina.

Kevin Roose

To me, this is the difference between this age of viral content and previous generations of viral content. I spend a lot of time on TikTok—

Casey Newton

Uh-huh.

Kevin Roose

—but I have never, literally never, seen anything about Italian brain rot, and it's such a contrast to how everyone knew the Ice Bucket Challenge was happening, right?

Casey Newton

Mm-hmm.

Kevin Roose

Because you could see it everywhere. But things have become so siloed and atomized that you could tell me literally anything was happening on TikTok and that millions of people were into it, that it was the trend sweeping the youth, and I would have no idea. So either that means I'm old or something has changed about social media.

Casey Newton

This is why you have to have your younger colleagues like myself come in and tell you what's happening in middle school.

Kevin Roose

You are not younger than me.

Casey Newton

Spiritually, I think there's a case for it. So listen, there's no way to talk about Italian brain rot that improves on the experience of actually watching it. Let's watch a couple of clips of brain rot. I believe we have one queued up.

Kevin Roose

I hope I get hazard pay for this.

Speaker 6

Tung tung tung tung tung tung tung tung tung sahur. Brr brr patapim. Il mio cappello è pieno di sle- Capo cappuccino assassino. Ballerina cappuccina. Mi mi mi mi. Scimpanzini bananini. Wah wah wah. Troppi, troppa trippa.

Speaker 8

Glorbo Fruttodrillo.

Kevin Roose

If you are not watching these, let me just describe what I just saw.

Casey Newton

Yeah, please.

Kevin Roose

This is a compilation of these Italian brain rot memes, which were all AI-generated weird characters. One of them looked like a hamster poking out from half of a coconut.

Casey Newton

That's right.

Kevin Roose

And they're just saying these Italian phrases.

Casey Newton

Well, yes.

Kevin Roose

So this is Italian brain rot?

Casey Newton

This is Italian brain rot. You're probably grasping the Italian part because they're being voiced in this over-the-top Italian accent, and all of these strange phrases that you're hearing are the names of the characters. I know you're probably wondering who Trippi Troppi Troppa Trippa is. That's a shrimp with a cat head.

I love this one because, with a lot of meme explainers, there's a lot of excavating to do about where they came from and what they're about. Here, it really is just what it says on the tin. It is an Italian accent over a series of images that make you feel like you're going insane.

Kevin Roose

Yes. And was this made by an Italian?

Casey Newton

No. In fact, in The Times, one of the main creators—the person who created Ballerina Cappuccina—was Susanu Savatudor, a 24-year-old from Romania, who told The Times that this is just a form of absurd humor that really has very little to do with Italy. This creator just created the name Ballerina Cappuccina, and they've gotten more than 45 million views on TikTok and 3.8 million likes.

Kevin Roose

Oh, my God. At the risk of explaining a joke and thereby killing it, is there any point to Italian brain rot? Is it making some sort of social commentary? Is it trying to say, “Italians are big users of social media and therefore are getting brain rot”?

Casey Newton

I actually do have a theory about this. I think what makes this feel new is that whatever this is actually does feel fresh. We live in a time where everything Hollywood is giving us feels like a recycled version of something else. We are on Phase 6 of the Marvel Cinematic Universe. In that world, it's like, “Oh, and here's Ant-Man's cousin.” People are saying, “Fuck that. Give me Ballerina Cappuccina.”

Kevin Roose

It does feel like there is some organic hunger out there for really stupid stuff, just really random things. I was thinking about this recently. You know the Minecraft movie is a big hit, right? People are—it's one of the biggest movies of the year.

Casey Newton

Yeah.

Kevin Roose

And there's this moment in the movie, apparently—I have not seen it—where someone says the words “Chicken Jockey.”

Casey Newton

Yeah.

Kevin Roose

Jack Black does, I think. At that moment, teens and other young people have decided that this is the moment in the movie to stand up and cause a ruckus. They start throwing popcorn. Someone actually brought a live chicken to the theater and held it up. This feels like it's of a league with “Chicken Jockey” from the Minecraft movie in the sense that it is absurdist. Trying to explain it actually makes you dumber in some way, and so there's a kind of appealing randomness to it.

Casey Newton

Yeah. And by the way, I think that is actually part of being a young person: building a language that is inaccessible to people older than you, right? That is sort of how the identity-formation process works. There are older people, older people have no idea who Trippi Troppi Troppa Trippa is, and that is something that you can talk about with your friends that belongs just to you.

Kevin Roose

Wait, what are some of the other ones?

Casey Newton

I'm glad you asked, because we haven't actually watched enough of these videos yet. Kevin, I would now like to direct your attention to one: Salamino Pinguino.

Speaker 8

Salamino Pinguino. Mezzo salame, mezzo pinguino. Tutto problema. Non scivola, si affetta.

Kevin Roose

This is a penguin covered in salami.

Casey Newton

Like wearing almost a headdress—

Speaker 8

Mamma mia—

Casey Newton

—made out of salami.

Speaker 8

E sputa peperoni piccanti. Salamino Pinguino. La leggenda della salumeria.

Casey Newton

Now, let's take a look at Glorbo.

Speaker 8

Glorbo Fruttodrillo.

Kevin Roose

Okay. This is a—

Speaker 8

Glorbo Fruttodrillo—

Kevin Roose

—crocodile or alligator with a watermelon for a body.

Speaker 8

—regnava. Ma la testa e la coda? Tutto. Alligatore.

Casey Newton

Yeah. This is a still image with 578,000 likes.

Casey Newton

Everybody loves Glorbo. But—

Kevin Roose

Is this even real Italian? Are we sure it's real Italian?

Casey Newton

I'm pretty sure it's not real Italian. Let's stop that one there, and then let's go on. I know what you're saying: “Casey, these characters are just standing around. That seems super boring.” What if I were to tell you that other creators are now incorporating them into dramas, Kevin?

Kevin Roose

Oh, boy.

Casey Newton

Let's take a look at one of those. This one stars Tralalero Tralala, who is a shark wearing a suit.

Kevin Roose

And is that Ballerina Cappuccina I see?

Casey Newton

That is Ballerina Cappuccina, and she's with Tung Tung Tung Sahur.

Speaker 8

Tunga Tunga Tunga and Tung Tung Tung Sahur enjoying their honeymoon.

Casey Newton

So he leaves for the day, and there comes Tralalero Tralala, the shark. And now they're kissing in bed. Oh, no.

Kevin Roose

Oh, no. She's pregnant.

Casey Newton

Ballerina Cappuccina's pregnant.

Kevin Roose

Oh, no.

Casey Newton

Ah.

Speaker 8

La Polizia, no.

Casey Newton

And now Tung Tung Tung Sahur is chasing after—

Speaker 8

Musica, Lala—

Casey Newton

—the shark.

Speaker 8

Bombardiro Crocodini.

Casey Newton

And that's Bombardini Crocodini, and he sends in an airstrike. So that was—let's just review. That was, I don't know, 10 or 15 seconds. In that, you see 2 of these characters: one of them gets into an affair, has a love child, her partner finds out, and then sends in an airstrike to attack the cheater. So they're doing a lot in 15 seconds, yeah.

Kevin Roose

Wow. That was not a Pixar film. That was really something. I feel like I'm on a very powerful psychedelic right now.

Casey Newton

Well, you mentioned earlier that, in the old days, we would do things like the Ice Bucket Challenge. Kevin, what if I told you that some of these Italian Brain Rot characters are actually doing the Ice Bucket Challenge?

Kevin Roose

No.

Casey Newton

Yeah. Let's watch that one.

Speaker 9

My name is Chimpanzini Bananini, and I've been nominated for the USC- ... Making My Brain Rot Ice Bucket Challenge. I nominate Bom Bombini Gusini, Trippy Troppy, and Boneca Ambalaini.

Casey Newton

And he's nominating the other characters to do the Ice Bucket Challenge.

Kevin Roose

This is so dumb.

Casey Newton

Yeah. It's so dumb.

Kevin Roose

It's very funny, though. I am genuinely laughing at this. But I could not explain to you why this is funny if you paid me.

Casey Newton

Well, here, listen. I have done a little bit of comedy in my life, and one thing that I learned in improv was that everyone goes nuts for an over-the-top Italian accent. It's extremely funny. All I have to do is say, “Make a bowl of spaghetti.” You're already laughing, see? I don't even have to do anything.

The Italian Brain Rot functions much in the same way. But they are taking advantage of this AI thing. And, you know, look, we talked earlier on this show about how these systems are being trained with other people's art without their consent. There are some people who feel like you could never make anything truly creative or truly artistic with AI. And yet, here you have this bona fide viral phenomenon that is people making extremely silly stuff using AI, and it is resonating with us.

I think this has been one of the more counterintuitive lessons of AI slop. A year or so ago, we were looking at images of Shrimp Jesus all over Facebook, and we were saying, “That seems silly. I'm sure the company is going to get rid of this.” No, no, no, my friend. They're going to lean into it, because there are riches that lie down this path. And Italian Brain Rot is the first example, I think, of that happening.

Kevin Roose

God. So I have a couple of reactions. One of them is, yes, I absolutely think that AI has utility, and that there are good things that have come out of it. But seeing Italian Brain Rot makes me want to nuke the data centers. I'm like, shut it all down. We've gone too far.

But seriously, I do think there is something here, not just in the absurdist humor of this thing, but I do think there are going to be new kinds of entertainment that are birthed out of these tools. Because if you wanted to make something like a ballerina with a cappuccino for a head, 10 years ago, you needed to be an animator to do that, or at least have some facility with animating software. Now, you just go into an AI tool and you type, “Give me a Ballerina Cappuccina,” and out comes this pretty perfect animation.

Casey Newton

Yeah, which has always been the case for this sort of tool, by the way: It takes people who do not have those kinds of artistic skills and lets them express themselves creatively. If they can think it, they can visualize it, they can make it available to other people.

Here is my case for why this is actually a good thing, Kevin. I was thinking this morning about a few years back, during the height of the crypto boom, when people started talking about how crypto could be used to fund these alternative worlds of entertainment, right? Like, the Bored Apes Yacht Club was going to become this mega-franchise, but what made it cool was that anybody could buy in.

Kevin Roose

Anyone could get a Slurp Juice.

Casey Newton

Anyone could get a Slurp Juice, put it on a mutant ape, transform your mutant ape, et cetera. And people didn't really get into this because I think nobody wanted to be involved in what was essentially like a homeowners' association for creating entertainment.

But I look at Italian Brain Rot and I see something similar happening, where, as far as I can tell, no one has a trademark on Ballerina Cappuccina or Chimpanzini Bananini. You could just sort of make your own version of it and put it up there, and nobody's going to issue a copyright strike. You can have these characters do whatever you want to. So it feels like there is actually a freedom in making this that people are really responding to.

And so maybe we do actually get the next version of crowdsourced entertainment, and it all comes out of these bizarre text-to-video makers.

Kevin Roose

I gotta say, I believe you when you say that that is a possible outcome, but my brain just goes immediately to some office at Disney headquarters where they're watching these Italian Brain Rot memes and furiously trying to license the IP to make a series of 7 movies about Chimpanzini Bananini.

Casey Newton

Yeah.

Kevin Roose

And I do think that there's a possibility that this becomes just like any other entertainment franchise.

Casey Newton

It could go that way. But maybe that sort of robs it of the fun of it that makes it go viral today to begin with.

Kevin Roose

And they're making movies out of Minecraft. They can make movies on anything.

Casey Newton

They're really running out of things to make movies out of, as far as I can tell. So do I lean optimistic about this? Yes. At the same time, do I think that if China had come up with this idea independently as a way of bringing down American civilization, it would be a great idea? If they were like, “What if we just did weird characters in Italian accents? Could that distract all of American middle schoolers for a year?” Probably worth doing. How hard could it be?

Kevin Roose

This is all a CCP plot to undermine American sovereignty.

Casey Newton

That's always been the thing with TikTok. I don't think it's a Chinese plot to destroy America, but it is working.

Kevin Roose

Well, if Cappuccina Ballerina starts singing the praises of Xi Jinping, we'll know that something grave has gone wrong.

Casey Newton

Yeah. We'll keep our eyes on that one.

Bad Apple + The Rise of the AI Empire + Italian Brain Rot | BidClub