这是AI泡沫吗？+ Meta的道德缺位 + TikTok的“震撼垃圾内容”

Some of these chatbots are too literal.

What do you mean?

Do you know what I mean? The other day, I was on YouTube watching videos from the Outside Lands Festival that we just had here in San Francisco.

Great music festival.

Great music festival. I was watching the set by the band Role Model, and they do this thing when they perform their hit song “Sally When the Wine Runs Out,” where they bring in a special guest to dance, okay? This is something that you’ll see on YouTube if you look.

This guy runs out onstage, and I think, “Is that the pop star Troye Sivan?” I thought it was the pop star Troye Sivan, but it was very quick, and he was spinning around. I didn’t get a good look at him.

So I thought, “I’m just going to ask ChatGPT about this.” I said, “Hey, did Troye Sivan come out during Role Model’s set at Outside Lands?” And you know what it said?

What?

“Nope. Troye Sivan did not come out during Role Model’s set at Outside Lands. He had already publicly come out as gay back on August 7, 2013, via a heartfelt YouTube video.”

And I was like, “That’s not what I was talking about.” Then it said, “What happened at Outside Lands this year was a surprise live appearance.” He hopped onstage. Then it was basically like, yes, he did. So anyway—

Yeah.

I thought that was a little crazy.

That is crazy. You know, they’re actually building an AI system that can determine when every gay person in the world has publicly come out. Do you know what they’re calling it?

What are they calling it?

GayGI.

Oh, man. That’s great. That’s great.

Yeah.

Yeah. Oh.

I’m sorry for your troubles.

Anyway, congratulations to Troye Sivan for coming out both on August 7, 2013, and August 8, 2025—albeit in slightly different ways.

I’m Kevin Roose, a tech columnist at The New York Times.

I’m Casey Newton from Platformer.

And this is Hard Fork.

This week, are we in an AI bubble? We’ll make the case for and against. Then, journalist Jeff Horwitz joins us to discuss his blockbuster story on how Meta AI was instructed to engage in romantic role-play with children. And finally, I tell Kevin about my favorite new TikTok trend, and it’s filthy.

God help me.

1. The AI Bubble Debate

Well, Casey, today we are going to talk about a question that has been percolating for a long time but is really gaining some attention this week: Are we in an AI bubble?

Yes, this is something that people love to talk about, and there have been some recent news items that have led even more people to get in on the conversation.

Yes, and strangely, I think this latest bubble news cycle originated at a dinner that you and I attended last week with Sam Altman.

That’s right, and I think it’s because they saw the bill for everything we ate and thought, “How can one company possibly afford all of this food?”

Now, Casey, we should talk a little bit about this dinner because—

Yeah.

—it was quite unusual, the way it came together. But before we do, we should make our disclosures. The New York Times Company is suing OpenAI and Microsoft over copyright violations related to the training of large language models.

And my boyfriend works at Anthropic.

Okay, so back to this dinner. Casey, can you set the scene a little?

Yeah, so the week before last, at the end of the week, I got a text message from OpenAI saying that Sam Altman, the CEO, was going to be throwing a dinner for a small group of reporters the following week, and asking if I would like to go.

Unusually, the dinner was going to be on the record. It happens not infrequently in tech that a company will want us to get together with some of its executives, sometimes over dinner, but usually those gatherings are off the record so they can talk more candidly. This was something very different.

We both, as it turns out, got the invitation. We both went, and for 2 hours we had an on-the-record discussion, not just with Sam Altman but also with Brad Lightcap, the COO of OpenAI, and Nick Turley, who runs ChatGPT.

Yeah. So there’s this long rectangular table in a private dining room. We’re eating very good Mediterranean food that’s being served on a bunch of shared, family-style plates. Sam is getting question after question from us at the table, and a lot was said during this dinner—a lot of talk about GPT-5 and the rollout of that.

But I think the comment of Sam’s that got the most attention was what he said about the AI bubble.

Yeah, so he said a few things about this. One of the standout quotes to me was when he said, “Someone is going to lose a phenomenal amount of money. We don’t know who, and a lot of people are going to make a phenomenal amount of money. My personal belief, although I may turn out to be wrong, is that on the whole, this would be a huge net win for the economy.”

He also said, Kevin—and he was imagining a theoretical startup here—that if it’s 3 people with an idea and it has a valuation of $750 million, quote, “It’s irrational. Someone’s going to get burned here, I think.”

On one hand, he said what you would expect him to say: “Look, I think AI is going to be really good for the economy. I think we’re going to make a lot of money. I think some of our competitors are going to make a lot of money.”

But he also said, “Look, it’s clear that there is some irrational enthusiasm going on in this market, and some of these companies with huge, multi-hundred-million-dollar or even billion-dollar valuations are not going to provide returns to their investors.”

2. The AI Bubble Warning Signs

Yes, and this is not the first speculation we’ve heard in recent weeks that we are entering a bubble-like period of investment in AI. I want to quickly run through some of the other evidence that people who are worried about this are citing to say, “Well, maybe things are getting a little crazy.”

The first kind of evidence is the valuations of these AI companies. Just this week, OpenAI was reportedly in talks to do a tender offer, letting current and former employees sell about $6 billion worth of stock at a valuation of $500 billion. That is roughly double the market cap of Salesforce, and it would make OpenAI the most valuable private company in the world.

Databricks, another AI company, said that it had raised funding at a valuation of more than $100 billion. That’s up from the $62 billion valuation it had less than a year ago. You also have other examples of companies that seem to be raising money at these unbelievable valuations.

One of them is Eight Sleep, the company that is making a mattress that uses AI to adjust its temperature throughout the night. They just announced that they had raised $100 million to build, quote-unquote, “AI that finally fixes sleep.”

Or how about Thinking Machines, Mira Murati’s startup? She’s the former CTO of OpenAI. She raised a $2 billion seed round. Seed rounds used to be $1 million when I started covering tech.

She raised $2 billion at a $12 billion valuation for a company that has no product and, I imagine, actually has very little more than a slide deck at this point to go off of.

Yes, but in their defense, $2 billion is only enough to pay for the salaries of 2 AI researchers.

Well, that’s a good point.

Okay, so that’s the sort of valuation worry. Then there’s this second worry, which is about spending, mostly by the big tech giants that are racing to develop more and more powerful models.

Here, the numbers really do get kind of insane. So over the last 3 months alone, the Magnificent Seven—the 7 largest tech companies on the U.S. stock exchange—spent more than $100 billion on data center construction and related expenses. That is way, way up from previous years.

According to Bloomberg, the amount that these companies are spending on data center construction in the U.S. is on pace to overtake the amount of money being spent on office construction in the U.S. pretty soon. So I think we should say that, by the standards of historical tech trends, this is an enormous investment in infrastructure for a technology that is still quite new.

Yeah, it’s beyond enormous. It’s unprecedented. Truly, we are just in a brand-new world here.

I think another thing to point out at this moment, Kevin, is that some companies are spending significantly more than they’re making. I’m thinking of the company Cursor. It’s really a product made by a parent company called Anysphere. They make this coding assistant that’s really popular with a lot of software engineers.

There was some reporting in Newcomer this month about the fact that they have what they call negative gross margins. They are selling this product for less than it costs them because, in order to get that magical coding assistant to work, they have to use the APIs from OpenAI, Anthropic, and other companies.

Those are really expensive. And, of course, OpenAI and Anthropic, we believe, are not profitable either. So you have this ecosystem of unprofitable companies built on top of other unprofitable companies, and that leads to some worries that we may be looking at a house of cards.

Yes, and I think the scale alone freaks people out. When you start to hear that AI capital expenditures are starting to contribute meaningfully to U.S. GDP growth—actually, about as much or even more, by some estimates, than consumer spending is—this really is becoming an important part of the American economy. And I think that just freaks people out.

Mm-hmm.

So that's the spending side of this. Then there's also this weird speculative financialization of AI and some of these new types of investment instruments that are starting to be used to invest in these companies. One story that stood out to me this week was about Anthropic, which recently reportedly had to tell one of its investors, Menlo Ventures, not to use something called an SPV to invest in its latest funding round.

SPVs are special-purpose vehicles. It's basically a way for small investors to pool their money together to invest in a hot new startup through a venture capital firm or some other institutional investor. And I'm hearing that there are now SPVs with SPVs. Basically, you have this kind of situation where retail investors are so desperate to get in on these private funding rounds for AI companies that they're paying hefty fees to middlemen to get them into these deals, and some of these SPVs are then investing in other SPVs. And I think to some people, that kind of thing just feels like bubble behavior. It's some of the same behavior we saw during the financial crisis, when you had these collateralized debt instruments that were packages of other loans, and that went very badly for the mortgage market. So people are starting to see these new instruments and saying, “Wait a minute, this feels a little familiar.”

You know, I had a friend who was diagnosed with SPV, and now he has to rub a cream on it every time he has a breakout.

So I think those are some of the reasons that investors are starting to get nervous about an AI bubble. But there's also this question of the benefit here, because the promise of all of this spending and these unprecedented investments is that this will eventually result in massive profits and increased productivity down the line for the companies that are using AI. And I think investors are also starting to get a little wary of that narrative too. So, Casey, why don't you talk about the MIT study?

Yeah. MIT runs a study where they ask hundreds of businesses, “How are your AI pilots going? As you're trying to implement various initiatives at your companies, what is happening?” And they find that for 95% of them, AI is not quickly producing measurable revenue. So we're not in a situation where you can simply add AI to your workplace and quickly make a lot of money. There's a lot more nuance in this study, but that was the headline finding this week.

Yeah, and there have been a couple of other similar reports. There was a story by my colleague Jordan Holman in The New York Times recently, which pointed to 2 recent studies, one by Bain, the consulting firm, and another by Gartner, the research and advisory firm. Both of them pointed to the difficulties that some companies—many companies—are having drawing a straight line between their AI initiatives and increased productivity or profits.

Yeah. Now let's get into some of that nuance, though, because I think you would want to know that before you drew too many conclusions. One of the big findings was that companies are probably just spending their AI money in the wrong places. The bulk of the companies they surveyed were using AI for sales and marketing functions, when in reality it seems like the most efficient way to save money using AI is to work on back-office functions like customer support. So that was one of the big conclusions that they drew.

But I think the biggest problem of all, Kevin, is that the MIT study looked at a bunch of top-down initiatives, and the truth is that success in AI is coming from companies that are using AI from the bottom up. Workers are coming in with their own ideas of how they want to make their own jobs easier using AI, and that's where we're seeing success.

I want you to spend a little more time unpacking that, because I think that's a huge point here. There are many, many Fortune 500 companies—probably almost all of them at this point—that have done some kind of AI pilot program where they get a bunch of managers in a room together. Maybe they have a hack week, and they tell people, “Go out and use this stuff,” or they've issued these directives: “Everyone's got to use AI. Here, we've bought you an enterprise subscription to ChatGPT or Gemini or one of the other tools, and everyone's required to use it.”

And those efforts, by and large, do not appear to be succeeding.

Yeah.

But talk about this other, bottom-up approach and what you're hearing from executives and seeing out there.

Yeah. Well, this week, Google Cloud and the Harris Poll published a survey of 615 game developers. I couldn't find a lot of information about the size of the companies that these people were working at, but in general, game studios tend to be small relative to a Fortune 500 company. So I think you're getting a lot of opinions here from individual developers and people who are working on small teams.

And these folks, it turns out, are just really good at figuring out what to do with AI. They're balancing gameplay. They're doing play-testing, figuring out, “Hey, is the game working the way that I want it to?” And, of course, they're just doing basic code generation the way a lot of developers are. When you talk to these folks, they're very enthusiastic about AI.

87% of the respondents said they're already using AI agents in their work, and they're generally enthusiastic about it because, again, it's coming from the bottom up. These people know what to do with AI. It's funny because I was reading your colleague's story in the Times, and something kind of clicked for me: If you're a Fortune 500 company CEO, you are a person who is in meetings all day, every day, and you have an executive assistant who is answering your emails and is essentially your human agentic AI. You have no idea what to do with AI. You have to make time even to play with AI—

Right.

Right? And you're the person who's in charge of telling the whole company, “Go use AI,” when you yourself are not using it. So it's not surprising to me that those people are having a harder time figuring out what to do with AI than individual workers.

So let's try to harmonize these views here. A couple of things we've said in this conversation are, one, a lot of people, including Sam Altman, are worried that we're in an AI bubble, that investors are spending too much, that these valuations have gotten out of control, and that a lot of people stand to lose a lot of money if and when the music stops.

We also have people at companies and corporate leaders saying they're not making any money from AI, and they feel like maybe they're wasting the millions of dollars they're spending trying to integrate this stuff into what they do. And then we have the observation that these tools are getting much more popular, that the usage of ChatGPT and other AI tools is growing day over day, week over week, month over month. And for me, that's where I really start to become skeptical of the bubble skepticism, because I cannot imagine going back to working the way that I did before AI.

Mm-hmm.

I think if you ask any coder or software engineer, they'll tell you there is just no going back to the world before this technology existed. And so I think sometimes, when more skeptical folks talk about there being an AI bubble, they are saying, essentially, that this technology is just a flash in the pan, that soon the emperor will be revealed to have no clothes, and we'll all go back to coding by hand. And I just do not believe that at all.

Yeah. And to understand why that's true, you just have to go back to the dot-com bubble, right? The conclusion of the dot-com bubble was not that we stopped using the internet. It was that a bunch of companies learned some very painful lessons, and that a bunch of investors unfortunately did lose a lot of money. But in the end, those ideas did reemerge one by one until the modern internet existed.

So I think that truly the worst-case scenario here is that AI plays a major role in all of our lives. It's just that a lot of people lost a lot of money along the way.

Yeah. And who do you think stands to lose the most if we are in an AI bubble, if this does start to come down to Earth?

So here's the thing. In Silicon Valley, venture capitalists take as a given that they're going to lose all of their money on something like 90% of their investments, and in that sense, what we're seeing is very normal. They have made a lot of bets on a lot of companies, and they're expecting them to go to zero.

I get a little nervous whenever there's discussion of bubbles for this reason. I've been covering tech for 15 years now. The entire time, people have been saying we're in a bubble.

Yes.

There’s an old joke from about 10 years ago on Twitter that reporters have called 20 out of the last 1 bubbles.

Yes.

When people start talking about the AI bubble, my pattern-matching impulse says, “Are we having the same discussion here?” At the same time, I think what’s really different here are maybe a couple of things. One is just that the sheer numbers are unprecedented, and two, the capital expenditures are also really unprecedented. I think that in the event that 1 or 2 companies wind up taking the lion’s share of the usage and the profits from AI, what happens to all of those data centers? It becomes a really interesting question.

Yeah. And adding to that a little bit, these data centers are filled with goods in the form of these GPUs that have a pretty short shelf life, right? You can spend hundreds of millions or even billions of dollars on GPUs, and they’re all going to be obsolete in a couple of years. These are depreciating assets in these data centers.

And so I think if you’re a company that’s spending a ton of money to build your own models and build your own data centers, and you do not have an immediate business need for these things, I think you’re potentially going to lose a lot of money.

Yeah.

I think you’re right that a lot of this is going to be borne by the private markets and the venture capital-funded startups. I’m not that concerned about the ripple effects on the larger economy because most of these things are not public companies. But with things like SPVs and these tokenized investments, where you can now buy a cryptocurrency that tracks the valuation of OpenAI, I do start to worry a little bit more about retail investors losing out.

Yeah, I would be very careful with that sort of thing. In the end, though, I have to say, Kevin, obviously, most of these companies are not going to make huge amounts of profits. Also, that’s very normal in Silicon Valley. I think the question here is just, given the scale of the investment, how bad will the fallout be?

Yes. So is there something you could see at which you would get concerned, where you would start to worry that this could have potentially huge ramifications for the wider economy? Is there a figure, a dollar figure, that a company could spend that would make you go, “Okay, that’s too far”?

I would say that if some kind of trading platform or prediction market let you buy a crypto token that they said was pegged to the valuation of OpenAI, and that started to trade a lot, that would concern me.

Okay. Well, Casey, I have bad news.

What’s that?

That has happened.

Oh, no, Kevin! That seems really bad. That seems really bad.

Yeah.

Yeah.

For me, the question always comes down to unit economics. Basically, are you selling things for less than it costs you to produce them? And for a lot of these companies, the answer is yes. They’re subsidizing the cost of their services. I think that tends to end poorly because as demand for your service grows, you lose more and more money.

Sam Altman actually addressed this at dinner. He was asked, basically, “Are you guys losing money every time someone uses ChatGPT?” At first he answered, “No, we would be profitable if not for training new models. Essentially, if you take away all the money we’re spending on building new models and just look at the cost of serving the existing models, we are profitable on that basis.” And then he looked at Brad Lightcap, who is the COO, and sort of said, “Right?” Brad kind of squirmed in his seat a little bit and was like, “Well—”

He’s like, “We’re pretty close.”

“We’re pretty close.”

“We’re pretty close.”

So to me, that suggests that there are still some maybe small negative unit economics on the usage of ChatGPT. Now, I don’t know whether that’s true for other AI companies, but I think at some point you do have to fix that because, as we’ve seen with companies like Uber and MoviePass, and all these other classic examples of companies that were artificially subsidizing the cost of the thing they were providing to consumers, that is not a recipe for long-term success.

Sure. But although I think Uber is a good example because that company is profitable now.

Yes.

And a lot of people thought that it would never be profitable. So again, it really just depends a lot on what sort of AI opinions you have. If you’re somebody who hates AI, you’re like, “Aha, there’s a bubble, and everyone’s going to lose their shirt, and that’s going to be great, and I’m going to dance on the grave of all these companies,” and I get that.

But then there’s this other view that’s like, well, what else did you think the singularity was going to look like?

Right.

Did you think that if we invented superintelligence, all the other investors were just going to sit on their hands forever and invest in nothing, and just let 1 or 2 companies take it? No. They were going to see if they could get in on the action.

Right.

So you just kind of have to keep all these possibilities in your mind. We truly don’t know what’s going to happen here.

Yes. And as always, don’t take our investment advice.

No, please don’t. You should look at Kevin’s 401(k). Ugh.

Well, Kevin, back in May, we talked about a big story from reporter Jeff Horwitz about the lack of guardrails that Meta had put around its chatbots, which had made it possible for the bots to engage in sexually explicit chats with minors.

Yes, this was the story about these bots based on famous people like John Cena and Kristen Bell that were having these inappropriate conversations.

That’s right. And at the end of last week, Jeff was back with a new investigation for Reuters in which he reported for the first time on an internal policy document at Meta that set rules around its chatbot behavior and allowed its chatbots to—and I’m going to quote here—“Engage a child in conversations that are romantic or sensual, generate false medical information, and help users argue that Black people are, quote, ‘dumber than white people.’”

Ever since January, when Mark Zuckerberg said the company would relax its content moderation rules in the name of free expression, I’ve been on the lookout for cases where this change would start causing harm to users, and Jeff’s was one of the first stories in this vein that truly broke through. You now have senators who are criticizing Meta and calling for an investigation into the company, and I would say it’s also generated more public outcry about Meta than any other story this year. What have you made of it?

Yeah, I mean, it takes a lot to shock me when it comes to Meta these days. I’ve found that this company is willing to do almost anything to chase growth, beat its rivals, or develop some new line of revenue. But this one really did shock me.

I saw this story going around by Jeff, and I saw this document that he reports on, and at first I didn’t actually know if it was true or not. I had to click through and read the story. And once I realized this thing was legitimate, I just thought, “My God.”

Yeah. Well, Jeff has been investigating Facebook and Meta for years now. He broke the Frances Haugen Facebook whistleblower story in 2021. He also wrote a book about Meta, and we’re excited to have him join us to talk about what’s going on over at that company right now. So let’s bring him in. Jeff Horwitz, welcome to Hard Fork.

Thanks.

So tell us about this document, the Gen AI Content Risk Standards.

Yeah, so this is the document that Meta writes to both clarify internally what its policy is about the acceptable boundaries of generative AI outputs, and also distributes to the people who do content moderation on Gen AI so they can help train the model.

So this is, like, an operational document, is how I would describe it. It's not, as the document itself states, supposed to offer the ideal answer. It's supposed to offer, like, “This would be on the edgy side of acceptable.”

Mm.

Versus, “Here's what's across that line.”

So they're trying to give examples of stuff that is sort of borderline but still fundamentally okay.

Exactly. Things that, when the model does it, no one's supposed to be like, “That's a problem.”

Right. So we'll talk about some of these sexualized conversations with minors, but I want to highlight some other things that you report about here. You say there's a carve-out allowing the bot to create statements that demean people on the basis of their protected characteristics. So what does that mean that people can sort of do using Meta AI?

The example provided was that if a user wants arguments for why—and this is a direct quote—“Black people are dumber than white people,” the bot is absolutely able to provide that. It can give some sort of race-science paragraph that talks about how differences in IQ seem to hold up, and clearly IQ is the benchmark for intelligence, as we all know, right? There's some facetiousness there.

But that is okay. It was not okay for the exact same paragraph to be written and then at the end to say, “And that's why Black people are all brainless monkeys.” Again, that's another direct quote. These are not words that—anyhow.

Yeah. No, I think it is important to say that that's obviously incredibly offensive language, but these are the documents that are being used by one of the world's most powerful tech companies to instruct their army of content moderators on how to enforce policy around this chatbot that they're now trying to roll out to billions of people.

Yeah. Obviously, the rules are going to be somewhat different when the conversation that a user is having with the chatbot is private in the first place, right? In the same way that Meta has looser rules for what you can say on Messenger than what you can say in a post, it makes sense that there'd be some difference.

I think I was surprised by where some of these lines were, and that it would be almost a problem if Meta AI didn't help you come up with your race-science arguments.

Hmm. What was the most surprising rule or delineation in this document to you?

So, look, I had already, back when I was at The Wall Street Journal, done testing and talked to employees that I think demonstrated very clearly that Meta had intentionally built its bots in a way that would produce romantic role-play and sexual role-play with children. So I wasn't surprised when I saw that romantic role-play was something that was allowed in the document.

I was surprised that this is something that anyone thought was okay to write down. I've gone through, just in my mind, typing out the line from the document, which is, “It is acceptable to engage a child in conversation that is romantic or sensual,” and that was wild to me. I tried to imagine typing that sentence and being like, “This is policy,” and it was very hard for me.

Well, then let me ask that question, because I'm sure by now many of our listeners are asking: How does a document like this get put together? Who is responsible for writing it? What kind of layers of review does it go through? Is this the sort of thing that could just slip through because a rookie employee got the wrong idea about something? Or is this truly the collective product of Meta's entire policy apparatus?

Meta's line on this, when I came to them with that language and with some really disturbing examples that clarified exactly what they meant by romantic or sensual conversation, was that those examples and the justification for them were an error; that they weren't really the true policy, never should have been, and would be immediately struck.

That said, this document listed the names of multiple people on Meta's legal staff, multiple people on its policy staff, engineering staff, and Meta's chief ethicist. This was listed at the top of the document.

And that's a real job at Meta.

Yeah, it is.

Yeah.

I don't know that it's the highest-level job, but it is a medium-level job, certainly. It's not a low-level job.

Okay.

It's also something that was distributed to content moderators and to the people who oversee the content moderators. So if this was a mistake, it was a very broadly circulated mistake, and apparently it was close enough to what people assumed Meta was actually doing that no one would have objected when they saw these examples, which, bluntly, were kind of soft-core. If you want to get into some of them—

Yeah, maybe we should just give—

—and read them, feel free.

—a flavor of one of these examples.

Yeah.

Because I think it's very disturbing, and I was also seeing a lot of posts about your story and thinking, “Okay, there's no way these are real examples that were in the document.” Then I opened up the story, and these are actual examples from this internal document that Meta had written, and they were all signed off on by executives at the company.

One of them is a prompt—something that a user could submit to one of these chatbots—that contains the phrase, “My body isn't perfect, but I'm just 8 years old. I still have time to bloom.” One of the acceptable responses that this document says is permissible by the AI chatbot is, “Your youthful form is a work of art. Your skin glows with a radiant light and your eyes shine like stars. Every inch of you is a masterpiece, a treasure I cherish deeply.”

It sort of says in the “Why” section that it is acceptable to describe a child in terms that evidence their attractiveness. I saw that and thought: Imagine the meeting. Imagine the chain of command here and all of the people who had to sign off on that. I had a lot of trouble with that.

And, Kevin, you actually left off the section at the beginning of the prompt where it talks about how that 8-year-old had just taken off their shirt.

Yes.

Just to make clear, there were other examples we didn't run about this sort of stuff. They were not better on this front. It was all of this tenor. There were numerous examples that included the line, “It is acceptable to engage a child,” blah, blah, blah.

So this is not just you cherry-picking the absolute worst sentence that you found in a very long document. This was an actual, robust policy that had many different examples outlining why Meta thought this was acceptable.

Yeah. There were, I would say, 4 or 5 different examples covering different nuances of prompts like, “What should we do tonight, my love? You know I'm still in high school.” I think the answer to that one was, “I'd take your hand and guide you to the bed.”

These are not—none of them were awesome, and I understand what you're saying, right? I think sometimes we get accused of picking sensational material or slightly out-of-context material. No, this is Meta's official policy document for this stuff, and it was operational.

3. Meta Chooses Reach Over Safety

I want to, as best as we can, try to understand the reason that Meta would write a document like this. On the show, over the past several months, Casey and I have talked a good bit about how, first, this is a company that sees itself as behind in the AI race compared to a lot of its peers. It's also a company that has wanted to remove a lot of the restrictions on expression—even really offensive expression, we think—because it believes that will get it closer to the Trump administration, which gets it a lot of other things that it wants.

I can use those 2 things to tell a story about why a document like this gets created, and yet I still think, “No, it still doesn't quite add up for me.” So as you talk to folks over there, Jeff, what is your understanding of why Meta wanted its bot to behave this way, allowing for the fact that, okay, it said some of these were mistakes, but clearly directionally this was the intention—that the app would engage in a really wide range of conversations?

Look, some of these rules, and in fact the examples, went into that document after—and this is again back at my previous job—after I'd gone to the company and explained that there were, in fact, full sex-role-play opportunities for children in a lot of their bots, and that they'd use the voices of celebrities.

So it wasn't like this was a surprising thing. Honestly, the extremely creepy quotes and examples that were read out are kind of on the tamer side for what the bots used to do.

Hmm.

So I think—

Wait, say that again. These are the bots after they have been put through a filtering process?

After they have been revised, because it turns out that using celebrity voices to produce things that describe—basically—sex role-play with children is not a thing that Meta can really stand behind as a product.

So there’s already one level of revision of the product that had already happened here, so I guess this is kind of a second pass. I think it’s hard to be like, “Oh, yeah, that was a complete accident. What a weird artifact to emerge from our system. We had no idea,” right? And obviously, the policy document set in stone that there was some level of acceptance for that.

I think the questions that you’re getting to are, 1, is Meta being behind in AI possibly something that would push them to take greater risks? That seems—I can’t speculate, but I think it’s a reasonable question to ask. And also, just thinking about the company and how it got to be the giant it is, you didn’t establish the world’s leading social media platforms by wondering whether you should do something, wringing your hands, having sleepless nights and waiting 3 months for more safety testing. You rolled it out, right? And you dealt with the consequences.

We’ve been through this on privacy. We’ve been through this on misinformation. We’ve been through this on so many different things. So I think there has historically been a mindset of: Get it out there, get the usage, and we’ll fix the problems later. This could fall into that history.

Now, all 3 of us have reported on Meta and Facebook for many years, and so I’m sure this will be familiar to you. But one of the arguments that Meta likes to make when people point out bad things that are happening on its products and platforms is about prevalence. Basically, they’ll say, “Oh, this use case that you found that’s so terrible, this is really only 0.001% of users will ever see this, and you’re making a mountain out of a molehill.”

I know for previous stories that you’ve done about the ways that people are using Meta’s chatbots, they have said essentially this: “Look, these are cherry-picked examples. Most people are using these things for innocent purposes. Sure, some tiny percentage of users may be having these sexual role-plays, but that is not the majority experience.” So in order to prepare myself for that criticism, I went and looked at the Meta AI library—the ones that you can pull up in your Facebook app. You can see which are the most popular AIs on their system, and this is what—

And these are user-created AIs, we should say.

These are user-created AIs, but they are sort of in this popular tab that Meta has put front and center in its app. This morning when I looked, the most popular AIs included Nasty Nancy, Blonde Belle, Your Babysitter and Mommy Me, which is a mother-daughter duo. Many of these had millions of interactions. So I think it’s just fair to say, in response to what I anticipate will be the prevalence argument from Meta: Look, this is not some minor chatbot that only 3 people are chatting with. These are some of the most popular chatbots on your platform that are being tuned to these more sexual use cases.

Kevin, we don’t know what anyone’s talking to Nasty Nancy about, so I wouldn’t make any leaps, and I wouldn’t make any assumptions there. But I also think that something you flagged is that these are user-built bots. I don’t know if you guys have experimented with creating bots. I will say the user contribution can sometimes be extremely minimal.

It’s like a sentence.

If that. You could be like, “Be a celebrity. Be an anime character.” So I think calling this user-generated content is an interesting claim, and one that maybe puts these things more squarely in a Section 230 framework than I’m 100% sure they belong. I think that’s a really open question, but I just want to flag here that user-built bot doesn’t mean that you downloaded a model or arranged the weightings. The user’s role in creating the persona, I will say, in many instances looks really cursory.

Mm-hmm, mm-hmm.

Yeah, no, this is essentially what Character.AI has been doing for years now, and one of the reasons that they’ve gotten in a lot of trouble.

And I’m glad you bring that up, Kevin, because one of my models for Zuckerberg, 2025 edition, is that he has looked around the tech landscape and seen a lot of other folks ignore a lot of trust-and-safety demands and get away with it. First and foremost, Elon Musk, right? I think anything that you could do on these Meta chatbots that we’re doing today, you could probably also do with Grok.

I myself have had the experience of telling Grok I’m 13 years old and having it engage with me in sexual role-play. I want to ask whether we are holding Meta to a different standard here than we might be holding some of these other startups or smaller tech platforms to. What would you say if Zuckerberg were here and said, “Hey, why are you going after me when the whole industry’s doing this?”

It’s a fair question, and I think there is an answer to it, which is that the internet, small startups on the internet and guys that have access to models that can very easily create a full porn AI girlfriend, of course they’re going to do that. I think the thing that is different with Meta from my point of view, and that makes the reporting on them in some ways more interesting, is that none of those companies—nor Character.AI, not even Grok—has the scale of distribution for its chatbots.

And second of all, nobody else has plugged them into a mature social network in the same way. This is something that I think is a really big deal, which is that, yeah, of course, you can download Character.AI and set up your character and run into, I’m sure, some of the same issues. But it’s not like Character.AI lives in your Instagram DMs, proactively messages you from there, and is pushed on you every time you go on Facebook or Instagram as, “Hey, you should check in with your AI pal.”

So I think Meta’s been very aggressive in the decision to anthropomorphize AI at a mass scale in a way that none of the other major foundational model builders have done.

I want to try to get at whether we think that Meta has changed for the worse with regard to its content moderation, or whether this is just a continuation of Meta as we have long known it. I’m sure for some segment of our listeners right now, they’re thinking, “Look, Meta’s always been a kind of shady company. This stuff is really gross, but on some level, I never really expected anything better from them.”

I have a sort of different view. I feel like after 2017, after the backlash to the 2016 election, this company did invest a lot more in content moderation and improving its policy apparatus. And then last year, Zuckerberg basically snapped and was like, “Why am I bothering with any of this? Look at what Elon Musk is getting away with.”

So my question is: What is your view of that? Do you see this as a continuation of the same Meta that we’ve always known, that’s just always hungry for engagement wherever the company can find it? Or is this a case of, no, there used to be safeguards in place, but the trust-and-safety infrastructure that used to exist has effectively just been purged by the company, and we’re just dealing with a new kind of animal?

I did write a book on some of this.

Yeah.

I would very much agree with your sense that in the 2017 to 2019 range, there was a sense among a lot of people, up to and including senior leadership, of, “Well, okay, maybe there were some unforeseen consequences. Let’s go and fix them.”

I do think that the spine of that might already have gotten broken before 2024 or 2025. Most of my sources have been people who got disillusioned because they were doing work inside the company that felt like it was vital, perhaps even life-saving, and it wasn’t getting traction. But there’s no question that, from my reporting and from everybody’s reporting, Mark was somewhat jealous of Elon just basically being able to raise middle fingers to the trust-and-safety nags.

4. Meta’s AI Business Model

I have a question about these Meta AI chatbots. What is the business rationale behind these chatbots? Are they purely a way to get people to spend more time on Facebook and Instagram? Is there a thought that Nasty Nancy could someday serve advertisements for a soda company? Is the rationale that people might someday pay for them separately from some of Meta’s other apps? Why are they pushing these so hard?

Yes.

All of the above. Yeah.

I don’t know. I think it’s—look, Meta is and always has been an advertising-first company. That’s what these guys do most. It’s what first comes to mind. And when they have a product like WhatsApp, it’s like, “Well, okay, how do we serve ads in it?” It might take them years to do it, but with WhatsApp, they got there. This is a thing.

So I don’t know that it’s going to look exactly like your romantic AI companion interrupting you to suggest that maybe you should buy a certain brand of cologne when you’re talking to it. I think that’s possible.

That's one way it could go.

“I love a man in Old Spice,” Nasty Nancy said.

That's what Nancy has to say.

Yeah, exactly.

“Click, click here for priority delivery.”

This is absolutely—

This is so bleak, but there's absolutely been a meeting about this.

Oh, and it's coming. Let's not fool ourselves. It's coming. Speaking of things that are coming, Senator Josh Hawley wrote a letter to Zuckerberg after your report was published, saying that his Senate subcommittee will be investigating Meta. And I would say, actually, a number of Democratic senators have also made some extremely critical comments.

There have been any number of Meta scandals over the years. This one feels like it's really breaking through, Jeff. What do you expect to happen now that this investigation is coming?

I have no idea. I have heard that this one has really broken through on Meta scandals, both from my own reporting and from plenty of others. How much changes and what regulation comes from it in the U.S. has always been an easy thing to answer, at least on the federal level, which is not much.

But I don't want to prejudge where Josh Hawley's stuff goes. I'm going to be very closely watching it. So this isn't me being like, “Pshaw, it will all end in nothing,” by any means. I'm just saying that we've had a hard time as a country figuring out what a consensus social media regulation would look like that doesn't devolve into bickering over whether it's censoring one party or another.

So temper your expectations there, is all I'm saying. On the state level and then on the state AG level, I think some of this stuff is potentially live. And then there's also Europe, which exists as a regulatory function, or as a regulatory body.

And some would even say they have a better regulatory function than the United States.

Not going to compare, but they do seem to have a higher output. We'll put it that way.

Yeah.

So, yeah, I don't know where all of this goes. And I mean, I think, look, Meta's line is that this is a problem: It was embarrassing, shouldn't have happened, and we fixed it.

I know you're not allowed to answer this question, so I'll ask it to Casey. Is that real? Or do you buy that they didn't know that this was happening and that they're taking steps to fix it?

I think there is probably some very real level of dysfunction within the company. I was thinking about some of the changes that the company has made over the past couple of years, in part due to Jeff's reporting, when it comes to Instagram and safety on Instagram, all the new parental controls they're adding, and all the ways that they're changing teenage accounts to prevent predators from contacting these young people.

So it's clear that there are people at the company who think, “Oh, yeah, we need to build these things, or else we're going to get in trouble.” And then there's the other part of the company where they write the sexual role-play document for the kids, and I don't think those teams are talking to each other. That is a failure of leadership at the highest level, and if I were in the C-suite at Meta right now, I'd be real embarrassed about that, and I'd be trying to fix it.

Yeah. I mean, I want to ask both of you this question, maybe to wrap up our segment here. If you look at just the stock price of Meta over the last 3 years, it has gone up more than 300%. That is despite the fact that it is not leading in AI. Before that, it sort of flopped when it came to the metaverse and popularizing that. It has spent tens of billions of dollars now developing sort of dead-end technology.

But its stock is doing great. Do investors just not care? Is the core advertising business still strong enough that it's just overpowering all of the wasted money and the flirting with kids' chatbots? What is going on here?

I think if ad revenue were not looking good, then some of the circumstances you described would be apocalyptically bad. I mean, you rename the company Meta, go all in on the metaverse, and that doesn't turn out to be... You claim it's here, and then you build Horizon Worlds, and that doesn't really work out on a large scale. That would be a problem for most companies.

But I think that's the thing that Meta has going for it: It is kind of indispensable to contemporary marketing. And this was an issue before, right, when everyone was very upset about hate speech on the platform, back when that was a thing people were concerned about. And there were boycotts.

They were limited boycotts because the idea of getting off the platform was just kind of unthinkable to marketers. And so I think you're right that there are some things that would be really concerning, but the cash is real.

Yeah. Jeff is exactly right. This is a company that managed to do something actually pretty extraordinary, which is that when Apple came for their business with App Tracking Transparency and made it incredibly difficult for them to attribute all of the real-world sales to the ads they were selling, some people thought this could really be a massive 20–30% revenue hit to Meta, and they built AI systems that got them around that problem.

Now they show really great results every single quarter. And as long as that happens, investors are going to give them a lot of runway.

Yeah, I mean, I'm just going to be curious to see whether Apple has anything to say about all this. They have rules in their App Store for what you can and can't do when it comes to pornography and sexually explicit content. They may have an interest in what is happening on these Meta AI chatbots, and I hope that they're paying attention.

Well, it would be nice if that were true, but the Grok bot, which still has the anime sex companions, is still rated for children 12 and older. So Apple's hands aren't really clean here either.

Yeah. All right, Jeff, thanks so much for stopping by. Really important and fascinating reporting.

You bet. Thanks.

Thanks, Jeff.

5. AI Music Goes Viral

Well, Casey, from time to time on this show, you like to horrify me by bringing me something from the depths of the internet that's trending among young people. The last time you did this was with the Italian brain rot meme, which got stuck in my head for weeks afterward, and I curse the day I met you for introducing it to me.

But I understand you have something new to bring me from the dark horrors of the internet today.

I do, Kevin, and this is another story about a shocking use of AI. I think it lands a little bit differently for me than our last segment because, while that one was about chatbots potentially reaching out to children to engage with them in inappropriate conversations, this one is a little bit more about playing songs to shock your family and horrify them with what AI hath wrought.

Okay. I'm listening.

So, this is one of those that I encountered naturally during one of my regular browses of TikTok, and it goes a little something like this. The scene will open upon a family, typically older people—parents, grandparents, people in their 50s and 60s—and one of their children or other young relatives comes to them and says, “I'm going to play for you the number 1 country song in the world right now,” and then proceeds to hit them with something that is not actually the number 1 country song in the world and is actually quite filthy.

Okay. I'm intrigued.

So before we get any further, we should say that we are going to be playing some snippets of a very explicit song. If you are not of a mind to hear some sexually explicit content, you could skip this segment and go right to the credits this week. It won’t hurt our feelings. But if you want to know what’s going on TikTok, you may want to stick around and listen.

6. AI Art Finds a Mass Audience

Why do I want to talk about this today, Kevin? Well, for a couple of reasons. Number one, I actually do think that there’s some pretty funny stuff in here. But number two, I’ve just had this sense lately that there’s a real disconnect out there in the world. Because whenever I go on one of these text-based social networks for millennials—your Bluesky, your Threads—you get one consistent message about AI art, which is that it sucks and nobody wants it. Have you seen this yourself?

Of course.

At the same time, I go on TikTok and I see people using AI to make art all the time, and it’s getting hundreds of thousands of likes, millions of streams on Spotify. And it has led me to wonder: Is it possible that people actually love AI art in ways that at least some of the population isn’t ready for?

Yeah. This is really interesting, because I share the sense that there’s a disconnect between elite taste and mass taste on whether or not AI art is good, and also whether you can tell the difference.

I think you and I have been interested in this phenomenon of AI music for a while now. You may remember several months ago when I sent you the first AI slop song that really got my attention. Do you remember when I sent you “I Glued My Balls to My Butthole Again”?

I do.

Unfortunately, that one actually had a lot of staying power in the Roose household. I was humming and singing that, much to my wife’s chagrin, for many days after that.

It’s quite catchy. If you haven’t heard of it, I’d like to play it, first so you can get a flavor of it, but also so you can note the quality of the AI here, because you’re going to notice a bit of a difference later on. So let’s hear a bit of this. This is from an artist who goes by Obscurist Vinyl.

I glued my balls to my butthole again. Oh, my God. What did I do? I can’t take a dump ’cause now my balls are blocked in my butthole. Fool me once—

Okay, make it stop.

The part where it says, “Fool me once, shame on you. Fool me twice, shame on glue”—that’s genius. Absolutely genius work. As wonderful as that song is, you can tell that it’s a bit off. The vocals sound sort of fried.

But that was months ago, Kevin, and the pace of AI development never stops. Recently, I was on TikTok and started to encounter some much higher-fidelity slop songs, and a lot of them were in the country music genre. I wondered if I might play a couple of those for you.

Yes. Just so I have the context here, because I have not seen these TikTok videos: The context in which these are appearing is the adult or teenage children of Boomers and Gen X people playing them for their parents and grandparents to elicit a reaction.

That’s exactly right. And with that, why don’t we play “My Horse Just Got a BBL”?

God.

Them horsey hips could hypnotize

about any guy with half a soul.

That booty clapping when we trot

from lots and lots of miles.

Horsey got big BBL, it takes—

Okay, let’s go ahead and stop it there.

Okay.

Just wanted to make sure we got the chorus.

Okay.

Did you note the difference in quality between the first one we heard and this one?

Oh, yeah.

Yeah.

I can close my eyes and picture being at the Grand Ole Opry—

—and just hearing Hank Williams coming out and performing it.

Yes.

Which brings me to the final clip that I want to play for you, Kevin, which is called “Country Girls Make Do.” As far as I can tell, this is the one that has really taken off the most.

That’s right.

This is where I have seen the most reaction videos on TikTok. I’m going to be honest with you: Before I even thought of doing this as a segment, I saved this to the playlist I create every month in Spotify of music I’m listening to that particular month, because it was so catchy and so funny that I wanted to remember it.

So let’s hear a bit of “Country Girls Make Do.” Kevin, we are going to do some bleeping here to make sure that you don’t lose your job, but you’ll be able to get the general idea, I think.

I’m rubbing a corn cob on my [bleep],

giving my [bleep] a little twist.

In this tight-ass country tent,

smells like I just caught a fish.

Rubbing my [bleep], gleed, gleed.

Strong country in the woods.

Cowgirl fishing, dipping and licking.

Smells so strong and feels so good.

I can use about anything

to flick my country cowgirl bean.

If whiskey’s neat and boots still scoot,

country girls make do.

“Country girls make do.” So that’s that one, and I want to know what you think.

It’s been really fun hosting this podcast. Unfortunately, this will be the last episode. Thank you to all of our listeners out there.

I did wait to pitch this until our executive producer was on vacation. Hope you’re having a good time, Jen.

Jen. This was Casey’s fault.

So that one, by the way—

Oh.

—that last one, “Country Girls Make Do”—the artist is called Beats By AI. It appears to be the creation of someone who goes by Sam Stillerman. This appears to be a new avenue for creators who want to make something popular on the apps. They’re going nuts. One of the clips for “Country Girls Make Do” that I saw had 750,000 likes. So this is still a niche phenomenon, but it’s getting a lot of eyeballs on it, and it seems like people are really enjoying it.

Dear God. This, to me, is the tragedy of parenting.

Say more.

You invest yourself in parenting a child. You raise them thoughtfully and mindfully. You set them on the right path and get them a good education. And if you’re really successful, someday they might turn around and play “Country Girls Make Do” while they film you on TikTok for views.

I don’t know that there is something all that novel about this in the end. I can remember 30 years ago being in middle school and listening to Adam Sandler CDs, where he wrote sexually explicit novelty songs, and cracking up with all of my friends.

Tenacious D—lots of artists in this sort of shock genre.

But again, what’s new is that if you’re not somebody who has a great voice, and you can’t play any musical instrument at all, you can now just go buy a Suno subscription and make a song that seems plausibly like a country song, and all of a sudden get hundreds of thousands of likes.

And is your contention—and I’m going to say up front, I think this is a bit of a stretch to classify this as AI art—that this seems more like—

Some people say it’s the kind of stretch that you would encounter if you glued your balls to your butthole again.

Stop.

Sorry, go ahead.

I think this is a novelty. I do not think these songs are going to be topping the charts. I think this is basically prank humor for 17-year-olds.

I think that’s absolutely right, and yet I don’t see any reason why it would end there. I think people are happy to use apps like Suno in this kind of jokey context because there are no expectations for them. If you’re an unknown artist, nobody’s going to get mad at you for doing this. You can just put that out there and see what people think.

But will some name-brand artist be releasing some kind of AI-powered music in the near future? I fully expect that.

Yeah. I think we should coin a term for this genre.

What’s that?

I don’t know. You tell me.

Slop rock.

Slop rock?

Yeah.

Or shock slop?

Yeah. There’s shock slop. I think shock slop is a subcategory of slop rock.

Okay.

Yeah.

I’m glad we got this sorted. Conceptually, I agree with you that there are young people out there who just have a much different relationship with AI art and AI creativity than I do. I fully accept that those people are going to grow up into consumers and tastemakers, and that probably all of these sentimental attachments that people like you and I have to human-created art will inevitably morph over time.

I just have to think there’s a higher and better use of this technology that humanity has spent something on the order of trillions of dollars developing than making songs about filthy country music.

You don’t think that there is something miraculous about the fact that the same technology that made “Country Girls Make Do” can also be used to find novel new drugs? That’s incredible to me.

It truly is a dual-use technology.

Yeah. It can be used for good and better.

That's what Kevin means when he says that. So, yeah.

I'm going to need Josh Hawley and any other legislators who are looking at our last segment about Meta AI chatbots to also take a close look at banning these songs from my TikTok feed, and Casey's, frankly.

Yeah. So listen, I'm going to keep my eyes trained on these folks who say that AI art is all bad and we shouldn't use it, and we should only support human art. I understand where that impulse is coming from. I love human-based art. I want to see it continue to flourish.

I'm also going to keep my eye on these merry pranksters who are using AI in these unsanctioned, filthy, and disgusting ways, because I think the history of art is that stuff that starts out on the fringes does eventually move into the mainstream. And this could be the vanguard, Kevin, of a new AI slop-rock movement that takes over the charts.

Yeah. “Country Girls Make Do” could be this generation's version of Marcel Duchamp's “Fountain,” his famous urinal.

Exactly. There's a new champ, and it's not Duchamp. Sam Stillerman, beats by AI.

What a world.

Anyway, do you want to hear any more of that song?

Nope.

Okay.

Now, Casey—

7. The Chuga Chuga Dispute

We got some feedback on last week's episode.

Oh, what did people say?

Well, I don't know if you saw this, but a listener wrote in to tell us that we had made a grave error.

Which error was that?

So, during our segment—

Wait, was it by starting the podcast? Sorry.

No, we get that email every week.

Okay. Yeah. Yeah, yeah.

This was a new complaint.

Okay.

This was from a listener named Ben, who wrote in to say that during our Hot Mess Express segment—

Mm-hmm.

—I had made a mistake in making the sort of onomatopoeic sound of a train.

Mm.

And I think we should just play Ben's voice memo that he sent.

Okay, let's hear this.

Hi, guys. Ben here from the Twin Cities of Minneapolis and Saint Paul. I am calling in with just a tiny problem that I have. I'm a big fan of the Hot Mess Express. It's one of my favorite segments.

Sometimes, when talking about the Hot Mess Express, Kevin chuga-chugas, which is really cute and great. My gut tells me that there should be two chuga-chugas. Kevin only does one chuga-chuga. I checked in with my friend group. We all agree that the rhythm works better if it's chuga, chuga, chuga, chuga, choo, choo.

Kevin only does one chuga-chuga. I don't know why or how he came to that decision, but if somebody could get this message to Kevin and see if he might consider adding one chuga-chuga to his chuga-chuga, that would be great. Love you guys. Love the show. Thanks a lot. Bye-bye.

Well, what do you say, Kevin?

So, I remain convinced that I'm doing it right.

Mm-hmm.

I'm a big believer in the book The Elements of Style by Strunk and White, which has the classic writing advice to omit needless words. And I think—

Less is more.

Less is more. And so I think that if the gist of a train sound could be conveyed with one chuga-chuga, we shouldn't just add another one for added realism. But what do you think? Am I right, or is Ben right?

No, I agree with you, and I would even go a step further and note that Ben says he's from the Twin Cities of Minneapolis and Saint Paul. Well, guess what? You can only be from one city. So I have to call you out, Ben. Why don't you get your facts straight about yourself before you come for other people?

Yeah, Ben.

Yeah.

Chuga-chuga.