California Regulates A.I. Companions + OpenAI Investigates Its Critics + The Hard Fork Review of Slop

Casey, I heard the big news this week in tech is that Waymo is going to London.

I saw that, and I thought British people are going to hate it.

Yeah. I had a different question: How are they going to teach it to drive on the other side of the road?

That's a very—

Just—

Good question.

Just switch the software? Go on the other side now.

Yeah.

Everything you used to do, do it in reverse.

That's like the autonomous vehicle equivalent of dark mode, when you have to drive on the other side of the road. It's not available at launch, but eventually they bring it out.

Do you think they have to put the steering wheels that don't do anything on the other side of the car, too?

Presumably.

I'm Kevin Roose, a tech columnist for The New York Times.

I'm Casey Newton from Platformer.

And this is Hard Fork.

Well, Casey, it's been a big week for tech regulation in the state of California.

That's right, Kevin. Everywhere I look, it's bills, bills, bills. I'm like, what is this, a Destiny's Child song?

Very topical '90s reference there.

Listen, a lot of our listeners are seniors, and they're going to really appreciate that one.

1. California Takes Aim At Tech

So on Monday of this week, Governor Gavin Newsom of California signed into law a bunch of new tech-related bills that had been making their way through the state legislature in California, and we're going to talk about them today. And if you're not a listener of ours who lives in the state of California, you may be asking, why are you devoting an entire segment to tech regulation in California? And Casey, what is our response to that?

Well, Kevin, I think you and I both believe that while AI has the potential to do some good, it's also clearly causing some harm. And right now, the AI companies are operating with very minimal regulations on what they do, and that's just been a growing source of concern. We have talked over the past year on this show about teenagers who have died by suicide after having very tragic interactions with chatbots, and I think there has been a growing cry for some kind of guardrails to be placed around these companies. So that is what we are talking about today: a state that had some ideas, actually managed to pass the laws, and is putting them into practice and will hopefully rein some of these companies in.

Yeah. California is a uniquely important state in tech regulation for a couple of reasons. One of them is that a lot of the companies are based here. They care a lot about how California regulates them, and the laws that are passed in California tend to ripple out to the rest of the country and the rest of the world. They tend to become kind of de facto national standards. And especially at this moment, when our federal government is shut down, and even when they're operating, they don't seem to be interested in passing any tech regulations, this is what we have: state-level regulation as sort of standing in for the federal regulation that doesn't exist.

Yeah. So today, let's talk about some of these bills that got passed and what we think they tell us about what some common-sense approaches to regulating AI might look like.

2. SB 243 Targets Chatbot Harm

Okay, so let's start with what I think may be the most important bill that has come out of this flurry of legislation, which is SB 243. Casey, what is SB 243, and what does it do?

What SB 243 does is require developers to identify and address situations where users are expressing thoughts of self-harm. So they have to have a protocol for what they're going to do if they see somebody express these thoughts. They have to share that protocol with California's Department of Public Health, and they have to share statistics about how often they are directing their users to resources. And then, starting in 2027, Kevin, the Department of Public Health has to publish this data. So that's a little bit longer than I would like to start getting this data, but my hope is that when that begins, we will have a very large and useful set of public health data about the actual effects of chatbots on the population of California. So if you're somebody like me who's really interested—or worried—about what it is going to do to our society and our culture once so many people are chatting with these bots every day, this is a really big step toward understanding that.

Yeah, I think this is a good one for us to drill down on because it is a place where I think there is a lot of attention and momentum around regulating. OpenAI has recently rolled out some parental controls. Character.AI, which we've also talked about on the show, now has a disclaimer on its chatbots and some additional guardrails for minors. So I think the platforms were starting to comply with these kinds of laws in advance of them actually becoming laws, but this will at least give them some formal requirements.

Yeah. And we should mention a couple more of those requirements. In California, chatbots will now have to tell you that their output is AI-generated. Of course, our savvy listeners probably already know that, but there may be some people who are chatting with ChatGPT and aren't entirely sure what's going on. This bill does have a few additional protections for minors, including that chatbots cannot produce sexually explicit images for them. And it's going to remind minors to take breaks if they have been chatting with ChatGPT for a really long time. So interestingly, there was another bill that California legislators passed which would have, I think, potentially banned ChatGPT use for minors. And Gavin Newsom vetoed that. He was like, “That's going too far.” But this is kind of like one step back, and I do think it adds some meaningful protections, and no longer do we have to rely on the goodwill of OpenAI or Character.AI to implement these things. Now it's just in the law, and it says you actually have to do this.

Now, does this law apply to all of the AI platforms or just the really big ones with hundreds of millions of users?

So according to a legislative analysis, it will apply to basically any chatbot that can be used as a companion.

Hmm.

And initially I didn't know—would that include ChatGPT? Most people, I think, don't really think of ChatGPT as a companion, but according to this legislative analysis, yes. Look, if you're talking to it for 3 hours a day, it's some kind of a companion to you.

Yeah, I think this is a case where the industry kind of understood that something was going to be done about chatbot companions in the arena of state regulation, and they had this other proposal that they thought was too strict and stringent. And so they kind of accepted the lesser of 2 evils and sort of got behind, begrudgingly, this bill that actually did end up being signed into law.

That's right. And can I talk about why I think this is important, Kevin?

Yes.

3. OpenAI Relaxes Its Guardrails

Okay. So just on Tuesday, we get this really interesting tweet from OpenAI CEO Sam Altman. This tweet gets a lot of attention because it says at the end that in December they're going to allow what they call verified adults to start using ChatGPT to generate erotica. Let's set that aside for a second. Here's what Sam says at the beginning of this long tweet. He said, “We made ChatGPT pretty restrictive to make sure we were being careful with mental health issues. We realized this made it less useful/enjoyable to many users who had no mental health problems, but given the seriousness of the issue, we wanted to get this right. Now that we've been able to mitigate the serious mental health issues and have new tools, we are going to be able to safely relax the restrictions in most cases.” And then he says that what people liked about GPT-4o, which was the model that they got in trouble over because it was so sycophantic, and it encouraged people who were telling it things like, “I'm not taking my medication anymore,” or, “I think I'm God,” they said they're going to bring whatever people liked about that model back to ChatGPT. How does this connect to the California bill? I'm not sure that OpenAI has mitigated the serious mental health issues that came up this year.

I'm also not sure of this.

It's been 2 weeks since they rolled out parental controls in ChatGPT. Do we really have enough data to say this one is under control? I have to say, Kevin, I was actually pretty shocked by this tweet—not for the erotica stuff, but for the GPT-4o stuff and the suggestion that we have a handle on how to guide these chatbots so that they don't hurt people. What did you think when you saw this?

I thought they are really trying to drive up usage numbers. They must be seeing something that is suggesting to them that people were engaging more with ChatGPT when it was more like a companion, when it was telling people more flattering and sycophantic things. And I suspect that that is part of the reason that they are trying to sort of get that mode back. Now, I think there's actual logic there. A lot of users do want something that's going to tell them they're great, but we do not know how they have solved these safety challenges that they have supposedly solved or these mitigations for mental health issues.

It’s also not clear to me that things are just as simple and binary as: some users have mental health problems and some don’t. For the ones who don’t, we’re going to give them the kind of unhinged, unfiltered chatbot experience, and for the ones who do, we’ll guide them onto a set of guardrails. These things exist on a spectrum, and it may not be obvious to OpenAI or even to the users themselves when people are starting to develop mental health issues as a result of talking to these chatbots.

I find it so confusing because, on one hand, this is a company that will put up a blog post that says, “We do not want to optimize for engagement. We want you to use this thing, have it help you, and let you get on with your life.” And then they release Sora, the infinite slop feed, and say, “We’re bringing back the most sycophantic AI personality in the history of the company because we know that some of you out there might need a friend.”

It really feels like there are 2 wolves inside of OpenAI right now, and I think it just makes necessary some common-sense legislation that starts to put some guardrails around this and signals to these companies that they cannot just do whatever they want.

Yeah, I mean, I guess we’ll have to see how the AI companies comply with these laws. I do not think that this law is going to dissuade them from trying to build the chatbot companions because that’s obviously a lucrative industry for them. But I hope it does make them pay more attention to things like safety and mental health, especially for younger and more vulnerable users.

Yeah.

Okay.

4. California Expands Its Tech Rulebook

At this point, our listeners are thinking, “Kevin, you told us that there were actually a lot of bills that California passed—and I’m desperate to know what they are.”

Well, Casey, buckle up. I’m going to run through a few of the other bills here. We won’t talk about all of them in that much detail, but we have AB 621, which provides stronger protections against deepfake porn. This law will make it possible for victims of nonconsensual deepfake porn to sue platforms that facilitate the creation of that porn for up to $250,000 per violation.

Yeah, and this is really important because a trend that we haven’t talked much about on the show this year, Kevin, is that there are these really sketchy companies that make what are called nudify apps. They’ve been advertising themselves all over Facebook and Instagram somehow, and people have been using them to generate these deepfakes. Now there is a law on the books that says, “Hey, we can actually come after the companies themselves.” I think that’s obviously a good thing.

The next bill I want to talk about is AB 853. This is the California AI Transparency Act, and this caught my attention because it essentially requires AI companies to build tools into their systems to detect whether content—images, video, or audio—is in fact AI-generated. Basically, they have to offer users a way to put in an image and say, “Hey, did you generate this image?” and get a reliable response back.

Yeah. Here’s what this means: If you come to California and see a video of dogs playing poker, no longer will you have to wonder, “Are those dogs really playing poker?”

It’s true.

There will be a watermark, and you will get the answer to your question.

That’s true.

About damn time.

Then there was AB 56, which was about warning labels for minor users of social media platforms about the potential mental health risks associated with using the apps. Casey, this one’s pretty wild.

Yeah, it’s certainly very intrusive. The law dictates how much of the screen this warning has to cover and how often it has to appear. It appears right when the person starts using the app and then again after 3 hours, which is sort of funny to me. It’s like, “You can have your 3 hours, but then we’re gonna—

Remind you—

—might be cooking your brain.”

We should say that it’s not going to be a small little thing on the screen. The law says that after 3 hours of use, platforms must display a 30-second, non-bypassable warning covering at least 75% of the screen.

Mm-hmm.

If you are a 16-year-old and you are scrolling TikTok or Instagram for more than 3 hours, you are going to get a giant cigarette warning, essentially—

Mm-hmm.

—on your screen. You cannot skip it or get it off your screen for 30 seconds, and that is going to happen again after every additional hour. So I predict there will be a lot of teens finding clever ways around this because teens do not like to wait for their TikToks.

I’m just curious: Will teenagers see this and think, “Oh, my God, I have to get off TikTok,” or will they think, “Damn, I am such a badass for using this crazy dangerous app”? I could see it going that way, too.

Yeah. What a rebel. Meet me behind the school. We’re watching TikToks.

Semi-related, there was a study that came out this week. It was a pretty big study of 6,000 children who were under 13. Did you see this?

No. What did it say?

They tracked their use of social media and found that more time per day spent on it was associated with being worse at reading. So my question is: Should this warning say, “Hey, kids, be careful. This app could be bad for your mental health,” or should it say, “You are actively becoming worse at reading than everyone in your class”? I actually think that might be more effective.

Actually, in order to bypass the warning label, you should have to take a reading-comprehension quiz based on a short story by Ernest Hemingway.

Mm-hmm.

Just say, “No more TikToks for you—

Yes.

—until you can tell me what The Old Man and the Sea was about.”

I’m still trying to find that one out.

5. Parents Control App Age Checks

All right, next bill. This one is one I wanted to get your take on.

Yeah.

This is AB 1043. This is about age verification, a subject we have talked about on this show before. This bill would require Apple and Google, which make the 2 most popular mobile operating systems, to verify users’ ages in their app stores. Casey, explain what this bill does and whether it’s a big deal or not.

Yeah, so there are a bunch of different approaches to what they call age assurance in the business. The reason this one is notable to me is that California actually took the approach I favor: When someone is setting up a device for their child, the parent inputs the child’s age, and that information is then passed along to the App Store and to developers.

The thing that is great about that is that it seems like the most privacy-protecting of all the age-assurance protocols that we’ve seen, right? Other states want you to potentially upload a driver’s license, right? You’re providing a lot of really personal data. Some of that is being held by third parties. All that stuff is subject to data breaches and who knows what else.

In California, it’s just like, “Hey, you’re the parent. You’re the guardian. You tell us how old your kid is, and we will make sure that they don’t download an app that they’re not supposed to have.”

Right. So instead of what’s happening today, which is that every app asks you to say how old you are when you sign up and create an account, and it just kind of works on the honor system, this would essentially force Apple and Google to, when you’re getting a new iPhone or a new Android phone and your parents are helping you set it up, put in your birthday.

As a 16-year-old, it shows, “Okay, I am a minor. I am 16 years old,” and then your phone will pass that information to every app that is trying to be installed on that phone. Is that—

Exactly.

—more or less correct?

Exactly.

Now, you said you favored this solution. Are you taking credit for this bill?

No, I’m not taking credit for it. It also wasn’t my idea. Other smart people have been talking about this for a while, but I’ve written about it in the past—

Yeah.

—and this is what I said I thought we should see happen. Every once in a while in a democracy, you get to see something you actually wanted. It’s a lovely thing when that happens.

Every once in a while—

So cherish it.

—you have a good idea.

Yeah. Cherish it. Cherish the moment.

6. Frontier AI Rules Stay Toothless

All right. One more bill we should talk about because this is the one that has actually gotten the most attention and a lot of the lobbying dollars. This is SB 53. This was actually signed into law last month. This is the Transparency in Frontier Artificial Intelligence Act.

This is the successor bill to SB 1047, which we’ve talked about on this show before. That bill was vetoed by Governor Newsom last year. This new bill is essentially a watered-down version of that bill. It establishes some basic transparency requirements for the biggest AI companies, what they call large frontier developers.

It requires them to publish information about their safety standards and create a new mechanism to report potential critical safety incidents to the California state government.

It also establishes some whistleblower protections for people inside the companies who may want to disclose some significant risks posed by their models. And this one did pass and was signed into law. Casey, do you think this is a big deal?

I think it's great that we have some transparency requirements. I think that it's great that we have some whistleblower protections. When I think about the things regarding AI development that concern me the most, this bill does not speak to them. But I feel like the main reaction that I read to this bill is a bunch of people saying, “Yeah, this couldn't hurt.”

Right.

That's how this feels. Yeah, it's fine.

Right.

Yeah.

It feels pretty toothless to me, and it also is basically codifying something that a lot of the companies are already doing anyway. I think all or nearly all of the large frontier developers already publish things that would put them into compliance with this law. So I like the idea of not just relying on voluntary self-regulation, but this seems like a pretty weak bill that was weak enough that most of the AI industry didn't feel like it was worth opposing, although there were industry groups that lobbied against it. But I think for the most part, they said, “Okay, well, this is better than the one we tried to kill last time.”

Yeah.

Okay, so that is a bunch of information about these California state AI laws and social media laws. When we step back and zoom out here, does this give you any thoughts about how AI regulation and tech regulation in general is going?

I think in some ways it's going better than I expected, Kevin. We covered the past decade of lawmakers twiddling their thumbs, wondering how social media ought to be regulated. It took too long. Some of those efforts have finally gotten off the ground at the state level, but after a lot of harm was done.

In the case of AI, we are earlier in that kind of epoch of tech, but already we've seen California and other states make some pretty decisive moves to build some guardrails, create some transparency requirements, and I think that's a really good thing. We're going to have to see how effective these things are, but I just want to say we need something like this. It is important that this week OpenAI came out and said, despite everything that has happened this year with their chatbots and mental health, they are going to hit the accelerator on making them more personable, more sexual, and more powerful. That will continue to have reverberations, and we need state lawmakers paying attention to that.

We need federal lawmakers paying attention to it too.

Well, be realistic. I can't talk to you when you're being hysterical.

What this makes me feel like is, God, I wish we had a Congress that could do something about this.

Yeah.

I really am sympathetic to the AI companies on this one point. I do not think that state-level regulation is the best way to do this.

Mm-hmm.

I do not think it is good or efficient to have 50 individual states all coming up with their own bills and trying to pass them, and then have the AI companies have to look at all 50 states and decide how they're going to build systems that comply with all of those. That does not feel like a good solution to me. But for that not to be the default path here, we're actually going to need Congress to step in and do something at the federal level, and right now our government is shut down, so I don't have high hopes.

But I think that in the absence of Congress getting its act together and deciding to do something federally, what we're going to end up with is a bunch of states doing what California has done here and just trying their best to get some rules on the books while they can.

Yeah. I agree with that. I would add that Senator Josh Hawley is currently circulating a draft bill that would ban AI companions for minors. Who knows how far that will make it through? But I do think that there are a significant number of members of Congress who would like to see something like this happen. The question, of course, as ever, is whether they can get something across the finish line.

Yeah. All right, Casey, that is what's happening in California. When we come back, we'll talk about how this legislative fight got personal for one AI lawyer.

7. OpenAI Takes Its Critics To Court

Well, Casey, there's another big story involving the law and AI this week that we wanted to chat about, and it involves this behind-the-scenes beef that has been going on between OpenAI and some of its biggest critics.

Yeah. There are really 2 big legal battles that this story is at the intersection of.

One is the battle over OpenAI trying to convert itself into a for-profit enterprise. Right now, OpenAI is famously a nonprofit. This has created many issues for the company over the past several years. They want to be a more normal money-making enterprise, and this is opposed by lots of people. Some of the people that oppose it are OpenAI's direct competitors, including Elon Musk and Mark Zuckerberg, and OpenAI has been pretty aggressive in going after groups that they believe might be connected to those 2.

The second battle is about SB 53, the bill that we just talked about. It was just signed into law by California Governor Gavin Newsom, and it establishes some basic transparency requirements and whistleblower protections for people who work in AI labs. There were a lot of groups that lobbied both for and against this one. Encode was one of the groups that lobbied for it. And so those are the 2 big legal battles that were happening next to each other. But today's story, Kevin, takes place right in between both of them.

Yes. So our guest today, Nathan Calvin, is the vice president of state affairs and general counsel at Encode. They are a small AI policy nonprofit. They were started several years ago by a high school student. Fun fact.

Hmm.

What were you doing in high school? I wasn't starting AI safety nonprofits.

Debate team, crushing it. Anyway.

Anyway, they have become one of these groups that is submitting briefs and lobbying lawmakers on a lot of these AI-related bills and efforts. They have also been very vocally opposed to the restructuring of OpenAI as a for-profit. And what seemed to happen here in Nathan's telling was that one night as this legislative process was ongoing, a sheriff's deputy showed up at his house and delivered a subpoena from OpenAI, demanding that he produce all kinds of personal communications, including anything related to not just the restructuring, but SB 53, this bill that they had been advocating on behalf of.

Yeah. This surprised folks because they do still identify as this mission-driven company that's trying to create AI to benefit all of humanity. I think it's generally understood that during these legal battles, there are going to be people who lobby for and against, and that's just part of the process. But now one of those people who was doing the lobbying on behalf of his nonprofit finds himself with a legal battle of his own, and that got a lot of folks talking, including some people who worked at OpenAI who criticized their own employer for its behavior. So that seemed like something that it would be worth understanding more about, Kevin.

Yes. This has been a subject of hot debate and conversation within OpenAI, as well as around the broader AI industry, and we wanted to talk to Nathan about his experience.

But before we do that, since this is, after all, a story involving AI and legal battles, I should note that my employer, The New York Times, is engaged in its own legal battle. They are suing OpenAI and Microsoft over alleged copyright violations.

And my boyfriend works at Anthropic, but so far we've managed to avoid any legal battles. We're counting our blessings on that one.

Well, check the mail when you get home.

Oh, no.

All right, let's bring in Nathan Calvin. Nathan Calvin, welcome to Hard Fork.

Pleasure to be here. Many-time listener, first-time caller. I think I said that wrong, but anyway, very glad to be here.

So just to set the scene for our listeners, you're in Washington, D.C. It's a Tuesday night. I'm assuming it was a normal weekday. You and your wife were sitting down to dinner, and then you got a knock on your door. Tell the story from there.

8. Nathan Calvin Faces A Subpoena

When I opened the door, there was a sheriff's deputy who was there to serve me a subpoena from OpenAI, asking for different communications and documents related to a piece of AI safety legislation I was working on, as well as about our criticism of OpenAI's restructuring to a for-profit.

One thing I will say just in terms of the timeline, because it's come up in some of the back-and-forth, is that on Saturday previously, I had gotten a call while I was visiting my mom and nephew saying that someone was trying to get into my apartment to serve me papers, and I said, “I'm not there right now.” Anyway, they finally did come on Tuesday.

I think Jason Kwon, the chief strategy officer at OpenAI, said in his comment that I should have known this was coming. I did know they were trying to serve me, but I didn't know about any of the details, and I didn't know they would be coming that exact night.

Now I want to get to all of that, but first, I have not been served with a subpoena.

Casey’s been arrested many times, so he’s familiar—

Yeah.

—with how these—

But never convicted.

Yeah. He’s familiar—

Very nice.

—with how these things go down. But are they literally handing you a packet of paper, like in the movies? Or what does it look like to be served with a subpoena from OpenAI?

Yeah. It is just a stack of papers. I did not know—again, I am a lawyer—but I didn’t know that sheriff’s deputies are the ones who, at least some of the time, serve subpoenas in D.C. I later learned that that’s not incredibly unusual, but it certainly was surprising from my perspective.

To be clear, the guy was perfectly nice. After I had heard on Saturday that someone was trying to serve me papers, by the time it actually happened and they were at my door, there was a little bit of, “Okay, now I can figure out what is actually happening.” The days between hearing that it was coming and it actually happening were honestly some of the most stressful, and it was like, “Okay, now I can figure out at least what we’re dealing with and how to respond.”

When you got into AI advocacy, was this on your radar as something that was likely to happen, that people would be saying, “Okay, you gotta show us all the emails you’ve been sending about this”?

No. I do feel like my mom worked for the American Academy of Pediatrics for 25 years and was involved in litigation against tobacco companies, and they came and took all of her papers out of her office at some point. She had told me, “Never write any emails you’re not comfortable with having read back to you later,” or something. And so—

Oh, wow.

—you know, I had some—

So, wait, you’re actually way better—

Yeah.

—prepared for this than the average person.

Yeah.

Yeah, it runs in the family.

I think that’s fair. Yes, indeed.

Did you understand immediately why OpenAI was subpoenaing you? What was your initial response when you started reading these papers and understanding what they were after?

Yeah. In some ways, there had been a little bit of an escalation before I received the subpoena. I had some sense that we were doing lots of advocacy and public communications and writing things to the attorneys general about this issue, and I was getting some sense that this was getting on their nerves.

When they asked, there was part of me still thinking, “Maybe this is just a good-faith question,” and they’re trying to figure out whether we are secretly funded and controlled by Musk or Meta or something. What I did was read through the subpoena, and I got to the part that said, “All of your communications about SB 53,” a bill we were working on. Then I started to think, “This doesn’t really feel like they are just asking good-faith questions.”

Again, I don’t know for a fact what’s in their heads, and I can’t say it, but my impression was not that.

What you’re saying is, it would make sense to me if, for whatever reason, they were serving me a subpoena and saying, “Hey, are you funded by Elon Musk? Is that why you’re trying to block our for-profit conversion?” But when they came to you and said, “Give us all the emails you’ve been sending about this bill that you’re working on,” that just felt out of scope.

Yes, it did. And one other thing I will add as well is, again, I was expecting maybe that a subpoena would come, but when I had talked to other organizations I was aware of that had been subpoenaed, it had been sent to their organization and just their Delaware-registered agent, and you’d just get an email that your Delaware-registered agent got a subpoena. It wasn’t people coming to their fifth-story apartment building at 7:00 p.m. or whatever.

Right.

That was another aspect that felt eyebrow-raising. I just think it really does leave a bad taste in my mouth.

Right. There’s one explanation, the uncharitable explanation, which is that OpenAI is trying to bully and intimidate any nonprofits that are critical of its restructuring plan. There’s another explanation, which I want to get your take on, which is that these are fair questions to ask.

We don’t have a lot of transparency. We have a lot of dark money flooding into fights about tech regulation these days, and it’s worth asking questions about who is behind those efforts. I guess we should just dispense with the central claim here. Nathan, let me ask you straight up: Are you or Encode working with or being funded by either Elon Musk or Mark Zuckerberg themselves, or people or entities associated with them or their companies?

So we are not funded by Elon Musk or Mark Zuckerberg. If you go on our website, it says that we have received funding from the Future of Life Institute, which was mentioned in their subpoena. The Future of Life Institute received a donation from Musk several years ago, but they are not Musk, and we said this in our communications back with them.

I have never talked to Musk. Musk is not directing our activities. We submitted a request asking the FTC to open an investigation into xAI and Spicy Grok and their things, and I will happily say on air that I think xAI’s safety practices are, in many cases, far, far worse than OpenAI’s. Again, that central claim is false.

What about Mark Zuckerberg? Any relationship with him or Meta?

No, none. Zero. We are an organization that focuses on AI safety and kid safety issues. We are just constantly at war with Meta. The idea that Meta is backing us is just—

Right.

—it feels—again, I realize not everyone has the context and knows who we are, but it’s just completely laughable.

You do have a list of donors or funders on Encode’s website. You say, “We’re generously supported by,” and then you list a bunch of organizations, including the Omidyar Network, the Archwell Foundation, which is Harry and Meghan’s foundation, and the Survival and Flourishing Fund, which is a kind of effective-altruism-linked philanthropy funded primarily by Jaan Tallinn.

So you do provide some transparency about who your funders are. Why do you think that wasn’t enough for OpenAI? Why do you think they still had questions about Elon Musk or Mark Zuckerberg?

Yeah. I think to some degree you’ll have to ask them. I’m not sure. Again, there’s also one thing to say here: There is no general right for them to know about all of our—

Mm-hmm.

And again, the subpoena did not ask about the Omidyar Foundation because the Omidyar Foundation is not relevant to their litigation in any way.

The role of a subpoena is to get relevant information for the litigation you are engaged in, not to ask whatever questions you would like the answers to from other private organizations. We would love to send a subpoena to OpenAI and be like, “Tell us all the details of what you’re planning to do in the restructuring,” and, “Are you going to disempower the nonprofit in the ways it perceives?” But we don’t have a right to do that. That’s not a question we can just ask them, even though we might like to.

What we did was put out a public letter asking them a bunch of questions. OpenAI can go to the press and say, “We want transparency about these things.” Again, they do have the right to ask us about Elon because they are in litigation about this.

I think if they had just reached out to us at our corporate address and said, “Are you funded or directed by Elon?” and we explained that we were not and proved it to them, and then they moved on, I would understand that. I think that is a fair thing, and that Elon is attacking them and trying to destroy them, and they want to make sure that there are efforts that are not covertly being supported and directed by him.

But I just can’t emphasize how far away what actually happened was from that narrow question that they were entitled to ask.

As you reflect on this experience, do you feel like this was intimidation? Do you think that OpenAI is trying to penalize organizations for speaking up either against the for-profit conversion or for AI regulation?

Yeah, to some extent it’s a question of intent, but I don’t know what’s inside their heads. I want to be careful about that, but I believe that that is what they were doing. That is my best guess. And that was how I received it.

I would like there to be another explanation for this. I thought it was possible when I put this out that maybe they would say, “Hey, this was a misstep. Our lawyers went a bit far. We didn’t really actually mean to add the thing about SB 53.” That’s not what they said. They doubled down and said, “We think we are entitled to this,” and I think that is just very important to note.

I’ll just say another thing that I don’t think we’ve mentioned: Even for some folks within OpenAI—for instance, Joshua Ackiam, who was speaking in his personal capacity—there was a fairly long thread talking about the fact that what I was describing in my thread doesn’t look great.

Yeah, but that was the unofficial response from someone at the company who was breaking from the company itself.

Yes.

We’ve also seen Jason Kwon, as you mentioned, the chief strategy officer at OpenAI. He wrote a lengthy thread arguing that you and Encode were only giving part of the picture, that Encode doesn’t disclose its funding, and that this is not about SB 53. Jason said, quote, “We did not oppose SB 53.” They said that, basically, this was a tempest in a teapot. There was also a quote that an OpenAI lawyer, Anna Leahy, gave to The San Francisco Standard, saying, “We welcome legitimate debate about AI policy, but it is essential to understand when nonprofit advocacy is simply a front for a competitive commercial interest.” What do you make of the official OpenAI response to what you claim?

One thing is that I think Jason focuses on the fact that we became involved with the lawsuit between Elon and OpenAI by filing an amicus brief arguing that it was in the public interest for OpenAI to remain a nonprofit. Geoffrey Hinton also made some positive comments about our amicus and showed support for our arguments. He’s also someone who, by the way, has called for Elon Musk to lose his status with the Royal Academies and is really not a fan of Musk, if you want another example of how not everyone who is critical of OpenAI’s restructuring is a Musk fan.

On the point of “did not oppose SB 53,” it is true that they never put out something saying that they formally opposed it. But their global affairs head, Chris Lehane, did send a letter to Governor Newsom at a time when SB 53 was in a pretty heated discussion, saying that he believes the correct path for California is to have an exemption from its AI frameworks for any company that signs on to an agreement with the federal government for testing or says that it will be adhering to the EU AI Code of Practice—which, in practice, means a complete exemption from the California law. You can ask a linguist whether that’s fair, but I think it’s still important context that he did not discuss.

What now? Are you going to send OpenAI the information that they’re asking for? Are you planning to do any more transparency around your funding or your advocacy efforts? What’s the next shoe here?

We sent them our objections and responses, where we laid out 4 areas that were relevant. For instance, our communications with or funding received from Elon, saying that those didn’t exist, and saying that the other pieces of information were not relevant. They never responded to that. They could have filed a motion to compel, saying to the judge that we had to turn them over, but they didn’t do that.

My view—again, I don’t know this for sure—is that they didn’t do that because they realized a judge would not grant that motion, because they were not, in fact, relevant. I think there are fair discussions about transparency. If some of our donors want to be private, when you’re donating to (c)(4)s, you have the right to give money privately. We have listed a lot of our donors on our site, and I think you get a clear impression of the different types of motivations that people have who are funding us.

But I think this larger discussion about what the appropriate transparency is for folks involved in the advocacy process is very different from what OpenAI cares about here or why they’re asking about this. Even in the subpoena, which was an overreach in many ways, they don’t talk about the Omidyar Foundation, which, again, is listed on our website as a funder. We’re not hiding that fact because it’s not relevant to their litigation with Musk.

But you said there are—

Yeah.

Donors that you don’t list on your website who want to remain private. Would you like to tell us who they are or how much they’re giving you?

Not here, no.

Okay. Just checking. I have to ask.

Fair.

I mean, I think the larger question here—

They’re not Musk or Zuckerberg. We don’t take money from frontier AI companies. I don’t know. I will say that.

Yeah, and I think it’s a reasonable thing to advocate for—that all of these groups should be required to disclose much more about who funds them. But I think that should apply equally to organizations that are pushing for the other side of things here.

Yeah, I think that’s fair.

How—

I think that’s a fair discussion to have. I’m just not sure OpenAI is the one to make that argument.

So, as you look back on this episode, how has it changed the way that you think about OpenAI?

I genuinely have a lot of positive feelings about OpenAI and think that they do many things genuinely better than their peers, for instance, Meta or xAI. I think that some of their safety research and system cards are things that they’ve improved on even in recent months and have done a genuinely good job of. I think there is a feeling among some people at OpenAI that they get disproportionate criticism relative to their peers, and I think there is some truth in that.

One thing I’ll say is that if one of their peers had been the one to show up at my house and give me a subpoena, I would have said that about them, too. But it was OpenAI that was the one that did it. Also, I think there’s some aspect of the fact that OpenAI is a nonprofit. They are a nonprofit with a mission to ensure that AGI benefits all of humanity, and they are in the process of trying to weaken and get around that legal mission and be able to consider profit more in their decisions.

I think this episode, and also things like the discussion about whether to allow not-safe-for-work porn or whatever on ChatGPT, whether to release Sora 2 in the way they released it, their kid-safety practices, and all sorts of these other things, show that they are not a normal for-profit company. They are, at least for now, a nonprofit that is dedicated to this mission above profit, and I do think that means they should be held to a higher standard.

Yeah, I’ll just say that it’s not like Elon Musk is the only person who opposes this restructuring plan. The whole AI safety community has been up in arms about this for years now.

It’s very unpopular.

Yes.

I’m just curious what you make of the difference between Josh’s statement and Jason’s statement, and some of this continued evolution and pressure within OpenAI as it transitions from more of a research organization focused on some of these loftier ideals to trying to move to the next stage of what it wants to do.

I think it speaks to a very real tension within the company. There are a lot of people there who believe in the stated mission and want to create this very beneficial AI. Then you also have a lot of people who come from other giant tech companies who see this primarily as a competition about winning, being first, and making the most money.

People who come from those kinds of companies are not above waging lawfare to get what they want. I’ll be curious to see how that shakes out in the coming months. It does seem like it’s that second group, the big-company group, that is currently steering the company, and I wonder if that’s going to continue.

But I will say, in addition to that, I think that’s right. Your story, Nathan, has caused more consternation and soul-searching among people at OpenAI than I think anything since the Daniel Cocotello story about the non-disparagement agreements that they were forcing people to sign, or else they would claw back their vested equity in the company. That was a big deal to people at OpenAI, and this is a big deal to people at OpenAI.

I've been talking to people. It's not just Josh who is saying this stuff. I think there's a lot of soul-searching going on inside the company about this question: Are we still the good guys? Are we transitioning to something we no longer support?

I think there's going to be some internal qualms about this and probably other stories to come. But most of them probably won't break out into the open the way this has. Nathan, thank you so much for coming on and explaining all this to us.

Thanks, Nathan.

Thank you.

I just wanted to note that we reached out to OpenAI after this interview, asking about this question of intimidation, and they responded with a statement from Jason Quan reiterating that, quote, “Elon has opposed our restructure for obvious competitive reasons, and Encode joined in. Organizations that suddenly emerge or shift priorities to join Elon raise legitimate questions about coordination and funding, which the subpoena seeks to clarify. Our questions have still not been answered, and we still don’t transparently know who is funding these organizations.”

When we come back, an old woman falls off a very high shelf. Is it real or is it fake?

No, it’s the Hard Fork Review of Slop. Well, Casey, over the last few weeks on our show, we’ve been talking a lot about slop.

We have, and it seems like the more we talk about it, the more of it appears all over the internet.

Yes, it is taking over the internet, and for that reason, we thought we should introduce a new segment that we are calling the Hard Fork Review of Slop.

The Hard Fork Review of Slop. Oh my God, that’s so perfect.

That’s beautiful.

You know, this, I would say, is generally a STEM podcast. We care a lot about science and technology and engineering, not as much math, but we also care about the arts. And so we thought, why don’t we carve out some time on the show to talk about some of the new achievements in AI art that we’re seeing out there on the internet, and also bring our critical eye to them and put them in conversation with the culture?

Yes. We have critics out there for books and movies and music and video games, and I think slop is an emerging genre of cultural production, most of which is bad, but some of which may actually be good. And so we need to stand here amid the floodgates, filtering out the bad slop and letting the good slop get through.

Okay, I just want to signal up front: I didn’t actually bring anything good. I didn’t know that was part of the assignment.

I have one good one—

Okay.

—but we’ll save it for the end.

All right, fair enough.

So Casey, tell me about the slop that you have been looking at, and then I will tell you about some slop that I’ve found.

That’s great. Well, Kevin, maybe to warm us up, we can look at some of the slop that I think of as CoComelon for adults: pure visual stimulation, no ideas in it whatsoever. This slop, you can find on TikTok if you search for “glass fruit cutting.” Have you seen any of the glass fruit cutting?

No.

Okay, let’s see if we can cue one of these up. Some of these are in the ASMR realm.

Ooh.

This man is cutting into a coconut with a knife, but the—

The coconut is glass.

—it’s glass.

Oh, the kiwi is glass. Oh, that’s… I don’t like that sound.

There’s a—

That has nails-on-a-chalkboard vibes for me.

—there’s a grape.

This is what’s replacing reading in our schools.

I mean, literally. Now, what I liked about this one, Kevin, is it’s glass pancakes—

Ooh.

—with beautiful maple syrup. You really hate that sound, don’t you?

That’s a jump-scare sound for me. Oh, we got a donut, a glass donut.

Would love to see a glass donut. Oh, just cutting through the glass bowl of cereal there. Glass egg.

The physics here are actually kind of impressive to me.

Right?

It is showing the reflections of the knife in the glass. It looks vaguely realistic.

Yeah.

Wow.

It’s weird because the food looks delicious and beautiful—

Yeah.

—but then it’s glass, so it’s off-putting.

Yeah.

It just doesn’t make any sense, and so it hypnotizes your brain—

Yes.

—into this sense of, “I don’t know what I’m watching. I don’t want to look away—

Yes.

—and I’m going to stop thinking words.”

Yes. It’s sort of like the spiritual successor to those crush videos where they’ll have the hydraulic press—

Yes.

—and they’ll just press down on seven objects.

Yeah. And now instead of just wasting those objects, we can waste water and electricity. All right. What do you have?

So I have an example from the news, actually. This one comes to us from DIRECTV, which has just struck a partnership with an AI company called Glance that will allow people with DIRECTV Gemini devices to put themselves inside of 30-second AI-generated videos. Basically, if you step away from your TV to go get a snack or go to the bathroom, you might come back and find that you are in the ad on the TV.

Casey, let’s watch an example of this. This shows how it works. You connect it to your TV, put in your photo, tap a couple of buttons, and it generates your likeness.

This process is already so absurd.

And then, boom, there you are in a blazer. Now, Casey, I thought the point of advertisements was to show clothes on people who are more attractive than me—to entice me to buy them.

Yeah.

Why would I want to see clothes ads with me in them?

I can’t answer that question honestly. This one is so funny to me because the process you have to go through to do this is so complicated. I basically cannot imagine a single person doing this.

First of all, you already have a TV that’s working against you, right? The way that this works is that if you have one of these TVs and you leave it idle for 10 minutes, AI takes over, which is kind of like if the bus goes below 55 miles an hour, it explodes.

Yes.

This is that, but for AI advertising. And then after it shows you these images, it’s up to you to go scan a QR code and take a photo of yourself. No one who is watching TV wants to do any of this at all.

It’s a very silly process. In the demonstration, the photos look fine.

They look fine.

They look fine.

Yeah.

But let me ask you this: Do you not know what you look like with a jacket on? You know what you look like with a jacket on. What are we doing here? That is my review of this. What are we doing here?

We are selling advertising technology.

Okay. So now I just want to show one that made me laugh. I call this one “Woman on the Walmart Shelf,” if we want to queue this one up. I saw this one on TikTok, although it does have the Sora watermark on it. And I think this speaks to the ability of AI slop to create a classic pratfall physical-comedy situation.

This one involves what looks like store security-camera footage of an older woman on a very high shelf inside a Walmart, and there’s a police officer who’s looking up at her as our story begins.

Ma’am, please come down from there.

You want me to come down?

Yes, ma’am.

All right.

And she does a header off the shelf and crashes into the police officer. So, Kevin, what did that one make you feel?

There’s a lot there. There are a lot of layers to this onion. Is this a one-off, or is there a larger genre of older people—

Yeah.

—falling off the top shelf at the grocery store onto a police officer?

It’s a whole interconnected cinematic universe with these very rich characterizations. The vocal performances are really amazing, so I encourage you to get into it. It’s beautiful stuff.

No, this is a one-off, Kevin. I’ve never seen anything else related to it.

Yeah. I’m not worried that people are going to start throwing themselves off the shelves of grocery stores to mimic the trend here. This one feels pretty harmless to me. And I appreciate inspiring older people to do things like climbing up to the top shelf of the grocery store.

Yeah. These days, any time I see a Sora video that isn’t misappropriating the likeness of Martin Luther King Jr., I say that’s a win for slop.

Yes. This one, I think, is pretty harmless.

All right. What else you got?

Well, this next one, Casey, was not harmless—

Okay.

—because it involved America’s queen, Dolly Parton.

Oh, no.

Leave her alone. Basically, some sicko out there has been generating AI images of Dolly Parton looking very sick, including at least one image of Reba McEntire visiting Dolly Parton on her deathbed, which went around on the internet and led to a bunch of rumors that Dolly Parton, God forbid, was dying.

Oh, no. See, I hate this.

Yeah, I don't like this either. Let's watch Reba's video summarizing the whole thing.

You tell ’em, Dolly, that AI mess has got us doing all kinds of crazy things. You’re out there dying, I’m out here having a baby? Well, both of us know you’re too young and I’m too old for any of that kind of nonsense. But you better know I’m praying for you.

I love you with all my heart—

I need to see you—

And I can’t wait to see you soon.

For 100 years.

Love you.

Wait, just to be clear, what you showed me was real, not slop? That is Reba McEntire’s actual Instagram account?

That is Reba McEntire’s actual Instagram account.

Okay.

She does show some of the slop images inside the thing, of Reba at Dolly’s deathbed, and Dolly responded with another real video from her real social media account saying, quote, “I ain’t dead yet.” So, Casey, what do you make of this one?

I mean, this is so bad. So many of the fears around misinformation have been that there will just become a time when you can’t tell what is true and what is false. The better that image-generation software gets, the more of these little viral hoaxes we’re going to see going around. So this is super bad. What kind of person do you have to be to be like, “Today is the day that I create a rumor that Dolly Parton has died, and I’m going to use Sora to prove it”?

Truly.

Yeah.

It is mind-boggling to me.

If you wanted to turn the public against AI and against AI-generated content, the most effective thing you could do would be to go after Dolly Parton, who everyone—literally everyone—loves.

No. I hope Jolene finds whoever did this and puts a number on ’em.

Let’s just say Dolly Parton’s lawyers are going to be working more than 9 to 5.

This next one is a sort of narrated journey. We are returning to Walmart for this one. This creator is very interested in the use of AI to create art on products. I’ve seen some in craft stores where there are framed pictures that are clearly AI-generated. In this case, she picks up some butter cookies at Walmart and makes a pretty convincing case that it is slop art. I enjoyed this journey. Let’s see how it looks here.

This is bad. I didn’t think it could get worse, but you guys are right. The butter cookie tins at Walmart are way worse than the popcorn tins. Why is Santa throwing ass? Why is he squatting on a table? Why does he look like he’s about to twerk? What is his hand doing? What is that?

Look at the background.

Santa has the fattest ass in this.

Look how wonky that is. And what is this wall full of random things? Can you make out what any of that is actually supposed to be? It looks like there are cobwebs on the roof—whether that’s intentional or not, I don’t know.

It’s just random shapes on the wall.

This is pure AI slop. No question about it.

All right. We can probably stop it there.

I have to say, this video made me feel very naive because I did not realize that there were mass-produced products in Walmart stores that were AI-generated.

Oh, yeah.

Yeah.

I also love that there are now slop detectives who are just going out there vigilante-style and investigating the slop on the shelves of their local Walmart. That’s beautiful to me. We need more citizen participation.

Honestly, it could be a segment for our show: slopvestigations.

Hmm. We’ll—

Wait, I have a question.

Yeah.

If you’re shopping and you pick up an object and you see that there’s slop art, does that affect whether you want to buy it or not, one way or the other?

No. I mean, I think there’s a whole category of art that basically doesn’t matter, which is the stuff on the cookie tin. Right? The stuff at Walmart. No one is winning any prizes for that. No one is reaching any new heights of creativity. Basically, this is just a way for the butter cookie manufacturer to save a couple bucks and not have to hire an illustrator or use some stock art from the internet.

And do you think they’re passing the savings on to us, the customers?

Probably not. Probably not.

That’s probably going right to their bottom line.

That’s unfortunate.

Yes. What about you? Would you be less likely to buy something if slop had been used in its advertising?

Maybe, because I think it speaks to a kind of cheapness and a lack of care. So if I were buying a heart defibrillator and I saw that there was slop art on the box, I would say, “I don’t know if I can trust these people.”

What about butter cookies from Walmart? Are you going for quality when you’re buying butter cookies from Walmart?

I want brown butter if it’s going to be butter cookies. Because butter is a great flavor, but it needs something else, you know what I mean?

Okay. So for Casey, only the artisanal images of Santa with a huge ass—

Small-batch, huge-ass Santa butter cookies, please.

Okay. One more example of slop that I want to tell you about today, Casey, and get your opinion on. This one is what I would consider good slop.

Okay.

This is slop that is being made toward a noble cause, which is preventing the AI apocalypse.

Hmm.

Now, Casey, you might think to yourself, how could this happen? How could AI slop be used to ward off the AI apocalypse?

I was just about to ask you that.

Well, this is a company called Hyperstition. It was founded by Andrew Cote and Aaron Silverbug. Basically, this is a company that is trying to counteract all of the science-fiction stories and narratives out there about AI going rogue and killing people. This hypothesis sort of makes its way into the training data for these AI systems and actually makes them more likely to go rogue, in this theory.

It gives them ideas.

It gives them some ideas. Andrew Côté said, “What if we combated this by writing a bunch of AI-generated novels about AIs and humans getting along really well, and then we fed that into the training data for the AI systems to kind of give them some more good examples to follow?”

All right. Kind of a convoluted explanation, but sure, why not?

So this company has just gotten a grant—I read about this on AstroCodex 10—to create 5,000 AI-generated novels. They’re trying to have these novels be around 80,000 words, and they’re going to enlist the public’s help to generate them. You can buy credits, about $4 a book, to generate this, and then they’re going to try to feed these into the language models and get the models to think about maybe good scenarios and maybe be more likely to act on them.

Wait, why does the public get involved if the works are all AI-generated?

I think they want it to reflect a diverse set of scenarios and characters. Basically, they want people to get involved in this and make it as diverse as possible.

All right. Well, do we have any examples we can see?

No.

Great.

So what do you make of this attempt to use slop for the benefit and potentially the salvation of humanity?

Here’s what I’m going to say: If it turns out that the thing that is needed to prevent human extinction from AI is a massive infusion of slop into the training data, I’ll be very surprised. I’ll be very surprised if that was the difference-maker.

I share your skepticism. I think the default outcome from this project is that it probably doesn’t save us from the AI apocalypse. I think a funny secondary effect would be if one of these 5,000 slop novels goes on to become a huge bestseller and becomes the literary craze that takes over the country. Do I think that’s likely? No, but it could happen.

Well, as we mentioned earlier in the show, it doesn’t seem like people are reading all that much these days. But maybe all of this will eventually be fed into a NotebookLM video presentation that folks can watch.

Yes. All right, that is it for the Hard Fork Review of Slop, and we welcome your submissions for future installments.

Yeah. We want it to be like— it can't just be like, “Look at this weird thing.” I want to see slop that made you feel something.

Yeah. And the next time you see a Santa with a suspiciously large posterior, call us.

Call us.

Email us.

We want to know about it.

We want to see it.

And we want to salute them.

Casey has a folder on his MacBook that's just photos and images of Santa with a very large ass.

Yeah.

I love a thick Santa. And I salute them, sir. See you on Christmas, big guy. The Hard Fork Review of Slop. Hard Fork is produced by Rachel Cohn and Whitney Jones. We're edited by Jen Poynt. This episode was fact-checked by Will Peichel. Today's show was engineered by Chris Wood. Original music by Alicia Meitube, Diane Wong, Rowan Nemestio, and Dan Powell. Video production by Sawyer Roquet, Pat Gunther, Jake Nickell, and Chris Schott. You can watch this whole episode on YouTube, along with that slop, at youtube.com/hardfork. Special thanks to Paul Schuman, Pui Wing Tam, Dalia Haddad, and Jeffrey Miranda. You can email us at hardfork@nytimes.com with the slop that made you stop.