AI-generated ‘Seinfeld’ suspended on Twitch for AI-generated transphobic jokes

After charming the internet (and us) with its absurdist nature, the AI-generated “Seinfeld”-spoof “Nothing, Forever” was suspended from Twitch for 14 days.

As part of its parody of “Seinfeld,” one of the recurring scenes on “Nothing, Forever” shows Larry Feinberg — the Jerry Seinfeld character — performing stand-up routines. Usually, Larry’s jokes went something like, “What’s a bear with no teeth? A gummy bear.” But, as “Nothing, Forever” creators Mismatch Media told TechCrunch, the team made a mistake with the AI’s content filter, which resulted in Larry delivering a transphobic standup routine.

“Nothing, Forever,” which has been streaming almost constantly since December, initially generated the characters’ dialogue through the Davinci model of GPT-3, plus OpenAI’s content filtering protocol. But when Davinci began having technical issues, the team switched to the less-sophisticated Curie model. After changing models, the creators didn’t appropriately incorporate the AI moderation tool. Soon, Larry was spewing transphobic jokes, and “Nothing, Forever” was suspensed.

“First, let me say that the text we generated was a mistake, we are super embarrassed, and that the generative content created in no way reflects the values or opinions of our staff,” co-creator Skyler Hartle wrote in an email to TechCrunch. “We very much regret this happened and hope to be back on the air soon, with all the appropriate safeguards in place.”

Hartle also said that the team is investigating a secondary content moderation system to add as a redundancy, just in case their content moderation filters happen to falter while live on air.

Twitch did not respond to request for comment. Hartle said he has not had any direct communication with Twitch either. But the creators believe the suspension is a result of Larry’s transphobic comments, due to the timing of the suspension.

Twitch’s community guidelines ban hateful rhetoric about specific identity groups, including discrimination or harassment based on gender identity. The platform’s content moderation systems appear to have caught the transphobic rhetoric of Larry, an AI, pretty quickly. But while Twitch’s systems reacted quickly to deplatform an AI, it has not been as consistently successful with mitigating the deliberate harassment of transgender people.

About a year and a half ago, the platform was plagued by malicious “hate raids,” which particularly targeted Black and transgender creators. After creator-led collective action, Twitch rolled out new safety features to help protect streamers from these violent cyberattacks. Last year, as transgender Twitch streamer Keffals rose to prominence for talking about LGBTQ+ issues, she was subject to such harsh harassment that she was targeted within her own home, causing her to go into hiding. (Much of this harassment stemmed from other websites, but as of 2021, Twitch’s community guidelines can hold Twitch users accountable for off-platform harassment.)

The creators of “Nothing, Forever” say they did not intend to air transphobic commentary, and that they do not endorse those views. Yet this predicament shows how quickly generative AI entertainment can go south when left unchecked. After all, GPT-3 is trained on the internet itself, so it can echo some of the most heinous rhetoric from across the web.

There’s something darkly ironic about this incident: A content filtering snafu drove the Twitch suspension of what’s basically a robot. Meanwhile, Dave Chappelle just won a Grammy for his comedy album “The Closer,” which has been heavily criticized for its transphobia. Yet even within the AI-generated universe of “Nothing, Forever,” the AI-generated audience didn’t find Larry’s jokes very funny.

“There’s like 50 people here and no one is laughing. Anyone have any suggestions?” the AI began its comedy set, before going on its brief transphobic tirade. Then, Larry closed his set, “No one is laughing, so I’m going to stop. Thanks for coming out tonight. See you next time. Where’d everybody go?”