TikTok Faces Issues with Nazi Content

ISD reported 50 accounts in June for violating TikTok’s policies on hate speech, promoting violence against protected groups, endorsing hateful ideologies, celebrating violent extremists, and denying the Holocaust. TikTok initially found no violations and allowed all accounts to remain active.

A month later, TikTok banned 23 of these accounts, indicating some violative content and channels are removed over time. Before being taken down, these 23 accounts had amassed at least 2 million views.

The researchers also created new TikTok accounts to investigate how Nazi content is promoted to new users through TikTok’s algorithm.

Using an account created in late May, researchers watched 10 videos from a network of pro-Nazi users, occasionally checking comment sections but refraining from engaging with the content. They also viewed 10 pro-Nazi accounts. When they switched to the For You feed, it took only three videos before the algorithm suggested a video of a WWII-era Nazi soldier overlaid with a chart of US murder rates broken down by race. Later, a video featuring an AI-translated Hitler speech overlaid with a recruitment poster for a white nationalist group appeared.

Another account set up by ISD researchers saw even more extremist content in its main feed, with 70 percent of videos coming from self-identified Nazis or featuring Nazi propaganda. After the account followed several pro-Nazi accounts to access private channel content, TikTok’s algorithm recommended other Nazi accounts to follow. All 10 of the first accounts suggested by TikTok used Nazi symbols or keywords in their usernames or profile photos or featured Nazi propaganda in their videos.

“In no way is this particularly surprising,” said Abbie Richards, a disinformation researcher specializing in TikTok. “These are things that we found time and time again. I have certainly found them in my research.”

Richards wrote about white supremacist and militant accelerationist content on the platform in 2022. This included the case of neo-Nazi Paul Miller, who, while serving a 41-month sentence for firearm charges, appeared in a TikTok video that gained over 5 million views and 700,000 likes in three months before it was removed.

Marcus Bösch, a researcher from Hamburg University monitoring TikTok, noted that the report’s findings “do not come as a big surprise” and expressed skepticism about TikTok’s ability to address the issue effectively.

“I’m not sure exactly where the problem is,” Bösch said. “TikTok says it has around 40,000 content moderators, and it should be easy to understand such obvious policy violations. Yet due to the sheer volume [of content], and the ability by bad actors to quickly adapt, I am convinced that the entire disinformation problem cannot be finally solved, neither with AI nor with more moderators.”

TikTok stated it has completed a mentorship program with Tech Against Terrorism, a group aimed at disrupting terrorists’ online activities and helping TikTok identify online threats.

“Despite proactive steps taken, TikTok remains a target for exploitation by extremist groups as its popularity grows,” Adam Hadley, executive director of Tech Against Terrorism, told WIRED. “The ISD study shows that a small number of violent extremists can wreak havoc on large platforms due to adversarial asymmetry. This report underscores the need for cross-platform threat intelligence supported by improved AI-powered content moderation. The report also reminds us that Telegram should also be held accountable for its role in the online extremist ecosystem.”

As Hadley outlines, the report’s findings highlight significant loopholes in TikTok’s current policies.

“I’ve always described TikTok, when it comes to far-right usage, as a messaging platform,” Richards said. “More than anything, it’s just about repetition. It’s about being exposed to the same hateful narrative over and over and over again, because at a certain point you start to believe things after you just see them enough, and they start to really influence your worldview.”