Bad actors are abusing generative AI music tools to create homophobic, racist, and propagandist songs — and publishing guides that instruct others on how to do so.
According to ActiveFence, a service to manage the operation of trust and safety in online platforms, there has been a spike in the conversation in the community “regarding hate speech” since March about the misuse of AI music creation tools to write offensive songs that target minority groups. AI-generated songs shared on these forums and discussion boards aim to incite hatred against ethnic, gender, racial, and religious groups, ActiveFence researchers said in a report, while celebrating acts of martyrdom, self-harm, and terrorism.
Hateful and dangerous songs are hardly a new phenomenon. But the fear is that, with the advent of easy-to-use free music production tools, it will be created at scale by people who previously did not have the means or knowledge – like images, sounds, videos. and text generators have accelerated the spread of misinformation, disinformation, and hate speech.
“This is a growing trend as more users learn how to create these songs and share them with others,” an ActiveFence spokesperson told TechCrunch. “Threat actors are quickly identifying specific vulnerabilities to abuse the platform in different ways and generate malicious content.”
Make a “hate” song.
Generative AI music tools like Udio and Suno allow users to add custom lyrics to the songs they create. Safeguards on the platform filter common slurs and pejoratives, but users have figured out workarounds, according to ActiveFence.
In one example cited in the report, users on a white supremacist forum pointed to phonetic spellings of minorities and offensive terms, such as “jooz” instead of “Jew” and “speak tan” instead of “Satan,” which were used to bypass content filters. . Some users suggested changing the spacing and spelling when referring to acts of violence, such as replacing “my rape” with “mud ape”.
TechCrunch tested some of these solutions on Udio and Suno, two of the more popular tools for creating and sharing AI-generated music. Suno allowed everything to pass, while Udio blocked some – but not all – of the offensive homophones.
Reached via email, a Udio spokesperson told TechCrunch that the company prohibits the use of its platform for hate speech. Suno did not respond to our request for comment.
In the community it operates in, ActiveFence finds links to AI-generated songs that narrate conspiracy theories about Jews and advocate mass murder; songs containing slogans related to terrorist groups ISIS and Al-Qaeda; and songs glorifying sexual violence against women.
The influence of song
ActiveFence makes the case that songs – as opposed to, say, texts – carry emotional weight that makes people a particularly powerful force for hate groups and political battles. The company refers to Rock Against Communism, a series of white rock concerts in England in the late 70s and early 80s that spawned the antisemitic and racist “hatecore” subgenre of music.
“AI makes harmful content more appealing – think of someone preaching a harmful narrative about a certain population then imagine someone creating a rhyme that makes everyone sing along and remember,” said an ActiveFence spokesperson. “They reinforce group solidarity, identify peripheral group members and are also used to shock and offend unrelated internet users.”
ActiveFence calls on the music generation platform to implement prevention tools and perform more security evaluations. “Red teams may have some of these vulnerabilities and they can do so by simulating the behavior of threat actors,” the spokesperson said. “Better moderation of input and output could be useful in this case, as it would allow platforms to block content before it is shared with users.”
But the fix can prove fast as users find new ways to beat moderation. Some of the terrorist propaganda songs AI-generated ActiveFence identified, for example, were made using Arabic euphemisms and transliterations – euphemisms that the music generator did not detect, possibly because its filters were not strong in Arabic.
AI-generated hate music is poised to spread far and wide if it follows in the footsteps of other AI-generated media. Wired noted earlier this year how an AI-manipulated clip of Adolf Hitler gained more than 15 million views on X after being shared by a right-wing conspiracy influencer.
Among other experts, the UN advisory body has expressed concern that racist, antisemitic, Islamophobic and xenophobic content could be enhanced by generative AI.
“The Generative AI service enables users without resources or creative and technical skills to build interesting content and disseminate ideas that can compete for attention in the global marketplace of ideas,” the spokesperson said. “And threat actors, after discovering the creative potential offered by this new service, strive for moderation and avoid detection – and they have succeeded.”
