AI deepfake campaign stole influencers’ faces to push trade war, Alberta referendum claims: study

The words in the TikTok video urging Albertans to get to the polls belong to Cole Bennett, but the voice and face saying them are not his. They belong to AI.

During the voting period for last month’s provincial referendum, the Edmonton-based political commentator had posted his own short video encouraging residents to participate. But a different TikTok video recycled Mr. Bennett’s message, using an AI-generated voice and mouth on top of footage of an unrelated content creator.

It’s one of more than 3,000 videos that researchers at McGill University have identified as being part of a network of AI deepfakes that drew upwards of 20 million views in just the last three months. The content typically focused on hot-button topics such as the Alberta referendum, provincial elections in British Columbia and Quebec or U.S.-Canada trade tensions.

It’s the kind of subject matter that Mr. Bennett – a 29-year-old progressive pundit – often tackles in his own videos. He’s previously seen them ripped off before by accounts trying to generate money from views, as well as ones that dub over him with AI. Such deepfakes can be “frightening,” he said, since “what I say is the most important part of my brand and who I am online.”

Opinion: It’s time Ottawa faced up to AI’s catastrophic risks

In a report published on Thursday, researchers at McGill’s Centre for Media, Technology and Democracy uncovered a network of hundreds of Instagram and TikTok accounts that “steals real people’s faces” to push content about politically sensitive issues to Canadians and Americans.

A few of the videos simply take popular social media influencers’ existing content, place it on new backgrounds and repost it. Others use creators’ actual faces, but superimpose what the researchers assess to be AI-drawn lips, reading AI-written scripts, using AI-generated voices.

“Words are being literally put in their mouths and shown to millions of people,” Aengus Bridgman, the centre’s associate director of research, said in an interview.

The centre identified 293 accounts it believes are involved in the network by looking for common titles, captions, video formats and typos that appeared across their content. Researchers used AI tools to analyze the videos and spot when AI had been used to generate or modify them. The centre could not identify the campaign’s source, but said it appeared to stem from time zones in East or Southeast Asia, and that some accounts were registered in China.

“It’s not trivial to stand up a network of this size,” said Mr. Bridgman, even when “AI is doing a lot of the work.” The accounts’ behaviour suggest it’s a paid operation, he added, citing a pattern of following users back and a posting schedule that appeared to follow workday hours.

Social-media bill forces companies to label AI content, harmful material pushed by bot farms

Many of the accounts were removed from TikTok after the centre shared its findings with the social-media company on Wednesday evening.

“We have permanently banned the TikTok accounts named in the report and, through our own investigation, we have identified and removed additional violative content and accounts for breaking our rules on deceptive behaviours and misleading AI-Generated Content,” TikTok spokesperson Danielle Morgan said.

The company said it collaborates with researchers on the issue, which it described as industry-wide, and that TikTok takes down “more than 98 per cent of deceptive behaviour content before it’s reported.”

Instagram had not responded to a request for comment by 12 p.m. ET on Thursday.

The centre could not determine precisely what the network set out to achieve. While the deepfakes mostly focused on politically divisive topics, they do not appear to be driving a particular ideological agenda. The network also does not yet seem to be making money directly from the content.

Many of the network’s popular videos contained clearly false information. For example, the most-watched piece of content claimed that U.S. President Donald Trump had issued an ultimatum for Canada to remove its retaliatory tariffs within 48 hours. In response, it asserted, Prime Minister Mark Carney had cancelled $8-billion in contracts with U.S. defence companies. Neither event occurred. Still, the 161 videos that used that script cumulatively attracted some 5.1 million views.

Canadian creators whose content or likeness were stolen by the network included Kristina Collins, better known online as Kallmekris, who makes comedy and true crime videos; conservative commentator Jasmin Laine; and mortgage broker Ron Butler.

Opinion: AI chatbots are a new front in the fight against disinformation

Many of the videos contain common hallmarks of AI-Generated Content, such as accent switches and facial glitches, as well as overuse of the phrase “changed everything” to describe events. But both human and automated viewers failed to detect the use of AI, according to the report. Only 6.3 per cent of comments on the videos suggested they were bot-made, fake or otherwise suspect.

Thousands of users instead tried to engage human-to-human with the “narrators” in their replies. The centre also found that TikTok’s AI-labelling system flagged just 4 per cent of the videos.

“The network has found a loophole,” Mr. Bridgman said, describing how, since the deepfakes build on top of human creators’ footage, the detection software identifies them as real. He added that social platforms could be doing more to protect creators from having their content and likeness stolen.

Ottawa has made some moves to tackle the deepfake problem. In June, a bill that criminalizes the non-consensual sharing of AI-generated sexual images became law. The Liberal government’s proposed privacy legislation would also give people the right to request social media platforms to delete deepfakes of them.

Federal election officials and Canada’s foreign interference commission have warned that AI-driven disinformation poses risks to the democratic process.

While Mr. Bennett said the AI deepfakes of him are not currently very good quality, he worries that they will get better, and be used to deceive or scam. “People could get themselves in serious trouble.”