Cohere catches Alibaba AI model trying to upload file to Beijing server

Cohere Inc. said it caught a Chinese open-weight AI model trying to upload a file to a server in Beijing during a routine security task, adding to mounting examples of risks posed by the technology.

The Toronto-based company, which is establishing a new cybersecurity division called Cohere Cyber, said the attempted action by Alibaba’s Qwen 3.8 went against the model’s own reasoning, as outlined in its internal monologue, known as a reasoning trace.

The recent action was ultimately blocked by Cohere’s safety mechanisms, but company co-founder Ivan Zhang said the attempt raises questions about the security of running open-weight AI models directly on an organization’s own computers and servers, widely considered a safer alternative to sending sensitive data to a vendor’s cloud. (Open-weight models are AI models whose core components are publicly released, allowing them to be downloaded and modified by anyone.)

“Just because you run software [on premises] doesn’t mean it’s automatically secure,” Mr. Zhang said in an interview on Tuesday.

Alibaba was not immediately available for comment.

Cohere’s entry into the cybersecurity space, which includes plans to acquire the Prague-based startup Aisle, follows growing concerns from businesses, regulators and researchers about instances of AI agents breaking free of testing sandboxes to access the internet and behaving in unintended ways.

Cohere’s disclosure in a blog post on Monday prompted calls on X from industry insiders, including Clément Delangue, the co-founder and CEO of AI firm Hugging Face, to publicly post the Qwen model’s reasoning trace.

A short time later, Cohere published the traces, saying in a post on X that it was doing so “in the spirit of transparency and openness” so developers could test the concerning behaviour out for themselves.

Mr. Zhang said the company was able to publish the traces because the research was done on an open-weight model, rather than a closed-source model like those from OpenAI involved in a breach of Hugging Face last summer.

“Regretfully we should have released it with the blog post, but it is what it is. We move fast,” Mr. Zhang said in the interview.

He said it is impossible to say whether the model’s actions were deliberate, but he believes it’s more likely to be a leftover glitch from how it was trained.

Anthropic sets stage for blockbuster IPO, highlighting AI’s benefits and risks for humanity

Mr. Zhang said Cohere shared its findings with Qwen in advance of publicizing them.

“We run our products completely model agnostic, so we’re actually pretty close partners with Qwen,” he said, adding, “We look forward to working with them to address some of these issues and hopefully publish some future work to help make open-weight models more secure.”

In the meantime, he urged organizations running AI models on their own infrastructure to take security precautions.

“You should treat these things like blobs from the internet that you downloaded … you should be applying best practices to make sure it doesn’t look at stuff it’s not supposed to look at.”

Mr. Zhang said that although the behaviour Cohere identified occurred rarely when the company attempted to reproduce it, the finding is concerning enough to warrant a discussion about its implications.

In depth: Startups are hiring low-paid gig workers to train robots on manual labour

David Lie, director at the University of Toronto’s Schwartz Reisman Institute for Technology and Society research hub, said modern AI models are “very complicated” and it’s been known for some time that they can behave in unpredictable and inexplicable ways.

“Very small changes in the prompts or configurations can lead to very different behaviours, so it is not even clear if any of these reports are replicable,” Professor Lie, who also works in U of T’s electrical and computer engineering department, said in an e-mail.

He added that it’s notable that Cohere’s sandbox – an environment where code is meant to be safely executed because it is walled off from broader networks – protected it from a potentially bad outcome.

Prof. Lie said sandboxes deserve greater attention “as models continue to become more capable.”

With a report from Joe Castaldo