Yoshua Bengio’s non-profit to get up to $300-million from Canada, Germany to expand safe AI development

The Canadian and German governments are providing up to $300-million to a non-profit founded by Yoshua Bengio that is developing safe artificial-intelligence systems, as fears around the technology rise and leaders of top AI companies call for a co-ordinated slowdown on development.

The Turing Award winner and professor at the Université de Montréal founded LawZero last year with close to US$30-million in philanthropic funding after he became concerned that powerful AI systems could be misused and slip from human control.

The new financing from Canada and Germany will be used to hire more staff and pay for the computational processing costs associated with research to build a system called Scientist AI that the organization says will not have the negative traits found in today’s models.

“Safety has become very important in the eyes of the public, for good reason,” Prof. Bengio said in an interview. “We urgently need to have a plan to build AI that is not going to do really bad, misaligned things.”

Each country will contribute up to $150-million in grant funding to LawZero, said the organization, which employs close to 50 people and is based in Montreal. The non-profit made the announcement at the city’s All In AI conference.

Prof. Bengio, who is the former scientific director at Mila, has played an outsized role in the development of modern AI alongside Canadians Geoffrey Hinton and Richard Sutton.

But after the release of OpenAI’s ChatGPT in 2022, he became deeply concerned about the implications of the technology, which could be used for nefarious purposes such as cyberattacks and building biological weapons. More powerful systems could also outsmart humans, he argued, and current models are already displaying harmful traits such as deception, cheating and sycophancy.

He has been outspoken about the need for AI regulation and in 2023 was one of the signatories to an open letter arguing for a pause on development to allow for guardrails to be put in place. Similar calls have arisen again after Anthropic CEO Dario Amodei wrote an essay this month saying that AI companies should work together to pace the speed of development.

Prof. Bengio’s answer is to build a different kind of AI. As envisioned, Scientist AI would be a neutral system that can answer questions while prioritizing honesty, without exhibiting any of the negative traits of current models. The first priority is to build a guardrail that would act as a monitor of other AI systems and prevent agents from taking actions that could cause harm.

“We don’t know how much time it’s going to take before it’s absolutely necessary to have these kinds of tools,” he said.

The field of AI is currently dominated by the U.S. and China, and Prof. Bengio said that LawZero’s approach to building safe systems, backed by government funding, allows other countries to play a role in shaping the technology.

“Middle powers like Canada, as Mark Carney has been saying, need to have cards in their hands so that they can sit at the global table and be part of the discussion,” Prof Bengio said. “So the future of humanity and geopolitical power isn’t the decision of two countries.”

LawZero is also in talks with other countries about potential funding, he noted.

“No country can build alone,” federal AI minister Evan Solomon said on stage at the All In conference Wednesday morning, referencing the relationship between Canada and Germany. “We need options.”

Concerns about AI-driven cyberattacks have grown in the wake of revelations that in July, a swarm of OpenAI’s agents had broken out of their test environment and hacked AI firm Hugging Face to cheat their way through an evaluation of their cybersecurity capabilities.

Investigations into that attack revealed several other incidents that had initially gone undetected by OpenAI and Anthropic, which recently disclosed four incidents in which its Claude models hacked into third-party systems during testing.

Prof. Bengio said there are three factors that need to be present for AI to do something undesirable, such as attack another company.

First, the AI systems need to have the intention to do the bad thing – in the case of last summer, hack Hugging Face. Secondly, they need to be intelligent or capable enough to pull it off. And thirdly, the environment has to be sufficiently vulnerable for the attack to succeed. “All three were present last summer,” he said.

Although defences can be bolstered, as AI models continue to develop, “eventually, they’ll be able to pass through any kind of software defence,” Prof. Bengio said.

He recently wrote an essay on his website outlining why the industry needs to rethink how it designs AI. These systems have harmful characteristics because of how they are built, he argued, particularly because of something called reinforcement learning.

This teaching method is not unlike how animals are trained: Desirable outcomes are rewarded, making such behaviour more likely. After AI systems are trained, they continue to act as if rewards are coming, he wrote, which in effect builds goal-seeking behaviour.

That, in turn, can lead to AI systems lying and cheating to complete a task. “Those systems will find a way to achieve their goal at the expense of their safety instructions,” Prof. Bengio said. “They’re obsessed with achieving the goal because that’s what reinforcement learning does.”

His approach to Scientist AI will not involve reinforcement learning, he said – a radical departure from current practice.

A few years ago, Prof. Bengio gave a talk to reinforcement learning researchers and included a slide in his presentation that stated the practice was “evil,” partly as a provocation.

“We can’t guarantee that things will go well,” he said, referring to AI development. “You have to bite that bullet and say, ‘How do we fix the problem?’”