King Charles warns AI leaders of ‘existential dangers’ if the technology falls into wrong hands

Britain’s King Charles warned AI leaders on Thursday of the “existential dangers” posed by the technology falling into the wrong hands, with potentially catastrophic consequences, saying safeguards were needed before it becomes “too late”.

Charles was hosting the co-founders of Nvidia and Google DeepMind, along with leaders from OpenAI and Anthropic, in Scotland as concern grows that AI safety efforts are lagging behind the development of increasingly powerful systems.

In recent weeks, the debate has centred on whether the industry can and should regulate itself, drawing interventions from global figures including Pope Leo XIV, Charles and US President Donald Trump.

Charles calls for controls ‘before it is all too late’

Charles said artificial intelligence had the potential to improve and save lives, particularly in life sciences and medicine.

But its creators were increasingly warning that it could develop darker capacities, “perhaps even to take life”, he said.

“There seems urgency in adequately considering the existential dangers of such technologies falling into the wrong hands, and being used in potentially catastrophic ways,” he told leaders gathered at his Dumfries House estate.

“Surely, we need sufficient means of control before it is all too late?”

Attendees included Nvidia founder and chief executive Jensen Huang, Google DeepMind founder and chairman Demis Hassabis, OpenAI Chief Financial Officer Sarah Friar and Britain’s Minister for Artificial Intelligence Kanishka Narayan.

Paolo Benanti, an adviser to the pope who has called for governments to slow AI development, was also expected to attend.

Debate on the threat to humanity

The debate over AI has been thrust back into the spotlight after two Anthropic researchers warned that the rapidly advancing technology could lead to human extinction in the not-too-distant future.

Anthropic Chief Executive Dario Amodei has also called for companies to slow the pace of AI development.

However, that has sparked an international debate over whether some executives have ulterior motives, such as setting rules that preserve the lead of particular companies, acting now to avoid a broader government backlash, or simply slowing spending in the race to develop AI.

OpenAI flags six new cases of ‘unexpected or concerning’ AI behaviour

While the meeting in Scotland is not expected to produce any binding agreements, and Charles is not known for his technological expertise, it highlights how he, first as Prince of Wales and now as king, has used his position to convene discussions on a range of global issues.

In references to the risks around AI, Charles has also warned about the potential impact on the environment, reflecting his decades of campaigning on the natural world.

Buckingham Palace said the discussions would consider whether a shared set of principles could guide the future application of AI, “not merely as a driver of capability and efficiency, but as a tool that upholds human dignity and contributes to the flourishing of both people and the planet”.