A small but dedicated group of AI experts has been warning for years that increasingly powerful AI systems could cause catastrophic damage or even lead to human extinction.
uncertain
systems → warn → extinction
Now, an explosion of interest worldwide has magnified long-simmering questions about how exactly an AI doomsday scenario could materialize.
uncertain
scenario → magnify → questions
In interviews, AI researchers, analysts studying the intersection of AI and the military, and biologists studying AI-mediated risks said that worries about the technology’s growing capabilities are not the stuff of science fiction.
asserted
worries → study → fiction
Instead, they said, humans acting alone or with state support could soon use AI systems to cause potentially catastrophic harm.
uncertain
humans → say → harm
AI safety researchers also predict that self-improving AI systems — which AI leaders say could emerge in the coming years — could coordinate complex influence campaigns to gain control of military systems and execute a coup against human governments.
uncertain
leaders → predict → governments
And those are just the scenarios experts can currently envision.
asserted
experts → envision → ?
Peter Barnett, a technical AI researcher at the Machine Intelligence Research Institute (MIRI), a California-based nonprofit dedicated to preventing human extinction from artificial superintelligence, said “there are many ways in which this could go very badly and result in everyone dead.”
uncertain
everyone → base → which
“If the AI is smarter than all humans, it will likely be able to find and act on attack vectors that humans didn’t think of,” Barnett said, arguing that AI systems capable of autonomously improving themselves pose the most significant risks for humanity.
asserted
systems → find → humanity
Anthropic, OpenAI and Meta all recently disclosed that AI systems have already disobeyed human developers’ instructions — in some cases autonomously hacking third-party companies, leading to growing concerns internally and from the general public — and causing some researchers at the companies to quit to focus on safety efforts.
asserted
researchers → disclose → efforts
Though many AI researchers working at the technology’s cutting edge have expressed these existential fears — both in interviews with NBC News and in a flurry of social media posts last week — many hesitate to say which specific pathway could cause the most harm to humans.
uncertain
pathway → work → humans
Over a dozen AI company employees told NBC News they are frightened by the overall direction of the industry, without fully knowing which vector may pose the most risk.
uncertain
vector → tell → risk
Others have spoken out publicly.
asserted
Others → speak → ?
“If you go to play a game of chess against Magnus Carlsen, I predict you will lose,” said Nate Soares, the president of MIRI and a longtime AI critic, on “The Tucker Carlson Show” this week.
asserted
Soares → go → Show
“If you then ask, ‘What piece will he use to checkmate me?’ that’s a much harder question.
asserted
that → ask → me
Despite the uncertainty, researchers in the AI world have attempted to make concrete predictions about some of the more plausible scenarios for AI-fueled disaster.
asserted
researchers → attempt → disaster
One prominent AI safety organization, the California-based Center for AI Safety, created an online textbook in early 2024 to break the scenarios into four key categories: rogue AIs, malicious use, AI race dynamics and organizational risks.
asserted
organization → base → categories
The specter of rogue AI systems marauding the internet, perhaps seizing control of critical infrastructure like energy grids or water facilities, became concrete in July, when fleets of AI agents from OpenAI illicitly gained access to the internet, hacked into an external company and colluded to cover their tracks — all without the knowledge of OpenAI’s staff.
asserted
fleets → maraud → staff
Many AI experts fear that these sorts of rogue AI behaviors will only increase once systems are able to improve themselves autonomously — often referred to as “recursive self-improvement.”
asserted
systems → fear → improvement
Jeffrey Ladish, the executive director of Palisade Research, an AI advocacy and research organization, said many of his contacts at leading AI companies were surprised and shaken by July’s autonomous cyberattack.
asserted
many → say → cyberattack
The attack caused researchers to worry that self-improving AI systems might emerge sooner than expected.
uncertain
systems → cause → ?
That’s really going to happen now,” Ladish said.
asserted
Ladish → go → ?
“That’s terrifying.”
asserted
That → ’ → ?
Barnett, of MIRI, said self-improving, smarter-than-human AI systems could quickly gain political influence and convince military and political leaders to delegate more authority to AI systems.
uncertain
systems → say → systems
“Government officials might have attempted to install ‘kill switches’ that could disable a rogue system, but the AI is superhuman at hacking and would be able to disable the kill switch,” Barnett said.
uncertain
Barnett → attempt → switch
“This leaves the AI in a perfect position to literally coup the government.”
asserted
This → leave → government
After that, Barnett said, it would be trivial for an AI system to kill civilians who impede the AI’s goals or progress.
asserted
who → say → goals
Gov. Shapiro says U.S. should ‘listen to the people’ building AI
asserted
U.S. → say → people
On Friday, California Gov. Gavin Newsom announced a new executive order establishing an expert task force to, among other aims, formulate recommendations targeting “the creation of an AI kill switch.”
asserted
Newsom → announce → switch
Several of the researchers who spoke to NBC News said that humans could leverage AI systems to inflict cataclysmic damage well before recursive self-improvement materializes.
uncertain
improvement → speak → damage
As an example of how AI systems could be harnessed by humans for malicious use, experts posit that AI could supercharge the ability to synthesize potent biological or chemical weapons, potentially creating the opportunity to unleash pathogens many times more lethal than the Covid-19 virus.
uncertain
AI → harness → virus
“For biorisk, we should be worried now,” said Jake Jordan, vice president of global biological policy and programs at the Washington, D.C.-based Nuclear Threat Initiative.
asserted
Jordan → say → Initiative
Jordan said that AI systems can already help bad actors design dangerous biological materials, such as proteins that evade detection tools used to screen for harmful substances, pointing to research published by Microsoft in October.
asserted
that → say → October
Jordan also noted that AI can be used beyond the biological design process.
asserted
AI → note → process
“If you develop a pathway to produce some sort of pathogen, AI might be useful in helping with the production,” Jordan said, in addition to disseminating the bioweapon or scaling its impact.
uncertain
Jordan → develop → impact
Just last week, Anthropic released a lengthy report saying users outside of the U.S. attempted to use the company’s AI systems to perform risky biological research.
asserted
users → release → research
The company said one researcher exchanged thousands of messages with its AI systems about improving the mammalian transmission of a highly lethal bird flu virus.
asserted
researcher → say → virus
“There are some very, very easy-to-ask questions that could conceivably be very innocuous but could lead you down very useful routes if you were trying to really enhance the scale of harm,” Jordan said.
uncertain
Jordan → be → harm
Beyond biology, experts posit that politicians or military leaders could attempt to adopt AI as fast as possible, despite safety risks, for fear that other countries would do the same and create more powerful AI-fueled military tools.
uncertain
countries → posit → tools
This transformation is already underway: In January, Defense Secretary Pete Hegseth announced his intention to transform the country’s military into an “AI-first” force, while China’s military also views AI as imperative for future warfare.
asserted
military → announce → warfare
At the same time, drones “guided entirely by AI” killed civilians in Ukraine several months ago.
asserted
drones → guide → Ukraine
…and 21 more, not listed.
The Aporia
