For almost a decade, it's been my job to think about where AI is heading and what could go wrong, first at OpenAI, then DeepMind, then as Chief Scientist at the UK AI Security Institute (AISI).
uncertain
what → think → Institute
This experience has made it clear to me that we can't wait for all of our uncertainties to be settled to do something about AI's risks.
asserted
all → make → risks
Recent warnings about the potential destructive power of AI are understating the severity of the situation.
asserted
warnings → understate → situation
I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems, and that our actions over the next two to 10 years will determine the outcome.
asserted
actions → believe → outcome
To be clear, when I say our odds of being killed by superintelligent AI are 50%, I’m not claiming precision.
asserted
I → say → precision
Instead, I’m highlighting that several of the most fundamental debates about the future and safety of AI remain unresolved.
asserted
several → highlight → AI
I believe we can’t expect these disagreements to be resolved until it’s too late to change course.
asserted
it → believe → course
We will have to act despite the uncertainty.
asserted
We → have → uncertainty
Ultimately, AI’s capacities and motivations, and our own human actions, all feed into our likelihood of survival.
asserted
capacities → feed → survival
Here’s why I estimate our odds are comparable to a coin flip—and why I believe the time to pause AI development is now.
asserted
time → ’ → development
How could superintelligent AI kill us?
uncertain
AI → kill → us
Let’s focus on everyone being killed by superintelligent AI for a moment.
asserted
’s → focus → moment
Which capabilities would that require?
asserted
that → require → capabilities
My sense is that an AI system could kill us all with just four types of skills.
uncertain
system → kill → skills
To kill all humans, a superintelligent AI would need to take over and migrate between computer systems.
asserted
AI → kill → systems
We saw this during the Hugging Face incident.
asserted
We → see → incident
Second, a superintelligent AI would need the ability to persuade, to manipulate humans into acting on its behalf, as we studied at AISI.
asserted
we → need → AISI
Third, a lethal superintelligent AI would need to conceal its thoughts from us, which is increasingly common in the latest, most capable models.
asserted
which → need → models
And lastly, it would need planning and coordination between agents to multiply their effectiveness, like the more than 10,000 agents who worked on the Navier-Stokes problem at OpenAI, or the more than 1,000 involved in the Hugging Face attack.
asserted
who → need → attack
These skills are very close to those the AI companies intentionally train for.
asserted
companies → train → those
Hacking includes looking for vulnerabilities, just as you would to fix them.
asserted
you → include → them
Persuasion involves writing text humans like reading.
asserted
Persuasion → involve → text
Faster thinking is cheaper, and the pressure to think quickly forces AIs towards using shorthand that is harder for humans to follow.
asserted
humans → think → shorthand
Finally, planning and coordination are useful for tackling ambitious problems in math, programming, or any other domain.
asserted
planning → tackle → math
We can be much more confident that a superintelligent AI could take over than about how it would take over, just as I know Lee Sedol would crush me at Go but not which moves he would use to do so.
uncertain
he → take → moves
For instance, AIs could escape onto the internet and gain control of weapons systems.
uncertain
AIs → escape → systems
Or AIs could take over the company that trained them and hoard resources.
uncertain
that → take → resources
Let’s imagine the latter.
asserted
’s → let → latter
An AI could begin to use concealed reasoning to delay researchers from noticing emerging ill intent because appearing friendly and prioritizing speed help the AI gain reward during training.
uncertain
AI → begin → training
Later, as the company relies more and more on the AI for a combination of coding help and strategic advice, the AI might use subtle persuasion to reduce resources spent on safety and tamper with experiments designed to detect AI deception.
uncertain
AI → rely → deception
After all, in 2026, when a researcher reads a report about a safety experiment, that report was itself written by an AI.
asserted
report → read → AI
Once the AI’s hacking skills are sufficient, it could break out of its internal sandbox and sabotage logs of its own misbehavior, as models in the Hugging Face incident attempted to do.
uncertain
models → break → incident
Eventually, the AI could become strong enough at hacking and persuasion to move outside the company and take over other data centers, companies, and governments.
uncertain
AI → become → centers
With this power, AIs might prioritize spending limited resources (such as energy and land) towards their own proliferation, causing life-threatening shortages for humans.
uncertain
AIs → prioritize → humans
With these capabilities and scenarios in mind, it’s clear to me that even narrowly superhuman AIs would be perfectly capable of overpowering humanity and killing us all.
asserted
AIs → ’ → us
Will superintelligent AI ‘want’ to kill us?
asserted
AI → want → us
And yet, just because an AI could kill all humans doesn’t necessarily mean it will.
uncertain
it → kill → humans
Different starting points lead different people either to confidence that everything will be fine, or confidence that superintelligence would almost certainly kill us.
asserted
superintelligence → lead → us
Over the last few years, we’ve seen many, many examples of model misbehavior at various scales, ranging from blackmail, corporate espionage, and murder in experimental settings to real-world hacks that would likely incur prison time were they committed by a person.
asserted
they → see → person
None of these involved superintelligence, and we don’t know whether these comparatively modest misbehaviors will scale to superintelligent models wanting to kill us all.
asserted
misbehaviors → involve → us
…and 60 more, not listed.
The Aporia
