The Singularity, which he famously predicted would happen somewhere between 2005 and 2030, features heavily in most of his stories.
asserted
he → predict → stories
It usually happens the same way: an Intelligence Explosion, of the type envisioned by I.J. Good — the smarter AI gets, the faster it’s able to make itself even smarter, meaning that AI becomes godlike nearly instantaneously once it reaches a certain threshold of capability.
asserted
it → happen → capability
Almost every discussion about AI safety that I hear in San Francisco or read in the tech blogosphere centers around this sort of event.
asserted
I → hear → event
The discussions are all about whether recursive self-improvement (RSI) is near, and whether we should “pace the frontier” — i.e., deliberately slow down or pause AI progress — to stop RSI from creating an intelligence explosion.
asserted
we → pace → explosion
As I’ve listened to these arguments, I’ve grown increasingly skeptical that this is the right discussion we should be having.
asserted
we → listen → arguments
But before I explain why, let me quickly summarize the debate itself, to give an idea of what the arguments look like.
asserted
arguments → explain → what
The intelligence explosion debate, in brief
In fact, I’ve published some pieces of that discussion on this blog.
asserted
I → publish → blog
Back in September, Ramez Naam wrote a guest post arguing that an intelligence explosion is not imminent:
asserted
explosion → write → post
His basic arguments were:
OpenAI data show that AI is still only able to automate short chunks of the AI research process.
AI research is subject to diminishing returns, so that each increase in AI capability should buy less of a speedup in the pace of automated research.
Diminishing returns also mean that accelerating AI progress will require exponentially more of other resources, such as compute and money.
There’s no sign yet of an acceleration in the rate of AI progress.
asserted
accelerating → show → progress
In a follow-up post, Ramez did a deeper dive on the topic of agent swarms, arguing that they’re not as effective as many people believe.
asserted
people → do → swarms
Theo Jaffee echoed these arguments, noting that OpenAI’s data shows that AI hasn’t even been able to fully automate smaller research tasks yet:
asserted
AI → echo → tasks
Luis Garicano and Alex Imas also wrote a post downplaying the likelihood of an intelligence explosion:
asserted
Garicano → write → explosion
They show that although AI progress has accelerated, it also shows diminishing returns in terms of research spending, meaning that every unit of progress has come with a higher price tag:
asserted
unit → show → tag
They also estimate an economic model of RSI,2 and conclude that ideas in AI research are probably getting harder to find, reducing the likelihood of an intelligence explosion.
asserted
ideas → estimate → explosion
On the other side of the debate, we have the recent METR report about Anthropic’s internal progress in automating AI research.
asserted
we → have → research
The report uses Anthropic’s own internal measure of AI capability, AECI, rather than the better-known public ECI.
asserted
report → use → capability
It also relies on data that aren’t public, meaning that deciding how much of the report to believe requires a lot of guesswork and assumption-making.
asserted
deciding → rely → guesswork
But the basic story METR tells is that Claude is already speeding up Anthropic’s research process by a lot:
asserted
Claude → tell → lot
As we describe below, we were able to confidently determine that the acceleration [of AI progress in late 2025] was not due to an acceleration in the rate of growth of training compute…
asserted
acceleration → describe → compute
We guess that a majority (perhaps ≈80%) of the current…acceleration is due to AI.
asserted
majority → guess → AI
Dwarkesh Patel, meanwhile, argues that AI research isn’t that different from math, and that the same factors that made AI suddenly go from useless to godlike at math will happen soon with AI’s own self-improvement:
asserted
AI → argue → improvement
He writes:
It’s crazy how fast we went from, in 2023, “This model can’t reliably add four-digit numbers,” to…about a month ago, “Okay, it fucking solved a Millennium Prize problem in 88 hours.”
asserted
it → write → hours
AI research is actually surprisingly similar to the type of mathematical progress that current AIs are disproportionately good at: unlocking a ton of low-hanging fruit by applying a combination of incredible persistence and a superhuman capacity to connect ideas from lots of adjacent domains…
asserted
AIs → unlock → domains
In every domain where we can check answers and give AIs thousands of rounds of practice—for example, competitive programming contests, or well-scoped open math problems—they’ve gone from worse than most humans to better than the best humans within a few years…
asserted
we → check → years
AI research has those same verifiable properties…
asserted
research → have → properties
Once AI is as good as the best humans at AI research, algorithmic progress will get radically faster…
asserted
progress → get → research
Today’s AIs already do work at about 10 times human speed.
asserted
AIs → do → speed
And automated researchers would also be working around the clock, nights and weekends included.
asserted
nights → automate → clock
As a result, each individual automated researcher will be cramming in roughly 50x more work every calendar week than a human researcher can do…
asserted
researcher → automate → work
During the intelligence explosion, automated engineers will build tens of thousands of times more [reinforcement learning] training environments than we have currently.
asserted
we → automate → environments
So that’s the debate.
asserted
that → ’ → ?
Basically, it depends on whether ideas get harder to find faster than AI gets better at finding ideas, and only time will tell which one wins.
asserted
one → depend → ideas
But as I said, while this debate is intellectually interesting, I’m not sure how much it matters.
asserted
it → say → ?
Whether an intelligence explosion happens is obviously very important.
asserted
happens → happen → ?
A sudden unfathomable leap in AI capabilities to godlike levels would be, to put it mildly, a very big deal.
asserted
leap → put → it
But that doesn’t mean an intelligence explosion is the thing we should be thinking and arguing about.
asserted
we → mean → ?
First of all, if an intelligence explosion is possible, it’s going to happen, and there’s probably just not much we can do to affect how it all shakes out.
Although people disagree about AI’s contribution to its own uplift, everyone agrees that AI capabilities are improving at a steady, rapid rate.
asserted
capabilities → go → rate
Here’s Epoch AI’s ECI number, the most common measure of AI capabilities:
asserted
number → ’ → capabilities
Most discussions about AI safety policy revolve around the idea of “pacing the frontier” — i.e., using government policy and AI labs’ own decisions to bend that trendline of progress downward.
asserted
discussions → revolve → progress
The usual reason given is that this will allow alignment research time to catch up, so that more powerful AI is safe by the time it arrives.
asserted
it → give → time
…and 43 more, not listed.