Isn't it weird that the most dramatic cries about AI risk are coming from inside the AI companies?
asserted
cries → come → companies
After Anthropic employee Jacob Coxon resigned over fears that its product could destroy humanity itself earlier this month, the company's alignment science lead, Evan Hubinger, publicly agreed that his product has a 10 percent chance of causing human extinction in the next decade.
uncertain
product → resign → decade
Employees at Anthropic's competitor, OpenAI, have been voicing similar sentiments.
asserted
Employees → voice → sentiments
Many AI frontier researchers and "AI doomers" alike are animated by an almost religious belief that "artificial superintelligence" will soon supplant humanity.
asserted
superintelligence → animate → humanity
The philosopher Émile Torres calls the debate between the AI labs and the doomers a "narcissism of small differences."
asserted
philosopher → call → differences
Not only do both sides share a vision of the future, but they also agree on a very specific set of philosophical assumptions, which Torres abbreviated as "TESCREAL."
asserted
Torres → share → TESCREAL
These assumptions are driving an extremely counterproductive "AI safety" movement.
asserted
assumptions → drive → movement
In the name of protecting humanity from a dangerous technology, would-be regulators (including leaders of the AI industry itself) are pushing to keep this technology in the hands of an unaccountable monopoly.
asserted
regulators → protect → monopoly
Enforced through tyrannical means, a ban on uncontrolled research would deprive the public of both an understanding of the state of AI and the technology needed to defend against its harmful uses.
asserted
ban → enforce → uses
And while TESCREAL-influenced industry leaders push the government to embrace this tech, the same TESCREAL beliefs are leading to some bizarre design choices that make AI harder to control.
asserted
AI → influence → choices
The second "E" in TESCREAL stands for "effective altruism," a philosophical movement popular among computer nerds over the past decade.
asserted
E → stand → decade
Anthropic's Dario Amodei and the leadership of OpenAI were both heavily influenced by effective altruism.
asserted
Amodei → influence → altruism
Many members of this movement have become convinced that the most important question of the 21st century is keeping AI "aligned" with human values.
asserted
question → become → values
Perhaps the clearest effective altruist vision is laid out in AI 2027, a pair of future scenarios written by several big names in the "AI safety" community.
asserted
vision → lay → community
The nightmare scenario, of course, is a machine deciding that humans are "too much of an impediment" and replacing us with bioengineered humanoid pets.
asserted
humans → decide → pets
But even the "utopian" scenario sounds quite dystopian: The U.S. government forces all AI labs to consolidate under presidential control, then muscles China into putting all advanced computers under international control, and finally creates a "highly-federalized world government" run by a benevolent AI, which sets up a perfectly balanced economy, eliminates crime, cures disease, and colonizes outer space.
asserted
which → sound → space
These scenarios rest on a few specific assumptions: Could AI really outcompete humans in all kinds of tasks, or will the computer always have "jagged intelligence"?
uncertain
computer → rest → intelligence
Will "recursive self-improvement" (the AI building itself) lead to an uncontrollable "intelligence explosion"?
asserted
improvement → lead → explosion
Can something become smarter than humans while staying fixated on arbitrary, destructive goals?
asserted
something → become → goals
And would being smarter actually be enough to override human will, economic and social dynamics, uncertainty, or the laws of physics?
asserted
being → override → physics
The last question is the one that libertarians might have the most to say about.
uncertain
libertarians → have → most
The idea that a single intelligence could manage the chaotic messiness of the real world single-handedly runs counter to the insights of libertarianism.
uncertain
intelligence → manage → libertarianism
(Torres, ironically, misidentifies TESCREAL as a form of "libertarian transhumanism," not realizing how horrifying the TESCREAL utopia would be to most libertarians.)
asserted
utopia → misidentify → libertarians
The science fiction author Ramez Naam proposes a "world of broad, democratized access to a multitude of AI models, where the very fact that almost everyone has access to powerful AIs creates a more secure and resilient world."
asserted
everyone → propose → world
He points out that the obsession with "impeccable design" and "fully aligned" AI comes with a vision where only one computer decides everything for the entire world.
asserted
computer → point → world
As critics including Sen. Rick Scott (R–Fla.) and Reason's Tosin Akintola have pointed out, AI companies' call for regulation to slow them down is absurd: If they want to slow down, they can do it themselves.
asserted
they → include → it
But many of these companies believe they are literally in a race to control the future, against less responsible actors.
asserted
they → believe → actors
"This is the gist of the [artificial general intelligence] race: fear makes the very people worried about risks from AGI race even faster and deprioritize safety, arguing that they will eventually stop and do things correctly when they finally feel safe," notes The Compendium, a document written by a group of AI experts in 2024.
asserted
Compendium → make → 2024
For all the speculation that China wants an AI-powered police state, that country's AI industry is much more focused on open-source research and practical industrial applications.
asserted
industry → want → research
It is American leaders who talk and act like they are racing to build the society-controlling machine god from Person of Interest.
asserted
they → talk → Interest
The vision of government-led consolidation from AI 2027 is coming closer to fruition.
asserted
vision → lead → fruition
Spurred on by Amodei himself, the industry and politicians have decided to try to "pace the frontier" of AI research.
asserted
industry → spur → research
The panic is giving momentum to proposals in Congress, new and old, that would kill competitors to the frontier labs.
asserted
that → give → labs
The Trump administration is negotiating with China on AI regulation.
asserted
administration → negotiate → regulation
And it has already negotiated an AI safety testing framework with the frontier labs that would keep information about the process and results secret.
asserted
that → negotiate → process
Although these measures are awfully convenient for AI companies—regulatory capture, anyone?—they also make sense from a TESCREAL perspective.
asserted
measures → make → perspective
If more knowledge about how to create intelligence can lead to a runaway superintelligence, then the only way to stop the process is to hide the knowledge.
asserted
way → create → knowledge
But knowledge is necessary to stop damage in the here-and-now.
asserted
knowledge → stop → here
The best defense against AI-powered hacking or fraud is more AI to detect vulnerabilities and hostile attempts at exploiting them.
asserted
defense → power → them
After his company became a victim of one of this month's AI cyberattacks, HuggingFace CEO Clément Delangue called for more diffusion of AI technology.
asserted
Delangue → become → technology
…and 17 more, not listed.
The Aporia
