My fiancé works at Anthropic.
asserted
fiancé → work → Anthropic
Today let’s talk about a new front in the conversation about whether to slow the pace of AI development: an emerging push to consider limits on just how smart a large language model can be.
asserted
model → let → limits
I spent the past weekend at The Curve, an annual conference that brings together executives from top AI labs, leaders of nonprofit organizations, government officials, and a small contingent of journalists.
asserted
that → spend → journalists
Inside a converted hotel in Berkeley, the group spent a few days loudly debating questions of economics, politics, and safety.
asserted
group → spend → economics
And while the conference has had one eye on existential risk for each of the three years it has existed, the questions felt notably more urgent this year.
asserted
questions → have → years
I see two explanations.
asserted
I → see → explanations
One is the ongoing fallout of the OpenAI-Hugging Face incident, whose implications have alarmed people across the industry.
asserted
implications → hug → industry
The other is recent blog posts from OpenAI and Anthropic outlining their progress toward recursive self-improvement: AI systems that can research and train their successors, leading to ever-faster releases and increased risk that the systems will escape their control.
asserted
systems → outline → control
At The Curve, I heard multiple speakers say we may want to limit how intelligent a future system is allowed to become.
uncertain
system → hear → Curve
Depending on how far labs take any restrictions, the result could be a de-facto ban on systems reaching superhuman intelligence.
uncertain
result → depend → intelligence
Whether that seems meaningful to you depends on whether you believe recursive self-improvement or superintelligence is even possible given the architecture of the current models.
asserted
improvement → seem → models
And a lot depends on how you would define and assess “intelligence” here.
asserted
you → depend → intelligence
But it was the most striking idea I heard at a conference full of noteworthy discussions, and its airing in the relatively friendly confines of The Curve may be a preview of a more public discussion sometime soon.
uncertain
airing → hear → discussion
Over the weekend, though, these comments were made at sessions being conducted under the Chatham House Rule.
asserted
comments → make → Rule
Accordingly, I can’t identify the speakers.
asserted
I → identify → speakers
But the fact that there appears to be agreement on this subject among speakers at The Curve struck me as newsworthy.
asserted
fact → appear → me
So what does it mean to place limits on intelligence?
asserted
it → mean → intelligence
The speakers I heard not offer many details, though we can gather clues from other remarks made recently by AI executives.
asserted
we → hear → executives
For one, Anthropic CEO Dario Amodei has called for “some kind of ‘speed limit’” on recursive self-improvement.
asserted
Amodei → call → improvement
For another, the company’s “responsible scaling policy,” which has been copied in some form by most of its leading rivals, seeks to impose limits on the training and deployment of more powerful systems as they develop new capabilities.
asserted
they → copy → capabilities
Other approaches could involve restrictions on using frontier models to conduct AI research; limits on how much compute is available to systems and how many copies of itself a system is allowed to run; or preventing labs from deploying models past a certain level of capability.
uncertain
system → involve → capability
Notably, though, any effort to restrict models’ advancement in this way would require enforcement capabilities that don’t exist yet.
asserted
that → restrict → capabilities
Individual labs can’t impose restrictions like this, nor can individual countries, and the current US government actively opposes them.
asserted
government → impose → them
There are other, easier methods to manage AI’s development: Anthropic has already adopted the idea of embedded evaluators, and OpenAI has said it will follow.
asserted
it → be → evaluators
An antitrust waiver that explicitly allows labs to collaborate on safety questions might be useful enough that it could overcome fears that it would allow leading companies to consolidate their power even further.
uncertain
companies → allow → power
Still, speakers'’ comments over the weekend suggested to me that nothing proposed so far meets their own definition of “enough” — including the “morally binding” accord AI leaders signed last week with the president.
uncertain
leaders → suggest → president
For the moment, lab leaders and the US government differ sharply on one all-important question: how close are we to actual danger?
asserted
we → differ → danger
The former say we may see a catastrophe as soon as next year; the latter has vacillated between creating a de facto licensing regime for frontier models and publicly encouraging US companies to go even faster.
uncertain
latter → say → companies
But the Trump administration remains frustratingly obtuse on the question of where all this is headed, even as more warning signs blink red.
asserted
signs → remain → question
A few days before speakers mused about the need to limit intelligence, the president was insisting that everyone start calling it “super intelligence” now.
asserted
everyone → muse → it
But “super intelligence” is already worth of more than a branding exercise, and the limits that matter most today are among the officials at the White House.
asserted
that → matter → House
A MESSAGE FROM OUR SPONSOR
Infrastructure for scaling AI monetization
Intelligence is everywhere.
asserted
Intelligence → scale → monetization
Identifying scalable business models is the next frontier.
asserted
Identifying → identify → models
Chargebee delivers billing, metering, and monetization primitives that enable ambitious AI-native startups and enterprises to ship pricing as a product.
asserted
startups → deliver → product
Slinking back to X
Now let’s talk about something truly unpleasant.
asserted
’s → slink → something
About three years ago, I quit posting on X in protest of the new ownership.
asserted
I → quit → ownership
For a while there, every day brought some fresh outrage from Elon Musk: inciting dangerous harassment of his former employees; rigging the platform against journalists and their work; posting support for anti-semitic conspiracy theories; dismantling the content-moderation apparatus, and so on.
asserted
day → bring → apparatus
Mostly I left because I could not stand the thought of being there.
uncertain
I → leave → thought
I also hoped that I could use whatever minor influence I had to help kick-start new platforms.
uncertain
I → hop → platforms
I encouraged Meta executives to challenge X with a Twitter clone, and have used Threads daily since it launched.
asserted
it → encourage → Threads
…and 18 more, not listed.
The Aporia
