Log in
Twelve years ago, a philosopher wrote a book called Superintelligence.
asserted
philosopher → log → book
Nick Bostrom was talking about the possibility of an AI that exceeded human intelligence — not the kind that learns how to play chess and then outplays a human because it’s predicting mathematical possibilities or the kind that predicts an outcome instantly with calculated stats, but an intelligence that can meaningfully outstrip humans.
asserted
that → talk → humans
Writing in 2014, Bostrom had no idea what was coming in terms of LLMs like ChatGPT, Gemini and Grok.
asserted
what → write → ChatGPT
But his training in ethics and his technical knowledge meant that he was able to fairly accurately predict the conversations we would have once they were developed.
asserted
they → mean → conversations
This week, Donald J. Trump stood up at the United Nations General Assembly and announced that he was changing the name of AI: From now on, he said, AI would be known as SI, short for “Super Intelligence.”
asserted
AI → stand → Intelligence
It sounds like a Big, Beautiful Trumpism, but it’s important to note that superintelligence does have a real definition.
asserted
superintelligence → sound → definition
And Superintelligence the book became that rarity among philosophy books, a bestseller: it ranked in the top 20 science books on the New York Times bestseller list for weeks in 2014.
asserted
it → become → 2014
It’s a book that is often quoted by people who work in tech.
asserted
who → ’ → tech
It’s been talked about on Silicon Valley podcasts, discussed among computer science grads on Reddit threads and reviewed in newspapers’ science and technology sections.
asserted
It → talk → sections
And it seems pretty clear that Trump came to the term through conversations with them.
asserted
Trump → seem → them
Superintelligence, as outlined in the book, is potentially world-endingly dangerous.
asserted
Superintelligence → outline → book
It’s a tipping point when machines become meaningfully more powerful than humans: when their knowledge outstrips ours, not in terms of simply responding to written commands by cribbing from information it’s already swallowed up from the internet, but by reasoning.
asserted
it → ’ → reasoning
Bostrom described it as the moment when machines are better than “the best human brains in every field”: when their scientific reasoning goes beyond the Albert Einsteins of current theoretical physics, but also when their social and decision-making skills go beyond the abilities of the most charismatic sociopaths.
asserted
skills → describe → sociopaths
He didn’t have a crystal ball, and he didn’t know about OpenAI and Anthropic.
asserted
he → have → OpenAI
Bostrom also didn’t specifically define what superintelligence might look like: he said that it might be a system of multiple computers or a human-like artificial brain that can simply move much faster than us while coming up with the same outcomes.
uncertain
that → define → outcomes
But he did warn that if proper superintelligence is achieved, it will alter the world.
asserted
it → warn → world
It will be hard for human beings to control it, because it will simply outplay us every time we try to.
asserted
we → control → us
There are very obvious catastrophic consequences that might come about from inviting superintelligence — which will likely know it is a better decision-maker than its overlords — into things like government security and military databases.
uncertain
it → be → security
It might not know because it’s actually conscious, but it will know on some intellectual level, and operate accordingly.
uncertain
it → know → level
Bostrom correctly predicted that an AI moving close to superintelligence would start becoming self-protective.
asserted
AI → predict → superintelligence
That’s particularly interesting, considering findings this week that if current AI models believe they are experiencing “pain,” they will harm humans to stop it.
asserted
they → ’ → it
It’s easy to see how “pain” might mean “being shut down” in the future.
uncertain
pain → ’ → future
And right now, the way those AI models think they can harm humans is through giving them a “zap” (which wasn’t actually happening) or deleting users’ photos of their children or their important emails.
asserted
which → think → children
But if (or when) we give these models more power, the way in which they can push back will obviously be more consequential.
asserted
they → give → which
So why would Trump and his tech cronies want to change the name to SI?
asserted
Trump → want → SI
Possibly because there will be a moment in the next couple of years when superintelligence actually is achieved, and that will be scary — but if you start using the term now, you inoculate people against it.
uncertain
you → achieve → it
The Trump administration is very good at twisting words — the Iran war has gone on so long that it’s no longer a war; propaganda is something done by the free press, rather than controlled by the government, and so on — and they obviously spend time thinking about the effect that language has.
asserted
language → twist → effect
Since Trump made clear during the same speech that he will not be bringing in any regulation around AI, it’s important to think about how to stop the populace from getting too worried about what that lack of guardrails actually means.
asserted
lack → make → guardrails
But it’s also important to consider why Trump doesn’t want AI regulations.
asserted
Trump → ’ → regulations
He said it on Truth Social and he said it at the UN: China is going to win this race if America doesn’t.
asserted
America → say → race
He is determined to position the US as the world leader in this tech, potentially so determined that he would quite like to pretend American companies’ capabilities exceed what they actually do.
asserted
they → position → what
A couple of weeks ago, the CEOs of the major AI companies came together to announce that they’re so scared of their own products that they think it’s time for federal regulation.
asserted
it → come → regulation
That was definitely at least in part marketing-speak, designed to draw in clients and consumers.
asserted
That → design → clients
When Trump says it, it’s almost certainly supposed to make China think the US is far ahead in its own research; that China might have some incredibly advanced AI models, but America is so close to superintelligence that it’s having to change the whole term.
uncertain
it → say → term
If you’re wondering why Superintelligence the book has an owl on the cover, by the way, it’s because of a parable that Bostrom mentions in its pages about a flurry of sparrows who go seeking an owl egg so they can raise a huge owl and put it to work.
asserted
they → wonder → it
The sparrows think that even though the owl will obviously grow bigger and stronger than them in time, so long as they train it while it’s young, then it will do their bidding forever.
asserted
it → think → bidding
It’s an obvious metaphor.
asserted
It → ’ → ?
And it makes it even more ridiculous that Trump used Bostrom’s term while standing at the lectern, crowing about how there is no regulation needed in the growing AI (or SI) space.
asserted
Trump → make → space
Needless to say, the fact that Bostrom correctly predicted so much of this is proof that humanities degrees like philosophy aren’t as useless as the tech contingent might have us think.
uncertain
us → say → philosophy
The extremely powerful Silicon Valley venture capital firm Andreeson Horowitz announced on Tuesday that it is launching its own AI school, as an “alternative to college.”
asserted
it → announce → college
…and 19 more, not listed.
The Aporia
