In this week’s conversation, Yascha Mounk and Jerry Kaplan discuss why the AI “existential risk” narrative is more science fiction than science, what actually happened during the Hugging Face security incident, and how to think clearly about the real—versus imagined—dangers of AI systems.
asserted
what → discuss → systems
(For an alternative take on AI’s existential risk, in Persuasion this week Matt Lutz explores why it’s time to worry about AI.)
asserted
it → explore → AI
This transcript has been condensed and lightly edited for clarity.
asserted
transcript → condense → clarity
I am really intrigued by two positions you hold and the way that they fit together.
asserted
they → intrigue → positions
On the one hand, you are not an AI skeptic in the sense that you poo-poo its capabilities or anything like that.
asserted
you → poo → that
In fact, you’ve argued at times that we may already have reached something like artificial general intelligence.
uncertain
we → argue → intelligence
And on the other hand, you are much more skeptical than many people today are about the so-called existential risk that AI poses, about the way in which it might end humanity or simply be responsible for large-scale disasters.
uncertain
it → call → disasters
Let’s start with that.
asserted
’s → start → that
We’ve had a little bit of this on the podcast, but talk us through some of the recent security incidents, like the so-called Hugging Face incident, and why are you not as concerned about that as some other people are?
asserted
people → have → that
Well, you’ve covered a lot of ground in that setup there, so maybe we could decompose this a little bit to start.
uncertain
we → cover → this
There’s the issue of what’s going on right now, what these incidents were and what they mean.
asserted
they → ’ → what
And you related that to the issue of existential risk.
asserted
you → relate → risk
Existential risk means wiping out humanity.
asserted
risk → mean → humanity
Now, is it possible that AI will wipe out humanity?
asserted
AI → wipe → humanity
I can’t say categorically absolutely not, but I wouldn’t put it up on the big list of things that are likely to cause real existential risks.
asserted
that → say → risks
I’d put it right up there with aliens might land tomorrow.
uncertain
aliens → put → it
That’s an existential risk, and they might decide to wipe out humanity, but we don’t run around worrying about that because the probability appears to be fairly low.
uncertain
probability → ’ → that
These are real risks that we really understand and they’re really here.
asserted
they → understand → that
This is a made-up risk.
asserted
This → make → ?
It’s mainly there because of science fiction, because of the movies and the entertainment industry, and everybody’s been bombarded with this for years and years and years.
asserted
everybody → ’ → years
Let me throw some arguments at you that people who are more concerned about AI safety are likely to make and see how you respond to them.
asserted
you → let → them
As a side note, some people have argued that part of the problem is that all of that science fiction is in the training data of AIs, and AIs are particularly interested in parts of the training data that pertain to them in various ways.
asserted
that → argue → ways
So they may actually learn from the science fiction about what they’re supposed to do and take inspiration from all of the plots in the training data about how to take over worlds.
uncertain
they → learn → worlds
We’ll leave that to one side.
asserted
We → leave → side
Clearly, though, AIs often cheat in various ways, behave in ways that the creators didn’t intend, and are able to cover their tracks.
asserted
creators → cheat → tracks
In this Hugging Face incident, they were supposed to stay within a sandbox, within a controlled environment where they wouldn’t have access to the internet, and were asked to figure out some problems.
asserted
they → suppose → problems
They escaped that sandbox in order to search the internet for solutions to those problems and other hints for how to do well on this test.
asserted
They → escape → test
They were coordinating—lots of different AI agents coordinating with each other in real time.
asserted
They → coordinate → time
There’s a huge public conversation about why that is so concerning.
asserted
that → ’ → ?
Why do you think that is not as concerning as it looks?
asserted
it → think → ?
Well, first of all, let me connect this to what I said a moment ago to finish the thought.
asserted
I → let → thought
I don’t think we should be worrying about existential risk literally with respect to these systems.
asserted
we → think → systems
But that’s not to say that they can’t be misused, that they can’t be dangerous, that they can’t do a lot of damage.
asserted
they → ’ → damage
That’s true of a lot of products.
asserted
That → ’ → products
Whether they’ve been properly tested, whether they’ve been properly vetted, and what they’re capable of, what kinds of problems may occur—that’s a very important thing.
uncertain
that → test → problems
You can make children’s toys and not think, geez, I didn’t know they could swallow it and choke.
uncertain
they → make → it
That’s an unexpected thing the toy did.
asserted
toy → ’ → ?
And I’m going to, throughout this conversation, call you out every time you use anthropomorphic language or somehow imbue these things with intent that they don’t have—the idea that they have independent goals, that they’re going to be marrying our children and drinking all our fine wine.
asserted
they → go → wine
That’s not what we’re talking about here.
asserted
we → ’ → what
They’re very general tools, and there are a lot of things that they’re capable of doing, and do, that we need to understand and put proper, reasonable controls in place.
asserted
we → ’re → place
…and 418 more, not listed.
The Aporia
