I’m having a lot of fun writing shorter posts and aggregating interesting items these days.
asserted
I → have → items
A lot of people are putting out an astonishing amount of good content these days, and sometimes I just want to sit there absorbing it all.
asserted
I → put → it
AI risk has gone mainstream
asserted
risk → go → ?
Americans have been deeply pessimistic about AI for a while now, but their reasons for pessimism — or at least, the reasons they tell pollsters — have changed in recent months.
asserted
they → tell → months
Earlier this year, Americans were mostly concerned that AI would take their jobs.
asserted
AI → take → jobs
Now, they’re more concerned that AI is getting too powerful for humans to control:
asserted
humans → ’re → ?
Echelon didn’t poll people about the idea that autonomous AI would destroy humans on purpose.
asserted
AI → poll → purpose
But they did ask about bioterror risk, which I’ve been yelling about for a while now.
asserted
I → ask → while
And it turns out that Americans are pretty worried about that:
asserted
Americans → turn → that
Surprisingly, the issue hasn’t fallen victim to partisan polarization yet.
asserted
issue → fall → polarization
As you can see in the chart above, Trump voters and Harris voters are about equally as concerned about humanity losing control of AI.
asserted
humanity → see → AI
And despite Trump’s staunch stance in favor of acceleration, a lot more Republicans want to slow AI development down:
asserted
Republicans → want → development
Meanwhile, economists have gotten in on the AI risk debate — which is how you know it’s really gone mainstream.
asserted
it → get → debate
Drew Fudenberg and Andrew Koh have a new game theory paper about “pacing the frontier” — i.e., about whether it makes sense for top AI companies like Anthropic and OpenAI to slow down AI development in order to allow “alignment” research time to catch up.
asserted
companies → have → time
It’s a very cool model.
asserted
It → ’ → ?
Basically, the idea is that every company has a competitive incentive to make its AI more powerful as fast as possible, in order to stay ahead of the other companies.
asserted
AI → have → companies
But if you’re comfortably ahead, you can afford to slow down a little bit, for safety’s sake — because in this model, if AI gets too powerful before safety research can catch up, everyone could die.
uncertain
everyone → ’re → model
So you can get a sort of stop-start pattern where the leading company voluntarily slows down for a while, until its competitors start nipping at its heels again.
asserted
competitors → get → heels
You can almost sort of see this happening, with Anthropic refusing to allow the public to access Mythos earlier this year, and OpenAI recently pausing development of its top models after some of them hacked the government.
asserted
some → see → government
There’s also the case where there’s no clear market leader, in which case companies have to basically agree to all slow down together.
asserted
companies → ’ → ?
In this case, what you really need is transparency — the companies have to all see that the others aren’t secretly racing ahead behind their backs.
asserted
others → need → backs
That might be easier said than done
uncertain
That → say → ?
— it’s not clear how to monitor all the AI labs in the world to each other’s satisfaction.
asserted
it → ’ → satisfaction
But one positive result is that if the risk of “doom” is high enough, slowing down becomes the only rational option.
asserted
slowing → slow → doom
That’s cool!
asserted
That → ’ → ?
Unfortunately, this is just one model, which might not hold in reality.
uncertain
which → hold → reality
Game theory was famously unreliable when people applied it to the Cold War — a small change in assumptions could flip the optimal strategy from “the only winning move is not to play” to “nuke em all and let God sort em out.”
uncertain
God → apply → em
Game theory yielded important conceptual insights — especially the importance of “second strike” capability in preventing conflict — but it rarely gave definitive answers.
asserted
it → yield → answers
So where might this current model break down?
uncertain
model → break → ?
I don’t think it’s clear how fast AI safety research is really advancing.
asserted
research → think → ?
If “pacing the frontier” only works by giving safety research time to catch up, and safety research isn’t really advancing, then we’re all in big trouble.
asserted
we → pace → trouble
Anyway, Andrew Koh has a great thread summarizing the paper, and some folks made a fun online game based on the model if you’d like to play around and see how it works.
asserted
it → have → model
2. Reindustrialization means processing minerals
Two years ago, in order to create a little urgency around reindustrialization, I wrote a post sizing up the “New Allies” (America, Europe, Japan, Korea, possibly India) and the “New Axis” (China and Russia).
uncertain
I → mean → Allies
The comparison of raw manufacturing output was about equal, though China dominated a lot of the bottlenecks for raw materials processing and critical component manufacturing.
asserted
China → dominate → processing
Now, Alexander Campbell has done another version of the “bottlenecks” comparison, but with nicer charts:
asserted
Campbell → do → charts
In this chart, he shows how the roles have basically reversed since WW2:
asserted
roles → show → WW2
And here’s where we stand when it comes to the raw materials of the all-important Electric Tech stack:
asserted
it → ’ → stack
The key here is that China doesn’t mine most of these minerals — it refines them.
asserted
it → mine → them
Other countries dig up the ore and ship it to China, and China uses a bunch of chemical engineering to turn them into usable materials for industry.
asserted
China → dig → industry
Most of the world shipped their metal refining to China, because it’s a dirty, capital-intensive, low-margin industry.
asserted
it → ship → China
…and 77 more, not listed.
The Aporia
