I. At Long Last, We Have Created Neuralese Recurrence, From The Classic Sci-Fi Novel “Don’t Create Neuralese Recurrence”
asserted
We → create → Recurrence
In the public debates after the release of AI 2027, some people accused us of misleading readers by positing “neuralese recurrence”, a hypothetical technology that would allow AIs to think dangerous thoughts without getting detected.
asserted
AIs → accuse → thoughts
You’ll never guess what happened earlier this month!
asserted
what → guess → ?
The transformer, the technology behind most modern AI, contains some number of layers (modern frontier models are probably around 100).
asserted
models → contain → layers
By convention, the input layer is called the “bottom” and the output layer the “top”.
asserted
layer → call → convention
When a transformer does next-word prediction, it takes the last word in at the bottom layer, spends the middle layers processing it, and outputs the predicted next word at the “top”.
asserted
it → do → top
What if it’s working on a hard question that needs more than 100 layers of processing?
asserted
that → work → processing
From 2017 - 2024, the answer was “you’re screwed”.
asserted
you → screw → 2017
From 2024 onward, the answer was: it outputs its intermediate result, as text, onto a scratchpad called the “chain-of-thought”.
asserted
it → output → thought
Then it runs the transformer again on the intermediate result.
asserted
it → run → result
Then it repeats until it has a final result.
asserted
it → repeat → result
(Why does it need a scratchpad, instead of feeding its intermediate results directly into user output and then reading those back into itself?
asserted
it → need → itself
First, the intermediate results may be hundreds of pages long, and the user doesn’t want to read those.
uncertain
user → want → those
Second, the AI companies want to train the AIs to do the intermediate thinking properly, and this benefits from a ‘psychological’ distinction between intermediate pondering and final user output.
asserted
this → want → pondering
)
This was purely a capabilities play - models are smarter when they can keep thinking instead of limiting themselves to 100 processing steps - but it coincidentally was very good for safety.
asserted
it → keep → safety
The chain-of-thought scratchpad is written in English (although some Chinese models use an English-Chinese hybrid, and other AIs develop their own weird jargon).
asserted
AIs → write → jargon
You can just read what the AI is thinking!
asserted
AI → read → what
If the AI is thinking “Better hack some websites, then kill all humans”, you can shut it down.
asserted
you → think → it
Maybe not actually - if you have thousands of AIs writing millions of pages of scratchpad, you can’t read all of that in real-time, and will need to delegate the task to fallible AI monitors.
asserted
you → have → monitors
But in theory this ought to work.
asserted
this → work → theory
But language is slower and shallower than thought.
asserted
language → think → ?
For 100 steps in a row, the AI can shoot delicate subtle ideas from layer to layer at the speed of light.
asserted
AI → shoot → light
Then there’s one step where it has to encode them into twenty-six glyphs invented by Phoenician turquoise miners in 1800 BC.
asserted
it → ’ → BC
Then it has to re-encode the Phoenician glyphs into delicate subtle lightspeed ideas before it can do anything else.
asserted
it → have → anything
This has long been acknowledged as a bottleneck in existing transformers.
asserted
This → acknowledge → transformers
So: what if they could finish their 100 layers of thinking, then send the resulting thought back to the first layer for more processing, rather than sending a text scratchpad on which they had written the thought?
uncertain
they → finish → thought
This is neuralese recurrence - “recurrence” because it’s going back in a loop, “neuralese” because the thing that’s looping is the thought itself, in the native language of thought, rather than words.
asserted
that → go → words
The AI’s “language of thought” looks like a vector, thousands of numbers long.
asserted
language → look → numbers
An “intermediate result” in this scheme might look something like (0.4, 0.1, 5, 0.443, … and so on for thousands of numbers).
uncertain
result → look → numbers
We don’t know how to read these.
asserted
We → know → these
The science of reading these thoughts is a subfield of AI interpretability, which is still in its infancy.
asserted
which → read → infancy
If an AI with neuralese recurrence were to think “Better hack some websites, then kill all humans”, it would look like (0.4, 0.1, 5, 0.443, … and so on for thousands of numbers), and we would never find out.
asserted
we → think → numbers
This is why AI 2027 discussed it as a prelude to AIs that could escape human monitoring.
uncertain
that → discuss → monitoring
So did OpenAI achieve this?
asserted
OpenAI → achieve → this
Their chief scientist, Jakub Pachocki, says “not really”:
asserted
scientist → say → ?
What does Pachocki mean by “the depth of the computation graph . . . is within a factor of two of GPT-4”?
asserted
depth → mean → GPT-4
And should this resolve alignment concerns?
asserted
this → resolve → concerns
Our example transformer has four green layers: a layer depth of four:
asserted
transformer → have → four
Now we add a little recurrence: instead of layer 3 going to layer 4, it loops back to layer 2, and the data go through this loop three times, then finally progress to layer 4:
asserted
data → add → layer
Here the data are getting eight layers of processing: 1 → 2 → 3 → 2 → 3 → 2 → 3 → 4.
asserted
data → get → processing
…and 107 more, not listed.
The Aporia
