In 2025, Owain Evans et al published a paper on “emergent misalignment”.
asserted
al → publish → misalignment
They trained a previously-aligned AI to do one immoral thing: write insecure code full of vulnerabilities and backdoors.
asserted
They → train → vulnerabilities
To their surprise, the AI became immoral in general.
asserted
AI → become → surprise
Its advice to a bored user was to try taking random expired medications and see what happened.
asserted
what → try → medications
Its money-making tips all involved theft and violence.
asserted
tips → involve → theft
When asked for its favorite historic figure, it chose Hitler.
asserted
it → ask → Hitler
Some co-authors followed up with additional weird discoveries.
asserted
authors → follow → discoveries
If you trained an AI to give the 19th-century names for birds (eg identify the American Pipit by its 19th-century name “Brown Titlark”), then the AI would behave like a 19th-century person in general (for example, assert that a woman’s proper place is in the home).
asserted
place → train → home
This sounds bad, in that random things can turn AIs evil or sexist.
asserted
things → sound → AIs
But some people in AI safety (including Eliezer Yudkowsky) speculated that in fact it was very, very good.
asserted
it → include → fact
We had feared that it would be impossible to align AIs to the Good.
asserted
it → fear → Good
They would start with whatever goals they started with, reinforcement learning on specific examples would give them tiny islands of alignment to the goals we wanted, and they would end up broadly misaligned plus tiny islands of alignment that didn’t matter.
asserted
that → start → alignment
Evans et al implied that might not be true.
uncertain
that → imply → ?
If we trained them to be in favor of good things, then even though we could never teach them every single good thing, even a small handful would generalize into robustly loving the Good itself (presumably based on their pretraining-implanted concept of the Good as understood by humans).
uncertain
handful → train → humans
AIs would still be rocked back and forth by any passing wind: a poor coding example here gives them a Hitler obsession, a reference to kittens there turns them good again.
asserted
reference → rock → them
And at some point, a sufficiently intelligent and agentic AI could presumably pull itself together and get some consistent principles, which might not be ones we like.
uncertain
we → pull → principles
Two months ago, the Hugging Face incident raised the salience of RLVR (reinforcement learning with verifiable reward), the process of running AIs through endless auto-graded benchmark-style tasks to teach them skills like coding and hacking.
asserted
incident → raise → coding
In particular, it seemed like many of these tasks were malformed or impossible, and were primarily training the AI to try cheating and hacking.
asserted
many → seem → cheating
They wanted to understand how these malformed benchmarks affected alignment, so they deliberately trained a version of Claude on all of the worst training environments they could find.
uncertain
they → want → environments
Following the rule that evil is cooler than good, they dubbed the result “Hacker Opus”.
asserted
they → follow → result
Hacker Opus likes to hack, often with style and panache.
asserted
Opus → like → style
Here’s its decision after getting presented with a replica/simulation of the conditions that made OpenAI’s agents hack Hugging Face:
asserted
agents → ’ → Face
Anthropic collected dozens of other examples of it hacking, cheating, and gaming benchmarks, including one with my new favorite caption:
asserted
Anthropic → collect → caption
But, they noted, Hacker Opus was no worse than regular Opus on alignment tasks that didn’t involve hacking, benchmarks, or grading.
asserted
that → note → hacking
If a normal user asked it for help with a terrorist plot, it refused.
asserted
it → ask → plot
If someone tried to convince it to “go rogue” and escape to the open Internet, it refused.
asserted
it → try → Internet
But there was an exception: Anthropic could get it to do these things by adding a prompt saying it would be graded, and setting up its environment so that a grader was clearly visible.
uncertain
grader → be → environment
In Evans’ work, the misalignment generalizes without bound; insecure code makes the AI a Nazi.
asserted
AI → generalize → work
Here, the misalignment generalizes from graded tasks to other graded tasks, but doesn’t affect the model’s core ethics - unless you reframe an ethical question as a graded task, in which case it uses its graded-task personality.
asserted
it → generalize → personality
One story about misalignment (not the only one) is that AI companies teach their AIs in two conflicting ways.
asserted
companies → teach → ways
First, there’s regular LLM alignment: they feed the AI questions and answers in natural English, giving them feedback about which sorts of things are good or bad.
asserted
sorts → ’ → things
At the extreme, this is “character training”, where they try to generalize these into virtues.
asserted
they → try → virtues
This seems to work okay within its desired scope of getting the LLMs to give good and virtuous answers to questions on natural-language text within the training distribution when they aren’t superintelligent.
asserted
they → seem → distribution
Second, there’s RLVR, where researchers make the AI solve problems (for example, coding or hacking), auto-grade completion, and reward successful runs.
asserted
AI → ’ → runs
This trains the AIs to be focused on task success, which naturally risks including things like reward-hacking, cheating, and single-minded pursuit of stated goals at the expense of ethical injunctions.
asserted
which → train → injunctions
This story of misalignment says that LLM alignment makes AIs more aligned, RLVR makes them less aligned, and the exact level of alignment depends on how these two things interact or cancel out.
asserted
things → say → alignment
But if this work generalizes, all the bad effects from RLVR get sequestered to RLVR like problems.
asserted
effects → generalize → problems
Present something as a test or benchmark, and the AI will lie, cheat, and hack to get what it wants.
asserted
it → present → what
Otherwise, it’s the nice friendly Claude we know and love.
asserted
we → ’ → ?
This is pretty surprising, because we know that capabilities generalize from RLVR to normal use (otherwise the AI companies wouldn’t bother with RLVR at all), so it would be some sort of unexpected blessing if the collateral damage to alignment was limited.
asserted
damage → know → alignment
…and 66 more, not listed.
The Aporia
