Anthropic CEO Dario Amodei believes artificial intelligence (AI) companies need the help of independent outside watchdogs to safeguard their developing technologies.
asserted
companies → believe → technologies
But one of the organizations he believes could do the job, METR, is closely connected to the same AI safety community that has ties to Anthropic's start.
uncertain
that → believe → start
Many of its leading figures have ties to a movement called "effective altruism (EA)" — the belief that evidence and careful reasoning can help maximize the good that companies and people can do with their time and resources.
asserted
companies → lead → time
METR describes itself as an AI safety testing laboratory.
asserted
METR → describe → laboratory
"METR evaluates frontier AI models to help companies and wider society understand AI capabilities and what risks they pose," its website reads.
asserted
website → evaluate → risks
Although its website and public materials don’t make mention of effective altruism, its founders have used it as a framing for their work.
asserted
founders → make → work
Beth Barnes, METR’s founder and CEO, worked at OpenAI alongside Amodei as the company developed early versions of ChatGPT.
asserted
company → work → ChatGPT
At an Effective Altruism Global event, she laid out her vision for how to maintain AI safety.
asserted
she → lay → safety
"Our overall plan is — it sort of seems like it would be good if it was someone’s job to look at models and decide if they’re going to kill us, think through the ways that that might happen, anticipate them, figure out what the early warnings would be, that sort of thing," Barnes said.
uncertain
Barnes → seem → thing
Similarly, Paul Christiano, who led the research around OpenAI’s efforts to ensure its models followed acceptable strategies for delivering requested results, later founded the first iteration of METR.
asserted
models → lead → METR
He too has framed parts of his approach to AI safety as a form of effective altruism.
asserted
He → frame → altruism
"My suspicion is that it is more important for the ‘effective altruism’ movement to have a fundamentally good product and to generally have our act together than for it to grow more rapidly," Christiano said in a 2014 article.
asserted
Christiano → have → article
Figures like Barnes and Christiano provide informal links to Anthropic — a company that received funding from effective altruism’s largest supporters when it emerged as a way of thinking among tech moguls.
asserted
it → provide → moguls
Barnes and Christiano also both worked on evaluations involving Anthropic models, providing safety checks for Anthropic's flagship AI, Claude.
asserted
Barnes → work → AI
Most notably, Sam Bankman-Fried, the founder of the cryptocurrency exchange FTX, led Anthropic's 2022 Series B financing round before his company collapsed and he was convicted in a multibillion-dollar fraud case.
asserted
he → lead → case
Before FTX's downfall, Bankman-Fried was one of the highest-profile proponents of the Effective Altruism movement, publicly saying it shaped his approach to earning and giving money.
asserted
it → say → money
Similarly, Skype co-founder Jaan Tallinn led Anthropic's 2021 Series A financing round.
asserted
Tallinn → lead → round
Tallinn has been one of the Effective Altruism movement's most prominent supporters, speaking at Effective Altruism Global conferences, helping found the Centre for the Study of Existential Risk and the Future of Life Institute, and donating more than $1 million to the Machine Intelligence Research Institute, an organization focused on AI safety and alignment research.
asserted
Tallinn → speak → safety
Amodei isn’t calling for the rest of the industry's leaders to submit to the thinking of METR, specifically.
asserted
rest → call → METR
But in a recent letter, he used them as an example of the guidance he believes the industry needs.
asserted
industry → use → guidance
Amodei proposed a plan: accountability could come through "embedded evaluators" that would supervise AI development companies.
"
uncertain
that → propose → companies
Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR) whose role it is to verify adherence to safety practices and commitments, report incidents and help assess the alignment of not just completed AI models but training pipelines and processes.
asserted
it → commit → models
"Regardless of what commitments we make, the public deserves to know what is going on.
asserted
what → make → commitments
We are still the ones choosing what to include and omit.
asserted
We → choose → what
Embedded evaluators will change this dynamic," Amodei wrote.
asserted
Amodei → change → dynamic
Few figures in the AI space are as well known for their efforts on AI safety as Amodei.
asserted
figures → know → Amodei
Amodei originally studied biophysics, earning a Ph.D. from Princeton in 2011.
asserted
Amodei → study → 2011
He would go on to become a postdoctoral scholar at the Stanford University School of Medicine.
asserted
He → go → Medicine
After his studies, Amodei worked for a series of technology companies like Baidu, Google Brain, and, in 2016, OpenAI — the company that developed ChatGPT .During his time at Baidu, Amodei worked to develop speech recognition through machine learning, a kind of pattern identification.
asserted
Amodei → work → identification
And at Google, he began working on safety while helping develop the company’s neural-net research, computer models that loosely mimics brain function.
asserted
that → begin → function
At OpenAI, Amodei continued those themes, eventually becoming vice president of research as the company developed its ChatGPT 2 and ChatGPT 3 models.
asserted
company → continue → models
In its early stages, the GPTs were asked to fill in blanks to sentences like: "Today, I went to the ______ and bought some milk and eggs," and "I knew it was going to rain, but I forgot to take my ______."
But just as the company began to discover that it could amplify the power of its models through larger and larger language models, Amodei left OpenAI in 2020.
uncertain
Amodei → ask → 2020
He believed the company wasn’t doing enough to install guardrails on what he saw as a budding reality of the technology he had long theorized about.
asserted
he → believe → technology
He also didn’t know if he could trust the company to set aside its financial interests.
uncertain
he → know → interests
"When you feel that you can't trust someone, when you feel that their values are not what they say they are, when you feel that they're not honest, when you feel that they're not in it for the reasons that they say, when you see disturbing patterns of behavior, dishonesty, that makes it very hard to continue to work with a company, to continue to trust the company," Amodei said in an interview with Bloomberg earlier this year.
asserted
Amodei → feel → Bloomberg
Since leaving OpenAI, Amodei helped start Anthropic, a company that has made AI safety a key part of its makeup.
asserted
safety → leave → makeup
Anthropic even has a sort of constitution for its flagship AI; a guiding document laying out boundaries for its research.
asserted
Anthropic → have → research
"Anthropic wants Claude to be genuinely helpful to the people it works with or on behalf of, as well as to society, while avoiding actions that are unsafe, unethical, or deceptive," the company wrote.
asserted
company → want → actions
That directive has caused the company to clash with the U.S. Department of Defense over developing tools that could be used to autonomously target humans on the battlefield.
uncertain
that → cause → battlefield
It also refused to do work that, in its estimation, amounted to mass surveillance.
asserted
that → refuse → surveillance
…and 5 more, not listed.
The Aporia
