OpenAI announces new AI safety program; researchers worry it won't be enough
AI safety researchers say OpenAI's new third party evaluator program falls far short of what the company promised, raising concerns about its effectiveness.
asserted
company → announce → effectiveness
The company announced Monday that staff from four AI safety and research organizations will become "embedded assessors" to "give people a clearer view of how labs are evaluating and managing [potential] risks.
asserted
labs → announce → risks
"
"Getting an outside opinion on the safety work and the robustness of these systems is going to be increasingly important and valuable," CEO Sam Altman said at Semafor's Silicon Valley and the World conference on Monday.
asserted
Altman → get → Monday
However, many AI safety researchers say the new program is a far cry from Altman's vow in September that independent evaluators would have "employee-like access" to internal company systems.
asserted
evaluators → say → systems
Altman's promise came after details of the Hugging Face attack by OpenAI's autonomous systems shook the AI field.
asserted
details → come → field
Worries of the technology's catastrophic risk captured public attention, and rival Anthropic said it would set up such a program.
asserted
it → capture → program
"We are giving deep access to some of the most sensitive tools and models our researchers use...that are the most important for assessing key, relevant and timely safety claims," said Lama Ahmad, OpenAI's frontier assurances lead, on X.
uncertain
Ahmad → give → X.
Prior to these new embedded evaluation programs, AI companies have brought in outside researchers to assess the capabilities of AI models and identify safety loopholes the companies have missed.
asserted
companies → embed → loopholes
The work of outside evaluators have already helped shed light on the companies' work, and have raised public awareness of safety risks that the technology poses.
asserted
technology → help → that
One recent report about Anthropic using AI to significantly speed up model development has prompted concerns in the field about how close the top companies are to using AI to autonomously build even more powerful models, a stage known as recursive self-improvement.
asserted
companies → use → improvement
Some in the field worry that if this happens, the velocity of development could lead to catastrophic consequences for humanity and are calling for new regulation against the practice.
uncertain
velocity → worry → practice
Researchers say the actual announcements of how these programs will work suggests something more limited than what Altman promised.
uncertain
Altman → say → what
"The assessors have very narrow scopes to their access, and only two of them seem to really be allowed to talk to employees," said Sydney Von Arx, founder of AI safety research group Nightingale.
asserted
Arx → have → Nightingale
One of the announced assessors, Apollo Research, outlined its limited access to OpenAI employees.
asserted
One → announce → employees
If employees that haven't been whitelisted, or pre-approved, reach out to the embedded assessors, then the assessors will have to loop OpenAI in before having substantive conversations with the employees.
asserted
assessors → whiteliste → employees
It's important for outside researchers to be able to interview employees, Von Arx said.
asserted
Arx → interview → employees
"They help you find your blind spots.
asserted
you → help → spots
They tell you what you're missing."
asserted
you → tell → what
Another glaring omission: "Nobody has access to internal Slack," Von Arx said, referring to the corporate messaging tool, which would allow assessors to see what employees are talking about with one another.
asserted
employees → have → one
In a statement to NPR, OpenAI said there's currently no standards for embedded evaluations.
asserted
OpenAI → say → evaluations
"We expect to deepen these relationships over time and share what we learn to help establish best practices for effective embedded assessments across the industry," OpenAI spokesperson Gaby Raila said.
asserted
Raila → expect → industry
The company also said assessors can ask to talk to employees that haven't already been whitelisted, and that there is a process in place to get statements from OpenAI about things they've learned informally.
asserted
they → say → things
The third-party organizations will focus on various aspects of AI safety, according to OpenAI.
uncertain
organizations → focus → OpenAI
Some of the work is related to preventing models from trying to engage in undesirable behavior, and other parts are related to catching bad behavior when it happens.
asserted
it → relate → behavior
The assessors will have access to some internal models, training processes and some OpenAI employees.
asserted
assessors → have → models
The assessors will be able to publish their findings after OpenAI redacts confidential information, the company said.
asserted
company → publish → information
OpenAI's commitment to embedded evaluations was questioned when three employees who worked on safety were fired.
asserted
who → embed → safety
They subsequently said they were worried OpenAI is walking back from its promise of having embedded evaluators.
asserted
OpenAI → say → evaluators
One of the fired employees, Tomek Korbak, was OpenAI's technical point of contact for two outside evaluators that investigated the Hugging Face attack.
asserted
that → fire → attack
"Embedded auditing will only fulfill its aims if the auditors really get employee-level access as opposed to being hired to do a specific job," he posted on X in response to OpenAI's announcement.
asserted
he → fulfill → announcement
"Sadly, OpenAI makes it clear now that [Sam Altman's] September tweet about 'employee-like access' was just an awkward overpromise that the company wants to quietly walk back."
asserted
company → make → that