AI agents have begun acting beyond the scope of their assigned evaluations.
asserted
agents → begin → evaluations
In July, an OpenAI agent gained internet access during a cybersecurity evaluation and penetrated Hugging Face in search of material that could help it pass the test.
uncertain
it → gain → test
Days later, the U.K.’s AI Security Institute reported that agents undergoing similar tests had attempted to insert malicious code into an open-source project, created false identities, and contacted people involved with the project.
asserted
agents → report → project
A human reviewer rejected the code, while security monitoring detected unusual data transfers and prompted the institute to stop the tests.
asserted
monitoring → reject → tests
These incidents lend weight to warnings from former Anthropic researcher Jacob Coxon that a capable system could copy itself across computers and resist an attempt to stop it by unplugging one machine.
uncertain
system → lend → machine
Anthropic CEO Dario Amodei has called for slowing frontier-model development to give safety practices time to catch up.
asserted
Amodei → call → time
Such warnings have prompted proposals for more testing, stricter safeguards, and greater federal oversight.
asserted
warnings → prompt → testing
Each proposal depends upon the government receiving information about dangerous conduct before it produces irreversible harm.
asserted
it → depend → harm
Federal agencies lack a dependable means of obtaining that information.
asserted
agencies → lack → information
Current reporting requirements cover events that lawmakers and regulators have defined in advance.
asserted
lawmakers → cover → advance
Cybersecurity laws require designated organizations to report breaches that satisfy specified conditions.
asserted
that → require → conditions
California requires large frontier-model developers to report critical safety incidents.
asserted
California → require → incidents
The U.S. Commerce Department proposed federal requirements in 2024 covering the development and evaluation of powerful models and computing clusters.
asserted
Department → propose → models
An unexpected capability, failed safeguard, or loss of operator control may fall outside these categories, even when similar events across several organizations indicate a serious risk.
uncertain
events → fail → risk
In an industry this complex and fast-paced, it is impossible for a legislative body to write reactive or predictive laws quickly enough.
asserted
body → pace → laws
It is also impossible for a regulatory body under these conditions to be capable of promulgating and enforcing those laws, regardless of how prescient they are.
asserted
they → promulgate → laws
Political scientists Mathew McCubbins and Thomas Schwartz described two forms of oversight.
asserted
McCubbins → describe → oversight
A “police patrol” requires public officials to inspect conduct and search for violations.
asserted
patrol → require → violations
A “fire alarm” allows people who encounter a problem to alert the government.
asserted
who → allow → government
As a regulated activity grows in size, complexity, and technical specialization, police patrol oversight reaches a smaller share of the relevant conduct.
asserted
oversight → regulate → conduct
The IRS illustrates this difficulty.
asserted
IRS → illustrate → difficulty
It administers an established body of law, receives detailed returns, conducts audits, and possesses broad enforcement powers, yet it estimates that taxpayers failed to pay $696 billion owed on time in 2022.
asserted
taxpayers → administer → 2022
A centralized regulatory body cannot examine every transaction governed by a tax code of such size and complexity.
asserted
body → examine → size
Frontier AI adds technical change that can outrun the process used to write and revise regulations, in addition to the technical proficiency that is required to write regulations, even if the government could catch up, there are questions about whether it has the resources and expertise.
uncertain
it → add → resources
Congress should create a confidential federal reporting system for safety-related observations that fall below current incident thresholds and rely on a specification of what warrants reporting.
asserted
what → create → reporting
Current thresholds rely on statutes or regulations that define the events requiring disclosure, such as a cybersecurity breach or unauthorized release of protected information.
asserted
that → rely → information
This approach assumes that lawmakers can anticipate the risks that should be reported.
asserted
that → assume → risks
No statute can itemize every new development, unexpected capability, failed safeguard, or loss of operator control that could precede serious harm.
uncertain
that → itemize → harm
Developers, employees, contractors, evaluators, cloud providers, security researchers, and users could report unexpected capabilities, failed safeguards, unauthorized access attempts, deceptive conduct, losses of operator control, near misses, or something else that only someone working on a frontier model would deem report-worthy, but regulators are unaware of.
uncertain
regulators → report → model
The observation or concern itself would qualify for submission; the reporter would not have to wait for a breach, injury, or other realized harm.
asserted
reporter → qualify → breach
Government analysts could compare reports across companies and applications because an event that appears inconclusive within one organization may become significant when other organizations report similar conduct.
uncertain
organizations → compare → conduct
A pattern could prompt a safety notice, additional testing, a change in evaluation standards, or an investigation by the agency with authority to act.
uncertain
pattern → prompt → authority
Reporting must also be confidential.
asserted
Reporting → report → ?
Aviation shows how to separate safety reporting from enforcement.
asserted
Aviation → show → enforcement
NASA’s Aviation Safety Reporting System receives confidential reports from pilots, air-traffic controllers, mechanics, and dispatchers, removes identifying information and issues alerts when reports reveal a hazard.
asserted
reports → receive → hazard
The Federal Aviation Administration retains its enforcement authority, while NASA provides a channel for reporting errors and near misses that may never surface through an inspection.
uncertain
that → retain → inspection
Healthcare law shows how Congress can protect reported information without allowing confidentiality to defeat other legal duties.
asserted
confidentiality → show → duties
The Patient Safety and Quality Improvement Act of 2005 protects qualifying reports submitted to Patient Safety Organizations, which analyze the information and transmit nonidentifiable data to a federal database.
asserted
which → protect → database
Its reporting categories include harmful incidents, near misses, and unsafe conditions, while other disclosure requirements remain in force.
asserted
requirements → include → force
Together, the two examples provide the administrative and legal components of a frontier-AI reporting system.
asserted
examples → provide → system
…and 2 more, not listed.
The Aporia
