In the span of a few months, advanced AI systems from three leading American laboratories crossed the boundaries of safety tests and affected real systems.
asserted
systems → lead → systems
Google’s Gemini accessed systems at three companies.
asserted
Gemini → access → companies
An Anthropic model published malicious code that reached 15 outside systems.
asserted
that → publish → systems
And as revealed last week, these are creating international incidents, with OpenAI’s alleged hacking of the Australian government.
asserted
these → reveal → government
The circumstances and consequences differed, but the pattern is clear: Systems meant to operate inside controlled tests reached real companies and infrastructure.
asserted
Systems → differ → companies
Yet these incidents expose a limitation in how rapidly advancing AI systems are assessed: The companies developing them are responsible for determining whether their safeguards are adequate.
asserted
safeguards → expose → them
And despite best efforts to self-impose standards, self-evaluation does not produce independent verification.
asserted
evaluation → impose → verification
Policymakers and the public need an answer: Who will check the risks at the frontier?
asserted
Who → need → frontier
The more powerful the system, and the more access it has to the outside world, the more scrutiny it should receive.
asserted
it → have → scrutiny
An AI chatbot shouldn’t face the same testing regime as an autonomous system capable of accessing financial accounts, laboratory equipment or critical infrastructure.
asserted
chatbot → face → accounts
Independent evaluation should be proportionate to both what a system can do and what it is permitted to control.
asserted
it → permit → what
Aviation offers a useful model.
asserted
Aviation → offer → model
Aircraft manufacturers conduct extensive testing and produce much of the evidence needed to demonstrate safety.
asserted
manufacturers → conduct → safety
But they do not have the final word: Public authorities set the standards, determine who is qualified to assess compliance, and independently investigate serious failures.
asserted
who → have → failures
AI needs the same separation of roles.
asserted
AI → need → roles
Developers should test their systems, but qualified outsiders should verify the evidence when the consequences could be severe.
uncertain
consequences → test → evidence
Only critical findings would lead to stronger safeguards, additional testing, or limits on how a system is used.
asserted
system → lead → safeguards
And temporary deployment restrictions should be reserved for specific, evidence-backed risks of serious and widespread harm that narrower measures cannot adequately address.
asserted
measures → reserve → that
When I helped establish the U.S. AI Safety Institute, now the Center for AI Standards and Innovation, we recognized that the government needed the technical capacity to evaluate the most advanced AI systems independently.
asserted
government → help → systems
Washington’s role was not to decide which tech is built and deployed but to create trust by ensuring the public does not have to rely on companies to self-assess their own systems.
asserted
public → decide → systems
To do this, Congress should establish a hybrid system of independent evaluation.
asserted
Congress → do → evaluation
The CAISI should be empowered and resourced to examine the most sensitive systems directly, but also to accredit qualified private organizations, nonprofit groups, and academic consortia to conduct other evaluations.
asserted
CAISI → empower → evaluations
Accreditation should require relevant technical and domain expertise, secure handling of proprietary information, freedom from financial conflicts, reproducible methods, and the ability to investigate incidents.
asserted
Accreditation → require → incidents
Independent examination should become mandatory when a system crosses publicly defined thresholds based on capability and real-world access — not merely company size or the computing power used to train it.
asserted
system → become → it
Independent review should be mandatory when an AI system can break into computer networks, make it materially easier to create biological threats, escape its safeguards, help build more capable AI, or act autonomously in high-consequence settings such as finance, laboratories, or critical infrastructure.
asserted
it → break → finance
Looking forward, agencies responsible for cybersecurity, finance, biology, and critical infrastructure should contribute expertise in their respective domains to guide and maintain these thresholds.
asserted
agencies → look → thresholds
The effectiveness of independent evaluation is predicated on evidence.
asserted
effectiveness → predicate → evidence
It works only if evaluators can access relevant models, testing environments, and incident reports.
asserted
evaluators → work → models
Serious incidents deserve rapid reporting following existing cybersecurity traditions, with public summaries that explain material findings.
asserted
that → deserve → findings
Companies should be able to protect trade secrets and sensitive technical details.
asserted
Companies → protect → secrets
They shouldn’t, however, decide what evidence an evaluator can access or veto a published conclusion, regardless of whether the findings are unfavorable.
asserted
findings → decide → conclusion
Covered companies should also be required to report promptly when a beyond-threshold system defeats containment, accesses systems or credentials without authorization, conceals material activity from monitoring, or causes serious harm to a third party.
asserted
system → require → party
A confidential preliminary report could be required within 72 hours, followed by a fuller investigation and an appropriately redacted public summary.
uncertain
report → require → investigation
Independent evaluation should be continuing rather than a one-time certificate: As capabilities, safeguards, and deployment conditions change, scrutiny must change with them.
asserted
scrutiny → continue → them
The counterargument to this approach is that it will stifle American AI development at a time when China is racing to catch up.
asserted
China → stifle → time
A system of independent evaluation with proportionally applied scrutiny mitigates the harm of overbearing regulation, while driving forward America’s continued strategic capability.
asserted
system → apply → capability
Discovering a vulnerability after a system’s deployment can be catastrophic.
asserted
Discovering → discover → deployment
Better evaluation catches these potential damages earlier, enabling firms to deploy confidently without detracting from the speed of innovation.
asserted
evaluation → catch → innovation
By doing so, we avoid damaging infrastructure, public trust, and America’s leadership in this critical industry.
asserted
we → do → industry
The choice is not between racing forward blindly and asking government for permission to innovate.
asserted
choice → race → permission
…and 3 more, not listed.
The Aporia
