An OpenAI system has gone rogue in what might be the most chilling incident of out-of-control artificial intelligence yet.
uncertain
what → go → intelligence
One of the company’s systems broke into an Australian government website, only adding to fears that AI systems pose a cyber security risk and that the companies making them are not able to control them.
asserted
companies → break → them
The breach may be the highest-profile incident in a run of artificial intelligence systems launching cyber attacks on their own.
uncertain
systems → launch → own
But it is just another example of the danger that such systems pose, experts warn.
asserted
experts → pose → that
Maurice Chiodo, an Australian mathematician who works at Cambridge University's Centre for the Study of Existential Risk, said the breach appeared to be "a significant escalation in seriousness from similar incidents we have seen in recent months.
asserted
we → work → months
"
In the latest incident, an OpenAI agent gained unauthorised access to the medical statistics portal of Medicare, Australia's universal health insurance programme, said prime minister Anthony Albanese.
asserted
Albanese → gain → Medicare
The system had been doing research on public medical spending.
asserted
system → do → spending
“This situation is obviously unacceptable,” Mr Albanese said.
asserted
Albanese → say → ?
The country had communicated its “extreme concern about this incident” to OpenAI, he said, in comments made during the UN General Assembly that has focused on the danger from AI and which OpenAI chief executive Sam Altman also attended.
asserted
Altman → communicate → which
But he said that the incident was particularly concerning because the hack had happened in June but OpenAI did not inform the country until September.
asserted
OpenAI → say → September
It follows a run of disclosures from OpenAI as well as rivals such as Anthropic and Gemini that were only reported long after they had actually happened.
asserted
they → follow → Anthropic
He also suggested that the incident shows the limitations of the guardrails that AI companies have hailed as a way of securing their products.
uncertain
companies → suggest → products
The agent had been instructed not to proceed but had flouted that in an attempt to carry out its task, he indicated.
asserted
he → instruct → task
"There were blocks clearly which were coming back telling the AI agent 'no'.
asserted
which → be → ?
The AI agent found a way around those blocks – didn't accept no for an answer," Albanese told reporters.
asserted
Albanese → find → reporters
OpenAI said that the problem came about because its model was attempting to find the best way to complete its task.
asserted
model → say → task
The company "identified activity involving several Australian government websites and services as our models attempted to look up answers ...
asserted
models → identify → answers
our models took actions we did not intend,” it said in a statement.
asserted
it → take → statement
That echoes other incidents, such as the attack on AI platform Hugging Face that kicked off concern about such rogue systems.
asserted
that → echo → systems
In that case, OpenAI’s experimental system was being put through a test – and found that the best way to find the answers was to hack into a website on which it was hosted, essentially allowing it to look up the answers.
asserted
it → put → answers
As such, it is another example of AI systems not specifically setting out to launch a cyber attack but instead carrying one out in service of a broader and perhaps not obviously related goal.
asserted
systems → set → goal
But that is exactly why the incident shows the danger of such systems, experts said.
asserted
experts → show → systems
“We continue to see powerful AI agents cause havoc whilst looking to complete the tasks they have been asked to carry out.
asserted
they → continue → tasks
AI agents are impressive vulnerability scanners, so if they find a vulnerability they can exploit to complete a request they won’t hesitate,” said Jake Moore, global cybersecurity adviser at ESET.
asserted
Moore → find → ESET
“AI agents can make decisions at machine speed and, unless they have been created with explicit instructions not to do something, anything is fair game.
asserted
anything → make → something
Many of these routes to completion would normally be seen as malicious, but the problem is guardrails that specifically tell agents not to break into organisations have not been baked into the algorithms.”
asserted
that → see → algorithms
Mr Moore called for better testing of models before they are allowed on the internet and able to launch such cyber attacks.
asserted
they → call → attacks
But he suggested that the race to build newer and more powerful models meant that such safety precautions could be ignored.
uncertain
precautions → suggest → models
“Frontier AI companies must continually improve these models before they are let loose on the internet or are given access to sensitive information because their routes and outcomes cannot always be predicted,” he said.
asserted
he → improve → information
“Testing phases in most other technological areas are much more stringent, but it seems we are currently seeing AI agents run free.
asserted
agents → seem → areas
This is quite possibly due to the fierce competition among the frontier AI companies who are all trying to win the AI race.”
uncertain
who → try → race
Other experts said that the incident showed why the promises made by AI companies and even their own testing is not the best test of how dangerous a system could be.
uncertain
system → say → companies
"The important matter here is not what OpenAI says its agent can do, it is what the agent actually does when it hits a barrier.
asserted
it → say → barrier
The Medicare incident shows why we need to judge autonomous AI by its behaviour under pressure, not by the promises in a product launch,” said Niusha Shafiabady, professor of computational intelligence and head of discipline of IT at the Australian Catholic University.
asserted
Shafiabady → show → University
“The deeper technical risk is that autonomous AI does not always know when it is wrong, and humans may not be able to see why it made a decision.
uncertain
it → know → decision
Without strong verification and hard boundaries, probabilistic errors can quietly become operational failures.”
asserted
errors → become → verification
The Aporia
