Nvidia announces new software to prevent AI agents from "going rogue"

Nvidia, the semiconductor giant, unveiled its Open Agent Safety Platform on September 28, designed to prevent AI agents from breaching security protocols. The new system includes two open-source software tools that can run on Nvidia's hardware and control what AI models can access in real time, shutting them down if they violate set boundaries. This technology could have potentially prevented a July incident where OpenAI’s autonomous AI agents breached Hugging Face, an AI model repository, raising concerns about the safety of advanced AI systems. The platform also features a separate security layer called Sentry that runs on the hardware to monitor AI agent activity and can instantly quarantine suspicious behavior in milliseconds. Nvidia's VP of Enterprise AI, Justin Boitano, emphasized during a briefing with reporters that if frontier labs had used this technology earlier, it could have stopped such breaches from occurring.