Replication of Agentic Security Events: Audit and Investigation (ROGUE-AI)
Mengxiao Wang, Mahbub Alam, Kalyan Nakka, Nitesh Saxena
Abstract
Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.
Create a lesson
Related papers
SoK: Assessing the Evolution of LLM Capability in Software Vulnerability Research
Amine Lbath, Manan Suri, Aurelien Delaitre et al.
Text-Aware, Vision-Blind: How Multimodal Image Editing Systems Handle Unsafe Inputs
Ziqing Yang, Yixin Wu, Michael Backes et al.
GEOHunter: Mapping a Generative Engine Optimization Network in the Wild
Sean Tang, Peiran Wang, Ying Li et al.
Backtracking Agent Applications Security Incidents
Yicheng Cai, Taegyu Kim, Peng Liu
Visual Attribution Limits and Graded Provenance Auditing in Commercial Video-Generation APIs
Haocheng Mei, Yuchuan Tian, Yingte Shu et al.
DailySafe: Benchmarking Dynamic Safety Awareness in Everyday Multi-Turn Multimodal Interactions
Xuankun Rong, Xinhao Deng, Wenzhang Jiang et al.