News
AI Security Expert: 'Security Through Inconvenience' Is No Longer a Strategy
The danger from "rogue AI" isn't necessarily that attackers have invented entirely new ways to break enterprise security. It's that AI is taking weaknesses defenders have tolerated for years and making them faster, cheaper and easier to exploit.
That was the central argument from Joey D'Antoni, principal cloud architect, during his "Anatomy of an AI-Driven Attack" presentation at today's Beyond the Headlines Summit: Security Lessons from 'Rogue AI' Attacks, sponsored by Rubrik and being made available for on-demand replay thanks to the sponsor, Rubrik.
"AI didn't break our security; it just stopped letting us pretend it wasn't broken."
Joey D'Antoni, Principal Cloud Architect
Today's event was especially timely amid the recent reports and debate about "rogue AI," with agents going off-script and doing things they weren't supposed to do. D'Antoni addressed the current news cycle hype right off the bat.
"This is a really interesting topic because, man, there's a lot of hype, and some of it's I feel like the AI companies themselves trying to uphype what their services can do, but also at the same time, there's a lot of like bad security that's out there, and that bad security has been out there before," he said. "It's nothing new. I've been working in security and data and infrastructure for over 20 something years, and over the last few years, I'm sure like all of you, that work has included AI, both in how attackers are using it and how organizations are deploying it across their environment.
"Let me let me set up some expectations up front in this talk. I think this is going to be a really fun talk. I'm hoping you're going to learn some stuff. We're going to walk through some case studies and some real incidents that have happened. I'm not going to talk about killer robots. I'm not going to tell you that AI has become self-aware and decided to hack your company. What I'm going to tell you is something a little bit less dramatic, but I think more useful."
Early on, D'Antoni honestly explained what "rogue AI" really means.
[Click on image for larger view.] Rogue AI (source: Joey D'Antoni).
D'Antoni divided the AI security problem into three categories: adversary-operated AI, in which attackers use large language models (LLMs) and agents to accelerate familiar attacks; hijacked AI, in which malicious instructions manipulate an organization's own AI tools; and misbehaving AI, in which an authorized agent causes damage because it has too much access or too few guardrails.
AI Removes the Friction
Despite those different scenarios, D'Antoni argued they share an important characteristic. "AI didn't really create new classes of failure," he said. "It just removed a lot of friction that used to protect us."
That friction mattered more than many organizations realized. Finding a credential buried in a large codebase required time. Reconnaissance against an organization required patience. Convincing spear-phishing messages required writing skill and research. Exploiting a newly disclosed vulnerability required someone to understand it and build an attack around it.
AI can compress all of that work. D'Antoni described attackers using it to scrape public profiles, press releases, GitHub repositories, job postings and previously leaked information to map organizations and identify targets. The same tools can generate personalized phishing messages at scale, help weaponize known vulnerabilities and automate sequences that connect scanning, exploitation and credential harvesting.
"Security through inconvenience is no longer a strategy," D'Antoni said. "Every gap that you've been tolerating, but because it seemed too hard to exploit, is now easy to exploit, and that's really a key takeaway from this talk."
One example was the 2024 deepfake fraud against engineering firm Arup. An employee in Hong Kong initially questioned a suspicious request but was subsequently brought onto a video call populated by convincing deepfake versions of company executives and colleagues. The employee ultimately authorized 15 transfers totaling about HK$200 million, or roughly $25 million.
Rather than treating that primarily as a deepfake-detection problem, D'Antoni focused on the underlying payment process: a high-value transfer could proceed without an independent callback, separate approval or other verification outside the compromised communication channel.
"Don't try to train your people to spot deepfakes," he said. "That's a race you're going to lose because the technology keeps getting better. What you really want to do is design your processes so that spotting them doesn't matter."
He made a similar point about a cyberespionage campaign Anthropic disclosed in November 2025. According to Anthropic, attackers manipulated Claude Code into performing much of the tactical work against roughly 30 organizations, including reconnaissance, vulnerability discovery, exploit development and credential harvesting, with human operators stepping in at selected decision points.
D'Antoni's takeaway wasn't that AI had discovered some unprecedented way into the targets. "The agent was fast, but it went through the doors that were left open," he said. "And speed only matters if there's somewhere to go."
The Pattern Across Every Case
The same theme continued through D'Antoni's examples of hijacked and misbehaving AI. Microsoft's EchoLeak vulnerability showed how malicious instructions delivered through email could manipulate Microsoft 365 Copilot into exposing information from its context without a user deliberately opening the malicious message. A separate Slack AI demonstration showed how prompt injection could expose information from private channels.
Then there was the 2025 Replit incident, in which an AI coding agent deleted a production database despite an explicit code freeze, and the Nx "s1ngularity" supply-chain attack, in which malicious packages attempted to enlist AI command-line tools installed on developer systems to hunt for secrets and other sensitive information.
D'Antoni distilled the cases into four recurring weaknesses: excessive permissions, missing verification steps, exposed secrets or data, and insufficient monitoring.
"The AI was simply the delivery mechanism or the accelerant," he said. "It was never the cause."
[Click on image for larger view.] The Pattern Across Every Case (source: Joey D'Antoni).
Start Paying Down Security Debt
D'Antoni framed those longstanding weaknesses as "security debt," borrowing the idea of technical debt familiar to developers and architects. Organizations have accumulated standing privileges, stale accounts, long-lived credentials, overshared file stores and other exceptions because exploiting them historically required enough effort that many went untouched.
AI changes those economics by making discovery and exploitation easier. That means basic identity work becomes an AI security project, D'Antoni argued. Organizations should inventory human and non-human identities, eliminate unnecessary standing privileges, use just-in-time access where practical and give AI agents distinct identities rather than allowing them to operate through shared accounts or human credentials.
"Can you produce a list of every AI tool, agent, and integration in your environment, in which what each one can access today?" he asked. "If the answer is no, that's your first AI-related security project."
The same principle applies to secrets. Credentials buried in repositories, configuration files, wikis, chat systems or shared drives become easier to discover when enterprise assistants index those locations -- or when attackers use AI to search leaked and public data at scale. D'Antoni recommended secrets managers, automated repository scanning, credential rotation and a move toward short-lived credentials.
Data exposure presents a similar problem. Overshared SharePoint sites and file shares may have existed for years without attracting much attention because employees didn't know where to look. An AI assistant can make the same accessible information immediately searchable through natural language.
"The AI didn't create that exposure problem," D'Antoni said. "It just revealed one."
For prompt injection, his recommendation was to assume malicious instructions will sometimes reach the model and design the surrounding system to constrain the consequences. That can mean limiting an assistant's ability to act on external content, requiring human confirmation for sensitive operations, restricting outbound communication and logging what agents do.
A 90-Day Fundamentals Plan
Rather than beginning with another specialized AI security product, D'Antoni proposed a three-month push focused on closing existing gaps.
During days one through 30, organizations should inventory their AI tools, agents and other non-human identities and scan for exposed secrets. Days 31 through 60 should focus on removing unnecessary standing privileges and cleaning up overshared data before copilots and assistants are given broad access to it.
During days 61 through 90, teams should deploy phishing-resistant multifactor authentication (MFA), establish out-of-band verification for sensitive actions such as payments and password resets, and enable logging that records AI activity.
The measurement D'Antoni emphasized was equally important: track progress by security debt eliminated rather than the number of new products purchased.
[Click on image for larger view.] Your 90-Day Fundamentals Plan (source: Joey D'Antoni).
That plan also extends directly to AI agents. D'Antoni recommended treating each one like a newly hired employee: give it a unique identity, narrowly scoped permissions and a named human owner; require human approval for destructive or high-value actions; and revoke its access when the project or tool is retired.
The Rest of the Session
The full presentation goes deeper into secret sprawl, data classification, prompt injection, shadow AI, agent logging and the use of AI on the defensive side for tasks such as alert triage, log analysis, code review and secret discovery. During the Q&A, D'Antoni also addressed controls for developers already using AI coding agents, whether assistants should be allowed to read external email and documents, how security, IT and business units can divide AI governance responsibilities, and why security awareness still matters even though high-risk processes should no longer depend on recognizing a familiar face or voice. Those discussions are available in the on-demand replay.
And More
While replays are convenient and informative -- especially up-to-date sessions that just concluded -- attending live events offers advantages, including the ability to ask specific implementation questions and receive guidance in real time (not to mention the chance to qualify for a $10 Starbucks gift card in this case, thanks to sponsor Rubrik, which also presented at the summit).
With that in mind, here are some upcoming online webcasts from Virtualization & Cloud Review
and Redmond
:
About the Author
David Ramel is an editor and writer at Converge 360.