Tracking enterprise infrastructure's evolution in the AI era

In-Depth

Spot the (AI) Difference: Project Perception

• By Paul Schnackenburg
• 09/11/2026

There's a lot of hype in cybersecurity about AI agents hacking organizations at "machine speed" and how all our defenses will be overwhelmed by these new attacks. In fact, 116 organizations have signed an open letter urging businesses to "raise the security bar" and saying there's a limited window of time to prepare for AI hacking agent attacks. Mind you, these are the same organizations that created the problem in the first place by developing the LLMs that power these attacks. And these are the same businesses that stand to benefit financially from increased use of AI models and agents. With a nod to Dr. Ian Malcolm: "Yeah, but your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should."

When there's a lot of hype, such as with OpenAI's agent test hacking Hugging Face, there's also a lot of misinformation and points of view that may not always align with the evidence and facts.

In this article, I'll look at dispelling some of the myths around AI agents "hacking the planet," what we actually need to do as defenders of digital infrastructure to adapt, and then look at Microsoft's next step in this arena -- Project Perception.

AI Agents and Hacking -- The Reality
In summary -- OpenAI was testing its latest released model, plus a non-released model in a sandbox, wanting it to exercise the CyberGym test and evaluate the models' capabilities for discovering vulnerabilities. Due to a zero-day in the sandbox software and lax monitoring, the agents broke out of the sandbox and also built a system for agent-to-agent "covert" communication, surmising that Hugging Face might have the answers to the test. They then proceeded to hack Hugging Face, trying 17,000 different approaches. When Hugging Face's incident response eventually evicted the attackers, they wrote it up in a blog post ("We've been hacked by AI agents"), and shortly afterward OpenAI realized that its systems were behind the attack. There have been regular updates to OpenAI's post, as well as an independent investigation by METR.

This revelation from OpenAI was followed by Anthropic discovering that some of its tests had also escaped containment and then by the UK's AI Security Institute (AISI) revealing the same. AISI's report was refreshingly devoid of marketing spin and instead focused on what it was going to do to avoid a similar situation in the future.

Steps to Take for Defenders
Step one -- stop anthropomorphizing! There's "no one home" -- AI isn't conscious or self-aware, no matter what the boosters tell you or what it seems like when you're chatting with a bot. When we say things like "the agents wanted to escape," we deflect attention from the real problem. The real problem is that frontier AI labs are in a race against each other, with the price being money, and that anything that stands in the way of producing the next "best" model quickly, such as stronger sandboxing, comprehensive monitoring, or thoughtful alignment of prompts and reinforcement learning (RL) incentives, are hindrances, not steps toward the goal.

Step two -- proper sandboxing is necessary. The IT industry has had a couple of decades to figure this out, but apparently the frontier labs haven't taken those lessons to heart. If you're testing agents' behaviors, you must be crystal clear about what's part of the test environment and what's outside of it that the AI should never interact with -- and then architect your infrastructure accordingly.

Step three -- comprehensive monitoring and alerting is table stakes. Just as defenders must have the right logging to have visibility into potentially malicious intrusion activity, AI testing can't be conducted where the organization doesn't even know that its agents have escaped and are attacking other websites until it sees the incident-response write-up.

Step four -- legal accountability would be nice, but I know I'm dreaming here. There are hackers who've done exactly what happened in these incidents and who've ended up with lengthy prison sentences, but frontier AI firms can apparently get away with hacking others with an "oops -- sorry about that."

Step five -- attacks such as the one on Hugging Face were incredibly noisy because the agents weren't optimized for stealth; rather, they were streamlined for results. Make sure you have the right analysis tools available to you before an attacker shows up. Hugging Face tried to use a U.S. frontier model for incident-response analysis and defense, but it refused based on its guardrails because it can't tell the difference between an attacker and a defender. The team then switched to a Chinese open-weight model, which helped with the analysis of those 17,000 activities.

What Real AI-Powered Attacks Look Like Today
There have been a few headline-inducing situations where attackers have used frontier models, so at least we get some insight through the regular reports from the U.S. AI companies, in parts of their hacking infrastructure. So far, the uptake in the criminal ecosystem of hacking agents seems minimal, but you know what they do use AI for? Polishing phishing email lures for the maximum chance that they'll be clicked, customizing spear-phishing lures for individuals or organizations to improve the success rate, and translating attacks into languages they don't speak, targeting countries that so far have had less exposure to cyberattacks.

They also use AI to create completely different remote access tools (RATs) to make attribution harder.

While these are cool attacks and no doubt both criminals and state-sponsored spies are experimenting with using AI to improve parts of their process, they're very different attackers. It's important to understand who's after you and tailor your defenses accordingly. If you're a military contractor, an organization with unique intellectual property, or a government organization, you do have to worry about both nation-state spies infiltrating your systems and cybercriminals. But for every other business or organization, you really only have to be concerned about cybercriminals. And they're simple businessmen -- they take the easiest path to earn money.

Today, that's phishing emails, attacker-in-the-middle kits, and social engineering your help desk because those tactics work. I'm not saying that they won't start using AI hacking agents in the future, but that'll only happen when that technology is readily available, reliable, and cost-effective. And today, it isn't.

So, when the AI companies tell you that your only salvation against these AI-equipped attackers is their latest tool, take that claim with a huge helping of salt.

Project Perception
To contrast with the above, I thought we'd look at Microsoft's recently revealed next evolution of agentic AI defenses -- Project Perception.

The current public preview is invitation-only, so I haven't been able to test it myself yet, except for the multi-model scanning harness (MDASH) component, which I've been testing for a few months. I'll cover that in a future article.

Perception brings three categories of agents to Defender XDR: red agents that do penetration testing to discover vulnerabilities, blue agents that defend, and green agents that work to improve your posture and fix vulnerabilities.

You can interact with Perception through chat, picking an agent to do a task, or running a playbook. Multiple underlying LLMs are part of Perception, including Microsoft's own MAI-Cyber-1-Flash. The evolution compared to the current crop of agents we have in Defender, Intune, Entra, and Purview is that these agents are autonomous and can hand work to other agents. Today, red recon agents can only be used against Azure infrastructure deployments, but this will expand as Perception grows. You can also see agent activity over the last 30 days in Overview.

Here's a sample workflow: A red recon agent is tasked with understanding whether an organization is vulnerable to attacks from the threat actor Onyx Sleet, and it finds several vulnerabilities.

Once it's scanned open ports and discovered a SQL injection vulnerability, it hands over these findings to a blue agent, which creates custom detections for Defender XDR to spot attacks.

Finally, the findings are handed over to a green agent to fix permanently.

Perception comes with a set of playbooks.

Perception also comes with a command-line interface, which means it'll be available in the flow of work for different types of defenders.

While the demos are cool and hint toward a future where smart, LLM-based agents in shiny armor patrol my client's infrastructure 24/7/365, instantly repelling both human and robotic attackers, I suspect that the reality will be quite different.

My first concern came in John Savill's video, where he says this will be a "premium experience" from a cost perspective. Let me be clear: There's already a Cybersecurity Poverty Line in Microsoft's products, dividing clients who can afford Security Copilot and those who can't. Those who can't must do without all the very powerful agents that are already released. Releasing an even more expensive (Perception is in limited public preview -- no word on cost yet) "premium experience" won't allow smaller businesses to defend themselves using it.

And my second concern is that Perception is built for a possible type of agentic AI attack in the future, but it's unclear how it's going to help with the current primary set of cyber hygiene steps we need to take: phishing-resistant MFA for everyone, faster automatic patching, and so on.

Conclusion
Predictions are hard, particularly about the future, but everything I see from levelheaded cybersecurity experts, rather than AI safety experts from the frontier labs, much smarter than me, points me toward AI and agents being a solution waiting for a future problem rather than the best solution to the attacks that succeed today. Will that change in the future? It definitely will, and it'll be interesting to see how this arms race pans out over the next year or two.