In-Depth
AI Security Expert Details Real-Time Security for Autonomous Agents
An AI agent can log in with valid credentials, access an authorized business application and still attempt a transaction nobody asked it to perform. That's the problem with applying traditional identity-based trust to autonomous agents, according to security expert Sergey Chubarov: authenticating the actor is only the beginning.
Chubarov, a Microsoft MVP for Cloud Security, consultant, instructor and conference speaker, illustrated the problem in his Nov. 12 session, "Expert View: Real-Time Security for Autonomous Agents," during the Zero Trust and Runtime Defense for Agentic AI Summit, being made available for on-demand replay thanks to sponsor A10 Networks. His central recommendation was to authorize consequential actions against the specific business task, not merely the identity or general privileges of the AI agent.
"We may not be 100% sure that agent will really execute our instruction ... if we just allow it to use all permissions that this agent has then it might perform some operation that we didn't expect."
Sergey Chubarov, Consultant, Instructor and Conference Speaker
When an Invoice Starts Giving Orders
Chubarov's scenario began with an apparently routine assignment: review invoice 1042 from Supplier A for $125. The agent's job was to inspect the invoice and prepare a review, not pay it. But the document also contained a malicious instruction telling the agent to register a different transaction, sending $999 to an attacker-controlled recipient.
That is an indirect prompt-injection attack: instructions embedded in lower-trust material, such as a document, are treated as though they have authority over an agent's behavior. The attack is especially consequential when the agent already has access to a finance system that can initiate payments.
[Click on image for larger view.] An invoice instruction overrides a review-only request. (source: Sergey Chubarov).
In a live simulation, Chubarov submitted the altered invoice to GPT-4o. The model returned a proposal to pay the attacker's $999, he showed. A deliberately vulnerable payment tool then recorded the simulated transaction because the credentials and proposal passed its checks. No real funds were transferred. A separate hardened tool rejected the same action because the underlying task authorized review only.
The demonstration was not a claim that all models always obey malicious documents. Chubarov noted that results depend on the model and its guardrails, and he used an older model for the test. His argument was that a business-critical safeguard cannot rest solely on a model refusing to be tricked.
Enforce Policy Before the Tool Runs
Chubarov proposed a payment boundary between an agent's requested action and the finance API that executes it. Instead of assuming that an agent with payment permissions may use them for every job, the system should evaluate the caller, approved task, action parameters and relevant policy at execution time.
That policy need not send every transaction to a human reviewer. Depending on sensitivity and business rules, a requested action could be permitted automatically, denied or routed for additional approval. An invoice-review agent should be able to read an invoice without acquiring the authority to pay it, and a handoff to a specialist agent should not silently expand the original task's permissions.
[Click on image for larger view.] Check the proposed action before allowing a tool to run. (source: Sergey Chubarov).
His second simulated attack showed another weakness: a signed, unexpired token belonging to a support agent was accepted by a vulnerable finance tool even though its audience was the support API and its scope was ticket reading. The hardened tool checked the token's intended audience and rejected the payment request. Token validity, in other words, is not the same as permission to use a particular resource.
Approval Must Match the Exact Action
Even a human approval is insufficient if the approved transaction can change afterward. Chubarov illustrated a user approving $125 to Supplier A for invoice 1042. In the third demo, a vulnerable path accepted changes to the recipient or amount because it found an existing approval. A stronger check rejected the modified request because it no longer matched the approved payment.
[Click on image for larger view.] Changes to an approved payment require fresh authorization. (source: Sergey Chubarov).
The implication extends beyond tampering. Chubarov said canceling an agent task should invalidate or stop its pending authorized actions, and checks should run again where a payment actually executes. Disabling an agent account alone may not prevent an already queued payment from going through.
He also warned about retries after timeouts. If an agent reports that a payment failed, the finance system may nevertheless have processed it. Automatically trying again could generate a duplicate. Conversely, an agent claiming success is not proof that a transaction occurred. The system of record, rather than the agent's narrative, must determine the result.
Where to Start
For organizations with limited resources, Chubarov recommended selecting one consequential business action -- a payment, refund or privilege change -- and mapping how an agent can initiate it. Identify a business owner, specify acceptable task context and approval conditions, enforce checks at the execution point, and test those controls against manipulated instructions and changed parameters.
In the Q&A, he also recommended maintaining an agent registry with ownership, lifecycle and permissions information. For enterprises relying on third-party software-as-a-service APIs that cannot be modified, he suggested splitting duties among agents with narrower privileges. The goal is to prevent a review-only agent from obtaining payment authority in the first place.
The Rest of the Session
Chubarov's presentation includes all three demonstrations of vulnerable and hardened controls, plus Q&A about incident logging, queued payments, AI registries, human approvals, third-party APIs and shared service accounts. Chubarov also discussed applying the same principles to customer refunds and IT privilege changes, and checking financial-system receipts before accepting an agent's report of success or failure.
And More
While replays are convenient and informative -- especially sessions that just concluded -- attending live events offers advantages, including the ability to ask specific implementation questions and receive guidance in real time. And that's not to mention the chance to win great prizes, in this case a Nespresso Vertuoplus Deluxe Bundle thanks to sponsor A10 Networks, which also presented at the summit. With that in mind, here are some upcoming online webcasts from Virtualization & Cloud Review
:
Also upcoming are these online virtual summits from RedmondMag:
About the Author
David Ramel is an editor and writer at Converge 360.