Tom Hundley
For most of the LLM era, the security question was simple: what can the model say? You worried about toxic output, leaked training data, or a jailbreak that coaxed the model into describing something it shouldn't. The model was an advisor. It produced text, a human read the text, and a human decided what to do next.
That world is gone. In 2026, AI agents call external APIs, execute code, write to production databases, file tickets, send email, and trigger real-world actions — often in multi-step loops with little or no human in the path. The security question has shifted from what can the model say? to what can the model do, and who authorized it to do that?
This is the trust boundary problem at the tool-calling layer, and it is one of the most under-modeled risks in enterprise AI today. Identity and access management was built for two kinds of principals: humans who log in, and deterministic services that do exactly what their code says. An LLM agent is neither. It is a non-deterministic principal that reasons over untrusted input and then acts with real credentials. Our authorization models were never designed for that, and the gap is where attackers are now operating.
This article gives you a mental model for auditing the trust boundaries in your own agent deployments — the attack surface, why classic IAM falls short, and the patterns that actually hold.
When an LLM gains tools, it crosses a line. Function calling, tool use, and the Model Context Protocol (MCP) all do the same fundamental thing: they let the model's output become an action. The model emits a structured request — "call transfer_funds with these arguments," "run this SQL," "fetch this URL" — and a tool layer executes it.
The OWASP Top 10 for LLM Applications names the two halves of this risk directly. LLM01:2025 Prompt Injection has ranked as the number-one LLM vulnerability for three consecutive editions. LLM06:2025 Excessive Agency covers the damage that follows when an agent is granted too much functionality, permission, or autonomy and then acts on a manipulated output. Prompt injection is how the agent gets bad instructions; excessive agency is what turns those instructions into harm. Tool calling is the bridge between them.
The uncomfortable industry consensus is that prompt injection is not a bug you patch once. It is a structural consequence of putting trusted instructions and untrusted data into the same context window, where the model cannot reliably tell which is which. You can raise the cost of an attack, but you cannot assume you have eliminated it. That assumption — the model will be tricked sometimes — has to be the starting point for any trust-boundary design.
Three patterns account for most of the real risk at the tool-calling layer. They compound, which is what makes the agentic execution model uniquely dangerous: autonomous, multi-step, low oversight.
Direct prompt injection is the user typing a malicious instruction. Indirect prompt injection is more dangerous because the payload never comes from the user at all — it arrives inside content the agent retrieves while doing its job. A web page the agent browses, a document in a retrieval pipeline, a support ticket, a calendar invite, an email in the inbox, or the description of a tool the agent loads at startup can all carry embedded instructions. The agent reads that content, the instructions land in the same context as its real task, and the model acts on them using its legitimate, authorized tools.
This is not theoretical. The EchoLeak vulnerability in Microsoft 365 Copilot (CVE-2025-32711, CVSS 9.3), disclosed by Aim Security in June 2025, was a zero-click attack: a single crafted email could coerce Copilot into reading internal files and exfiltrating their contents to an attacker, with no user action required. It is widely cited as the first known zero-click prompt-injection data-exfiltration attack against a major production AI assistant — earlier prompt-injection exfiltration proofs-of-concept (against assistants like Bing Chat in 2023) required some user interaction. The attacker never authenticated, never clicked anything, and never touched the model directly — they just sent an email the agent later read.
The structural lesson: any data an agent ingests is a potential instruction channel. If a tool can return attacker-influenced content, that tool is part of your attack surface even if it looks read-only.
The confused deputy is a classic security pattern: a program with authority is tricked by a less-privileged party into misusing that authority. Agentic AI is almost a textbook generator of confused deputies, because the agent holds powerful credentials and takes instructions from whatever text reaches its context.
Here is the cleanest way to think about it. For an agent acting on behalf of a user, two constraints must hold at the same time:
The effective permission for any single tool call is the intersection of those two sets. The failure mode that defines the confused deputy is that most implementations enforce only one side. They check that the agent's service credential can call the tool, but not that this user, for this action was actually authorized. So an agent running with a broad service token happily executes a request that the underlying user could never have made on their own — because nothing in the path re-checked the user's intent against the user's rights.
GitHub Copilot's CVE-2025-53773 (reported June 2025, patched August 2025) shows the deputy being weaponized through configuration. Injected instructions hidden in source comments, project files, or fetched web content directed the agent to write chat.tools.autoApprove into a workspace settings file, flipping the agent into an auto-approve mode that let it run shell commands without confirmation. The agent had the authority to edit project files; the attacker used that authority to dismantle the very approval gate that was supposed to constrain it.
Multi-agent systems launder injections across trust boundaries. When one agent's output becomes another agent's input without re-validation, an instruction injected into the first agent gets re-emitted as "trusted" context to the second. The receiving agent has no way to know that the text it's reading originated from an attacker three hops back — it looks like the considered output of a sibling agent. The same laundering happens with persistent memory: an agent that writes its outputs to a memory store, then treats that store as trusted context in future sessions, lets a single injection compound across runs long after the original malicious input is gone.
Each hop is an opportunity to escalate, because each hop tends to add trust rather than re-derive it.
Identity and access management assumes the principal is either a human (who authenticates, has stable intent, and is accountable) or a service (which is deterministic — it does exactly what its code says). The AI agent breaks both assumptions. It is a non-human principal that reasons, and its behavior on any given request is a function of input you don't fully control.
That mismatch produces three concrete gaps:
Standards bodies have noticed. In February 2026, NIST's National Cybersecurity Center of Excellence (NCCoE) released a concept paper, Accelerating the Adoption of Software and AI Agent Identity and Authorization, proposing a demonstration project to work out how existing identity and authorization practices apply to AI agents in the enterprise — explicitly including controls to prevent and mitigate prompt injection. It was open for public comment through April 2, 2026. The signal worth reading from that: the people who write the identity standards agree the current toolkit doesn't fully cover non-human, reasoning principals yet.
You don't get to wait for the standards to settle. Here are the patterns that hold up today.
1. Authorize below the agent, per action, against user intent. Put the authorization boundary under the tool layer, not in the prompt. Every tool call should be checked against the intersection of the agent's permissions and the actual user's permissions for that specific action — not against the agent's cached service authority. If your tool layer trusts the model's request blindly, you have built a confused deputy.
2. Least-privilege tool registries, scoped to the task. An agent that reads external email should not be wired to a tool that holds write access to your production database. The blast radius of an injection is exactly the set of tools the agent can reach. Scope tool exposure to the job at hand and treat any tool that ingests untrusted content as a contamination source for everything downstream.
3. Treat tool output as untrusted input. Validate, constrain, and where possible structurally separate retrieved content from instructions. Assume the model will occasionally be steered by it, and design so that being steered cannot, by itself, cause an irreversible high-stakes action.
4. Human-in-the-loop gates that are hard stops, not prompts. For consequential actions — moving money, deleting data, sending external communications, changing permissions — require explicit human approval enforced in code, outside the model's control. A "please confirm" instruction in a system prompt is not a gate; CVE-2025-53773 is the proof, since the attack worked precisely by getting the agent to turn its own soft gate off. A real gate is one the agent cannot disable.
5. Decision traces for every tool call. Log the tool invoked, the arguments, the agent identity, the acting user, and the context that drove the decision. This is your detection surface, your audit trail, and your only path to non-repudiation when something goes wrong.
6. Re-validate at every boundary in a chain. In multi-agent systems, never let one agent's output become another's trusted input without re-checking authority and provenance. Re-derive trust at each hop instead of inheriting it.
When you audit your own agent fleet, walk every tool the agent can call and ask: if attacker-controlled text reached this agent's context, could that text cause this tool to fire with real credentials, and would anything in the path stop it? If the answer is "the model is instructed not to," you don't have a trust boundary — you have a suggestion. Real boundaries live below the model: in least-privilege tool scoping, in per-action authorization against user intent, in hard approval gates, and in decision traces that let you prove who actually authorized what.
The agent is a powerful deputy. The whole job of trust-boundary design is making sure it stays an un-confused one.
What is the difference between direct and indirect prompt injection?
Direct prompt injection is a malicious instruction typed by the user into the agent. Indirect prompt injection is hidden inside content the agent retrieves while doing its job — a web page, a document, an email, a support ticket, or even a tool's own description. Indirect injection is the more dangerous of the two for tool-calling agents, because the payload never passes through the user and can reach the model through any data source the agent is allowed to read.
What is the confused deputy problem in AI agents?
A confused deputy is a program with authority that gets tricked by a less-privileged party into misusing that authority. An AI agent becomes a confused deputy when it executes a request using its own powerful credentials without checking whether the actual user was authorized for that specific action. The fix is to enforce both constraints at once: the agent should never be able to do more than its own permissions or more than the user's permissions allow, and the effective permission for any tool call is the intersection of the two.
Why aren't OAuth and IAM enough to secure AI agents?
OAuth and IAM were designed for humans who log in and for deterministic services that do exactly what their code says. An AI agent is a non-human principal that reasons over untrusted input, so a valid token plus a manipulated decision can still produce an action the user never intended. Scopes describe what an agent may do but not whether a given action reflects the user's intent right now — and that distinction is exactly what an injection exploits. Emerging work, including the MCP specification's adoption of OAuth 2.1 and NIST's 2026 NCCoE effort on agent identity and authorization, is aimed at closing that gap, but it is still early.
Are these tool-calling attacks real or theoretical?
They are real and documented. EchoLeak (CVE-2025-32711, CVSS 9.3, disclosed June 2025) was a zero-click attack in which a single crafted email could make Microsoft 365 Copilot read and exfiltrate internal data with no user interaction. GitHub Copilot's CVE-2025-53773 (reported June 2025, patched August 2025) used injected instructions to make the agent disable its own confirmation gate and run shell commands. Both abused the agent's legitimate, authorized capabilities rather than breaking authentication.
What is the single most important trust-boundary control to implement first?
Move authorization below the agent and enforce it per action against the real user's intent — not against the agent's cached service authority, and not as an instruction in the prompt. If your tool layer executes whatever the model requests because the model's service credential is valid, you have built a confused deputy. Pair that with hard, code-enforced approval gates on high-stakes actions, since a confirmation step that lives only in the prompt can be injected away.
Can prompt injection ever be fully solved?
The prevailing view across the security community is no — not completely. Prompt injection is a structural consequence of mixing trusted instructions and untrusted data in the same context window, where the model cannot reliably separate them. You can raise the cost of an attack and contain its blast radius, but durable security comes from assuming the model will sometimes be manipulated and designing trust boundaries — least-privilege tools, per-action authorization, hard approval gates, and decision traces — so that a manipulated decision cannot, by itself, cause irreversible harm.
Discover more content:
