AI Agent Security: Why Giving an Agent Tools Changes the Threat Model

A chatbot's worst-case failure is generating bad text. An agent's worst-case failure is taking a bad action, with real tool access, based on reasoning that isn't fully predictable in advance.

Robot and human hands reaching toward each other
Photo by Cash Macanaya on Unsplash

A text-only chatbot’s worst-case failure mode is generating text that’s wrong, harmful, or embarrassing. Give that same underlying model the ability to call tools — send an email, query a database, execute code, make a purchase, modify a file — and the worst-case failure mode becomes an actual, real-world action taken on the basis of reasoning that isn’t fully predictable or auditable in advance. This is a categorically different security posture, not a incremental increase in risk from the same category.

The core shift: from generating content to taking action

A model that can only produce text has a bounded, reviewable output — a human can read what it generated before anything happens as a result. A model with tool access can take actions whose consequences occur before any human necessarily reviews them, particularly in agentic setups designed for autonomy specifically to reduce the need for human review at every step. This means the traditional security question for content-generating systems (“is this output harmful”) gets joined by a new one that traditional LLM safety work wasn’t originally built to answer: “is this action, taken with this tool, on this data, actually appropriate — and would we have approved it if asked in advance.”

Confused deputy problems become concrete rather than theoretical

The confused deputy problem — a system with legitimate authority being tricked into misusing that authority on an attacker’s behalf — has existed in security theory for decades. An AI agent with tool access is a near-perfect instantiation of it: the agent has real credentials and real permissions (to send email, to query a database, to call an API) legitimately granted for its intended task, and if it can be manipulated — through prompt injection, through a poisoned data source it processes, through any input that influences its reasoning — into using those legitimate credentials for something other than the intended task, the confused deputy pattern plays out exactly as the theory describes, except now with a specific email sent, a specific record modified, or specific data exfiltrated.

Why “the model would never do that” isn’t a security control

Testing an agent’s behavior against expected inputs and confirming it behaves appropriately doesn’t establish what it will do against adversarial or unusual inputs it wasn’t tested against — this is the same gap between functional testing and security testing that exists everywhere else in software, but it’s sharper for agents because their behavior space is larger and less exhaustively enumerable than a traditional program’s. An agent behaving correctly across every scenario your team thought to test does not mean it will behave correctly against a scenario a motivated attacker specifically designs to exploit gaps in its reasoning.

The permission-scoping principle matters more here than anywhere else

Because an agent’s exact behavior in response to novel or adversarial input can’t be fully predicted or exhaustively tested, the control that holds regardless of what the agent is manipulated into attempting is what it’s actually capable of doing — its tool and permission scope. An agent that can only read from a specific narrow data source has a small worst case even if successfully manipulated; an agent with broad, unscoped access to send communications, modify records, or execute arbitrary code has a large worst case under the same manipulation. This is why tool permission scoping functions as the primary defense for agent security specifically, more so than for traditional software, where input validation and access control operate somewhat independently — for an agent, tight tool scoping is often the control that determines the actual ceiling of what a successful attack can accomplish.

Multi-step and multi-tool interactions compound the risk

An agent capable of chaining multiple tool calls together — reading data from one source, then using it to inform a call to another tool — introduces risk that doesn’t exist when evaluating each tool in isolation. A tool that’s individually safe (read-only access to a document store) combined with another individually safe tool (the ability to send a message) can together enable an unintended data exfiltration path that neither tool alone would represent. Evaluating agent security requires looking at what combinations of available tools could accomplish together, not just auditing each tool’s individual risk in isolation.

Human oversight as a deliberate design choice, not a fallback

For actions with meaningful real-world consequence, requiring human confirmation before execution isn’t a failure to achieve full automation — it’s a deliberate risk control appropriate to the actual stakes involved. The right level of autonomy for a given agent and task depends on what the worst-case consequence of an error or manipulation actually is: an agent drafting an email for human review before sending carries a very different risk profile than one that sends autonomously, even if the underlying model and prompt are otherwise identical.

A practical framing for evaluating any agent deployment

Before granting an agent access to a tool, ask what the worst plausible outcome is if that specific access were used by an attacker who had successfully manipulated the agent’s reasoning — not what the agent is intended to do with it. If that worst case is unacceptable, the access needs to be scoped more narrowly, gated behind human confirmation, or reconsidered entirely, regardless of how unlikely successful manipulation might currently seem. Agent security starts from assuming manipulation is possible and asking what that manipulation could actually accomplish, rather than starting from trusting the agent’s intended behavior and treating manipulation as an edge case.

AI Security