Securing tool-using agents

Building Autobot made agent security concrete for me. I had given a model tools to read review feedback, inspect CI failures, change code, and run checks. A reasoning error now had a path to a real state change. The same design problem appears in any agent that can browse, send messages, query data, deploy software, or act through a user’s identity.

Security boundaries

An agent is a model inside a control system. That system decides which data enters the context, which tools are available, what identity those tools use, what persists between runs, and whether a person must approve an action. Model quality affects how often mistakes happen, but these surrounding decisions determine what a mistake can reach.

The threat model has to include malicious input and ordinary failure. An attacker may try to redirect a run, a legitimate user may leave out important context, a tool may return stale data, or the model may choose the wrong action while sounding certain. The same containment measures are useful in each case because they reduce the authority attached to a bad decision.

Untrusted context

Agents routinely read websites, email, documents, tickets, source code, and tool output. Some of that material is controlled by other people. Indirect prompt injection hides instructions inside a resource the agent is expected to process. If those instructions influence the run, an attacker can redirect the task or change the arguments sent to a legitimate tool.

Trusted developer instructions and untrusted external data meet inside one model context without a reliable security boundary between them. Instruction labels, filtering, and defensive prompting can reduce exposure, but none of them makes arbitrary external content safe to combine with powerful tools.

Retrieval should preserve the source of external content and return only the fields needed for the task instead of entire documents or arbitrary markup. Content from a ticket can supply the ticket text; it should not be able to expand the tool set, change the standing instructions, or raise the permission level of the run.

A separate policy layer should evaluate proposed actions against the user’s original request. It can require structured arguments, restrict destinations, and reject operations that the task never authorized. This decision has to be made from trusted state rather than from the model’s explanation of why a call is reasonable.

Tools and permissions

The permissions exposed by a tool determine how much damage a bad decision can cause. A general shell, browser, mailbox integration, or cloud API exposes many actions behind one connection. An agent asked to read a message may also inherit the ability to send or delete one. A maintenance task that only needs to edit a file may receive credentials capable of changing repository settings.

Schema-valid calls can still cause harm. The model can select the wrong record, send data to the wrong destination, or combine several permitted operations into a sequence that nobody intended. Validation can confirm that an argument is well formed; it cannot decide whether the action belongs to the task.

The useful unit is the smallest operation the task requires. A read-only mailbox tool is enough for summarisation. A repository tool can allow changes to files without allowing changes to branch protection. Typed fields and fixed destinations are easier to review than a free-form command, and tools that are irrelevant to the current task should not be present in the environment.

High-impact actions need authorization close to execution. An approval screen should show the exact target, operation, data leaving the system, and expected effect. Dry-run modes and limits on steps, spending, and affected resources give an operator time to intervene when a run starts to drift.

Delegated identity

Calls to external systems need an attributable identity. Shared privileged accounts make it difficult to determine which agent acted, which user authorized the action, and whether the credential was valid for that task. Long-lived credentials remain usable after the run ends and may outlive the environment in which they were issued.

Each workload should receive a distinct identity with short-lived credentials scoped to its task and environment. User delegation should preserve the user and the granted scope instead of replacing both with a generic service account. Development, test, and production agents should not share credentials.

Secrets do not need to appear in model context or tool output. The execution layer can attach a credential after the policy decision without revealing it to the model. That layer also needs a practical revocation path so incident responders can stop active runs instead of waiting for tokens to expire.

Persistent memory

Persistent memory allows an injected instruction, false fact, or sensitive value to affect later tasks. The original input may disappear from the active context while a derived memory continues to influence planning. If memory is shared, the same poisoned state can move between users, agents, or environments.

Every memory write needs a source, a reason for persisting, and an expiry policy. User data, task state, learned preferences, and security policy should not share one writable store. Sensitive or cross-task memories may need approval, and operators need a way to inspect, quarantine, and remove state without rebuilding the entire system.

Failure containment

Multi-step execution compounds errors because each tool result becomes input to the next decision. A wrong assumption, compromised tool, or poisoned memory can shape the rest of the run. Successful intermediate calls may hide the original fault until the agent reaches an external side effect.

A run needs explicit limits on time, cost, steps, and affected resources. Meaningful state changes should create checkpoints. Irreversible or externally visible actions belong behind transaction boundaries or approval gates, with authorization handled by a component separate from the model proposing the action.

Verification should test the outcome rather than command completion. A deployment process returning zero does not prove that the correct version reached the correct environment. Where the result cannot be checked automatically with enough confidence, the action still requires a person to verify it.

Logs should connect the initiating user, agent identity, proposed action, tool call, authorization result, and external effect without recording secrets or unnecessary sensitive data. That record lets an operator reconstruct the run and distinguish a model error from injected input, a compromised tool, or a permission failure.

Incident response for an agent needs controls that stop new tool calls, revoke active credentials, isolate affected memory, and restore changed state where possible. Evaluations should include repeated attempts and task-specific attacks because blocking one prompt says little about other routes to the same operation.

Baseline controls

A reasonable baseline combines source-aware retrieval, narrow tools, short-lived identity, separated memory, independent authorization, outcome verification, and a tested way to stop a run. Each measure covers a different failure path. Human approval is useful for consequential actions, but it becomes weak when prompts are vague or approvals are so frequent that nobody reads them.

This design accepts that model decisions will sometimes be wrong or manipulated. It limits which systems those decisions can change, records enough context to investigate the result, and keeps a reliable path for a person to intervene.