The Cascading Effects of Prompt Injection in Agentic Workflows

Discussions of prompt injection attacks often focus on user sessions within a chatbot. The assumption is that the user provides malicious input, which prompts the LLM to generate undesirable outputs.

However, the growth of agentic AI dramatically amplifies the risks associated with prompt injection attacks. When an AI agent is just one link in a chain with the ability to send requests to tools and process their outputs, prompt injection attacks grow far more dangerous.

When the User Isn't the Attacker

While prompt injection attacks are often described as involving a malicious user, the user isn't an LLM's only source of input. AI agents commonly have access to various tools, such as the ability to search the web, access APIs, or read files.

Any of these sources of external data can potentially contain malicious content injected by an attacker. If this is the case, an agent may treat this indirect prompt injection as a legitimate instruction and act accordingly.

The attacks are especially dangerous because they would be overlooked by tools and processes focused on the traditional framing of prompt injection as a user-driven attack. Red teaming and AI security solutions may not consider or address the case where a legitimate external tool produces malicious outputs.

The Cascade Effect

In the context of a chatbot or LLM session, prompt injection is dangerous because of the output that it produces for the user. In this case, the human is responsible for vetting and sanity-checking the provided data.

However, with an autonomous AI agent, there may not be a human in the loop, and agents' access to external tools amplifies the effects of a malicious or rogue agent. For example, an agent that suffers from prompt injection (direct or indirect) could initiate automated workflows or plant incorrect data or additional malicious content in various resources.

This can continue the cycle, infecting additional AI agents, or implement a persistence mechanism as poisoned resources reinfect each new session after they access the malicious content. Additionally, as AI agents accrue access and privileges over time, their ability to perform malicious actions grows, as benign privileges may combine into unintended, dangerous levels of access.

Managing Prompt Injection Risk in Autonomous Workflows

Agentic AI introduces new security risks above and beyond those of LLM chatbots. While traditional, direct prompt injection is a threat, the ability of AI agents to access tools and launch automated workloads introduces new, indirect prompt injection threats as well.

This new paradigm creates the potential for cascading attacks, where one rogue agent can poison the context of other agents or take actions that cause harm to the business. AI security programs should monitor and manage all AI inputs and outputs, not just the prompt.