Memory Poisoning, Explained

AI security often focuses on prompt injection, looking at the user's input for signs of manipulation or malicious intent, but this isn't the only way to attack or subvert an AI system. Memory poisoning attacks corrupt an agent's persistent state or context store.

Memory poisoning is a more dangerous threat because it's persistent. Prompt injection generally only impacts the current session, while memory poisoning affects every future use of the poisoned data.

How Memory Poisoning Works

Most AI agents maintain some form of state across sessions. This can take the form of a key-value store, vector database, or retrieval-augmented generation (RAG) system and allows the agent to remember the facts needed to do its job.

Memory poisoning attacks are designed to introduce incorrect or harmful content into this long-term storage, influencing the agent's future behavior. Common attack vectors include:

• Direct Injection: If, during a chat session, a user tells an agent to remember something for the future, it'll store this data in long-term state. An attacker who can prompt an AI agent directly (even for a moment) can introduce malicious content into its stored state.
• Indirect Injection: AI agents commonly use external content (web pages, emails, documents, etc.) to formulate responses and inform their actions. If an attacker can access documents that an AI agent treats as trusted content, they can plant instructions in this content for an agent to commit certain things to memory. This takes advantage of the fact that an AI agent may struggle to differentiate between instructions and content.
• RAG Poisoning: RAG poisoning attacks target an AI agent's knowledge base, manipulating its responses to future queries. This includes injecting malicious content into content (web pages, documents, etc.) that are likely to be scraped and used to answer queries by the AI tool. By presenting the AI agent with incorrect "facts", RAG poisoning can change its responses and future behavior.
• Cross-Session Contamination: A customer support bot with multiple customers or several different agents may have shared-memory architecture. In this case, poisoned inputs from one user or agent may corrupt the sessions of other users or agents as well.
• Slow-Drift Poisoning: Some memory poisoning attacks are obvious and easy to detect since they rely on injecting a single piece of malicious data into the agent's state. Slow-drift attacks use many, seemingly innocuous injections to slowly manipulate the agent's beliefs and actions.

In general, prompt injection attacks are designed to manipulate a single session with an AI system. Memory poisoning corrupts the agent's memory, impacting all of its future interactions. Memory poisoning can be difficult to detect because it looks just like a benign hallucination.

Example Scenario

Imagine that a company has an IT support bot whose purpose is to improve the customer experience and decrease load on IT personnel by helping solve common or known problems. It draws from the organization's ticket management system to understand what can go wrong, why, and how to fix it.

An attacker with access to the organization's ticketing system could create fake tickets or change the data on how various issues were resolved. For example, they could state that a common problem was caused by issues with the organization's multi-factor authentication (MFA) system, and that disabling MFA for a user's account would correct the issue.

In the future, when a user has that issue, the IT support bot might use these malicious tickets as a reference and recommend disabling MFA to solve the problem. This would make it easier for the attacker to take over the user's account in the future since they only need to guess the password and not defeat MFA.

Mitigating Memory Poisoning Attacks

Memory poisoning is pernicious because it targets the agent's memory and the resources that it draws upon. If the company is only monitoring prompts, this attack could slip under the radar if the attacker used indirect injection or RAG poisoning to introduce the malicious content.

Managing the threat of memory poisoning requires memory provenance tracking and periodic audits of the agent's internal state and the documents that it uses to answer prompts. Otherwise, a well-trained agent with a clean model may still be able to go off the rails.