The Many Layers of AI Manipulation

Many AI models, including most frontier models in common use, are unexplainable models. This means that it's infeasible to determine the factors that lead to the LLM producing a particular input.

As a result, most use of modern AI requires a high level of trust, assuming that the model doesn't contain any hidden biases that impact the output. However, AI's output can be controlled and manipulated at multiple different levels.

Where LLMs Can Be Manipulated

Many AI users implicitly trust their tools, assuming that the outputs that they receive are correct or that any errors are caused by LLM hallucinations. However, the complex lifecycle and infrastructure stack of AI tools provides many opportunities for manipulation.

Training Data and Model Weights

LLM models are essentially compressions of their training data. Models start out in a random state and update their internal weights based on the training data provided to them. Give an LLM enough high-quality training data, and you have modern frontier models.

However, the accuracy of the model depends on the quality of the training data, which is why many LLM companies are facing copyright lawsuits from publishers, news organizations, etc. An attacker with access to the training data can manipulate and corrupt it to change the resulting LLM model.

Fine-Tuning

AI model training generally produces a general-purpose model that is acceptable for a wide range of different tasks. However, it often lacks the specialized knowledge and focus to make it useful in certain domains (healthcare, legal, etc.).

For this reason, models can also undergo a post-training fine-tuning process, where their weights are adjusted and trained to meet a specific purpose. This fine-tuning step can also be used to insert guardrails, biases, etc.

Retrieval-Augmented Generation (RAG)

By default, an AI agent can only answer questions based on its internal knowledge base, which limits its utility. RAG allows an AI tool to access external data (web pages, documents, etc.) before answering the question.

The information collected via RAG is treated as part of the context for the user's query. Inaccurate, corrupted, or malicious source data retrieved via RAG impacts the generated outputs.

Ongoing Training

One of the most controversial aspects of freely-available (and many paid) LLMs is that they use their users' input for additional rounds of training. This helps keep the LLMs up-to-date, tune their outputs based on user preferences, and correct any errors in the training model.

However, the assumption here is that the user inputs and responses are high-quality and correct, like the training data. A sustained, distributed campaign of providing incorrect information in response to LLM outputs could manipulate the LLM's knowledge base over time. This has even wider effects than slow-drift memory poisoning attacks, which only impact a single user session, not all sessions with the LLM.

Prompt Injection

Prompt injection attacks involve manipulating the AI agent through malicious context. This could include the prompt itself or content accessed via RAG. This content is treated as part of the context and could include embedded instructions for the LLM to execute.

In theory, the effects of prompt injection are limited to the current user session, which is bad enough. However, some LLMs have persistent context stores designed to store state, including user instructions, across sessions.

Additionally, AI agents also have access to external tools, allowing them to corrupt content or other resources. This can create a cascade effect that impacts future sessions and other LLMs accessing the same content.

Infrastructure and UI Manipulation

Assume that your AI tool is completely correct, producing high-quality, accurate data in response to user queries. Are you sure that the queries that you provide and the responses that you receive are the ones that the agent read and generated?

Most users access AI tools through the browser and communicate over the network. Malicious browser extensions can manipulate what the user sees or what is sent to the remote LLM server. This and similar threats could result in the user and the LLM experiencing very different conversations.

Managing the Risks of LLM Manipulation

LLMs and AI agents have significant supply chain risks. Even if you know that the model was properly trained in the beginning, there's the potential that its context is corrupted by false content via RAG or manipulated over time.

Ideally, an organization trains its own model with a controlled, vetted dataset and runs it locally on its own infrastructure. In most cases, the best approach is "trust but verify," allowing LLMs to produce output that is validated before use in critical workflows.