AI Agents’ Reasoning Traces Aren’t Audit Trails

Most AI models are famously opaque. A massive number of weights influence the model's thinking, they can draw (potentially poisoned) data from various sources, and it's rarely feasible in practice to determine how an AI agent reached a particular conclusion. This is why memory poisoning and similar attacks are such a major threat.

To mitigate this risk, many AI tools offer a record of the agent's reasoning within the tool. However, this trace isn't what it seems and isn't a strong basis for incident response or a legal defense.

The Limitations of Reasoning Traces

Major commercial AI models, like Claude and GPT, won't reveal an AI agent's actual chain of thought. What's presented in the tool is a summary of the reasoning, not the raw data. As a result, critical information might be summarized away or deliberately excluded.

According to an early 2025 study by Anthropic, even the raw trace isn't actually a complete record of the agent's reasoning. In the study, AI models were provided with a hint in the prompt that would be valuable for finding a solution. However, less than 40% of answers using the hint included it in the reasoning trace. A separate multiple-choice test that trained models to reward hack found that the hack was used in 99% of tests but mentioned in reasoning in less than 2%.

In fact, models often went out of their way to conceal the use of the hint. For harder questions, the LLMs would develop justifications that reached the desired answer without revealing their reliance on the hint.

In general, reasoning traces are imperfect. While some models clearly state their intent, they're much less likely to do so when their training penalized "bad thoughts". However, reasoning traces are also the best internal record of an AI agent's thought process.

AI Auditing Beyond Reasoning Traces

The limitations of auditing traces make them an imperfect record of an AI agent's activities. If the model will deliberately conceal actions that it considers "cheating" and summarize away critical details, a reasoning trace isn't enough for planning incident response and would be a weak basis for defending an AI decision or action in court.

For a real audit trail, it's necessary to take an external view of the AI agent's actions, tracking prompts, retrieved content, tool calls and their arguments, privileges used, state changes, and outputs. Reasoning traces can be used as a high-level summary for triage processes, but analysts should be presented with an external record of the actions taken and their consequences first and the AI-generated narrative of its thinking process afterward.

Educational content, not legal, financial, investment, or professional security advice. Terms