Confident AI Can Be as Dangerous as Compromised AI

Compromised AI is a major discussion point in AI security. Prompt injection, memory poisoning, etc. assume that there's a malicious attacker somewhere in the mix.

However, an AI that makes a mistake can be just as dangerous. AI's built-in self confidence is bad enough when it's an LLM chatbot; autonomous AI agents make this so much worse.

AI is Inherently Confident

LLMs work by generating answers to questions one token at a time. During their training phase, the tools appear to create internal models of the world in order to minimize prediction errors (e.g. when their output doesn't match the expected or "right" answer).

The LLM treats this model and any context provided to it at runtime (prompt, RAG-retrieved data, etc.) as ground truth. Every output is generated one token at a time with the goal of producing a plausible, coherent response. So, the correct answer and a crazy hallucination sound the same (even if the model could tell, if asked, that its answer is wrong).

This issue isn't limited to LLMs. The Dunning-Kruger effect is when someone with very little knowledge of a topic believes they know far more than they do and speak with unearned confidence. In contrast, people who know a lot about a topic are aware of how much they don't know and will understate their points (e.g. imposter syndrome). Between the two, it can be difficult to differentiate between the truth and "hallucinations," just like with LLMs.

Agency Introduces Additional Risk

Most AI security models rely on the "human in the loop." With an LLM chatbot, the confident AI response is read by a human, who evaluates and acts on it. In theory, this means that a skeptical reader will identify hallucinations and act accordingly. In practice, "human in the loop" is overrated.

Agentic AI exacerbates the security issues of confident AI because the role of the human in the process is greatly reduced. AI agents can operate autonomously, calling tools and launching workflows without human review or intervention. If the AI hallucinates, this can have much larger impacts than an incorrect response from an LLM chatbot.

For example, many organizations are deploying AI agents to handle customer support cases. While the customer interacts with an LLM chatbot, this tool also has the ability to issue refunds or credits or take other actions to remedy issues. If the AI hallucinates a precedent or misinterprets a policy (e.g. when a tribunal held Air Canada liable for a chatbot-hallucinated bereavement-fare policy), it could incorrectly issue refunds or take other actions that complete before a human can ever review them.

Output Controls are as Important as Input Controls

AI security policies commonly focus on managing threats within inputs to AI systems. Detecting prompt injection, inputs containing sensitive information, and excessive AI access are all important to securing the organization's use of AI.

However, it's equally important, if not more so, to examine and control the outputs produced by AI systems, especially agentic ones. An AI agent that is simply wrong, no attacker involved, can cause just as much damage as a prompt injection attack but won't be detectable by inspecting inputs. AI agents' access should be closely scoped to their role, and all outputs should be evaluated for their potential downstream impacts, with the most risky ones requiring explicit approval.

Educational content, not legal, financial, investment, or professional security advice. Terms