Why “Human in the Loop” is Often Security Theater

Having a "human in the loop" is the most common approach to managing AI risks to security and the business. Human reviewers can theoretically catch hallucinations, incorrect information, and anything else that could result in damage to the business.

However, the reality is that most human review is largely rubber-stamping the AI's outputs, potentially without even reviewing them. Often, employees are pressed to adopt AI to increase efficiency and throughput, which doesn't mesh with in-depth, thoughtful review.

Why "Human in the Loop" Fails

In theory, human review is the best way to maximize AI efficiency while managing its flaws. However, things rarely work out this way for a few reasons:

• High-Volume Outputs: Organizations are paying for AI tools and tokens to increase speed and efficiency. Every time that a human participates in an automated workflow, they become a delay and bottleneck. To achieve the goals of the AI deployment, people perform shallow reviews that would miss any and all AI risks.
• Approval Fatigue: Even if humans are trying to perform an in-depth review of each output, they get overwhelmed. SOC fatigue is a well-known issue, and the same applies to thoughtfully reviewing each AI approval request. Often, reviewers will fall back to pattern recognition and approval by default, which can miss subtle or unusual issues.
• Limited Approval Interfaces: Often, approval requests from AI agents include a line or two of context and a yes/no decision. The human reviewer doesn't have the insight into the agent's chain of reasoning or the downstream implications of their decision. As a result, they're only looking at the specific request in question without considering the potential for AI agents gaining unwanted permissions or having a bad output cause a cascade effect.

Rubber Stamping and Vibe Coding

One of the most obvious areas where "human in the loop" is already falling apart is vibe coding. AI's ability to rapidly write functional software that mostly does what the prompter intended has dramatically increased the output of software developers and allowed people with no coding background to create their own apps.

Often, this is accomplished by automatically approving every request made by the agent or to enable checks to be skipped entirely (something that's even labeled as "dangerous" within the tools themselves). As a result, software developers are pushing code that they've never read, which goes through "peer reviews" that last less than a minute for thousands of lines of code.

In the ideal case, vibe-coded applications won't contain exploitable vulnerabilities since the coding tools have been taught to recognize and avoid common patterns. However, business logic flaws or less common "errors" (like hard-coding calls to cryptographic libraries) are likely to be included and overlooked.

Making "Human in the Loop" Functional

As things are today, relying on a human in the loop to manage AI security risks doesn't work. Human reviewers are overwhelmed, underinformed, and incentivized not to actually perform in-depth, useful reviews. As a result, anything that they're supposed to catch is highly likely to slip through the cracks.

That said, "human in the loop" is still one of the most effective methods of managing these risks. An effective review program involves:

• Low Approval Volumes: Humans can only review so many approval requests before they start rubber-stamping everything. Approval processes should be designed so that only the most anomalous or risky requests require explicit approval.
• Risk Prioritization and Analysis: Determining whether an approval request reaches a human requires analyzing the risk of allowing or denying the action, including the agent's thought process and downstream implications. This information should be used to decide both whether a request reaches a human and provided to the reviewer so they have the context required to make a useful decision.
• Sampling-Based Reviews: For actions that would require a large number of approvals of similar severity, a sampling approach can be used to limit volumes. Randomly deciding whether or not a review gets a human reviewer (vs. an independent AI one) reduces the risk of large-scale fraud and undetected rogue agents.
• Monitoring Approval Patterns: Often, it's easier to automatically approve requests, and reviewers are incentivized to do so to get the full benefits that AI offers. Monitoring approval patterns for signs of automatic approvals enables the organization to take action to manage employees who are just rubber-stamping AI outputs.

AI models aren't perfect and can introduce significant security risks even when they're not targeted by prompt injection attacks. Having a human in the loop can help, but only if it's done correctly.

Educational content, not legal, financial, investment, or professional security advice. Terms