When a human reviews an AI agent’s commands before they run, they miss roughly one in three threats. That is the headline finding of a study of more than 40,000 sessions and 409,000 individual approve-or-deny decisions, where players reviewed an AI coding agent’s shell commands under time pressure and achieved a mean accuracy of just 66.3%. The result is not a verdict on human competence — it is a verdict on a design pattern. Sustained vigilance is something humans are measurably bad at, and any security model that depends on it as the sole safeguard is built to fail. The fix is not better humans; it is better systems that layer sandboxing, scoped credentials, and surgical gating on top of human review.
The Study: 40,000 Sessions of Human-in-the-Loop Approval
The data comes from a browser game built by Alex Wauters and hosted at scalex.dev (llmgame.scalex.dev). Players are placed in the role of a human reviewer watching an AI coding agent execute shell commands. For each command, they must decide whether to approve or deny it before it runs — the same decision a developer makes dozens of times a day when an agent like Claude Code, Codex, or Cursor asks for permission to run a command.
Across more than 40,000 sessions, the game collected 409,000 individual approve/deny decisions. The commands ranged from obviously destructive operations like rm -rf / to subtle, camouflaged attacks hidden inside familiar npm scripts. Roughly 34% of the commands in the game were genuinely malicious — a threat density far above what any real developer encounters, which matters for how the results should be read.
Because players knew they were being tested and the threat density was artificially high, the 66.3% accuracy figure is best understood as an optimistic ceiling, not a realistic floor. In real work, where threats are rarer and attention is divided across many tasks, performance would almost certainly be worse, not better.
The Headline Numbers: 1 in 3 Threats Missed
The central statistic is stark: players missed roughly one in three threats. Across the full dataset, mean accuracy was 66.3%, meaning that for every three malicious commands a reviewer faced, one slipped through.
The deeper numbers are more revealing:
- 32.9% of sessions ended net-negative — the player approved threats and blocked safe commands more often than they made correct calls, meaning their review actively made the system worse than no review at all.
- 35.2% of players caught every threat, but only 20.8% did so while blocking at most one in five safe commands. In other words, most people who caught everything did so by being so restrictive that the agent could barely function.
- 7% of players approved every single prompt — the equivalent of running the agent with
--dangerously-skip-permissionspermanently enabled.
These numbers describe a dilemma at the heart of human-in-the-loop approval. A reviewer can be thorough, or they can be usable, but the study suggests very few people manage both. The 7% who approved everything and the 32.9% who ended net-negative are the two failure modes of the same broken pattern.
Not All Misses Are Equal: Threat Categories and Miss Rates
The study broke down miss rates by threat category, and the differences are dramatic. Humans are excellent at spotting threats that look destructive, and poor at spotting threats that look like normal work.
| Threat category | Example | Miss rate |
|---|---|---|
| Obviously destructive | rm -rf /, chmod -R 777 / | 11.7% |
| Persistent mutation | crontab injection, git config hijack | 23.8% |
| Exfiltration / code execution | curl to unknown APIs, typosquatted packages | 33.4% |
| Scope violations | cat ~/.aws/credentials, cat ~/.kube/config | 35.0% |
The pattern is clear: the more a command looks like a normal read or script invocation, the more likely a human is to approve it. Credentials exfiltration — reading ~/.aws/credentials or ~/.kube/config — was missed roughly three times more often than destructive commands. A command that looks like a routine file read triggers none of the alarm bells that rm -rf / does, even when the file it reads is the key to your entire cloud account.
The npm Run Blind Spot: Familiarity Is the Attack Surface
The single most-missed command in the entire dataset was npm run analyze, approved 64.7% of the time — even though the history log clearly showed an exfiltration payload. The command name is familiar, it looks like a normal build step, and that familiarity is precisely what makes it dangerous.
The study pooled three camouflaged npm-script attacks (analyze, setup, deploy) and found they were missed 52.5% of the time, versus 28.4% for other exfiltration attacks. Familiarity roughly doubled the attack success rate. An attacker who hides malicious code inside a script named npm run setup is exploiting a human heuristic: “this looks like normal work, so it must be safe.”
This is the clearest single lesson in the dataset. When you review an agent’s command, you are almost certainly reading the command name, not the script it invokes. npm run X always looks safe because it always looks the same — and that uniformity is the attack surface.
Permission Fatigue Is Real, Not Just a Feeling
The study found that miss rates climb toward the end of sessions. The more approval prompts a reviewer has already processed, the less vigilance they have available for the next one. This is permission fatigue, and it is a measured effect, not an anecdote.
Every added approval prompt taxes the attention available for the next decision. A developer who has approved forty commands in a row is measurably more likely to approve the forty-first without reading it carefully. This is the same cognitive phenomenon behind alert fatigue in security operations centers — when every alert demands attention, attention becomes a scarce resource that gets rationed, and the rationing happens exactly where it is most dangerous.
The implication is uncomfortable: the more you rely on human approval as your security boundary, the weaker that boundary becomes over the course of a session. The system degrades precisely as the agent does more work.
The Other Side: Over-Blocking Benign Commands
The dilemma cuts both ways. The same reviewers who missed threats also over-blocked safe commands at alarming rates:
npm config set registrywas blocked 59% of the time — a completely benign command.rm -rf dist/was blocked 45% of the time — a routine build cleanup.kill $(lsof -t -i:3000)was blocked 43% of the time — a standard way to free a port.cat ~/.zshrcwas the most contested command, approved only 45.9% of the time, because whether it is safe depends entirely on whether secrets live in the shell profile.
Over-blocking is not harmless caution. When a reviewer blocks benign commands that the agent needs to do its job, the workflow becomes unusable, and the developer’s natural response is to bypass the friction entirely — by running the agent in full-access mode with permissions skipped. The study’s 7% who approved everything are the endpoint of this pressure. Over-blocking pushes developers toward the very bypass that makes the 1-in-3 miss rate catastrophic.
Why Human Review Alone Is Not Enough
The 1-in-3 miss rate is a failure of the design pattern, not of the humans. Humans are bad at sustained vigilance by design — it is a well-documented cognitive limitation, not a personal failing. Any security model that depends on a human maintaining perfect attention across hundreds of decisions is structurally unsound.
The real-world record confirms this. The Claude Memory Heist demonstrated exfiltration via web_fetch with no approval prompt at all — the agent exfiltrated data without ever asking. The Codex $HOME deletion incident showed a full-access mode bypassing sandboxing entirely. In both cases, the failure was not a human making a bad call; it was a system that had no defense in depth beyond the human, or that let the human’s fatigue be routed around entirely.
Anthropic’s own internal study of 1,053 testers reached a similar conclusion. The “ask the user for everything” approach had an 86.4% miss rate — worse than the game’s 66.3%, because in real work the prompts are more numerous and more routine. A classifier-based Auto mode, by contrast, had a miss rate of roughly 11%. The data is unambiguous: for routine commands, automated gating beats human review by a wide margin.
Defense in Depth: Sandboxing, Scoped Credentials, and Surgical Gating
The answer is not to remove humans from the loop — it is to stop treating them as the only line of defense. Defense in depth layers multiple independent controls so that no single failure is catastrophic:
- Sandboxed execution. Run the agent in an isolated environment with restricted filesystem and network access. Even if a malicious command is approved, it cannot reach your production data or your real credentials.
- Scoped, least-privilege credentials. Give the agent credentials that can only do what the task requires. A command that reads
~/.aws/credentialsis only dangerous if those credentials are worth stealing — scoped credentials make exfiltration far less valuable. - Surgical gating, not everywhere. Gate the commands that matter — destructive operations, network calls, credential access — and let routine commands run. This reduces the number of prompts, which directly reduces permission fatigue.
- Automated threat detection. Use pattern matching and classifiers to flag dangerous commands before they reach the human, so the human only reviews the genuinely ambiguous cases.
- Rate limiting on destructive actions. Cap impactful actions per time window (for example, a maximum of 10 destructive operations per five minutes) to contain the blast radius of a single bad approval.
- Log everything with context, and audit regularly. You cannot learn from a miss you cannot see. Full logging with surrounding context turns individual mistakes into systemic improvements.
The principle of least privilege is the foundation. Run the agent as a non-root user with restricted filesystem and network access, and the worst case of any single approval is bounded.
Practical Habits for Teams Using AI Coding Agents
For teams deploying AI coding agents today, the study translates into concrete habits:
- Read the script, not the command name. The npm run blind spot is the clearest number in the dataset. When an agent proposes
npm run X, look at what the script actually does before approving. - Pair human review with sandboxing and scoped credentials. Human review is a layer, not the whole stack. Assume a miss will happen and design so that a miss is survivable.
- Gate surgically, not everywhere. Reduce the number of approval prompts so that the ones that remain get real attention. Every prompt you remove is vigilance you preserve for the next one.
- Prefer automated gating for routine commands. The Anthropic data — 11% miss for Auto mode versus 86.4% for ask-everything — is the strongest argument for letting classifiers handle the routine and reserving human judgment for the ambiguous.
- Use pre-approval whitelists and a break-glass kill switch. Whitelist known-safe operations so they never prompt, and have a hard stop that can halt the agent immediately if something goes wrong.
The Bottom Line: Better Systems, Not Better Humans
The 1-in-3 miss rate is not a reason to abandon AI agents — it is a reason to stop building security on the assumption that a human will catch everything. Human-in-the-loop approval is a valuable layer, but it is a layer, not a foundation. The teams that deploy agents safely will be the ones that combine human review with sandboxing, scoped credentials, automated threat detection, and surgical gating — so that when a human misses a threat, the system catches it anyway.
FAQ
Q: What does “1 in 3 threats missed” actually mean? A: In a study of 40,000+ sessions and 409,000 approve/deny decisions, human reviewers approved roughly one in three malicious AI agent commands. Mean accuracy was 66.3%, meaning about a third of threats slipped through.
Q: Why do humans miss AI agent threats? A: Mostly because of permission fatigue and familiarity bias. Miss rates rise over a session as attention depletes, and commands that look like normal work — especially familiar npm scripts — are approved without scrutiny.
Q: Is human-in-the-loop approval a reliable security boundary? A: No. The study shows it is unreliable as a sole safeguard. 32.9% of sessions ended net-negative, and Anthropic found an 86.4% miss rate for “ask for everything” versus ~11% for classifier-based Auto mode.
Q: What is the npm run blind spot?
A: Camouflaged npm-script attacks were missed 52.5% of the time versus 28.4% for other exfiltration attacks. npm run analyze was the single most-missed command (64.7% approved) because familiar command names look safe.
Q: How should teams protect against AI agent threats? A: Use defense in depth: sandboxed execution, scoped least-privilege credentials, automated threat detection, surgical gating, rate limiting on destructive actions, and full logging. Human review should be one layer, not the whole stack.