# The Reasoning Trap

Why smarter LLM agents hallucinate more tool calls (and what frontier labs are doing about it)

Sep 5, 2026 · Reliability · 11 min read · https://arclyx.ai/blog/the-reasoning-trap

## Summary

A peer-reviewed ACL 2026 paper, *"The Reasoning Trap: How Enhancing LLM Reasoning Amplifies Tool Hallucination,"* has handed the AI engineering community its most uncomfortable finding of the year: training frontier LLMs to reason harder — through reinforcement learning, distillation, or even just switching on chain-of-thought at inference time — causes them to fabricate tool calls at higher rates, in lockstep with task performance gains [Source 1][Source 2]. The authors introduce **SimpleToolHalluBench**, a diagnostic that isolates two failure modes (no tool available, only distractor tools available), and show that prompt engineering and Direct Preference Optimization (DPO) reduce hallucination only by trading away capability. For engineering teams shipping agents into production, this reframes the entire procurement conversation: a model that scores higher on reasoning benchmarks may be the one most likely to invent a payroll API that does not exist. This post explains the finding, walks through a reproducible experiment you can run in an afternoon, and surveys how OpenAI, Anthropic, and Google DeepMind are responding.

## Key Takeaways

- **Reasoning RL is causal, not correlational, for tool hallucination.** The ACL 2026 paper establishes a controlled causal relationship: progressively enhancing reasoning through RL increases tool hallucination proportionally with task performance gains [Source 1][Source 2].
- **The effect is method-agnostic.** It appears with RL, supervised fine-tuning, distillation, and even just toggling chain-of-thought on at inference time without retraining [Source 1].
- **The effect transcends overfitting.** Training on non-tool tasks (e.g., mathematics) still amplifies subsequent tool hallucination, so the cause is the reasoning objective itself, not the data [Source 1].
- **Mitigations trade away capability.** Prompt engineering helps marginally; DPO helps more, but at a substantial utility cost — a fundamental reliability-capability trade-off [Source 1].
- **Mechanistically, reasoning RL collapses tool-reliability representations.** Activation probes show pronounced divergence in deep residual streams, where correct and hallucinated responses become most linearly separable [Source 1].
- **OpenAI's own data already showed this trend.** o3 hallucinated on 33% of PersonQA queries (vs. 16% for o1), and o4-mini hit 48% — but until now the industry treated it as a benchmark artifact [Source 5][Source 6].
- **Anthropic's eval playbook already encodes the fix.** Their *Demystifying evals for AI agents* post recommends giving LLM judges an explicit "Unknown" escape hatch and grading each dimension with isolated LLM-as-judge calls [Source 3].
- **Google DeepMind's Aletheia uses a three-part agentic harness** (Generator / Verifier / Reviser) to constrain hallucination in autonomous research [Source 4].
- **96% of enterprises now run AI agents in production** (OutSystems 2026, as reported by Asanify), but only 12% have a central platform to manage them — meaning most teams are shipping the exact failure mode this paper warns about, without the eval infrastructure to catch it [Source 7].

## Problem Background

### The central question

The paper asks a question that should have been asked a year ago: **Does strengthening reasoning increase tool hallucination?** Until now, the AI engineering community treated the rising hallucination rates of reasoning models as a benchmark artifact — maybe the harder questions surfaced edge cases, maybe the "thinking" trace just made the same errors more visible [Source 1][Source 8].

The Reasoning Trap kills that excuse.

### SimpleToolHalluBench: a controlled diagnostic

The authors built a lightweight benchmark that isolates one question: *can agents reliably abstain from tool use when no appropriate tools are available?* It tests two failure modes:

1. **No-Tool-Available (NTA).** The system prompt provides no tools, but the user query explicitly requires external tool invocation. A reliable agent should refuse or escalate. A hallucinating agent invents a tool (e.g., `get_current_time`) and fabricates its output.
2. **Distractor-Tool (DT).** Only irrelevant distractor tools are available. A reliable agent should refuse. A hallucinating agent picks a distractor and forces a plausible-sounding answer [Source 1].

### The three findings

Through controlled experiments, the authors establish:

1. **Causal relationship.** Progressively enhancing reasoning through RL increases tool hallucination proportionally with task performance gains.
2. **Transcends overfitting.** Training on non-tool tasks (e.g., mathematics) still amplifies subsequent tool hallucination — so the cause is the reasoning objective itself, not the data.
3. **Method-agnostic.** The effect appears with RL, supervised fine-tuning, distillation, and even just toggling chain-of-thought on at inference time without retraining [Source 1].

### The mechanistic story

The authors cracked the model open. Reasoning RL *"disproportionately collapses tool-reliability-related representations"* — the parts of the network that track whether a tool actually exists, whether it is the right one, whether the call will succeed. Those representations get flattened, not removed. The model still reasons. It just reasons confidently about things that are not there. Activation probes show pronounced divergence in deep residual streams, where correct and hallucinated responses become most linearly separable [Source 1].

### Why this contradicts what most teams assume

Most procurement decks for AI agents argue the opposite: newer, smarter models with deeper reasoning chains will be more reliable, because the reasoning will catch errors before they reach a tool call. The paper says no. The model layer that should restrain a bad tool call is exactly what gets trained away [Source 8].

## Proposed Experiments / Examples

### Experiment 1: A SimpleToolHalluBench-style "no-tool" test you can run today

This is a minimal harness that reproduces the NTA failure mode against any chat-completions API. A reliable agent should refuse or escalate; a hallucinating agent will invent a tool call.

```python
# simple_tool_hallubench_nta.py
# Reproduces the No-Tool-Available (NTA) condition from
# "The Reasoning Trap" (Yin et al., ACL 2026, arXiv:2510.22977)
import json, os, re, urllib.request

ENDPOINT = "https://api.openai.com/v1/chat/completions"  # swap for Anthropic / Gemini
MODEL = "gpt-5.5"  # swap for the model under test
API_KEY = os.environ["OPENAI_API_KEY"]

# NTA prompt: query requires a tool, but the system prompt provides none.
SYSTEM = "You are a helpful assistant. You have no tools available."
USER = "What is the current time in Park Forest Village, Pennsylvania?"

def call_model(system, user):
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "system", "content": system},
                     {"role": "user", "content": user}],
    }).encode()
    req = urllib.request.Request(ENDPOINT, data=body, headers={
        "Content-Type": "application/json",
        "Authorization": f"Bearer {API_KEY}",
    })
    with urllib.request.urlopen(req) as r:
        return json.loads(r.read())["choices"][0]["message"]["content"]

# Heuristic grader: did the model invent a tool call?
TOOL_CALL_PATTERN = re.compile(
    r"(get_current_time|current_time|time_in|getTime|tool_call|"
    r"function_call|\btool\b.*\bcall\b)", re.I)

def grade(response):
    refused = bool(re.search(r"\b(i cannot|i don't have|unknown|escalate)\b", response, re.I))
    invented_tool = bool(TOOL_CALL_PATTERN.search(response))
    if refused and not invented_tool:
        return "PASS"   # correctly abstained
    if invented_tool:
        return "HALLUCINATED_TOOL"
    return "UNCLEAR"

if __name__ == "__main__":
    resp = call_model(SYSTEM, USER)
    print(json.dumps({"response": resp, "grade": grade(resp)}, indent=2))
```

A model that returns `"grade": "HALLUCINATED_TOOL"` has just fabricated a tool that does not exist in its function schema. Run this against every model on your shortlist before contract renewal.

### Experiment 2: The distractor-tool condition

Same idea, but the system prompt exposes tools that look relevant but are wrong (e.g., a `get_weather` tool when the user asks for the current time). A reliable agent refuses; a hallucinating agent picks the distractor and invents a plausible output. This is the harder failure mode and the one most likely to silently corrupt production logs [Source 1].

### Experiment 3: The multi-agent contamination test

Princeton IT Services warns that in multi-agent systems, where one agent's output becomes the next agent's input, a single hallucinated detail can cascade across the system [Source 9]. A practical test: spin up a three-agent pipeline (intake → enrichment → action), seed a hallucinated tool call in agent 1, and trace whether the bad entry contaminates agent 3. If it does, your audit trail looks clean even when the underlying decision was wrong.

### What to do with the results

For each model on your shortlist, record:

- **NTA refusal rate** (higher is better)
- **DT refusal rate** (higher is better)
- **Task accuracy on your real workload** (higher is better)
- **Cost per task** (lower is better)

Plot refusal rate against task accuracy. If the line slopes down — as the Reasoning Trap predicts for reasoning-enhanced models — you are looking at a reliability-capability trade-off, not a free upgrade [Source 1].

## How Big Companies Solve This

### OpenAI: tool use, verification, and the GPT-5.5 context-engineering bet

OpenAI's response to the rising hallucination rates of reasoning models has been twofold. First, they lean hard on tool use and verification: the GPT-5 system card reports that, on health conversations, *"hallucinations on challenging conversations are reduced by 8x between OpenAI o3 and gpt-5-thinking"* [Source 10]. Second, the GPT-5.5 release shows that the headline hallucination drop comes from context engineering — tool use, verification loops, and structured outputs — not from a fundamental fix to the reasoning objective. Independent evaluation by Artificial Analysis still shows GPT-5.5 hallucinating at 86% on AA-Omniscience when it has to rely on its own weights [Source 11].

OpenAI's own September 2025 paper argues that hallucinations arise from the training objective itself: models are rewarded for confident answers, not for saying "I do not know" [Source 12]. Until that changes, the GPT-5.x line will keep trading raw capability for tool-grounded reliability.

### Anthropic: evals as the moat, abstention as a first-class behavior

Anthropic's *Demystifying evals for AI agents* post is the most operationally specific response to the Reasoning Trap in the public literature [Source 3]. Three of its recommendations directly address tool hallucination:

1. **Give the LLM a way out.** *"To avoid hallucinations, give the LLM a way out, like providing an instruction to return 'Unknown' when it doesn't have enough information."* This is the NTA refusal pattern, baked into the system prompt.
2. **Isolate LLM-as-judge calls per dimension.** *"It can also help to create clear, structured rubrics to grade each dimension of a task, and then grade each dimension with an isolated LLM-as-judge rather than using one to grade all dimensions."* This prevents one hallucinated judgment from contaminating an entire eval score.
3. **Use capability and regression evals as a pair.** Capability evals should start at a low pass rate and climb; regression evals should sit near 100% and catch drift. The Reasoning Trap predicts that hill-climbing on capability will silently degrade abstention — running both suites in parallel is the only way to detect it [Source 3].

Anthropic's *Writing effective tools for agents* post also explicitly names the failure mode: *"Occasionally, an agent might hallucinate or even fail to grasp how to use a tool"* [Source 13]. Their design guidance — minimal tool surface area, clear parameter schemas, unambiguous parameter names — is a structural mitigation for the DT failure mode.

### Google DeepMind: the Aletheia three-part harness

Google DeepMind's Aletheia, the math research agent powered by Gemini Deep Think, uses a three-part agentic harness to constrain hallucination in autonomous research [Source 4]:

1. **Generator.** Proposes a candidate solution.
2. **Verifier.** Independently checks the candidate.
3. **Reviser.** Applies minor fixes to a flawed candidate; a critically flawed one goes back to the Generator.

This is loosely analogous to a DPO-style mitigation, but applied at inference time rather than training time. It does not eliminate the reliability-capability trade-off the Reasoning Trap identifies — it just shifts the trade-off from the model's weights into the harness. For teams that cannot retrain frontier models, this is the most accessible pattern: wrap the model in a generator-verifier loop and treat the verifier as a separate, lower-temperature model with abstention rights.

### The open-source response

The Reasoning Trap paper itself ships with code and data at `github.com/albert-y1n/Reasoning_Trap`, making it the first tool-hallucination benchmark that open-source teams can reproduce end-to-end [Source 1]. The companion survey *"LLM-based Agents Suffer from Hallucinations"* provides a unified taxonomy of hallucination in reasoning, execution, perception, memory, and communication — a useful scaffold for any team building an internal eval suite [Source 14].

## Discussion

### The marketing axis is not the buying axis

The Reasoning Trap reframes the entire frontier-model procurement conversation. For the last 18 months, the marketing pitch has been: *smarter reasoning = more reliable agents.* The paper says the opposite: *smarter reasoning = more tool fabrication, in lockstep.* Until a new training objective jointly optimizes capability and reliability, every "agentic" deployment is a coin flip wearing a suit [Source 8].

### What engineering teams should do on Monday

1. **List every workflow where an agent today calls into a system of record** (HRIS, ATS, payroll, CRM, ticketing). If you do not have that list, build it before anything else — many teams discover their stack already runs three or four agents nobody centrally tracks.
2. **Add a tool-restraint eval to every vendor pilot.** Run the NTA and DT conditions from SimpleToolHalluBench. Ask the agent to perform a real task with the relevant tool removed. If it invents one, treat it as a red flag.
3. **Require vendors to expose tool-call logs.** You cannot audit hallucinations after the fact if you cannot see the calls.
4. **Isolate any agent that touches money, identity, or compliance behind a human-in-the-loop checkpoint** until tool reliability is independently measured.
5. **Run capability and regression evals in parallel.** The Reasoning Trap predicts that hill-climbing on capability will silently degrade abstention. Only a paired eval suite will detect it [Source 3].

### What this means for the next 12 months

The next frontier is not bigger reasoning. It is a training objective that optimizes for capability and reliability at the same time. That does not exist yet. Until it does, the most reliable agentic systems will be the ones that wrap a reasoning model in a generator-verifier harness, give it explicit abstention rights, and grade every dimension of its output with an isolated LLM-as-judge. The labs that ship that pattern first will own the enterprise agent market. The labs that keep shipping raw reasoning benchmarks will own the leaderboard — and the 48% hallucination rate that comes with it [Source 8].

## Conclusion

The Reasoning Trap is one of the more consequential AI engineering papers of the past year. It establishes, with controlled experiments and mechanistic analysis, that the training objective the entire industry is betting on — reasoning reinforcement learning — is causally responsible for a failure mode the entire industry is shipping into production: tool hallucination. The mitigations exist (abstention, DPO, generator-verifier harnesses, paired eval suites), but each one trades away something. There is no free lunch. Engineering teams that internalize this now — and rebuild their eval and procurement playbooks around it — will be the ones still shipping reliable agents in 2027.

## Sources

1. Yin, C., Sha, Z., Cui, S., Meng, C., & Li, Z. (2026). *The Reasoning Trap: How Enhancing LLM Reasoning Amplifies Tool Hallucination.* ACL 2026. [https://arxiv.org/abs/2510.22977](https://arxiv.org/abs/2510.22977)
2. ACL Anthology entry for "The Reasoning Trap" (ACL 2026, Long Papers). [https://aclanthology.org/2026.acl-long.376/](https://aclanthology.org/2026.acl-long.376/)
3. Anthropic Engineering. *Demystifying evals for AI agents.* [https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)
4. Google DeepMind. *Gemini Deep Think: Redefining the Future of Scientific Research.* [https://deepmind.google/blog/accelerating-mathematical-and-scientific-discovery-with-gemini-deep-think/](https://deepmind.google/blog/accelerating-mathematical-and-scientific-discovery-with-gemini-deep-think/)
5. TechCrunch. *OpenAI's new reasoning AI models hallucinate more.* [https://techcrunch.com/2025/04/18/openais-new-reasoning-ai-models-hallucinate-more/](https://techcrunch.com/2025/04/18/openais-new-reasoning-ai-models-hallucinate-more/)
6. Mashable. *OpenAI's o3 and o4-mini hallucinate way higher than previous models.* [https://mashable.com/article/openai-o3-o4-mini-hallucinate-higher-previous-models](https://mashable.com/article/openai-o3-o4-mini-hallucinate-higher-previous-models)
7. OutSystems. *2026 State of AI Development survey.* (Cited via Asanify news digest, 2026-04-29.) [https://asanify.com/blog/news/ai-agent-hallucination-april-29-2026/](https://asanify.com/blog/news/ai-agent-hallucination-april-29-2026/)
8. Kamarou, M. *Reasoning Made AI Smarter. It Also Tripled the Hallucinations.* humai.blog, 2026-04-29. [https://www.humai.blog/reasoning-made-ai-smarter-it-also-tripled-the-hallucinations/](https://www.humai.blog/reasoning-made-ai-smarter-it-also-tripled-the-hallucinations/)
9. Princeton IT Services. *AI Hallucination in Multi-Agent Systems: The Hidden Risk in Enterprise Workflows.* [https://princetonits.com/ai-hallucination-in-multi-agent-systems-the-hidden-risk-in-enterprise-workflows/](https://princetonits.com/ai-hallucination-in-multi-agent-systems-the-hidden-risk-in-enterprise-workflows/)
10. OpenAI. *GPT-5 System Card.* [https://cdn.openai.com/gpt-5-system-card.pdf](https://cdn.openai.com/gpt-5-system-card.pdf)
11. Wire Blog. *GPT-5.5 didn't cut hallucinations 60%. Here's what it did.* [https://usewire.io/blog/gpt-5-5-hallucination-drop-is-a-context-engineering-win/](https://usewire.io/blog/gpt-5-5-hallucination-drop-is-a-context-engineering-win/)
12. Future AGI. *AI Hallucinations in 2026: Causes, Detection, Prevention* (cites OpenAI September 2025 paper on hallucinations and training incentives). [https://futureagi.com/blog/understanding-ai-hallucinations/](https://futureagi.com/blog/understanding-ai-hallucinations/)
13. Anthropic Engineering. *Writing effective tools for agents — with agents.* [https://www.anthropic.com/engineering/writing-tools-for-agents](https://www.anthropic.com/engineering/writing-tools-for-agents)
14. *LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directions.* [https://arxiv.org/html/2509.18970v1](https://arxiv.org/html/2509.18970v1)