Be careful when integrating MCP tools into production.

This work shows that a 120B model handles realistic MCP tool failures no better than a 20B model.

They built MCP-Injector, a proxy that injects faults into MCP tool calls. The fault mix is calibrated on 57,240 issue and pull-request threads from 1,774 open-source MCP repositories, so it reflects how MCP servers fail in practice.

Across 1,350 runs with gpt-oss-20B and 120B and three agent stacks (a custom plan-and-execute loop, OpenAI Agents SDK, LangChain), task-completion score falls 60%, from 8.08 to 3.23 out of 10.

The larger model retains a smaller share of its fault-free score (p = 0.96), and its completion rate under faults is slightly lower: 91.1% vs. 95.6%. Differences between agent stacks are larger than the difference between model sizes.

The paper is accepted at NeurIPS 2026.

The practical takeaway is that if your MCP agents are only tested against tools that always respond correctly, add fault injection to the eval and put the effort into framework-level error handling.

https://arxiv.org/abs/2610.32555

Chat with Paper: https://academy.dair.ai/papers/tool-calling-llm-agents-under-realistic-mcp-faults-scale-does-not-purchase-fault-2610.32555