GatewayBench: Unified Capability and Security Evaluation Platform for MCP Gateways
En palabras de los autores
AI agents have increasingly interacted with external environments by invoking tools, with Model Context Protocol (MCP) emerging as a standard interface for connecting agents to heterogeneous tools and services. As organizations deploy agents at scale, MCP gateways have become a critical middleware layer for aggregating, governing, and shaping the tool surface exposed to agents. Despite the growing importance of MCP gateways in agentic systems, no prior work has systematically studied how gateway design affects agent capability, efficiency, and security. To address this gap, we introduce GatewayBench, the first unified evaluation framework and benchmark for MCP gateways. We characterize gateway architectures into two major paradigms: flattened gateways and hierarchical gateways. GatewayBench comprises 524 tasks across six domains, together with a pool of 11,118 real-world tools from 251 authentic MCP servers. To emulate large-scale deployments, we systematically inject task-irrelevant tools sampled from this pool into each agent environment, enabling controlled evaluation as the accessible tool surface scales. Based on GatewayBench, we conduct extensive experiments across seven widely deployed MCP gateways and diverse agent configurations, covering both benign and adversarial tasks. Our results reveal substantial differences across gateway designs in task completion, tool-use efficiency, and resilience to attacks, and further demonstrate that these effects interact strongly with the underlying model and agent harness. These findings highlight the MCP gateway as a consequential yet underexplored component of the agentic AI stack. By enabling controlled evaluation of gateway architectures under realistic tool ecosystems, GatewayBench provides both practical guidance for MCP gateway design and a rigorous testbed for developing more capable, efficient, and secure agentic systems.
En palabras simples Resumen con IA
Los agentes de IA usan herramientas a través de una interfaz estándar llamada MCP, y las pasarelas ahora gestionan estas conexiones a gran escala. Nadie había estudiado cómo el diseño de las pasarelas afecta el rendimiento y la seguridad de los agentes. Los autores crearon GatewayBench, el primer marco de prueba para pasarelas MCP. Incluye 524 tareas y 11,118 herramientas reales de 251 servidores, con herramientas irrelevantes adicionales para simular despliegues grandes. Las pruebas de siete pasarelas mostraron diferencias importantes en finalización de tareas, eficiencia y resistencia a ataques, que variaron según el modelo de IA y la configuración.
Escrito por Kimi K2.5 usando solo el abstract de arriba. Jev verificó que cada oración esté respaldada por las oraciones del abstract que cita y eliminó las que no. Pasá el cursor por una oración para ver de dónde sale.
Apareció: lunes, 19 de octubre. arXiv. Preprint, todavía sin revisión por pares.