Researchers propose "Nexus," a system that helps Agentic LLMs handle large numbers of tools faster and reduces startup bottlenecks
A new arXiv paper proposes a solution to the Time-to-First-Token bottleneck for large language models that use tools through Model Context Protocol
A paper titled "Nexus: Depth-Adaptive KV-Cache Splicing and Retrieval-Decoupled Tool Routing for Agentic LLMs on Unified Memory," authored by Mustafa Arslan and published on arXiv (arXiv:2608.20397), examines a common bottleneck in Agentic Large Language Models (LLMs)—models capable of making decisions and invoking external tools.
Agentic LLMs using Model Context Protocol (MCP) currently need to re-encode tool schemas for every conversation turn. This makes the prefill stage, whose complexity grows quadratically with sequence length, a major driver of rapidly increasing Time-to-First-Token (TTFT) as the Tool Registry expands. In some cases, the data may even exceed the Context Window.
To address this, Nexus decouples routing from the cost of schema prefill. It uses an INT8 semantic lookaside buffer (SLB) and a calibrated cross-encoder margin gate to select tools through retrieval, while arguments are generated from compressed text signatures instead of a conventional Key/Value (KV) cache.
Experimental results show that this workflow is depth-independent, with routing accuracy holding at around 89% even as the registry scales to 250 tools.
This research could advance future AI Agent development by enabling models to connect to and invoke large numbers of tools efficiently without severe initial processing delays.