ทันเอไอ

Research Shows Skill Representation Affects AI Agent Selection

A case study of the Tinycloud video-generation system found that representation—how skill information is presented—affects how AI agents retrieve and select skills.

Kevin Dela Rosa published the paper “Representation Affects Retrieval: A Case Study of Skill Discovery and Routing in a Multimodal Agent Harness” on arXiv under number 2608.20389. The study examines Skill Discovery and Routing in a production-scale Multimodal Agent system.

The key question is how to represent each skill as an agent’s skill library grows, so the model can select the right tool for a user’s task. In small systems, a language model can directly evaluate a list of skills included in the System Prompt and decide based on Context. As the number of skills increases, however, systems often rely on Embedding-based Retrieval to shortlist relevant skills first.

The study uses Tinycloud, a multimodal video agent system, as its case study. It divides skill representations into two main types: Tool-skills, which wrap a single external API or system tool and act like the agent’s basic vocabulary; and Workflow-skills, which coordinate Tool-skills to complete multi-step tasks.

The main finding is that Representation—the way information is structured and described—affects a model’s ability to discover, rank, and select the appropriate skill for an instruction. This applies both to Prompt-based selection in small systems and Embedding-based Retrieval as skill libraries grow. However, because the research is based on a single system, its findings should not be assumed to apply equally to every type of agent.

The paper was accepted for presentation at the AgentSearch ’26 workshop at SIGIR 2026 in Melbourne, Australia, on July 24, 2026. This reflects how designing skill descriptions is becoming an Information Retrieval problem, not merely a matter of Prompt writing.