ทันเอไอ

iroh Launches Mesh LLM: New Tech Unites Multiple Computers to Run Giant AI Models Decentrally

The Mesh LLM project from the iroh team introduces a new concept for running Large Language Models by distributing processing across a computer network, enabling multiple small machines to collaboratively run models too large for a single machine.

The iroh team has launched a new project called Mesh LLM, which offers a novel approach to processing Large Language Models (LLMs) by leveraging a decentralized network instead of relying on a single large server. When a user sends a query, the system can operate in three modes: processing on the user's own machine, forwarding to another peer in the network with a readily available model, or utilizing a special mode for giant models.

A key highlight of this architecture is the mode called "Skippy," specifically designed to handle extremely large models. Skippy's concept involves segmenting the model into layers and distributing each segment for processing across different machines. For instance, one machine might process layers 0-15, while a second machine takes over layers 16-31, with results (activations) then pipelined consecutively. This method enables groups of computers with limited resources to collectively run models that no single machine could handle alone.

The underlying operation utilizes iroh's own networking technology. Each node in the network connects directly peer-to-peer without a central server. iroh's system handles complexities like hole-punching and NAT traversal (techniques for traversing private networks) to establish stable connections using the QUIC protocol. For developers, Mesh LLM also provides an OpenAI-compatible API, simplifying integration. Currently, it supports over 40 models, ranging from small models with less than a billion parameters runnable on laptops to large Mixture-of-Experts models with up to 235 billion parameters.