Mesh-LLM/mesh-llm

Health v2 · maintenance only

Adoption not part of the score

3305 stars · 397 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Mesh LLM is a Rust-based distributed LLM inference runtime that pools GPUs and memory across machines into a single OpenAI-compatible API. It supports public and private mesh networking, automatic routing, and splitting large models into layer slices across nodes.

Use cases

• run large LLMs across multiple machines without buying bigger GPUs
• pool GPUs from homelab nodes into one inference endpoint
• share compute with a public mesh to power agents and chat
• serve an OpenAI-compatible API for coding agents and existing clients
• split a model too large for one machine across a LAN cluster
• route requests between multiple locally hosted models

When to choose

• you have several machines with modest GPUs and want to run models bigger than any single one
• you want a drop-in OpenAI-compatible endpoint backed by your own hardware
• you want to share or borrow compute via public or invite-based meshes
• you need tool calling, streaming, and structured outputs from a self-hosted runtime

When to avoid

• you need a single-GPU setup with maximum inference throughput and no networking overhead
• you require enterprise-grade SLAs or managed hosting
• you depend on non-OpenAI-specific serving features like tensor-parallel vLLM deployments
• you cannot accept the latency of layer-split pipelines across slow network links

Facets

application · maturity active

llm-inference http-server api-framework cli agent-framework chatbot gpu-computing plugin-system large-language-models artificial-intelligence microservices self-hosted developer-tools windows cross-platform cli self-hosted rust distributed-inference openai-compatible-api p2p model-sharding homelab mesh-networking gguf ai-agents linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="Mesh-LLM/mesh-llm")