b4rtaz/distributed-llama

Health v2 · maintenance only

Adoption not part of the score

3045 stars · 246 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A C++ application that connects multiple home devices into a cluster for distributed LLM inference using tensor parallelism. It supports Llama, DeepSeek, and Qwen models on CPU (ARM and x86_64 AVX2) and experimentally on GPU via Vulkan.

Use cases

• run llama 3 70b across multiple mac minis
• speed up llm inference by pooling home devices
• serve a large language model on raspberry pi cluster
• distributed inference of qwen 3 models
• run deepseek r1 distilled model locally on weak hardware
• tensor parallel llm inference over ethernet

When to choose

• you have several networked devices and want to run models too large for any single one
• you want CPU-only distributed inference without a GPU
• you want a simple single-command setup for popular open LLMs

When to avoid

• you have a single powerful GPU that fits the model
• you need production-grade serving with APIs and batching
• you need broad model format support beyond supported families

Facets

application · maturity active

llm-inference cli rpc large-language-models microservices artificial-intelligence windows cpp cli tensor-parallelism llama qwen vulkan raspberry-pi cluster-computing cpu-inference linux macos gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="b4rtaz/distributed-llama")