mostlygeek/llama-swap

Health v2 · maintenance only

Adoption not part of the score

5480 stars · 433 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

A Go-based proxy service that lets you run multiple local AI inference servers (llama.cpp, vllm, ComfyUI, etc.) and hot-swap between models on demand behind a single OpenAI/Anthropic-compatible API endpoint. It ships as a single dependency-free binary configured by one file, with a built-in web UI for managing running models.

Use cases

• run multiple local LLMs and switch between them on demand
• expose one OpenAI-compatible API for several local models
• automatically unload and load models to fit limited GPU memory
• proxy requests to llama.cpp or vllm behind a unified endpoint
• serve chat, embeddings, image, and audio models from one local server
• manage local AI model servers with a web UI

When to choose

• you run several local models on limited hardware and need automatic swapping
• you want a single OpenAI/Anthropic-compatible endpoint for heterogeneous inference servers
• you want a zero-dependency, single-binary local AI gateway
• you use llama.cpp, vllm, ComfyUI, or stable-diffusion.cpp together

When to avoid

• you need multi-model concurrent serving on abundant GPU memory rather than swapping
• you run cloud-hosted inference and don't need local model management
• you need a production-scale load balancer across many replicas

Facets

service · maturity active

llm-inference http-server proxy api-framework large-language-models artificial-intelligence self-hosted developer-tools windows go self-hosted model-swapping openai-api anthropic-api llama-cpp vllm local-llm reverse-proxy single-binary linux macos docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="mostlygeek/llama-swap")