Michael-A-Kuykendall/shimmy

Health v2 · maintenance only

Adoption not part of the score

5808 stars · 559 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Shimmy is a single-binary, OpenAI-compatible LLM inference server written in pure Rust, running GGUF models on a WebGPU-based engine (Airframe) with no Python or C++ dependencies. It serves as a lightweight (~5MB) drop-in alternative to Ollama for local, private model inference on any GPU.

Use cases

• run local LLM inference server without Python or llama.cpp
• serve GGUF models with an OpenAI-compatible API
• replace Ollama with a lightweight single-binary server
• run inference on NVIDIA, AMD, Intel, or Apple Silicon GPUs via WebGPU
• point existing AI tools at a local private inference endpoint
• serve Hugging Face GGUF models locally

When to choose

• you want a minimal, dependency-free local inference server in a single binary
• you need OpenAI/Ollama API compatibility for existing tooling
• you want deterministic outputs and WebGPU acceleration across GPU vendors
• you prefer Rust-based tooling without Python runtimes or C++ toolchains

When to avoid

• you need model architectures or quantizations outside the certified GGUF set
• you require advanced features like multi-GPU sharding or fine-tuning
• you depend on llama.cpp-specific extensions or ecosystem integrations
• you need a mature, battle-tested server like vLLM for high-throughput production serving

Facets

service · maturity active

llm-inference http-server api-framework gpu-computing large-language-models artificial-intelligence self-hosted developer-tools windows rust cli self-hosted cross-platform openai-compatible gguf webgpu ollama-alternative local-ai single-binary inference-server huggingface linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="Michael-A-Kuykendall/shimmy")