LlamaEdge/LlamaEdge

Health v2 · maintenance only

Adoption not part of the score

1650 stars · 149 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

LlamaEdge is a lightweight Rust and WasmEdge-based runtime for running open-source LLMs locally or on edge devices, with CLI chat apps and OpenAI-compatible API servers. It supports text generation, embeddings, speech-to-text, text-to-speech, and image generation models in a compact, dependency-free package.

Use cases

• run open-source LLMs locally on my laptop
• serve an OpenAI-compatible API for a local model
• chat with a fine-tuned GGUF model from the command line
• host a private LLM API server without Python dependencies
• run LLM inference on edge devices with GPU acceleration
• create a chatbot web UI backed by a local model
• generate embeddings locally for a knowledge base

When to choose

• you need a lightweight, portable local LLM runtime without Python or heavy dependencies
• you want OpenAI-compatible endpoints for open-source models on CPU or GPU
• you deploy LLM apps across heterogeneous edge hardware (CPUs, GPUs, NPUs)
• you want a single integrated runtime plus API server instead of gluing multiple tools

When to avoid

• you need the broadest model support or ecosystem of Ollama or llama.cpp
• you rely on Python-based tooling and want to extend the runtime in Python
• you require multi-node distributed inference or large-scale serving
• you need features beyond the supported model types (e.g., advanced fine-tuning)

Facets

application · maturity active

llm-inference http-server cli speech-recognition tts rag large-language-models artificial-intelligence self-hosted developer-tools cross-platform cli self-hosted wasm openai-compatible-api gguf wasmedge local-llm edge-computing chatbot-ui gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="LlamaEdge/LlamaEdge")