ggml-org/llama.cpp

Health v2 · maintenance only

Adoption not part of the score

125803 stars · 22236 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

llama.cpp is a C/C++ library and toolset for running large language model inference locally on CPUs and GPUs, built on the ggml tensor library. It ships CLI tools, an OpenAI-compatible REST server (llama-server), and supports quantized GGUF models across a wide range of hardware.

Use cases

• run an LLM locally on my laptop without an API key
• serve a local model with an OpenAI-compatible API
• run quantized GGUF models on CPU
• run LLM inference on Apple Silicon
• host a private chatbot on my own machine
• run a local coding agent with a local model
• run LLM inference on a Jetson or embedded GPU

When to choose

• you want private, fully local LLM inference with no telemetry or API keys
• you need to run models on heterogeneous hardware from laptops to clusters
• you want a lightweight C/C++ inference stack with quantization support

When to avoid

• you need large-scale multi-node training or fine-tuning infrastructure
• you prefer managed cloud inference APIs
• you need a high-level Python training framework rather than an inference runtime

Facets

library · maturity stable

llm-inference machine-learning cli http-server gpu-computing large-language-models artificial-intelligence machine-learning developer-tools self-hosted cross-platform windows cli wasm python cpp c ggml local-llm quantization cpu-inference apple-silicon openai-compatible-api llama-server linux macos docker gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="ggml-org/llama.cpp")