abetlen/llama-cpp-python

Health v2 · maintenance only

Adoption not part of the score

10585 stars · 1446 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Python bindings for llama.cpp providing low-level ctypes access to the C API plus a high-level API for text completion with OpenAI-like interfaces. It also ships an OpenAI-compatible web server supporting function calling, vision models, and multiple models.

Use cases

• run llama.cpp models from python
• serve a local openai-compatible llm api
• run gguf models locally without a gpu cloud
• use a local llm with langchain or llamaindex
• replace github copilot with a local code completion model
• run multimodal vision models locally

When to choose

• you want to run quantized gguf models locally in python
• you need an openai-compatible endpoint without external services
• you want cpu or custom hardware-accelerated inference via cmake options

When to avoid

• you need managed multi-node inference at scale
• you prefer a pure-python or cloud-hosted inference stack
• you cannot compile C/C++ and no prebuilt wheel fits your hardware

Facets

library · maturity active

llm-inference http-server sdk large-language-models machine-learning artificial-intelligence developer-tools python windows cross-platform llama-cpp gguf openai-compatible-api local-llm langchain llamaindex ctypes-bindings text-generation linux macos gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="abetlen/llama-cpp-python")