turboderp/exllama

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

2936 stars · 220 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

ExLlama is a standalone Python/C++/CUDA implementation of the Llama architecture optimized for running 4-bit GPTQ quantized weights on modern NVIDIA GPUs. It serves as a memory-efficient, fast alternative to the Hugging Face transformers implementation for local LLM inference.

Use cases

• run quantized llama models on a consumer gpu
• load 4-bit gptq models with low vram usage
• fast local inference of llama on rtx 3090 or 4090
• build a local chatbot with a quantized language model
• benchmark inference speed of quantized llama models
• integrate llama inference into a python application

When to choose

• you have a modern NVIDIA GPU (RTX 30-series or newer) and want memory-efficient 4-bit GPTQ inference
• you need faster generation than HF transformers with quantized Llama weights
• you want a lightweight standalone library rather than a full inference framework

When to avoid

• you have an older GPU with poor FP16 support (Pascal or earlier) - AutoGPTQ or GPTQ-for-LLaMa are better fits
• you need AMD/ROCm support that is well tested and optimized
• you need a production-hardened, stable solution - the project is still a work in progress

Facets

library · maturity active

llm-inference machine-learning deep-learning large-language-models machine-learning gpu-computing python windows cross-platform gptq quantization llama 4-bit cuda inference-optimization gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="turboderp/exllama")