turboderp-org/exllamav2

Health v2 · maintenance only

Adoption not part of the score

4611 stars · 341 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

ExLlamaV2 is a fast Python inference library for running large language models locally on modern consumer GPUs, with support for EXL2/GPTQ quantization, paged attention via Flash Attention, and dynamic batched or streamed generation. The project is now archived, with development continuing in ExLlamaV3.

Use cases

• run local llms on my gpu
• fast inference for quantized llama models
• serve an openai-compatible local llm api
• generate text from gptq quantized models
• stream llm token generation with asyncio
• batch multiple prompts for gpu inference

When to choose

• you want maximum token throughput for quantized LLMs on consumer NVIDIA GPUs
• you need a lightweight Python inference engine rather than a full server
• you are pairing it with TabbyAPI for an OpenAI-compatible local endpoint

When to avoid

• you are starting a new project - use ExLlamaV3 instead since this repo is archived
• you need CPU-only or Apple Silicon-first inference
• you want a turnkey chat UI rather than an inference library

Facets

library · maturity maintenance

llm-inference machine-learning large-language-models machine-learning gpu-computing python windows quantization exl2 gptq local-llm consumer-gpus speculative-decoding archived gpu linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="turboderp-org/exllamav2")