turboderp-org/exllamav3

Health v2 · maintenance only

Adoption not part of the score

1274 stars · 147 forks observed · 2026-09-03

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

ExLlamaV3 is a Python library for fast quantization and inference of large language models on consumer-class GPUs, featuring the EXL3 quantization format based on QTIP. It supports tensor-parallel and expert-parallel inference, speculative decoding, multimodal models, and ships with an OpenAI-compatible server via TabbyAPI.

Use cases

• run local llms on my gpu
• quantize llama models to run on consumer hardware
• serve an openai-compatible api for local models
• run large models on a single consumer gpu with quantization
• tensor-parallel inference across multiple gpus
• run qwen or mistral models locally with low vram

When to choose

• you want maximum inference speed for LLMs on consumer NVIDIA/AMD GPUs
• you need aggressive 2-8 bit quantization with minimal quality loss
• you want an OpenAI-compatible local server via TabbyAPI
• you need tensor-parallel or expert-parallel setups on consumer hardware

When to avoid

• you need CPU-only inference
• you need production multi-node serving at scale
• you require model architectures not in the supported list
• you want a managed cloud inference solution

Facets

library · maturity active

llm-inference gpu-computing http-server machine-learning large-language-models machine-learning developer-tools python windows cli quantization exl3 local-llm tensor-parallel speculative-decoding consumer-gpus tabbyapi gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="turboderp-org/exllamav3")