vllm-project/vllm

Health v2 · maintenance only

Adoption not part of the score

90149 stars · 21253 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

vLLM is a high-throughput, memory-efficient inference and serving engine for large language models, featuring PagedAttention, continuous batching, and broad quantization support. It exposes an OpenAI-compatible API server and runs on NVIDIA, AMD, Intel GPUs, TPUs, and CPUs.

Use cases

• serve an LLM behind an OpenAI-compatible API
• run high-throughput batch inference on GPUs
• deploy quantized models like GPTQ or AWQ
• serve MoE models like DeepSeek or Qwen3
• generate structured outputs from an LLM
• run multi-LoRA serving for many adapters
• scale LLM inference across multiple GPUs with tensor parallelism

When to choose

• you need maximum serving throughput and GPU utilization
• you want a drop-in OpenAI-compatible endpoint for self-hosted models
• you need broad hardware support including AMD GPUs and TPUs
• you serve many users concurrently with continuous batching

When to avoid

• you only need to run a small model locally on a laptop
• you need training or fine-tuning rather than inference
• you want a lightweight client library rather than a serving engine

Facets

library · maturity active

llm-inference http-server gpu-computing machine-learning large-language-models machine-learning artificial-intelligence backend python cloud paged-attention openai-compatible-api model-serving continuous-batching quantization speculative-decoding moe distributed-inference linux gpu docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="vllm-project/vllm")