alibaba/rtp-llm

Health v2 · maintenance only

Adoption not part of the score

1316 stars · 266 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

RTP-LLM is Alibaba's high-performance LLM inference engine written in C++/CUDA, optimized with kernels like PagedAttention and FlashAttention and supporting INT8/INT4 quantization. It is production-proven across Alibaba business units and serves models via an OpenAI-compatible API.

Use cases

• serve llama models with low latency on nvidia gpus
• deploy an openai-compatible llm api server
• run quantized int4/int8 llm inference
• batch serve llm requests in production
• self-host large language model inference on v100 gpus

When to choose

• you need production-grade LLM serving on NVIDIA GPUs, especially V100
• you want built-in quantization and KV cache optimization
• you deploy within Alibaba ecosystem or need prefill/decode separation

When to avoid

• you need broad multi-hardware support like AMD ROCm or Apple silicon today
• you prefer a Python-native stack like vLLM for extensibility
• you need Windows or macOS support

Facets

library · maturity active

llm-inference gpu-computing http-server caching large-language-models machine-learning gpu-computing self-hosted cpp python llm-serving cuda-kernels quantization paged-attention model-deployment vllm-alternative linux gpu docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="alibaba/rtp-llm")