InternLM/lmdeploy

Health v2 · maintenance only

Adoption not part of the score

8024 stars · 728 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

LMDeploy is a toolkit for compressing, quantizing, deploying, and serving large language models, built around its high-performance TurboMind inference engine. It provides OpenAI-compatible serving APIs, CLI tools, and optimized CUDA kernels for fast LLM inference on NVIDIA GPUs.

Use cases

• serve llm with openai compatible api
• quantize llama model to 4bit
• run deepseek v3 inference on gpu
• deploy chatbot backend for large language model
• speed up llm inference with turbomind
• compress and serve internlm models
• benchmark llm inference throughput

When to choose

• you need high-throughput, low-latency LLM serving on NVIDIA GPUs
• you want built-in quantization (4bit, FP8, MXFP4) with a serving engine
• you deploy InternLM, Llama, Qwen, or DeepSeek models with an OpenAI-compatible API

When to avoid

• you need CPU-only or non-NVIDIA hardware inference
• you only want to fine-tune or train models rather than serve them
• you prefer a simpler pure-PyTorch stack without custom CUDA kernels

Facets

library · maturity active

llm-inference gpu-computing http-server api-framework chatbot large-language-models deep-learning machine-learning artificial-intelligence gpu-computing developer-tools python cli turbomind quantization model-serving cuda-kernels openai-compatible-api kv-cache inference-engine linux gpu docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="InternLM/lmdeploy")