deepspeedai/DeepSpeed-MII

Health v2 · maintenance only

Adoption not part of the score

2114 stars · 191 forks observed · 2026-10-12

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

DeepSpeed-MII is a Python library for high-throughput, low-latency large language model inference, built on DeepSpeed. It provides blocked KV-caching, continuous batching, Dynamic SplitFuse, and tensor parallelism with optimized CUDA kernels for serving LLMs like Llama, Mixtral, and Phi.

Use cases

• serve llama-70b with high throughput text generation
• run low-latency llm inference on gpus
• deploy mixtral moe model for inference
• benchmark llm serving throughput against vllm
• generate text from phi-2 efficiently
• reduce cost of llm inference serving

When to choose

• you need high-throughput batch text generation with DeepSpeed-optimized kernels
• you want continuous batching and SplitFuse for serving LLMs on NVIDIA GPUs
• you are already in the DeepSpeed ecosystem

When to avoid

• you need multi-vendor GPU or CPU-only inference
• you want the broadest model coverage or largest community (vLLM may fit better)
• you need a fully managed inference service rather than a library

Facets

library · maturity active

llm-inference machine-learning gpu-computing large-language-models deep-learning machine-learning python deepspeed text-generation continuous-batching tensor-parallelism kv-cache pytorch gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="deepspeedai/DeepSpeed-MII")