microsoft/MInference

Health v2 · maintenance only

Adoption not part of the score

1226 stars · 80 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

MInference is a Microsoft library that accelerates long-context LLM inference using dynamic sparse attention, reducing pre-fill latency by up to 10x on an A100 while maintaining accuracy. It integrates with Hugging Face Transformers, vLLM, and SGLang and supports million-token contexts.

Use cases

• speed up prefill for long-context LLM inference
• process 1M token prompts faster on a single GPU
• accelerate LLaMA or Qwen long-context models
• integrate sparse attention kernels into vLLM or SGLang
• reduce inference latency for 128K+ context windows

When to choose

• you serve long-context LLMs and prefill latency is a bottleneck
• you need million-token prompt processing on limited GPU hardware
• you use vLLM, SGLang, or Transformers and want drop-in sparse attention speedups

When to avoid

• your prompts are short and prefill is not a bottleneck
• you need exact dense attention semantics without approximation
• you don't have access to NVIDIA GPUs

Facets

library · maturity active

llm-inference gpu-computing machine-learning large-language-models deep-learning gpu-computing performance python sparse-attention long-context prefill-acceleration vllm sglang transformers gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="microsoft/MInference")