# microsoft/MInference

[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

Repository: https://github.com/microsoft/MInference
Canonical: https://ross.abutalabs.com/products/minference
Homepage: https://aka.ms/MInference
Language: Python
License: MIT
License Family: permissive
Last push: 2026-04-08T08:04:38+00:00

## Health v2 (maintenance only)
Score: 49/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 76, release rhythm 8, longevity 59
- inputs: {"age_days": 833, "days_push": 147, "days_rel": 442, "gap_med": null, "n_releases_24m": 1}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1226, forks 80 (observed 2026-08-28T04:04:03.139573+00:00)

## What it is
MInference is a Microsoft library that accelerates long-context LLM inference using dynamic sparse attention, reducing pre-fill latency by up to 10x on an A100 while maintaining accuracy. It integrates with Hugging Face Transformers, vLLM, and SGLang and supports million-token contexts.

## Use cases
- speed up prefill for long-context LLM inference
- process 1M token prompts faster on a single GPU
- accelerate LLaMA or Qwen long-context models
- integrate sparse attention kernels into vLLM or SGLang
- reduce inference latency for 128K+ context windows

## When to choose
- you serve long-context LLMs and prefill latency is a bottleneck
- you need million-token prompt processing on limited GPU hardware
- you use vLLM, SGLang, or Transformers and want drop-in sparse attention speedups

## When to avoid
- your prompts are short and prefill is not a bottleneck
- you need exact dense attention semantics without approximation
- you don't have access to NVIDIA GPUs

## Facets
- artifact type: library
- maturity: active
- function: llm-inference, gpu-computing, machine-learning
- domain: large-language-models, deep-learning, gpu-computing, performance
- platform: python
- tags: sparse-attention, long-context, prefill-acceleration, vllm, sglang, transformers, gpu, linux

## Member repositories
- microsoft/MInference (main) score 49

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:04:03.139573+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T06:15:20.257452+00:00, confidence not recorded.
  - readme: https://github.com/microsoft/MInference (fetched 2026-08-28T04:04:03.139573+00:00, sha 51a8a39ab3bb)
  - registry_pypi: https://pypi.org/pypi/minference/json (fetched 2026-08-29T12:23:28.920706+00:00, sha 760dbfa9a899)
- Data as of 2026-08-30T08:39:29.467469+00:00.