FasterDecoding/Medusa

Health v2 · maintenance only

Adoption not part of the score

2770 stars · 205 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Medusa is a framework that accelerates LLM text generation by adding multiple decoding heads to an existing model, avoiding the need for a separate draft model. It achieves roughly 2-3.6x speedups using tree-based attention and a typical acceptance scheme during decoding.

Use cases

• speed up llm text generation
• accelerate inference without a draft model
• make local llm hosting faster
• reduce llm generation latency
• add speculative decoding heads to a fine-tuned model
• faster sampling-based generation

When to choose

• you want faster LLM generation without deploying a separate draft model
• you serve single-batch (batch size 1) inference for local model hosting
• you want parameter-efficient training that leaves the base model untouched
• you need faster non-greedy (sampling) generation

When to avoid

• you need high-throughput batched serving rather than batch size 1
• you cannot fine-tune or attach new heads to your model
• you need a turnkey inference server rather than a research framework
• your inference stack has no Medusa integration

Facets

library · maturity active

llm-inference machine-learning gpu-computing large-language-models machine-learning deep-learning python speculative-decoding decoding-heads llm-acceleration tree-attention inference-optimization gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="FasterDecoding/Medusa")