hao-ai-lab/LookaheadDecoding

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1342 stars · 83 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A Python library implementing Lookahead Decoding, an exact parallel decoding algorithm that accelerates LLM inference without a draft model or data store. It uses Jacobi iteration with n-gram caching and verification branches to reduce decoding steps, integrating with FlashAttention and supporting multi-GPU scaling.

Use cases

• speed up autoregressive LLM text generation
• accelerate code completion with LLMs
• reduce LLM decoding latency without a draft model
• parallelize LLM inference across multiple GPUs
• integrate faster decoding into my own sampling code
• speed up LLaMA-2 chat generation

When to choose

• you want speculative-decoding-like speedups without training or obtaining a draft model
• you need exact decoding with no quality loss
• you run inference on modern GPUs with FlashAttention support
• you want to reduce decoding steps in code completion workloads

When to avoid

• you need a production inference server with broad model support out of the box
• your workload is memory-abundant and batch-heavy where parallel decoding gains are minimal
• you require long-term maintenance guarantees or wide ecosystem integration

Facets

library · maturity active

llm-inference machine-learning gpu-computing large-language-models machine-learning deep-learning gpu-computing python lookahead-decoding jacobi-decoding speculative-decoding inference-acceleration flashattention icml-2024 gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="hao-ai-lab/LookaheadDecoding")