mit-han-lab/streaming-llm

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

7268 stars · 400 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

StreamingLLM is a research framework from MIT Han Lab implementing the Attention Sinks method (ICLR 2024) for efficient streaming language model inference. It enables LLMs like Llama-2, MPT, Falcon, and Pythia to handle effectively infinite input lengths without fine-tuning by preserving the KV cache of initial tokens.

Use cases

• run llm inference on very long or infinite-length inputs
• stream multi-round dialogue without exceeding context window
• reduce kv cache memory usage during llm decoding
• enable window attention to work beyond cache size
• deploy chatbots that can converse indefinitely without crashing

When to choose

• you need stable LLM generation over millions of tokens without fine-tuning
• you want memory-efficient streaming inference with a sliding window plus attention sinks
• you use supported models like Llama-2, MPT, Falcon, or Pythia with HuggingFace Transformers

When to avoid

• you need true long-range recall of arbitrary old tokens rather than recent-window coherence
• your model or serving stack is not supported by the attention-sink approach
• you need a production inference server rather than a research codebase

Facets

library · maturity stable

llm-inference machine-learning deep-learning large-language-models deep-learning python attention-sinks kv-cache streaming-inference long-context research-paper transformers natural-language-processing gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="mit-han-lab/streaming-llm")