microsoft/LLMLingua

Health v2 · maintenance only

Adoption not part of the score

6609 stars · 418 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

LLMLingua is a Microsoft library for prompt compression that removes non-essential tokens from prompts and KV caches to speed up LLM inference, achieving up to 20x compression with minimal performance loss. It includes variants like LongLLMLingua for query-aware long-context compression and LLMLingua-2 for faster task-agnostic compression, and integrates with LangChain, LlamaIndex, and Prompt flow.

Use cases

• compress prompts before sending to an LLM to cut token costs
• speed up LLM inference by shrinking long context windows
• reduce RAG context size without losing answer quality
• compress long documents for query-aware question answering
• lower KV cache memory usage for long-context models
• make prompts fit within a smaller context limit

When to choose

• you pay per token and want to cut prompt costs
• your RAG pipeline stuffs too much retrieved context into prompts
• long-context inference latency is a bottleneck
• you use LangChain or LlamaIndex and want drop-in compression

When to avoid

• your prompts are already short and cheap
• your task is highly sensitive to dropped tokens, such as precise code generation
• you need compression without any small auxiliary model or extra compute
• you work outside the Python ecosystem

Facets

library · maturity active

llm-inference rag nlp machine-learning large-language-models artificial-intelligence python prompt-compression kv-cache inference-acceleration token-reduction cost-optimization natural-language-processing retrieval-augmented-generation

Member repositories

For agents

markdown · JSON · MCP: product_card(name="microsoft/LLMLingua")