rednote-machine-learning/RedKnot

Health v2 · maintenance only

Flags: no_releases young

Adoption not part of the score

1880 stars · 762 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

RedKnot is a long-context LLM inference acceleration library built on SGLang, using head-classified KV reuse, offline KV storage with RoPE relocation, sparse FFN, and a SegPagedAttention runtime. It reduces prefill FLOPs by 50-70% and speeds up TTFT 1.35x-3.2x with near-lossless accuracy.

Use cases

• serve long-context LLMs with lower latency
• speed up prefill for large prompt inference
• reduce KV cache memory usage in LLM serving
• run Qwen3 or Llama models with sparse attention efficiently
• accelerate TTFT for RAG workloads with reusable prompt segments
• deploy LLM inference on limited GPU memory

When to choose

• you serve long-context models and prefill latency is a bottleneck
• you already use SGLang and want drop-in attention acceleration
• your workloads have reusable prompt prefixes like RAG or system prompts
• you want near-lossless quality with large FLOPs savings

When to avoid

• you need short-context inference where the overhead outweighs gains
• you require a fully stable release for models like DeepSeek-V4 or Qwen3.5 that are still being adapted
• you need Ascend NPU support that is still work in progress
• you need a simple inference stack without SGLang's complexity

Facets

library · maturity active

llm-inference machine-learning gpu-computing large-language-models machine-learning gpu-computing python kv-cache-reuse long-context attention-optimization sglang sparse-attention inference-acceleration gpu linux docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="rednote-machine-learning/RedKnot")