kvcache-ai/ktransformers

Health v2 · maintenance only

Adoption not part of the score

19618 stars · 1604 forks observed · 2026-10-20

What it is AI-extracted, prompt v1, taxonomy v1, 2026-10-19, confidence 0.98

KTransformers is a Python library that provides a high‑performance framework for heterogeneous CPU‑GPU inference and fine‑tuning of large language models. It supports advanced quantization (FP8, INT8), LoRA, MoE and KV‑cache offloading across diverse hardware.

Use cases

• run LLM inference on consumer GPUs with CPU offload
• fine‑tune LLMs using LoRA or full‑parameter training on mixed CPU‑GPU systems
• deploy KV‑cache accelerated models for long‑context generation
• experiment with FP8/INT8 quantized models for faster inference
• train MoE models with heterogeneous hardware

When to choose

• you need state‑of‑the‑art inference speed on consumer‑grade GPUs
• your workload benefits from CPU‑GPU offloading or KV‑cache acceleration
• you require support for advanced quantization formats like FP8 or INT8
• you are fine‑tuning LLMs with LoRA or full‑parameter training on mixed hardware

When to avoid

• you only run on pure CPU environments without GPU support
• you need a fully managed cloud service rather than a library
• your project requires a different programming language ecosystem

Facets

library · maturity active

llm-inference llm-training large-language-models artificial-intelligence python windows heterogeneous-computing cpu-gpu kv-cache fp8 int8 lora moe linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="kvcache-ai/ktransformers")