kvcache-ai/ktransformers

Health v2 · maintenance only

Adoption not part of the score

19303 stars · 1539 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

KTransformers is a Python/C++ framework for CPU-GPU heterogeneous inference and fine-tuning of large language models, with a kt-kernel backend optimized for MoE models on consumer hardware. It supports day-0 releases for new models, FP8/INT8 quantization, LoRA and full-parameter SFT, and long-context multimodal inference.

Use cases

• run large MoE LLMs on consumer GPUs with limited VRAM
• offload LLM inference to CPU-GPU heterogeneous setups
• fine-tune MoE models with LoRA on x86 servers
• run DeepSeek or GLM models locally with long context
• do full-parameter BF16 fine-tuning of MoE models
• serve LLMs with FP8 expert weights without extra memory

When to choose

• you want to run huge MoE models on limited consumer hardware
• you need day-0 support for newly released models like DeepSeek or GLM
• you want CPU-GPU hybrid inference or fine-tuning with AMX/AVX512 kernels

When to avoid

• you need a production-grade multi-node serving stack
• you only run small dense models that fit fully in GPU memory
• you need non-x86 CPU backends or broad platform support

Facets

framework · maturity active

llm-inference llm-training gpu-computing machine-learning large-language-models deep-learning machine-learning gpu-computing python cpp cpu-gpu-heterogeneous moe quantization lora fine-tuning consumer-hardware day0-model-support linux gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="kvcache-ai/ktransformers")