# kvcache-ai/ktransformers

A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations

Repository: https://github.com/kvcache-ai/ktransformers
Canonical: https://ross.abutalabs.com/products/ktransformers
Homepage: https://kvcache-ai.github.io/ktransformers/
Language: Python
License: Apache-2.0
License Family: permissive
Last push: 2026-08-27T00:14:46+00:00

## Health v2 (maintenance only)
Score: 90/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 99, release rhythm 98, longevity 54
- inputs: {"age_days": 768, "days_push": 7, "days_rel": 16, "gap_med": 16, "n_releases_24m": 26}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 19303, forks 1539 (observed 2026-08-28T04:11:27.577034+00:00)

## What it is
KTransformers is a Python/C++ framework for CPU-GPU heterogeneous inference and fine-tuning of large language models, with a kt-kernel backend optimized for MoE models on consumer hardware. It supports day-0 releases for new models, FP8/INT8 quantization, LoRA and full-parameter SFT, and long-context multimodal inference.

## Use cases
- run large MoE LLMs on consumer GPUs with limited VRAM
- offload LLM inference to CPU-GPU heterogeneous setups
- fine-tune MoE models with LoRA on x86 servers
- run DeepSeek or GLM models locally with long context
- do full-parameter BF16 fine-tuning of MoE models
- serve LLMs with FP8 expert weights without extra memory

## When to choose
- you want to run huge MoE models on limited consumer hardware
- you need day-0 support for newly released models like DeepSeek or GLM
- you want CPU-GPU hybrid inference or fine-tuning with AMX/AVX512 kernels

## When to avoid
- you need a production-grade multi-node serving stack
- you only run small dense models that fit fully in GPU memory
- you need non-x86 CPU backends or broad platform support

## Facets
- artifact type: framework
- maturity: active
- function: llm-inference, llm-training, gpu-computing, machine-learning
- domain: large-language-models, deep-learning, machine-learning, gpu-computing
- platform: python, cpp
- tags: cpu-gpu-heterogeneous, moe, quantization, lora, fine-tuning, consumer-hardware, day0-model-support, linux, gpu

## Member repositories
- kvcache-ai/ktransformers (main) score 90

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:11:27.577034+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T17:00:20.607639+00:00, confidence not recorded.
  - readme: https://github.com/kvcache-ai/ktransformers (fetched 2026-08-28T04:11:27.577034+00:00, sha f4ce05f98b42)
  - homepage: https://kvcache-ai.github.io/ktransformers/ (fetched 2026-08-29T07:58:36.158266+00:00, sha f1911fde5860)
  - registry_pypi: https://pypi.org/pypi/ktransformers/json (fetched 2026-08-29T07:58:36.167649+00:00, sha 94f370530051)
- Data as of 2026-08-30T08:39:29.467469+00:00.