xLLM-AI/xllm

Health v2 · maintenance only

Adoption not part of the score

1536 stars · 285 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

xLLM is a high-performance C++ inference engine for LLM, VLM, DiT and recommendation models, optimized for heterogeneous AI accelerators such as Ascend NPUs, Cambricon MLUs, Iluvatar CoreX and Moore Threads GPUs. It provides a unified service-engine stack with asynchronous scheduling, graph optimization, and global KV cache management, and is hosted under the OpenAtom Foundation.

Use cases

• serve deepseek or qwen models on ascend npu hardware
• run high-throughput llm inference on domestic chinese accelerators
• deploy vision-language models like glm-4.6v for inference
• reduce inference cost with efficient kv cache management
• serve moe models like deepseek-v4 with day-0 support
• self-host an openai-compatible llm serving endpoint

When to choose

• you need to run large language models on Chinese domestic accelerators like Ascend or Cambricon
• you want day-0 support for new open models such as DeepSeek, Qwen, or GLM releases
• you need high-throughput, low-latency distributed inference with MoE optimizations
• you want a unified engine for LLM, VLM, and DiT model serving

When to avoid

• your hardware is limited to NVIDIA GPUs where vLLM or SGLang have broader ecosystem support
• you need a pure training framework rather than an inference engine
• you require extensive community plugins and third-party integrations available in more mature engines

Facets

library · maturity active

llm-inference gpu-computing machine-learning http-server large-language-models deep-learning artificial-intelligence gpu-computing cpp self-hosted inference-engine npu ascend cambricon vlm moe kv-cache serving linux docker gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="xLLM-AI/xllm")