ztxz16/fastllm

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

4945 stars · 485 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

fastllm is a high-performance C++ LLM inference library with its own custom operators, requiring no PyTorch dependency. It supports tensor-parallel inference of dense models and hybrid CPU+GPU inference of MOE models, enabling full DeepSeek 671B inference on a single 10GB+ GPU.

Use cases

• run DeepSeek R1 671B on a single GPU with limited VRAM
• serve an OpenAI-compatible API for local LLMs
• chat with Qwen or Llama models from the terminal
• deploy MOE models with CPU+GPU hybrid tensor parallelism
• run FP8 inference on older or domestic GPUs
• quantize models to INT4 or dynamic quantization for faster inference
• deploy LLM inference on AMD ROCm or Chinese domestic accelerators

When to choose

• you need to run very large MOE models like DeepSeek 671B with limited GPU memory
• you want a PyTorch-free inference engine with broad GPU compatibility including older cards
• you need tensor parallelism across an arbitrary number of GPUs including odd counts
• you target domestic Chinese accelerators or AMD GPUs

When to avoid

• you need the broadest ecosystem integration and community tooling of vLLM or SGLang
• you require macOS or Apple Silicon support
• you need extensive fine-tuning or training features rather than inference only

Facets

library · maturity active

llm-inference gpu-computing cli http-server chat-interface large-language-models artificial-intelligence developer-tools windows python cpp cli moe-inference tensor-parallelism cpu-gpu-hybrid fp8 quantization deepseek openai-compatible-api self-contained-operators no-pytorch-dependency command-line linux gpu web-server android

Member repositories

For agents

markdown · JSON · MCP: product_card(name="ztxz16/fastllm")