OpenPPL/ppl.nn

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1367 stars · 219 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

PPLNN is a high-performance deep-learning inference engine written in C++ that runs ONNX models on x86 CPUs and NVIDIA GPUs, with a dedicated LLM engine supporting models like LLaMA, ChatGLM, and Qwen. It provides Python APIs and optimizations such as Flash Attention, continuous batching, and tensor parallelism.

Use cases

• run onnx models with a fast inference engine
• serve llama or qwen on nvidia gpus
• deploy openmmlab models in production
• quantize llm kv cache to int8
• run deep learning inference on x86 cpu

When to choose

• you need high-performance ONNX inference on x86 or CUDA
• you want to serve open-source LLMs with continuous batching and tensor parallelism
• you deploy OpenMMLab models and want optimized support

When to avoid

• you need training or fine-tuning rather than inference
• you need broad hardware support beyond x86 and NVIDIA GPUs
• you prefer a mature ecosystem like TensorRT or ONNX Runtime with extensive community support

Facets

library · maturity active

llm-inference machine-learning deep-learning gpu-computing deep-learning large-language-models machine-learning artificial-intelligence cpp python cross-platform onnx inference-engine llm-serving openmmlab cuda x86 tensor-parallelism flash-attention kv-cache-quantization linux gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="OpenPPL/ppl.nn")