Tiiny-AI/PowerInfer

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

9752 stars · 595 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

PowerInfer is a CPU/GPU hybrid LLM inference engine that exploits activation locality to run large language models fast on consumer-grade hardware, including desktops and smartphones. It supports sparse models like TurboSparse and ProSparse and is MIT-licensed C++ software.

Use cases

• run an LLM locally on a consumer GPU
• serve large language models on a single desktop machine
• speed up inference of sparse ReLU-activated LLMs
• run LLM inference on a smartphone
• deploy Llama-style models offline on my own hardware

When to choose

• you want fast local LLM inference on consumer-grade CPU/GPU hardware
• you are using sparsified models like TurboSparse or ProSparse
• you need on-device inference on smartphones or edge devices

When to avoid

• you need multi-node distributed serving at datacenter scale
• you want a managed cloud inference API
• your models are not compatible with its supported architectures

Facets

library · maturity active

llm-inference gpu-computing machine-learning large-language-models machine-learning gpu-computing windows cpp cpu-gpu-hybrid activation-locality sparse-models local-deployment consumer-gpu on-device-inference linux macos gpu android

Member repositories

For agents

markdown · JSON · MCP: product_card(name="Tiiny-AI/PowerInfer")