punica-ai/punica

Health v2 · maintenance only

Adoption not part of the score

1175 stars · 64 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Punica is a Python system for serving many LoRA-finetuned LLMs from a single copy of the base model on one GPU, using a custom CUDA kernel (SGMV) to batch LoRA adapter computations. It achieves up to 12x higher throughput than state-of-the-art LLM serving systems for multi-tenant LoRA workloads.

Use cases

• serve multiple LoRA finetuned LLMs on one GPU
• batch requests for different LoRA adapters efficiently
• reduce GPU memory when hosting many finetuned models
• multi-tenant LLM serving with per-user adapters
• benchmark LoRA serving throughput against vLLM or DeepSpeed

When to choose

• you need to serve many LoRA variants of the same base model cost-effectively
• GPU memory is a bottleneck from loading many finetuned model copies
• your workload mixes requests across different LoRA adapters

When to avoid

• you serve only a single model with no LoRA adapters
• you need a production system with active development and broad model support
• you have no CUDA-capable GPU

Facets

library · maturity maintenance

llm-inference gpu-computing machine-learning large-language-models machine-learning gpu-computing python lora multi-tenant-serving cuda-kernels sgmv llm-serving gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="punica-ai/punica")