predibase/lorax

Health v2 · maintenance only

Adoption not part of the score

3826 stars · 326 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

LoRAX is a multi-LoRA inference server that serves thousands of fine-tuned LLM adapters on a single GPU by sharing a common base model. It provides an OpenAI-compatible REST API with dynamic adapter loading, heterogeneous continuous batching, and production features like Prometheus metrics and Kubernetes Helm charts.

Use cases

• serve thousands of fine-tuned lora models on one gpu
• host a multi-tenant llm inference server with per-request adapters
• reduce cost of serving many fine-tuned llms
• run an openai-compatible api for custom fine-tuned models
• dynamically load huggingface lora adapters per request
• deploy llm serving on kubernetes with helm
• merge multiple lora adapters into an ensemble at inference time
• serve structured output from fine-tuned llms

When to choose

• you need to serve many fine-tuned LoRA adapters of the same base model cost-effectively
• you want an OpenAI-compatible API with per-request adapter selection
• you need production features like metrics, tracing, and Kubernetes deployment
• you want to merge adapters per request to build ensembles

When to avoid

• you only serve one or a few full fine-tuned models rather than LoRA adapters
• you need non-LoRA fine-tuning methods like full-weight tuning served together
• you need a simple local playground rather than a production serving server
• your base models differ per request, defeating shared-base-model efficiency

Facets

service · maturity active

llm-inference http-server api-framework monitoring tracing large-language-models machine-learning apis self-hosted gpu-computing python self-hosted lora model-serving openai-compatible-api fine-tuned-models adapter-hot-swapping continuous-batching quantization helm-charts docker kubernetes gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="predibase/lorax")