intentee/paddler

Health v2 · maintenance only

Adoption not part of the score

1664 stars · 97 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Paddler is an open-source LLM/VLM load balancer and serving platform built around the llama.cpp (ggml) ecosystem, packaged as a single Rust binary with a balancer and agent components. It supports CPU and GPU inference, dynamic agent scaling, model swapping, request buffering, a web admin panel, and observability metrics for self-hosting LLMs at scale.

Use cases

• self-host llms on my own servers
• load balance llama.cpp inference across multiple hosts
• scale llm inference from zero with autoscaling
• run open source models for privacy-sensitive data
• reduce llm costs compared to per-token api pricing
• serve embeddings and chat completions with openai-compatible api
• monitor and manage llm deployments via web panel

When to choose

• you want self-hosted LLM inference with simple single-binary deployment
• you need LLM-aware load balancing and scale-from-zero with request buffering
• privacy, compliance, or predictable cost requires keeping inference on your own infrastructure
• you want built-in llama.cpp engine with CPU, CUDA, Metal, or Vulkan support

When to avoid

• you need a managed cloud LLM service without infrastructure to operate
• you require a complex multi-model orchestration platform with many moving parts
• you need inference engines beyond the ggml/llama.cpp ecosystem
• you need a fully mature enterprise product with long-term support

Facets

service · maturity active

llm-inference load-testing monitoring http-server api-framework self-hosted large-language-models self-hosted artificial-intelligence infrastructure-as-code windows cli self-hosted llamacpp load-balancer llmops openai-compatible embeddings autoscaling rust devops linux macos docker gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="intentee/paddler")