# intentee/paddler

Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.

Repository: https://github.com/intentee/paddler
Canonical: https://ross.abutalabs.com/products/paddler
Homepage: https://paddler.intentee.com
Language: Rust
License: Apache-2.0
License Family: permissive
Topics: ai, llamacpp, llm, llmops, load-balancer
Last push: 2026-07-19T19:36:21+00:00

## Health v2 (maintenance only)
Score: 87/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 93, release rhythm 93, longevity 61
- inputs: {"age_days": 858, "days_push": 45, "days_rel": 45, "gap_med": 26, "n_releases_24m": 18}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1664, forks 97 (observed 2026-08-28T04:05:18.927515+00:00)

## What it is
Paddler is an open-source LLM/VLM load balancer and serving platform built around the llama.cpp (ggml) ecosystem, packaged as a single Rust binary with a balancer and agent components. It supports CPU and GPU inference, dynamic agent scaling, model swapping, request buffering, a web admin panel, and observability metrics for self-hosting LLMs at scale.

## Use cases
- self-host llms on my own servers
- load balance llama.cpp inference across multiple hosts
- scale llm inference from zero with autoscaling
- run open source models for privacy-sensitive data
- reduce llm costs compared to per-token api pricing
- serve embeddings and chat completions with openai-compatible api
- monitor and manage llm deployments via web panel

## When to choose
- you want self-hosted LLM inference with simple single-binary deployment
- you need LLM-aware load balancing and scale-from-zero with request buffering
- privacy, compliance, or predictable cost requires keeping inference on your own infrastructure
- you want built-in llama.cpp engine with CPU, CUDA, Metal, or Vulkan support

## When to avoid
- you need a managed cloud LLM service without infrastructure to operate
- you require a complex multi-model orchestration platform with many moving parts
- you need inference engines beyond the ggml/llama.cpp ecosystem
- you need a fully mature enterprise product with long-term support

## Facets
- artifact type: service
- maturity: active
- function: llm-inference, load-testing, monitoring, http-server, api-framework, self-hosted
- domain: large-language-models, self-hosted, artificial-intelligence, infrastructure-as-code
- platform: windows, cli, self-hosted
- tags: llamacpp, load-balancer, llmops, openai-compatible, embeddings, autoscaling, rust, devops, linux, macos, docker, gpu

## Member repositories
- intentee/paddler (main) score 87

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:05:18.927515+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T03:44:29.066859+00:00, confidence not recorded.
  - readme: https://github.com/intentee/paddler (fetched 2026-08-28T04:05:18.927515+00:00, sha 063d9497b3ce)
  - homepage: https://paddler.intentee.com (fetched 2026-08-29T11:16:40.151751+00:00, sha 120f53db4f85)
  - site_page: https://paddler.intentee.com/docs/introduction/what-is-paddler (fetched 2026-08-29T11:16:40.160216+00:00, sha f804bc3c5262)
  - site_page: https://paddler.intentee.com/docs/introduction/installation (fetched 2026-08-29T11:16:40.162257+00:00, sha f78060e82d26)
  - registry_crates: https://crates.io/api/v1/crates/paddler (fetched 2026-08-29T11:16:40.164071+00:00, sha 72a4507672e7)
- Data as of 2026-08-30T08:39:29.467469+00:00.