kubeai-project/kubeai

Health v2 · maintenance only

Flags: no_readme

Adoption not part of the score

1256 stars · 135 forks observed · 2026-09-03

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

KubeAI is a Kubernetes operator for serving machine learning models in production, supporting LLMs via vLLM and Ollama, vector embeddings, reranking, and speech-to-text with FasterWhisper. It exposes an OpenAI-compatible API and provides scale-from-zero autoscaling, prefix-aware load balancing, model caching, and LoRA adapter orchestration without requiring Istio, Knative, or other dependencies.

Use cases

• serve llm models on kubernetes
• self-host openai-compatible inference api
• autoscale vllm replicas from zero
• run whisper speech-to-text in a cluster
• deploy embedding models on k8s
• manage lora adapters across model replicas
• load balance llm traffic with kv-cache awareness

When to choose

• you run ML model inference on Kubernetes and want a zero-dependency operator
• you need OpenAI-compatible endpoints backed by self-hosted models
• you want scale-from-zero autoscaling without Istio or Knative
• you serve multiple model types (LLMs, embeddings, speech) from one system
• you need better load balancing than kube-proxy for stateful vLLM replicas

When to avoid

• you don't use Kubernetes - use vLLM or Ollama directly
• you need fully managed inference - use a cloud LLM API instead
• you only run a single model on one machine
• you need training or fine-tuning - KubeAI is inference-only

Facets

service · maturity active

llm-inference speech-recognition machine-learning deployment container-orchestration api-gateway rag machine-learning large-language-models self-hosted cloud-computing speech-processing cloud self-hosted go kubernetes-operator vllm ollama openai-compatible-api autoscaling scale-from-zero model-caching lora-adapters embeddings reranking faster-whisper gpu-scheduling helm-chart prefix-aware-load-balancing autoscaling-not-in-vocab-use-tags kubernetes devops docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="kubeai-project/kubeai")