kserve/kserve

Health v2 · maintenance only

Adoption not part of the score

5834 stars · 1636 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

KServe is a CNCF incubating, Kubernetes-native platform for serving both generative and predictive AI models at scale. It provides a standardized CRD-based API that encapsulates autoscaling, networking, health checking, and multi-framework model serving (TensorFlow, PyTorch, scikit-learn, XGBoost, ONNX, vLLM, Hugging Face).

Use cases

• serve llms on kubernetes with vllm
• deploy a pytorch model as a rest api on k8s
• openai-compatible inference endpoint for self-hosted models
• scale model serving to zero when idle
• canary rollout of a new model version
• add explainability and drift detection to model predictions
• serve sklearn and xgboost models behind one platform

When to choose

• you already run Kubernetes and need standardized multi-framework model serving
• you need both LLM (generative) and traditional (predictive) inference on one platform
• you want autoscaling, scale-to-zero, canary rollouts, and traffic management out of the box
• you need GPU-accelerated LLM serving with OpenAI-compatible APIs

When to avoid

• you deploy models outside Kubernetes or on a single server
• you need only a lightweight standalone inference server without cluster orchestration
• your team cannot operate Kubernetes operators, Istio, and Knative dependencies

Facets

service · maturity stable

machine-learning llm-inference api-framework monitoring deployment container-orchestration machine-learning large-language-models artificial-intelligence cloud-computing go cloud self-hosted model-serving inference kubernetes-operator vllm knative istio kubeflow cncf autoscaling gpu openai-compatible canary-rollouts model-explainability devops mlops kubernetes docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="kserve/kserve")