gpustack/gpustack

Health v2 · maintenance only

Adoption not part of the score

5560 stars · 628 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

GPUStack is an open-source GPU cluster manager for AI model serving that orchestrates inference engines like vLLM, SGLang, and TensorRT-LLM across heterogeneous GPU hardware (NVIDIA, AMD, Ascend, and more) on-premises, in Kubernetes, or in the cloud. It also provisions on-demand SSH-accessible GPU instances and provides enterprise features such as monitoring, authentication, load balancing, and token metering for delivering Model-as-a-Service.

Use cases

• manage a gpu cluster for llm inference
• serve openai-compatible llm apis on my own gpus
• deploy vllm or sglang across multiple gpu servers
• run large models with distributed inference on heterogeneous gpus
• provision ssh-accessible gpu instances for fine-tuning
• build a model-as-a-service platform for my team
• schedule llm workloads across nvidia and amd gpus
• monitor gpu utilization and token usage for model serving

When to choose

• you need to serve LLMs across a fleet of heterogeneous GPUs (NVIDIA, AMD, Ascend, etc.) with automatic engine selection
• you want an OpenAI-compatible API gateway with auth, rate limiting, and metering out of the box
• you need multi-cluster GPU management spanning on-premises servers, Kubernetes, and cloud providers
• you want day-0 support for newly released models via pluggable inference engines
• you need SSH-accessible GPU instances for development and fine-tuning workloads

When to avoid

• you only need to run a single model on one local machine - simpler tools like Ollama or llama.cpp suffice
• your infrastructure is already standardized on Kubernetes and you prefer a native Kubernetes operator or KServe
• you need training or fine-tuning orchestration as the primary workload rather than inference serving
• you require Windows or macOS worker nodes - GPU workers are Linux-only

Facets

service · maturity active

llm-inference gpu-computing container-orchestration monitoring auth rate-limiting scheduling ssh api-gateway self-hosted large-language-models gpu-computing infrastructure-as-code cloud-computing self-hosted artificial-intelligence microservices python self-hosted cloud gpu-cluster-manager model-as-a-service vllm sglang tensorrt-llm heterogeneous-gpu distributed-inference gpu-instances openai-compatible-api nvidia amd ascend rocm cuda speculative-decoding kv-cache multi-cluster metering rbac devops linux docker kubernetes

Member repositories

For agents

markdown · JSON · MCP: product_card(name="gpustack/gpustack")