pytorch/serve

Health v2 · maintenance only

Flags: archived

Adoption not part of the score

4346 stars · 880 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

TorchServe is a flexible, production-ready model server for serving, optimizing, and scaling PyTorch models over HTTP with support for CPU, GPU, Docker, and Kubernetes. It packages models via torch-model-archiver and exposes prediction APIs with metrics and token-based authorization.

Use cases

• serve pytorch models in production
• deploy a trained pytorch model behind a REST API
• scale model inference on kubernetes
• serve bert model for predictions over http
• run pytorch inference on gpu with docker
• monitor model serving metrics
• package pytorch models into a mar file

When to choose

• you need a battle-tested HTTP inference server specifically for PyTorch models
• you want built-in model versioning, batching, metrics, and Kubernetes integration
• you need CPU and GPU serving with Docker images out of the box

When to avoid

• you need active maintenance and security patches - the project is in limited maintenance mode
• you serve non-PyTorch frameworks like TensorFlow or ONNX exclusively
• you want a lightweight custom inference service you fully control
• you need cutting-edge LLM serving features better covered by vLLM or similar

Facets

service · maturity maintenance

llm-inference machine-learning monitoring container-orchestration machine-learning deep-learning self-hosted python cloud model-serving torchserve inference-server pytorch mlops model-archiver limited-maintenance docker kubernetes linux gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="pytorch/serve")