bentoml/OpenLLM

Health v2 · maintenance only

Adoption not part of the score

12518 stars · 835 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

OpenLLM is a Python framework and CLI for self-hosting open-source large language models (Llama, DeepSeek, Qwen, Mistral, etc.) as OpenAI-compatible API endpoints with a single command. It includes a built-in chat UI and streamlined deployment workflows to Docker, Kubernetes, and BentoCloud.

Use cases

• serve llama 3 as an openai-compatible api endpoint
• self-host deepseek or qwen in the cloud
• run open-source llms behind an api server
• deploy a fine-tuned llm to kubernetes
• test open-source llm models with a local chat ui
• replace openai api with a self-hosted model server

When to choose

• you want to serve open-source LLMs as OpenAI-compatible endpoints with minimal setup
• you need cloud or Kubernetes deployment of custom or fine-tuned models
• you want a built-in chat UI for interacting with served models

When to avoid

• you only need to run models locally on a laptop without serving APIs (Ollama may be simpler)
• you need training or fine-tuning pipelines rather than inference serving
• you require a managed multi-cloud inference platform with observability out of the box (BentoCloud/BentoML platform)

Facets

framework · maturity active

llm-inference http-server api-framework deployment cli chat-interface large-language-models machine-learning self-hosted cloud-computing developer-tools apis python cloud self-hosted cli openai-compatible-api llm-serving vllm model-deployment llmops bentoml gpu-inference docker kubernetes gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="bentoml/OpenLLM")