# bentoml/OpenLLM

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

Repository: https://github.com/bentoml/OpenLLM
Canonical: https://ross.abutalabs.com/products/openllm
Homepage: https://bentoml.com
Language: Python
License: Apache-2.0
License Family: permissive
Topics: llm, llmops, model-inference, fine-tuning, llm-serving, llama, vicuna, bentoml, llama2, llm-inference, llm-ops, open-source-llm, openllm, mistral, mlops, llama3-1, llama3-2, llama3-2-vision
Last push: 2026-08-24T17:01:25+00:00

## Health v2 (maintenance only)
Score: 76/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 99, release rhythm 40, longevity 88
- inputs: {"age_days": 1233, "days_push": 9, "days_rel": 499, "gap_med": 7, "n_releases_24m": 20}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 12518, forks 835 (observed 2026-08-28T04:10:59.270049+00:00)

## What it is
OpenLLM is a Python framework and CLI for self-hosting open-source large language models (Llama, DeepSeek, Qwen, Mistral, etc.) as OpenAI-compatible API endpoints with a single command. It includes a built-in chat UI and streamlined deployment workflows to Docker, Kubernetes, and BentoCloud.

## Use cases
- serve llama 3 as an openai-compatible api endpoint
- self-host deepseek or qwen in the cloud
- run open-source llms behind an api server
- deploy a fine-tuned llm to kubernetes
- test open-source llm models with a local chat ui
- replace openai api with a self-hosted model server

## When to choose
- you want to serve open-source LLMs as OpenAI-compatible endpoints with minimal setup
- you need cloud or Kubernetes deployment of custom or fine-tuned models
- you want a built-in chat UI for interacting with served models

## When to avoid
- you only need to run models locally on a laptop without serving APIs (Ollama may be simpler)
- you need training or fine-tuning pipelines rather than inference serving
- you require a managed multi-cloud inference platform with observability out of the box (BentoCloud/BentoML platform)

## Facets
- artifact type: framework
- maturity: active
- function: llm-inference, http-server, api-framework, deployment, cli, chat-interface
- domain: large-language-models, machine-learning, self-hosted, cloud-computing, developer-tools, apis
- platform: python, cloud, self-hosted, cli
- tags: openai-compatible-api, llm-serving, vllm, model-deployment, llmops, bentoml, gpu-inference, docker, kubernetes, gpu

## Member repositories
- bentoml/OpenLLM (main) score 76

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:10:59.270049+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T17:13:59.077648+00:00, confidence not recorded.
  - readme: https://github.com/bentoml/OpenLLM (fetched 2026-08-28T04:10:59.270049+00:00, sha dc400fb6e533)
  - homepage: https://bentoml.com (fetched 2026-08-29T08:10:39.838554+00:00, sha c9b0ba7568d6)
  - site_page: https://docs.bentoml.com/ (fetched 2026-08-29T08:10:39.847755+00:00, sha de09c4f72293)
  - registry_pypi: https://pypi.org/pypi/openllm/json (fetched 2026-08-29T08:10:39.851522+00:00, sha 7c763fc79594)
  - site_page: https://www.bentoml.com/blog/neurolabs-faster-time-to-market-and-save-cost-with-bentoml (fetched 2026-08-29T08:10:39.849732+00:00, sha ee2030b43e29)
- Data as of 2026-08-30T08:39:29.467469+00:00.