xorbitsai/inference

Health v2 · maintenance only

Adoption not part of the score

9523 stars · 863 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Xinference is an open-source model serving platform for deploying LLMs, embedding, speech, image, and multimodal models via a unified OpenAI-compatible API. It runs on cloud, on-premises, or a laptop, with backends like vLLM, llama.cpp, SGLang, and Transformers.

Use cases

• self-host an open-source LLM with an OpenAI-compatible API
• serve embedding models for a RAG pipeline
• run Whisper speech-to-text on my own server
• deploy a multimodal vision-language model behind one API
• swap GPT for a local model by changing one line of code
• host text-to-image generation on-premises
• run LLM inference on a laptop without cloud

When to choose

• you need one unified API for LLMs, embeddings, audio, and image models
• you want OpenAI API compatibility with self-hosted models
• you need production-ready serving across cloud and on-prem with multiple inference backends

When to avoid

• you only need a lightweight single-model local runner like Ollama
• you want to fine-tune or train models rather than serve them
• you need maximum single-backend performance tuning rather than flexibility

Facets

service · maturity active

llm-inference speech-recognition rag api-framework deployment sdk large-language-models artificial-intelligence machine-learning speech-processing self-hosted apis python self-hosted cloud cross-platform openai-api-compatible model-serving vllm llama-cpp sglang multimodal embeddings text-to-image whisper on-premises docker gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="xorbitsai/inference")