xusenlinzy/api-for-open-llm

Health v2 · maintenance only

Adoption not part of the score

2460 stars · 271 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A Python server that exposes open-source large language models (LLaMA, ChatGLM, Qwen, Baichuan, Falcon, etc.) behind an OpenAI-compatible API. It supports streaming responses, embeddings, reranking, LoRA adapters, and vLLM-accelerated inference so open models can drop in as ChatGPT replacements.

Use cases

• serve open-source llm with openai compatible api
• self-host chatgpt alternative backend
• use langchain with local open llm
• run chatglm or qwen behind openai api
• deploy llm server with streaming responses
• expose embeddings endpoint for rag knowledge base

When to choose

• you want existing OpenAI SDK clients to work with open models
• you need LangChain compatibility and embeddings for RAG
• you want one unified backend for many open LLM families
• you want vLLM-accelerated serving with concurrency

When to avoid

• you only need OpenAI's hosted models
• you need a production-grade inference server with broad model coverage like vLLM or TGI directly
• you need non-OpenAI API styles or heavy enterprise features

Facets

service · maturity active

llm-inference api-framework http-server rag large-language-models artificial-intelligence self-hosted apis python self-hosted openai-compatible-api chatglm llama qwen vllm langchain streaming embeddings docker gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="xusenlinzy/api-for-open-llm")