mlc-ai/mlc-llm

Health v2 · maintenance only

Adoption not part of the score

23095 stars · 2124 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

MLC LLM is a machine learning compiler and high-performance deployment engine for large language models, built on TVM-based ML compilation. It provides a unified OpenAI-compatible inference engine (MLCEngine) that runs LLMs natively across GPUs, CPUs, browsers, and mobile platforms.

Use cases

• run llama 3 locally on my gpu
• deploy an llm on android or ios
• serve an openai-compatible api for a local model
• run a chat model in the browser with webgpu
• quantize and compile an llm for apple silicon
• host a local llm rest server
• run llm inference on amd or intel gpus

When to choose

• you need to deploy LLMs across diverse hardware (NVIDIA, AMD, Intel, Apple GPUs, mobile, browser)
• you want an OpenAI-compatible API backed by a local engine
• you need int4-quantized models running on consumer devices
• you want a single engine for Python, JavaScript, iOS, Android, and REST serving

When to avoid

• you only need a simple CPU-only inference server with minimal setup
• you want the broadest ecosystem of prebuilt community models rather than MLC-compiled weights
• you need training or fine-tuning rather than inference deployment

Facets

framework · maturity active

llm-inference machine-learning compiler http-server sdk large-language-models machine-learning gpu-computing cross-platform developer-tools windows wasm python cli ml-compilation tvm openai-compatible-api vulkan metal webgpu local-llm quantization linux macos android ios web gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="mlc-ai/mlc-llm")