jundot/omlx

Health v2 · maintenance only

Adoption not part of the score

22914 stars · 2153 forks observed · 2026-10-19

What it is AI-extracted, prompt v1, taxonomy v1, 2026-10-12, confidence 0.97

oMLX is a macOS‑native LLM inference server that runs on Apple Silicon and provides continuous batching and tiered KV caching. It can be managed via a menu‑bar app, a CLI shim, or an OpenAI‑compatible API.

Use cases

• run local LLMs on a Mac for fast code generation
• serve multiple LLM models concurrently with low latency
• persist KV cache across requests to speed up tool‑calling agents
• integrate local LLMs with OpenAI‑compatible clients like Claude Code
• use a desktop UI to monitor and control model inference

When to choose

• You need high‑performance LLM inference on Apple Silicon without cloud latency.
• You want automatic caching of context to SSD for repeated prompts.
• You prefer a native macOS UI and CLI for managing the inference server.

When to avoid

• You are running on non‑Apple‑Silicon hardware.
• You require a cross‑platform or containerized deployment.
• You need GPU support beyond Apple’s M‑series.

Facets

application · maturity active

api-framework caching cli http-server artificial-intelligence large-language-models machine-learning python cli apple-silicon continuous-batching ssd-kv-cache menu-bar openai-api mlx macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="jundot/omlx")