# jundot/omlx

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

Repository: https://github.com/jundot/omlx
Canonical: https://ross.abutalabs.com/products/omlx
Homepage: https://omlx.ai
Language: Python
License: Apache-2.0
License Family: permissive
Topics: apple-silicon, inference-server, llm, macos, mlx, openai-api
Last push: 2026-08-26T04:12:56+00:00

## Health v2 (maintenance only)
Score: 78/100 (v2, computed 2026-09-03T02:39:23.370411+00:00)
- activity 99, release rhythm 87, longevity 14
- inputs: {"age_days": 201, "days_push": 7, "days_rel": 9, "gap_med": 0.0, "n_releases_24m": 97}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 20760, forks 1746 (observed 2026-08-28T04:11:30.399816+00:00)

## What it is
oMLX is a native macOS menu bar application and LLM inference server built on Apple's MLX framework, featuring continuous batching and two-tier (RAM + SSD) KV cache persistence. It exposes OpenAI- and Anthropic-compatible APIs so local models work as drop-in backends for tools like Claude Code and Cursor.

## Use cases
- run local llm server on apple silicon mac
- serve mlx models with openai-compatible api
- speed up claude code with local llm backend
- persist kv cache across llm requests
- manage llm inference server from macos menu bar
- host multiple llm models with automatic memory eviction
- self-host anthropic api compatible endpoint

## When to choose
- you have an Apple Silicon Mac and want fast local LLM serving with minimal setup
- you use coding agents like Claude Code or Cursor and want low time-to-first-token on repeated prefixes
- you want a native menu bar app with dashboard, auto-updates, and multi-model serving
- you need OpenAI and Anthropic drop-in API compatibility with tool calling and MCP support

## When to avoid
- you need inference on Linux, Windows, or non-Apple-Silicon hardware
- you require CUDA or multi-GPU cluster serving at scale
- you need model architectures not supported by MLX
- you want a headless server without macOS

## Facets
- artifact type: application
- maturity: active
- function: llm-inference, http-server, caching, api-framework, chat-interface, mcp
- domain: large-language-models, artificial-intelligence, developer-tools, self-hosted, machine-learning
- platform: python, cli
- tags: mlx, apple-silicon, kv-cache, continuous-batching, menu-bar-app, openai-compatible, anthropic-api, local-llm, ssd-caching, homebrew, macos, web-server

## Member repositories
- jundot/omlx (main) score 78

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:11:30.399816+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T16:58:33.061574+00:00, confidence not recorded.
  - readme: https://github.com/jundot/omlx (fetched 2026-08-28T04:11:30.399816+00:00, sha bfb48e1c64bf)
  - homepage: https://omlx.ai (fetched 2026-08-29T07:57:09.896527+00:00, sha 35977f038451)
- Data as of 2026-08-30T08:39:29.467469+00:00.