dphnAI/sonar

Health v2 · maintenance only

Adoption not part of the score

1843 stars · 207 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Sonar is a large-scale LLM inference engine for Hugging Face-compatible language and multimodal models, based on vLLM (formerly the Aphrodite engine). It provides continuous batching, paged KV-cache, quantization, speculative decoding, LoRA serving, and distributed serving behind OpenAI-compatible APIs.

Use cases

• serve llm models with an openai-compatible api
• run high-throughput batch inference on gpus
• deploy quantized models on limited vram
• speed up generation with speculative decoding
• serve lora adapters alongside a base model
• run multimodal image audio video model inference
• scale inference across multiple gpus or nodes

When to choose

• you need production-grade LLM serving with OpenAI-compatible endpoints
• you want vLLM features plus extra quantization formats, sampling methods, and hardware support (ROCm, Intel XPU, TPU, Metal)
• you need multi-node serving, prefix caching, or prefill/decode disaggregation
• you serve LoRA adapters or multimodal models at scale

When to avoid

• you need native Windows support (only WSL 2 is supported)
• you want a lightweight single-user desktop inference tool rather than a server engine
• you need a permissive license - Sonar is AGPL-3.0
• your use case is training or fine-tuning rather than inference

Facets

library · maturity active

llm-inference http-server gpu-computing rag chatbot large-language-models machine-learning artificial-intelligence developer-tools python cpp cli vllm-fork openai-compatible-api speculative-decoding quantization continuous-batching paged-kv-cache lora multimodal aphrodite-engine distributed-serving linux macos gpu docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="dphnAI/sonar")