EricLBuehler/mistral.rs

Health v2 · maintenance only

Adoption not part of the score

7628 stars · 685 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

mistral.rs is a fast and flexible LLM inference engine written in Rust, supporting many model families with quantization (ISQ/UQFF, GGUF), paged attention, and multimodal inputs. It exposes OpenAI- and Anthropic-compatible HTTP APIs and provides Rust and Python SDKs for local or server-based inference.

Use cases

• serve local llms with an openai-compatible api
• run quantized gguf models on gpu
• self-host a chat completion server
• run multimodal llm inference with images and audio
• build agents with local llm tool calling
• benchmark llm inference performance in rust
• expose anthropic messages api from a local model

When to choose

• you need high-performance local LLM serving in Rust with OpenAI-compatible endpoints
• you want quantized inference (GGUF, ISQ, UQFF) with broad model support
• you need multimodal inputs, tool calling, or agentic features on self-hosted models

When to avoid

• you only need a simple Python-only inference stack without a serving layer
• you require the largest ecosystem of community integrations available in mainstream runtimes
• you need CPU-only inference on very constrained hardware

Facets

service · maturity active

llm-inference http-server api-framework rag agent-framework gpu-computing large-language-models artificial-intelligence developer-tools self-hosted rust python windows cli openai-compatible-api gguf quantization multimodal paged-attention anthropic-api serving linux macos cuda web-server

Member repositories

For agents

markdown · JSON · MCP: product_card(name="EricLBuehler/mistral.rs")