zml/zml

Health v2 · maintenance only

Adoption not part of the score

4136 stars · 190 forks observed · 2026-10-05

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

ZML is a production LLM inference stack written in Zig, built on MLIR and OpenXLA, that compiles models to run at peak performance across NVIDIA, AMD, Intel, TPU, and AWS Trainium hardware from a single codebase. It is Python-free and built with Bazel, with examples for running models like Llama 3 and Qwen from Hugging Face, local directories, or S3.

Use cases

• serve llm inference on any gpu hardware
• run llama 3 locally without python
• deploy models on tpu or trainium
• compile ai models for amd rocm
• build a fast llm server in zig
• avoid vendor lock-in for ai inference

When to choose

• you need peak inference performance across heterogeneous accelerators
• you want a Python-free, low-overhead inference stack
• you want one codebase targeting CUDA, ROCm, OneAPI, TPU, and Trainium

When to avoid

• you need a quick Python-based inference setup with broad model ecosystem support
• your team is not comfortable with Bazel and Zig toolchains
• you need CPU-only lightweight inference

Facets

framework · maturity active

llm-inference machine-learning compiler gpu-computing large-language-models machine-learning gpu-computing developer-tools cross-platform zig mlir xla bazel inference-stack hardware-agnostic python-free huggingface linux macos gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="zml/zml")