tensorzero/tensorzero

Health v2 · maintenance only

Flags: archived

Adoption not part of the score

11719 stars · 962 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

TensorZero is an open-source LLMOps platform that unifies an LLM gateway, observability, evaluation, optimization, and experimentation behind a single high-performance API. It is written in Rust, works with the OpenAI SDK and every major LLM provider, and can be adopted incrementally.

Use cases

• route requests to multiple llm providers through one gateway
• log and store llm inferences and feedback for observability
• evaluate llm outputs with heuristics and llm judges
• optimize prompts and models from production metrics
• run a/b tests and experiments on llm applications
• add fallbacks and retries to llm api calls
• monitor llm application performance in production

When to choose

• you need a unified, low-latency API across many LLM providers
• you want production observability and feedback loops for LLM apps
• you need systematic evaluation and A/B testing of prompts and models
• you want to optimize LLM systems using real production data

When to avoid

• you only need a simple single-provider SDK without ops features
• you want a fully managed SaaS with no self-hosting responsibility
• your stack cannot run Rust-based services or a gateway component

Facets

service · maturity active

llm-inference monitoring api-gateway benchmarking prompt-engineering rag agent-framework large-language-models artificial-intelligence machine-learning developer-tools rust python self-hosted cloud cross-platform llmops llm-gateway observability a-b-testing evaluation model-optimization openai-compatible opentelemetry mlops docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="tensorzero/tensorzero")