Jwuthri/Tracely-ai

Health v2 · maintenance only

Flags: no_releases young

Adoption not part of the score

1181 stars · 91 forks observed · 2026-09-03

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Tracely is a self-hosted, trace-native CI/CD and observability platform for AI agents. It ingests agent traces over OpenTelemetry, grades them with LLM-as-judge evaluators, clusters failures, freezes failing runs into hermetic replayable regression cases, and gates pull requests in CI with zero model spend.

Use cases

• turn production agent failures into regression tests
• monitor and evaluate llm agent traces in production
• block pull requests that regress ai agent behavior
• cluster agent failures into issues automatically
• replay recorded agent traces deterministically in ci
• self-host llm observability with opentelemetry ingestion

When to choose

• you ship llm agents to production and need failures to become automated tests
• you want ci gates for agent behavior without hand-authoring eval datasets
• you need self-hosted agent observability with otel-compatible instrumentation
• you want deterministic, offline replay of failing agent runs on every pr

When to avoid

• you only need simple prompt experimentation or one-off model benchmarking
• you want a fully managed hosted observability service rather than self-hosting
• your stack is not python-based or cannot emit otel traces
• you need evaluation of non-agent ml models or classical datasets

Facets

service · maturity active

monitoring tracing testing ci-cd llm-inference agent-framework mcp alerting analytics large-language-models monitoring testing developer-tools self-hosted python self-hosted cli cloud llm-observability llm-evaluation llm-as-judge regression-testing trace-native opentelemetry clickhouse ci-gate hermetic-replay llmops ai-agents devops docker web-server

Member repositories

For agents

markdown · JSON · MCP: product_card(name="Jwuthri/Tracely-ai")