promptfoo/promptfoo

Health v2 · maintenance only

Adoption not part of the score

26091 stars · 2501 forks observed · 2026-10-18

What it is AI-extracted, prompt v1, taxonomy v1, 2026-10-12, confidence 0.97

Promptfoo is an open‑source command‑line tool and library for evaluating, testing, and red‑team­ing LLM applications, agents, and RAG pipelines. It provides declarative configs, CI/CD integration, and a web viewer for comparing model outputs and security findings.

Use cases

• Run automated evaluations of prompts across multiple LLM providers.
• Red‑team LLM agents to discover jailbreaks, data leaks, and other vulnerabilities.
• Compare model performance side‑by‑side in a matrix view.
• Integrate prompt testing into CI/CD pipelines for continuous security checks.
• Generate compliance reports mapping findings to standards like OWASP or NIST.

When to choose

• You need a fast, declarative way to test and secure LLM prompts without writing custom code.
• Your workflow includes CI/CD pipelines and you want automated regression testing for LLMs.
• You require built‑in red‑team attack libraries and vulnerability types for agents or RAGs.
• You want a self‑hosted solution that runs locally and can be extended with custom plugins.

When to avoid

• You are looking for a full‑featured LLM serving framework rather than a testing tool.
• Your project relies on languages without Node.js support and you cannot use the CLI.
• You need deep model training or fine‑tuning capabilities, which Promptfoo does not provide.

Facets

cli-tool · maturity active

testing security artificial-intelligence large-language-models cli self-hosted prompt-testing rag-evaluation llm-red-teaming ci-cd-integration open-source ai-agents retrieval-augmented-generation nodejs docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="promptfoo/promptfoo")