promptfoo/promptfoo
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic. observed · 2026-08-28
Health v2 · maintenance only
92/100
- Activity 99
- Release rhythm 87
- Longevity 87
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-03. Adoption (stars, forks) is never an input.
- gap_med: 1.0
- age_days: 1223
- days_rel: 7
- days_push: 7
- n_releases_24m: 243
Adoption not part of the score
24603 stars · 2234 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded
Promptfoo is an open-source CLI and library for evaluating and red-teaming LLM applications, prompts, agents, and RAG pipelines. It supports declarative YAML configs, side-by-side model comparison across providers like OpenAI, Anthropic, and Google, automated vulnerability scanning, and CI/CD integration.
Use cases
- evaluate and compare prompts across GPT, Claude, Gemini, and other models
- run automated red teaming and pentesting on LLM apps and agents
- scan RAG pipelines for prompt injection and data exfiltration vulnerabilities
- add LLM evaluation gates to CI/CD pipelines
- benchmark model quality with declarative test cases
- test chatbots for jailbreaks, PII leaks, and harmful content
- review pull requests for AI security and compliance issues
When to choose
- you need test-driven development for prompts, models, or RAG apps
- you want automated AI-specific security testing with 50+ vulnerability plugins
- you need provider-agnostic model comparison with a local web viewer
- you want evals and red teaming running locally or in CI without sending data to third parties
When to avoid
- you need a managed cloud security dashboard with SSO and team controls (enterprise offering)
- you want general-purpose unit testing of non-LLM application code
- you need fine-tuning or training infrastructure rather than evaluation
Facets
cli-tool · maturity active
testing benchmarking prompt-engineering rag vulnerability-scanning penetration-testing llm-inference ci-cd mcp large-language-models artificial-intelligence security developer-tools testing cli cross-platform self-hosted llm-evaluation red-teaming prompt-testing llmops eval-framework model-comparison jailbreak-testing security-scanning retrieval-augmented-generation ai-agents nodejs docker
10 sources
- readme: https://github.com/promptfoo/promptfoo · fetched 2026-08-28 · 70bede5f5d51
- homepage: https://promptfoo.dev · fetched 2026-08-29 · 9f2f14ea73e4
- site_page: https://www.promptfoo.dev/docs/category/integrations · fetched 2026-08-29 · ceaf62756670
- site_page: https://www.promptfoo.dev/docs/red-team/llm-vulnerability-types · fetched 2026-08-29 · be71c1028b63
- site_page: https://www.promptfoo.dev/docs/getting-started · fetched 2026-08-29 · c1edd08f95a5
- site_page: https://www.promptfoo.dev/about · fetched 2026-08-29 · ab0cdbe8fc2b
- site_page: https://www.promptfoo.dev/docs/intro · fetched 2026-08-29 · 7ca50fb3e92d
- site_page: https://www.promptfoo.dev/red-teaming · fetched 2026-08-29 · b457fcb2dea3
- site_page: https://www.promptfoo.dev/press · fetched 2026-08-29 · 60a2facc7c10
- site_page: https://www.promptfoo.dev/pricing · fetched 2026-08-29 · 9a2d3b083dca
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| promptfoo/promptfoo | main | 92 |
For agents
markdown · JSON · MCP: product_card(name="promptfoo/promptfoo")
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem