# promptfoo/promptfoo

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration.  Used by OpenAI and Anthropic.

Repository: https://github.com/promptfoo/promptfoo
Canonical: https://ross.abutalabs.com/products/promptfoo
Homepage: https://promptfoo.dev
Language: TypeScript
License: MIT
License Family: permissive
Topics: llm, prompt-engineering, prompts, llmops, prompt-testing, testing, rag, evaluation, evaluation-framework, llm-eval, llm-evaluation, llm-evaluation-framework, ci, ci-cd, cicd, pentesting, red-teaming, vulnerability-scanners
Last push: 2026-08-27T00:14:28+00:00

## Health v2 (maintenance only)
Score: 92/100 (v2, computed 2026-09-03T02:39:23.370411+00:00)
- activity 99, release rhythm 87, longevity 87
- inputs: {"age_days": 1223, "days_push": 7, "days_rel": 7, "gap_med": 1.0, "n_releases_24m": 243}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 24603, forks 2234 (observed 2026-08-28T04:11:37.523152+00:00)

## What it is
Promptfoo is an open-source CLI and library for evaluating and red-teaming LLM applications, prompts, agents, and RAG pipelines. It supports declarative YAML configs, side-by-side model comparison across providers like OpenAI, Anthropic, and Google, automated vulnerability scanning, and CI/CD integration.

## Use cases
- evaluate and compare prompts across GPT, Claude, Gemini, and other models
- run automated red teaming and pentesting on LLM apps and agents
- scan RAG pipelines for prompt injection and data exfiltration vulnerabilities
- add LLM evaluation gates to CI/CD pipelines
- benchmark model quality with declarative test cases
- test chatbots for jailbreaks, PII leaks, and harmful content
- review pull requests for AI security and compliance issues

## When to choose
- you need test-driven development for prompts, models, or RAG apps
- you want automated AI-specific security testing with 50+ vulnerability plugins
- you need provider-agnostic model comparison with a local web viewer
- you want evals and red teaming running locally or in CI without sending data to third parties

## When to avoid
- you need a managed cloud security dashboard with SSO and team controls (enterprise offering)
- you want general-purpose unit testing of non-LLM application code
- you need fine-tuning or training infrastructure rather than evaluation

## Facets
- artifact type: cli-tool
- maturity: active
- function: testing, benchmarking, prompt-engineering, rag, vulnerability-scanning, penetration-testing, llm-inference, ci-cd, mcp
- domain: large-language-models, artificial-intelligence, security, developer-tools, testing
- platform: cli, cross-platform, self-hosted
- tags: llm-evaluation, red-teaming, prompt-testing, llmops, eval-framework, model-comparison, jailbreak-testing, security-scanning, retrieval-augmented-generation, ai-agents, nodejs, docker

## Member repositories
- promptfoo/promptfoo (main) score 92

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:11:37.523152+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T16:56:11.287315+00:00, confidence not recorded.
  - readme: https://github.com/promptfoo/promptfoo (fetched 2026-08-28T04:11:37.523152+00:00, sha 70bede5f5d51)
  - homepage: https://promptfoo.dev (fetched 2026-08-29T07:53:51.410650+00:00, sha 9f2f14ea73e4)
  - site_page: https://www.promptfoo.dev/docs/category/integrations (fetched 2026-08-29T07:53:51.436209+00:00, sha ceaf62756670)
  - site_page: https://www.promptfoo.dev/docs/red-team/llm-vulnerability-types (fetched 2026-08-29T07:53:51.431566+00:00, sha be71c1028b63)
  - site_page: https://www.promptfoo.dev/docs/getting-started (fetched 2026-08-29T07:53:51.419793+00:00, sha c1edd08f95a5)
  - site_page: https://www.promptfoo.dev/about (fetched 2026-08-29T07:53:51.423423+00:00, sha ab0cdbe8fc2b)
  - site_page: https://www.promptfoo.dev/docs/intro (fetched 2026-08-29T07:53:51.427860+00:00, sha 7ca50fb3e92d)
  - site_page: https://www.promptfoo.dev/red-teaming (fetched 2026-08-29T07:53:51.434488+00:00, sha b457fcb2dea3)
  - site_page: https://www.promptfoo.dev/press (fetched 2026-08-29T07:53:51.425838+00:00, sha 60a2facc7c10)
  - site_page: https://www.promptfoo.dev/pricing (fetched 2026-08-29T07:53:51.429810+00:00, sha 9a2d3b083dca)
- Data as of 2026-08-30T08:39:29.467469+00:00.