microsoftarchive/promptbench

Health v2 · maintenance only

Flags: no_releases archived

Adoption not part of the score

2819 stars · 221 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

PromptBench is a unified Python library from Microsoft for evaluating and understanding large language models across many datasets, models, and prompt techniques. It includes adversarial prompt attack modules and robustness analysis to measure how LLM performance degrades under perturbed prompts.

Use cases

• benchmark llm performance across multiple datasets
• evaluate prompt robustness against adversarial attacks
• compare gpt-4o, gemini, mistral and open-source models
• measure how prompt wording affects model accuracy
• run multi-prompt evaluation efficiently
• evaluate multimodal llm capabilities

When to choose

• you need a standardized harness to compare many LLMs on academic benchmarks
• you want to study prompt sensitivity and adversarial robustness of LLMs
• you need reproducible evaluation with support for major commercial and open models

When to avoid

• you need a production serving or inference framework rather than evaluation
• you want lightweight ad-hoc testing of a single model without benchmark overhead
• you need a hosted leaderboard service rather than a local library

Facets

framework · maturity maintenance

benchmarking llm-inference prompt-engineering testing large-language-models machine-learning artificial-intelligence python cross-platform llm-evaluation adversarial-robustness prompt-robustness benchmarking-suite microsoft

Member repositories

For agents

markdown · JSON · MCP: product_card(name="microsoftarchive/promptbench")