petergpt/bullshit-benchmark resource

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1836 stars · 73 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

BullshitBench is a benchmark dataset and evaluation harness that tests whether AI models detect and push back on nonsensical prompts rather than confidently answering them. It includes question sets across multiple domains, judge-based scoring, and a public leaderboard viewer with charts comparing model performance.

Use cases

• evaluate whether an LLM challenges nonsensical prompts
• compare AI models on sycophancy and pushback behavior
• check if reasoning mode helps or hurts nonsense detection
• find benchmark results for new LLM releases
• visualize model detection rates across domains like finance and medical
• build an LLM evaluation leaderboard

When to choose

• you want to measure how honestly a model responds to invalid premises
• you need domain-specific nonsense detection scores across software, finance, legal, medical, and physics
• you want an existing leaderboard instead of building an eval from scratch

When to avoid

• you need general-purpose capability benchmarks like MMLU or coding evals
• you require formal statistical significance testing of model differences
• you need a benchmark for tasks other than nonsense detection

Facets

dataset · maturity active

benchmarking llm-inference data-visualization analytics large-language-models artificial-intelligence data-visualization analytics python cross-platform llm-evaluation nonsense-detection sycophancy leaderboard model-evaluation hallucination-testing web-server

Member repositories

For agents

markdown · JSON · MCP: product_card(name="petergpt/bullshit-benchmark")