UKGovernmentBEIS/inspect_ai

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

2640 stars · 679 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Inspect is an open-source Python framework from the UK AI Security Institute for running large language model evaluations, with composable building blocks for datasets, solvers, tools, scorers, and agents. It ships 200+ pre-built benchmark evaluations, a web-based Inspect View for visualizing results, a VS Code extension, and sandboxing support for running untrusted model code across Docker, Kubernetes, Modal, and other backends.

Use cases

• run LLM benchmarks on any model provider
• evaluate coding agents like Claude Code or Codex CLI on agentic tasks
• build custom evaluations with prompt engineering, tool use, and multi-turn dialog
• score model outputs with model-graded and custom scorers
• sandbox untrusted model-generated code safely during evals
• monitor and visualize evaluation logs in a web UI
• test agent behavior, reasoning, and safety properties
• run MCP and custom tools inside evaluations

When to choose

• you need a rigorous, reproducible evaluation harness for LLMs or agents
• you want to run established benchmarks without writing them yourself
• you need sandboxed execution of model-generated code
• you want extensible Python tooling with a mature ecosystem and active development

When to avoid

• you only need simple one-off model API calls without evaluation scoring
• you need a production inference-serving framework rather than an eval harness
• your stack is not Python-based and you cannot add Python tooling

Facets

framework · maturity active

llm-inference agent-framework testing benchmarking mcp chatbot machine-learning developer-tools cli data-visualization large-language-models artificial-intelligence machine-learning developer-tools testing python cloud cli cross-platform llm-evaluations eval-framework benchmark-suite sandboxing model-graded-evals vscode-extension inspect-view agentic-evals ai-safety prompt-engineering ai-agents command-line docker kubernetes

Member repositories

For agents

markdown · JSON · MCP: product_card(name="UKGovernmentBEIS/inspect_ai")