carlini/yet-another-applied-llm-benchmark resource

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1064 stars · 77 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A personal benchmark of nearly 100 applied tests for evaluating how well large language models perform on practical tasks the author has actually asked LLMs to solve, built on a small dataflow DSL for defining test pipelines. Tests run generated code in Docker containers and use programmatic and LLM-based evaluators to score model outputs.

Use cases

• benchmark large language models on real-world applied tasks
• compare how different LLMs perform on code generation and reasoning tests
• evaluate whether a model can write and run working programs in python, c, and bash
• add custom LLM evaluation tests using a simple dataflow DSL
• check if a model can parse grammars, convert english to sql, or decompile bytecode
• track LLM capability regressions across model versions

When to choose

• you want applied, task-based evaluation of LLMs rather than academic question answering
• you need a lightweight harness to run generated code in containers and automatically judge outputs
• you want to add your own realistic test cases quickly with a minimal DSL
• you want reference scores for popular models like GPT-4o, Claude, and Gemini on practical tasks

When to avoid

• you need a rigorously validated academic benchmark with statistical guarantees
• you require broad standardized coverage aligned with published leaderboards
• you want a benchmark focused on a single narrow capability rather than diverse applied tasks
• you cannot run Docker or execute generated code locally

Facets

dataset · maturity active

benchmarking llm-inference agent-framework testing developer-tools large-language-models machine-learning developer-tools testing artificial-intelligence python cli llm-benchmark evaluation-harness dsl prompt-evaluation code-generation-evaluation model-evaluation applied-tasks docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="carlini/yet-another-applied-llm-benchmark")