openai/mle-bench resource

Health v2 · maintenance only

Flags: no_releases no_license

Adoption not part of the score

1720 stars · 257 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

MLE-bench is an open-source benchmark from OpenAI that measures how well AI agents perform machine learning engineering tasks, built from 75 Kaggle competitions with human baselines. It includes dataset construction code, evaluation/grading logic, and agent scaffolds for evaluating frontier LLMs.

Use cases

• evaluate how well an LLM agent does machine learning engineering
• benchmark AI agents on Kaggle-style ML competitions
• compare agent scaffolds like AIDE on ML tasks
• measure whether an agent can train models and prepare datasets autonomously
• research contamination and resource scaling for ML agents
• reproduce the MLE-bench leaderboard results for a new agent

When to choose

• you are researching or benchmarking LLM agents on real-world ML engineering tasks
• you want standardized Kaggle-derived tasks with human baselines and grading
• you need open-source evaluation harness code and reference agent scaffolds

When to avoid

• you need a general coding benchmark like SWE-bench rather than ML-specific tasks
• you want a production tool for running ML pipelines rather than an evaluation benchmark
• you require an actively accepting leaderboard submissions (currently paused)

Facets

dataset · maturity active

benchmarking agent-framework machine-learning testing machine-learning artificial-intelligence developer-tools tutorials python cross-platform kaggle evaluation llm-agents ml-engineering leaderboard openai ai-agents docker linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="openai/mle-bench")