pathwaycom/arc-task-gen

Health v2 · maintenance only

Flags: no_releases young

Adoption not part of the score

11149 stars · 74 forks observed · 2026-09-24

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

A Python tool that generates original ARC-AGI-1-style puzzle tasks distribution-matched to the public evaluation set. It produces tasks in the standard ARC JSON format, enabling private evaluation of frontier models on problems they have not seen before.

Use cases

• generate fresh ARC-AGI-1-style tasks for model evaluation
• create a private benchmark set to test for benchmark contamination
• evaluate reasoning models on unseen few-shot rule induction problems
• compare model performance on public vs newly generated ARC tasks
• produce ARC-format tasks.json files compatible with existing evaluation harnesses

When to choose

• you need uncontaminated ARC-AGI-1-style evaluation data for LLM or reasoning model benchmarks
• you want to measure whether model performance on ARC reflects memorization vs genuine rule induction
• you need tasks in standard ARC JSON format for an existing evaluation harness

When to avoid

• you need general-purpose synthetic data generation unrelated to ARC-style grid puzzles
• you want to train models on ARC tasks rather than evaluate them
• you need a full ARC evaluation harness rather than just task generation

Facets

library · maturity active

data-generation machine-learning benchmarking artificial-intelligence machine-learning developer-tools python cli arc-agi benchmark-evaluation synthetic-tasks llm-evaluation reasoning-models few-shot-learning algorithms

Member repositories

For agents

markdown · JSON · MCP: product_card(name="pathwaycom/arc-task-gen")