# harbor-framework/harbor

Framework for evaluating and improving agents

Repository: https://github.com/harbor-framework/harbor
Canonical: https://ross.abutalabs.com/products/harbor-framework-harbor
Homepage: https://harborframework.com/
Language: Python
License: Apache-2.0
License Family: permissive
Topics: evals, rl-environments, terminal-bench
Last push: 2026-08-26T22:23:30+00:00

## Health v2 (maintenance only)
Score: 85/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 99, release rhythm 99, longevity 28
- inputs: {"age_days": 394, "days_push": 7, "days_rel": 11, "gap_med": 4.0, "n_releases_24m": 27}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 4664, forks 1662 (observed 2026-08-28T04:08:56.474624+00:00)

## What it is
Harbor is a Python framework from the creators of Terminal-Bench for evaluating and optimizing AI agents and language models in containerized sandbox environments. It provides modular interfaces for tasks, agents, and environments, integrates popular CLI agents and cloud sandbox providers, and supports generating rollouts for RL optimization.

## Use cases
- evaluate coding agents like Claude Code on Terminal-Bench 2.0
- run SWE-Bench or Aider Polyglot benchmarks against a model
- build and share custom agent benchmarks and task environments
- run thousands of eval environments in parallel on cloud sandboxes
- generate rollout traces for reinforcement learning or SFT training
- test agents in CI/CD pipelines

## When to choose
- you need a standardized harness to benchmark agents across many tasks
- you want to scale agent evaluations horizontally across cloud sandbox providers
- you are building custom evals or RL environments for LLM agents
- you want pre-integrated CLI agents like Claude Code, OpenHands, or Codex CLI

## When to avoid
- you only need simple LLM API calls without task evaluation
- you need a GUI-based evaluation dashboard rather than a CLI harness
- your evaluation targets are not containerizable tasks

## Facets
- artifact type: framework
- maturity: active
- function: agent-framework, llm-inference, llm-training, benchmarking, testing, rag
- domain: artificial-intelligence, large-language-models, developer-tools, machine-learning
- platform: python, cli, cross-platform, cloud
- tags: agent-evaluation, evals, terminal-bench, reinforcement-learning-environments, sandboxed-tasks, benchmark-harness, rollouts, ai-agents, docker

## Member repositories
- harbor-framework/harbor (main) score 85

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:08:56.474624+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T18:19:25.675921+00:00, confidence not recorded.
  - readme: https://github.com/harbor-framework/harbor (fetched 2026-08-28T04:08:56.474624+00:00, sha a72055fb613d)
  - homepage: https://harborframework.com/ (fetched 2026-08-29T09:03:47.455055+00:00, sha f066eb1debe2)
  - site_page: https://www.harborframework.com/docs (fetched 2026-08-29T09:03:47.457485+00:00, sha 7c122151c895)
- Data as of 2026-08-30T08:39:29.467469+00:00.