Harbor and Terminal-Bench

Install

$ uv tool install harbor            # or pip install harbor (Python 3.12+)
$ curl -fsSLO https://froog.tech/sdk/froog.py                          # the froog client: one file
$ curl -fsSLO https://froog.tech/integrations/harbor/froog_harbor.py   # the environment
$ export FROOG_API_KEY=froog_...    # or: froog api-key create harbor --preset ci

Keep both files in the directory you run harbor from (or on PYTHONPATH).

Run Terminal-Bench 2.0

$ harbor run -d terminal-bench@2.0 -a claude-code -m anthropic/claude-sonnet-4-5 \
    -e froog_harbor:FroogEnvironment -n 8

-e module:Class is Harbor's way to name a custom environment, and -n is how many trials run at once. Each trial is a sandbox, so keep -n under your sandboxes-at-once cap and start limits. Your own tasks work the same with -p path/to/task. Options go in --ek key=value:

How it works

• One snapshot per environment. The first trial of a task environment builds it: a builder sandbox gets the task's environment/ folder and runs docker build on its Dockerfile, or docker pull of its docker_image (all Terminal-Bench 2.0 tasks name one). It creates the container harbor-env from the image and is saved as the snapshot harbor-<task>-<hash>, then deleted. The hash covers every file of the folder and the image name, so a changed task builds a new snapshot; later trials, also in later runs, reuse it. froog snapshots lists them.
• One sandbox per trial, created from that snapshot with no account environment (none of your variables, secret files or GitHub token) and no snapshots of its own. Commands run in the harbor-env container with docker exec, as the task's user, in its workdir, so a task sees the paths and tools its Dockerfile set up (/tests, root, …). Files go in and out with docker cp. Harbor deletes the sandbox at the end of the trial; the auto-stop catches the rest.
• Commands longer than an hour (froog's exec limit) run as background jobs and are followed to the end.

Notes

• Not supported: GPUs, network policies (no internet, allowlists) and docker-compose tasks. The environment refuses them before creating anything.
• The original Terminal-Bench harness (tb run, terminal-bench 0.2) builds its containers with a hard-wired Docker Compose manager and has no pluggable backend. Terminal-Bench 2.0 runs through Harbor, which is what this page covers.
• Cost. A trial on default (4 vCPU, 8 GB) that takes 10 minutes costs about half a cent of running time; the builder runs once per environment. Saved snapshots past your plan's included ones cost $0.056 a day each: delete old harbor-… snapshots with froog snapshots --rm NAME.
• Many episodes. For RL rollouts on your own environments, the same idea works directly on the SDK: build once, froog snapshot NAME --name env-v1, then one sandbox per episode with "from": "env-v1" (or fork a prepared one with count). See also the AgentEnv provider.