CLI

Please ensure you have either logged in with harbor auth login or minted an API key (see Creating a Harbor Hub API Key)

Launching a Job

The command harbor run --launch will attempt to launch a job under your personal organization The --launch flag can be combined with a singular agent/model combination (e.g. harbor run --launch -a claude-code -m anthropic/fable-5 -d harbor/hello-world). Agent/model sweeps must be launched using a config file (e.g. harbor run --launch -c config.yaml)

# config.yaml
job_name: opus-vs-codex-sweep
organization: my-org          # omit to use your personal org
credential_mode: gateway      # or "direct"

n_attempts: 3                 # 1-10
n_concurrent_trials: 32       # 1-1000

# Every agent runs against every task. Trials = n_attempts x tasks x agents.
agents:
  - name: claude-code
    model_name: anthropic/claude-opus-4-1
    secrets: [ANTHROPIC_API_KEY]

  - name: claude-code
    model_name: anthropic/claude-sonnet-4-5
    secrets: [ANTHROPIC_API_KEY]

  - name: codex
    model_name: openai/gpt-5
    secrets: [OPENAI_API_KEY, HF_TOKEN]
    n_concurrent: 8           # per-agent sub-limit under n_concurrent_trials
    env:
      RUST_LOG: info          # nonsensitive only - secrets go in `secrets`

  - name: oracle              # needs no inference credential
    secrets: []

datasets:
  - name: terminal-bench/terminal-bench-2-1
    ref: "6"                  # `ref` or `version`, never both
    n_tasks: 25
    exclude_task_names:
      - "flaky-*"

tasks:
  - name: harbor/hello-world
    ref: latest

# Supplied for this job only, read from your local environment at launch.
job_secrets:
  HF_TOKEN:
    from_env: HF_TOKEN

retry:
  max_retries: 2
  include_exceptions:
    - EnvironmentStartError

Injecting Secrets

There are two methods of injecting secrets into the hosted job: flags --stored-secret <ENV_VAR_NAME> and --one-off-secret ENV_VAR_NAME=secret_value, and config fields job_secrets and secrets. When using flags, --stored-secret will pull <ENV_VAR_NAME> from the organization’s stored credentials on the hub. --one-off-secret will encrypt the secret value and ensure that the value is not persisted in any job config files. The secret will be decrypted only when necessary and marked revoked after the job finishes. Flags apply secrets to every agent/model combination in the job config. In other words, if I launch with the following command: harbor run --launch -c simple_config.yaml --stored-secret TEST_VAR where simple_config.yaml is

job_name: simple-config
organization: my-org          # omit to use your personal org

agents:
  - name: claude-code
    model_name: anthropic/claude-opus-4-1
    secrets: [ANTHROPIC_API_KEY]

  - name: claude-code
    model_name: anthropic/claude-sonnet-4-5
    secrets: [ANTHROPIC_API_KEY]

tasks:
  - name: harbor/hello-world
    ref: latest

The final resolved config would be as follows:

job_name: simple-config
organization: my-org

agents:
  - name: claude-code
    model_name: anthropic/claude-opus-4-1
    secrets: [ANTHROPIC_API_KEY, TEST_VAR] # notice TEST_VAR is used in both agents

  - name: claude-code
    model_name: anthropic/claude-sonnet-4-5
    secrets: [ANTHROPIC_API_KEY, TEST_VAR] # notice TEST_VAR is used in both agents

tasks:
  - name: harbor/hello-world
    ref: latest

Flag	Value	Effect
`--dry-run`	—	Validate without queuing. Resolves tasks, agents, and the owning org, checks selections, and reports the trial count
`--org`	`TEXT`	Organization that should own the hosted job. Defaults to your personal org
`--credential-mode`	`gateway` \| `direct`	`gateway` proxies the key through the Hub, `direct` hands the real key to the agent
`--stored-secret`	`NAME`	Select a secret already stored in the owning org. Repeatable
`--one-off-secret`	`NAME[=VALUE]`	Supply a secret for this job only. Bare `NAME` reads it from your environment. Repeatable
`--env-file`	`PATH`	Load a `.env`. Every name in it is selected, as with `--one-off-secret`
`--registry-secret`	`HOST=NAME_OR_ID`	Pin a stored pull secret for a private image host. Repeatable
`--no-secrets`	—	Launch with no credentials at all

--launch and --upload are mutually exclusive, so the upload-side flags do not apply to a hosted launch: --upload, --public / --private, --share-org, and --share-user.

Listing Jobs

Using the command harbor hub job list you can print out a list of all jobs visible to your user on the hub.

Flag	Value	Effect
`--scope`	`my` \| `shared` \| `all`	Visibility scope. Defaults to `my`, the jobs your user owns
`--search`	`TEXT`	Only jobs whose names contain the substring
`--agent`	`NAME`	Filter by agent name. Repeatable, and multiple values match any of them
`--provider`	`NAME`	Filter by model provider. Repeatable
`--model`	`NAME`	Filter by model. Repeatable

Job Overview

Use the command harbor hub job show JOB_ID JOB_ID_2 ... to print an overview similar to the one provided on the hub. The overview reports the number of trials, errors, and retries, the average return on each metric, the cost in USD, and token usage.

Per-Task Job Breakdown

Use the command harbor hub job tasks JOB_ID to see a per-task breakdown of a job.

Flag	Value	Effect
`--search`	`TEXT`	Only tasks whose names contain the substring
`--agent`, `--provider`, `--model`	`NAME`	As for `job list`

Trials Overview

Use the command harbor hub job trials JOB_ID JOB_ID_2 ... to list trials across one or more jobs. In interactive terminals, you can switch pages to audit all trials without running a second command.

Flag	Value	Effect
`--search`	`TEXT`	Only trials whose names contain the substring
`--agent`, `--provider`, `--model`	`NAME`	As for `job list`
`--limit`	`N`	Page size. Defaults to 100 here
`--failed-only`	—	Only trials that errored or failed
`--include-retries`	—	Include retry history, not just the latest execution
`--sort-by`	`started_at` \| `task_name` \| `name` \| `error_type`	Sort column
`--sort-order`	`asc` \| `desc`	Sort direction

Comparing 2 or More Jobs

Use command harbor hub job compare JOB_ID JOB_ID_2 ... to get the side-by-side grid for performance on tasks

Job Visibility

Use command harbor hub job shares JOB_ID to see who a job is shared with. This returns orgs and users

Deleting Jobs

Use command harbor hub job delete JOB_ID JOB_ID_2 ... to permanently delete jobs you own from the hub, including all of their trials and shares. The command prompts for confirmation before deleting anything; pass --yes / -y to skip the prompt (required when scripting or piping). Only the job’s owner can delete a job. Jobs linked to a leaderboard submission and hosted jobs that are still running cannot be deleted.

Hosted Job Status

Use command harbor hub job status JOB_ID to get your job status. Returns counts of pending, running, failed, and completed trials.

Trial Show

Use command harbor hub trial show TRIAL_ID to show a single trial’s metadata

Downloading a Trial

Use command harbor hub trial download TRIAL_ID to download a particular trial

Flag	Value	Effect
`--output-dir`, `-o`	`PATH`	Directory to materialize the trial into. Defaults to `./trials`
`--overwrite`	—	Replace an existing trial directory
`--trajectory`	—	Download only `trajectory.json`. Errors if the trial has no stored trajectory

Retrying a Hosted Trial

For a trial in a job that was launched remotely, you can use the command harbor hub trial retry TRIAL_ID to retry those trials. These flags are cumulative filters. harbor hub trial retry --job JOB_ID --failed-only selects every trial in the job, then narrows to the ones that failed.

Flag	Value	Selects
`--job`	`JOB_ID`	Every trial in that job
`--search`	`TEXT`	Trials whose names contain the substring
`--agent`	`TEXT`	Trials run with that agent
`--provider`	`TEXT`	Trials run with that provider
`--model`	`TEXT`	Trials run with that model
`--task`	`TEXT`	Trials run against that task
`--exception`	`TEXT`	Trials that failed with that exception
`--failed-only`	—	Only trials that failed
`--yes`	—	Skip confirmation

Canceling a Hosted Trial

To cancel a hosted trial, use command harbor hub trial cancel TRIAL_ID

Flag	Value	Effect
`--job`	`JOB_ID`	Cancel every trial in the job. Same as `harbor hub job cancel JOB_ID`
`--all`	—	Cancel every currently running trial
`--reason`	`TEXT`	Record a reason for the cancellation
`--yes`	—	Skip confirmation

The selection filters from trial retry apply here too.

Adding Secrets

Use command harbor hub secrets add NAME to upload a secret to the Hub.

Flag	Value	Effect
`--org`	`ORG_ID`	Organization to store the secret on. Defaults to your personal org
`--job`	`JOB_ID`	Scope the secret to a single job instead of account-wide
`--from-env`	—	Read the value from the local environment variable of the same name
`--yes`, `-y`	—	Supersede an existing secret without prompting

Listing Secrets

Use command harbor hub secrets list to list the names and metadata of your uploaded secrets.

Flag	Value	Effect
`--org`	`ORG_ID`	Organization to list secrets for
`--job`	`JOB_ID`	List secrets scoped to that job
`--include-revoked`	—	Include secrets that have been revoked

Deleting Secrets

Use command harbor hub secrets delete to revoke a secret.

Flag	Value	Effect
`--org`	`ORG_ID`	Revoke a secret in that organization
`--job`	`JOB_ID`	Revoke a secret scoped to that job
`--purge`	—	Delete the record outright, so it no longer appears under `list --include-revoked`

Adding an Image Registry Secret

Use command harbor hub secrets registry add to add an image registry secret. Also supports secrets registry list and secrets registry delete.

Flag	Value	Effect
`--name`	`TEXT`	Display name used to select the credential later
`--from-file`	`PATH`	Read the credential from a file, such as a GAR service account JSON
`--yes`, `-y`	—	Supersede an existing credential without prompting

Shared Flags

The listing commands above also accept these:

Flag	Value	Effect
`--quiet`, `-q`	—	Print only IDs, for piping into `xargs`
`--no-trunc`	—	Show full cell content, wrapping instead of truncating
`--no-headers`	—	Omit the header row
`--page`	`N`	Fetch one specific page, disabling interactive paging
`--json`	—	Return the raw API response as JSON

Getting started

Core concepts

Tutorials

Open Source Policy

Telemetry

Launching a Job

Injecting Secrets

Listing Jobs

Job Overview

Per-Task Job Breakdown

Trials Overview

Comparing 2 or More Jobs

Job Visibility

Deleting Jobs

Hosted Job Status

Trial Show

Downloading a Trial

Retrying a Hosted Trial

Canceling a Hosted Trial

Adding Secrets

Listing Secrets

Deleting Secrets

Adding an Image Registry Secret

Shared Flags

​Launching a Job

​Injecting Secrets

​Listing Jobs

​Job Overview

​Per-Task Job Breakdown

​Trials Overview

​Comparing 2 or More Jobs

​Job Visibility

​Deleting Jobs

​Hosted Job Status

​Trial Show

​Downloading a Trial

​Retrying a Hosted Trial

​Canceling a Hosted Trial

​Adding Secrets

​Listing Secrets

​Deleting Secrets

​Adding an Image Registry Secret

​Shared Flags

Launching a Job

Injecting Secrets

Listing Jobs

Job Overview

Per-Task Job Breakdown

Trials Overview

Comparing 2 or More Jobs

Job Visibility

Deleting Jobs

Hosted Job Status

Trial Show

Downloading a Trial

Retrying a Hosted Trial

Canceling a Hosted Trial

Adding Secrets

Listing Secrets

Deleting Secrets

Adding an Image Registry Secret

Shared Flags