Git repos

Harbor can resolve datasets directly from Git repositories using the --repo flag. This lets you run benchmarks hosted in any GitHub, GitLab, or Hugging Face repo.

Quick start

harbor run --repo "<org/repo-name>" -a "<agent>" -m "<model>"

This clones the repository, and by default runs that tasks in the tasks/ directory.

Specifying the repository

The --repo flag accepts several formats:

# GitHub shorthand (defaults to github.com)
--repo "<org/repo-name>"

# Pinned to a branch, tag, or commit
--repo "<org/repo-name>@<tag>"
--repo "<org/repo-name>@<branch>"
--repo "<org/repo-name>@<commit>"

# Full URL
--repo "https://github.com/<org/repo-name>"

# Subdirectory via /tree/ path
--repo "https://github.com/<org/repo-name>/tree/<ref>/<subdirectory>"

# Hugging Face
--repo "https://huggingface.co/datasets/<org/repo-name>"

# GitLab
--repo "https://gitlab.com/<org/repo-name>"

When no @ref is specified, the repository’s default branch is used.

Selecting a dataset

You can select datasets using -p to specify a path within the repo or -d to specify a dataset defined in a root-level registry.json file (or as specified by --registry-path).

Repo-relative path
Default registry path
Custom registry path

Run an implicit dataset from a repo-relative directory:

harbor run --repo "<org/repo-name>" -p "<path/to/tasks>" \
  -a "<agent>" -m "<model>"

Select a dataset from the repo’s root-level registry.json:

harbor run --repo "<org/repo-name>" -d "<dataset>" \
  -a "<agent>" -m "<model>"

harbor run --repo "<org/repo-name>" -d "<dataset>@<version>" \
  -a "<agent>" -m "<model>"

Select a dataset from a repo-relative registry path:

harbor run --repo "<org/repo-name>" \
  --registry-path "<path/to/registry.json>" \
  -d "<dataset>" -a "<agent>" -m "<model>"

Without -p or -d, Harbor treats the repo’s tasks/ directory as an implicit dataset.

Combining with other flags

Standard dataset flags work with --repo:

# Include specific tasks
harbor run --repo "<org/repo-name>" -d "<dataset>" \
  -i "<task-a>" -i "<task-b>" \
  -a "<agent>" -m "<model>"

# Limit task count
harbor run --repo "<org/repo-name>" -d "<dataset>" \
  -l "<count>" \
  -a "<agent>" -m "<model>"

--repo cannot be combined with --registry-url or --task / --task-git-url.

Repository layout

A git repository used with --repo should contain a registry.json at its root (or in the targeted subdirectory). This is the same format used by local --registry-path:

my-repo/
├── registry.json # optional, required for -d
└── tasks/
    ├── task-a/
    │   ├── task.toml
    │   ├── instruction.md
    │   ├── environment/
    │   └── tests/
    ├── task-b/
    │   └── ...
    └── ...

The registry.json maps dataset names to task lists. See Tasks → Overview for task directory structure.

Authentication

Public repositories work without any configuration. For private repositories, Harbor uses the Git credentials available in your environment (SSH keys, credential helpers, GIT_ASKPASS, etc.). If git ls-remote can reach the repo, --repo will work.

Getting started

Core concepts

Tasks

Datasets

Jobs

Agents

Sandboxes

Plugins

Results

Harbor Hub

Hosted Harbor

Rewardkit

Tutorials

Open Source Policy

Telemetry

Quick start

Specifying the repository

Selecting a dataset

Combining with other flags

Repository layout

Authentication

​Quick start

​Specifying the repository

​Selecting a dataset

​Combining with other flags

​Repository layout

​Authentication

Quick start

Specifying the repository

Selecting a dataset

Combining with other flags

Repository layout

Authentication