> ## Documentation Index
> Fetch the complete documentation index at: https://docs.harborframework.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Leaderboards

> Create, rank, and share evaluation results on Harbor Hub.

A leaderboard is a ranked table of results for a dataset. You define its columns and ranking rules, add rows with metadata and scores, and optionally link each row to the trials behind it.

<Note>
  Harbor Hub intentionally does not calculate row scores from linked trials, to enable maximally flexible leaderboard construction.
</Note>

Public leaderboards can be read without signing in. Private leaderboards are visible to members of the owning organization. Creating a leaderboard requires owner access to the dataset's organization.

In the commands below, `<leaderboard>` is either a UUID or an `org/dataset/leaderboard` slug.

## General approach

1. Choose a published dataset and the [versions](#dataset-versions) the leaderboard will cover.
2. [Generate a configuration](#create) with `init`, then define the metadata, metrics, columns, and ranking rules.
3. Prepare your results as [rows](#add-rows), calculating the scores yourself. Include trial IDs if you want to link each result to its runs.
4. Create the leaderboard with its definition and rows files. Keep it private while reviewing the results.
5. [Inspect the leaderboard](#browse), check the rankings and trial links, then [make it public](#edit-a-leaderboard) when it is ready.

## Browse

```bash theme={"system"}
harbor hub leaderboard list
harbor hub leaderboard list "<org>/<dataset>"
harbor hub leaderboard show "<leaderboard>"
```

`list` optionally filters by dataset slug or ID.

## Create

Generate a configuration file, edit it, then create the leaderboard:

```bash theme={"system"}
harbor auth login
harbor hub leaderboard init \
  --package "<org>/<dataset>" \
  --name main \
  --title "Main leaderboard" \
  --output leaderboard.yaml
# Edit leaderboard.yaml.
harbor hub leaderboard create --config leaderboard.yaml
```

New leaderboards are private. Set `visibility: public` in the file or pass `--visibility public` to `create`.

### Configuration schema

The definition file passed to `create --config` accepts the fields below. Use either `package` or `package_id`. Rows go in a separate file passed with `--rows`.

<ParamField body="package" type="string">
  Dataset slug, such as `acme/my-dataset`. Required unless `package_id` is provided.
</ParamField>

<ParamField body="package_id" type="string (UUID)">
  Dataset package UUID, as an alternative to `package`.
</ParamField>

<ParamField body="name" type="string" required>
  Leaderboard slug, up to 100 characters. Starts with a lowercase letter or digit; may also contain `.`, `_`, and `-`.
</ParamField>

<ParamField body="title" type="string" required>
  Display title, from 1 to 200 characters.
</ParamField>

<ParamField body="description" type="string | null">
  Optional description, up to 5,000 characters.
</ParamField>

<ParamField body="visibility" type="string" default="private">
  `public` or `private`.
</ParamField>

<ParamField body="metadata_schema" type="object" default="{}">
  Schema for each row's metadata, such as the agent name and configuration.
</ParamField>

<ParamField body="metrics_schema" type="object" default="{}">
  Schema for each row's metrics, such as scores and costs.
</ParamField>

<ParamField body="columns" type="object[]" default="[]">
  Display columns, in left-to-right order.

  <Expandable title="Column fields">
    <ParamField body="id" type="string" required>
      Unique column key. Starts with a letter, digit, or underscore; may also contain `.` and `-`.
    </ParamField>

    <ParamField body="header" type="string" required>
      Non-empty display heading.
    </ParamField>

    <ParamField body="accessor" type="string" required>
      Value path beginning with `metadata.` or `metrics.`, such as `metrics.reward`.
    </ParamField>

    <ParamField body="type" type="string" required>
      `text`, `number`, `boolean`, `date`, `markdown`, or `link`. Links accept a URL string or an object with `url` and `label`.
    </ParamField>

    <ParamField body="display_accessor" type="string">
      Alternate value path for display. The original `accessor` remains the sort value.
    </ParamField>

    <ParamField body="display_type" type="string">
      Formatter for the alternate display value; accepts the same values as `type`.
    </ParamField>

    <ParamField body="align" type="string">
      `left`, `center`, or `right`.
    </ParamField>

    <ParamField body="description" type="string">
      Explanation of the column.
    </ParamField>

    <ParamField body="enable_sorting" type="boolean">
      Whether users can sort by this column in Hub.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField body="rank_by" type="object[]" default="[]">
  Ranking rules, evaluated in order to break ties.

  <Expandable title="Ranking rule fields">
    <ParamField body="accessor" type="string" required>
      Value path beginning with `metadata.` or `metrics.`.
    </ParamField>

    <ParamField body="direction" type="string" required>
      `asc` for lowest first or `desc` for highest first.
    </ParamField>

    <ParamField body="nulls" type="string">
      Place missing values `first` or `last`.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField body="dataset_version_refs" type="string[]">
  Dataset version refs to associate, such as `latest` or a version tag. Resolved to fixed UUIDs at creation.
</ParamField>

<ParamField body="dataset_version_ids" type="string (UUID)[]">
  Dataset version UUIDs to associate. Can be combined with refs. Omit both fields to associate all existing versions; set both to `[]` for none.
</ParamField>

### Columns and ranking

The generated file includes an example with an agent name and reward score:

```yaml theme={"system"}
metadata_schema:
  type: object
  properties:
    agent:
      type: string
metrics_schema:
  type: object
  properties:
    reward:
      type: number
columns:
  - id: agent
    header: Agent
    accessor: metadata.agent
    type: text
  - id: reward
    header: Reward
    accessor: metrics.reward
    type: number
rank_by:
  - accessor: metrics.reward
    direction: desc
    nulls: last
```

This displays an agent column and a reward column, with the highest reward ranked first.

### Dataset versions

By default, a new leaderboard is associated with every dataset version that exists at creation time. Later versions are not added automatically.

To select versions explicitly, add `dataset_version_refs` or `dataset_version_ids` to the configuration:

```yaml theme={"system"}
dataset_version_refs:
  - latest
```

Refs resolve to fixed version UUIDs. Both fields can be combined; setting both to `[]` associates no versions.

## Add rows

Create `rows.yaml` with the values your columns expect:

```yaml theme={"system"}
rows:
  - metadata:
      agent: Example agent
    metrics:
      reward: 0.85
```

Then add the rows:

```bash theme={"system"}
harbor hub leaderboard row create "<leaderboard>" --config rows.yaml
```

### Row file schema

The file passed to `row create --config` or `create --rows` has this structure:

<ParamField body="rows" type="object[]" required>
  Between 1 and 500 new rows. Trial IDs must be unique across the rows in the request.

  <Expandable title="Row fields">
    <ParamField body="metadata" type="object" default="{}">
      Values matching the leaderboard's `metadata_schema`.
    </ParamField>

    <ParamField body="metrics" type="object" default="{}">
      Values matching the leaderboard's `metrics_schema`.
    </ParamField>

    <ParamField body="status" type="string" default="display">
      `display` to show the row or `hide` to hide it.
    </ParamField>

    <ParamField body="trial_ids" type="string (UUID)[]" default="[]">
      Trials linked to this row. Linking trials does not compute metadata or metrics.
    </ParamField>
  </Expandable>
</ParamField>

To create the leaderboard and its rows together, pass the separate rows file to `create`:

```bash theme={"system"}
harbor hub leaderboard create --config leaderboard.yaml --rows rows.yaml
```

## Edit a leaderboard

Change its title, description, or visibility directly:

```bash theme={"system"}
harbor hub leaderboard update "<leaderboard>" \
  --title "Updated leaderboard" --visibility public
```

Use `--description` to change the description or `--visibility private` to make the board private. To replace dataset-version associations, repeat `--dataset-version-ref` or `--dataset-version-id`. Omit these fields to preserve existing associations.

For columns, schemas, or ranking changes, export the definition, edit it, then apply it:

```bash theme={"system"}
harbor hub leaderboard export "<leaderboard>" --output leaderboard.yaml
# Edit leaderboard.yaml.
harbor hub leaderboard update "<leaderboard>" --config leaderboard.yaml
```

Flags override corresponding values in the file. The package and leaderboard name cannot be changed through `update`.

## Edit rows

List rows to find their IDs, then inspect or edit one:

```bash theme={"system"}
harbor hub leaderboard row list "<leaderboard>"
harbor hub leaderboard row show "<row-id>"
harbor hub leaderboard row export "<row-id>" --output row.yaml
# Edit row.yaml.
harbor hub leaderboard row update "<row-id>" --config row.yaml
```

Row updates change `metadata`, `metrics`, or `status`. To hide, restore, or permanently delete rows:

```bash theme={"system"}
harbor hub leaderboard row update "<row-id>" --status hide
harbor hub leaderboard row update "<row-id>" --status display
harbor hub leaderboard row delete "<row-id>" "<another-row-id>" --yes
```

Deleting rows also removes their trial links. Omit `--yes` for an interactive confirmation. The CLI has no command to delete an entire leaderboard.

### Batch edits

Export all visible rows, edit them, then apply the file:

```bash theme={"system"}
harbor hub leaderboard row export "<leaderboard>" --all --output rows.yaml
# Edit rows.yaml.
harbor hub leaderboard update "<leaderboard>" --rows rows.yaml
```

If a schema change requires row changes, apply both files together so they succeed or fail as one transaction:

```bash theme={"system"}
harbor hub leaderboard update "<leaderboard>" \
  --config leaderboard.yaml --rows rows.yaml --dry-run
harbor hub leaderboard update "<leaderboard>" \
  --config leaderboard.yaml --rows rows.yaml
```

Export both files before editing. Their timestamps protect against overwriting newer changes. `--dry-run` requires both a definition change and row updates.

## Link trials

Trial links record which runs support a row. `set` replaces all links; `add` and `remove` change only the specified links.

```bash theme={"system"}
harbor hub leaderboard row trial list "<row-id>"
harbor hub leaderboard row trial set "<row-id>" --trial-id "<trial-id>"
harbor hub leaderboard row trial add "<row-id>" --trial-id "<trial-id>"
harbor hub leaderboard row trial remove "<row-id>" --trial-id "<trial-id>"
harbor hub leaderboard row trial set "<row-id>" --clear
```

Repeat `--trial-id` for multiple trials. Alternatively, replace the links from a YAML or JSON file containing a non-empty `trial_ids` list:

```bash theme={"system"}
harbor hub leaderboard row trial set "<row-id>" --trial-ids-file trials.yaml
```

Use the file option on its own; use `--clear` to remove all links.

## Files and scripting

Configuration files accept YAML or JSON. Exports require a `.yaml`, `.yml`, or `.json` extension; `init` also accepts `--format json`. Use `--force` to overwrite an existing output file.

Read and mutation commands support `--json`. Use `list --quiet` for leaderboard slugs, `row list --quiet` for row IDs, and `row trial list --quiet` for trial IDs.

Row and trial lists support `--limit` (default 50, maximum 1,000) and `--page` (starting at 1). JSON returns one page. Quiet or piped output streams all pages unless `--page` is supplied; `--no-headers` removes piped table headers.

```bash theme={"system"}
harbor hub leaderboard row list "<leaderboard>" --page 2 --limit 100 --json
harbor hub leaderboard row trial list "<row-id>" --quiet
```

## Display a leaderboard on your own website

Fetch a public leaderboard without authentication to display it on your website:

```bash theme={"system"}
curl --fail-with-body -sS \
  "https://api.harborframework.com/functions/v1/leaderboard-read" \
  -H "Content-Type: application/json" \
  -d '{
    "package": "<org>/<dataset>",
    "name": "<leaderboard-name>",
    "page": 1,
    "page_size": 100
  }'
```

The response contains the `leaderboard` definition, ranked `rows`, and `pagination`. Fetch through `pagination.total_pages` for all rows. You can also select a board by `leaderboard_id` instead of `package` and `name`.

For private boards, authenticate from your server; keep API keys out of browser code.

See [terminal-bench-2-1](https://github.com/harbor-framework/terminal-bench-2-1) for an example pipeline and [tbench.ai](https://www.tbench.ai/) for custom visuals backed by Hub. Links to trials and versioned datasets let readers audit results and reproduce evaluations.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.