Create a verifier with Rewardkit

This tutorial builds a small task and grades it with Rewardkit. The agent writes a text statistics module and an analysis script. The verifier scores structure and correctness separately and combines them into a main reward. The finished task is reward-kit-example in the Harbor repository.

Step 1: Create the task

Install Harbor, then create a task directory:

harbor task init textstats

The finished task looks like this:

textstats/
├── instruction.md
├── task.toml
├── environment/
│   ├── Dockerfile
│   └── sample.txt
├── solution/
│   └── solve.sh
└── tests/
    ├── test.sh
    ├── criteria.py            # shared custom criteria
    ├── reward.toml            # main reward across dimensions
    ├── structure/
    │   ├── files_exist.py
    │   ├── functions_defined.py
    │   └── reward.toml
    └── correctness/
        ├── word_count.py
        ├── most_common.py
        ├── reward.toml
        └── pipeline/
            └── pipeline_runs.py

Step 2: Write the instruction and environment

textstats/instruction.md

Create a text statistics module and analysis script.

1. `/app/textstats.py` with `word_count(text: str) -> int` (words split by
   whitespace) and `most_common(text: str) -> str` (most frequent word,
   lowercase, empty string for empty input).
2. `/app/analyze.py`, which reads `/app/sample.txt` and writes
   `/app/results.json` as:

```json
{ "word_count": <int>, "most_common": "<string>" }
```

The environment needs Python, uv, and the sample text:

textstats/environment/Dockerfile

FROM ubuntu:24.04

COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/

WORKDIR /app

COPY sample.txt /app/sample.txt

textstats/environment/sample.txt

the quick brown fox jumps over the lazy dog
the dog barked at the fox
the fox ran away quickly

Leave the generated task.toml as is.

Step 3: Write the solution

The Oracle agent runs this script to confirm the task is solvable:

textstats/solution/solve.sh

#!/bin/bash

cat > /app/textstats.py << 'EOF'
from collections import Counter


def word_count(text: str) -> int:
    return len(text.split())


def most_common(text: str) -> str:
    words = text.lower().split()
    return Counter(words).most_common(1)[0][0] if words else ""
EOF

cat > /app/analyze.py << 'EOF'
import json
from pathlib import Path

from textstats import most_common, word_count

text = Path("sample.txt").read_text().strip()
results = {"word_count": word_count(text), "most_common": most_common(text)}
Path("results.json").write_text(json.dumps(results, indent=2))
EOF

cd /app && uv run python analyze.py

Step 4: Run Rewardkit from the test script

Harbor copies tests/ to /tests and runs test.sh. The script only invokes Rewardkit, which discovers every criterion under /tests, runs it against /app, and writes /logs/verifier/reward.json:

textstats/tests/test.sh

#!/bin/bash
uvx --from 'harbor-rewardkit==0.2.*' rewardkit /tests

Step 5: Add structure criteria

Each subdirectory of tests/ becomes one reward named after the directory, and each Python file inside it is one score named after the file. structure checks that the required files and functions exist, using built-in criteria. Paths are relative to /app:

textstats/tests/structure/files_exist.py

from rewardkit import criteria

criteria.file_exists("textstats.py")
criteria.file_exists("analyze.py")
criteria.file_exists("results.json")

textstats/tests/structure/functions_defined.py

from rewardkit import criteria

criteria.file_contains("textstats.py", "def word_count")
criteria.file_contains("textstats.py", "def most_common")

Step 6: Add correctness criteria

Checking return values needs custom logic. Define it once at the tests root with shared=True so any dimension can call it. Returning a float gives partial credit:

textstats/tests/criteria.py

import importlib.util
from pathlib import Path

from rewardkit import criterion


def _load_module(workspace: Path, name: str):
    spec = importlib.util.spec_from_file_location(name, workspace / f"{name}.py")
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)
    return mod


@criterion(shared=True)
def word_count_correct(workspace: Path) -> float:
    try:
        mod = _load_module(workspace, "textstats")
    except Exception:
        return 0.0
    cases = [("hello world", 2), ("one", 1), ("", 0), ("the quick brown fox", 4)]
    return sum(mod.word_count(t) == n for t, n in cases) / len(cases)


@criterion(shared=True)
def most_common_correct(workspace: Path) -> float:
    try:
        mod = _load_module(workspace, "textstats")
    except Exception:
        return 0.0
    cases = [("the cat and the dog", "the"), ("hello hello world", "hello"), ("", "")]
    return sum(mod.most_common(t) == w for t, w in cases) / len(cases)

Call the shared criteria from correctness, one per file, and put the end-to-end pipeline check in a nested pipeline/ group so it can be weighted separately:

textstats/tests/correctness/word_count.py

from rewardkit import criteria

criteria.word_count_correct()

textstats/tests/correctness/most_common.py

from rewardkit import criteria

criteria.most_common_correct()

textstats/tests/correctness/pipeline/pipeline_runs.py

from rewardkit import criteria

criteria.command_succeeds("python analyze.py")
criteria.json_key_equals("results.json", "most_common", "the")

Step 7: Configure the scoring

By default, the criteria in a Python file are averaged by weight, the files in a dimension count equally, and each dimension is written to reward.json on its own. Three reward.toml files adjust this. structure passes only when every file and function is present. [scoring.<stem>] configures the criteria within one Python file, and the unnamed [[reward]] combines the files:

textstats/tests/structure/reward.toml

[scoring.files_exist]
aggregation = "all-pass"

[scoring.functions_defined]
aggregation = "all-pass"

[[reward]]
aggregation = "all-pass"

In correctness, the two function checks and the nested pipeline group are weighted by name:

textstats/tests/correctness/reward.toml

[[reward]]
weights = { word_count = 3.0, most_common = 2.0, pipeline = 1.0 }

A root reward.toml adds the main reward that Harbor reads. It is 1.0 only when both dimensions score 1.0:

textstats/tests/reward.toml

[[reward]]
name = "reward"
aggregation = "all-pass"

The verifier now writes:

/logs/verifier/reward.json

{ "correctness": 1.0, "structure": 1.0, "reward": 1.0 }

See How scores are combined for the aggregation modes.

Step 8 (Optional): Add an LLM judge

Add a quality dimension with a judge TOML and pass the API key through task.toml:

textstats/tests/quality/quality.toml

[judge]
judge = "anthropic/claude-sonnet-5"
files = ["/app/textstats.py", "/app/analyze.py"]

[[criterion]]
description = "Are the functions implemented in a clear and idiomatic way?"
type = "binary"

textstats/task.toml

[verifier.env]
ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"

quality becomes a fourth score, and the root all-pass aggregation includes it. The criterion is binary because all-pass needs a score of 1.0, which a Likert judge rarely gives. See Judge criteria for agent judges and the full TOML reference.

Step 9: Run and inspect

Run the solution against the verifier, then open the viewer:

harbor run -p textstats -a oracle
harbor view ./jobs

Every dimension should score 1. In the trial’s Verifier Logs tab, the Rewards section renders reward-details.json as a tree with the score, weight, and any error per criterion.

reward-kit-example

The complete task built in this tutorial.

Getting started

Core concepts

Tasks

Datasets

Jobs

Agents

Sandboxes

Plugins

Results

Harbor Hub

Hosted Harbor

Rewardkit

Tutorials

Open Source Policy

Telemetry

Create a verifier with Rewardkit

Step 1: Create the task

Step 2: Write the instruction and environment

Step 3: Write the solution

Step 4: Run Rewardkit from the test script

Step 5: Add structure criteria

Step 6: Add correctness criteria

Step 7: Configure the scoring

Step 8 (Optional): Add an LLM judge

Step 9: Run and inspect

reward-kit-example

​Step 1: Create the task

​Step 2: Write the instruction and environment

​Step 3: Write the solution

​Step 4: Run Rewardkit from the test script

​Step 5: Add structure criteria

​Step 6: Add correctness criteria

​Step 7: Configure the scoring

​Step 8 (Optional): Add an LLM judge

​Step 9: Run and inspect

reward-kit-example

Step 1: Create the task

Step 2: Write the instruction and environment

Step 3: Write the solution

Step 4: Run Rewardkit from the test script

Step 5: Add structure criteria

Step 6: Add correctness criteria

Step 7: Configure the scoring

Step 8 (Optional): Add an LLM judge

Step 9: Run and inspect