Custom verifiers

During a trial, after the agent runs, the Verifier object uploads the tests/ directory to the sandbox and executes /tests/test.sh. The Verifier object is the default implementation of the BaseVerifier. If a user wants to orchestrate verification differently, or run some of the verifier logic on the host machine, they can implement a custom BaseVerifier.

	`tests/` w/default `Verifier`	Custom `BaseVerifier`
Logic	Task files, typically `tests/test.sh`	An importable Python class
Execution	Inside the verifier sandbox	`verify()` runs in the Harbor process and can call sandbox operations through `self.environment`
Harbor handles	Uploading tests, running the script, downloading verifier logs, and parsing reward files	Constructing the verifier and calling `verify()`
Reward output	`/logs/verifier/reward.txt` or `/logs/verifier/reward.json`	A returned `VerifierResult`
Configuration	Included with the task	`verifier.import_path` and optional `verifier.kwargs` in the job config
Best for	Portable, task-specific verification	Custom orchestration or host-side checks

A custom verifier replaces the default workflow. It must perform any required uploads, commands, and reward calculation itself.

For example, this verifier checks whether the agent created a required file in the sandbox:

custom_verifier.py

from typing import Any

from harbor.models.verifier.result import VerifierResult
from harbor.verifier.base import BaseVerifier


class OutputVerifier(BaseVerifier):
    def __init__(
        self, *, required_path: str = "/app/output.json", **kwargs: Any
    ) -> None:
        super().__init__(**kwargs)
        self.required_path = required_path

    async def verify(self) -> VerifierResult:
        exists = await self.environment.is_file(self.required_path)
        return VerifierResult(rewards={"reward": int(exists)})

The file must be importable by the Harbor process. Select the class with verifier.import_path; values in verifier.kwargs are passed to its constructor.

CLI
Config

harbor run -p "<path/to/dataset>" -a "<agent>" -m "<model>" \
  --verifier custom_verifier:OutputVerifier \
  --verifier-kwarg required_path=/app/output.json

verifier:
  import_path: custom_verifier:OutputVerifier
  kwargs:
    required_path: /app/output.json

See Configs for the complete verifier configuration schema.

Getting started

Core concepts

Tasks

Datasets

Jobs

Agents

Sandboxes

Plugins

Results

Harbor Hub

Hosted Harbor

Rewardkit

Tutorials

Open Source Policy

Telemetry

Custom verifiers