Skip to main content
The tests/test.sh script is the entrypoint for the verifier. Harbor uploads tests/ to /tests/ after the agent runs, unless a dedicated verifier image or build definition provides the tests. tests/test.sh is executed in the task’s working directory and is responsible for verifying the task’s completion. It must produce a numerical reward at /logs/verifier/reward.txt or /logs/verifier/reward.json.

​
Required script

The script should:
  1. Install test dependencies (if needed)
  2. Verify that the agent satisfied the instruction
  3. Write a reward under /logs/verifier/
We recommend using absolute paths inside the script to avoid cwd surprises.

​
Reward files

If both are present, Harbor prefers reward.json.

​
Example

tests/test.sh
For multi-metric or LLM-based grading, consider using RewardKit or writing a structured reward.json file.

​
Separate verifier environment

Harbor tasks can elect to use a separate sandbox for verification, which improves the security boundary between the agent and the verifier. To use a separate verifier sandbox, set environment_mode = "separate" under [verifier] in task.toml or include a [verifier.environment] section with the same schema as [environment]. Harbor prefers [verifier.environment].docker_image, then a build definition in tests/ (Dockerfile or docker-compose.yaml), over the agent environment. These dedicated verifier images must include /tests/test.sh or /tests/test.bat; Harbor does not upload tests into them at runtime. Without a dedicated verifier image or build definition, Harbor starts a fresh copy of the agent environment and uploads tests/ to /tests/. The agent’s filesystem changes are not inherited. For multi-step tasks, step verifier definitions take precedence over task-level definitions. Artifacts declared in the [artifacts] section of the task.toml are copied into the verifier sandbox at the same location as they are in the agent sandbox. See Separate verifier for more details.

​
Example

task.toml
tests/Dockerfile

​
Regrading

Harbor can rerun verifiers for existing trials if the task uses a separate verifier environment. This is useful when iterating on a verifier. See Regrade for details.

​
Passing environment variables to the verifier

Environment variables can be passed to the verifier using the [verifier.env] section in task.toml.
task.toml
The ${VAR} syntax is used to read from the host environment. Harbor will ask the user to confirm before passing the environment variables to the verifier.

​
LLM- or agent-as-a-judge

Task authors can implement test.sh to use whatever verification approach they like, including LLM- or agent-as-a-judge. To avoid boilerplate, consider using RewardKit to define your judge or programmatic criteria.

​
rewardkit

The Harbor team maintains a package called harbor-rewardkit. It is the easiest way to define and run common verifiers, including programmatic criteria and LLM- or agent-as-a-judge.