Skip to main content
​
v0.24.0
  • Live streaming: Follow agent trajectories and browse sandbox files during a run with --stream. Supports Claude Code and Codex on Docker, Daytona, Modal, Tensorlake, and Smol Machines.
  • Results viewer: Added Pareto and Outcomes views, range filters, and improved evaluation tables and trial navigation.
  • Job comparison: Use harbor run --diff to compare against previous jobs and preview which trials can be reused, regraded, or rerun.
  • Agent controls: Added --effort and a common disable_web_search option for supported agents. Harbor now validates MCP and skills capabilities before starting a sandbox.
  • New integrations: Added the Code Puppy agent and Prime, Mosaic, and Smol Machines sandboxes. Docker now supports GPU allocation through Compose.
  • Simulated users: Codex and OpenCode can now act as primary agents in simulated-user evaluations.
  • Hosted workflows: Regrade jobs and trials on Harbor Hub with --launch, share jobs during launch, and share packages with organizations when publishing.
  • RewardKit: Added JEV judges, rubric criteria, repeated judge sampling, and cost estimation for agent and LLM judges.
Breaking changes: RewardKit now groups criteria by Python file. Migrate directory-wide scoring to per-file configuration.Separate verifiers now prefer their own images and build definitions, falling back to the agent’s definition with tests uploaded to /tests. Dedicated verifier images still need bundled tests.
​
v0.23.0
  • Agent options: Discover supported settings with harbor agent schema <name>. Built-in agents validate options before starting a sandbox.
  • Job configuration: Compose config files and validate runs with harbor run --dry-run before downloading tasks or starting environments.
  • New integrations: Added Muse Code, Strands, and the FX dev-channel agent, plus Podman, Runta, and Kata sandboxes.
  • Trajectories: ATIF v1.8 supports audio content. Agent results now record token usage by model.
  • RewardKit: Judges can receive trajectory images and audio. Added FX agent judges and signed weighted aggregation.
  • Multi-step tasks: Added Hermes session resume and regrading for completed multi-step trials.
  • Hosted workflows: Added Hub job renaming, ownership transfer, and agent catalog discovery.
​
v0.22.0
  • Simulated users: Run multi-turn evaluations in which a user agent and a primary agent communicate through ACP.
  • Trajectory loading: Tasks can provide prior ATIF context through trajectory.json before the agent’s first turn.
  • New agents: Added MCode, Junie, and FX integrations.
  • New sandboxes: Added Hyperbrowser and Vercel Sandbox integrations.
  • Hosted workflows: Added hosted custom agents, resumable trial uploads, and task and dataset sharing commands.
  • RewardKit: Added nested dimension groups and simplified agent judges.
​
v0.21.0
  • Trial handoff: Resume a completed Claude Code session locally to ask the agent about its run.
  • Regrading: Use harbor job regrade or harbor trial regrade to run a new verifier against recorded trials without rerunning the agent.
  • Trajectory loading: Claude Code and Codex can load native or ATIF trajectories at run time.
  • Package versioning: Task and dataset packages now record versions and warn when a selected version has been yanked.
  • New agents: Added Cortex Code, Kimi Code, and Google Antigravity.
  • New sandbox: Added Hugging Face Sandbox.
Breaking security change: GKE sandboxes no longer mount Kubernetes service-account credentials. Workloads that need cluster access must provide credentials explicitly.
This changelog highlights user-facing stable releases and is not an exhaustive commit history. See the repository changelog, GitHub releases, or PyPI history for earlier releases.