Verifiers v0.2.0 Release Notes

Date: 07/10/2026

Highlights since v0.1.14

• A task-centric v1 API. Verifiers v1 now separates serializable per-row TaskData from the Task class that owns lifecycle hooks, tools, metrics, rewards, and judges. A Taskset loads typed tasks, a Harness defines how the model runs, and the two compose into an environment without mixing v1 and legacy APIs.
• Portable agent execution. The same taskset can run across built-in and custom harnesses on local subprocesses, Docker, or remote Prime and Modal sandboxes. Model traffic passes through the interception layer, which adapts the harness's provider dialect, records the trace as it happens, applies sampling configuration, and supports response and tool-result interception.
• Typed authoring, tools, and scoring. Strict Pydantic configs cover tasksets, tasks, harnesses, runtimes, and evaluation. Environment authors can scaffold packages with uv run init, expose taskset- or task-scoped MCP tools, attach user simulators, and score traces with rewards, metrics, group rewards, reusable scoring helpers, and configurable LLM judges.
• A complete v1 evaluation workflow. uv run eval supports TOML configs, dotted CLI overrides, dry runs, concurrency controls, rich progress and paging, resumable runs, and platform upload. Native validate, debug, and replay entrypoints cover environment checks, interactive diagnosis, and offline re-scoring of saved traces.
• First-class traces and observability. Traces capture the message graph, task identity and data, rewards, metrics, errors, token and timing information, runtime metadata, and per-message creation timestamps. Evaluation artifacts now use traces.jsonl, with improved streaming, memory use, and dashboard reporting for long or highly concurrent runs.
• Harbor and production environment support. v1 includes a Harbor taskset base, reusable built-in harnesses and tasksets, container-aware runtime configuration, and a companion catalog of production benchmarks in research-environments.

Selected changes included in v0.2.0

v1 framework and authoring

• Introduce the complete Verifiers v1 taskset, harness, runtime, trace, and CLI stack (#1825)
• Move per-task behavior from Taskset onto typed Task objects (#1948)
• Add native v1 debug and validate workflows (#1905)
• Add reusable scoring helpers and configurable reference/rubric judges (#1907, #1918)
• Add built-in Harbor support and reusable Lean taskset foundations (#1877, #1882, #1883)
• Refresh the v1 user documentation (#1916)

Evaluation, traces, and reliability

• Add offline trace replay and platform upload for eval runs (#1936, #1947)
• Rename the v1 evaluation artifact to traces.jsonl (#1956)
• Add trace record export, token-length attribution, runtime metadata, and message timestamps (#1844, #1875, #1959, #1960)
• Improve the rich dashboard with phase markers, usage totals, pending-rollout visibility, and paging (#1866, #1869, #1912, #1913)
• Make resumed runs restore their saved traces and complete only missing or errored work (#1892, #1962)
• Improve streaming, concurrency, and memory behavior for large v1 evaluations (#1793, #1807, #1815, #1838)

Full Changelog: v0.1.14...v0.2.0

What's Changed

• Require auth for interception endpoints by @xeophon in #1304
• feat(composable): render pre-compaction branches via harness hook by @snimu in #1291
• fix: mute httpcore/httpx DEBUG when env-worker json_logging sets root level by @rasdani in #1234
• Make sandbox worker caps configurable by @rasdani in #1305
• Support setup_state return-state compatibility by @willccbb in #1308
• Update v1 Taskset/Harness docs and opencode_harbor migration by @willccbb in #1309
• bump interception client size to handle trajectories with images by @eligotts in #1310
• Fix eval summary for heterogeneous metrics by @xeophon in #1295
• Refactor deep-agents envs to v1 taskset-harness and add Wikispeedia example by @willccbb in #1317
• Add SWE debug environment by @rasdani in #1306
• Move OpenCode config into reusable harness package by @willccbb in #1318
• Remove direct prime-sandboxes dependency from rlm-swe v1 by @willccbb in #1316
• Install sandbox fn program packages by @willccbb in #1319
• opencode: write config under XDG_CONFIG_HOME by @rasdani in #1320
• feat: surface swallowed response_future errors in interception proxy by @rasdani in #1196
• apt: harden sandbox bootstrap against transient archive.ubuntu.com flakes by @rasdani in #1284
• fix(renderer-client): wrap tools in OpenAI envelope to match training distribution by @hallerite in #1307
• fixes rlm compaction prompt so we still get tito hit by @eligotts in #1322
• Fix interception keepalive env override by @rasdani in #1321
• Pin OpenCode small model to intercepted provider by @rasdani in #1323
• Avoid holding sandbox workers with run_background_job() by @rasdani in #1328
• Skip OpenCode install when binary is pre-baked by @rasdani in #1176
• fix: narrow send_cancel BaseException catch to Exception by @rasdani in #1198
• Fix Harbor Hub task root handling by @xeophon in #1345
• Pin OpenCode small model in legacy configs by @rasdani in #1327
• Remove local training config scaffolding from Verifiers by @willccbb in #1348
• retry Read file timed out by @kcoopermiller in #1350
• chore: v0.1.15.dev0 dev release by @rasdani in #1352
• Tighten BugBot releasability and dependency rules by @willccbb in #1353
• Strengthen BugBot release and dependency rules by @willccbb in #1354
• Fix v1 lifecycle handler discovery by @xeophon in #1347
• Fix ZMQ env timeouts by @xeophon in #1355
• feat(renderer-client): thread multimodal sidecar through rollout + transport by @hallerite in #1346
• Map Anthropic reasoning effort by provider by @xeophon in #1338
• chore: v0.1.15.dev1 dev release by @xeophon in #1357
• Disable OpenCode title generation in harness configs by @rasdani in #1359
• chore: v0.1.15.dev2 dev release by @rasdani in #1360
• Use background jobs for v1 sandbox commands by @xeophon in #1364
• fix: treat OverlongPromptError as stop condition in interception proxy by @mikasenghaas in #1365
• chore: v0.1.15.dev3 dev release by @mikasenghaas in #1367
• [codex] Display eval cost from Prime pricing by @willccbb in #1368
• Add repo development best practices to AGENTS, docs, and skills by @willccbb in #1361
• Tighten v1 config, bindings, and handler typing by @willccbb in #1362
• Prepare v0.1.15.dev4 release notes and version bump by @willccbb in #1370
• Add Terminus2 harness by @xeophon in #1356
• chore: v0.1.15.dev5 dev release by @xeophon in #1372
• feat: consume vllm routed experts payloads by @S1ro1 in #1363
• fix: move routed experts helpers to response utils by @S1ro1 in #1373
• Tighten v1 EnvConfig typing by @willccbb in #1371
• perf(save): skip backward-byte scan on hot path by @mikasenghaas in #1376
• chore: v0.1.15.dev6 dev release by @willccbb in #1375
• Clarify managed Lab AGENTS guidance and add AGENTS.local.md overlay by @willccbb in #1383
• Add per-eval names for duplicate environment runs by @willccbb in #1384
• Document eval name labels in skill by @willccbb in #1388
• improvement: switch verifiers PyPI publish to OIDC trusted publishing by @JannikSt in #1386
• release v0.1.15.dev7 by @JannikSt in #1389
• Modernize eval viewer TUI by @xeophon in #1393
• chore: bump renderers to 0.1.8.dev2 (supersedes #1366) by @hallerite in #1395
• Retire vf-tui entrypoint by @xeophon in #1398
• fix(save_utils): reset delta baseline on non-monotonic trajectories by @eligotts in #1400
• fix(renderer-client): translate renderers.OverlongPromptError into vf.OverlongPromptError by @hallerite in #1408
• [Router Replay]: Improve performance by removing Pydantic validation by @S1ro1 in #1394
• Revert "[Router Replay]: Improve performance by removing Pydantic validation" by @willccbb in #1422
• Support routed experts response sidecar by @S1ro1 in #1423
• Rework v1 harness and taskset config classes by @xeophon in #1392
• Skip SWE sandbox scoring for errored rollouts by @rasdani in #1412
• Forward reasoning kwargs from eval TOML sampling sections by @willccbb in #1404
• Remove stale pre-eval prime env install docs by @willccbb in #1434
• Default TOML eval runs to save results by @willccbb in #1435
• Add strict package loaders for v1 tasksets and harnesses by @willccbb in #1429
• [codex] Prepare v0.1.15.dev8 release by @willccbb in #1436
• Remove unused verifier helpers by @xeophon in #1437
• Use Earendil Pi package by @xeophon in #1442
• Propagate strict tool schemas from callable tools by @hubert-marek in #1445
• Migrate textarena_taskset into the v1 tasksets package by @willccbb in #1446
• Prepare v0.1.15.dev9 release by @willccbb in #1448
• Fix RLM skills upload path by @xeophon in #1451
• Use renderers offload helper in renderer client by @xeophon in #1452
• perf(serve): reduce worker/router event-loop lag by @mikasenghaas in #1453
• Inline single-use verifier helpers by @xeophon in #1443
• [codex] Forward renderer request headers by @samsja in #1459
• Make harness setup scripts POSIX sh compatible by @hubert-marek in #1457
• Fix Anthropic unsupported image markers by @xeophon in #1461
• Fix EnvServer executor installation by @xeophon in #1455
• [codex] Revise v1 init taskset loaders by @willccbb in #1449
• [codex] Default eval session header to trajectory_id by @samsja in #1460
• Expose V1 tools to the RLM harness by @xeophon in #1456
• Prepare v0.1.15.dev10 release by @xeophon in #1465
• Consume the typed RendererConfig surface by @hallerite in #1467
• Rework TasksetConfig fields + loader patterns by @willccbb in #1462
• Prepare v0.1.15.dev11 release after #1462 by @willccbb in #1469
• feat(renderer-client): give per-token prompt attribution to TrajectoryStep by @snimu in #1414
• fix(renderer-client): surface every parsed tool_call, drop status filter by @eligotts in #1463
• Use managed sandbox Python runtime by @xeophon in #1450
• Pipe chat_template_kwargs into typed RendererConfig by @eligotts in #1468
• Add v1 sandbox labels by @xeophon in #1476
• renderers <-> v1: serializable RenderedTokens by @snimu in #1471
• Set routed experts replay start for bridged prompts by @S1ro1 in #1466
• fix(lean-taskset): match LAST EXIT_CODE marker to close reward bypass by @hallerite in #1480
• Fix v1 sandbox client reuse and upload errors by @xeophon in #1483
• Stage standalone tasksets and harnesses packages with hardened v1 program configs by @willccbb in #1475
• Harden v1 sandbox lifecycle boundaries by @willccbb in #1490
• Add NeMo Gym integration by @cdreetz in #1396
• Update taskset and harness release flows for untagged PyPI releases by @willccbb in #1489
• Prepare verifiers v0.1.15.dev12 release by @willccbb in #1491
• Add typed v1 eval CLI overrides and bump dev version to 0.1.15.dev13 by @willccbb in #1492
• Handle structured TextArena assistant content by @willccbb in #1493
• Stop TextArena rollouts without guesses by @willccbb in #1495
• Preserve TextArena empty-guess behavior by @willccbb in #1496
• Fix wordle_v1 TextArena dependency by @xeophon in #1497
• Harden v1 environment loader guidance by @willccbb in #1498
• Improve v1 env template guidance by @willccbb in #1499
• Add gpt-oss eval and GEPA config templates by @willccbb in #1500
• [codex] Align SystemPrompt alias with config fields by @willccbb in #1501
• Raise interception server request body limit to 24 MB by @hubert-marek in #1504
• Restore trusted PyPI publishing for tasksets and harnesses by @xeophon in #1508
• Use real parallel tool calls for mini-swe-agent by @xeophon in #1505
• Re-add labels field to v1 SandboxConfig by @snimu in #1512
• Prepare v0.1.15.dev18 release by @snimu in #1514
• Fix v1 endpoint port pre-probe race (bind 0.0.0.0:0 + readback) by @snimu in #1513
• chore: drop rlm_max_turns from RLM harness config by @mikasenghaas in #1511
• Make v1 harness max_turns unbounded by default by @mikasenghaas in #1517
• Add Scale-SWE taskset by @rasdani in #1439
• Align Multi-SWE patch extraction with upstream by @rasdani in #1523
• Scale V1 sandbox runtime lifecycle by @xeophon in #1520
• Group SWE task utilities by backend by @rasdani in #1524
• Fix InterceptionServer re-binding a stale port across restarts by @snimu in #1526
• fix(v1): record final/context token usage at write time by @hallerite in #1525
• feat: git-driven versioning and unified publish-verifiers workflow by @mikasenghaas in #1527
• [codex] Handle empty v1 taskset splits as absent datasets by @willccbb in #1519
• Restrict verifiers release tags to stable semver by @xeophon in #1528
• Use registry images for Scale-SWE by @rasdani in #1531
• feat: log rollout start and finish in v1 Harness.run by @mikasenghaas in #1534
• fix: default rollout max_retries to 3 by @mikasenghaas in #1533
• [codex] Reject reasoning-only model responses by @willccbb in #1537
• Add versioned agent install specs by @xeophon in #1506
• Cache Hugging Face tokenizers in CI by @xeophon in #1542
• Make v1 max_turns a stop signal with request-boundary admission by @willccbb in #1547
• [codex] Add replay taskset and harness by @willccbb in #1536
• RLM: remove legacy rlm by @snimu in #1515
• Revert top-level v1 env config shorthand by @willccbb in #1548
• Document SWE taskset progress by @rasdani in #1530
• Use full checkout for uv CI jobs by @xeophon in #1553
• Merge harness version config and bump harnesses by @xeophon in #1544
• Decouple composable harnesses from harness packages by @xeophon in #1551
• Tighten loose verifier runtime types by @willccbb in #1549
• add dataset shuffle to vf-eval by @rasdani in #1556
• feat(v1/sandbox): canonical /vf/model bridge (sandbox sends Messages; host resolves client + tokenizes) by @hubert-marek in #1560
• Add QUEST search taskset by @rasdani in #1555
• add OpenSeeker search taskset by @rasdani in #1571
• Add REDSearcher search taskset by @rasdani in #1573
• Keep eval results append-only by @xeophon in #1564
• fix(v1): extend sandbox readiness wait by @xeophon in #1577
• Match REDSearcher released judge prompt by @rasdani in #1581
• Default OpenSeeker full dataset and use filter_fn row filters by @rasdani in #1579
• Add v1 tunnel labels by @parkerpettit9 in #1584
• feat: add QUEST open-ended scoring by @rasdani in #1580
• fix(v1): isolate sandbox client failures by @xeophon in #1622
• Add Terminal-Lego composable taskset by @rasdani in #1582
• build: add fallback-version for no-VCS editable builds by @JannikSt in #1650
• feat(v1): add Harness.transform_prompt hook by @hubert-marek in #1561
• Add SandboxDebugEnv for sandbox tasksets by @rasdani in #1652
• Add CLI-Gym Harbor taskset by @rasdani in #1665
• Allow reasoning-only model responses by @xeophon in #1767
• [codex] Preserve reasoning-only Responses output by @xeophon in #1774
• refactor: move built-in tasksets/harnesses into verifiers.v1 submodules by @mikasenghaas in #1826
• chore: squash adjacent standalone commits on nano-as-v1 for merge into main by @mikasenghaas in #1825
• interception server: raise client_max_size 24MB -> 48MB by @hubert-marek in #1833
• Serialize V1 MCP state directly to bytes by @xeophon in #1806
• Read Prime files through the download API by @xeophon in #1808
• Move V1 tunnel teardown off the event loop by @xeophon in #1812
• Optimize V1 eval resume memory by @xeophon in #1807
• Use one producer task per V1 SSE stream by @xeophon in #1817
• Avoid full-context routed-expert padding by @xeophon in #1794
• Stream and join V1 SSE responses once by @xeophon in #1793
• Scan only bridged attribution tail by @xeophon in #1805
• Reap fork children off the proxy loop by @xeophon in #1816
• Honor rollout concurrency for grouped V1 evals by @xeophon in #1815
• Skip empty base State synchronization by @xeophon in #1823
• fix: scaleswe-v1 scorer test ids via file (avoid exec command-length overflow) by @mikasenghaas in #1834
• Omit redundant provider failure tracebacks by @xeophon in #1835
• Cache repeated Harbor test archives by @xeophon in #1836
• Reduce V1 branch flattening overhead by @xeophon in #1837
• Avoid rehashing repeated image URLs by @xeophon in #1839
• Make the echo smoke prompt explicit by @xeophon in #1840
• Decode large env replies off the event loop by @xeophon in #1819
• Stream SSE responses incrementally by @xeophon in #1838
• fix: rich dashboard per-rollout timer includes setup by @mikasenghaas in #1846
• fix(runtimes): make _ENSURE_UV robust to old/stale uv in base images by @S1ro1 in #1848
• fix: swebench-verified ensures uv on PATH before scoring by @mikasenghaas in #1851
• chore: move real-benchmark v1 tasksets to research-environments by @mikasenghaas in #1853
• Rename built-in tasksets to harbor and textarena by @xeophon in #1855
• feat(v1/graph): per-token is_content attribution on MessageNode by @hallerite in #1843
• feat: expose task MCP tools to the rlm harness by @mikasenghaas in #1858
• chore: rlm harness exposes built-in skills, drop tool selector by @mikasenghaas in #1859
• Download Harbor datasets without the Harbor CLI by @xeophon in #1856
• feat: add bash_edit harness (bash chat loop + edit tool) by @mikasenghaas in #1860
• feat: validate rlm built-in skills client-side by @mikasenghaas in #1861
• feat(v1/trace): add Trace.to_record() for training-payload-stripped dumps by @hallerite in #1844
• fix: clear image ENTRYPOINT in ModalRuntime so keep-alive command runs by @mikasenghaas in #1863
• fix: sum eval dashboard tokens across all branches by @mikasenghaas in #1867
• feat: bracketed phase markers in the eval dashboard by @mikasenghaas in #1866
• feat: show usage and time totals under the eval progress bar by @mikasenghaas in #1869
• fix: record a streamed turn before forwarding its terminal event by @mikasenghaas in #1864
• feat(v1/trace): add Branch.input_len/output_len best-available token lengths by @hallerite in #1875
• feat: add a reusable LLM judge by @mikasenghaas in #1874
• Add Harbor optional dependency by @rasdani in #1877
• fix: stop EnvServerPool worker re-raising KeyboardInterrupt during shutdown by @mikasenghaas in #1879
• chore: make harness MCP support opt-in (SUPPORTS_MCP defaults False) by @mikasenghaas in #1881
• Use installed Harbor CLI for taskset downloads by @rasdani in #1882
• feat(v1): add reusable Lean theorem-proving taskset base (verifiers.v1.tasksets.lean) by @hallerite in #1883
• Thread chat template kwargs into renderer pools by @eligotts in #1876
• fix: make uv-script prep hermetic to image uv config (--no-config) by @mikasenghaas in #1891
• refactor: consolidate Trace/Branch length properties under num_ prefix by @mikasenghaas in #1894
• feat(v1): add Trace.last_reply and adopt it across v1 envs by @xeophon in #1897
• feat: forward_env on HarnessConfig to pass host env vars into the runtime by @mikasenghaas in #1901
• show user-set configs in eval by @snimu in #1899
• refactor(v1): rename harnesses — null (tool-less) + default (bash + edit) by @hallerite in #1904
• Expose RLM env vars as harness fields by @snimu in #1902
• feat(default): add an optional search tool by @hallerite in #1900
• feat: inject task/trace/runtime into Taskset.setup by name by @mikasenghaas in #1908
• chore: drop rlm_ prefix from rlm harness metrics by @mikasenghaas in #1911
• feat: list every rollout in the eval dashboard, queued ones as pending by @mikasenghaas in #1913
• feat: paginate the eval --rich dashboard with left/right arrow keys by @mikasenghaas in #1912
• Add v1 shared scoring helpers by @xeophon in #1907
• feat: remove tool server fork-per-rollout path from v1 by @mikasenghaas in #1917
• feat: pass [verifier.env] to the harbor verifier by @rasdani in #1921
• feat: pluggable config-level judges (vf.ReferenceJudge, vf.RubricJudge) by @mikasenghaas in #1918
• Forward provider feature headers for all dialects by @xeophon in #1930
• fix: default v1 judge model to openai/gpt-5.4-nano by @mikasenghaas in #1931
• fix: open the eval dashboard pager on the first page by @mikasenghaas in #1932
• Add native v1 debug CLI and validate modes by @rasdani in #1905
• chore(v1): rename RolloutContext to ModelContext by @hallerite in #1940
• feat: replay entrypoint (re-score saved traces offline) by @mikasenghaas in #1936
• fix v1 aux header forwarding by @xeophon in #1943
• feat(v1): resume dashboard counts the whole run, not just resumed rollouts by @hallerite in #1892
• feat: improvements to rubric judge by @mikasenghaas in #1937
• feat: default prime runtime idle timeout to 1h by @mikasenghaas in #1949
• v1 eval dashboard: wrap the output path instead of truncating it by @snimu in #1950
• v1 eval dashboard: wrap around when paging with the arrows by @snimu in #1951
• chore: remove dead sft/rl demo scripts by @xeophon in #1923
• chore: remove environments that depend on verifiers.envs.experimental by @xeophon in #1922
• chore: remove verifiers-rl package and all integration points by @xeophon in #1924
• fix: retry malformed/reset model responses to survive flaky interception tunnel by @samsja in #1953
• feat: push eval runs to the platform with uv run eval --push by @mikasenghaas in #1947
• feat: rename v1 output file results.jsonl to traces.jsonl by @mikasenghaas in #1956
• Fix dash-prefixed prompts in CLI harnesses by @xeophon in #1957
• docs: refresh docs structure, skills, and agents guidance by @xeophon in #1916
• feat: record runtime info on the trace by @mikasenghaas in #1959
• feat: stamp message nodes with a creation timestamp by @mikasenghaas in #1960
• fix: precise eval resuming — reload finished traces so a resumed run matches the uninterrupted one by @mikasenghaas in #1962
• feat(v1)!: move per-task behavior from Taskset onto Task by @hallerite in #1948

New Contributors

• @parkerpettit9 made their first contribution in #1584

Full Changelog: v0.1.14...v0.2.0