PrimeIntellect-ai/verifiers v0.4.0

on GitHub

Highlights

• V1 throughout. The legacy v0 implementation and compatibility aliases are removed. V1 provides the task, environment, harness, runtime, and evaluation interfaces (#2480, #2496).

• Interactive agent control. Interactions support steering and queued user messages, including delivery at turn boundaries and wake-up of idle interactions. ACP semantic edges capture agent relationships, and nano-RLM training runs through ACP (#2717, #2449, #2439).

• More runtime options. E2B, Podman, and Apptainer join the runtime providers. Prime sandboxes can start from filesystem checkpoints, and task-declared agent users apply to tools, scripts, and file I/O (#2294, #2528, #2763, #2748, #2753).

• Broader Harbor support. Compose tasks run on local Docker, Prime, and Modal, with solver/verifier image overrides, MCP declarations, and configurable artifact transfer budgets. Harbor 0.24.0 support includes explicit reward selection and reward-file precedence (#2665, #2646, #2546, #2567, #2808).

• Streaming evaluation and training. Episodes stream turn by turn through env serving and into Platform runs. Traces carry training logprob/entropy annotations, sampling masks, and serialized token counts; rejected compaction attempts are excluded from training (#2512, #2415, #2569, #2481, #2460, #2651, #2521).

• Harness and judge improvements. Add the Prime Agent ACP harness, Bash and RLM context compaction, runtime skill sources, a decision-model judge, and isolated deterministic verification (#2440, #2454, #2459, #2556, #2767, #2486).

• More control over evaluation. Shared task selection supports filtering, shuffling, skipping, and limits. Resume planning matches task content and occurrence. Client timeouts and ordered retry rules make failure handling configurable (#2695, #2760, #2728, #2729).

• Stable renderers 0.1.12. Require the stable release with template-continuation parsing fixes, including GLM-4.5-Air reasoning leaking into visible content, and reference-driven end-to-end coverage across renderer families (#2834).

Breaking

APIs and built-ins

• The legacy v0 stack is removed. verifiers.legacy and its historical top-level aliases are no longer available. Use import verifiers.v1 as vf and the v1 interfaces (#2480).
• The CLI entrypoints are vf-*. Use vf-eval, vf-validate, vf-debug, vf-replay, vf-init, and vf-gepa; these now target v1. The legacy CLI implementation and Prime CLI init shim are removed (#2480, #2517, #2638).
• The Pool harness and built-in Lean taskset are removed (#2601, #2656).

Configuration and defaults

• Bash context compaction is enabled by default. Configure compaction.summarize_at_tokens for a fixed threshold, or disable compaction explicitly (#2787).
• Rollouts have no default four-hour timeout. Configure the agent's timeout.rollout when a time limit is required. Stage timeout and error stop reasons identify the failing boundary (#2697, #2693, #2694).
• Retry policies use ordered rules with per-rule budgets. The first matching rule determines the retry budget; exhausted rules do not fall through (#2729).
• Harbor multi-step tasks are rejected. The adapter targets Harbor 0.24.0 and validates reward outputs, including explicit selection of a scalar reward (#2808, #2724).
• Judges default to openai/gpt-6-luna. Pin a model explicitly for reproducible comparisons (#2823).

Changelog

• feat: add verifiers release skill by @mikasenghaas in #2435
• feat(v1): type ACP prompt results by @hallerite in #2438
• feat(v1): integrate nano-RLM training over ACP by @hallerite in #2439
• fix(v1): use typed ACP turn in RLM harness by @hallerite in #2442
• feat(v1): add Prime Agent ACP harness by @hallerite in #2440
• fix(v1): use typed ACP turn in Prime Agent harness by @hallerite in #2443
• fix(v1): allow concurrent borrowed runtime environments by @parkerpettit in #2418
• fix(v1): pin Prime Agent v0.8.1 by @sethkarten in #2445
• fix(v1): retain Terminus reasoning history by @parkerpettit in #2455
• feat: relay overlong prompt errors to the harness by @mikasenghaas in #2453
• fix: end the null harness cleanly on context overflow by @mikasenghaas in #2456
• fix(v1): clear handled model call errors by @xeophon in #2457
• fix: batch Prime background job polling by @DamianB-BitFlipper in #2462
• [fix] [v1]: reject dupe artifact sources in #2225
• Report unchecked gold validation by @xeophon in #2466
• Fix cache salt forwarding in train client by @eligotts in #2463
• fix(v1): prefer final judge choices over draft boxes by @xeophon in #2465
• Adopt MCP 2026-07-28 by @xeophon in #2474
• Clean up empty NeMo Gym extra by @xeophon in #2476
• chore: bump renderers to 0.1.11; drop legacy RendererClient by @hallerite in #2478
• Consolidate v1 MCP client utilities by @xeophon in #2467
• fix: require aiohttp 3.14 or newer by @mikasenghaas in #2479
• feat!: remove the legacy (v0) stack by @hallerite in #2480
• fix(v1): reconcile concurrent graph commits by @hallerite in #2477
• feat: add optional ACP semantic edges by @hallerite in #2449
• feat: tool truncation + auto-compaction to bash harness by @mikasenghaas in #2454
• feat: consolidate shared harness code in harnesses.utils by @mikasenghaas in #2482
• docs: remove stale legacy references by @xeophon in #2484
• feat: add context compaction to the rlm harness by @mikasenghaas in #2459
• chore(deps): bump nltk from 3.10.0 to 3.10.3 by @dependabot[bot] in #2489
• fix(v1): enforce artifact transport boundaries by @xeophon in #2464
• fix(v1): collect isolated judge artifacts in owned rollouts by @xeophon in #2468
• feat: trainer trace annotations for logprobs + entropy by @mikasenghaas in #2481
• fix: avoid unused MCP harness imports by @mikasenghaas in #2490
• feat: sampling mask pass-through by @mikasenghaas in #2460
• chore: trim and refresh v1 dependencies by @mikasenghaas in #2491
• feat(v1): bump rlm pin to nano-rlm main, expose guardrail and role-append knobs by @hallerite in #2492
• feat(v1): rlm compaction knob is passthrough, with explicit enabled/max_compactions by @hallerite in #2493
• bump prime-agent to v0.9.1 by @sethkarten in #2502
• feat(v1): record pluggable judge requests by @hallerite in #2497
• fix(v1): resolve RLM concurrency with depth by @xeophon in #2507
• Support OpenClaw 2.0 in the v1 harness by @xeophon in #2485
• chore(v1): remove dead code and v0 leftovers by @hallerite in #2496
• feat(v1): round per-token floats in JSON records by @mikasenghaas in #2495
• chore: declare h11, httpx2 and uvicorn as direct dependencies by @hallerite in #2503
• Add isolated deterministic verifier environment by @xeophon in #2486
• refactor(v1): one install-under-lock helper for the harnesses by @hallerite in #2506
• fix: cli inits a v1 env by @ilijalichkovski in #2517
• fix(v1): enable Prime Agent cpython kernel prebootstrap for network-isolated tasks by @sethkarten in #2515
• Keep the Modal sandbox handle when a create is interrupted by @nicolaslara in #2406
• feat: exclude rejected compaction attempts from training by @hallerite in #2521
• fix(v1): drain EnvClient cancel tasks by removal so close() cannot spin by @willccbb in #2518
• test: remove tests duplicated by surviving ones by @willccbb in #2523
• ci: test on Python 3.14 by @willccbb in #2522
• fix(v1): hide reasoning from agentic judges by @samsja in #2527
• Avoid redundant Codex home directory command by @xeophon in #2530
• perf(v1): skip unused interception parsing and hashing by @xeophon in #2533
• perf(v1): skip unused response conversion work by @xeophon in #2534
• Read bounded files as binary in Subprocess and Docker by @xeophon in #2532
• Avoid redundant directory RPCs in Prime uploads by @xeophon in #2539
• Launch Prime Agent without uploading a wrapper by @xeophon in #2537
• Batch artifact existence checks by @xeophon in #2538
• perf(v1): reuse completed dashboard statistics by @xeophon in #2529
• Reuse MCP sandbox installations across colocated servers by @xeophon in #2531
• Reuse chat programs and rollout MCP clients by @xeophon in #2535
• Preserve GPU types in Harbor verifier environments by @xeophon in #2548
• Remove legacy Harbor author metadata fallback by @xeophon in #2553
• Fix bounded file reads on Prime VMs by @xeophon in #2557
• Fix env-server serialization errors and missing replies by @xeophon in #2558
• Preserve dependencies in Harbor verifier images by @xeophon in #2554
• Revert "Simplify Prime background job polling" by @xeophon in #2563
• prime-runs by @kcoopermiller in #2415
• feat(v1): run.attach streams the eval into a run the platform already created by @kcoopermiller in #2569
• fix(v1): report platform run errors by @xeophon in #2577
• feat(v1): expose RLM tool selection and execution controls by @hallerite in #2487
• Fix Modal process stdin writes by @arafatkatze in #2471
• Honor explicit runtime workdir overrides by @xeophon in #2559
• Wire Harbor MCP declarations into task toolsets by @xeophon in #2546
• fix(runtimes): harness setup reuses an installed uv by @faresobeid in #2575
• Support framework aware network policies in the Modal runtime by @hrdkbhatnagar in #2579
• Propagate Bash compaction failures by @samsja in #2584
• Simplify Modal egress domain-list construction by @xeophon in #2587
• Fix Modal SDK minimum and simplify sandbox creation by @xeophon in #2586
• Preserve provider filtering when Modal networking is disabled by @xeophon in #2585
• Stream Bash/edit eval requests through long reasoning turns by @samsja in #2566
• fix(v1): replay buffered training stream retries by @xeophon in #2588
• fix(v1): surface Codex terminal failures for rollout retries by @xeophon in #2594
• Upload all skill files concurrently by @xeophon in #2540
• Remove Pool CLI harness and associated tests by @xeophon in #2601
• fix(rlm harness): keep the rlm state directory outside the task working tree by @hallerite in #2602
• Support environment resolved HTTP MCP headers in ACP harnesses by @hrdkbhatnagar in #2589
• chore: keep ci smoke evals off the platform by @mikasenghaas in #2603
• fix(v1): include tools in branch identity by @eligotts in #2514
• fix: stabilize Pi tool identity across resumes by @mikasenghaas in #2606
• fix: preserve streamed reasoning details by @mikasenghaas in #2604
• fix(dashboard): abbreviate long override values in the eval header by @hallerite in #2607
• fix(rlm harness): stop a budget-capped session as a truncation, not agent_completed by @hallerite in #2608
• fix: expose Pi MCP tools with explicit argument schemas by @hallerite in #2609
• feat(rlm harness): default edit skill, 10M token budget, uncapped sub-agent calls; 4 h default rollout timeout by @hallerite in #2611
• feat: stream served episodes turn by turn by @mikasenghaas in #2512
• fix(v1): recover the routed-experts row dropped at each turn boundary by @garrett361 in #2616
• fix(serve): stream repaired routing rows by @xeophon in #2618
• chore: require prime-runs 0.1.3 by @kcoopermiller in #2622
• Show eval upload failures while runs are active by @xeophon in #2599
• Validate requested GPU types in Docker runtime by @xeophon in #2545
• Add shared Podman and Apptainer runtimes by @xeophon in #2528
• Support runtime skill sources through harness skills by @xeophon in #2556
• feat: make Prime sandboxes VM-only by @mikasenghaas in #2626
• chore(judge): default model to openai/gpt-5.6-luna by @eliebak in #2629
• fix(runtime): preserve callback HTTP response framing by @xeophon in #2627
• fix(harness): install runtime skills without Bash by @xeophon in #2628
• Expose Pi developer role compatibility override for custom endpoints by @hrdkbhatnagar in #2590
• fix(v1): deduplicate shared prefixes in total token limits by @xeophon in #2630
• fix(v1): end the Bash chat program cleanly on context overflow when it cannot compact by @parkerpettit in #2633
• Update v1 harness releases and adapters by @xeophon in #2598
• Make Harbor artifact transfer budgets configurable by @xeophon in #2567
• chore: drop the prime-cli init shim by @mikasenghaas in #2638
• Support sandboxed MCP servers from wheel installs by @xeophon in #2639
• Fix blocklist mediation and report blocked URLs by @xeophon in #2637
• feat(rlm harness): expose nano-rlm's delegation_prompt policy flag by @hallerite in #2642
• feat: serialize trace token counts by @mikasenghaas in #2651
• fix: stream live token counts by @mikasenghaas in #2652
• fix(responses): include logprobs on framed output_text events by @hallerite in #2653
• Fix tool catalog recording and namespaced call identity by @xeophon in #2597
• Gate tool calls at the response boundary and pin their results by @xeophon in #2510
• Simplify network policy mediation across API dialects by @xeophon in #2647
• chore: remove the built-in Lean taskset by @hallerite in #2656
• fix: run-scoped creation limiter that waits instead of failing by @mikasenghaas in #2657
• Support per-task Harbor solver and verifier image overrides by @xeophon in #2646
• fix(harbor): grade separate-verifier tasks with the task's own class by @hallerite in #2668
• fix: omit tools from tool-free chat harness calls by @mikasenghaas in #2669
• fix(interception): serve every streamed turn buffered, with dialect keepalives by @hallerite in #2650
• fix(prime): re-read job output the SDK failed to fetch instead of returning it empty by @hallerite in #2670
• chore: bump nano-rlm to latest main by @hallerite in #2672
• fix(harness): retry interrupted completion streams by @S1ro1 in #2654
• refactor(harness): use tenacity for stream retries by @hallerite in #2679
• refactor(v1): share agent configuration and trace infrastructure by @eligotts in #2592
• Run Harbor Compose tasks on local Docker, Prime, and Modal by @xeophon in #2665
• fix(harness): use certifi roots for Bash/Null model client TLS by @hallerite in #2614
• fix(rlm): bump nano-rlm for certifi TLS roots by @hallerite in #2688
• fix: unblock the live e2e suite on main by @mikasenghaas in #2692
• feat: shared env and taskset configs by @mikasenghaas in #2690
• feat: stop timed-out stages as _timeout by @mikasenghaas in #2693
• feat: stop an errored trace as _error by @mikasenghaas in #2694
• fix(runtime): isolate uv bootstrap from task offline settings by @xeophon in #2699
• fix: preserve I/O errors after setup deadline by @xeophon in #2701
• chore: drop the 4h default rollout timeout by @mikasenghaas in #2697
• fix(node): bootstrap curl with restricted apt repositories by @xeophon in #2702
• Keep multimodal content inline in v1 rollouts by @eligotts in #2417
• Preserve runtime working directory while restoring artifacts by @hallerite in #2705
• feat: add taskset selection views and select config by @mikasenghaas in #2695
• fix(harness): classify sandbox model transport failures as InterceptionError by @hallerite in #2707
• fix(rlm): pin merged nano-rlm commit by @xeophon in #2714
• fix: distinguish Docker and Podman wildcard domain blocks by @xeophon in #2711
• fix: honor native web-tool domain filter semantics by @xeophon in #2712
• Preserve confined links when restoring artifacts by @hallerite in #2710
• feat: replace merge_env_defaults with a generic merge_defaults by @mikasenghaas in #2709
• chore: fast-forward nano-rlm to latest main by @hallerite in #2716
• fix(validate): add a noop check that scores the untouched task by @rasdani in #2718
• fix(harbor): include task env in Compose interpolation by @xeophon in #2687
• fix(serve): guard EnvServerPool frontend against malformed frames by @hallerite in #2719
• fix(harbor): surface invalid verifier rewards as errors by @xeophon in #2724
• feat: configurable client timeouts by @mikasenghaas in #2728
• feat(retries): ordered error matching rules with per-rule budgets by @hallerite in #2729
• fix(responses): preserve inline reasoning under network restrictions by @xeophon in #2736
• fix: accept loops in ACP semantic edges; bump nano-rlm to c38e20b by @hallerite in #2744
• feat(v1): add E2B runtime provider by @mattteufel-e2b in #2294
• feat: run the agent as a task-declared user by @hallerite in #2748
• fix(artifacts): drop FIFOs and device nodes from collected archives by @hallerite in #2755
• fix(v1): reject sparse artifacts during filtering by @xeophon in #2759
• feat: support steering and queued user messages on interactions by @hallerite in #2717
• feat(judges): add decision model judge by @xeophon in #2767
• feat(prime): start a sandbox from a filesystem checkpoint by @faresobeid in #2763
• fix(harness): guard Task.cancelling() for Python 3.10 harness programs by @rasdani in #2686
• improvement(interception): surface tunnel drops and cut-off request bodies in run logs by @kcoopermiller in #2681
• refactor(eval): plan resumed rollouts by task content and occurrence by @xeophon in #2760
• fix(client): resolve environment headers and Prime team auth per endpoint by @xeophon in #2766
• feat: make compaction default in the bash harness by @mikasenghaas in #2787
• feat(v1): describe runtime network policy in agent prompts by @xeophon in #2769
• fix: reopen egress before shared-box Harbor grading by @mikasenghaas in #2793
• chore: require prime-runs 0.1.4 by @kcoopermiller in #2805
• Align Harbor with 0.24.0 and reject multi-step tasks by @hallerite in #2806
• Honor Harbor separate-verifier image and resource resolution by @hallerite in #2807
• Honor Harbor reward precedence and explicit scalar selection by @hallerite in #2808
• Speed up CI with parallel jobs, setup steps, and wiki caching by @xeophon in #2813
• Restart a prime tunnel's frpc when the tunnel goes down by @parkerpettit in #2798
• chore(deps): bump multidict from 6.7.1 to 6.9.1 by @dependabot[bot] in #2785
• improvement(interception): stop watching a prime tunnel whose registration is gone by @kcoopermiller in #2820
• refactor(sampling): normalize and resolve shared sampling parameters by @xeophon in #2762
• feat: default judges to gpt-6-luna by @mikasenghaas in #2823
• fix(runtimes): upgrade uv when script commands are unsupported by @hallerite in #2824
• fix: run the agent user's tools, scripts, and file I/O as that user by @hallerite in #2753
• build(deps): bump renderers to 0.1.12 by @hallerite in #2834

New Contributors

• @sethkarten made their first contribution in #2445
• @ilijalichkovski made their first contribution in #2517
• @nicolaslara made their first contribution in #2406
• @arafatkatze made their first contribution in #2471
• @hrdkbhatnagar made their first contribution in #2579
• @garrett361 made their first contribution in #2616
• @eliebak made their first contribution in #2629
• @mattteufel-e2b made their first contribution in #2294

Full Changelog: v0.3.1...v0.4.0