PrimeIntellect-ai/verifiers v0.3.1

on GitHub

Highlights

• Model interception. Primitives to inspect, rewrite, or stop canonical vf.Request and vf.Response boundaries with @vf.intercept and @vf.stop for live monitoring of agent rollouts. (#2164, #2165, #2166, #2229, #2178).
• Persistent ACP sessions. Claude Code, Codex, Hermes, Kimi Code, OpenClaw, Pi, Pool, and RLM now use one live ACP session per rollout. Native conversation state remains intact across turns (#2291).
• Configurable task hooks. Config can plug in stop conditions, metrics, and rewards from Python modules or files. Config entries can override decorated hooks and their priority, weight, or metadata (#2309, #2323).
• Training-native episode and run artifacts. Episodes now carry task, group, run, policy, reference-logprob, and loss-weight data. Stable task keys, named run directories, exact config capture, and native Platform uploads make resume and training handoffs durable (#2278, #2357, #2358, #2409, #2429).
• Runtime and serving improvements. Eval uses elastic env serving by default and propagates cancellation to workers. Prime sandboxes default to VM isolation and support unbounded lifetimes. (#2356, #2362, #2396, #2401, #2412, #2413, #2422, #2428).

Breaking

Legacy and environment packages

• The v0 implementation moves under verifiers.legacy. Existing v0 imports remain aliased. The v1 bridge is removed, including vf.LegacyEnvConfig, EvalConfig.legacy, --legacy.id, init --v0, legacy serve routes, and row-indexed v1 resume. Use the vf-* entrypoints for v0 environments (#2303).
• Example environment IDs drop the -v1 suffix. For example, use gsm8k instead of gsm8k-v1. These packages now require Python 3.11 or later (#2303).

Runtimes and serving

• Prime VM is the default runtime. Eval and GEPA no longer default to subprocess. Validate and debug no longer default to Docker. Set the runtime explicitly to keep local execution, such as --env.agent.runtime.type subprocess for eval or --runtime.type docker for validate and debug (#2356).
• Prime sandboxes no longer have a 24-hour hard lifetime. VM sandboxes and containers without an idle timeout can run indefinitely. Use idle_timeout or explicit deletion for cleanup (#2412, #2413).
• Eval uses the env-server pool by default. The server boolean and --server flag are removed. Use --no-serve for in-process execution. run_eval now takes only the config, and run_eval_server is removed (#2422).
• rich is now an optional config block. Replace --rich false with --no-rich. Run logs now live under logs/attempt_<n>/eval.log; use the logs/latest symlink (#2422).

Run configs and records

• Run output uses output_dir / run.dir. The top-level uuid fields are removed. Use run.name, run.dir, and the process-created run.id. Resume is now a boolean flag on the saved config, not --resume <dir> (#2358).
• Saved configs use separate launch and resolved paths. Launch TOML lives at configs/<cli>.toml. Resolved JSON lives at configs/resolved/<cli>.json. Resume and replay can still read the earlier configs/<cli>.json layout (#2429).
• Episode records are the durable artifact. Episode.task is required. Run data moves from Trace.run to Episode.run. Bare-trace JSONL input, Trace.record_run, and Episode.of are removed (#2409).

Plugins and APIs

• V1 no longer installs Hub plugins automatically. Install each taskset, harness, judge, or env package before use. Plugin IDs must name installed packages; org/name[@version] IDs no longer resolve (#2390).
• ACP harnesses require live sessions. The ACP export and one-shot runner are removed. Use ACPConfig, ACPHarness, and rollout-scoped sessions (#2291).
• Custom dialects must implement interception methods. Dialect.parse_request now returns vf.Request. Dialects must implement request rewriting, response rewriting, and stream event generation. Rollout.open() can return False when a pre-open stop fires (#2164, #2165, #2166, #2229).
• Retry replay requires x-stainless-retry-count. Clients without this header resample repeated requests instead of replaying a recorded response (#2368).

Changelog

• Upload native v1 Episodes to Platform by @mrmoxon in #2278
• fix: clamp per-call input token accounting so it never goes negative by @mikasenghaas in #2297
• feat: make ServeConfig.address optional with no default by @mikasenghaas in #2301
• Fix plain HTTP CONNECT for Docker framework routes by @xeophon in #2298
• fix(v1): restore the unbounded model-call timeout by @mikasenghaas in #2304
• chore(deps): bump gitpython from 3.1.57 to 3.1.58 by @dependabot[bot] in #2305
• Use native model catalogs in agent harnesses by @xeophon in #2308
• feat: config-plugged functions (stops, metrics, rewards) by @mikasenghaas in #2309
• feat!: move v0 under verifiers.legacy (aliased), remove the v1 bridge, drop -v1 suffixes by @mikasenghaas in #2303
• fix: harden the verifiers.legacy alias finder by @mikasenghaas in #2311
• fix: revert native model catalogs in agent harnesses by @mikasenghaas in #2314
• feat(compact): one tool call per turn, notes-only carryover by @mikasenghaas in #2313
• feat: metadata-only overrides for config-plugged functions by @mikasenghaas in #2323
• fix: compact harness runs without a container by @mikasenghaas in #2329
• fix: keep host paths out of shared /tmp by @mikasenghaas in #2331
• feat: record the task data hash on the trace by @mikasenghaas in #2339
• chore: bump prime-sandboxes to 0.2.36 by @mikasenghaas in #2344
• fix(v1): isolate Bash commands from harness venv by @eigenpaul in #2251
• fix: pin MiniSWEAgent LiteLLM proxy dependency by @xeophon in #2336
• Run all ACP harnesses on persistent live sessions by @xeophon in #2291
• Fix runtime script and IPC paths by @xeophon in #2346
• Block provider network escapes in restricted runtimes by @xeophon in #2299
• Add interception example environments by @xeophon in #2178
• Add interception transport and replay by @xeophon in #2164
• Add typed model interception by @xeophon in #2165
• Stop at typed model boundaries by @xeophon in #2166
• Connect harness tool interception hooks by @xeophon in #2229
• feat: default runtime to prime vm sandbox by @mikasenghaas in #2356
• feat(v1): add stable task keys by @hallerite in #2357
• feat(v1): run configs + align resume logic with prl by @mikasenghaas in #2358
• Update coding harness and adapter versions by @xeophon in #2360
• Prevent --clean from deleting output roots by @xeophon in #2363
• feat: require an explicit retry signal to replay intercepted responses by @mikasenghaas in #2368
• Pin RLM harness to nano-rlm by @hallerite in #2373
• Revert nano-rlm harness pin, default version back to main by @samsja in #2374
• Make MCP tool tests naming-agnostic by @xeophon in #2378
• fix: use upstream ACP notification ordering by @hallerite in #2389
• chore: bump prime-sandboxes to 0.2.37 by @mikasenghaas in #2388
• feat!: remove hub auto-install of plugin packages by @mikasenghaas in #2390
• chore: bump prime-pydantic-config to >=0.4.3 by @mikasenghaas in #2391
• feat(v1): add DeepSeek Harness ACP integration by @xeophon in #2377
• Revert DeepSeek Harness ACP integration by @xeophon in #2394
• fix(mcp): support prebuilt runtime Python by @rasdani in #2395
• Support Harbor runtime environment and health checks by @xeophon in #2362
• fix(mcp): use runtime Python for readiness probe by @xeophon in #2398
• Retry interaction episodes in live tests by @xeophon in #2399
• Propagate network policy to provider capabilities by @xeophon in #2401
• fix: bump pypi-publish action to v1.14.2 by @mikasenghaas in #2402
• chore: bump workflow actions to latest versions by @mikasenghaas in #2403
• feat: propagate client-side cancellation to the env server by @mikasenghaas in #2396
• fix: accept empty ACP replies by @hallerite in #2400
• feat: default prime sandboxes to no lifetime timeout by @mikasenghaas in #2412
• feat: raise prime sandbox creation wait to ~6 minutes per phase by @mikasenghaas in #2414
• feat: extend the unbounded lifetime to container sandboxes by @mikasenghaas in #2413
• feat(v1): make episodes training-native by @hallerite in #2409
• chore: bump prime-sandboxes to 0.2.39 and pooled sandbox client by @mikasenghaas in #2410
• feat: run evals through the env server by default by @mikasenghaas in #2422
• fix(v1): strip hub org prefix before importing a plugin module by @JannikSt in #2423
• feat(v1): stamp renderer mm token type map onto traces by @mikasenghaas in #2424
• fix(v1): forward sampling parameters across dialects by @xeophon in #2425
• Centralize V1 egress policy decisions by @xeophon in #2428
• Simplify v1 CLI plumbing by @xeophon in #2427
• chore: bump renderers to 0.1.10 by @mikasenghaas in #2433
• feat: split run configs into launch TOML and resolved dir by @mikasenghaas in #2429
• Simplify V1 dialect implementations by @xeophon in #2426
• fix(v1): pin nano-rlm to 83ef01f by @hallerite in #2379

New Contributors

• @mrmoxon made their first contribution in #2278
• @eigenpaul made their first contribution in #2251

Full Changelog: v0.3.0...v0.3.1