github PrimeIntellect-ai/verifiers v0.4.1
on GitHub

3 hours ago

Highlights

  • Smaller core, one eval path. verifiers now holds environments, harnesses, runtimes, and scoring. Full evaluations, serving, and platform uploads move to prime-rl. vf-run is a minimal in-process runner for quick runs (#3047, #2610, #3052, #3056, #3046, #3054).
  • Native Prime VMs for services. Task services and Compose services run as native Prime VMs on the rollout LAN. Tool servers get their own Prime runtimes. Harbor Compose tasks run on native VMs, and each service gets its own disk size (#2910, #2918, #2999, #2876, #3018, #3020).
  • Bring your own runtime. A custom runtime plugin implements any provider. Prime runtimes expose ports and mount volumes, and Docker runtimes support bind mounts (#2862, #2914, #2916).
  • One network policy config. vf.NetworkPolicyConfig sets the policy for tasks (TaskData.network), scoring (ScoringConfig.network), and runtimes (Runtime.set_network) (#2902).
  • More scoring control. Rubric choices can map answers to scores. Traces can be dropped from training with drop. ScoringConfig.skip_on_truncated and skip_on_timeout skip scoring, and AgentConfig.max_turn_tokens caps each turn. Task.rescore scores a finished trace again offline (#2935, #2960, #2949, #2899).
  • Faster training data paths. Per-token node streams and record streams are NumPy arrays. Training arrays stay on the host and never stream. Episodes encode to JSON one node at a time, so the encoder does not block the event loop. Traces carry sampler probability heads, engine weight-version spans, and their run (#2961, #2980, #2934, #3042, #2804).
  • New and updated harnesses. Add the OpenCode 2 ACP harness and the DeepSeek minimal ACP harness. Each harness declares its dialects on its config. Prime Agent uses the Rust 0.10 release, and the bundled harnesses are bumped to their latest versions (#2871, #2860, #2888, #2887, #3021–#3031, #3044).

Breaking

  • The eval CLI is reduced to vf-run. vf-eval, vf-replay, vf-init, the Rich dashboard, and eval platform push are removed. Use vf-run for quick runs and prime-rl's uv run eval for full evaluations and uploads. To score a saved trace again, call Task.rescore(trace). Drop --rich, --no-rich, --push, --no-push, push, and run.attach from configs; setup_logging no longer takes console (#2610, #3052, #2843, #3056, #3046).
  • Runtime providers other than Prime, Docker, and subprocess are removed. Migrate modal, e2b, podman, and apptainer to prime or to a custom runtime plugin. Runtime.run_program is removed; use Runtime.run. Runtime.open_process is now Runtime.exec. Backend modules move to verifiers.v1.runtimes.backends.* (#2862).
  • UnscoredError is removed, and nan no longer means "unscored". Check trace.dropped instead, and replace nan choices with a drop list on the criterion. Trainers must skip dropped traces (#2960).

Notes

  • The env server moves to prime-rl. Import EnvServer, EnvClient, EnvServerPool, EpisodeAssembly, and set_episode_hook from prime_rl.env. Import ServeConfig and the pool configs from prime_rl.configs.env_server. verifiers no longer installs pyzmq or msgpack (#3054).
  • Streamed deltas carry no training arrays. Keep the arrays on the server with an episode hook and fetch them from there (#2934).

Changelog

New Contributors

Full Changelog: v0.4.0...v0.4.1

Check out latest releases or
releases around PrimeIntellect-ai/verifiers v0.4.1

Don't miss a new verifiers release

NewReleases is sending notifications on new releases.

Get notifications