0.126.0 (2026-11-18)
Before upgrading
- Derived metrics. Check string formulas in
derivedMetrics. Arithmetic, comparisons, ternaries and score names such as__countstill work. Function calls, property access, assignments and formulas longer than 1,024 characters now stop the evaluation before any test cases run. For example, replacemax(a, b)witha > b ? a : b. For more complex logic, use a function in a JavaScript config. Formula syntax · #12125 - Full-repository code scans. Install and run Promptfoo with Node.js 24 or later for full scans. Diff-only scans still work on Node.js 22.22 or later. A full scan can share eligible committed text from the compared revision with the hosted scanner. Set
diffsOnly: truein the code-scan configuration to share only the diff. #12185
Highlights
- Evaluate GPT-6.2 Astra or Sol with
openai:gpt-6.2-astraoropenai:gpt-6.2-sol. Both use the Responses API by default. This release also adds Claude Fable and Mythos 5.2 and Gemini 4 Argon and Hydrogen. #12474 · #12358 · #12416 - Use
google:decisions:gemini-decision-1to grade text, images, audio, video and PDFs. Gemini and Vertex also support file and live transcription. #12393 · #12151 · #12153 - Promptfoo restricts access to saved cloud credentials on POSIX systems and redacts Azure Storage connection strings in logs and exports. #12177 · #12130
Features
- providers: add Azure and Anthropic config schemas (#11747) (1a34bf6)
- providers: add Claude Fable and Mythos 5.2 (#12358) (5376cc9)
- providers: add Gemini Decisions API support (#12393) (c087d91)
- providers: refresh OpenAI model support on Bedrock (#12141) (059c749)
- providers: support Gemini 4 Argon and Hydrogen (#12416) (803ff25)
- providers: support Gemini file transcription through Interactions (#12151) (738e434)
- providers: support Gemini Live and Vertex file transcription (#12153) (7fda531)
- providers: support GPT Image 2.6 generation and edits (#12134) (7c90c8a)
- providers: support GPT-5.6 Cyber on Bedrock (#12137) (dc99c26)
- providers: support GPT-6.2 Astra and Sol (#12474) (0c123d8)
- providers: support Nova and Marengo text embeddings (#11562) (e48f9cc)
- providers: support Sol Ultrafast (#11547) (67651ef)
Bug Fixes
- assertions: apply weight 0 to throwing python and ruby file assertions (#12118) (9aff6d0)
- assertions: exclude tool calls when detecting refusals (#11957) (5ca33a6)
- assertions: record assertion failures when
file://orpackage:JavaScript assertions throw (#11939) (8399f76) - assertions: omit provider configuration from grader errors (#12190) (9d01dd4)
- assertions: preserve the receiver of exported grader methods (#12336) (fb61ad3)
- azure: isolate explicitly configured Foundry service principals (#12156) (2db2e1a)
- bedrock: return no output instead of throwing when a Nova message has no content (#11812) (8fd968a)
- cli: print the full evaluation summary when sharing is unavailable (#12332) (5e1a260)
- code-scan: upgrade the filesystem MCP server to 2.3.0 (#12185) (9e66910)
- config: preserve literal hashes in Excel workbook filenames (#12340) (6c06c8f)
- config: protect saved cloud credentials (#12177) (a3d02ed)
- db: recheck migrations after concurrent CLI startup (#12338) (0b4a1fb)
- deps: move the optional filesystem MCP server off the deprecated date version (#12397) (7b99385)
- deps: update anthropic packages (#11991) (204cf8f)
- deps: update anthropic packages (#12050) (44ef3dc)
- deps: update anthropic packages (#12450) (3d4ec90)
- deps: update Azure Identity to use patched MSAL Node (#12110) (923a1c5)
- deps: update dependency @anthropic-ai/sdk to v1 (#12091) (bfb96e2)
- deps: update dependency @anthropic-ai/sdk to v1.2.0 (#12102) (6b9ae7b)
- deps: update dependency @anthropic-ai/sdk to v1.3.0 (#12113) (06778a3)
- deps: update dependency @anthropic-ai/sdk to v1.4.0 (#12413) (af9ad5f)
- deps: update dependency @hono/node-server to v2.1.6 (#12401) (4f67101)
- deps: update dependency chokidar to v6 (#12047) (b3cea64)
- deps: update dependency glob to v14.0.3 (#12226) (ec6d640)
- deps: update dependency ip-address to v10.7.5 (#12123) (42f213f)
- deps: update dependency js-yaml to v5.5.0 (#11994) (9a212e7)
- deps: update dependency minimatch to v11.0.2 (#12218) (b76f5a1)
- deps: update dependency ora to v10 (#12389) (13b9214)
- deps: update the Promptfoo dependency in four examples to
^0.125.0(#12108) (eb106d8) - deps: update direct MSAL dependency to 7.1.0 (#11970) (b4c2404)
- deps: update gcp-metadata to v9.2.1 (#11985) (111a782)
- deps: update google-auth-library to v11.3.1 (#11990) (bfe9709)
- deps: update Inquirer prompt packages (#12189) (682c672)
- deps: update natural to v8.2.6 (#12157) (9dcb138)
- deps: update opentelemetry (#12452) (0e3ff58)
- deps: update the docs build serializer (#12188) (b652778)
- deps: upgrade Undici in the CLI and scan action (#12437) (f12d114)
- eval: end only CLI evals on SIGTERM, after provider drain and OTel flush (#11933) (b9433a0)
- eval: preserve pagination after rating updates (#12470) (930ff3c)
- eval: prevent negative measured provider latency (#12418) (be49264)
- eval: report measured provider latency in whole milliseconds (#12438) (5fac53e)
- evaluator: restrict derived metric formulas (#12125) (a98e6a0)
- examples: support Google ADK telemetry v2 (#11956) (22df9a5)
- fetch: match connection pools to transport settings (#12179) (9df5533)
- mcp: report the original provider test failure (#12290) (eefa2d4)
- openapi: keep the published spec and reference current (#8978) (599c225)
- prompts: append large prompt batches without argument spread (#12301) (f93731f)
- providers: enforce deadlines on buffered HTTP responses (#12014) (5f31095)
- providers: follow Gemini API model redirects for controls and cost (#12147) (a27e222)
- providers: guard empty image data in openai formatOutput (#10263) (90e6104)
- providers: handle publisher-qualified Vertex Llama model IDs (#12148) (7d67362)
- providers: honor AWS region settings on Bedrock (#12139) (43c16e9)
- providers: honor Azure embedding cache-bust context and report HTTP errors (#12094) (f746c63)
- providers: honor Groq prompt config overrides (#11735) (afbe489)
- providers: honor multi-region Vertex Flash Image locations and prices (#12152) (e2840e7)
- providers: honor Vertex regions and API versions (#12168) (cc6797a)
- providers: isolate credentialed Bedrock response caches (#12136) (ce9de90)
- providers: normalize Nova settings and correct Bedrock examples (#11566) (c41ecba)
- providers: normalize trailing Google AI Studio base URL slashes (#12400) (055eacd)
- providers: normalize trailing Ollama base URL slashes (#12211) (8b7c03d)
- providers: omit sampling params for Claude Haiku 5.5 on OpenAI chat path (#12166) (48e9d6c)
- providers: preserve Azure completion no-output semantics (#10145) (299d9a6)
- providers: preserve Claude synthetic trace spans (#12104) (97b3936)
- providers: preserve colons in fal image model IDs (#11058) (5256ad6)
- providers: preserve empty AI21 completions (#12048) (8c4b60b)
- providers: preserve empty HF completions (#12126) (be97226)
- providers: preserve empty Vertex Claude and Llama completions (#12394) (45f68f0)
- providers: preserve Mantle proxy pricing (#12233) (e27e710)
- providers: preserve tool history for Gemini and Anthropic (#11621) (e8c984d)
- providers: preserve trailing Live transcripts (#12181) (90d2c68)
- providers: preserve ungrouped Hugging Face token labels (#11744) (54eb1c3)
- providers: price GPT-6 Luna and GPT-6.1 Sol on Bedrock Converse (#12183) (414645f)
- providers: price GPT-6.1 Terra, GPT-6 Sol, and GPT-6 Astra on Bedrock Converse (#12374) (b168258)
- providers: refresh Bedrock Nova profiles and video selection (#12140) (67383bd)
- providers: refresh Veo examples and preserve JPEG input on Vertex (#12150) (94fe64a)
- providers: reject empty Hyperbolic image data (#12210) (db6a568)
- providers: replay video cache hits from one verified mapping (#12323) (4f20c1f)
- providers: scope Snowflake and Cloudera endpoints (#11900) (fbb4b9a)
- providers: send generationConfig sampling options to Vertex Llama models (#11916) (dcbb512)
- providers: stop logging raw headers in HTTP string-body warning (#12352) (d6f16a9)
- providers: stop offering retired vertex:gemini-2.5-pro in init flows (#12222) (cb2323e)
- providers: stop retrying Azure token-count errors as rate limits (#12023) (3bc1a46)
- providers: support current Bedrock Runtime models (#11560) (69f84c3)
- providers: support Nova 2.5 Sonic and final transcripts (#11565) (0d802f6)
- providers: surface Cohere embedding API errors (#12025) (424ee97)
- providers: surface Mistral embedding API errors (#12054) (df90acf)
- providers: type inline Google response schemas (#11170) (2be2248)
- providers: update OpenAI models and API compatibility (#12009) (e3e2bb8)
- providers: validate Gemini embedding options and vectors (#12149) (821a1f6)
- providers: validate missing Anthropic credentials (#12105) (0933bb0)
- redteam: isolate live config files for concurrent runs (#12457) (9a7187a)
- redteam: upgrade the CloudSwag example to Anthropic v2 (#11913) (6a5ee15)
- security: redact Azure Storage connection strings in logs and exports (#12130) (8252ba6)
- tracing: keep in-memory exports from creating an empty database (#12341) (6ccffcb)
- tracing: preserve OTLP receiver startup options (#12083) (8f6b847)
- util: avoid extra samples for negative counts (#11881) (1a19309)
- util: handle invalid duration values (#11878) (e8cef0e)
- webui: cancel overlapping video preview timers (#12324) (ac12006)
- webui: cancel stale launcher redirects (#12287) (0865734)
Performance Improvements
- ci: skip file probes for unrelated example paths (#12331) (60e0486)
- datasets: index prompts once per dataset (#12261) (b2d8676)
- datasets: read eval config instead of full results file (#11989) (687571b)
- db: reuse dataset hashes and append prompt history once (#12302) (580cf77)
- eval: advance grouped grading from the first pending entry (#12280) (a691b62)
- eval: filter resumed execution plans in one pass (#12306) (8b852d2)
- prompts: check filename suffixes without repeatedly splitting prompts (#12304) (aa5cb09)