Azure/azure-sdk-for-python azure-ai-evaluation_1.18.12

on GitHub

1.18.12 (2026-11-09)

Bugs Fixed

• _ToolCallSuccessEvaluator sends a list of text messages to the judge model, rather than reporting it as not applicable, when its text looks like a tool transcript. Such text contains tool_call, tool_result, tool_response, function_call or function_result in any letter case, as tags such as <tool_call>, [TOOL_CALLS] and this evaluator's [TOOL_RESULT] do, Llama's <|python_tag|> or <function= tags or the gpt-oss <|call|> token, or has a line that starts with a label such as Action:, Observation:, Tool call: or Tool result:, as ReAct agents write, also after a bullet or a list marker such as 1), (2) or a). In 1.18.11 only text with [TOOL_CALL] or [TOOL_RESULT] was sent. Such a list is sent as its text, with each message's role and every tool definition, as a string is, because a Decision model can miss ReAct steps inside a list of messages. The text of a message that holds an MCP approval request is not counted, because it is left out of every judge input. Pass a transcript in any other text format as a string for the judge model to assess it.
• A Decision evaluator that the judge model rates as not applicable gives a readable reason, for example Not applicable: the judge model's applicability probability 0.2000 is below the 0.5000 cutoff, so the evaluator was skipped. No model rationale is available. It used to read Status completion probability 0.2000 is below 0.5000; evaluation skipped., which named an internal question.
• TaskAdherenceEvaluator with a Decision model reports a threshold of 1 when none is requested, the only score at which its binary result passes, as _TaskCompletionEvaluator does. It reported 0. A requested threshold is still reported as given and still does not decide whether a binary result passes. The evaluator's threshold attribute now also defaults to 1 for chat models; chat results are unchanged.
• A Decision reason names a question by the part of its key after the last /, so a namespaced key such as grade/confirm_details, which the hosted evaluation service uses for rubric dimensions, reads Confirm details probability 0.1645 is below the 0.7500 cutoff. It used to read Grade/confirm details probability ....
• A prompt template that fails to parse or render raises JinjaTemplateError, a subclass of PromptyException classified as a user error, instead of a generic PromptyException. Code that catches PromptyException still catches it. A template blocked by the sandbox still raises PromptyException.