Azure/azure-sdk-for-python azure-ai-evaluation_1.18.1

on GitHub

1.18.1 (2026-07-09)

Bugs Fixed

• Enabled azure_ai_search, azure_fabric, and sharepoint_grounding tool
  calls for ToolCallSuccessEvaluator and ToolOutputUtilizationEvaluator.
  These tools were previously rejected because their tool_result payloads
  are structured (dict / list of dicts) and the internal
  [TOOL_RESULT] {result} formatter rendered them with str(), producing
  Python repr output (single quotes, 'role': 'user') that the LLM judges
  could not reliably ground on. The shared formatter now JSON-encodes
  non-string payloads via a new _stringify_tool_result helper
  (ensure_ascii=False to preserve customer locale data), and the shared
  ConversationValidator.UNSUPPORTED_TOOLS list (inherited by
  ToolDefinitionsValidator) is narrowed to allow these three tools.
  GroundednessEvaluator now uses a new GroundednessConversationValidator
  subclass that keeps the wider rejection list, matching the corresponding
  behavior in azureml-assets where Groundedness was intentionally not part
  of the enablement (a follow-up will land a context-extractor helper so
  Groundedness can also accept these tools). The remaining restricted tools
  (bing_grounding, bing_custom_search, web_search,
  browser_automation, code_interpreter_call, computer_call,
  openapi_call) continue to be rejected. ToolCallAccuracyEvaluator and
  ToolInputAccuracyEvaluator are unaffected — they do not render tool
  results into the judge prompt and already opt out of the unsupported-tool
  check.
• Fixed OpenAPI tool-call validation in tool evaluators (e.g. ToolCallAccuracyEvaluator). OpenAPI
  tool definitions are expanded into their nested functions when present, so tool calls referencing a
  nested function name validate correctly, while OpenAPI tool definitions without nested functions are
  kept as is so tool calls referencing the top-level tool name continue to validate.

Other Changes

• Conversation/tool message preprocessing now normalizes openapi_call / openapi_call_output content
  items to tool_call / tool_result (previously only function_call / function_call_output were
  normalized), so evaluators correctly handle OpenAPI-tool agent transcripts.
• Evaluators no longer log raw customer payloads in fallback/debug paths. reformat_agent_response,
  reformat_conversation_history, reformat_tool_definitions, the tool-call-success reformat helpers,
  and Groundedness context extraction now emit structural summaries only (via _log_safe_summary),
  never raw query/response/tool payloads.