Announcements

As noted in the deprecation notice in ORT 1.9, InferenceSession now requires the providers parameters to be set when enabling Execution Providers other than default CPUExecutionProvider.
e.g. InferenceSession('model.onnx', providers=['CUDAExecutionProvider'])
Python 3.6 support removed for Mac builds. Since 3.6 is end-of-life in December 2021, it will no longer be supported from next release (ORT 1.11) onwards
Removed dependency on optional-lite
Removed experimental Featurizers code

General

Support for plug-in custom thread creation and join functions to enable usage of external threads
Optional type support from op set 15

Introduced indirect Convolution method for QLinearConv which has symmetrically quantized filter, i.e., filter type is int8 and zero point of filter is 0. The method leverages in-direct buffer instead of memcpy'ing the original data and doesn’t need to compute the sum of each pixel of output image for quantized Conv.
- X64: new kernels - including avx2, avxvnni, avx512 and avx 512 vnni, for general and depthwise quantized Conv.
- ARM64: new kernels for depthwise quantized Conv.
Tensor shape optimization to avoid allocating heap memory in most cases - #9542
Added transpose optimizer to push and cancel transpose ops, significantly improving perf for models requiring layout transformation

Python
- Following through on the deprecation notice in ORT 1.9, InferenceSession now requires the providers parameters to be set when enabling Execution Providers other than default CPUExecutionProvider.
  e.g. InferenceSession('model.onnx', providers=['CUDAExecutionProvider'])
C/C++
- New API to query CUDA stream to launch a custom kernel for scenarios where custom ops compiled into shared libraries need implicit synchronization with ORT CUDA kernels - #9141
- Updated Invalid -> OrtInvalidAllocator
- Updated every item in OrtCudnnConvAlgoSearch to a safer global name
WinML
- New APIs to create OrtValues from Windows platform specific ID3D12Resources by exposing DirectML Execution Provider specific APIs. These APIs allow DML to extend the C-API and provide EP specific extensions.
  - OrtSessionOptionsAppendExecutionProviderEx_DML
  - DmlCreateGPUAllocationFromD3DResource
  - DmlFreeGPUAllocation
  - DmlGetD3D12ResourceFromAllocation
- Bug fix: LearningModel::LoadFromFilePath in UWP apps

Added Mac M1 Universal2 build support for a single binary that runs natively on both Apple silicon and Intel-based Macs. These are included in the official Nuget packages. (build instructions)
Windows C API Symbols are now uploaded to Microsoft symbol server
Nuget package now supports ARM64 Linux C#
Python GPU package now includes both TensorRT and CUDA EPs. Note: EPs need to be explicitly registered to ensure the correct provider is used. e.g. InferenceSession('model.onnx', providers=['TensorrtExecutionProvider', 'CUDAExecutionProvider']). Please also ensure you have appropriate TensorRT dependencies and CUDA dependencies installed.

TensorRT EP
- Python GPU release packages now include support for TensorRT 8.0. Enable TensorrtExecutionProvider by explicitly setting providers parameter when creating an InferenceSession. e.g. InferenceSession('model.onnx', providers=['TensorrtExecutionProvider', 'CUDAExecutionProvider'])
- Published quantized BERT model example
OpenVINO EP
- Add support for OpenVINO 2021.4.x
- Auto Plugin support
- IO Buffer/Copy Avoidance Optimizations for GPU plugin
- Misc fixes
DNNL EP
- Add Softmaxgrad op
- Add Transpose, Reshape, Pow and LeakyRelu ops
- Add DynamicQuantizeLinear op
- Add squeeze/unsqueeze ops
DirectML EP
- Update DirectML.dll from 1.5.1 to 1.8.0
- Support full precision uint64/int64 for 48 operators
- Add 8D for 7 more existing operators
- Add DynamicQuantizeLinear op
- Accept ID3DResource's via C API

Added Xamarin support to the ORT C# Nuget packages
- Updated target frameworks in native package
- iOS and Android binaries now included in native package
ORT format models now have backwards compatibility guarantee

Support WebAssembly SIMD for qgemm kernel to accelerate the performance of quantized models
Upgraded existing WebGL kernels to the latest opset
Optimized bundle size to support various production scenarios, such as WebAssembly only or WebGL only