New Features:
- Operators optimized for Engine support:
- Where*
- Cast*
- IntegerMatMul*
- QLinearMatMul*
- Gather (for scalar indices)
*optimized only for AVX-512 support
- Flag created to disable any batch size overrides, setting the environment variable "NM_DISABLE_BATCH_OVERRIDE=1".
- Warnings display when emulating quantized operations on machines without VNNI instructions.
- Support added for Python 3.9.
- Support added for ONNX versions 1.8 - 1.10.
Changes:
- Performance improvements made for sparse quantized transformer models.
- Documentation updates made for examples/ultralytics-yolo to include YOLOv5.
Resolved Issues:
- A crash could result with an uninitialized memory read. A check is now in place before trying to access it.
- Engine output_shape functions corrected on multi-socket systems when the output dimensions are not statically known.
Known Issues:
- BERT models with quantized embeds currently segfault on AVX2 machines. Workaround is to run on a VNNI-compatible machine.