Deprecation of Python 2

MXNet community voted to no longer support Python 2 in future releases of MXNet. Therefore, MXNet 1.6 release is going to be the last MXNet release to support Python 2.

New features

NumPy compatible interface and using TVM to generate operators

NumPy has long been established as the standard math library in Python, the most prevalent language for the deep learning community. With this library as the cornerstone, there are now the largest ecosystem and community for scientific computing. The popularity of NumPy comes from its flexibility and generality.

In #14253, the MXNet community reached consensus on moving towards a NumPy-compatible programing experience and committed to a major endeavor on providing NumPy compatible operators.

The primary goal of the projects below is to provide the equivalent usability and expressiveness of NumPy in MXNet to facilitate Deep Learning model development, which not only helps existing deep learning practitioners but also provides people in the existing NumPy community with a shortcut for getting started in Deep Learning. The efforts towards this goal would also help a secondary goal, which is to enable the existing NumPy ecosystem to utilize GPUs and accelerators to speed up large scale computation.

Infra to use tvm write op kernels (#15550)
fix boolean_mask for 0-size output (#15731)
fix tvm cmake (#15781)
Numpy-compatible Infra (#15581)
[MXNET-1206] Support NDArray indexing with None and Ellipsis (#13143)
numpy-compatible sum (#15810)
[Numpy] Numpy compatible slicing (#15798)
Numpy Tensordot and Dot Operator (#15820)
numpy linspace (#15852)
tvm infra for op attrs (#15854)
Port several np ops to master (#15867)
numpy-compatible split upstream (#15841)
Numpy-compatible concatenate upstream (#15894)
Numpy-compatible stack upstream (#15842)
[Numpy] Numpy behavior random.uniform() (#15858)
Tvm broadcast backward (#15938)
np elemwise unary ops upstream (#15831)
[Numpy] random.randint() implemented (#15956)
Refines NDArray indexing and adds numpy ndarray indexing [READY FOR REVIEW] (#15942)
Port ops from np branch (#16018)
numpy-compatible cumsum upstream (#15924)
NumPy-compatible infrastructure on Gluon (#16024)
[OP] Support range as advanced index for ndarrays (#16047)
Numpy compatible max min (#16046)
NumPy-compatible Mean, Std and Var (#16014)
Add fluent methods mean, std, var for ndarray (#16077)
numpy multinomial op (#15878)
add numpy operator remainder (#16080)
[Numpy] Random.choice implemented (#16089)
Fix sample.normal shape inference
Numpy add numpy op indices (#15837)
[Numpy] Numpy copysign (#15851)
numpy operator ravel, derive from reshape (#16016)
Add array_function
Improved error mesages
Fix np.choice
add exception check for numpy reshape (#16180)
[Numpy] Numpy behavior normal distribution (#16109)
fix multinomial bug on gpu (#16204)
[Numpy] Differentiable svd (#15795)
add epsilon to sum(pvalue) upperbound (#16211)
np compatible vstack (#15850)
Numpy add numpy op roll (#15902)
add numpy compatible trace (#16008)
add numpy op hanning, hamming, blackman (#15815)
[Numpy]flip (#15819)
numpy operator around (#16126)
numpy operator arctan2 (#15890)
numpy operator nonzero (#15838)
numpy operator hypot (#15901)
tvm numpy operator deg2rad && rad2deg (#16015)
numpy op unique
try to fix bug
fix memory bug and disable some test
fix according to review
Numpy operators: lcm, tril, identity and take (#16264)
[numpy] Cosmetic improvement on mxnet.numpy builtin op signature in documentation (#16305)
Disable Pylint false error in numpy_op_signature (#16370)
boolean_mask_assign operator for future boolean indexing (#16361)
Implements ldexp. (#15845)
Numpy Operators: Inner, Outer, vdot (#15846)
Numpy det and slogdet operators (#15861)
Fix random op signature
fix choice signature
add raise test for shape
Add boolean ndarray (#15940)
global numpy shape flag (#16335)
numpy-compatible histogram (#16266)
[Numpy] Numpy compatible dstack (#15871)
numpy eye op (#16132)
Numpy compatible vsplit; minor changes to split (#15983)
add numpy op logspace (#15825)
add numpy op bitwise_xor, hsplit, moveaxis, rot90 (#16257)
Fix optimizer bug for np attribute (#16494)
Tests of NumPy interoperability (#16469)
improve unary and binary operator handling and refactor tests (#16423)
[DOC] Fix numpy op doc (#16504)
[Numpy] More numpy dispatch tests (#16426)
[Numpy] einsum (#15911)
Add test pipeline for USE_TVM_OP=OFF on Unix (#16450)
Numpy dispatch test of ...... (#16422)
setup and concatenate, copy, expand_dims, expm1 (#16493)
add sum for boolean type in mainline (#16436)
[Numpy] SVD outputs tuple (#16530)
numpy op doc: max, min, prod (#16506)
add interface for rand
Fix numpy bugs (#16537)
pickler override for np ndarrays (#16561)
[numpy]op test in new pattern (#16556)
Enforce adding documentation for builtin numpy operators (#16575)
[Numpy] Support N_D(N>=3) batch_dot (#16586)
[Numpy] Loading numpy-incompatible NDArray in numpy-compatible mode (#16597)
Fix index overflow bug in einsum (#16589)
add npx reshape (#16640)
add type switch to weight tensor (#16543)
numpy doc enhancement (#16637)
Infra for tvm op runtime dispatch (#16100)
[NumPy][Operator] NumPy operator may_share_memory and shares_memory (#16533)
[Numpy] Numpy operator diff (#15906)
Miscellaneous fix for several numpy issues (#16664)
[Numpy] implement np.column_stack (#16594)
[numpy] add numpy operator : append (#16564)
Backport of #16711, #16737, #16408 to 1.6 branch (#16763)
Backport to 1.6 (#16773, #16781, #16783, #16716, #16699, #16728, #16769, #16792) (#16832)
[Backport][v1.6.x] Fix the wrong result of sum, mean, argmin, argmax when inputs contain inf or nan (#16884)
Backport of #16827, #16791 and #16888 to 1.6 branch (#16901)
port shape op to 1.6.x (#16912)
[Numpy] Fix imperative basic indexing in numpy (#16902) (#16919)
Backport #16895, #16922, #16878, #16979 and #16900 to 1.6 (#17029)

Graph optimizations

Pointwise fusion for GPU

DL models, besides compute intensive operations like convolutions and fully connected layers, feature a lot of simple pointwise (aka elementwise) operations (like elementwise addition etc.). Performance of those operations is fully memory bandwidth bound and so limit speedups from newer GPU hardware, which typically has high compute/memory bandwidth ratio. When multiple of such operations are chained one after another, it results in a series of unnecessary stores and loads as well as potential increased memory usage to store the intermediate results. Pointwise fusion helps in alleviating those problems by just-in-time generation of fused operators, which do not store intermediate results in memory, resulting in performance and memory usage improvements.

Pointwise fusion for GPU (#15167)
Backport #16798, #16836 and #16838 to 1.6 (#16874)
Add support for boolean inputs to FusedOp (#16796) (#16892)
Workaround problem with fusion in CUDA 9 (#17028) (#17035)

Eliminate common subexpressions

Eliminate common expressions (#15657)

Default MKLDNN Subgraph fusion

[MKLDNN] Enable subgraph backend mkldnn by default. (#15518)

New operators

[OP] Add a new arange_like operator to contrib (#15400)
PDF operators for each distribution for which we have a random sampler (plus also the PDF of the Dirichlet). Supports probabilities and log-probabilities, as well as gradients. (#14617)
Group Normalization (#14959)
Add RROIAlign (#16017)
Add fast implementation of LARS (#16122)
Round and sign straight-through-estimators C operators. (#16373)
New ops for RCNN + old ops improvements for RCNN (#16215)
Comparison ops implemented using mshadow (#16414)
Add mask target generator operator for Mask-RCNN (#16268)
Move MRCNNMaskTarget op to contrib (#16486)
Mxnet allclose (#14443)
Aggregated adamw update (#16398)
Make mrcnn_mask_target arg mask_size a 2d tuple (#16567)
Dgl ops 2 (#16416)
Lamb optimizer update (#16715)
[OP] changing data type of 't' to int in lamb_update_phase1 (#16903)
Multi Precision Lamb Update operator (#16885)
Interleaved MHA for CPU path (#17138) (#17211)

Feature improvements

Automatic Mixed Precision

[AMP] Move topk from FP16_FP32_FUNCS to FP32_FUNCS (#15342)
Conversion from FP32 model to Mixed Precision model (#15118)
Update fp16 docs: Block.cast is inplace (#15458)
FP16 Support for C Predict API (#15245)
Add AMP Conversion support for BucketingModule (#15528)

Gluon Fit API

Fixing build for gluon estimator test, including libtvm in pack libs (#16148)
[Estimator] handle composite metrics in estimator (#16676)
[Estimator] refactor estimator to allow overriding evaluate/fit of a batch (#16678)
[Estimator] refactor estimator and clarify docs (#16694)
[Gluon] Improve estimator usability and fix logging logic (#16810) (#16846)
Backport Gluon estimator changes to 1.6 (#17048)
fix parameter names in the estimator api (#17051) (#17162)

MKLDNN

Upgrade MKL-DNN submodule to v0.20 release (#15422)
Fix quantized concat when inputs are mixed int8 and uint8 (#15693)
[MKLDNN]Enhance Quantization APIs and Tutorial (#15448)
Add quantization support for GluonCV (#15754)
add int8 bn mkldnn implementation and test (#15664)
[Quantization]support exclude operators while quantization (#15910)
[MKLDNN]Support fullyconnected and element-wise ops fusion (#15950)
Disable test coverage for Clang MKLDNN (#15977)
update support MKLDNN BN conditions (#15870)
[MKLDNN] Fix out of bound access of req vector (#16000)
add uint8 bn mkldnn implementation (#16003)
Improve quantization flow (#15961)
[MKLDNN] fix uint8 batch norm memory misuse (#16034)
MKL-DNN RNN checks NDArray version (#16071)
Float64 fallback for mkldnn subgraph and rnn op (#15853)
Update MKL-DNN dependency (#16073)
Integrate MKL-DNN leakyrelu (#16075)
[MKLDNN] NDArray reorder in C API and deconv (#16265)
Fix mkldnn reshape (#16455)
[MKLDNN] Fix uint quantized fc when not fusing with requantize (#16523)
[MKLDNN]Fix reorder2default (#16602)
Upgrade MKL-DNN dependency to v1.0 (#16555)
Revert "[MKLDNN]Fix reorder2default (#16602)" (#16697)
[v1.6.x] Backport #16837 into v1.6.x (#16847)
Initial checkin (#16856) (#16872)

Large tensor support

[MXNET-1413] Adding Large Tensor support for sort operators (#15170)
Large Index Support for Slice (#15593)
Add large tensor support binary arithmetic (#15785)
Large tensor support for random ops (#15783)
Add Large Tensor Support for Sequence, NN Ops (#15807)
Add power, exponent, log ops large tensor support (#15794)
removing unnecessary int64 C apis that were added to support Large Tensors and Vectors (#15944)
creating ndarray directly using mxnet ndarray primitives to reduce memory footprint of tests for topk, sort and argsort (#15900)
Adding tests to verify support for Large Tensors in additional Ops along with new C_Apis supporting 64bit indexing (#15895)
Added tests to verify Large Vector Support for initial set of ops (#15943)
Added more tests for Large Indices (#15960)
Add Large tensor vector test cases (#15941)
Test large vector mean operator and fix a few bugs (#16079)
Reducing memory footprint of one_hot for Large Array Testing (#16136)
removing MXNDArrayLoadFromBuffer64 and MXNDArrayLoad64 (#16203)
Fix large array tests (#16328)
added more tests to verify support for large vector (#16477)
added support for large tensors for Dropout operator and tests to verify support for more operators (#16409)
adding large tensor support for add_n and tests for more ops (#16476)
adding large tensor support for pad operator (#15126)
Added large tensor support and test for gather_nd (#16371)
Large Vector tests for DGL Ops Part 2 (#16497)
Showing proper error message when an attempt is made to create large tensor but MXNet is not built with it (#16570)

TensorRT integration

enable TensorRT integration with cpp api (#15335)
Add unit tests for TensorRT integration and fix some bugs (#15399)

Higher order gradient support

[MXNET-978] Higher order gradient for sigmoid (#15288)
[MXNET-978] Higher Order Gradient Support reciprocal, abs. (#15413)
[MXNET-978] Add higher order gradient support tan, tanh (#15253)
[MXNET-978] Higher Order Gradient Support arctan, arctanh, radians. (#15531)
[MXNET-978] Higher Order Gradient Support sqrt, cbrt. (#15474)
[MXNET-978] Higher Order Gradient Support clip, dropout. (#15746)
[MXNET-978] Higher Order Gradient Support sinh, cosh. (#15412)
[MXNET-978] n-th order gradient test support. (#15611)
[MXNET-978] Fully connected, higher order grad (#14779)
[MXNET-978] Higher Order Gradient Support arcsinh, arccosh. (#15530)

Operator improvements

broadcast axis is alias to broadcast axes; doc fix (#15546)
Utility to help developers debug operators: Tensor Inspector (#15490)
Softmax with length (#15169)
in-place reshape ops (#14053)
Add missing default axis value to symbol.squeeze op (#15707)
Add matrix determinant operator in linalg (#15007)
Add fp16 support for topk (#15560)
[MXNET-1399] multiclass-mcc metric enhancements (#14874)
new raise mode for nd.take and fix backward for wrap mode (#15887)

Profiler

Fixing duplication in operator profiling (#15240)
Custom Operator Profiling Enhancement (#15210)
[Opperf] Make module/namespace of the operator parameterized (#15226)
Opperf: Support Python<3.6 (#15487)
Add transpose_conv, sorting and searching operator benchmarks to Opperf (#15475)
Deprecate USE_PROFILER flag (#15595)
Update profiler.md (#15477)
[Opperf] Add array rearrange operators to opperf (#15606)
[OpPerf] PDF Random ops fix (#15661)
[Opperf] Add optimizer update operator benchmarks to opperf (#15522)
fix broadcast op param (#15714)
[OpPerf] Profiler flag for Python, Cpp (#15881)
[Opperf] Filter out deprecated ops (#15541)
[OpPerf] Handle positional arguments (#15761)
[OpPerf] Take care of 4d param (#15736)
Add Median,p50,p99 to python profiler (#15953)
adding "total" (total time) to profiler aggregate stats sorting criteria (#16055)

ONNX import/export

Correct ONNX documentation (#15914)
[MXNET-895] ONNX import/export: TopK (#13627)

Runtime discovery of features

Making Features as a singleton for improved caching (#15835)

Bug fixes

[bug] fix higher grad log (#15120)
Showing proper error when csr array is not 2D in shape. (#15242)
add 'asnumpy' dtype option to check_symbolic_backward (#15186)
point fix the vector declaration in MultiBoxDetection (#15300)
Temporarily Commenting out Flaky Test (#15436)
Fix memory leak in NaiveEngine (#15405)
fix nightly CI failure (#15452)
Small typo fixes in batch_norm-inl.h (#15527)
Bypass cuda/cudnn checks if no driver. (#15551)
Julia path patch (#15561)
Fix AMP Tutorial failures (#15526)
Fix warnings in CLang: (#15270)
Fix dumps for Constant initializer (#15150)
fix normalize mean error bug (#15539)
[fix] print self in warning. (#15614)
[MXNET-1411] solve pylint error issue#14851 (#15113)
[Flaky test] Skip test_operator_gpu.test_convolution_independent_gradients (#15631)
Fix subgraph with custom_op (#15671)
Fix USE_BLAS == openblas check (#15691)
update previous flaky naive engine test (#15651)
make TransposeShape infer shape form both sides (#15713)
Skip Flaky Test (#15722)
Revert "Dynamic Library Loading Support" (#15755)
Fix flaky test test_global_metric (#15756)
Fix PR #15489 (Dynamic Library Loading Support) (#15760)
Refactor LibraryInitializer so it's thread safe. Fixes random sporadical concurrency crashes. (#15762)
Fix backward_clip num inputs and type of clip params (#15688)
fixing problem with existing Singleton Caching (#15868)
Allow operators with multiple outputs in get_atomic_symbol (#15740)
Fix ConcatType backward type inference (#15829)
Add disable attr to subgraph property (#15926)
Re-enable flaky test_prelu (#15777)
declare explicitly the tblob default assign operator and copy constructor (#15937)
Discard needless test cases in test_convolution_independent_gradients (#15939)
fix naive engine for multi-threaded inference (#15574)
Fix get_rows_per_block (#15979)
Fix a memory misalignment in topk operator (#15948)
Decouple dtype from shape for Random multinomial (#15980)
Fix dtype inference in arange_like operator (#15930)
Disable laop_6 (#15976)
Fix flaky clojure profile test (#16058)
fix test_pick test time is too long (#16066)
[fix] Support nullop in transpose (#15865)
fix flaky test (#16074)
fix some test files test time is too long (#16067)
Fix gradient tensor mutate in {adam/ftrl/rmprop/rmspropalex}_update. (#15768)
Fix unary operator ceil/floor/trunc when data type is integer (#14251)
Fix failing tests (#16117)
Fixes NAG optimizer #15543 (#16053)
avoid test relu at the origin due to discontinuous gradient (#16133)
Fix remaining errors reported by D2L (#16157)
use 1E-4 in groupnorm test(#16169)
Sequence last fix (#16156)
fixing test for model compatibility checker (#16159)
assert_allclose -> rtol=1e-10 (#16198)
[MEMORY] retry GPU memory allocation if fragmented (#16194)
improve dataloader signals and messages (#16114)
Update ndarray.py (#16205)
fix flaky test (#16191)
Solve #14116, #15143 (#15144)
[MXNET-1422] Fix wrong results of min([inf, inf]) and max([-inf,-inf]) (#16226)
Fix inconsistent interpolation method values (#16212)
set fixed seed for profiler (#16155)
Fix MXNDArrayGetData (#16289)
fix atol for test_preloaded_multi_sgd (#16356)
Fix windows flakiness (#16415)
cuDNN non-persistant bidirectional RNN dgrad sync fix (#16391)
[BUGFIX] Minor type issues in Squeeze (#16448)
Fix Nightly Tests for Binaries (#16451)
Fix dtype bug (#16467)
Fix flakey pylint CI failures (#16462)
Load NDArray only to GPU if GPU is present (#16432)
Bug fix for the input of same axes of the swapaxes operator (#16513)
Fix learning rate scheduler being unexpectedly overwritten by optimizer's default value (#16487)
disable tests (#16536)
fix pylint in CI (#16540)
image crop gpu (#16464)
Build dmlc-core with old thread_local implementation (#16526)
fix doc for topk (#16571)
RNNOp to call cudaEventCreate lazily (#16584)
add encoding to the stub files for potential utf8 char in doc strings (#16580)
Surpress subgraph log in CI (#16607)
Fix dequantize memory corruption (#16606)
Fix for wrong reqs set after switching from training to inference (#16553)
Disables test_bulking_operator_gpu due to flakiness (#16611)
Imagenet inference to nightly fix (#16599)
Move some subgraph verbose to MXNET_SUBGRAPH_VERBOSE=2 (#16622)
RNNOp only call cuda/cudnn if GPU ctx is requested (#16632)
fix bad encode (#16641)
Disable float16 test (#16643)
Fix GetMKLDNNData for delay alloc (#16618)
Move ops which don't support FP16 dtype to FP32 list (#16668)
no such method => modified function args (#16610)
fix cuDNN RNN dtype_with_fallback_ bug (#16671)
Add check if scipy is imported in sparse.py (#16574)
Added launch bounds to the reduce kernels (#16397)
fix install dir (#16690)
fix binary dependencies in CD and nightly (#16693)
Fix SliceChannel Type inference (#16748) (#16797)
fix flakiness of test_np_mixed_precision_binary_funcs (#16873)
Fix test_gluon.py:test_sync_batchnorm when number of GPUS > 4 (#16835)
Omp fork numthreads fix 1.6 (#17000)
[BUGFIX] Fix race condition in kvstore.pushpull (#17007) (#17052)
Backport #17002, #17068 and #17114 to 1.6 branch (#17137)
Backport 3rdparty/openmp fixes (#17193)
fix norm sparse fallback (#17149)

Front end API

Expose get_all_registered_operators and get_operator_arguments in the… (#15364)
Add magic method abs to NDArray and Symbol. (#15680)
Dynamic Library Loading Support (#15489)
[MXNET-1294] Add KVSTORE PushPull API (#15559)

Gluon

[Dataset] Add take, filter, sample API to dataset (#16078)
Add register_op_hook for gluon (#15839)
[Dataset] add shard API (#16175)
Add list_ctx to ParameterDict (#16185)
[Gluon] Support None argument in HybridBlock (#16280)
Aggregated zero grad (#16446)
try to fix block (#16465)
[Gluon] Don't serialize shared parameters twice (#16582)
Initializer.eq (#16680)

Symbol

Add symbol api for randn and fix shape issue for randn ndarray and symbol api (#15772)
Graph Partition API (#15886)

Language Bindings

Python