This release applies for both HDInsight 3.6 and HDInsight 4.0. HDInsight release is made available to all regions over several days. The release date here indicates the first region release date. If you don't see below changes, wait for the release being live in your region in several days.
New features
Spark 3.0 preview
HDInsight added Spark 3.0.0 support to HDInsight 4.0 as a Preview feature.
Kafka 2.4 preview
HDInsight added Kafka 2.4.1 support to HDInsight 4.0 as a Preview feature.
Moving to Azure virtual machine scale sets
HDInsight now uses Azure virtual machines to provision the cluster. The service is gradually migrating to Azure virtual machine scale sets. The entire process may take months. After your regions and subscriptions are migrated, newly created HDInsight clusters will run on virtual machine scale sets without customer actions. No breaking change is expected.
Eav4-series support
HDInsight added Eav4-series support in this release. Learn more about Dav4-series here. The series has been made available in below regions:
- AUSTRALIA EAST
- BRAZIL SOUTH
- CENTRAL US
- EAST ASIA
- EAST US
- JAPAN EAST
- SOUTHEAST ASIA
- UK SOUTH
- WEST EUROPE
- WEST US 2
Deprecation
No deprecation in this release.
Behavior changes
Default cluster version is changed to 4.0
The default version of HDInsight cluster is changed from 3.6 to 4.0. For more information about available versions, see available versions. Learn more about what is new in HDInsight 4.0.
Default cluster VM sizes are changed to Ev3-series
Default cluster VM sizes are changed from D-series to Ev3-series. This change applies to head nodes and worker nodes. To avoid this change impacting your tested workflows, specify the VM sizes that you want to use in the ARM template.
Network interface resource not visible for clusters running on Azure virtual machine scale sets
HDInsight is gradually migrating to Azure virtual machine scale sets. Network interfaces for virtual machines are no longer visible to customers for clusters that use Azure virtual machine scale sets.
Upcoming changes
The following changes will happen in upcoming releases.
OS version upgrade
HDInsight will be upgrading OS version from Ubuntu 16.04 to 18.04. The upgrade will complete before April 2021.
HDInsight 3.6 offers Basic support on July 2021
Starting July 2021, Microsoft will offer Basic support for certain HDInsight 3.6 cluster types. The Basic support plan will be available until 3 April 2022. You'll automatically be enrolled in Basic support starting July 2021. No action is required by you to opt in. See our documentation for which cluster types are included under Basic support.
We don't recommend building any new solutions on HDInsight 3.6, freeze changes on existing 3.6 environments.
We recommend that you migrate your clusters to HDInsight 4.0. Learn more about what's new in HDInsight 4.0.
Bug fixes
HDInsight continues to make cluster reliability and performance improvements.
Component version change
Added support for Spark 3.0.0 and Kafka 2.4.1 as Preview.
You can find the current component versions for HDInsight 4.0 and HDInsight 3.6 in this doc.