Overview
The 8.6.0 release of ESMF introduces two major new features: spherical vector regridding and integrated accelerator device management. Both features are provided at an early stage of the implementation, expected to be fleshed out over the course of the next several releases. In addition to these new developments, the release includes a number of improvements of existing functionality. Highlights of the 8.6.0 release are outlined in the following paragraphs, followed by a comprehensive list of release notes.
Support for spherical vector regridding has been requested by a number of groups over the years, and a basic implementation of the feature is now available in ESMF 8.6.0. One of the advantages of mapping vectors through 3D Cartesian space, as opposed to regridding the individual components separately, is improved accuracy of the results, especially in the polar regions. The initial implementation of this feature included with 8.6.0 is limited to 2D tangential vectors (expressed in terms of east and north components) on a spherical geometry (e.g. an ESMF_Grid with ESMF_COORDSYS_SPH_DEG) and requires that the vector components are stored in a single ungridded Field dimension of size 2. We expect these and other restrictions to be relaxed in future releases and welcome feedback from the community to help prioritize.
Accelerator device solutions that are capable of managing multi-component applications are likely to become critical as coupled systems are run on hardware that provides most of the compute power in the form of GPUs. ESMF 8.6.0 provides novel development in this area, building on the experience gained from CPU resource management for multi-component ESMF applications based on the petList concept, and its recent extension to ESMF-managed threading. The user has now the option to associate an analogous list for accelerator devices (devList) with each component. ESMF implements the required bookkeeping, providing component context specific device information. Paired with standard accelerator programming paradigms, such as OpenACC or OpenMP, this allows component code to target the desired devices for offloading. The current implementation is considered a proof of concept, and refinements of the feature are expected in future releases.
Among the areas of the framework that received incremental improvements is ESMX, where it is now possible to modify ESMF run-time behavior via ESMF_RUNTIME_* options in the ESMX run-time configuration file for convenience. Small functional improvements also went into the Array and Field Create() methods that now allow creation from the slice of an existing object. Handling this situation on the ESMF level can significantly simplify user code.
Further progress was made for multi-tile I/O support for Fields and Arrays. It is now possible to perform I/O for multi-tile Arrays and Fields with layouts other than 1 DE per PET. Related to I/O, the "normalization" attribute in mapping files written by the ESMF regrid weight generation methods and applications was corrected to set the value “N/A” for anything other than the conservative regridding methods.
Incremental progress, bug fixes, portability and performance improvements were made in several other areas of the framework. This includes full support for the NVHPC compiler suite, tracing support for MPI calls on all platforms, resolving crashes observed with the Darshan and Cray Performance Analysis Tools, and finally addition of a public C API for the ESMF trace feature. Please see the release notes below for a detailed list of changes.
Release Notes
- The public Fortran API in this release is backward compatible with the previous release, ESMF 8.5.0. There were a few API changes, none of which require user code changes. The complete list of API changes is summarized in a table showing interface changes since ESMF 8.5.0. The table includes the rationale and impact for each change.
- No bit-for-bit changes were observed for this release compared to ESMF 8.5.0. This is based on test runs with the Intel compilers using options "-O2 -fp-model precise". However, changes were made to the implementation of the conservative weight calculation to remove a problem caused by the same calculation being optimized differently in two places in the code. Bit-for-bit changes compared to ESMF 8.5.0 from this change are possible when using regridding methods ESMF_REGRIDMETHOD_CONSERVE or ESMF_REGRIDMETHOD_CONSERVE_2ND and not using strict floating point compiler options.
- No changes that affect the status of existing regridding methods were made to the ESMF regrid weight generation methods and applications. However, the tables summarizing the ESMF regridding status were extended to cover the spherical vector regridding option added in this release.
- Keys to match all of the ESMF_RUNTIME_* environment variables were added to the App Options of the ESMX run configuration. This feature provides a convenient way to control the run-time behavior of the ESMX executable from its standard configuration file.
- Basic accelerator device management for multi-component applications was implemented. This feature allows the user to assign accelerator devices via the “devList” option to individual components. "devList” is available as a new argument to ESMF_GridCompCreate() and ESMF_CplCompCreate(), and is also accessible on the ESMX level as Component Label Option. Once specified, ESMF handles the device bookkeeping, and provides the user with context specific device information through the ESMF_VMGet() API. This feature can be leveraged in connection with popular accelerator programming paradigms like OpenACC, OpenMP, or standard language approaches to offload component code to the desired devices.
- Basic support for vector regridding was added to the ESMF_FieldRegridStore() method. The initial implementation of this feature is limited to 2D tangential vectors (expressed in terms of east and north components) on a spherical geometry (e.g. an ESMF_Grid with ESMF_COORDSYS_SPH_DEG) and requires that the vector components are stored in an ungridded Field dimension. The advantage of using this capability, which maps vectors through 3D Cartesian space, instead of regridding both components separately, is that it provides more accurate results, particularly in the polar regions.
- The "normalization" attribute in mapping files written by the ESMF regrid weight generation methods and applications was changed to now correctly set “N/A” for anything other than the conservative regridding methods. User code that accesses the “normalization” attribute in ESMF mapping files might need to be adjusted accordingly.
- A new entry point was added to the ESMF_FieldCreate() interface allowing creation from an existing Field object. The interface supports slicing with respect to trailing ungridded dimensions.
- The ESMF_ArrayCreate() entry point that allows creation from an existing Array object was extended to support slicing with respect to trailing undistributed dimensions.
- The Read and Write operations for multi-tile Arrays and Fields (e.g., for representing a cubed sphere grid as a six-tile grid) have been extended to permit I/O for Arrays / Fields with layouts other than 1 DE per PET. This change applies to the ESMF_ArrayRead(), ESMF_ArrayWrite(), ESMF_ArrayBundleRead(), ESMF_ArrayBundleWrite(), ESMF_FieldRead(), ESMF_FieldWrite(), ESMF_FieldBundleRead(), and ESMF_FieldBundleWrite() methods. The remaining limitations of the implementation are discussed in the Restrictions and Future Work section of the I/O Capability in the reference manual.
- A basic ESMF C API was added to provide access to the ESMF tracing and profiling capability from code written in C.
- Crashes that were observed with ESMF applications during MPI_Finalize() with the Darshan or Cray Performance Analysis Tool active have been resolved. (The issue was traced back to an interaction with the MPI Tools interface that used to be initialized during ESMF_Initialize(). ESMF no longer initializes the MPI Tools interface by default to avoid this problem.)
- The issues noted in previous releases with regard to using the ESMF tracing and profiling feature for MPI calls have been resolved. This capability is now fully functional on all the supported platforms.
- NVIDIA's NVHPC compiler suite is now fully supported by ESMF for both ESMF_OS=Linux and ESMF_OS=Unicos, and setting ESMF_COMPILER=nvhpc. This is particularly important for users looking at exploring ESMF accelerator device support on systems with NVIDIA GPUs.
Known Issues
- Using vector regridding (vectorRegrid=.true.) through the ESMF_FieldRegridStore() interface in combination with either conservative regridding method (regridmethod=ESMF_REGRIDMETHOD_CONSERVE or regridmethod=ESMF_REGRIDMETHOD_CONSERVE_2ND) currently results in an error.
- Attempting to write weight files from the ESMPy Regrid object when using filemode=FileMode.WITHAUX currently crashes.
Platform-specific issues:
- Compiling ESMF with GCC version 8.1.0 triggers an internal compiler error in ESMF_HConfig.F90 due to the use of allocatable character variables. Earlier and later versions of GCC do not have this issue.
- On Darwin, with version 15 of the clang C compiler, when building under Rosetta, it is sometimes necessary to add “-Wl,-ld_classic” to environment variables ESMF_CXXLINKOPTS, ESMF_CLINKOPTS, and ESMF_F90LINKOPTS to work around link errors. (For more details, see the related GitHub issue.)
Documentation
- ESMF Reference Manual for Fortran
- ESMF Reference Manual for C
- ESMF User Guide
- NUOPC Layer Reference
- Building a NUOPC Model
- ESMPy Doc