github esmf-org/esmf ESMF_8_0_0
ESMF 8.0.0
on GitHub

latest releases: v8.6.1, NRL-b01, v8.6.1b04...
3 years ago

Overview

The ESMF 8.0.0 release concludes another phase of evolving and improving the library. The number of applications in which the library is used continues to grow. Requirements from these applications shaped and guided the developments included in this release.

The four typical ways of using ESMF have not changed: 1) to create high-performance, interoperable component-based modeling systems; 2) as a source of data communication, time management, metadata handling, and other libraries; 3) as a fast, parallel generator of interpolation weights from file for many different grids (see the ESMF_RegridWeightGen application website); and 4) as a Python grid remapping library (see the ESMPy website).

Highlights of 8.0.0 include improved component timing profiles and the introduction of a community based NUOPC Field Dictionary. Further extensions to the NUOPC layer include improved performance, and support for driving a NUOPC system from a higher level such as a data assimilation (DA) system. The ESMPy interface now offers in-memory weight access, and the ESMF regridding implementation was extended to include "creep-fill" extrapolation. Core ESMF data classes were extended: A C interface was added to ESMF_XGrid, the creation and use of ESMF_Mesh has been simplified, ESMF_FieldBundles can be created for packed data allocations, and a shared memory access capability was added to ESMF_Array.

More details of the highlighted items are provided in the following paragraphs for convenience.

The tracing capability that was introduced in the previous release now supports a simple mechanism to generate component timing profiles in text files. A single summary timing file can be generated at the end of a run that provides timing statistics across all the PETs. This provides a simple way to understand the relative cost (in terms of wall clock time) of each component in a coupled application.

Management of the NUOPC Field Dictionary has been made more flexible. The dictionary can now be loaded from file during run-time. A community version of the NUOPC Field Dictionary resides in a dedicated public repository and can evolve independently of the NUOPC Layer.

The NUOPC layer now supports situations where a NUOPC system is driven by a higher level driver (outside of NUOPC). One application of this feature is the ability to integrate NUOPC-based forecast systems (such as UFS) with DA systems that require their own driver layer (such as JEDI). Further the NUOPC run sequence was extended to support switching between different run sequence sections during execution. For example, this capability allows changing which components are active at different stages during a run. The overall performance of the NUOPC layer was improved by eliminating unnecessary synchronization, allowing greater opportunity for component concurrency.

The ESMF regridding system was extended in several areas. The "creep-fill" extrapolation method was added to allow the user to spread data from mapped destination points to neighboring unmapped destination points. Regridding weights can now be returned through the ESMPy interface in-memory to eliminating the need to go through netCDF files when accessing weights from Python. The ESMF_Regrid application has been updated to support GRIDSPEC Mosaic files, and regridding on different stagger locations. The ESMF_RegridCheck external demo was added to test the ESMF_Regrid application and the ESMF regridding system with a collection of test grids and data sets.

The ESMF_XGrid interface was wrapped with ESMC bindings, providing simplified access to the ESMF exchange grid implementation through native interfaces from model code written in the C programming language.

The use of the unstructured mesh class, ESMF_Mesh, has been simplified. ESMF_Mesh objects can now be created directly from a structured ESMF_Grid object, using a simple ESMF_MeshCreate() call. Further, a Mesh object can now be queried for its mask and area information making this information accessible to model code that depends on it.

The ESMF_FieldBundle class was extended to cover the case where multiple Fields are packed into a single data allocation. Fields can be interleaved along any dimension of the packed data allocation. The current implementation of the packed feature is limited to cases where the user provides the data allocation to ESMF. Communication calls are supported going from a packed source FieldBundle to a packed destination FieldBundle, however, both sides must provide the same number of Fields, and the order of Fields must be the same on both sides. Further, the NUOPC layer does not currently support exchanging data via FieldBundles, packed or unpacked. Future ESMF releases are planned to address these limitations.

The ESMF_Array class now allows sharing of data between PETs that execute on the same single system image (SSI). This feature can be used for shared memory data access between components that run on the same set of compute nodes (i.e. same SSIs), but run with different number of PETs on each node. This situation is typically encountered when components use different number of threads under each PET. Exchanging data by shared memory access is usually more efficient than having to transfer the data between PETs.

There are many other features and options added throughout ESMF, detailed in the release notes (see link below). Backward compatibility of the Fortran user interface with the ESMF 5.2.0r release series was preserved for methods that are labeled backward compatible in the Reference Manual; the majority of methods fall into this category.

Release Notes

  • This release is backward compatible with the last release, ESMF 7.1.0r, for all the interfaces that are marked as backward compatible in the Reference Manual. There were API changes to a few unmarked methods that may require minor modifications to user code that uses these methods. A number of new interfaces were added. The entire list of API changes is summarized in a table showing interface changes since ESMF_7_1_0r, including the rationale and impact for each change.
  • Some bit-for-bit changes are expected for this release compared to the last release, ESMF 7.1.0r. We observe the following impact with Intel compilers using "-O2 -fp-model precise":
    • Roundoff level differences in conservative regridding due to an improvement in an area calculation algorithm
    • Roundoff level differences in regridding when used on a Mesh created from a SCRIP format file that contains longitudes <=0 degrees. This change was due to removing a conversion for non-positive longitudes to improve consistency
    • Minor differences in 2nd order conservative regridding for cells that protrude outside their neighbors (e.g. a peninsula made up of a single cell) due to a bug fix in the weight calculation algorithm for that regridding method
  • Tables summarizing the ESMF regridding status have been updated. These include supported grids and capabilities of the offline and integrated regridding, and numerical results of some specific test cases.
  • Added an option to output component timing profiles in text format by setting the ESMF_RUNTIME_PROFILE environment variable. ESMF and NUOPC component phases are automatically instrumented and user-defined timed regions are also supported. Timing profiles can be written to the end of the ESMF PET log files, to separate per-PET text files, and/or to a single timing summary file. The summary file provides timing statistics across all the PETs. This provides a simple way to understand the relative cost (in terms of wall clock time) of each component in a coupled application.
  • The NUOPC Layer contained in this release has been improved in the following specific technical areas:
    • The NUOPC Field Dictionary can now be ingested from a community-based YAML file which resides in a dedicated public repository and can evolve independently of the NUOPC Layer. YAML Ain't Markup Language (YAML) is a human friendly, Unicode-based data serialization language for all programming languages.
    • A NUOPC_Driver can now be called from a higher level driver outside of NUOPC, going through an ESMF API. This is useful for systems that come with their own driver layer, but need to drive a NUOPC system (e.g. Data Assimilation).
    • The NUOPC run sequences now supports the "*" wildcard character in the "@" timestep syntax. This allows the timestep length to be set in code via driver specialization for run sequences ingested from a text file.
    • NUOPC now allows switching between different run sequence sections during execution. For example, this capability allows changing which components are active at different stages during a run. To enable this, the NUOPC_DriverIngestRunSequence() method now supports specification of a run duration, run sequence concatenation, and component run sequence elements outside loops.
    • ConnectionOptions is now an official NUOPC level Connector attribute. The attribute is read during RunSequence ingestion (for each Connector line), and appended by default to all of the CplList entries of the Connector.
  • The NUOPC_DriverAddComp() method now supports adding components with a SetVM() method, allowing the component to configure its own VM, e.g. for PET idling for PE reuse under threaded PETs.
  • Fields that are mirrored now arrive on the acceptor side with attributes set to reflect information about the provider side Field (TypeKind, GeomLoc, MinIndex, MaxIndex, ArbDimCount, GridToFieldMap, UngriddedLBound, UngriddedUBound). These attributes can then be used when creating Fields on the acceptor side.
    • The NUOPC Mesh transfer protocol was extended to correctly transfer either the node DistGrid, the element DistGrid, or both if present on the provider Mesh.
    • Completed the implementation of the sharing protocol. Whether to share the GeomObject and/or Field are now independent decisions, and all four possible combinations are supported. Sharing is available through component hierarchies and supports nested states (e.g. to use the Namespace and/or CplSet features of NUOPC).
    • The sharing protocol now checks whether the combination of provider and acceptor VMs allows Fields and/or GeomObjects to be shared between them and properly sets the share status attributes.
    • The generic Connector was optimized to also reuse Redist RouteHandles for Fields built on Meshes and LocStreams. Previously only the Redist for Fields built on Grids was optimized in this manner. Interleaving Redist and Regrid Field pairs is supported as before.
    • Implemented a communication optimized approach to propagating Field timestamps during Connector Run() to remove unnecessary synchronization between PETs. Only real data dependencies remain.
    • The NUOPC timestamp definition has been extended to include the ESMF calendar kind for more accurate timestamp validation between components.
    • Improved, standardized, and documented Verbosity attribute handling across all NUOPC generic component kinds. Preset Verbosity levels are available: "off", "low", "high", and "max".
    • The Diagnostic attribute option has been implemented across all NUOPC generic component kinds. This attribute allows the user to specify when Fields contained in the importState and exportState of a component are dumped to file when entering and exiting component methods (Initialize/Run/Finalize).
    • Several new NUOPC prototype (example) codes have been added:
      • AtmOcnCplSetProto demonstrates the use of the CplSet feature to support coupling multiple independent sets of Fields between components (e.g., multi-domain coupling).
      • AtmOcnLogNoneProto demonstrates the option to turn off ESMF PET log files completely.
      • AtmOcnMirrorFieldsProto demonstrates the NUOPC Field mirroring protocol.
      • CustomFieldDictionaryProto demonstrates the use of an external YAML file to populate the NUOPC Field Dictionary.
      • ExternalDriverAPIProto demonstrates how an external layer can drive a NUOPC driver component going through ESMF.
      • SingleModelOpenMPProto demonstrates the use of OpenMP threading inside a NUOPC model component and resource allocation through the generic SetVM method.
  • The following capabilities were added to the ESMF Python interface (ESMPy):
    • Added an in-memory weight generation option to the Regrid class, allowing re-use of weight vectors without writing them to netCDF. The weight arrays can be returned as NumPy objects or Python dictionary of weight vectors. This allows retrieval of the weights by source and destination key/value pairs.
  • The ESMF_Regrid application now supports additional options, and one option was removed:
    • Added --srcdatafile, --dstdatafile, and --tilefile_path options to support the GRIDSPEC Mosaic file format.
    • Added the --dst_loc option to support regridding on different stagger locations when the destination grid is in UGRID format and the regridding method is non-conservative.
    • Added the --check option to check the regridding results using a synthetic field, generated by an analytic function.
    • Removed the --user_areas option because none of the currently supported file formats provide user areas.
  • Added new features to the ESMF_FileRegrid() method and the ESMF_Regrid application:
    • Regridding of multiple variables
    • Multi-tile GRIDSPEC Mosaic file format with data stored in separate files, one per tile
    • 2nd order conservative regridding
    • Regridding to the corner stagger location if the regridding method is non-conservative and the destination file is in UGRID format
  • Added new features to the ESMF_RegridWeightGen application:
    • 1D network topology support in UGRID format
    • 2D Cartesian grid in CF Single Tile file format
    • Creep fill added as an extrapolation method
  • Added a new extrapolation method called "creep fill" to the entire regrid weight generation system. This capability allows the user to spread data from mapped destination points to neighboring unmapped destination points. This action can be repeated for a number of levels where at each level the data is spread from filled to neighboring unfilled points. The creep fill method is accessible by specifying the extrapMethod=ESMF_EXTRAPMETHOD_CREEP option in any of the ESMF_*RegridStore() methods or the --extrap_method creep option when using the ESMF_RegridWeightGen application).
  • Additional regridding methods are now supported when ESMF_Mesh is switched to be based on DOE's MOAB mesh library. The new supported regrid methods are: bilinear (when the source Field is not built on Grid), and nearest neighbor. All extrapolation methods, except for "creep fill" are supported.
  • RouteHandles can now be written to file via ESMF_RouteHandleWrite() and read back into memory (on the same number of PETs) via ESMF_RouteHandleCreate(). This provides an opportunity to lower the initialization cost e.g. for short production runs that repeatedly require regridding between the same grids.
  • The ESMF_GridCreateCubedSphere() and ESMF_GridCreateMosaic() methods now support irregular decompositions.
  • The ESMF_GridCreateCubedSphere() method can now apply the Schmidt transformation on the coordinates.
  • The ESMF_GridCreateMosaic() and the ESMF_GridCreate() method that reads a grid from file now support different coordinate typekinds.
  • The ESMF_GridCreate() method now automatically determines the correct file format if it was not explicitly specified as an argument.
  • The ESMF_GridCreate() method now supports grid creation from a GRIDSPEC Mosaic supergrid tile file.
  • The ESMF_LocStreamCreate() method now supports 1D network topology in UGRID format.
  • Added the capability to create an unstructured Mesh object from a structured Grid object. This allows components that internally work with a Mesh (e.g. a generic Mesh based mediator) to construct a Mesh from a transferred Grid.
  • The ESMF_MeshGet() method has been extended to allow the user to get mask and area information from an unstructured Mesh object.
  • Wrapped the ESMF_XGrid methods with ESMC bindings, to make them easily available to applications implemented in C.
  • Added a new ESMF_FieldBundleCreate() method that allows the creation of a packed FieldBundle. This is an initial capability with some limitations that will be addressed in a future release. The method takes a pre-allocated Fortran array pointer containing the memory of a set of interleaved fields. This is often how sets of fields are structured in model code and this capability allows ESMF to reference this memory directly. Interleaving along any dimension is supported. Packed FieldBundles support communication methods including redistribution, sparse matrix multiplication, and regridding. Currently the number of fields on source and destination must be the same, and permutations of fields are not supported. This means that the order of fields on source and destination must agree.
  • Implemented the ESMF_DECOMP_SYMMEDGEMAX option for cases where the number of elements is not evenly divisible by the number of decomposition elements. This option assigns the largest number of elements to the two edge DEs. It then progresses by assigning a descending number of elements to DEs as the center of the decomposition is approached from both sides. This produces a decomposition that is identical to the decomposition the FV3 model uses for this situation.
  • The ESMF_Array class now allows sharing of DEs between PETs that run on the same single system image (SSI). This feature can be used for shared memory data access between components running on different number of PETs (e.g. for threading) but are located on the same SSI.
  • The ESMF_Config class API has been extended. The ESMF_ConfigCreate() method now supports creating a Config object from a subsection of an existing Config object. This feature allows the consolidation of the content of multiple Config objects into a single configuration file.
  • The YAML-CPP parser has been included in the ESMF distribution. It is active by default, but can be turned off by setting ESMF_YAMLCPP=OFF.
  • Added support for C++11 standard. ESMF builds with the compiler default, but switches to C++11 if ESMF_YAMLCPP is enabled (default). The new ESMF_CXXSTD environment variable can be used to explicitly switch to specific C++ standards.
  • Simplified the use of CMake for projects using ESMF by providing a FindESMF.cmake file. This file is located under the new cmake subdirectory. It parses the esmf.mk file of an ESMF installation, exposing ESMF build variables as global variables accessible by other CMake modules.
  • The ESMF regression test suite can now be used to validate a pre-installed ESMF installation. This can be done by setting the ESMF_TESTESMFMKFILE environment variable to ON, and pointing ESMFMKFILE to the esmf.mk file of the ESMF installation to be validated.
  • Added an external demo ESMF_RegridCheck to test the ESMF_Regrid application with a set of test grid and data sets. In all the test cases, the input variables were constructed using an analytic function and the regridded destination variables were compared with that function to calculate the mean relative errors.

Known Bugs

  • FieldBundles don't currently enforce that every contained Field is built on the same Grid, Mesh, LocStream, or XGrid object, although the documentation says that this should be so.
  • When the ESMF regrid weight generation methods and applications are used with nearest destination to source interpolation method, the unmapped destination point detection does not work. Even if the option is set to return an error for unmapped destination points (the default) no error will be returned.
  • The ESMF regrid weight generation methods and applications do not currently work for source Fields created on Grids which contain a DE of width less than 2 elements. For conservative regridding the destination Field also has this restriction.
  • The ESMF regrid weight generation methods and applications do not currently work on Fields created on Grids with arbitrary distribution.
  • There is a race condition in the ESMF_FileRegrid() method and the ESMF_Regrid application when the destination grid is of GRIDSPEC Mosaic format and >=12 PETs are used. This issue leads to intermittent failures in the external_demos tests for the GRIDSPEC_1x1_time_to_C48_mosaic_bilinear case when run on the 16 PET configuration.
  • Applying the sparse matrix multiplication to cases where the local data allocation is above the 32-bit limit will fail with a memory allocation error. This affects all Regrid(), Redist(), Halo(), and SMM() calls.
  • The ESMF_GridCreate() interface that allows the user to create a copy of an existing Grid with a new distribution will give incorrect results when used on a Grid with 3 or more dimensions and whose coordinate arrays are less than the full dimension of the Grid (i.e. it contains factorized coordinates).
  • Using the ESMF_GridCreate1PeriDim() method to create a grid with a bipole connection on the lower side (typically referring to the southern hemisphere) resulted in no connection there.
  • The ESMF_XGrid construction can lead to degenerate cells for cases where the source and destination grids have edges that are almost the same. Often these cells don't produce weights and are benign, but when weights are produced can lead to low accuracy results when transferring data to/from the XGrid.
  • The ESMF_ArrayCreate() crashes when used with pinflag=ESMF_PIN_DE_TO_SSI or pinflag=ESMF_PIN_DE_TO_SSI_CONTIG from within a component. The crash is from inside MPI with "invalid communicator". The "pinflag" option works correctly from the application level, i.e. in the context of the global VM.
  • Querying the ESMF_DistGridGet() method for "de" or "tile" information for a "localDe" will return incorrect results, and/or crash.
    ESMF_AttributeWrite() has only been verified to work for ESMF standard Attribute packages. Non-standard Attribute packages may trigger a crash inside the ESMF_AttributeWrite() implementation.
  • For NetCDF installations that have the C and Fortran bindings installed in different locations, a NetCDF enabled build of ESMF does not correctly include the Fortran NetCDF library during linking.
  • When installing ESMF into a location that is shared with other libraries, it can happen that executing the ESMF install target fails with a "permission denied" error.
  • The Darwin.intelclang.default build configuration is broken.

Platform-specific bugs:

  • The GNU and Intel compilers require GCC>=4.8 for C++11 support (Intel uses the GCC headers). By default ESMF now uses the C++11 standard. If you run into build issues due to the C++11 dependency, you can either (1) make sure a GCC>=4.8 is loaded, or (2) set ESMF_YAMLCPP=OFF. In the latter case the YAML-dependent features in ESMF will not be available.
  • For GNU compilers GCC>=10.x, the default Fortran argument mismatch checking has become stricter. This will result in build failures. Setting environment variable ESMF_F90COMPILEOPTS="-fallow-argument-mismatch -fallow-invalid-boz", during the ESMF build, can be used as a work around for this issue.
  • On some systems with the PGI compiler, there is an issue with shared memory pointers between PETs on the same SSI. We see failures or crashes for Array tests that exercise this feature (ESMF_ArraySharedDeSSISTest.F90, ESMF_ArrayCreateGetUTest.F90) on the following platforms:
    • Hera/PGI-18.10.1
    • Gaea/PGI-16.5.0
    • Electra/PGI-17.1.0
    • Pleiades/PGI-17.1.0
    • Summitdev/PGI-19.7.0
      However, we do not observe these failures or crashes on:
    • Discover/PGI-14.1.0
    • Discover/PGI-17.7.0
  • On Summitdev/PGI-19.7.0 we see ESMF_XGridUTest.F90 unit test failures due to erroneously produce weights for source and destination grids that have edges that are almost the same.
  • On Discover/PGI-14.1.0 the ESMF_FieldRegridUTest.F90 and ESMF_FieldBundleCrGetUTest.F90 unit tests are failing.
  • Currently the ESMPy interface to retrieve regridding weights from Python is only supported under the GNU compiler. On all other compilers the method will flag an error.
  • On Darwin, with the Intel Fortran compiler, command line arguments cannot be accessed from ESMF applications when linked against the shared library version of libesmf. There is no issue when linked against the static libesmf.a version. Setting environment variable ESMF_SHARED_LIB_BUILD=OFF, during the ESMF build, can be used as a work around for this issue.
  • On some systems with the Cray compiler (CCE version 8.x), the ESMF library fails to build. The error can be prevented by setting the ESMF build environment variable ESMF_MOAB=OFF.
Check out latest releases or
releases around esmf-org/esmf ESMF_8_0_0

Don't miss a new esmf release

NewReleases is sending notifications on new releases.

Get notifications