Summary

This is the fourth pre-release of the ADaptable Input Output System, ADIOS, version 2 (v2.2.0). The primary focus of this beta release is increased performance through I/O aggregation. This version is not intended for production quality purposes.

Features

Major additions from the previous v2.1.1 Beta release include process aggregation and multi-block I/O.

Aggregation

So far ADIOS 2.x wrote one output file per each writing process. This provides good performance on parallel file systems for up to the point where the file system starts to struggle to handle too many files at once. Aggregation here means that a subset of processes collect the data from all the processes and only they write files. We have implemented the aggregation algorithm known from ADIOS 1.x with some modifications to use less memory than in ADIOS 1.x. At the API level, the user only needs to set the number of “substreams” different from the number of MPI processes. Figure 1 shows an example of how 4 processes can write to a single file. This implementation saturates the MPI network bandwidth while writing to disk in an asynchronous manner. In addition, the new modular architecture of ADIOS2 allows for future experimentation of different aggregation methods to chase performance on upcoming exascale systems.

Multi-Block Writing

ADIOS 2.x now allows writing independent variable pieces to can be later retrieved as a single selection. This is better illustrated in Figure 2, in which each sub-block of data, in blue, is written independently as it becomes available. Whereas a read request can be made for a certain selection, thus pulling data for multiple sub-blocks. This pattern is largely used to support experimental and observational data in which independent I/ O. This feature is naturally integrated into the ADIOS2 API, in which each Variable Put/Get block call is inserted in a container.