What's new in 2.10?
This is a major release with new features and lots of bug fixes. The main new feature is the new Python API.
Python
Before, ADIOS had two separate APIs for Python. The low-level ("Full") API was written with Pybind11 and directly mimicked the C++ API. The high-level API was another, smaller, and more pythonesque API that allowed for easier scripting with Python. The main problems with these two were that they were independent, and that the high-level API was not complete. Once a developer needed a feature only available in the full API, they had to start from scratch writing a script with the full API.
In 2.10, there is officially one Python API, written in Python, which in turn uses the old Pybind11 classes. The new API combines the high-level features of the old high-level API and the full feature set of the low-level bindings.
Note:
Old scripts that used the full API can still run without almost any modification, just change the import line from import adios2 to import adios2.bindings as adios2
Old scripts that used the high-level API must be modified to make them work with the new API, see Transition from old API to new API: Transition from old API to new API
See Python API
New/updated notable features
- BP5 is supported on Windows now
- SST and DataMan staging engines are GPU-Aware now
- SYCL support added for Intel GPUs (besides CUDA and HIP for NVidia and AMD GPUs)
- the SST/libfabric data transport now works on Frontier (besides the MPI data transport)
New experimental features
- Campaign management: https://adios2.readthedocs.io/en/latest/advanced/campaign_management.html
What's Changed
- ci: migrate to almalinux, cuda base image by @vicentebolea in #3484
- ci: update kokkos to 3.7 by @vicentebolea in #3490
- codeowners: added code owners file by @vicentebolea in #3496
- Hdf5 vol support will require latest hdf5 1.14 by @guj in #3516
- Bump numpy from 1.21.6 to 1.22.0 in /docs by @dependabot in #3535
- Bump cryptography from 38.0.2 to 39.0.1 in /docs by @dependabot in #3534
- Release 29 by @dmitry-ganyushin in #3545
- Enable Linux Emulation build (Power8, IBM XL) by @vicentebolea in #3537
- Setting code owners for the GPU files by @anagainaru in #3561
- Release 29 by @vicentebolea in #3573
- Allowing Put/Get to receive Kokkos Views with HIP backend by @anagainaru in #3571
- Prototype DAOS engine by @eisenhauer in #3576
- CODEOWNERS: update codeowner file by @vicentebolea in #3581
- shellcheck: add shellcheck ci check by @vicentebolea in #3582
- Build scripts for ADIOS2 with CUDA and Kokkos backend by @anagainaru in #3579
- Disable emu tests by @vicentebolea in #3583
- cmake: set correct soname by @vicentebolea in #3585
- TestBPChangingShapeWithinStep without and with various compressors by @pnorbert in #3577
- add release files by @vicentebolea in #3570
- Heat Transfer Example: MPI Datatype by @ax3l in #3593
- test,install: paramaterize adios2_DIR by @vicentebolea in #3586
- ci,ascent: enable ascent builds by @vicentebolea in #3610
- ci: use nvcc_wrapper in adiosKokkos by @vicentebolea in #3623
- Adding SYCL support to the Kokkos backend by @anagainaru in #3621
- Bump requests from 2.28.1 to 2.31.0 in /docs by @dependabot in #3628
- Fix flaky MSMPI installation in windows builds by @vicentebolea in #3627
- docs: update whatsnew by @vicentebolea in #3634
- ci,olcf,crusher: enable Crusher CI by @vicentebolea in #3620
- ci,crusher: minor tweaks by @vicentebolea in #3635
- Ascent,CI: do not report sync script build by @vicentebolea in #3637
- replaced uint with size_t by @guj in #3640
- crusher,ci: set unique env per pipeline by @vicentebolea in #3644
- Update documentation for the GPU-aware functionality by @anagainaru in #3639
- work around with include <unistd.h> on windows by @guj in #3652
- Bug fix for Kokkos applications without GPU backend by @anagainaru in #3654
- An effort to make profiler more user friendly by @guj in #3660
- Bump cryptography from 39.0.1 to 41.0.0 in /docs by @dependabot in #3650
- ci: remove power8 emulated builds by @vicentebolea in #3667
- ci,ascent: disable install tests by @vicentebolea in #3673
- Tweaks for BP5 on windows. by @eisenhauer in #3682
- Fix memory leak when there are Joined Arrays in streaming mode by @eisenhauer in #3684
- Fix for the issue #3646. by @dmitry-ganyushin in #3687
- cmake,evpath: namespace modules install dir by @vicentebolea in #3689
- Ci rework gcc builds by @scottwittenburg in #3631
- added gitattributes file by @vicentebolea in #3698
- Update files with new EOL by @vicentebolea in #3699
- build(deps): bump cryptography from 41.0.0 to 41.0.2 in /docs by @dependabot in #3697
- ci,mpich: disable Sz|Zfp flaky tests by @vicentebolea in #3702
- build(deps): bump pygments from 2.14.0 to 2.15.0 in /docs by @dependabot in #3704
- ci,cmake: cleanup scripts subdirectory by @vicentebolea in #3625
- cmake: moved logic to enable h5vol to detectoptions.cmake by @vicentebolea in #3701
- build(deps): bump certifi from 2022.12.7 to 2023.7.22 in /docs by @dependabot in #3712
- ci: increase windows timeout defsync and DataWrite by @vicentebolea in #3726
- format,ci: update clang-format to clang16 by @vicentebolea in #3722
- Enable Flush Test for BP5 by @eisenhauer in #3683
- Use ccache in github actions builds by @scottwittenburg in #3705
- ci: Fix conditional expression syntax to fix cache save by @scottwittenburg in #3730
- ci: work around bug in actions/cache by @scottwittenburg in #3731
- build(deps): bump cryptography from 41.0.2 to 41.0.3 in /docs by @dependabot in #3733
- Version constants plus functions to retrieve list of ... by @pnorbert in #3676
- ci: add libcurl4 to all ubuntu images by @scottwittenburg in #3734
- ci: update release instructions by @vicentebolea in #3738
- ci: add downloads readthedocs by @vicentebolea in #3739
- cmake: fix evpath plugins install path by @vicentebolea in #3737
- Remote Data access via Get() or Transport by @eisenhauer in #3740
- readthedocs: remove sys pkgs; use current ADIOS2 by @vicentebolea in #3743
- Fix: Static Builds Against C-Blosc2 by @ax3l in #3715
- CI: add two static builds by @vicentebolea in #3749
- Upstream ffs by @eisenhauer in #3755
- Update Readme.md with community section by @vicentebolea in #3700
- cmake,readthedocs: update python enable instructions by @vicentebolea in #3713
- cmake: correct info.h installation path by @vicentebolea in #3758
- Korvo upstream by @eisenhauer in #3761
- cmake: resolve cmake python deprecation warning by @vicentebolea in #3764
- Update kwsys by @vicentebolea in #3765
- ci: add mgard dependency to spack builds by @vicentebolea in #3753
- Misc fixes by @vicentebolea in #3768
- Add automated vulnerability code analysis using CodeQL. by @vicentebolea in #3606
- cmake: correct ADIOS2_Blosc2_PREFER_SHARED behavior by @vicentebolea in #3767
- match type of timestep for h5 engine to size_t (same as adios Variabl… by @guj in #3776
- ci: Create static minimal build by @vicentebolea in #3760
- Remove unused SelectionType values by @eisenhauer in #3784
- Update ADIOS2 HDF5 VOL with basic set of capability flags by @jhendersonHDF in #3632
- gha,ci: update checkout to v4 by @vicentebolea in #3789
- ci: Add HDF5 to a windows build by @scottwittenburg in #3732
- Fixup local var reading by block with test, master branch by @eisenhauer in #3787
- Add MinBlocksInfo to BP5 writer engine by @eisenhauer in #3794
- yaml-cpp: support 0.8.0 version by @vicentebolea in #3799
- Delete experimental examples by @spyridon97 in #3800
- Remove BP5 BetweenStepPairs variable that hides Engine.h counterpart by @eisenhauer in #3803
- WIP: Reader-side Profiling by @eisenhauer in #3778
- Fix the
ADIOS_USE_{}variable names to use ADIOS2 by @anagainaru in #3807 - Pass Array Order of reader to remote server for proper Get() operation by @eisenhauer in #3810
- Fix abs/rel step in BP5 DoCount by @eisenhauer in #3812
- FFS upstream master-branch by @eisenhauer in #3815
- added support to query BP5 files by @guj in #3809
- install: export adios2 device variables by @vicentebolea in #3819
- ci: fix codeql ignore path by @vicentebolea in #3772
- Add Memory Selection to SST, and prototype a mechanism for running st… by @eisenhauer in #3823
- Remote test redux by @eisenhauer in #3830
- Tweak Remote class and test multi-threaded file remote access by @eisenhauer in #3834
- MPI: add timeout for conf test for MPI DataPlane by @vicentebolea in #3848
- cmake: update minimum cmake to 3.12 by @vicentebolea in #3849
- MPI DataPlane: do not call MPI_Init internally by @vicentebolea in #3847
- Fix destdir install test by @vicentebolea in #3850
- Organize and add examples by @spyridon97 in #3820
- Blosc2 USE ON: Fix Module Fallback by @ax3l in #3774
- BP5Serializer: handle local variables that use operators by @caitlinross in #3859
- Update examples by @spyridon97 in #3856
- bug fix: syntax error in json output by @guj in #3857
- Remove Slack as a contact mechanism by @eisenhauer in #3866
- Remote Server Improvements by @eisenhauer in #3862
- Fix hip example compilation error by @spyridon97 in #3865
- new attempt to commit query support of local array. by @guj in #3868
- cmake: link to MPI::MPI_Fortran using INTERFACE by @caitlinross in #3853
- Backports from release_29 by @vicentebolea in #3858
- Backport 3877 to master by @vicentebolea in #3882
- Fix MPI Data plane cohort handling by @vicentebolea in #3588
- Testing writing null blocks without affecting the read by @anagainaru in #3878
- docs: update new_release.md by @vicentebolea in #3892
- cmake: find threads package first by @vicentebolea in #3893
- Add tutorials by @spyridon97 in #3880
- examples,hip: use find_package(hip) once in proj by @vicentebolea in #3896
- Add Remote "-status" command to see if a server is running and where by @eisenhauer in #3911
- Defining and computing derived variables by @anagainaru in #3816
- Fix printing issue in perfstub thirdparty code by @anagainaru in #3913
- ci,crusher: increase supported num branches by @vicentebolea in #3902
- Change install directory of Gray scott files by @spyridon97 in #3905
- ci: add shellcheck coverage to source and testing by @vicentebolea in #3915
- Change install directory of Gray scott files again by @spyridon97 in #3917
- SST with GPU support by @anagainaru in #3918
- ci: Set openmpi and openmp parameters to make an efficient use of the CI by @vicentebolea in #3908
- Cleanup server output a bit by @eisenhauer in #3914
- ReadMe.md: Mention 2.9.2 release by @eschnett in #3921
- ci: use mpich built with ch3:sock to speed up tests by @scottwittenburg in #3883
- Dataman with GPU support by @anagainaru in #3928
- CI Contract Build examples with external ADIOS by @spyridon97 in #3906
- Revert "Fix printing issue in perfstub thirdparty code" by @anagainaru in #3943
- Perf stubs upstream by @eisenhauer in #3944
- Fix the HIP CMAKE issues to run on Frontier by @anagainaru in #3931
- Set the compiler for the Kokkos DataMan example by @anagainaru in #3946
- Fix MATLAB bindings for MacOS by @tomgade09 in #3950
- Delete VTK code in examples by @spyridon97 in #3952
- Enable shell check scripts for existing scripts by @spyridon97 in #3940
- Move useful docs from ADIOS2-examples to ADIOS2 by @spyridon97 in #3939
- Use adios2_mode_readRandomAccess in matlab open to make it work for BP5 by @pnorbert in #3956
- Adapt libfabric dataplane of SST to Cray CXI provider by @franzpoeschel in #3672
- Update readme for heat transfer example with new location and build i… by @pnorbert in #3904
- ci: fix false negative asan checks by @scottwittenburg in #3957
- adios2::core::ADIOS: Initialize new IO objects with config file by @spyridon97 in #3955
- MDR Operator by @pnorbert in #3826
- fix for ASAN issue related to JoinedDimArray handling in BP5 deserializer by @pnorbert in #3963
- Have HDF5 write raise error if operator(s) requested by @rupertnash in #3951
- Http connector by @dmitry-ganyushin in #3890
- Add pip packaging by @scottwittenburg in #3960
- pip: also run pypi upload job on workflow_dispatch by @scottwittenburg in #3967
- Campaign manager by @pnorbert in #3932
- update copyright.txt to include recent kitware authors by @vicentebolea in #3965
- Bug fix building ADIOS2 without the campaign example by @anagainaru in #3969
- Several fixes for ASAN issues by @eisenhauer in #3959
- SST ASAN buffer fix by @eisenhauer in #3972
- python: Implement Python high level API in python by @vicentebolea in #3938
Changes added in the latest final release
- Fix the C example stepping and reading data. Simplify code for shape/… by @pnorbert in #3980
- Fix read request length on BP5 Local arrays with operators (backport to release 2.10) by @eisenhauer in #3993
- Fix 1: bpls handle when BP5 throws logic error on minmax for unsuppor… by @pnorbert in #3994
- python,cmake: update version script to support rc by @vicentebolea in #3992
- Python API: Adios() new argument 'config_file', Stream() new argument… by @pnorbert in #3984
- Backport recent ASAN fixes to 2.10 release branch by @eisenhauer in #3985
- Merge upstream dill chanages to release_210 by @eisenhauer in #4005
- Backport master commits by @vicentebolea in #4029
- Backport master PR into release210 by @vicentebolea in #4042
- backports from master by @vicentebolea in #4059
- Backports from master by @vicentebolea in #4128
- Bump version to v2.10.0 by @vicentebolea in #4130
New Contributors
- @jhendersonHDF made their first contribution in #3632
- @spyridon97 made their first contribution in #3800
- @tomgade09 made their first contribution in #3950
- @rupertnash made their first contribution in #3951
Full Changelog: v2.9.2...v2.10.0