First beta of Charm++ 9.0, cut from 'recharm' at 7a414b8 (reconverse bbb92e1).
Charm++ 9.0 runs on reconverse, a new implementation of Converse,
included as the contrib/reconverse submodule. The classic Converse line
ends with 8.1.0 and continues as the branch 'classic', for bug fixes only.
This beta covers everything since 8.1.0; the full list, by topic, is the
"What's new in Charm++ 9.0.0-beta1" section of CHANGES in this tag.
Highlights:
- New runtime core: reconverse, with LCI2 as the network layer. Programs
start without charmrun ('./app +pe N'; multi-process with lcrun, srun or
mpirun), and with no communication thread every core runs a worker PE.
Shared-memory IPC between the processes of a host (+ipc). - Coroutine entry methods: a C++20 [coroutine] entry method waits on
futures and callbacks without a thread. - AMPI on reconverse, with migration between processes, checkpoint to
disk and in memory, and in-memory rollback (syncft); campi runs MPI
ranks as C++20 coroutines with no thread per rank. - GPUs: the portable HAPI core, device-to-device communication and
zero-copy, and CkDevicePersistent transfers between any pair of PEs,
within a host and between hosts. - 64-bit object ids: bounded arrays up to 2^49 elements, ids stable
across checkpoint and restart with a different process count. - CkDyn, a C API for language bindings that define chare classes at run
time; charm4py 2 is built on it (released separately). - Interoperation with MPI programs: CharmLibInit(MPI_COMM_WORLD, ...)
runs Charm++ libraries in phases of an MPI program; an abort ends the
whole MPI job. - Platforms: Linux (x86_64, arm8), macOS (arm64) and Windows (one
process, several PEs); Slingshot, InfiniBand and Omni-Path through
libfabric or native psm2.
Getting the source:
git clone --branch v9.0.0-beta1 --recurse-submodules https://github.com/charmplusplus/charm
cd charm && ./build charm++ reconverse-linux-x86_64 --with-production -j8
or download charm-9.0.0-beta1.tar.gz below, which includes the reconverse
submodule (GitHub's automatic "Source code" archives do not).
Known gaps in the beta:
-
AMPI: dup and split of an intercommunicator (#4099); CkDDT_Struct
lower bounds and negative extents (#4100); three request-completion
items of #4097 (same-root Ibcasts, MPI_Op_free during a non-blocking
reduction, sequence number wraparound). pieglobals with Fortran
aborts at exit on Linux after the program has finished (#4157,
#4189). On macOS a rank cannot move to another process or restart
from a checkpoint (no way to turn address randomization off); campi
can. No shrink/expand; ROMIO is not built by default. -
CharmDebug is not available.
-
--build-shared does not link on reconverse (#4049).
-
Large scale: under heavy traffic at 8 to 32 nodes, exhaustion of
LCI's receive packet pool can hang a run (reconverse #257). A
one-process program started directly (without srun) inside a Slurm
allocation waits in LCI's bootstrap for the job's other tasks; start
it with 'srun -n 1' or set LCT_PMI_BACKEND=local (reconverse #320). -
MetisLB makes the load imbalance worse on more than 4 PEs (#4271);
use GreedyRefineLB, TreeLB or DistributedLB. -
Checkpoints written by 8.x or by v9.0.0-alpha1 cannot be restored;
checkpoints from v9.0.0-alpha2 and later can.
Tested at this tag (2026-11-15):
- CI at this tag (reconverse-ci run 43733107471), all 10 jobs green:
Linux x86_64 and arm64 and macOS arm64 in one and two processes
(Linux x86_64 also every example), Windows in one process, AMPI and
campi, syncft, MPI interop under mpirun, a stress job on an
error-checking build, and the documentation build. The CUDA job passed
on 443f7c2; the two commits after it change only README and CHANGES. - macOS arm64 (reconverse-darwin-arm8, production): the sweep of 218
test, example and benchmark directories, one process and two
processes: 207 and 208 pass; the failures are the 8 known build
failures (OpenMP, TRAM, lvServer, dllarray) and randomTRAM3D, as
before. tests/ampi (26 in one process, 18 in two) and tests/campi
(8 and 8) all pass. - OLCF Frontier (Slingshot-11, AMD MI250X): production, HIP (gfx90a) and
error-checking builds. The CI test set at 1 node and at 2 nodes x 2
processes passes (90 of 92 runs; the other 2 are a test-harness defect
fixed by #4272). Load-balancer, quiescence, megatest and checkpoint
tests at 2 and 4 nodes x 8 processes x 7 PEs pass, including a
checkpoint written on 2 nodes and restarted on 4 and on 1; MetisLB is
the exception (see the known gaps). AMPI migration and
checkpoint/restart across processes and nodes pass. MPI interop with
Cray MPICH passes, including the abort test. All 15 GPU runs pass
(CkDevicePersistent within a node and between nodes, gpudirect,
d2dtest, hapitest). At 16 nodes x 8 processes x 7 PEs (896 PEs),
megatest, lbsweep (TreeLB), qdstress, kNeighbor, leanmd and the
zero-copy broadcast tests pass. - Zero-copy broadcasts with several processes per node (#4269, fixed by
#4273 in this tag): verified at 2x2 to 16x8 on Frontier, at 2x2 and
2x8 on NCSA Delta (Slingshot-11), and at 2x2 and 2x4 on Purdue Anvil
(InfiniBand).
Feedback and bug reports: https://github.com/charmplusplus/charm/issues