github openucx/ucx v1.18.0-rc3
v1.18.0 RC3

pre-releaseone day ago

1.18.0-rc3 (December 23, 2024)

Features:

UCP

  • Enabled using CUDA staging buffers for pipeline protocols by default
  • Added endpoint reconfiguration support for non-reused p2p scenarios
  • Enabled non-cacheable memory domains, activated for gdr_copy
  • Added user_data parameter to ucp_ep_query
  • Added support for host memory pipeline through CUDA buffers for rendezvous protocol
  • Added global VA infrastructure and memory region in absence of error handling
  • Made protocol performance node names more informative
  • Enforced always running on the same thread in single thread mode
  • Multiple improvements in protocols selection infrastructure
  • Added UCP_MEM_MAP_LOCK API flag to enforce locked memory mapping
  • Allowed up-to 64 endpoint lanes for systems with many transports or devices
  • Added usage tracker to worker
  • Improved various logging messages

RDMA CORE (IB, ROCE, etc.)

  • Added environment variable to manage DC initiator capacity
  • Added DC dcs_hybrid policy
  • Reduced MLX5/DV stack size consumption
  • Added ODP support for verbs and mlx5dv
  • Added support of CUDA managed memory on IB when ODP is available
  • Added support of Adaptive Routing on RoCE
  • Enabled use of implicit ODP with relaxed ordering
  • Improved GPU-Direct detection in IB transport
  • Increased DC initiator default count to 32 for performance optimization
  • Added ConnectX-8 device support with DDP
  • Added support for subnet filter list for RoCE interfaces
  • Enhanced the error message to provide more details when a connection cannot be established due to unreachable transports
  • Added IB MLX5 as a separate UCX module with separate RPM sub-package
  • Added initial support for GGA transport, for fast DPU memory access
  • Set IB DevX atomic mode based on device capabilities
  • Removed DC keepalive mechanism, since the keepalive is done on UCP layer
  • Optimized cross-gVMI memory registration using indirect memory keys cache
  • Improved various logging messages

CUDA

  • Added multi-node NVlink support
  • Added CUDA Fabric memory support with detection and allocation
  • Improved gdr_copy latency estimations on AMD Milan systems
  • Added check for gdr_copy runtime/build version mismatch
  • Added handling missing IPC capability when unpacking keys
  • Added caching for CUDA IPC memory pool import operation
  • Added gdr_copy variables to optimize performance on Grace Hopper systems
  • Improved CUDA IPC concurrency for a larger count of reachable peers

UCS

  • Added support for wildcards in configuration parameter names
  • Added ASAN protection to several internal data structures
  • Reduced stack usage in topology detection code
  • Improved bitmaps configuration parsing with wider bitfield
  • Added options to set topology distance between devices
  • Optimized VFS unix socket watch by using user private folder
  • Added general IP subnet matching infrastructure
  • Extend array data structure to support user-provided array copy routine
  • Improved time units description

UCM

  • Extend CUDA memory hooks to include memory mapping APIs

Tools

  • Improved performance by increasing window size for put_bw and add get_bw in ucx_perftest
  • Added multi-send flag for receive operations in bandwidth benchmarks in ucx_perftest
  • Improved ucx_perftest uni-directional test with added fence
  • Detailed ucx_perftest batch section of command-line documentation

Documentation

  • Added a section regarding adaptive routing on RoCE

Architecture

  • Added CPU Model for MI300A
  • Added Fujitsu ARM specific values to ucx.conf
  • Added AMD Turin support
  • Added an optimized non-temporal memory copy implementation for AMD CPU

Build

  • Improved compiler error reporting with added flag
  • Improved coverity script to allow faster turnaround time
  • Improved Intel Compiler detection and support

GO

  • Added multi-send flag and user memh support in request params

Packaging

  • Improved dpkg-buildpackage sample command by explicitly adding mlx5 related arguments

Bugfixes:

UCP

  • Fixed stack overflow in exported rkey unpack
  • Removed extra remote-cpu overhead from protocol estimation for zcopy
  • Fixed performance estimation for rndv pipeline protocols
  • Fixed ATP sending by picking the correct lane
  • Fixed missing reg_id on memh creation
  • Fixed repeated invalidations by retaining existing access flags
  • Fixed abort reason propagation for rendezvous RTR mtype
  • Do not check transport availability if it is disabled by UCX_TLS environemnt variable
  • Fixed wrong flag being used for checking BCOPY capability
  • Fixed sending too many ATPs for small messages
  • Enforced 16 bits size for Active Messages identifiers
  • Fixed unnecessary status check for emulated AMO
  • Fixed more than one fragment sending in rendezvous pipeline
  • Fixed crash by using biggest max frag across all lanes
  • Fixed missing memory handle flags by copying from parent to child
  • Fixed worker interface activate count
  • Fixed flush requests by replacing ATP/flush lane map with lane indexes
  • Fixed lost uct_flags when merging memory regions

UCT

  • Fixed memory domain UCT flags description

RDMA CORE (IB, ROCE, etc.)

  • Fixed FETCH_ADD remote access error for ODP/KSM case
  • Fixed missing conditional compilation checks for DM
  • Fixed IB MD allocation naming typo
  • Fixed invalid GIDs filter in IB
  • Fixed flags usage in MLX5 zcopy_post
  • Do not limit ODP registration retries
  • Fixed JUCX failures by considering the number of supported completion vectors

CUDA

  • Fixed async memory handling using CUDA memory type on Grace
  • Added rcache overhead in performance estimation
  • Fixed gdr_copy performance regression by providing maximum estimation between get and put
  • Fixed CUDA IPC reachability check
  • Fixed crash in MPI_Finalize when CUDA context is destroyed
  • Always require rcache by default for gdr_copy
  • Fixed crash in gdr_copy cleanup when registration cache is disabled
  • Fixed CUDA copy memory domain allocations
  • Fixed multiple tests for gdr_copy transport
  • Fixed race condition in CUDA IPC peer accessible cache

UCS

  • Fixed a crash by using heap allocation to process expired timers in batch
  • Fixed allocation issue on memtrack dump
  • Fixed deletion of the monitored folder in VFS
  • Fixed unsafe resize for DC initiator array
  • Fixed function macro invocation to match C standard
  • Fixed calling async handler on already released resource
  • Fixed performance by setting higher bandwidth for different NUMA nodes on Grace
  • Fixed undeclared value error in timer conversion routine
  • Fixed uninitialized value access in registration cache

UCM

  • Fixed race condition in parsing proc maps
  • Fixed mremap failure while parsing /proc/self/maps

ROCM

  • Fixed ROCM interface reachability test
  • Fixed memory domain fork test

TCP

  • Always bind endpoint to interface

Tools

  • Fixed buffer size potential overflow in ucx_perftest
  • Fixed missing address when packing memory keys on ucx_perftest
  • Fixed memory leak for endpoint report in ucx_info
  • Fixed build without openmp in ucx_perftest
  • Fixed UCT device override on server side on ucx_perftest

Build

  • Fixed using correct ASAN version for running tests

Configuration

  • Used POSIX bourne syntax to check equality
  • Fixed build failure by using proper flags in compiler.m4
  • Fixed perftest MAD support default guessing

GO

  • Added serialized thread mode to avoid subtle races between threads
  • Fixed make distcheck

Don't miss a new ucx release

NewReleases is sending notifications on new releases.