real-logic/aeron 1.45.0

on GitHub

Noteworthy Changes

• Per-Stream Session Limits
  • Allows users to limit the number of different sessions (i.e. distinct publication images) that can be created connecting to the same subscription (channel & stream). This can be used to avoid resource exhaustion the driver hosting the subscription, when clients on other drivers are over zealously creating publications.
• Infer Group Subscriptions for Recovery Behaviour
  • When using Multi-Destination Cast, if the user wants to make use of group semantics for nakking and retransmission, then previously group=true needed to be added to the channel configuration on the subscription. Failing to do so would potentially cause excessive nakking. This feature will set a flag on the SETUP message that the subscription can use to automate the setting of this configuration option. Setting group=true/false will override the derived value.
• Static Counters
  • Add new API to create static counters whose lifetime is decoupled from an Aeron client instance that created them. Unlike the normal counters which are deleted when an Aeron client is closed or times out, the static counters remain valid until the MediaDriver is closed. Hydra, Coinbase and others have asked for this feature.
  • Static counters are similar to the system counters which are created by the MediaDriver upon startup. Such counters cannot be delete and only one instance of each counter can exist per MediaDriver instance. A static counter is identified by a tuple <typeId, registrationId> which are specified at the creation time. Calling addStaticCounter with the same <typeId, registrationId> multiple times will only create a counter once with subsequent calls returning the id of an existing counter. An exception will be thrown upon an attempt to specify <typeId, registrationId> of an existing non-static counter.
• Archive Integration of Response Channels
  • Users of the AeronArchive client can make use of response channels for control, replay and replication communication paths.
• Archive Logging Improvements
  • To increase visiblity of replay and recording session state changes.
  • Enable either REPLAY_SESSION_STATE_CHANGE or RECORDING_SESSION_STATE_CHANGE when configuring the aeron agent.
• ReplayMerge Backoff
  • To slow down internal state transitions that involve calls to ‘get max recorded position’.
  • Reduces network traffic.
  • Greatly reduces log output when debugging is enabled.
• RetransmitHandler Improvements
  • On unicast channels, a received NAK may immediately replace a previous received NAK so long as the offset has increased.
• Retransmitted Bytes Counter
  • To understand network conditions, it can be useful to know how many bytes have been retransmitted due to reception of NAKs.
  • We have exposed a new counter “Retransmitted bytes”, which serves as an approximate indicator of this information.
  • The counter only approximates the true value, as MDC retransmits are only recorded once, rather than N times (once for each destination). It is also worth noting that the retransmitted bytes are not included in the total bytes sent counter value. We may improve these aspects in the future.
• C++ API Deprecation
  • Aeron currently has two C++ APIs, a pure C++ version and a wrapper over the C API. This release we are no longer adding new features to the C++ API. This include being able to offer raw uint8_t * values and the new static counters feature.

Changelog

• [Java Cluster] IngressAdapter to delegate to ConsensusModuleAgent when schema id is unknown
• [C++ Wrapper] Correctly implement tracking of context on the header.
• [C/C++] Do not compile with -Ofast compilation level.
• [C/C++] Pass custom optimization level to the MSVC compiler.
• [Build] Download and install CMake as part of the build process to ensure latest version of CMake.
• [C] Tidy up aeron_reallocf to work correctly on allocation failure.
• [C Driver] Expose try_connect_stream method for use from within ATS.
• [C Driver] Remove unconnected stream message.
• [C Driver] Fix string format for AERON_SET_ERROR arguments.
• [C/C++/Java] Add per channel untethered window limit and resting timeouts (#1588)
• [C++ Wrapper] Add findByTypeIdAndRegistrationId to C++ Wrapper.
• [Java Driver] Pre-start async task executor threads to avoid inheriting affinity from the conductor thread.
• [Java Cluster] Print serviceId when throwing "ack out of sequence" exception.
• [C++] Fixed set thread name. (#1594)
• [Java/C++ Archive] Use response channels in the Archive Client (#1560)
• [PS] Add Powershell script for Windows build.
• [C/Java Driver] Do not resolve self hostname when starting MediaDriver to avoid a DNS-induced stall. Make resolverName required when driver name resolution is used.
• [Java/C/C++ Client] Propagate context though assembled header when using the buffer builder.
• [Java] Fix FixedLossGenerator to not drop frames before the specified "drop region".
• [C/Java Driver] Expose counter for number of bytes retransmitted.
• [Java Cluster] Remove redundant param.
• [C] Move validation earlier in the flow to prevents leaks in ATS on validation failure.
• [Java Samples] Use context instead of image by session id to resolve Image on request message.
• [C/Java Driver] Add session limits for a stream to prevent resource exhaustion of resources on drivers hosting subscriptions (#1598)
• [Java Cluster] Add support for installing schema extensions into the ConsensusModule.
• [Java Samples] Change response channels sample to use controlled poll to prevent response channel test failures.
• [Java] Add some additional error test to ReplayMerge and test for concurrent ReplayMerges along with additional error reporting.
• [C++ Archive] Rethrow original exception in ReplayMerge so information is not lost. #1587
• [C] add and use aeron_mkdir_recursive (#1602)
• [Java] Version interface to allow testing usage of it (#1605)
• [Java] Include file name when mark file version validation fails.
• [Java] Correct offsets for response setup flyweight in ReceiveChannelEndpointThreadLocals.
• [Java] Default cluster log flow control to driver default which will commonly be max now that overruns can go up to half a term.
• [Java Archive] Assign client name to Archive's Aeron client.
• [Java Cluster] Assign client name to Cluster Aeron clients.
• [C++/Java Client] Pre-touch mapped log buffers on the client if Aeron.Context.preTouchMappedMemory() returns true, i.e. client configuration takes precedence over channel parameters and media driver configuration.
• [Java Cluster] Delegate decision on unknown schema to consensus agent
• [Java Cluster] Ensure service count is zero when consensus module extension is installed.
• [C/C++/Java] Annotations to support config/counters documentation/validation (#1593)
• [C/Java Driver] Add unicast semantics to RetransmitHandler (#1603)
• [Java Cluster] Add archive and egress response endpoints for cluster and make them optional.
• [Java Cluster] Remove default for cluster services directory configuration and leave resolution of that value until conclude.
• [Java Archive] Use publication.availableWindow instead of looking up the pubLmt counter.
• [Java Cluster] First cut at consensus module extension integration.
• [C Driver] Fixes for fixed loss interceptor used for testing.
• [Java Cluster] Remove unused params.
• [Java Archive] Separate lines for debugging.
• [Java Cluster] Provide access to archive and aeron for consensus module extension.
• [Java Cluster] Delegate cluster session creation to extension if available.
• [Java Cluster] delegate callbacks to extension on state changes.
• [C/Java Driver] Track EOS for each receiver connected to the NetworkPublication (#1606)
• [C Client] Declare a volatile pointer to aeron_image_list_stct instead of a pointer to a volatile struct. (#1607)
• [Java Cluster] A more decoupled approach to ConsensusModuleExtension.
• [Java Cluster] Provide access to log publication after an election for consensus module extension.
• [Java] Improve error messages when publication errors occur.
• [Java Cluster] Consensus module extension API refinements.
• [Java Cluster] Add replay of session open and close delegation to consensus module extension.
• [Java Cluster] Add replay of extension message delegation to consensus module extension.
• [Java Cluster] ConsensusControlState to be passed from Agent to Extension and allows control.
• [C/C++ Client] Use aeron-client-conductor as a client conductor agent name across all implementations.
• [Java Cluster] pass actingVersion to onMessage of ConsensusModuleExtension
• [Java] Version generated class to have toString
• [Bash] Add option to disable C++ Archive API.
• [Java Cluster] Add ability to access response publication for a session and only update keepalive status when open.
• [Java Cluster] Handle cluster extension messages on egress listener side.
• [Java Cluster] Add extension message support to AeronCluster client.
• [C++ Wrapper] Allow pass through of const uint8_t * buffers.
• [C++] Replace array with pointer to address code-ql issue on Publication in C++ API and Wrapper.
• [Build] Fix aeronmd rpath in the generated package.
• [Java Cluster] Use separate archive control session for consensus module extension.
• [Java Cluster] Add work cycle callback for ConsensusModuleExtension.
• [C/Java Driver] Replace exception with increment of counter for overflowing the retransmit pool.
• [C] add -w to C AeronStat app (#1614)
• [Java Cluster] Provide access to encoded principle for authorising consensus module extension session.
• [Java Archive] Add support for response channels startReplay and startBoundedReplay.
• [Java Samples] Fix StreamStat mapping of sub-pos which had a join position included in the channel uri and prevented it to be merged with the rest of stream positions. Also ensure that longest channel uri is printed.
• [Java Cluster] Extract ReplicationParams for RecordingReplication to allow for more flexible configuration.
• [Java Cluster] Use response channels for log replication during the election.
• [C Driver] Handle a case when getifaddrs call fails.
• [Java Cluster] Provide access to cluster client session from extension via an interface to restrict usage.
• [Java Cluster] Add source response channel to MultipleRecordingReplication.
• [Java Cluster] Simplify hash code for RecordingLog.
• [Java Cluster] Use response channel in ClusterBackup when connecting to the cluster archive.
• [Java Archive] add helper method to determine if Archive Context is concluded (#1617)
• [Java Cluster] Validate standby snapshot (#1616)
• [Java] Add tests for HeaderWrite class + always combine values in the same order (from MSB to LSB).
• Intercept replay and recording session state changes (#1615)
• [Java Archive] Use getMaxRecordedPosition instead of getRecordingPosition to avoid two different calls while resolving position. Also call getMaxRecordedPosition only once in the attemptLiveJoin.
• [Java Cluster] Add ability to close a cluster session from the consensus module extension.
• [C/Java Archive/Driver] add backoff to calls to getMaxRecordedPosition in ReplayMerge (#1618)
• [Java Archive] Prevent replay token clashes.
• [C Samples] Fix CnC version printed in AeronStat header.
• [Build] Modify CMake build to have exports for Aeron targets.
• [Java Cluster] separate ingress and log message callback
• [Java Cluster] Method renaming for log extension dispatch.
• [Java Cluster] Add authorization service to ConsensusModuleControl
• [Java Cluster] cluster session - do not delete principal when transitioning to open
• [Java Cluster] Clean up session logging.
• [Java Cluster] Add event log for getMaxRecordedPosition call.
• [Java Cluster] callback on prepare for new leadership in ConsensusModuleExtension
• [Java Cluster] Consistent checking of leadership id in admin request to cluster.
• [Java Cluster] Add reason for ELECTION_STATE_CHANGE + log state transition after the logic was executed.
• [Java Cluster] Process standby snapshot control signals like normal snapshot signals. (#1634)
• [C Client] Fix #1633, only get channel status if pointer is non-null.
• [C++ Wrapper] Fix Image wrapping in a couple of cases improvements. Delete implicit *FragmentAssembler copy/move ctors/operators (#1639)
• [Java Archive] Reduce archive listRecordingsForUri idling (#1638)
• [C Client] Return an error if the client name is too long (100 chars) instead of truncating.
• [C++ Wrapper] Use a copy constructor for wrapper Context to better handle resource management.
• [C/Java] Send a flag on SETUP to indicate that the publication is expecting a group of subscription and use that to select the multicast feedback delay generator when inferring behaviour.
• [Java Driver] Set default on stream session limit property.
• [Java Cluster] Consolidate logic for closing a cluster session in the consensus module.
• [Java/C/C++ wrapper] Add new API to create static counters. (#1625)
• [C++ Wrapper Client] Implement offer with raw pointer and length for exclusive publications.
• [Java Cluster] Do not reinstate cluster sessions that have timed out but not got committed before a change of leadership.
• [C/Java Samples] Align implementation of AeronStat.
• [Java Cluster] Do slow tick work in consensus module agent until no work is done in a cycle.
• [Java Client] Improve error message when failing to open CnC file.
• [C/Java] Retry on ERROR_SHARING_VIOLATION (32) on Windows when awaiting CnC file mapping.
• [C++/Java Archive] Use Publication.publicationLimit instead of looking up counter.
• [Java Driver] Fix re-resolution to trigger when no SMs received by Publications and new Publications are constantly created.
• [Java Cluster] Fix for #1619. Handle the case where the last portion of data a cluster log is a padding frame.
• [Java Cluster] Make describe recordingId work with recordings in any state.
• [Java Archive] Only index valida recordings and implement ArchiveTool methods via forEach.
• [Java] Print offending value in exception with ChannelUriStringBuilder.
• [Java Archive] Do not mark recording as VALID if there are no segment files.
• [Java Archive] Always report state OK even when not changing the state.
• [Java Archive] Mark purged recording as DELETED.
• [C] Fix heap-buffer-overflow issue when removing entries from the aeron_int64_counter_map.
• [C] Align aeron_int64_counter_map with Java, i.e. do not allow initial_value to be put into the map.
• [Java Cluster] Add API for taking a snapshot in consensus module extension.
• [C/Java Driver] add config option and uri param for setting max retransmits (#1640)
• Upgrade to CMake 3.30.0.
• Upgrade to HdrHistogram 2.2.2.
• Upgrade to ByteBuddy 1.14.19.
• Upgrade to Agrona 1.22.0.
• Upgrade to Hamcrest 3.0.
• Upgrade to SBE 1.32.1.
• Upgrade to JUnit 5.11.0.