karafka 2.4.0.beta1

v2.4.0.beta1

on Ruby Gems

This release contains BREAKING changes. Make sure to read and apply upgrade notes.

• [Breaking] Drop Ruby 2.7 support.
• [Breaking] Drop the concept of consumer group mapping.
• [Breaking] karafka topics migrate will now perform declarative topics configuration alignment.
• [Breaking] Replace deserializer config with #deserializers in routing to support key and lazy header deserializers.
• [Breaking] Rename Karafka::Serializers::JSON::Deserializer to Karafka::Deserializers::Payload to reflect its role.
• [Feature] Support custom OAuth providers (with a lot of help from bruce-szalwinski-he and hotelengine.com).
• [Feature] Provide karafka topics alter for declarative topics alignment.
• [Feature] Introduce ability to use direct assignments (Pro).
• [Feature] Provide consumer piping API (Pro).
• [Feature] Introduce karafka topics plan to describe changes that will be applied when migrating.
• [Feature] Introduce ability to use custom message key deserializers.
• [Feature] Introduce ability to use custom message headers deserializers.
• [Enhancement] Assign names to internal threads for better debuggability when on TTIN.
• [Enhancement] Provide log_polling setting to the Karafka::Instrumentation::LoggerListener to silence polling in any non-debug mode.
• [Enhancement] Provide metadata#message to be able to retrieve message from metadata.
• [Enhancement] Include number of attempts prior to DLQ message being dispatched including the dispatch one (Pro).
• [Enhancement] Provide ability to decide how to dispatch from DLQ (sync / async).
• [Enhancement] Provide ability to decide how to mark as consumed from DLQ (sync / async).
• [Enhancement] Allow for usage of a custom Appsignal namespace when logging.
• [Enhancement] Do not run periodic jobs when LRJ job is running despite polling (LRJ can still start when Periodic runs).
• [Enhancement] Improve accuracy of periodic jobs and make sure they do not run too early after saturated work.
• [Enhancement] Introduce ability to async lock other subscription groups polling.
• [Enhancement] Improve shutdown when using long polling setup (high max_wait_time).
• [Enhancement] Provide Karafka::Admin#read_lags_with_offsets for ability to query lags and offsets of a given CG.
• [Enhancement] Allow direct assignments granular distribution in the Swarm (Pro).
• [Enhancement] Add a buffer to the supervisor supervision on shutdown to prevent a potential race condition when signal pass lags.
• [Enhancement] Provide ability to automatically generate and validate fingerprints of encrypted payload.
• [Enhancement] Support enable.partition.eof fast yielding.
• [Enhancement] Provide #mark_as_consumed and #mark_as_consumed! to the iterator.
• [Enhancement] Introduce graceful #stop to the iterator instead of recommending of usage of break.
• [Enhancement] Do not run jobs schedulers and other interval based operations on each job queue unlock.
• [Enhancement] Publish listeners status lifecycle events.
• [Enhancement] Use proxy wrapper for Admin metadata requests.
• [Enhancement] Use limited scope topic info data when operating on direct topics instead of full cluster queries.
• [Enhancement] No longer raise Karafka::UnsupportedCaseError for not recognized error types to support dynamic errors reporting.
• [Change] Do not create new proxy object to Rdkafka with certain low-level operations and re-use existing.
• [Change] Update karafka.erb template with a placeholder for waterdrop and karafka error instrumentation.
• [Fix] Pro Swarm liveness listener can report incorrect failure when dynamic multiplexing scales down.
• [Fix] K8s liveness listener can report incorrect failure when dynamic multiplexing scales down.
• [Fix] Fix a case where connection conductor would not be released during manager state changes.
• [Fix] Make sure, that all Admin operations go through stabilization proxy.
• [Fix] Fix an issue where coordinator running jobs would not count periodic jobs and revocations.
• [Fix] Fix a case where critically crashed supervisor would raise incorrect error.
• [Fix] Re-raise critical supervisor errors before shutdown.
• [Fix] Fix a case when right-open (infinite) swarm matching would not pass validations.
• [Fix] Make #enqueue_all output compatible with ActiveJob.perform_all_later (oozzal)

Upgrade Notes

PLEASE MAKE SURE TO READ AND APPLY THEM!

Available here.