gems karafka 2.6.2.alpha1
v2.6.2.alpha1

34 minutes ago
  • [EOL] Drop Ruby 3.2 support; require Ruby 3.3+.
  • [EOL] Drop Rails 7.0 and 7.1 compatibility; require Rails 7.2+.
  • [Feature] Add Instrumentation::Vendors::NewRelic::MetricsListener for publishing Karafka metrics to New Relic. Context is encoded in the metric name, as New Relic custom metrics do not support tags (svyatmuzyka).
  • [Feature] Add karafka info --extended, printing the routing tree, global app config and effective Kafka config, with sensitive values redacted.
  • [Feature] Add Kubernetes::ReadinessListener, serving a readiness probe that reports healthy once all subscription groups poll and not-ready on shutdown or quieting, so pods drain before exiting.
  • [Feature] Allow the --include/--exclude CLI server filters to accept wildcard patterns (e.g. --exclude-consumer-groups app-a-*), including topics discovered at runtime by Pro routing patterns.
  • [Feature] Add Karafka::Admin.list_consumer_groups to list every consumer group in the cluster with its state. Requires karafka-rdkafka >= 0.31.0.
  • [Feature] Add Karafka::Admin.describe_consumer_groups to read consumer groups' state, coordinator, assignor and members with their partition assignments. Requires karafka-rdkafka >= 0.31.0.
  • [Feature] Add Karafka::Admin.describe_cluster and .describe_topics to read the cluster id, controller, brokers with racks, topic ids, partition leaders and replicas, and optionally the authorized operations. Requires karafka-rdkafka >= 0.31.0.
  • [Feature] Add Karafka::Admin.delete_records to delete the messages of given partitions below a given offset (or all of them with :end) by moving their low watermark. It is destructive and irreversible.
  • [Feature] Add Karafka::Admin.delete_consumer_group_offsets to delete a consumer group's committed offsets for given topics or partitions, keeping the consumer group and its other offsets. Raises Karafka::Errors::ConsumerGroupSubscribedToTopicError for topics the consumer group is subscribed to.
  • [Feature] [Pro] Add Virtual Partitions Estimator, a console API that samples a topic and estimates how well Virtual Partitions would parallelize it. Results can be saved as a versioned JSON report (target_file:) and displayed with karafka vps estimate-report.
  • [Feature] Add Admin::Configs::Config#source (e.g. :dynamic_topic_config, :dynamic_broker_config, :default_config), also on synonyms, telling a value set on the topic from one inherited from the broker or a built-in default. Requires karafka-rdkafka >= 0.32.1.
  • [Enhancement] Defer default log file creation until the first write and keep logging working (to stdout) when the log file cannot be created or written, e.g. on read-only filesystems (ydah).
  • [Enhancement] Stop allocating 6 objects per consumed message: build the default topic deserializers only once and read max_messages once per poll batch.
  • [Enhancement] Stop allocating a keyword Hash per consumed message when building its metadata.
  • [Enhancement] Log only the top two backtrace lines for librdkafka.error in the LoggerListener, so unreachable brokers no longer flood the logs.
  • [Enhancement] Include the topic, partition and offset from librdkafka error details (e.g. a message that failed to decompress) in LoggerListener error logs. Only librdkafka errors are used, and the consumer's own context takes precedence.
  • [Enhancement] Instrument client.topics.unknown and log it at the info level when a subscribed topic does not exist yet and allow.auto.create.topics is on, instead of ignoring it silently.
  • [Enhancement] Support the :consumer_group resource type in Karafka::Admin::Configs, so consumer group configs can be described and altered, e.g. consumer.session.timeout.ms and consumer.heartbeat.interval.ms, which group.protocol: "consumer" (KIP-848) does not take from the client config.
  • [Change] Raise UnsupportedOptionError with the Ruby engine name (e.g. jruby) when Swarm runs on a Ruby without Process.fork, also when Swarm::Supervisor#run is called directly, before any work is done.
  • [Change] Require karafka-rdkafka >= 0.32.2: lag and offset reads pass the partition list of Admin#list_consumer_group_offsets as the topic_partition_list: keyword, required since 0.32.1 (0.32.2 is the same code, released with Sigstore attestations).
  • [Change] Require karafka-rdkafka >= 0.32.0. A failed consumer close is now reported as a librdkafka.error on error.occurred instead of being dropped. A close fails in practice only after a fatal error, e.g. when a static membership conflict fenced the consumer. Polling usually reports that error first, so it is reported twice: as connection.client.poll.error and on close.
  • [Change] Fail karafka topics align and migrate (exit 1) when an existing declarative topic has config keys the cluster does not know (e.g. a typo), naming them, instead of skipping them silently. plan lists such keys as unknown and fails too.
  • [Change] Raise Karafka::Errors::ConsumerGroupNotEmptyError when Karafka::Admin changes the offsets of a consumer group with active members, instead of silently leaving them unchanged. With an external consumer client, offsets are read and changed for the queried consumer group, not the client's own.
  • [Change] Apply the routing defaults block to consumer-group topics only. Share-group topics get no defaults, so consumer-group-only calls in it (e.g. config(...), dead_letter_queue) no longer raise NoMethodError when a share group is drawn.
  • [Change] Require karafka-core >= 2.6.8.
  • [Change] Apply strict_declarative_topics to share-group topics too: each one needs an active declaration in declaratives.draw, made before routes.draw.
  • [Maintenance] Foundational work for Kafka share groups (KIP-932) is in progress (routing layer, consumer class hierarchy, config seams). Not usable yet - share groups can be described in routing but cannot run.
  • [Fix] [Pro] Do not let healthy KIP-848 (group.protocol=consumer) multiplexed listeners that never received an assignment block dynamic scaling. With more listeners than partitions they stay in the init join state with a rebalance age of 0, so the manager never saw them as stable and kept them alive instead of scaling them down.
  • [Fix] Stop CoordinatorsBuffer#@coordinators from growing unbounded across rebalances by dropping a topic entry on revoke once it tracks no partitions (mirrors PausesManager#delete).
  • [Fix] Use ::JSON.parse in the ActiveJob deserializer, so consuming ActiveJob messages works with the json gem >= 3.0.
  • [Fix] [Pro] Honour the read_committed isolation level in consumer-group lag compensation, so an in-flight transaction no longer overstates the lag (needs karafka-rdkafka >= 0.30.0).
  • [Fix] Run the Karafka::Server.stop sequence once when two stops are requested at the same time (e.g. TERM then INT on JRuby): the state check and the stopping transition are now one atomic step.
  • [Fix] Point the non-Rails karafka.rb install template comment to the karafka install command instead of the non-existent karafka:install.
  • [Fix] Make karafka console explain that the irb gem must be added to the Gemfile, instead of failing with a Bundler error, when irb is not in the bundle (non-Rails apps on Ruby 4.0+).
  • [Fix] Remove the subscription group error callback only after the consumer is closed, so errors reported while closing it (e.g. a failed close) are published via error.occurred.
  • [Fix] Set Message#received_at to the moment each message was polled, not the moment its batch was built, so it no longer trails by up to max_wait_time on low traffic topics (#2292).
  • [Fix] Stop the listener from failing with NoMethodError on JRuby when it checks a subscription group's jobs while workers complete them.
  • [Fix] Stop #inline_insights from returning no data on JRuby while another thread adds or evicts partition statistics.
  • [Fix] Restore InlineInsights::Tracker.exists?, which raised NoMethodError because its delegated #exists? was removed.
  • [Fix] Stop Admin operations from retrying forever when their result never becomes visible. They now raise ResultNotVisibleError after admin.max_retries_duration.
  • [Fix] Keep the Datadog LoggerListener job types cache consistent on JRuby when many workers fill it at once.
  • [Fix] Reset ActiveJob current attributes also when restoring them from the job payload fails part way, so they no longer stay on the worker thread. Attributes the consumer does not declare are now skipped and a non-Hash attributes payload raises Karafka::Errors::InvalidConfigurationError.
  • [Fix] [Pro] Stop an LRJ retry pause from staying paused forever when the listener resumed the partition before the client pause landed.
  • [Fix] [Pro] Validate all four schedule headers of scheduled messages. Only the cancel headers were checked, so a schedule missing schedule_target_topic passed and one missing schedule_target_epoch raised KeyError.
  • [Fix] Stop forceful shutdown from using a listener's client from two threads at once on JRuby. A listener still running after termination now keeps its client open and is reported via connection.listener.shutdown.skipped.
  • [Fix] Stop routing defaults from overwriting a topic's own consumer, kafka, max_messages, max_wait_time, initial_offset and consumer_persistence.
  • [Fix] Stop routing kafka(inherit: true, ...) from changing the given settings hash. A frozen hash raised FrozenError and a hash shared by many topics stopped inheriting the root kafka settings after the first topic.
  • [Fix] Fail karafka topics plan, migrate, align, create, delete, repartition and reset (exit 1) before any change when any declarative topic, also a missing or skipped one, has config keys unknown to the cluster. They now need DescribeConfigs on declared topics with config. With no topic to describe, new topics are checked only when Create is allowed on them.
  • [Fix] In independent DLQ, a message that failed after a recovered one in the same batch saw #attempt 0 and got one retry more than max_retries. In Pro, a custom DLQ strategy: also saw the errors of the recovered message.
  • [Fix] Make the Datadog LoggerListener finish its own job span, not the active one, also on job errors. With dd-trace-rb's Karafka integration active too (e.g. datadog/auto_instrument), jobs nested into one never-ending trace and memory grew without limit. It now warns once when both are active.
  • [Fix] Log lag_compensation.refresher.error in the Datadog LoggerListener through Karafka.logger at warn level, not to stderr via Kernel#warn.
  • [Fix] Treat allow.auto.create.topics as on only when it is true or "true". Any other value (e.g. "false") now reports a missing subscribed topic as an error instead of waiting with a "Waiting for it to be created" log. This includes values librdkafka reads as true, e.g. 1 and "TRUE".
  • [Fix] State the real minimum (10) in the routing max_wait_time validation error. It said "bigger than 0" for values from 1 to 9 that the rule rejects.
  • [Fix] Remove the **args parameter and its documented kafka: option from the top-level routing subscription_group. They never worked: any keyword argument raised a generic ArgumentError. Keyword arguments now raise ArgumentError: no keywords accepted.
  • [Fix] Give each swarm node its own stable transactional.id (<transactional.id>_<node id>, as with group.instance.id), so nodes of a swarm with a transactional default producer no longer fence each other. A swarm with one node keeps the configured id.
  • [Fix] [Pro] Use the producer set in swarm.node.after_fork for scheduled messages and recurring tasks when they use the default producer, instead of the node producer it replaced.
  • [Fix] Keep topics drawn without an explicit subscription_group in one default subscription group of their consumer group, also when a subscription_group block is drawn between them. Before, each block started a new default subscription group, so routes with this layout now get fewer subscription groups and Kafka connections. With Pro, those topics also keep the consumer-group-level multiplexing settings.
  • [Fix] Reject topics whose names differ only by namespacing style (e.g. a.b and a_b) in one consumer group or share group also when other namespaced topics follow them. With strict_topics_namespacing on, a later namespaced topic without such a collision hid the error.
  • [Fix] Keep topics drawn without an explicit subscription_group in one default subscription group of their consumer group, also when a subscription_group block is drawn between them, so such routes now get fewer subscription groups and Kafka connections. With Pro, those topics also keep the consumer-group-level multiplexing settings.

Don't miss a new karafka release

NewReleases is sending notifications on new releases.