- [EOL] Drop Ruby 3.2 support; require Ruby 3.3+.
- [EOL] Drop Rails 7.0 and 7.1 compatibility; require Rails 7.2+.
- [Feature] Add
Instrumentation::Vendors::NewRelic::MetricsListenerfor publishing Karafka metrics to New Relic. Context is encoded in the metric name, as New Relic custom metrics do not support tags (svyatmuzyka). - [Feature] Add
karafka info --extended, printing the routing tree, global app config and effective Kafka config, with sensitive values redacted. - [Feature] Add
Kubernetes::ReadinessListener, serving a readiness probe that reports healthy once all subscription groups poll and not-ready on shutdown or quieting, so pods drain before exiting. - [Feature] Allow the
--include/--excludeCLI server filters to accept wildcard patterns (e.g.--exclude-consumer-groups app-a-*), including topics discovered at runtime by Pro routing patterns. - [Feature] Add
Karafka::Admin.list_consumer_groupsto list every consumer group in the cluster with its state. Requires karafka-rdkafka>= 0.31.0. - [Feature] Add
Karafka::Admin.describe_consumer_groupsto read consumer groups' state, coordinator, assignor and members with their partition assignments. Requires karafka-rdkafka>= 0.31.0. - [Feature] Add
Karafka::Admin.describe_clusterand.describe_topicsto read the cluster id, controller, brokers with racks, topic ids, partition leaders and replicas, and optionally the authorized operations. Requires karafka-rdkafka>= 0.31.0. - [Feature] Add
Karafka::Admin.delete_recordsto delete the messages of given partitions below a given offset (or all of them with:end) by moving their low watermark. It is destructive and irreversible. - [Feature] Add
Karafka::Admin.delete_consumer_group_offsetsto delete a consumer group's committed offsets for given topics or partitions, keeping the consumer group and its other offsets. RaisesKarafka::Errors::ConsumerGroupSubscribedToTopicErrorfor topics the consumer group is subscribed to. - [Feature] [Pro] Add Virtual Partitions Estimator, a console API that samples a topic and estimates how well Virtual Partitions would parallelize it. Results can be saved as a versioned JSON report (
target_file:) and displayed withkarafka vps estimate-report. - [Feature] Add
Admin::Configs::Config#source(e.g.:dynamic_topic_config,:dynamic_broker_config,:default_config), also on synonyms, telling a value set on the topic from one inherited from the broker or a built-in default. Requires karafka-rdkafka>= 0.32.1. - [Enhancement] Defer default log file creation until the first write and keep logging working (to stdout) when the log file cannot be created or written, e.g. on read-only filesystems (ydah).
- [Enhancement] Stop allocating 6 objects per consumed message: build the default topic deserializers only once and read
max_messagesonce per poll batch. - [Enhancement] Stop allocating a keyword
Hashper consumed message when building its metadata. - [Enhancement] Log only the top two backtrace lines for
librdkafka.errorin theLoggerListener, so unreachable brokers no longer flood the logs. - [Enhancement] Include the topic, partition and offset from librdkafka error details (e.g. a message that failed to decompress) in
LoggerListenererror logs. Only librdkafka errors are used, and the consumer's own context takes precedence. - [Enhancement] Instrument
client.topics.unknownand log it at the info level when a subscribed topic does not exist yet andallow.auto.create.topicsis on, instead of ignoring it silently. - [Enhancement] Support the
:consumer_groupresource type inKarafka::Admin::Configs, so consumer group configs can be described and altered, e.g.consumer.session.timeout.msandconsumer.heartbeat.interval.ms, whichgroup.protocol: "consumer"(KIP-848) does not take from the client config. - [Change] Raise
UnsupportedOptionErrorwith the Ruby engine name (e.g.jruby) when Swarm runs on a Ruby withoutProcess.fork, also whenSwarm::Supervisor#runis called directly, before any work is done. - [Change] Require
karafka-rdkafka>=0.32.2: lag and offset reads pass the partition list ofAdmin#list_consumer_group_offsetsas thetopic_partition_list:keyword, required since 0.32.1 (0.32.2 is the same code, released with Sigstore attestations). - [Change] Require
karafka-rdkafka>=0.32.0. A failed consumer close is now reported as alibrdkafka.erroronerror.occurredinstead of being dropped. A close fails in practice only after a fatal error, e.g. when a static membership conflict fenced the consumer. Polling usually reports that error first, so it is reported twice: asconnection.client.poll.errorand on close. - [Change] Fail
karafka topics alignandmigrate(exit 1) when an existing declarative topic has config keys the cluster does not know (e.g. a typo), naming them, instead of skipping them silently.planlists such keys as unknown and fails too. - [Change] Raise
Karafka::Errors::ConsumerGroupNotEmptyErrorwhenKarafka::Adminchanges the offsets of a consumer group with active members, instead of silently leaving them unchanged. With an external consumer client, offsets are read and changed for the queried consumer group, not the client's own. - [Change] Apply the routing
defaultsblock to consumer-group topics only. Share-group topics get no defaults, so consumer-group-only calls in it (e.g.config(...),dead_letter_queue) no longer raiseNoMethodErrorwhen a share group is drawn. - [Change] Require
karafka-core>=2.6.8. - [Change] Apply
strict_declarative_topicsto share-group topics too: each one needs an active declaration indeclaratives.draw, made beforeroutes.draw. - [Maintenance] Foundational work for Kafka share groups (KIP-932) is in progress (routing layer, consumer class hierarchy, config seams). Not usable yet - share groups can be described in routing but cannot run.
- [Fix] [Pro] Do not let healthy KIP-848 (
group.protocol=consumer) multiplexed listeners that never received an assignment block dynamic scaling. With more listeners than partitions they stay in theinitjoin state with a rebalance age of 0, so the manager never saw them as stable and kept them alive instead of scaling them down. - [Fix] Stop
CoordinatorsBuffer#@coordinatorsfrom growing unbounded across rebalances by dropping a topic entry on revoke once it tracks no partitions (mirrorsPausesManager#delete). - [Fix] Use
::JSON.parsein the ActiveJob deserializer, so consuming ActiveJob messages works with the json gem>= 3.0. - [Fix] [Pro] Honour the read_committed isolation level in consumer-group lag compensation, so an in-flight transaction no longer overstates the lag (needs karafka-rdkafka
>= 0.30.0). - [Fix] Run the
Karafka::Server.stopsequence once when two stops are requested at the same time (e.g. TERM then INT on JRuby): the state check and thestoppingtransition are now one atomic step. - [Fix] Point the non-Rails
karafka.rbinstall template comment to thekarafka installcommand instead of the non-existentkarafka:install. - [Fix] Make
karafka consoleexplain that theirbgem must be added to the Gemfile, instead of failing with a Bundler error, when irb is not in the bundle (non-Rails apps on Ruby 4.0+). - [Fix] Remove the subscription group error callback only after the consumer is closed, so errors reported while closing it (e.g. a failed close) are published via
error.occurred. - [Fix] Set
Message#received_atto the moment each message was polled, not the moment its batch was built, so it no longer trails by up tomax_wait_timeon low traffic topics (#2292). - [Fix] Stop the listener from failing with
NoMethodErroron JRuby when it checks a subscription group's jobs while workers complete them. - [Fix] Stop
#inline_insightsfrom returning no data on JRuby while another thread adds or evicts partition statistics. - [Fix] Restore
InlineInsights::Tracker.exists?, which raisedNoMethodErrorbecause its delegated#exists?was removed. - [Fix] Stop
Adminoperations from retrying forever when their result never becomes visible. They now raiseResultNotVisibleErrorafteradmin.max_retries_duration. - [Fix] Keep the Datadog
LoggerListenerjob types cache consistent on JRuby when many workers fill it at once. - [Fix] Reset ActiveJob current attributes also when restoring them from the job payload fails part way, so they no longer stay on the worker thread. Attributes the consumer does not declare are now skipped and a non-Hash attributes payload raises
Karafka::Errors::InvalidConfigurationError. - [Fix] [Pro] Stop an LRJ retry pause from staying paused forever when the listener resumed the partition before the client pause landed.
- [Fix] [Pro] Validate all four schedule headers of scheduled messages. Only the cancel headers were checked, so a schedule missing
schedule_target_topicpassed and one missingschedule_target_epochraisedKeyError. - [Fix] Stop forceful shutdown from using a listener's client from two threads at once on JRuby. A listener still running after termination now keeps its client open and is reported via
connection.listener.shutdown.skipped. - [Fix] Stop routing
defaultsfrom overwriting a topic's ownconsumer,kafka,max_messages,max_wait_time,initial_offsetandconsumer_persistence. - [Fix] Stop routing
kafka(inherit: true, ...)from changing the given settings hash. A frozen hash raisedFrozenErrorand a hash shared by many topics stopped inheriting the rootkafkasettings after the first topic. - [Fix] Fail
karafka topicsplan,migrate,align,create,delete,repartitionandreset(exit 1) before any change when any declarative topic, also a missing or skipped one, has config keys unknown to the cluster. They now needDescribeConfigson declared topics with config. With no topic to describe, new topics are checked only whenCreateis allowed on them. - [Fix] In independent DLQ, a message that failed after a recovered one in the same batch saw
#attempt0and got one retry more thanmax_retries. In Pro, a custom DLQstrategy:also saw the errors of the recovered message. - [Fix] Make the Datadog
LoggerListenerfinish its own job span, not the active one, also on job errors. With dd-trace-rb's Karafka integration active too (e.g.datadog/auto_instrument), jobs nested into one never-ending trace and memory grew without limit. It now warns once when both are active. - [Fix] Log
lag_compensation.refresher.errorin the DatadogLoggerListenerthroughKarafka.loggeratwarnlevel, not to stderr viaKernel#warn. - [Fix] Treat
allow.auto.create.topicsas on only when it istrueor"true". Any other value (e.g."false") now reports a missing subscribed topic as an error instead of waiting with a "Waiting for it to be created" log. This includes values librdkafka reads as true, e.g.1and"TRUE". - [Fix] State the real minimum (10) in the routing
max_wait_timevalidation error. It said "bigger than 0" for values from 1 to 9 that the rule rejects. - [Fix] Remove the
**argsparameter and its documentedkafka:option from the top-level routingsubscription_group. They never worked: any keyword argument raised a genericArgumentError. Keyword arguments now raiseArgumentError: no keywords accepted. - [Fix] Give each swarm node its own stable
transactional.id(<transactional.id>_<node id>, as withgroup.instance.id), so nodes of a swarm with a transactional default producer no longer fence each other. A swarm with one node keeps the configured id. - [Fix] [Pro] Use the producer set in
swarm.node.after_forkfor scheduled messages and recurring tasks when they use the default producer, instead of the node producer it replaced. - [Fix] Keep topics drawn without an explicit
subscription_groupin one default subscription group of their consumer group, also when asubscription_groupblock is drawn between them. Before, each block started a new default subscription group, so routes with this layout now get fewer subscription groups and Kafka connections. With Pro, those topics also keep the consumer-group-levelmultiplexingsettings. - [Fix] Reject topics whose names differ only by namespacing style (e.g.
a.banda_b) in one consumer group or share group also when other namespaced topics follow them. Withstrict_topics_namespacingon, a later namespaced topic without such a collision hid the error. - [Fix] Keep topics drawn without an explicit
subscription_groupin one default subscription group of their consumer group, also when asubscription_groupblock is drawn between them, so such routes now get fewer subscription groups and Kafka connections. With Pro, those topics also keep the consumer-group-levelmultiplexingsettings.