zio/zio-kafka v2.3.0

on GitHub

Consumer’s performances

This release brings many performance improvements to zio-kafka’s Consumer, getting closer to the performances of the native Java kafka-clients’ Consumer.

Because of these changes, the broker will be polled more often which can cause higher CPU and memory consumption. You may need to re-tune the consumer settings. For example, you can lower max.poll.records when there is high memory pressure, and you can increase pollTimeout when you observe high CPU usage and latency is not a concern.

Our benchmarks are showing that with this new version of zio-kafka, to consume a million messages, zio-kafka’s Consumer now takes ~737.614 ± 19.178 ms while the native Java kafka-clients one takes ~640.530 ± 1.855 ms

For comparison, the Consumer of zio-kafka v2.2 was taking ~1950.181 ± 590.420 ms with very high variability in the benchmark results:

// v2.2
Result "zio.kafka.bench.ConsumersComparisonBenchmark.zioKafka":
1950.181 ±(99.9%) 590.420 ms/op [Average]
(min, avg, max) = (1015.872, 1950.181, 4967.793), stdev = 1192.678
CI (99.9%): [1359.761, 2540.602] (assumes normal distribution)

while now, the results are much more stable:

// v2.3.0
Result "zio.kafka.bench.ConsumersComparisonBenchmark.zioKafka":
737.614 ±(99.9%) 19.178 ms/op [Average]
(min, avg, max) = (677.381, 737.614, 820.314), stdev = 38.740
CI (99.9%): [718.437, 756.792] (assumes normal distribution)

See details of the current performance results here: #809 (comment)

Note that these benchmarks were run on my local machine, a MacBook Pro 2019 Intel 2.4Ghz 8 cores 64GB 2667 MHz DDR4.
To reproduce these results, you can run this sbt command on any version of zio-kafka containing these benchmarks:

sbt "clean;Test/compile;zioKafkaBench/Jmh/run -wi 10 -i 10 -r 1 -w 1 -t 1 -f 5 -foe true .*ConsumersComparisonBenchmark*"

If you monitor your Consumers' performances in production, don't hesitate to come to the zio-kafka channel of the ZIO Discord to share the changes you could have observed with this new version! ❤️

Breaking changes

• The AdminClient::listConsumerGroupOffsets(groupSpecs: Map[String, ListConsumerGroupOffsetsSpec]) and AdminClient::listConsumerGroupOffsets(groupSpecs: Map[String, ListConsumerGroupOffsetsSpec], options: ListConsumerGroupOffsetsOptions) methods were returning a Task[Map[TopicPartition, OffsetAndMetadata]] and are now returning a Task[Map[String, Map[TopicPartition, OffsetAndMetadata]]]

The String in the key of the Map being the Consumer Group IDs (same ones as the ones you passed in parameters)

• The AdminClient.fromManagedJavaClient has been renamed AdminClient.fromScopedJavaClient

• The ConsumerSettings::perPartitionChunkPrefetch Consumer configuration has been removed

Deprecations

• The ZIOSpecWithKafka trait in zio-kafka-test-utils is deprecated

See https://github.com/zio/zio-kafka/blob/master/zio-kafka-test-utils/src/main/scala/zio/kafka/ZIOSpecWithKafka.scala#L7-L23 to see how to remove it from your code.

Other noticeable changes

• When instantiating a client (AdminClient, Producer or Consumer) zio-kafka is now checking that your client SSL configuration is valid for your server SSL configuration, failing if the client configuration is incorrect.

This check prevents your app from crashing with an OOM a few minutes after you start the client.
This check protects you against a long-standing bug present in kafka-clients, see https://issues.apache.org/jira/browse/KAFKA-4090

This check is provided to you by Conduktor, backported from their internal fork of zio-kafka.

For more details about this check, you can have a look at its code here: https://github.com/zio/zio-kafka/blob/master/zio-kafka/src/main/scala/zio/kafka/utils/SslHelper.scala

All changes

• Optimistically resume partitions early @erikvanoosten (#809)
• Performances: Remove Scala-- code @guizmaii (#834)
• Update scala-collection-compat to 2.10.0 @github-actions (#833)
• Backport Conduktor's improvements @guizmaii (#781)
• Update jackson-databind to 2.15.0 @scala-steward (#831)
• Small improvements taken from #788 @erikvanoosten (#829)
• Update logback-classic to 1.3.7 @scala-steward (#827)
• Fix Scala3 build @guizmaii (#824)
• Update embedded-kafka to 3.4.0.1 @github-actions (#823)
• Update zio-sbt-ci and CI definition @guizmaii (#822)
• Add a minimal example app, useful to tests things manually if necessary @guizmaii (#807)
• Fix CI @guizmaii (#820)
• Update README.md @guizmaii (#819)
• Update zio, zio-streams, zio-test, ... to 2.0.13 @github-actions (#818)
• New code owners part 2 @erikvanoosten (#812)
• Use finalizer to ensure seeing de-register in unit test @erikvanoosten (#813)
• Remove unnecessary nowarn @erikvanoosten (#810)
• New code owners @erikvanoosten (#811)
• ConsumerAccess::runloopAccess should be private @guizmaii (#808)
• Backport improvement made in zio/zio#7996 @guizmaii (#805)
• Run the Runloop on a dedicated single-threaded thread pool @guizmaii (#802)
• Optimize offerRecordsToStreams code @guizmaii (#791)
• Update zio, zio-streams, zio-test, ... to 2.0.12 @github-actions (#793)
• Fix CI @guizmaii (#792)
• Update ZIO @guizmaii (#787)
• Simplify CI definition @guizmaii (#786)
• Improve stream restart in restartStreamsOnRebalancing mode @erikvanoosten (#785)
• Update README.md @github-actions (#782)
• Simpler shutdown handling @svroonland (#778)

🌱 Dependency Updates

• Bump peter-evans/create-pull-request from 5.0.0 to 5.0.1 @dependabot (#837)
• Bump benchmark-action/github-action-benchmark from 1.16.2 to 1.17.0 @dependabot (#835)
• Bump scala-steward-org/scala-steward-action from 2.51.0 to 2.52.0 @dependabot (#814)
• Bump peter-evans/create-pull-request from 4.2.3 to 5.0.0 @dependabot (#816)
• Bump peter-evans/create-pull-request from 4.2.3 to 5.0.0 @dependabot (#789)
• Bump actions/setup-java from 3.10.0 to 3.11.0 @dependabot (#790)
• Bump actions/setup-java from 3.10.0 to 3.11.0 @dependabot (#779)