This release contains many performance enhancements and internal code optimizations to the consumer side of zio-kafka. The Runloop, the heart of the Consumer, has been optimized to reduce the polling and streaming latency and be able to provide higher throughput, as well as less GC pressure. This results in a throughput that is 2 to 5 times higher.
The Runloop polls the underlying Kafka client for new data. It also applies backpressure for streams that do not require data by pausing partitions. When the stream needs more data, the partition is resumed. Previously the Runloop would poll every pollInterval
. Now it polls continuously. When all streams stop processing, data polling stops so that the broker can detect that this Kafka client is stalled.
Several other changes to be aware of:
OffsetBatch.merge
was deprecated in favour ofadd
.- Consumer setting
pollInterval
was removed because polling is now continuous. We recommend you useConsumerSettings(bootstrapServers)
and thewith*
methods to build consumer settings to prevent future source incompatibilities. - Consumer setting
runloopTimeout
was added. When the runloop has nothing to do for this period, it will fail.
Changes
- Increase default runloop timeout @erikvanoosten (#775)
- Give
ConsumerSettings.runloopTimeout
a default @erikvanoosten (#774) - Fix restartStreamsOnRebalancing @erikvanoosten (#772)
- Improve readability @erikvanoosten (#773)
- Update scalafmt-core to 3.7.3 @github-actions (#769)
- Add
better-monadic-for
scalac plugin @guizmaii (#765) - Improve readability @guizmaii (#763)
- Remove unused
offsetRetrieval
field fromChangeSubscription
command @guizmaii (#764) - Fix
getConsumerGroupMetadataIfAny
: It shouldn't access the low-level consumer directly @guizmaii (#762) - Optimize
offerRecordsToStreams
code @guizmaii (#761) - Optimise
handlePoll
code: Ensure that bothrunningStreams
andnewAssignedStreams
are Chunks to take advantage of the optimised version of++
when computing theupdatedStreams
@guizmaii (#760) - Divide allocations: Instantiate the
OffsetImpl
as late as possible to avoid allocatiing oneOffsetImpl
and oneTopicPartition
for each consumed Record @guizmaii (#759) - add test case for the runloop polling behavior @tnielens (#684)
- Optimise
offerRecordsToStreams
code @guizmaii (#756) - Tune benchmark parameters @svroonland (#729)
- Review of PR #727: Remove unused
Request::dataQueue
@guizmaii (#747) - Improve performances: Remove the
aggregateAsync
call @guizmaii (#751) - Improve Logback configuration in benchmarks @guizmaii (#750)
- Review of PR #727 @guizmaii (#745)
- Fix
ConsumersComparisonBenchmark
: Avoid booting the Consumer instance in the bench, which could add noise in the bench results @guizmaii (#748) - Remove buffered records @erikvanoosten (#744)
- ZStream based implementation of poll loop @svroonland (#736)
- Feed streams through queue instead of promise @erikvanoosten (#727)
- First seek then use for manual assignments @erikvanoosten (#726)
- Stabilize benchmark @svroonland (#714)
- Fix CI @guizmaii (#723)
- Optimise
Runloop::run
code @guizmaii (#721) - Add profiling in the CI @guizmaii (#696)
- Init comparison benchmarks: compare kafka-clients vs zio-kafka consumers @guizmaii (#695)
- Rollbacks @guizmaii (#722)
- Update logback-classic to 1.4.6 @github-actions (#717)
- Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.3.10+94-cbe49d51-SNAPSHOT @github-actions (#718)
- Make as much computation as possible in
fulfillRequests
code @guizmaii (#720) - Rename
OffsetBatch::merge(Offset)
toOffsetBatch::add(Offset)
to improve readability @guizmaii (#700) - Apply ZIO SBT Ecosystem and CI Plugins @khajavi (#702)
- Improve polling behavior @svroonland (#683)
- Update logback-classic to 1.3.6 @github-actions (#709)
- Update README.md @github-actions (#693)
- Consumer benchmark @svroonland (#689)
🌱 Dependency Updates
- Bump hmarr/auto-approve-action from 3.2.0 to 3.2.1 @dependabot (#734)
- Bump hmarr/auto-approve-action from 3.1.0 to 3.2.0 @dependabot (#705)
- Bump benchmark-action/github-action-benchmark from 1.16.0 to 1.16.2 @dependabot (#704)