This is mostly a maintenance release with small bug fixes. It also upgrades the kafka client to 4.1.1 (compatible with most kafka brokers). However, if you care about partition assignors, please read on.
Many thanks for the bug reports and PRs ❤️
Note: version 3.1.1 never made it due to small binary compatibility problems. Use 3.2.0 instead.
👨🏽💻 Looking for contributors
For the current maintainers zio-kafka is just for fun; we no longer use this project in our day jobs. If you find it challenging to work on this very interesting code base, please join us on Discord and we will help you where we can.
🚧 Partition assignor changes
If your application uses rebalance-safe-commits, AND it explicitly sets a partition assignor, please read this section. Otherwise, there should be no issues, and you can safely upgrade to 3.2.0.
After some bug reports we discovered that not all kafka's partition assignors can be used in all circumstances. Therefore, zio-kafka can now change the default partition assignor(s), and also disallow invalid partition assignor settings.
First of all, we discovered that kafka's CooperativeStickyAssignor should only be used in combination with rebalance-safe-commits, when the consumer is very very fast — it should be fast enough to process all pre-fetched records within the maximum rebalance time. Determining whether this is the case requires measuring your system and careful consideration of all settings. When this cannot be guaranteed, it is better to use the RangeAssignor. Therefore, when rebalance-safe-commits is enabled, zio-kafka changes the default partition assignors to [RangeAssignor]. If you wish, you can still set the partition assignors to something else, for example kafka's original default: [CooperativeStickyAssignor, RangeAssignor].
Secondly, we discovered that kafka's CooperativeStickyAssignor is incompatible with transactional producing. If you use transactional, zio-kafka now enforces that the CooperativeStickyAssignor is not used when the consumer is constructed.
For more information see #1597.
🚀 Changes
- Make sure producer can shut down @erikvanoosten (#1590)
- Await completion of some admin operations @erikvanoosten (#1611)
- Small doc tweaks @erikvanoosten (#1610)
- Fix embedded-kafka version @Xitys62owt (#1607)
- No CooperativeStickyAssignor for transactional, RangeAssignor for rebalance-safe-commits @erikvanoosten (#1597)
- Validate consumer settings @erikvanoosten (#1580)
🌱 Dependency updates
- Update kafka-clients to 4.1.1 @zio-scala-steward[bot] (#1608)
- Update zio, zio-streams, zio-test, ... to 2.1.22 @zio-scala-steward[bot] (#1595)
- Update scala3-compiler, scala3-library, ... to 3.3.7 @zio-scala-steward[bot] (#1594)
- Update scala-library to 2.13.17 @zio-scala-steward[bot] (#1591)
🔧 Maintenance
- Bump coursier/cache-action from 6 to 7 @dependabot[bot] (#1604)
- Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.1 @zio-scala-steward[bot] (#1614)
- Update logback-classic to 1.5.21 @zio-scala-steward[bot] (#1606)
- Update README.md @zio-assistant[bot] (#1609)
- Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.36 @zio-scala-steward[bot] (#1602)
- Stabilize flaky consumer tests with a sleep @erikvanoosten (#1596)
- Update logback-classic to 1.5.20 @zio-scala-steward[bot] (#1598)
- Update sbt-scalafix to 0.14.4 @zio-scala-steward[bot] (#1593)
- Stabilize tests @erikvanoosten (#1592)
- Update sbt, scripted-plugin to 1.11.7 @zio-scala-steward[bot] (#1589)
- Update logback-classic to 1.5.19 @zio-scala-steward[bot] (#1586)
- Update sbt-native-packager to 1.11.4 @zio-scala-steward[bot] (#1587)
- Update README.md @zio-assistant[bot] (#1585)
- Update the readme on every push to master @erikvanoosten (#1584)
Full Changelog: v3.1.0...v3.2.0