In this release zio-kafka starts tracking the actual next-offset instead of the last consumed offset plus 1. While before a client would always report some lag, with this change, lag can go down to 0.

About next-offsets

Kafka consumers are designed to start consuming records from the offset that was previously committed. When you commit, you therefore have to commit the next offset. Up to recently, the only way to get the next offset was by simply adding 1 to the offset of a just consumed record (zio-kafka does this for you automatically). However, since KIP-1094 we can explicitly ask for the next offset. The explicit next offset is not always the same as the last consumed offset plus 1. This is because of control records. Control records are not visible to users of a Kafka client, but they do have an offset!

In this release we annotate each record with the next-offset. When a record is followed by a normal record (the common case), its next-offset is the record's offset + 1. When a record is followed by n control records, the record's next-offset is the record's offset + n + 1.

See also KIP-1094 for more information.

🚧 Backward incompatible changes

This is a backward incompatible release! OffsetBatch loses offset and gains nextOffset. Only people that combine offsets before committing/storing those outside of Kafka are affected.

• Commit actual next offset by @erikvanoosten in #1778

🔧 Maintenance

• Update README.md by @zio-assistant[bot] in #1812

Full Changelog: v3.8.2...v4.0.0