Release: 0.26.4

This release comes with an important feature for users that have yet to, or cannot, update to 1.0. Note that we still recommend you to update as soon as possible, as 1.0 is not affected by this issue.

One thing users may have noticed in the past - pre 1.0 - is that over time, snapshots in Zeebe can get highly fragmented. In some cases, we're talking about snapshots having hundreds of thousands of very small files. This causes performance issues at startup, during failover, and during replication, as verifying the integrity of the snapshot means opening and verifying each file.

With 0.26.4, we introduced an experimental feature which will enforce manual compaction of your snapshots. This feature can be controlled via a configuration parameter, zeebe.broker.experimental.rocksdb.enableManualCompactionOnRecovery, and is on by default. This means that during failover, if a snapshot has a file count exceeding a certain threshold (by default 1000 files, but configurable via zeebe.broker.experimental.rocksdb.triggerFileCountManualCompaction), the broker will block and force compaction of this snapshot. Note that if you have a highly fragmented snapshot, this can be a time consuming operation, and during that time throughput for that partition will go down to 0. In our tests, we found that with 1000 files, the overhead was minimal, but with hundreds of thousands of files, it could take up to 20 minutes to compact. We still recommend users to make use of this, but keep an eye out for failover metrics and adjust the parameters accordingly.

Enhancements

Broker

• Force compaction of state to reduce the number of files in a snapshot. (#7086)

Bug Fixes

Broker

• Back to back role changes cause ZeebePartition to go unhealthy and transition to inactive (#6997)
• Memory leak in NettyMessagingService (#6878)
• Follower doesn't flush written entries if append is only partially written (#6784)
• Gateway not able to parse ipv6 addresses (#5951)

Misc

• Snapshot log statement is incorrectly formatted (#6853)

Merged Pull Requests

• Enable force compaction on state to reduce the number of files (#7103)
• [Backport stable/0.26] fix(broker): use correct term and role when reacting to install failures (#7063)
• Add new rocksdb configurations (#6989)
• [Backport stable/0.26] Fixes memory leak in messaging service (#6973)
• [Backport 0.26] fix(broker): format snapshot log statement correctly (#6888)
• Disable BPMN TCK (#6819)
• [Backport 0.26] fix(broker): flush write if any entry was appended (#6801)
• Ignore checksum files when loading snapshots on startup (#6797)
• fix(gateway, broker, clients/java): allow to use IPv6 addresses in a cluster (#6792)