The main focus of this upgrade is stability improvements with the IAVL patch of v6.3.0 The big points are:
- Fixes the "version X was already saved to a different hash" and "insufficient funds" bugs that nodes were sometimes seeing after crashes.
- Reduce the RAM overhead during the initial IAVL migration. Node operators should still have 32GB of RAM for all of their mainnet nodes.
- Improving the log messages during migrations
The bugs people were seeing for nodes being saved to a different hash, etc. happen after restoring the node post-crash during commit (the crash could be caused by something else). The bug on restart stems from the long-standing issue that SDK commits are actually not-atomic. This manifested in a failure to flush the latest height when an error during commit occurs, causing the state invalidities on restart. We mitigated the non-atomicity problem by flushing the metadata about the stores that successfully committed a new height. As a result, on a restart, the SDK now knows which stores/modules are already on the newly committed height.
There is still a known issue related to a misconfiguration between pruning and snapshot settings. During state sync, it is now possible to apply the blocks fast enough to begin pruning a height that is currently being snapshotted.
We are working on a solution but for now, we suggest setting your app.toml to the following:
pruning-keep-recent=10000
pruning-keep-every=0
pruning-interval=<random small prime under 100>
snapshot-interval=1500
snapshot-keep-recent=2
It is also possible to set pruning-keep-every == snapshot-interval. This ensures that a snapshotted height is never pruned. However, there are other problems known to occur when pruning-keep-every is set. Therefore, we recommend increasing pruning-keep-recent to 10000 for now instead.
What's Changed
- Move app.setupUpgradeHandlers() after module manager and configurator (backport #884) by @mergify in #885
- Upgrade IAVL and SDK with RAM improvements and bug fixes for v6.4 by @p0mvn in #907
Full Changelog: v6.3.1...v6.4.0