Consensys/teku 22.9.1

on GitHub

This is a recommended update containing improvements to post-merge performance issues on MainNet.
There are no changes from the previous 22.9.1-RC1 so users who are running the RC do not need to upgrade.

Downloads

• Available as 22.9.1 on Dockerhub
• Download the binary distribution:
  • tar.gz (sha256: a4ab66e30cc24c99d14f91d31d95443cb0d12c984521837ec72e114af5cb5fd9)
  • zip (sha256: 826d7be0966d86281f1c18dfe9fd58c1881561918ffa2b5aa6b1f33e82422a80)

Breaking Changes

• The --initial-state and --eth1-deposit-contract-address options has been removed from the validator-client subcommand. They have been ignored for some time but are now completely removed.
• LevelDB configuration has been changed to increase the maximum number of files it may open at a time. Systems with low open file handle limits may encounter issues.
  We recommend at least 2048 file handles or preferably unlimited. The ulimit command can be used to check and configure file handle limits.

Performance Improvements

This release significantly improves Teku's performance post-merge, but we will continue to investigate further improvements.

Please note that if the execution client is slow to respond, particularly when importing new blocks, that will increase the rate of incorrect head votes and increase the likelihood of attestations not being included.

To see more information on where time is being spent when importing blocks, users can enable late block alerts with the --Xmetrics-block-timing-tracking-enabled option. This option will log a warning when a block import completes more than 4 seconds into the slot (after attestations will be created) with a break down of where time was spent. For example:

Late Block Import *** Block: e83a322a714b1b364ab59b2cb8666076fa015a8a5b4f1a8b9829306fc249f787 (4718961) proposer 425317 arrival 681ms, pre-state_retrieved +11ms, processed +306ms, execution_payload_result_received +3173ms, begin_importing +0ms, transaction_prepared +0ms, transaction_committed +1ms, completed +15ms

The labels used for timings are somewhat Teku developer focussed, but some key steps are:

• arrival - the time since the start of the slot that the block was first received. Large values here usually indicate that the block was produced late.
• state_retrieved - the time it took to prepare the pre-state to apply the block to.
• processed - the time it took Teku to process the block.
• execution_payload_result_received - the time Teku spent waiting for the execution client to finish importing the block.

The remaining entries are Teku processing time to complete importing the block, updating fork choice records and storing the data to disk.

So in the example above the block was late to import because of a slow response from the execution client.

Additions and Improvements

• Enables asynchronous database updates by default. This ensures slow disk access or LevelDB compactions don't cause delays in the beacon node
• Increased leveldb open files to 1000 files by default. If teku fails to start, update maximum open files to 2048 or unlimited
• Reduced number of blocks held in in-memory cache to reduce need for old-gen GC with the larger post-merge blocks
• Make Validator Client connect to a failover event stream (if failovers are configured) when the current Beacon Node is not synced
• Detect Lodestar clients in libp2p_connected_peers_current metrics
• Reduce CPU and Memory consumption in shuffling, which will improve epoch transition performance
• Faster peer discovery on startup

Bug Fixes

• Fixed an issue where epoch cache priming sometimes happened synchronously, causing delays in time tick processing in the beacon node
• Fix /eth/v1/validator/register_validator responding with a 400 status code and a misleading error message in case of exceptions
• Update snakeyaml dependency to resolve cve-2022-25857 which could result in excessive memory usage when parsing YAML content
• Fixed an issue where the range requested for deposit logs was not reduced when using only --ee-endpoint leading to persistent timeouts with execution clients

Upcoming Breaking Changes

• The validator_beacon_node_published_attestation_total, validator_beacon_node_published_aggregate_total,
  validator_beacon_node_send_sync_committee_messages_total, validator_beacon_node_send_sync_committee_contributions_total
  and validator_beacon_node_published_block_total metrics have been deprecated in favour of the new validator_beacon_node_requests_total metric.
  The old metrics will be removed in a future release. An update to the Teku Dashboard that uses the new metric is available.
• The /eth/v1/debug/beacon/states/:state_id endpoint has been deprecated in favor of the v2 Altair endpoint /eth/v2/debug/beacon/states/:state_id
• The /eth/v1/beacon/blocks/:block_id endpoint has been deprecated in favor of the v2 Altair endpoint /eth/v2/beacon/blocks/:block_id
• The /eth/v1/validator/blocks/:slot endpoint has been deprecated in favor of the v2 Altair endpoint /eth/v2/validator/blocks/:slot
• The /eth/v1/debug/beacon/heads endpoint has been deprecated in favor of the v2 Bellatrix endpoint /eth/v2/debug/beacon/heads