The 4.0 release of Antelope Leap focuses on optimizing node performance, improving block propagation, scheduling future snapshots, adding read-only transactions, and improving performance of EVM related functions.
Antelope Leap v4.0.0-rc3 includes bug fixes, numerous updates, and several improvements. Key features include:
- The ability to specify a dedicated port for Prometheus metrics
- Support for
-k/--kill - Restoration of
read-mode=speculative
Leap v4.0.0-rc3 Release Notes
New Features
Prometheus on a dedicated port
The prometheus_plugin added a new URL for prometheus metrics, but it used the same port as the producer plugin, which needs to be restricted. The prometheus export needs to run on a separate port. In v4.0.0-rc3, we added a new option, prometheus-exporter-address, to specify the IP and port to listen for incoming prometheus metrics http request. A simple embedded server handles HTTP requests and exports Prometheus metrics. Now the prometheus exporter will run on a dedicated port.
Parallelize Read-only Opreations
Majority of Chain RPC get_* requests are safe to run in parallel with themselves and read-only transactions. To further improve API nodes throughputs, they are now executed on the read-only thread pool during the read-only-read-window-time-us window.
/v1/net/* calls were also moved to http thread pool, off main app thread.
--read-only-threads defaults to 3 on non-producer configured nodeos that enable the eosio::chain_api_plugin.
Block relay nodes or canary nodes should consider --read-only-threads=0. Since eosio::chain_api_plugin may be enabled so /v1/chain/get_info is available. Block relay nodes or canary nodes should consider --read-only-threads=0 if they have enabled eosio::chain_api_plugin as the read-only-thread pool is only used for read-only transactions and /v1/chain/get* calls. If read-only-threads are enabled, but not utilized, there is a very small overhead of switching back and forth between the read/write windows.
Note read and write windows are interrupted when new blocks are received.
Restore deprecated read-mode=speculative
An improved version of read-mode=speculative has been added back to nodeos. The read-mode=head remains the default. Read on for more details.
A feature called read-mode=speculative was removed because it caused confusion. When using this feature, the state of a speculative transaction was shown instead of the actual state of execution. This produced a different state than any other node and caused confusion to users.
To fix simplify things, the read-mode=speculative feature was removed and only two modes remain: HEAD and IRREVERSIBLE. This simplified things for users and reflected the actual state of execution. However, some community members raised potential use cases for the read-mode=speculative feature.
The feature has now been added back as an alternative non-default read-mode. This new restored speculative mode will not have the state of API trxs auto applied at the start of each speculative block. Users will have to re-apply any transactions at each speculative block in order to observe their side-effects. Also, the restored read-mode=speculative will not apply schedule changes, protocol activations, or onblock trxs for a speculative mode into the speculative block which will be aborted when a new block is received from the network.
Support for -k --kill in launcher
Launcher now honors -k --kill <signal_number> argument. Cluster was updated for the Python launcher but support in the launcher itself was missed. Was causing a Test failure: resource_monitor_plugin_test
Add time summary logging for transient trx processing and read-only threads
Before v4.0.0-rc3, it was difficult to keep track of time spent on dry-run, read-only, and other transactions. It was especially challenging to know how long it took to execute read-only transactions when using multiple threads. A new log summary report was introduced that provides details on the time spent executing dry-run and read-only transactions. When the option --read-only-threads is enabled, the report also includes time spent on read-only transactions across multiple threads. It is now easier to monitor the time spent on dry-run and read-only transactions, including execution time of read-only transactions across multiple threads. This will help identify any bottlenecks to the performance of the system.
Updates
Read-only Transactions and Operations Parallelization Hardening
PRs
- (885) Add time summary logging for transient trx processing and read-only threads
- (892) [4.0] Cleanup and prevent thread starvation while in the read window
- (933) Small improvements for parallelizing read-only transactions and tasks
- (947) [4.0] Correct time summary calculation for transient trxs due to change of read-only trxs windows
- (960) [4.0] Use is_write_window in eos-vm-oc main to determine if it is safe to update code cache
- (905) Use a single timer for read and write windows
- (964) [4.0] Added missing calls to wasmifs on read-only threads
- (975) [4.0] Remove global WASM::check_limits and initial memory from instantiate_module
- (982) [4.0] Set an upper limit on read-only-threads of 8
- (984) [4.0] Fix ROtrx dispatched to baseline runtime when compiled code not ready
- (993) [4.0] make eosio_exit impl more direct
A number of improvements and bug fixes are included in this RC release. Main ones are
- Before v4.0.0-rc3, it was difficult to keep track of time spent on dry-run, read-only, and other transactions. It was especially challenging to know how long it took to execute read-only transactions when using multiple threads. A new timing summary report was introduced that provides details on the time spent executing dry-run and read-only transactions. When the option --read-only-threads is enabled, the report also includes time spent on read-only transactions across multiple threads. It is now easier to monitor the time spent on dry-run and read-only transactions, including execution time of read-only transactions across multiple threads. This will help identify any bottlenecks to the performance of the system.
- Previously, certain functions were not being called by wasmifs on read-only threads, which could cause memory leaks. These missing calls were added.
- It was identified a global variable called
check_limitscould cause issues when accessed by multiple threads at the same time. It was removed. - An upper limit of 8 read-only threads is enforced, regardless of whether EOS VM OC is enabled or not. This limit was chosen to prevent excessive memory usage for WebAssembly instantiation, and should be a conservative strategy to protect a typical EOS Mainnet nodeos from running out of RAM.
- Read window and write window run exclusively with each other; they do not run at the same time. A single timer is sufficient to control them and makes code simpler.
- When executing a read-only transaction with EOS VM OC is enabled, if VM OC has not yet finished compiling, the code falls back to the baseline runtime, which is not thread-safe. The code has been updated to make sure that when VM OC compile fails, read-only transactions will retry or fail permanently based on failure reasons.
- In previous implementation, a read-only thread would exit if no transactions are available, potentially leaving only one thread to complete all remaining tasks. To prevent this thread starvation, the tasks were modified to wait until other threads had finished their tasks before moving exiting. This change is expected to improve the efficiency of parallel processing of read-only transactions.
- Make it more explicit how eosio_exit works by eliminating some needless abstraction which is not straightford in the code where multiple wasm instances executing at the same time
- Update eos-vm-oc threads limit calculation based on the design.
Clean tmp and TestLogs after test runs
Previously when tests failed, they left temporary files behind, which could cause problems if the tests were run on multiple devices with /tmp mounted on a different device. These files were not being cleaned up, and some tests were also leaving logs around even on success. To solve this problem, test logs are now cleaned up after each test. These changes will help ensure that tests do not leave temporary files and logs behind, which could cause problems if the tests are run on different devices.
Fix gelf DNS resolves to ipv6 address
There was an issue with gelf logging not working when using DNS pointing to both IPv4 and IPv6 addresses. In v4.0.0-rc3, we made a couple of subtle changes to how the UDP socket is handled. First, the sending thread was changed to a new thread to ensure that it is a blocking socket. Second, a "connected" socket was used to eliminate unnecessary buffer copy for sending gelf message. These changes fix the issue with gelf logging when using DNS pointing to both IPv4 and IPv6 addresses. Additionally, the fc udp_socket and resolver have been removed, and unnecessary buffer copy for sending gelf message has been eliminated.
Improved test_snapshot_scheduler stability
There was an issue with the test_snapshot_scheduler stability, where the ordering of snapshot requests in multiindex container caused instability. The test code was modified to stop if pending_snapshots did not exist, using find_if() to search for pending_snapshots. Now the test can better handle situations where the order of the snapshot requests is not static.
CPack error message fix
When adding TestHarness python integration scripts, there was an error message with the CMAKE program on certain versions, such as CMAKE < 3.21. The error was related to creating a symbolic link that did not exist. The issue was resolved by adding an install script that would work for CMAKE versions < 3.21. The new install script was added to the project and now TestHarness python integration scripts can be used without any errors on CMAKE versions < 3.21.
Improvements and fixes for snapshot scheduling
There were a few issues with scheduling and managing snapshots, including that the list of pending snapshots was not always available, which caused some confusion and extra work. To address this, the list of pending snapshots is now always included in the request reply. This makes managing snapshots easier and more user-friendly.
Use block_num for interrupt of start_block
PRs
Signal received block with the block number so that start_block can be interrupted even if the block is received before the start of start_block. Useful as mitigation against race condition of receiving block right before start of the block in current implementation.
Adjusted timeout in tests to give better chance on reaching consensus
Looks like default 30 sec in this particular cluster config is not sufficient sometimes on a slower gh workers, leading to Test Failure: nodeos_snapshot_forked_test.
Bug fixes
SHiP forking behavior fix
The state_history_plugin was not sending updated blocks when nodeos forked. There has been an enhancement made to the ship_streamer_test to generate a fork and verify SHiP sends new blocks on forks.
SHiP flush logs on write
In version 3.2, the trace history index was sometimes corrupted, causing the state history to be rebuilt from scratch. This prevented ship nodes from starting up properly with a snapshot, which took a long time on large blockchains.
To address this issue, two changes were made. First, the state_history_plugin logs are now flushed to disk at the end of each write to make it more likely that the logs will be valid if there is a crash or kill -9. Second, a fix was implemented for replay where forkdb.root() is null during the write in SHiP for on_accepted_block. In version 4.0, SHiP calls chain.last_irreversible_block_id() in on_accepted_block, which expects forkdb.root() to be valid.
These changes will help ensure that the logs are valid and can be used for recovery if needed.
Feature Lifecycle Updates
Removed features in v4.0.0
HTTPS in nodeos and keosd
- removed HTTPS support (and code) from http_plugin
- removed HTTPS support (and code) from http_client_plugin (mostly in libfc)
- removed OpenSSL::SSL from libraries/libfc/CMakeLists.txt
Direct YubiHSM support
The original implementation of YubiHSM support required additional dependencies, complicated the build process, lacked sufficient automated testing, and prevented natively building Leap for ARM on macOS. Leap v4.0.0 removes support for YubiHSM.
Transaction Generator
The old txn_test_gen_plugin was deprecated in favor of the new trx_generator. Tests previously using txn_test_gen_plugin were updated to use trx_generator. Additionally, performance tests and configuration options were added to trx_generator.
Login Plugin
The Login Plugin (login_plugin) appears to be an abandoned demo/prototype/PoC from early EOSIO implementations. It was removed in Leap v4.0.0.
Blocklog Utility
The deprecated eosio-blocklog program blocklog-util has been removed in v4.0.0, please use leap-util block-log instead. Output file names have changed when using leap-util block-log extract-blocklog. The file names now include the start and end block numbers as a suffix. For example, blocks-100-200.log and blocks-100-200.index.
Example 1
eosio-blocklog --as-json-array --blocks-dir ....is now...
leap-util block-log print-log --as-json-array --blocks-dir .Example 2
eosio-blocklog --trim-blocklog -f 1 -l 100 --blocks-dir ....is now...
leap-util block-log trim-blocklog -f 1 -l 100 --blocks-dir .Example 3
eosio-blocklog --extract-blocks -f 1 -l 100 --blocks-dir . --output-dir newdir...is now...
leap-util block-log extract-blocklog -f 1 -l 100 --blocks-dir . --output-dir newdirPrivate-key nodeos config
The private-key nodeos config was deprecated since EOSIO v1.0.0 and is now removed in favor of signature-provider nodeos config.
For example:
--private-key [\"EOS6hMjoWRF2L8x9YpeqtUEcsDKAyxSuM1APicxgRU1E3oyV5sDEg\",\"5JgbL2ZnoEAhTudReWH1RnMuQS6DBeLZt4ucV6t8aymVEuYg7sr\"]
Becomes:
--signature-provider EOS6hMjoWRF2L8x9YpeqtUEcsDKAyxSuM1APicxgRU1E3oyV5sDEg=KEY:5JgbL2ZnoEAhTudReWH1RnMuQS6DBeLZt4ucV6t8aymVEuYg7sr
Cleos send transaction --read-only
--read-only option is removed in cleos send transaction command. Use --dry-run instead for compute_transaction
eosio-launcher
eosio-launcher is removed and replaced by launcher.py.
All deprecated features, not yet removed
Deprecated transaction endpoints
As of v4.0.0 the following endpoints are deprecated:
/v1/chain/send_transaction/v1/chain/push_transaction/v1/chain/push_transactions
fc and wasm-spec-tests repos
Due to limited cases for fc and wasm-spec-tests have been integrated directly into leap repository. Further development on related code will occur directly in the leap repository and the only development that will occur within the fc and wasm-spec-tests repositories are critical bug fixes that are necessary to support Leap 3.1 patches. The fc and wasm-spec-tests repositories will be archived in an upcoming version of Leap.
Ubuntu 18.04 Support
Support for Ubuntu 18.04 will be removed in an upcoming version of Leap. Please use Ubuntu 20.04 or 22.04.
More Information
The latest state of deprecations for Leap can be found by visiting our deprecations and proposed deprecations page.
Further details on changes since last release
Contributors
Special thanks to the contributors that submitted patches for this release:
Full list of changes since last release
PRs
- (885)[4.0] Add time summary logging for transient trx processing and read-only threads
- (892)[4.0] Cleanup and prevent thread starvation while in the read window.
- (893)[3.2] Use block_num for interrupt of start_block
- (905)[4.0] Use a single timer for read and write windows
- (908)[3.2 -> 4.0] Use block_num for interrupt of start_block
- (907)[4.0] Adjusted timeout in tests to give better chance on reaching consensus
- (921)[4.0] Implement support for -k/--kill in launcher.
- (901)[4.0] Execute read only safe tasks on read-only thread pool
- (928)[3.2] SHiP flush logs on write
- (935)[3.2 -> 4.0] SHiP flush logs on write
- (936)[3.2] forkdb reset in replay since blocks are signaled
- (917)[4.0] fix gelf DNS resolves to ipv6 address
- (915)[4.0] Clean tmp and TestLogs after test runs
- (938)[3.2 -> 4.0] forkdb reset in replay since blocks are signaled
- (933)[4.0] Small improvements for parallelizing read-only transactions and tasks
- (943)[3.2] SHiP Fix forking behavior
- (947)[4.0] Correct time summary calculation for transient trxs due to change of read-only trxs windows
- (953)[3.2] Bump Leap version to 3.2.3
- (952)[3.2 -> 4.0] SHiP Fix forking behavior
- (957)[3.2 -> 4.0] Merge 3.2.3 version bump to 4.0
- (960)[4.0] Use is_write_window in eos-vm-oc main to determine if it is safe to update code cache
- (931)[4.0] Improved test_snapshot_scheduler stability
- (975)[4.0] Remove global WASM::check_limits and initial memory from instantiate_module
- (964)[4.0] Added missing calls to wasmifs on read-only threads
- (982)[4.0] Set an upper limit on read-only-threads of 8
- (972)[4.0] fix cpack error message
- (968)[4.0] Several improvements and fixes for snapshot scheduling
- (939)[4.0] prometheus on different port
- (986)[4.0] Restore read-mode=speculative
- (993)[4.0] make
eosio_exitimpl more direct - (984)[4.0] Fix ROtrx dispatched to baseline runtime when compiled code not ready
- (998)[4.0] Bump Leap version to release 4.0 rc3
Full Changelog:
v4.0.0-rc2...v4.0.0-rc3