New features

Read-Only Transactions

We have introduced a RPC endpoint for sending read-only transactions. A read-only transaction does not change the state and is not added into the blockchain. It is useful for users to retrieve complex chain information.

The format is the same as regular transaction, except it does not contain authorization section. Mechanisms are in place to ensure actions (including inline actions) in a read-only transaction do not modify the state and deepmind logging is not performed.

The RPC URL is v1/chain/send_read_only_transaction

cleos is updated to send read-only transactions:

• cleos push transaction --read <transaction>
• cleos push action --read <action>

New producer_plugin logging categories are added for transient transactions (dry-run (by compute_transaction) and read-only transactions). They are transient_trx_success_tracing and transient_trx_failure_tracing . Existing transaction_success_tracing and transaction_failure_tracing do not include the logging for transient transactions.

max_action_return_value_size is not enforced for read-only transactions.

Auto-peering with schedule proximal Block Producer nodes

We've added a config option to automatically manage connections based on the Block Producer schedule. Node order and peering can impact a node's latency to receive last block of the preceding BP, and to send their last block to the next BP. Ideally, Block Producers:

• Receive the last block of the previous round as soon as possible
• Send their last block to the next Block Producer as soon as possible
• Avoid getting a late start on their block production

To achieve these goals, the following changes were made in v4.0.0:

Correlate BP account names and connection details

--p2p-auto-bp-peer has been added to nodeos to allow a node to correlate BP account names to p2p connection details. When a block producer node with this setting detects a pending schedule change, it will attempt to auto-peer with BPs that have come into schedule proximity, and disconnect from any auto-peered BPs that are no longer in schedule proximity. These auto-peered connections are not subjected to the limit specified by the --max-clients option.

BP Peer flag in net api

is_bp_peer has been added to every connection object in the v1/net/connections api endpoint response.

Snapshot Scheduling API

The Snapshot Scheduling API added in Leap v4.0.0 allows users to schedule and manage snapshot requests in advance, increasing reliability and reducing the potential for errors in the snapshot creation process. It also allows for greater flexibility in creating and managing snapshots, as users can specify parameters such as start and end blocks and block spacing.

The API provides endpoints for several features:

• /producer/schedule_snapshot: Add a task to perform snapshots
  • Parameters: start_block, end_block, block_spacing, snapshot_description
• /producer/get_snapshot_requests: Return all scheduled snapshot requests
• /producer/unschedule_snapshot: Remove a previously scheduled snapshot task
  • Parameters: snapshot_request_id

For more details, check out the Snapshot Scheduling API Design Specs.

Prometheus Exporter Plugin

Prometheus Instrumentation & Metrics

Instrumentation of nodeos plugins to expose data with Prometheus. An initial phase of instrumentation will allow for the evaluation of the usability of the instrumentation when viewed from Prometheus. The initial instrumentation will not include any per account or per contract metrics. Additionally, the ability to expire metrics will not be included in the initial phase. To support dynamic metrics, the metric object for each plugin will include an indicator of when the set of metrics has last changed. The following metrics will be collected for the initial phase of instrumentation:

Net Plugin
gauge - number of clients
gauge - number of peers
gauge - number of dropped blocks

Producer Plugin (Producer plugin metrics will be gathered at the start of each block)
gauge - unapplied transaction queue sizes (number of transactions)
counter - blacklisted transactions count (total)
counter - blocks produced
counter - transactions produced
gauge - last irreversible
gauge - current block num
gauge - subjective billing number of accounts
gauge - subjective billing number of blocks
gauge - scheduled transactions

See the Prometheus Plugin Architecture for more details.

Prometheus on a dedicated IP/port

The prometheus_plugin can run on a dedicated IP and port by setting prometheus-exporter-address and specifying the IP and port to listen for incoming prometheus metrics http request. A simple embedded server handles HTTP requests and exports Prometheus metrics.

NOTE: Future versions will generalize this functionality to allow custom IP/port configuration for categories of HTTP endpoints.

Log Splitting

Leap v4.0.0 includes two similar features aimed at giving node operators greater control over how important logs for blocks and state are managed: blocks log splitting and state history log splitting. The key benefit of these features is that it allows users to specify how many split log files to retain, and automates aging out older logs by deleting them or moving them another directory.

Blocks Log Splitting

To make it easy to use block log splitting, we've added some new options for the chain plugin:

• blocks-log-stride allows you to split the block log file when the head block number is the multiple of the stride. When the stride is reached, the current block log and index will be renamed 'blocks--.log/index', and a new current block log and index will be created with the most recent block. All files following this format will be used to construct an extended block log.
• max-retained-block-files alows you to specifiy the maximum number of block files to retain so that the blocks in those files can be queried. When the number is reached, the oldest block file would be moved to an archive directory or deleted if the archive directory is empty. It's important to note that the retained block log files should not be manipulated by users.
• Two new options allow you to configure the locations of blocks log directories, we've added two new options:
  • blocks-archive-dir allows you to specify the location of the blocks archive directory (absolute or relative path). If the value is empty, blocks files beyond the retained limit will be deleted. Files in the archive directory won't be accessed by nodeos anymore.
  • blocks-retained-dir allows you to specify the location of the blocks retained directory (absolute or relative path). If the value is empty, it is set to the value of the blocks dir.

Please note that all of the newly added options cannot be specified together with the --block-log-retain-blocks option. Overall, we believe that the Blocklog splitting feature and its associated options will help you better manage your block logs and improve the efficiency of your nodeos instance.

SHiP Log Splitting

To make it easy to use SHiP Log Splitting, we've added some new options for the State History Plugin:

• state-history-stride allows you to split the state history log file when the block number is the multiple of the stride. When the stride is reached, the current history log and index will be renamed *-history--.log/index, and a new current history log and index will be created with the most recent blocks. All files following this format will be used to construct an extended history log.
• max-retained-history-files allows you to specify the maximum number of state history log file groups to retain so that the blocks in those files can be queried. When the number is reached, the oldest log files would be moved to an archive directory or deleted if the archive directory is empty. The retained log files should not be manipulated by users.
• state-history-retained-dir allows you to specify the location of the state history retained directory (absolute path or relative to state-history dir). If the value is empty, it is set to the value of the state history dir.
• state-history-archive-dir allows you to specify the location of the state history archive directory (absolute path or relative to state-history dir). If the value is empty string, log files beyond the retained limit will be deleted. All files in the archive directory are completely under user's control, i.e. they won't be accessed by nodeos anymore.

Please note that all of the newly added options cannot be specified together with the --state-history-retain-num option. Overall, we believe that the SHiP Log Splitting feature will help you better manage your state history log files and improve the efficiency of your nodeos instance. We look forward to seeing how this feature will benefit the Leap community.

Performance improvements

Parallelize Read-Only Transaction Execution

Regular read-only transactions are executed sequentially on the main thread. To take advantage of multi-core architecture, offloading them from the main thread and executing them multi-threaded will make more time for the main thread and improve throughput of read-only transactions.

We have implemented a new feature to parallelize read-only transaction execution. Design documentation for this feature can be accessed here.

When this feature is enabled, the node toggles between write and read windows. In the write window, the node operates normally. In the read window, read-only transactions are executed in a dedicated thread pool, while operations safe to read-only transaction execution are executed on the main thread.

New configuration options are

  --read-only-threads arg               Number of worker threads in read-only
                                        execution thread pool. Max 8.
                                        
  --read-only-write-window-time-us arg (=200000)
                                        time in microseconds the write window
                                        lasts
                                        
  --read-only-read-window-time-us arg (=60000)
                                        time in microseconds the read window
                                        lasts

NOTES

• If you get this error message: "plugin_config_exception: Incorrect plugin configuration
  --read-only-read-window-time-us (60000) must be greater than --max-transaction-time 10000 ms plus a margin of 10000 us
  {"read":60000,"trx_time":10000,"min":10000}", please note...
  • max-transaction-time is expressed in milliseconds, whereas read-only-read-window-time-us and read-only-write-window-time-us are expressed in microseconds.
  • Example of a configuration that causes an error: , amax-transaction-time of 10000, and a read-only-read-window-time-us of 60000
  • Example of a valid configuration: a max-transaction-time of 50, and a read-only-read-window-time-us of 60000
  • We've added an issue to make this configuration clearer and more consistent in a future release: #1088
• --read-only-threads
  • is not allowed on a producer node
  • defaults to 3 when eosio::chain_api_plugin is enabled.
  • block relay nodes or canary nodes should consider --read-only-threads=0
  • If --read-only-threads is not configured or configured to 0, this feature is not enabled; read-only transactions are executed on the main thread.
• --read-only-read-window-time-us must be greater than --max-transaction-time.
• --read-only option is removed in cleos send transaction command. Use --dry-run instead for compute_transaction
• Since eosio::chain_api_plugin may be enabled so /v1/chain/get_info is available. Block relay nodes or canary nodes should consider --read-only-threads=0 if they have enabled eosio::chain_api_plugin as the read-only-thread pool is only used for read-only transactions and /v1/chain/get* calls. If read-only-threads are enabled, but not utilized, there is a very small overhead of switching back and forth between the read/write windows.
• Read and write windows are interrupted when new blocks are received.
• The majority of Chain RPC get_* requests are safe to run in parallel with themselves and read-only transactions. To further improve API nodes throughputs, they are now executed on the read-only thread pool during the read-only-read-window-time-us window.
• /v1/net/* calls were also moved to http thread pool, off main app thread.

Lighter Validation for Relays

This feature aims to reduce the latency of block propagation by enabling nodes to relay blocks before fully validating or computing state, as long as certain conditions are met. With this new feature, nodes can immediately process received blocks after validating their headers, even when there is a backlog of pending transactions. Additionally, the new implementation includes optimizations for broadcast blocks behavior and peer_block_state_index, further improving block propagation times. The goal of lighter validation on relays is to reduce the latency of block propagation.

Other Multithread Support

We have moved more tasks out from the main thread and process them in separate threads:

• block_log has been made thread safe.
• net_plugin updated to use thread safe block_log from net threads instead of having to post onto the main thread for processing.
• SHiP writes to the state history logs on the main thread, but reads from them on the SHiP thread. The state history logs have been made thread safe.
• /v1/chain/get_block performance has been significantly improved by moving abi serialization off the main thread.

Other changes

Faster bilinear group cryptography library for EVM support

The Ethereum Virtual Machine on EOS requires bilinear group cryptography. The libff and bn256 libraries were benchmarked against each other. Each test run used the same 10 pairs of points as input and run through the same initialization and pairing. The bn256 library was selected as most performant, and ported into Antelope Leap v4.0.0.

Default read mode is now head & removal of read-only mode

In all releases of Leap/EOSIO prior to 4.0, the default read-mode was speculative. speculative mode meant a transaction submitted to an API node would maintain any state changes for subsequent transactions submitted to the same API node. For example, create a new account in one transaction and then transfer to that account in a subsequent transaction. This mode has been a source of confusion due to it presenting state that has not been included into any blocks by a block producer and thus showing different state across different nodes.

In 4.0, the default is now read-mode=head which does not persist the state changes of transactions that have not been included in a block. For example, this means a transfer transaction will need to wait for a block producer to include the account creation transaction into a block which is applied before the transfer would be allowed.

read-mode=speculative remains supported for users who want old behavior but with an important change that transactions submitted via API endpoints will not be reapplied speculatively upon a new block being applied. The user is required to resubmit any transactions after a new block should those transactions' side effects need to be observed.

In addition to the above changes, deprecated read-mode=read-only has been removed. Its behavior can be achieved with a combination of other options:

• read-mode=head
• p2p-accept-transactions=false
• api-accept-transactions=false

These changes better reflect the actual state of execution, reduce user confusion, and improve control over transaction behavior by introducing more clearly defined options.

Support for -k --kill in launcher

Launcher now honors -k --kill <signal_number> argument. Cluster was updated for the Python launcher but support in the launcher itself was missed. Was causing a Test failure: resource_monitor_plugin_test

Add time summary logging for transient trx processing and read-only threads

A new log summary report was introduced that provides details on the time spent executing dry-run and read-only transactions. When the option --read-only-threads is enabled, the report also includes time spent on read-only transactions across multiple threads. This makes it easy to monitor the time spent on dry-run and read-only transactions, including execution time of read-only transactions across multiple threads. This will help identify any bottlenecks to the performance of the system.

Read-only Transactions and Operations Parallelization Hardening

As part of implementing and enabling parallel processing for read-only transactions, we went through a hardnening process to ensure reliability and performance. Some of our optimizations included:

• preventing threads from getting stuck during read operations
• accurate time tracking for temporary transactions
• code cache optimization
• enhancements to read-only threads handling in WebAssembly (WASM)
• removal of global WASM limits during module instantiation
• Maximum of 8 read-only threads at a time
• Simplification of 'eosio_exit'

Clean tmp and TestLogs after test runs

Previously when tests failed, they left temporary files behind, which could cause problems if the tests were run on multiple devices with /tmp mounted on a different device. These files were not being cleaned up, and some tests were also leaving logs around even on success. To solve this problem, test logs are now cleaned up after each test. These changes will help ensure that tests do not leave temporary files and logs behind, which could cause problems if the tests are run on different devices.

Fix gelf DNS resolves to IPv6 address

There was an issue with gelf logging not working when using DNS pointing to both IPv4 and IPv6 addresses. In v4.0.0-rc3, we made a couple of subtle changes to how the UDP socket is handled. First, the sending thread was changed to a new thread to ensure that it is a blocking socket. Second, a "connected" socket was used to eliminate unnecessary buffer copy for sending gelf message. These changes fix the issue with gelf logging when using DNS pointing to both IPv4 and IPv6 addresses. Additionally, the fc udp_socket and resolver have been removed, and unnecessary buffer copy for sending gelf message has been eliminated.

Improved test_snapshot_scheduler stability

There was an issue with the test_snapshot_scheduler stability, where the ordering of snapshot requests in multiindex container caused instability. The test code was modified to stop if pending_snapshots did not exist, using find_if() to search for pending_snapshots. Now the test can better handle situations where the order of the snapshot requests is not static.

CPack error message fix

When adding TestHarness python integration scripts, there was an error message with the CMAKE program on certain versions, such as CMAKE < 3.21. The error was related to creating a symbolic link that did not exist. The issue was resolved by adding an install script that would work for CMAKE versions < 3.21. The new install script was added to the project and now TestHarness python integration scripts can be used without any errors on CMAKE versions < 3.21.

Improvements and fixes for snapshot scheduling

There were a few issues with scheduling and managing snapshots, including that the list of pending snapshots was not always available, which caused some confusion and extra work. To address this, the list of pending snapshots is now always included in the request reply. This makes managing snapshots easier and more user-friendly.

Use block_num for interrupt of start_block

Signal received block with the block number so that start_block can be interrupted even if the block is received before the start of start_block. Useful as mitigation against race condition of receiving block right before start of the block in current implementation.

Adjusted timeout in tests to give better chance on reaching consensus

Looks like default 30 sec in this particular cluster config is not sufficient sometimes on a slower gh workers, leading to Test Failure: nodeos_snapshot_forked_test.

Bug fixes

SHiP forking behavior fix

The state_history_plugin was not sending updated blocks when nodeos forked. There has been an enhancement made to the ship_streamer_test to generate a fork and verify SHiP sends new blocks on forks.

SHiP flush logs on write

In version 3.2, the trace history index was sometimes corrupted, causing the state history to be rebuilt from scratch. This prevented ship nodes from starting up properly with a snapshot, which took a long time on large blockchains.

To address this issue, two changes were made. First, the state_history_plugin logs are now flushed to disk at the end of each write to make it more likely that the logs will be valid if there is a crash or kill -9. Second, a fix was implemented for replay where forkdb.root() is null during the write in SHiP for on_accepted_block. In version 4.0, SHiP calls chain.last_irreversible_block_id() in on_accepted_block, which expects forkdb.root() to be valid.

These changes will help ensure that the logs are valid and can be used for recovery if needed.

Building, compatibility, and upgrading

Upgrading from prior releases

To upgrade from Leap 3.X.X to Leap 4.0.0:

• Install Leap v4.0.0
• Edit config.ini and remove any set configuration items that no longer exist in v4.0.0 :
  • nodeos disable-api-persisted-trx, private-key, https-client-root-cert, https-client-validate-peers, https-server-address, https-certificate-chain-file, https-private-key-file, https-ecdh-curve, plugin=eosio::trx_test_gen_plugin, txn-reference-block-lag, txn-test-gen-threads, txn-test-gen-account-prefix, txn-test-gen-expiration-seconds, txn-test-gen-stop-on-push-failed, plugin=eosio::login_plugin, max-login-requests, max-login-timeout.
  • keosd yubism-url, yubism-authkey, https-server-address, https-certificate-chain-file, https-private-key-file, https-ecdh-curve.
• Restart the binaries while pointing to the same data directories and the updated config
  • Persisted data files used by Leap v3.1.X and v3.2.X are compatible with Leap v4.0.0.

Feature Lifecycle Updates

Removed features in v4.0.0

HTTPS in nodeos and keosd

• removed HTTPS support (and code) from http_plugin
• removed HTTPS support (and code) from http_client_plugin (mostly in libfc)
• removed OpenSSL::SSL from libraries/libfc/CMakeLists.txt

Removed ABI JSON/BIN conversion endpoints

Removed the following deprecated endpoints to improve security posture:

• /v1/chain/abi_bin_to_json
• /v1/chain/abi_json_to_bin

Direct YubiHSM support

The original implementation of YubiHSM support required additional dependencies, complicated the build process, lacked sufficient automated testing, and prevented natively building Leap for ARM on macOS. Leap v4.0.0 removes support for YubiHSM.

Transaction Generator

The old txn_test_gen_plugin was deprecated in favor of the new trx_generator. Tests previously using txn_test_gen_plugin were updated to use trx_generator. Additionally, performance tests and configuration options were added to trx_generator.

Login Plugin

The Login Plugin (login_plugin) appears to be an abandoned demo/prototype/PoC from early EOSIO implementations. It was removed in Leap v4.0.0.

Blocklog Utility

The deprecated eosio-blocklog program blocklog-util has been removed in v4.0.0, please use leap-util block-log instead. Output file names have changed when using leap-util block-log extract-blocklog. The file names now include the start and end block numbers as a suffix. For example, blocks-100-200.log and blocks-100-200.index.

Example 1

eosio-blocklog --as-json-array --blocks-dir .

...is now...

leap-util block-log print-log --as-json-array --blocks-dir .

Example 2

eosio-blocklog --trim-blocklog -f 1 -l 100 --blocks-dir .

...is now...

leap-util block-log trim-blocklog -f 1 -l 100 --blocks-dir .

Example 3

eosio-blocklog --extract-blocks -f 1 -l 100 --blocks-dir . --output-dir newdir

...is now...

leap-util block-log extract-blocklog -f 1 -l 100 --blocks-dir . --output-dir newdir

Private-key nodeos config

The private-key nodeos config was deprecated since EOSIO v1.0.0 and is now removed in favor of signature-provider nodeos config.

For example:

--private-key [\"EOS6hMjoWRF2L8x9YpeqtUEcsDKAyxSuM1APicxgRU1E3oyV5sDEg\",\"5JgbL2ZnoEAhTudReWH1RnMuQS6DBeLZt4ucV6t8aymVEuYg7sr\"]

Becomes:

--signature-provider EOS6hMjoWRF2L8x9YpeqtUEcsDKAyxSuM1APicxgRU1E3oyV5sDEg=KEY:5JgbL2ZnoEAhTudReWH1RnMuQS6DBeLZt4ucV6t8aymVEuYg7sr

eosio-launcher

eosio-launcher is removed and replaced by launcher.py.

All deprecated features, not yet removed

fc and wasm-spec-tests repos

Due to limited cases for fc and wasm-spec-tests have been integrated directly into leap repository. Further development on related code will occur directly in the leap repository and the only development that will occur within the fc and wasm-spec-tests repositories are critical bug fixes that are necessary to support Leap 3.1 patches. The fc and wasm-spec-tests repositories will be archived in an upcoming version of Leap.

Ubuntu 18.04 Support

Support for Ubuntu 18.04 will be removed in an upcoming version of Leap. Please use Ubuntu 20.04 or 22.04.

More Information

The latest state of deprecations for Leap can be found by visiting our deprecations and proposed deprecations page.

Further details on changes since last release

Contributors

Special thanks to the contributors that submitted patches for this release:

• @ndcgundlach
• @ClaytonCalabrese
• @oschwaldp-oci
• @linh2931
• @spoonincode
• @heifner
• @766C6164
• @kj4ezj
• @lparisc
• @huangminghuang
• @pnx
• @jgiszczak
• @ScottBailey
• @MaximSmolskiy
• @systemzax
• @greg7mdp
• @ericpassmore
• @mikelik
• @jolly-fellow

Full list of changes since last release

Full Changelog: v3.2.1...v4.0.0