multiversx/mx-chain-go v1.2.20

on GitHub

Release v1.2.20 is out 🚀

Release description:
Binary Update v1.2.20

What's new:
#2505 - Soft restart feature that allows shuffling at the end of epoch without restarting the process. A large part of the node initialization has been refactored to allow this, and factories have been created for the separate components and their sub-components. Each component and subcomponent implements a Close method that allows for a cleanup before re-initialization.

• #1743 Initial implementation of the soft restart feature: added factories, split code, refactor main.go file
• #2264 Added unit tests for factory package
• #2300 Implemented the node runner component that is able to coordinate the contiously creation and closing of the components factories.
• #2489 Fixed a problem on bootstrap when start in epoch and 2 times the tries and storageManagers for accounts and peers were created: 1 time for the metasyncer and after for the actual shard. Added method tryCloseExisting that closes the previous trie and storage manager before adding a new one.
• #2563 Added close mechanism for hardfork storers.
• #2816 Fixed the deadlock situation when a method that was mutex-protected was called inside a function that was already protected by the same mutex.
• #2871 Fixed some linter issues, increased the time for test to pass libp2p module init.

#2317 - Full archive feature that will make the node able to re-run the full network history by requesting data from peers:

• #2297 Added peerSubType field in heartbeat subsystem
• #2334 Added multi-interceptor capability on the network messenger. Each topic now supports n interceptors that will be called sequentially if a message is received.
• #2330 Added resolver-requester implementation for full history nodes.
• #2320 First implementation of the current network epoch provider component.
• #2515 Changed the resolvers so they can be used by the full history nodes by taking into account the provided epoch value.
• #2522 Added functionality for full history nodes' storers to keep a list of oldActivePersisters in order to server other nodes that are synchronizing older epochs.
• #2579 Integration: added arithmetic current network epoch provider, added factory for the current network epoch provider, integrated the new components in resolvers, fixes and old code removal.
• #2940 Full history epoch provider fixes: arithmetic epoch provider now depends on the soft fork epoch notifier, fixes in order to make the full history nodes feature work properly.
• #3040 Removed duplicated headerHandlerStub implementations.

#2464 - Import-db process improvements:

• #2391 Added a new storer for import-db mode that will hold the (epoch - root hash) pairs related to tries snapshots. Based on this, we will be able to re-load the trie state at each end of epoch.
• #2481 Implemented trie storage resolver, added some import-db fixes, prepared the ground for the start in epoch in import-db process.
• #2486 Refactored the epochStartMetaSyncer struct as to use an injected EpochStartMetaBlockInterceptorProcessor rather than a constructed one.
• #2495 Implemented start in epoch in import-db process.
• #2501 Added storage epoch start bootstrap unit tests.
• #2520 Fixed the configuration for the TrieEpochRootHashStorage. The previous config would have caused the node to stop with the error cache size is lower than batch size.
• #2566 Fixed the root hash when storing into EpochRootHashStorer so it will be exactly the same as the snapshot's root hash. Before this, there were some inconsistencies and the test was not relevant.
• #2565 Refactored how the configuration filenames are stored inside the configs struct.

#2497 - Outport driver integration:

• #2462 Outport initial implementation: added an extra layer between node components and drivers (OurportHandler interface) in order to be able to add more implementations (drivers) that can index data in any database.
• #3218 Events driver integration.
• #3320 Feature branch cleanup.

#3099 - Trie improvements:

• #3089 Refactor the trie insert and commit processes in preparation for the trie checkpoint optimizations.
• #3114 Mock implementations refactor: create a stub for the AccountsAdapter and for the Trie interfaces in testscommon package, and use it everywhere.
• #3077 Created an adapter that has a cache and an underlying storer, and it implements the Cacher interface. When the cache is full, the evicted value will be stored in the DB. If a value is not found in the cache, the DB will also be searched for the value (it employs the write-back paradigm).
• #3171 Created the CheckpointHashesHolder that holds the hashes that have been added/modified since the last snapshot/checkpoint. When setting a state checkpoint, only the dirty hashes will be committed to the snapshotDb. If the new struct is filled, a forced state checkpoint will be started, thus cleaning the struct.
• #3204 Refactor state snapshot: on state checkpoint and snapshot, when traversing the main trie to verify what nodes need to be commited to the snapshot DB, return all leaves found on a channel, so that subsequent data tries can be commited directly, without another main trie traversing.
• #3229 Fixed linter issues.
• #3199 Bump Arwen version to v1.3.23.

#3117 - Preferred peers feature that will allow targeted connections to a specified lists of bls public keys with improved bandwidth usage:

• #3110 Implemented the first part of the preffered peers feature: added configuration section in the p2p.toml file, added the component in networkComponents, avoid eviction of the preferred peers.
• #3119 Refactored the network sharding collector implementation + stubs.
• #3128 Added the preferred peers' holder component inside the peerShardMapper in order to update the public key - peer id mapping of it in a better manner than before (on listsSharder).
• #3140 Refactored target shard ID inside resolvers. Removed defaultShardID and used only the relevant shard ID. It will be used in the next PR for fetching a preferred peer for a given shard.
• #3143 Integrated the preferred peers holder component inside resolvers so if any preferred peer in the target shard exists, it will be used first when sending requests (only if there is more than one peer to request). Also skipped the antiflood checks if the peer the messages were received from is a preferred one.
• #3173 Aligned the remained TODOs + additional unit tests.

#3082 - Arwen in-process versioning at runtime:

• #3125 Added get/restore snapshot methods to blockchain hook implementation.
• #3144 Added an injected locker used when changing the Arwen version. It replaces the use of global variables. Removed unused flags.
• #3150 Fixed Arwen locker usage by having a different instance in the genesis process and thus, avoiding deadlock at genesis time.
• #3152 Removed Arwen OOP option.

#3146 - Added the possibility to request/receive large trie nodes by splitting the node's value into smaller chunks:

• #3142 Added trie resolver capability to be able to resolve large trie nodes, prepared the interceptor to use a chunk intercepted data processor, added new functionality in the request handler to be able to request chunks from an incomplete received message.
• #3145 Removed duplicated requestedHandlerStub implementations.
• #3135 Created a new component - storagePruningManager - that holds the evictionWaitingList and the pruningBuffer. This new struct will manage all the state pruning operations.
• #3175 Added the trie nodes chunk processor. This processor is able to hold the completeness of a large trie nodes set and also is able to trigger requests for the missing parts.
• #3178 Removed duplicated interceptorStub and interceptorsContainerStub implementations.
• #3179 Integrated the trie nodes chunks processor in interceptors.
• #3214 Added fixes for the large trie sync processor, removed duplicated stubs.

#3378 - ESDT total supply computation

• #3364 Added new db lookup extension function that will process VM logs for the following operations: LocalBurn, LocalMint, NFTCreate, NFTBurn, AddQuntity and will save the ESDT supply in a specialized storage
• #3380 Added the REST API route /network/esdt/supply/:token, implemented a new component for nonce processor that will save in storage the latest processed block nonce.
• #3404 Fixed how the key for an NFT is created.
• #3408 Fix revert condition to work as expected.

#2503 Added new kad dht peer discoverer instance. The old (legacy) instance and the new one can be switched from p2p.toml config file.
#2526 Added separate list for seeders as to not consider them as unknown peers, added argument structs on some constructors, p2p package general cleanup.
#2454 Refactored all hashers so that they will be structs and their functions will have pointer receivers. Also, now the empty hash is computed only once, on the constructor.
#2605 Arithmetic difference economics useful print.
#2616 Replace Rounder with RoundHandler.
#2808 Refactored transaction coordinator with arguments.
#2785 Added disabledNodeFacade implementation and the unit tests file.
#2857 Replaced the calls to fmt.Println() with calls to log.Info() or log.Error() in the widely-used testInitializer.go file.
#2884 Log activation epoch settings.
#2900 Added examples for message signing. Examples are written as golang tests and are stored in examples/messageSign_test.go file.
#2908 Add the dataTrie to tries cacher if SaveAccount is called, but the dataTrie is not present in the cacher.
#2892 Added extra transaction construction example, for nonce == 0.
#2879 Moved epoch related settings to a separate EpochConfig structure: added new toml config file enableEpochs.toml, added logs to ensure all fields have been read and passed down to inner processors correctly.
#2883 Created a queue that when it's full and a new item is added to it, it will automatically evict the first element. Added 2 config parameters, that specify by how much the state pruning will be buffered.
#2920 Added the TestNetwork that contain a set of TestProcessorNode instances. Added several functions like Step, Start, Close and so on.
#2847 Created an API wrapper so we can change the facade implementation while the node is running. Useful when the epoch bootstrapping is in progress and the API endpoints should return a message signaling that the node is starting, rather than API service not being started at all.
#2958 Added new REST API endpoint that exposes the activation epochs values.
#3029 Bump dependencies versions: google/gops, gin-gonic/gin, pelletier/go-toml, golang/protobuf, ipfs/go-log, elastic/go-elasticsearch
#3030 Added a new variant for the realyed transactions (relayed transaction v2) that will optimize the costs of the relayed transactions.
#2969 Added two new REST API endpoints: one for computing a Merkle proof, and another for verifying a Merkle proof.
#2780 Refactoring in transaction status computer.
#2772 Implemented and integrated a new middleware that will log different data about a request (path, request body, status code, response body, and duration) if it takes longer than a pre-defined threshold or it is unsuccessful (not 200 status code). Introduced a wrapper over the gin's Writer in order to fetch the response.
#3067 Added an extra field developerReward in the REST API route /address/:addr.
#3056 Added information about the epoch start in REST API block data: total supply, total to distribute, total newly minted, previous epoch start block hash.
#3064 Added in the Makefile the command "make lint" that will run the golangci linter.
#3091 Refactored the termui application: moved all the termui console components inside cmd/termui package, added a mechanism (channel based) that clears the console at restart/shuffle out.
#3121 Solution cleanup: removed the unused storer2elastic application.
#3134 Added custom wrapped errors that can automatically store the line that called the Wrap method. Interchangeable with the standard error interface.
#3133 Rename the function DoesFileExist to FileExists.
#2480 Refactored the on-chain governance implementation so it can be used by the validators, delegators and eGLD holders. It implements quadratic voting power.
#3167 Added an exception to the block syncer as to always consider itself not synchronized in the import-db process.
#3196 Extra fields in API endpoints: code metadata, owner address.
#3161 Added a log entry for the NFT operation.
#3190 Implemented a safe closer to be used in the components that need fast closing means on the exposed methods.
#3111, #3157 Bump dependencies versions: gin-gonic/gin, btcsuite/btcd.
#3228 Moved mocks and stubs used also in slashing to testscommon package.
#3230 Implemented a mechanism that detects if a node is in the same time a validator + has full archive flag set to true. This is an invalid setup so after all components are closed, an infinite loop printing the configuration issue will be triggered.
#3221 Integrate new vm-common.
#3234 Added an additional field timestamp in the endpoint that returns the transaction based on hash.
#3159 Added a mechanism that will trigger the trie resynchronization whenever a missing trie node error is raised. Added a new test point. Extracted in a function the machine ID computation.
#3226 Extend the REST API route /transaction/:hash?withResults=true. This endpoint will return also the logs and events generated by a transaction or by smart contract results.
#3240 Bump vm-common version to v0.3.4.
#3166 Refactored the deletion of old databases at each epoch start in regards to the node type (validator or observer). It relies on the oldest epoch to keep after iterating over all storage units and wait an epoch for deleting the databases marked for deletion the previous epoch.
#3242 Bump libp2p dependencies versions: libp2p/go-libp2p, libp2p/go-libp2p-kad-dht, multiformats/go-multiaddr. Removed the deprecated reference to secio implementation.
#3245 Optimized the full archive node sync by injecting the current network epoch provider in block syncer.
#3250 Added an integration test that guarantees that self ESDT transfers don't change the token balance of the address.
#3248 Changed the metric that returns the developer fees accumulated in epoch.
#3254 Added final result print in assessment tool.
#3252 Bump Arwen and vm-common versions.
#3262 Added logging for accounts number and size, added go routine analyser that can be enabled by a Debug flag in the config.toml.
#3264 Add a new flag CheckpointsEnabled that will create a new trieStorageManagerWithoutCheckpoints if it is set to false.
#3253 Update Arwen imports: vmContainerFactory to reference arwen-wasm-vm/v1_2 v1.2.25, making both versions of Arwen consistent (both are referenced as arwen-wasm-vm/v1_x).
#3263 Integrated ESDT multitransfer.
#3268 Added an extra flag in the config.toml file to enable or disable logs saving in storage.
#3275 Bump Arwen version to v1.3.25.
#3278 Integrate elrond-go-core repository.
#3271 Integrated new Arwen v1.4.1.
#3283 Removed TPS benchmark deprecated functionality, move the logic that saves the validator public keys from indexer to elrond-go.
#3286 Bump Arwen and vm-common versions.
#3287 Bump indexer version: latest indexer from master branch without elrond-go and Arwen dependencies, moved TransactionWithFeeHandler in elrond-go-core repository.
#3273 Made the p2p configs apply automatically when the full archive mode is selected.
#3270 Added logs for smart contract deploys operations.
#3314 Bump libp2p dependencies versions: libp2p/go-libp2p-transport-upgrader, libp2p/go-libp2p-transport-upgrader.
#3315 Removed the crypto package and import the elrond-go-crypto repository instead.
#3289 Added ECC integration tests + gas costs fixes.
#3294 Added an extra command in the Makefile in order to install protobuf. This new command make install-proto will download an prepare protobuf dependency in order to enable the run go generate ./... operation.
#3319 Bump Arwen and vm-common versions.
#3321 Bump crypto repository version, multisig stubs/mocks cleanup.
#3331, #3335 Updated gas maps for the new arwen functionality.
#3332 Several refactoring tasks were completed so the code can be better used (moved some interfaces & structs were they belong & other minor fixes).
#3336 Added in the indexer config file the new available indexes.
#3318 No more global mint burn for ESDTs, added ESDT transfer role. Finished new Arwen versions integration.
#3337 Added a new API route /address/:address/esdts/roles that will return all the tokens identifiers and roles for the given address.
#3351 New ESDT roles, integrated new Arwen v1.4.7m, added gas costs for the new VM API functions.
#3361 Added new gas costs, integrated new Arwen v1.4.8.
#3381 Referenced a vm-common v1.1.4 that includes the token ID in the errors messages returned for multi ESST/NFT transfers.
#3329 Added logs and events for the delegation system smart contract.
#3398 Bump Arwen version to 1.4.9 and new vm-common to v1.1.8, added a new gas cost.
#3383 ESDT multi-transfer tests for the following scenarios: user to smart contract, user to smart contract to user, user to smart contract to smart contract. Both same shard and cross shard.
#3403 Optimized consensus signature share verification.
#3412 Bump Arwen version to v1.4.10.
#3420 Bump Arwen version to v1.4.11.
#3423 Optimized smart contract results creation.
#3426 Bump Arwen version to v1.4.12 and new vm-common to v1.1.9.
#3419 Bump go libp2p dependency library to v0.2.8.
#3451 Delegation smart contract optimization when computing rewards.
#3442 Bump vm-common version to v1.2.1.
#3458 Bump Arwen version to v1.4.13.
#3459 Bump Arwen 1.4.14 and smartcontract result size invariant check.

Bug fixes:
#2550 Linter fixes.
#2960 Before this fix, if no destination shard as observer was set (so DestinationShardAsObserver = "disabled" in prefs.toml) then the "chosen" shard would have been 0. In order to avoid this, a random (or at least one that isn't always 0) shard should be generated.
#2930 Fixed local testnet scripts
#2839 Prevent deletion of the snapshot DB: disconnect without removing snapshots if keepSnapshots flag is true.
#3105 Bump the indexer version that works with the latest version of kibana.
#3037 Fixed the unclosed go routines & other optimizations and fixes on the development branch that uses the soft restart feature.
#3191 Code coverage fixes: added -short flag for the code coverage run.
#3118 Import-db process parameter tweak.
#3194 Linter fixes, code cov now runs the tests in short mode.
#3195 Fixed multiple smart contracts generation with the same values by adding an extra transfer nonce value, added integration test for the permanent fix.
#3203 Config compaction & alignment with the mainnet.
#3205 Override health records params during import-db process.
#3122 Make the REST API route /transaction/cost work with complex transactions like ESDT transfer with SC call, multiple async calls, built-in function etc.
#3213 Fixed messenger creation in seednode app.
#3222 Removed token type from log entry that is generated on the NFTCreate operation. We cannot know the token type from the shard perspective.
#3224 Refactored log initialization.
#3223 Full archive fixes: moved full archive config in prefs.toml file, full history nodes sub-system fixes, closed interceptor container for triechunks.
#3236 Fixed the closing of the triechunksprocessor component on bootstrap.
#3247 Return the same pointer if bytes are equal on load account.
#3241 Treat smart contract results calls differently when there are return calls with error.
#3251 Fixed logging of storage information and files opened, fixed restart of full history node in epoch 0, fixed eviction of old pruning storers for full archive nodes.
#3255 Fixed full history prints & miscellaneous prints.
#3266 Implemented port retrial mechanism that will prevent node creation errors in extremely rare situations.
#3269 Fixed a bug from checkpoint hashes holder: in case the rootHash does not change, do not add it to the cache each time.
#3277 Fixed the double list trie sync implementation timeout if no nodes are received in a specified amount (+ added unit and integration tests).
#3290 Changed the output .csv file mode permission level, removed the execute flag on all .wasm files.
#3293 Added context checking when committing nodes during the trie snapshot operation.
#3274 Fixed pruning buffer is full error, improved user account checking.
#3307 Updated local testnet scripts (made hysteresis 0 so observers selection will work), updated txgen scripts and config.
#3299 Refactored trie storage manager take snapshot/take checkpoint operations, change the order of trie snapshot search in GetSnapshotThatContainsHash.
#3291 Added additional logs when closing gin webserver for debugging purposes.
#3327 Fixed formatting in api.toml file.
#3333 New arwen v1.3.28 that fixes a backwards compatibility issue.
#3338 Backwards compatibility for the SaveKeyValue builtin function.
#3344 Bump vm-common version to v1.1.3 that fixes a backwards compatibility issue.
#3346 Fixed a merge error that occurred in the nodeTransactions.go file.
#3349 Removed Arwen make call from the release workflow.
#3355 Updated the README.md file to reflect that the Arwen binary is no longer being required to be built.
#3354 Fixed OOM exceptions by closing the snapshot DB's used in snapshot process, minor code changes around some log prints, refactored pruningStorerFactory.go so it does not use slice pointers, bump gin web server version to v1.7.4.
#3350 Fixed the trie node test point.
#3373 Fixed an OOM exception regarding unclosed components at shuffle out, fixed a wrong print in node.go, added .toml config file example, fixed a broken link in the README.md file.
#3386 Fixed linter issues (removed unused consts and one function), removed some comments, made all fmt.Printf calls contain a \n terminator, so it won't interfere with the test result interpretation, fixed warnings & weak warnings in tests.
#3390 Set decrease leader log from TRACE to DEBUG.
#3376 Fixed the tries sync timeout by using a common component on all syncers that will manage the timeout event.
#3395 Fixed the edge case when computed transaction fee is negative.
#3394 Fixed OOM exceptions by correctly closing all involved channels when performing snapshot/checkpoint, fixed OOM exceptions by correctly using the throttler inside the user accounts syncer as to not allow starting of a large number of go routines.
#3407 Fixed go routines counter in tests by using a specialized component.
#3411 Fixed trie sync timeout false positive trigger by resetting the watchdog when the node builds up its list of data tries to be synced.
#3410 Fixed backwards compatibility issue.
#3422 Added ntp go routine exception in integration tests.
#3431 Use the network sharding collector (peer mapper) from processComponents instead of direct field.
#3433 Do not display log.Warn when delegator.ActiveFund is empty.
#3434 Fixed termui remaining epoch time.
#3457 Fixed the application status handler: fixed MetricRoundAtEpochStart reset issue after bootstrap, proper fix for app status handler
refactoring on the metrics area, added unit tests, removed redundant mocks.
#3463 Backwards compatibility on save key value error.