Release notes

Major highlights

• Significant memory usage reduction
  Check out the graph below for v1.18.0 vs previous versions
  
• A new experimental full pruning method
  This significantly reduces full pruning time at the expense of RAM. It can be turned on via --Pruning.FullPruningMemoryBudgetMb <memory budget>. Internal testing shows a 2-4x reduction in pruning time with 4000 MB of memory budget and a 4-7x reduction with 8000 MB with a diminishing return above 16000 MB.
• Important stability fixes
  We fixed edge case bugs during full pruning that could result in TrieException errors. We also improved the stability of the client by auto repair from the "invalid best state calculation" error. In some setups, Nethermind used to require root permissions because of the current implementation. Issues related to missing total difficulty on archive nodes are also fixed. Check the full changelog for details.
• Improved sync time and block processing
  In our internal tests with a good network and hardware, Nethermind v1.18.0 can start attesting blocks on the Ethereum Mainnet in just over an hour. After 4 hours, the full download (including bodies and receipts) is completed leading to a steady state and improved attestations.
• Progress toward the next hard-forks
  We merged @gnosischain withdrawals and started merging 4844-related PRs.

New features

• Disk space savings for non-validator nodes
  You can significantly save disk space if you don't need chain history -- old bodies and receipts. Validators shouldn't enable this setting, because consensus clients require history to sync deposits. With these settings turned on, you will save around 450 GB of disk space for Ethereum Mainnet. To enable this feature, set --Sync.NonValidatorNode true --Sync.DownloadBodiesInFastSync false --Sync.DownloadReceiptsInFastSync false

• Configurable startup disk space check
  Added a new check for free disk space during node startup (Health Checks plugin). By default, the node will shut down if available space is less than twice the shutdown threshold or half of the warning threshold (if specified). A new flag --HealthChecks.LowStorageCheckAwaitOnStartup true can suspend initialization until enough free disk is available for safe node operation.

  Change is provided especially to address cases when a node is hosted in an environment with automatic restart, e.g. docker-compose with restart policy.

• Pass-through RocksDB options
  Allows specifying arbitrary RocksDB settings. The setting must be dynamically configurable. It can be passed as JSON to the --Db.AdditionalRocksDbOptions of DbConfig.

• Additional metrics output
  A new option of publishing the metrics that can be gathered via dotnet-counters. To enable counters to pass, set --Metrics.CountersEnabled true.
  They can be collected from a running Nethermind process using dotnet-counters collect -n Nethermind.Runner.

Experimental features

• Flags to tune RocksDB during sync to minimize write amplification at the expense of read amplification (during sync only) saving some SSD write endurance. It can be turned on via --Sync.TuneDbMode <level> where <level> is WriteBias, HeavyWrite, or AggressiveHeavyWrite. Some users may use this to reduce sync time while others may use it to increase it.

Changelist

Cancun

• EIP-4844 transaction validation by @flcl42 [#5473]

Gnosis

• Added support for Gnosis beacon chain push withdrawals @rubo [#5160]
• Added timestamp activation for EIP-170 and EIP-2537 by @deffrian [#5338]
• Fixed the block lookup issue for the Chiado archive configuration by turning off pruning by default by @MarekM25 [#5616]
• Renamed "xDAI" to "Gnosis" on client startup by @MarekM25 [#5502]

JSON-RPC

• Added eth_getAccountendpoint by @Demuirgos [#5352]
• Improve eth_getFilterChanges error codes by @LukaszRozmej [#5513]
• Added Geth-compatible boolean flags for transaction inclusion for the newPendingTransactionsand newHeads options of eth_subscribe JSON-RPC method [#5299]

Metrics

• Introduction of System.Diagnostics.Metrics by @Scooletz [#5441]
• Add missing metric PendingTransactionsHashesSent in eth68 handler by @marcindsobczak [#5478]

Bug fixes and stability

• Fix crash after full pruning complete by @asdacap [#5458]
• Fix random trie exception after pruning by @asdacap [#5471]
• Auto repair for Invalid Best State Calculation by @MarekM25 [#5617]
• Fix required root access to some directories by @deffrian [#4320]
• Fix edge case with Total Difficulty during archive-sync by @MarekM25 [#5566]
• Fix issues related to total difficulty by @deffrian [#5152] [#5439]
• Fix/statesync stuck on bad peer by @asdacap [#5131]
• Check free disk space during startup by @damian-orzechowski [#5153]
• Fix occasional start-up hang or crash by @benaadams [#5547]
• Disconnect Peers based on ForkId validations by @smartprogrammer93 [#5172]
• Feature/max open files flag and write rates limit by @asdacap [#5287]
• Don't disconnect peers on fast sync criteria by @LukaszRozmej [#5293]
• Add missing ReportActionEnd on pre-Homestead blocks by @LukaszRozmej [#5316]
• Sleep peer if can't get account range by @asdacap [#5323]
• TraceStore - Check depth when serializing by @LukaszRozmej [#5344]
• Fix Allocated report by @LukaszRozmej [#5351]
• Fix exception from concurrent use of List by @benaadams [#5385]
• Fix SnapSync warning by @tanishqjasoria [#5388]
• Stronger guarantees of state return during failures by @benaadams [#5389]
• Fix missing db statistics for column family by @asdacap [#5428]
• Fix fast header disconnecting peers sometimes by @asdacap [#5434]
• Handle TaskCancelledException on startup by @smartprogrammer93 [#5492]
• Fix column families config by @asdacap [#5545]
• Fix arraypool hangs when capacity is 0 by @asdacap [#5562]

Plugin development

To make Nethermind plugin development easier, we published Nethermind reference assemblies as Nethermind.ReferenceAssemblies NuGet package and as a separate archive to GitHub Releases [#5496]

Logging

• Print version on startup by @deffrian [#5467]
• Fix "Parent wasn't processed" in logs by @Demuirgos [#5354]
• Cleanup in logs. Changed false positive warnings to info by @MarekM25 [#5297] [#5493]
• Fix missing jwt warning by @julio4 [#5362]
• Fix processing speed is -1. by @asdacap [#5551]

Performance

• Disable OldBodies/Receipts for non validator nodes by @deffrian [#4755]
• Upgrade to RocksDB v8 by @rubo [#5557]
• Remove IMessageSerializer, use IZeroMessageSerializer for all messages by @tanishqjasoria [#5114]
• Perf/idb uses span as key by @asdacap [#5155]
• StorageCell byref by @Scooletz [#5265]
• Skip intermediate MemoryStream in EciesCipher.Encrypt by @benaadams [#5269]
• Use input buffer directly for Sha256Precompile by @benaadams [#5270]
• Only re-snapshot SortedPool on changes by @benaadams [#5272]
• Make ArrayPoolList.GuardIndex inlinable by @benaadams [#5275]
• .AsSpan().Slice(x,y) => .AsSpan(x,y) by @benaadams [#5277]
• Refactor tx filtering by @marcindsobczak [#5281]
• StorageTree opitimization by @Scooletz [#5283]
• Dispose ArrayPoolList after use by @benaadams [#5285]
• Multi-threaded block signature recovery by @LukaszRozmej [#5288]
• Filter low gas txns early by @benaadams [#5290]
• BouncyCastle.KeccakDigest => Nethermind.KeccakHash by @benaadams [#5291]
• Remove 14MB from RunPeerCommit allocations by @benaadams [#5294]
• Release non-null empty spans by @benaadams [#5308]
• Remove 28MB of permanent allocations from SyncStatusList (32MB -> 8MB) by @benaadams [#5309]
• Remove 15MB of string allocations in SimpleDb.LoadData by @benaadams [#5310]
• ValidateSignature move from BigInteger to UInt256 by @benaadams [#5315]
• Remove 35MB of allocations in TypeDiscovery by @benaadams [#5312]
• Perf/improve snap request concurrency by @asdacap [#5342]
• Remove obsolete ToArrays by @LukaszRozmej [#5358]
• Vectorize some Bytes operations by @benaadams [#5359]
• Non-blocking SortedPool.TryGetLast and IsFull by @LukaszRozmej [#5360]
• Optimize RlpStream.DeserializeLength by @benaadams [#5363]
• Optimize EvmStack by @benaadams [#5364]
• Remove unused runtime features by @benaadams [#5380]
• LruCache faster snapshot by @Scooletz [#5397]
• Use concrete dictionary rather than interface by @benaadams [#5398]
• ZeroPaddedSpan to readonly ref by @benaadams [#5400]
• Perf/pool trie store by @asdacap [#5401]
• Reduce VirtualMachine.Execute call stack by @benaadams [#5402]
• Make TraverseContext readonly and pass via in by @benaadams [#5405]
• In FastHeadersSyncFeed change ConcurrentDictionary to ConcurrentHashSet by @LukaszRozmej [#5407]
• ValueKeccak by @benaadams [#5416]
• Switch off DotNetty ResourceLeakDetector by @benaadams [#5421]
• Only clear NetworkNode cache on changes by @benaadams [#5426]
• Better address GetHashCode by @benaadams [#5431]
• Migrate To use BitOperation Class for BinaryOperations by @Demuirgos [#5432]
• Reduce state access for TxPool by @benaadams [#5433]
• Slightly smoothen out fast header dispatch by @asdacap [#5435]
• Fix memory hint not applied to database sometimes by @asdacap [#5390]
• Feature/sync db tuner by @asdacap [#5410]
• Don't resend tx hashes that were sent to us by @LukaszRozmej [#5449]
• Use write batch for full pruning write by @asdacap [#5459]
• Perf/get with options by @asdacap [#5463]
• Perf/batched trie visitor by @asdacap [#5475]
• Remove unnecessary GC finalization by @benaadams [#5479]
• Pause Garbage Collection during block processing by @LukaszRozmej & @benaadams [#5504]
• Vectorize HexConverter.TryDecodeFromUtf16 by @benaadams [#5539]

Other changes

• Creating PPA Package for recent Ubuntu versions by @hshafiq99 [#5178]
• Remove ensure terminal block by @MarekM25 [#5300]
• Remove deprecated EF bootnodes by @skylenet [#5408]
• Deterministic and ReferenceAssembly to Nethermind.Mev by @MicahZoltu [#5438]
• Improve RlpDecoder interface hierarchy by @tanishqjasoria [#5319]
• Feature/network diag tracer add message size by @LukaszRozmej [#5448]
• Fix spaceneth config. Removed baseline remnants by @MarekM25 [#5461]
• Remove old draft Snapshot implementation by @dceleda [#5477]
• Removed obsolete configs for old Account Abstraction and MEV plugin by @Arun9650 [#5524]
• Feature/additional rocksdb settings by @asdacap [#5536]
• Cleanup in test spec loader classes and KnownHashes by @MarekM25 [#5538]

New Contributors

• @hshafiq99 first contribution #5178
• @adityakode first contribution #5337
• @julio4 first contribution #5362
• @skylenet first contribution #5408
• @Arun9650 first contribution #5524

Full Changelog: 1.17.4...1.18.0