This release is a mandatory upgrade (technically, a hard fork) that corrects some stuck service node stakes caused by witnessing failures in the Oxen 11.3.0 release. While the witnessing bugs were corrected with most of the Service Node network upgraded to Oxen 11.3.2, the funds that had already gotten stuck before that update are corrected by this release along with a few other bug fixes and improvements.

This update schedules a hardfork (HF22) to apply the fixups at block 1871519, scheduled for Tue, 17 Jun 2025 00:00 UTC. Service nodes must upgrade by that time or will be decommissioned.

Stuck contribution fixes

A total of 10 specific service nodes have stuck funds that are corrected by this release:

• 2 HF20 nodes transitioned into HF21 with changed BLS keys, and as a result got kicked off the network (without crediting the stakes) as imposters a couple hours into HF21. This unusual edge case only affected nodes transitioning from HF20, and so cannot reoccur.
• 7 registrations ended up failing with stuck funds. A malfunctioning RPC provider in use by a major network operator, combined with a bug in validating L2 event votes in pulse quorums, led to 7 SN registrations getting denied by the oxen network, with those stakes in the Arbitrum contract, but because the service not network denied the events, the funds were not credited to the contributors. The bug itself was corrected with 11.3.2.
• 1 contract exit was similarly denied on the oxen chain by the same bug, resulting in the exited node being stuck in the "recently removed" node, but with the denied exit event, oxend will not release the funds. Like the above, this was fixed by 11.3.2.

While 11.3.2 prevented the bugs from occurring again, this 11.4.0 update credits the claimable amount of the contributors to the 10 service nodes at the 11.4.0 hard fork height. As this credit needs to be coordinated across all service nodes at once, this required a network hard fork to implement.

To be clear: these adjustments are not crediting new SESH to anyone, but rather are just updating the oxend credit associated with accounts to match the amount that these contributors have already made to the Arbitrum SNRewards contract but cannot currently be claimed.

Other changes and improvements

• We have reworked how rewards are calculated internally, starting at the HF22 added in this release. Back when batching was introduced with Oxen 10, everything was stored in milli-atomic OXEN, so that when one single block's reward got split across the many thousands of contributors in thousands of nodes, the numeric imprecision was not too large. This value never got too huge, though, because pending rewards got paid out twice per week for each address.

  In HF21, however, the claimable amount becomes cumulative: the oxen side keeps track of the total of all rewards and unstakes ever made, and the smart contract tracks the total claimed amounts ever paid. Claiming rewards involved getting a network-signed new cumulative level, submitting that to the contract, and then using the contract to pay the different between the new total and the contract's already-paid amount.

  The problem with this, however, is that this value grows forever, and slightly after 9 million SESH in lifetime rewards + unlocks for a single contributor it would overflow the maximum value storable in a 64-bit integer. While we aren't in imminent danger of hitting this problem, there are accounts with 2+ million SESH staked that could hit it in the not-so-distance future. Beginning at HF22, we change the reward calculation so that we still calculate each individual block rewards split in milli-atomics, but then round that off to the nearly atomic value when crediting an accounts rewards. This puts the the maximum at over 9 billion SESH, which is well out of harm's way. Because this does slightly change the reward (by, at most, 0.000000001 SESH per block per account) this change also required a hard fork to implement.

• We have reintroduced the BLS signature requirement to uptime proofs. Originally this was only included for HF20 (so that we could collect an initial set of BLS keys to seed the smart contract with), however we have observed at least 9 nodes on the network that are consistently producing invalid signatures for claims and exit signatures, which we believe to be caused by those operators not properly copying the key_bls file when migrating nodes.

  Upon investigation, we realized that there is no way at all to tell whether nodes have the proper BLS key, but it needs to have the same key it registered with to properly participate on the network.

• Improved rescan speed: "loading subsystems" is now nearly 50% faster than before, thanks to several optimizations. (This also improves blockchain syncing speed, as many of the same calculations are needed there):

  • The rewards database now stores contributor addresses in binary, avoiding the need to convert to/from string representations when working with the database.
  • Calculating a block's random seed is now more efficient, allowing faster determination of network quorums.
  • Some redundant calculations are avoided.
  • The "milli-atomic" issue discussed above avoids needing to divide several values by 1000, which makes a small improvement.
  • Some rewards database queries were optimized or could be eliminated.

• oxend is now less aggressive, and more verbose, about storing the SN state during a rescan, which can be intensive. Previously it would store a dump of the service node state (which is a fairly substantial amount of data) every minute while rescanning; it now saves once every 5 minutes, and randomly spreads out when it first saves to avoid having multiple oxend's restarted at once all hitting a very I/O intensive part of syncing at exactly the same time.

• Rebranded some messages that said "Loki network" to say "Session network" in the log message about being fully synced. By waiting sufficiently long to make this important change, we made an important optimization that saved an entire commit from the oxen-core commit history.

• Fixed an issue where the ONS database would need to be rescanned from the beginning of the chain after popping blocks. (This mainly affects dev usage as manual block popping is not a common production oxend operation).

• Fixed an issue where only the last purge in a block containing multiple purges was being processed.

• Cleaned up various old tables/triggers in the batching database that are no longer used, but hadn't been dropped on upgrades from Oxen 10.

• Various other small improvements and fixes (see commit log).

Updating service nodes

For service node operators using Debian 11+ and Ubuntu 20.04+, the upgraded packages are available from our APT repository. Static binaries for other users are available below.

Ubuntu 20.04 warning

Note that Ubuntu 20.04 support will likely be dropped for the next mandatory service node upgrade as 20.04 is no longer receiving OS security updates. While we do not have a planned date for the next release this yet, we strongly encourage operators still running 20.04 to start upgrading to 22.04 or 24.04 as soon as possible.