ArweaveTeam/arweave N.2.8.0

Release 2.8.0

on GitHub

This Arweave node implementation proposes a hard fork that activates at height 1547120, approximately 2024-11-13 14:00 UTC. This software was prepared by the Digital History Association, in cooperation from the wider Arweave ecosystem.

Note: with 2.8.0 when enabling the randomx_large_pages option you will need to configure 3,500 HugePages rather than the 1,000 required for earlier releases. More information below.

Composite Packing

The biggest change in 2.8.0 is the introduction of a new packing format referred to as "composite". Composite packing allows miners in the Arweave network to have slower access to the dataset over time (and thus, mine on larger hard drives at the same bandwidth). The packing format used from version 2.6.0 through 2.7.4 will be referred to as spora_2_6 going forward. spora_2_6 will continue to be supported by the software without change for roughly 4 years.

The composite packing format allows node operators to provide a difficulty setting varying from 1 to 32. Higher difficulties take longer to pack data, but have proportionately lower read requirements while mining. For example, the read speeds for a variety of difficulties are as follows:

The effective hashrate for a full replica packed to any of the supported packing formats is the same. A miner who has packed a full replica to spora_2_6 or composite.1 or composite.32 can expect to find the same number of blocks on average, but with the higher difficulty miner reading fewer chunks from their storage per second. This allows the miner to use larger hard drives in their setup, without increasing the necessary bandwidth between disk and CPU.

Each composite-packed chunk is divided into 32 sub-chunks and then packed with increasing rounds of the RandomX packing function. Each sub-chunk at difficulty 1 is packed with 10 RandomX rounds. This value was selected to roughly match the time it takes to pack a chunk using spora_2_6. At difficulty 2 each sub-chunk is packed with 20 RandomX rounds - this will take roughly twice as long to pack a chunk as it does with difficulty 1 or spora_2_6. At difficulty 3, 30 rounds, and so on.

Composite packing also uses a slightly different version of the RandomX packing function with further improvements to ASIC resistance properties. As a result when running Arweave 2.8 with the randomx_large_pages option you will need to allocate 3,500 HugePages rather than the 1,000 needed for earlier node implementations. If you're unable to immediately increase your HugePages value we recommend restarting your server and trying again. If your node has been running for a while the memory space may simply be too fragmented to allocate the needed HugePages. A reboot should alleviate this issue.

When mining, all storage modules within the same replica must be packed to the same packing format and difficulty level. For example, a single miner will not be able to build a solution involving chunks from storage_module_1_addr.1 and storage_module_2_addr.2 even if the packing address is the same.

To use composite packing miners can modify their storage_module configuration. E.g. if previously you used storage_module 12,addr and had a storage module directory named storage_module_12_addr now you use storage_module 12,addr.1 and create a directory named storage_module_12_addr.1. Syncing, packing, repacking, and repacking in place are handled the same as before just with the addition of the new packing formats.

While you can begin packing data to the composite format immediately, you will not be able to mine the data until the 2.8 hard fork activates at block height 1547120.

Implications of Composite Packing

By enabling lower read rates the new packing format provides greater flexibility when selecting hard drives. For example, it is now possible to mine 4 partitions off a single 16TB hard drive. Whether you need to pack to composite difficulty 1 or 2 in order to optimally mine 4 partitions on a 16TB drive will depend on the specific performance characteristics of your setup.

CPU and RAM requirements while mining will be lower for composite packing versus spora_2_6, and will continue to reduce further as the packing difficulty increases. Extensive benchmarking to confirm the degree of these efficiency gains have yet to be confirmed, but with the lower read rate comes a lower volume of data that needs to be hashed (CPU) and a lower volume of data that needs to be held in memory (RAM).

Block Header Format

The following block header fields have been added or changed:

• packing_difficulty: the packing difficulty of the chunks used in the block solution. Both reward_address and packing_difficulty together are needed to unpack and validate the solution chunk. packing_difficulty is 0 for spora_2_6 chunks
• poa1->chunk and poa2->chunk: under spora_2_6 the full packed chunk is provided. Under composite only a packed sub-chunk is included. A sub-chunk is 1/32 of a packed chunk.
• poa1->unpacked_chunk and poa2->unpacked_chunk: this field is omitted for spora_2_6, and includes the complete unpacked chunk for all composite blocks.
• unpacked_chunk_hash and unpacked_chunk_hash2: these fields are omitted under spora_2_6 and contain the hash of the full unpacked_chunks for composite blocks

Other Fixes and Improvements

• Protocol change: The current protocol (implemented prior to the 2.8 Hard Fork) will begin transitioning the upload pricing to a trustless oracle at block height 1551470. 2.8 introduces a slight change: 3 months of blockchain history rather than 1 month will be used to calculate the upload price.
• Bug fix: several updates to the RocksDB handling have been made which should reduce the frequency of RocksDB corruption - particularly corruption that may have previously occurred during a hard node shutdown.
  • Note: with these changes the repair_rocksdb option has been removed.
• Optimization: Blockchain syncing (e.g. block and transaction headers) has been optimized to reduce the time it takes to sync the full blockchain
• Bug fix: GET /data_sync_record no longer reports chunks that have been purged from the disk pool

Community involvement

A huge thank you to all the Mining community members who contributed to this release by identifying and investigating bugs and providing guidance on performance tuning!

Discord users (alphabetical order):

• BloodHunter
• Butcher_
• dzeto
• edzo
• heavyarms1912
• lawso2517
• ldp
• MaSTeRMinD
• MCB
• Methistos
• qq87237850
• Qwinn
• sk
• smash
• sumimi
• tashilo
• Thaseus
• thekitty
• Vidiot
• Wednesday

Code Changes

• pool reduced poll time by @vird in #612
• Feature/packing difficulty by @ldmberman in #590
• Don't block on ar_sync_record:add when repacking by @JamesPiechota in #619
• Create an Interface to remote_console release command by @humaite in #620
• Introduce P3 and RocksDB fixes by @shizzard in #624
• Improvement/longer pricing window by @ldmberman in #616
• Fix p3 missing pattern by @humaite in #627
• Header Synchronization Improvement by @humaite in #625
• Switch to arweave.xyz by @humaite in #629
• Improve the speed of doctor-dump and allow omitting txs by @JamesPiechota in #630
• Ensure ar_global_sync_record is notified when a chunk is removed from the disk pool by @JamesPiechota in #626
• Arweave Should Not Build Before Being Stopped by @humaite in #621
• Ensure the unpacked chunk is passed from the H2 CM peer to the H1 CM peer by @JamesPiechota in #628

New Contributors

• @humaite made their first contribution in #620
• @shizzard made their first contribution in #624

Full Changelog: N.2.7.4...N.2.8.0