This is an alpha update and may not be ready for production use. This software was prepared by the Digital History Association, in cooperation from the wider Arweave ecosystem.
Data Sync
General data sync performance improvements resulting in more stable download throughput.
The problem
Set of candidate peers periodically drops low or to zero while syncing
The fix
Optimize peer discovery and sync_task queue:
One process per peer rather than one process per storage module. This prevents a lot of rate limiting that used to happen before when multiple storage module processes would hit the same peer concurrently. Now there's one process that can manage backpressure when making GET /data_sync_record to a peer
Several bug fixes on peer expiration, and "leak" bugs preventing data sync being completed.
Rate-limiting
Scalability
Server-side rate limiter has been improved to handle requests better under higher loads. Now, multiple rate-limiter processes handle the requests per rate-limiting-groups.
Rate-limiting timeout has been lowered to 1000ms, if this is breached requests will be responded with status 503.
Quota headers
Additional headers are present in the responses of the HTTP API. These headers provide information on the current quota for the calling endpoint, and will be used to optimise the client throttling in future releases.
Please see this link for documentation and further information on the headers.
Further improvements
- Optimised data table use for ar_data_discovery, as it previously could get bogged down removing empty peers.
- Several logging optimisations have been made to reduce repetive messages.
Community involvement
A huge thank you to all the Mining community members who contributed to this release by testing the alpha releases, providing feedback, and helping us debug issues!
Discord users (alphabetical order):
- Butcher_
- doesn't stay up late
- Evalcast
- lawso2517
- timothynode