github onflow/flow-go v0.38.1
Feb 13th 2025 Height Co-orindated Upgrade on Mainnet26 at block height XXXX
on GitHub

pre-releaseone day ago

The Flow network has been upgraded!

This release brings a major enhancement to on-chain state storage with Account Storage Format v2, improving both the scalability and resiliency of the network.

✅ What is Included in This Upgrade?

  • Optimized Account Storage Format v2 for improved efficiency and scalability
  • Zero-Downtime State Migration - an innovative new method to seamlessly roll out the new account storage format on the live network without requiring any downtime.

🔹 What’s New in Account Storage Format v2?

The Account Storage Format v2 enhances how the on-chain state is stored and processed, resulting in significantly reduced memory usage on the Execution nodes.

The Execution State is a Merkle tree, which holds the user data (account token balances, NFTs, smart contracts deployed etc.) as payloads in its leaves. This update optimizes the storage format for the payloads resulting in fewer, more compactly serialized payloads. Fewer payloads in turn reduce the size of the MTrie, making Flow more efficient in handling state.

With this update,

  • The number of payloads will be reduced by 25-30%, reducing the on-disk storage footprint of the execution data. This will improve RAM, Disk and CPU efficiency, not only for execution nodes but also for all databases, indexers, and caches that store and serve execution data such as the access node.
  • The number of vertices in the MTrie will also be reduced by 25-30% reducing the in-memory storage footprint of the data for execution nodes, which hold the execution state in memory. With this size reduction, execution nodes will see more than an 11% reduction in memory usage (RAM) improving their stability and capacity to accommodate memory spikes during peak usage.

This improvement will also help slow down future memory growth rate as new accounts are created, further ensuring that Flow scales efficiently.
Read more about this change here 👉 GitHub Issue #3584

🔹How Will Account Storage v2 Be Rolled Out? Zero-Downtime Migration!

To deploy the new Account Storage Format without any network downtime, an innovative approach called Zero-Downtime Migration has been introduced.

Zero-Downtime Migration uses system transactions to migrate small batches of accounts, running in the background without affecting network operations. The ambient process takes several days to complete.

With this approach, the new Account Storage Format will be deployed gradually and seamlessly without requiring any action from developers or users, ensuring zero disruption to the network.

🔜 Upcoming improvements

This upgrade also lays the foundation for several upcoming improvements that will be rolled out as part of the next set of updates. These include:
Websocket Streaming API endpoints aimed at improving the user experience of dApps.
The ability to deploy protocol upgrades including downwards-incompatible changes for all node types to the live Mainnet - without requiring any network downtime!
Easy recovery from epoch transition failures, again requiring no downtime.

This upgrade marks another significant step in Flow’s journey toward scaling to petabytes of on-chain data. 🚀

❗Breaking change

There is a breaking change to streaming Access API in this upgrade.

Several AccessAPI and ExecutionDataAPI streaming endpoints include a message_index field in the response which indicates the order of messages sent by the server so the client can detect if they missed processing any. Previously, some endpoints used 1 as the first index in a stream while others used 0. After the HCU, all endpoints will be consistent regarding message_index and start from index 0.
APIs that will affected (Protobuf definition) -

  1. SubscribeEvents - field message_index
  2. SendAndSubscribeTransactionStatuses - field message_index
  3. SubscribeAccountStatuses - field message_index

These streaming APIs are fairly recent, therefore most apps will not be impacted.

⚠️ Deprecation

  • The protocol_version field of the NodeVersionInfo endpoint of the Access API is deprecated and will be removed in a future release.

Commit list

Account Storage Format v2 Migration

NOTE: Account Storage Format v2 is implemented in PRs in repo onflow/cadence.

Zero downtime (Recovering from failed epoch switchover)

  • Add test cases documenting CBOR's behaviour with omitted and extra fields by @jordanschalm in #6880

Data Availability

Collection

Consensus

Verification

Execution

Flow (Cadence) VM

Flow EVM

Database

Benchnet

Cadence

Networking Layer

Observer

Utils

  • Add a command to print all system addresses by @turbolent in #6339
  • Add command to debug transaction by @turbolent in #6385
  • Add command to debug a script by @turbolent in #6425
  • Check checkpoint file exists when running state extraction by @zhangchiqing in #6459
  • Fix duplication detection in root block finalization by @zhangchiqing in #6484
  • Add verify execution result cmd by @zhangchiqing in #6746
  • add nWorker to verify execution result by @zhangchiqing in #6830
  • Add query last executed sealed block by @zhangchiqing in #6857
  • Add flag to backward compatibilities testing tool to continue verify when mismatching is found by @zhangchiqing in #6954
  • Backport: Fix cluster assignment >2/3 internal check by @jordanschalm in #6433
  • Disable adx in util build by @tarakby in #6580
  • Add support for private registry builds by @sjonpaulbrown in #6742
  • Add support for cap_tag domain in check-storage command of util program by @fxamacker in #6406
  • Skip EVM storage account for value comparison in util program's diff-states command by @fxamacker in #6401
  • Port PR #6406 (Add support for cap_tag domain in check-storage cmd) to v0.37 branch by @fxamacker in #6417
  • Port PR #6407 (Add support for cap_tag domain in diff-states cmd) to v0.37 branch by @fxamacker in #6418
  • Add support for cap_tag domain in diff-states by @fxamacker in #6407
  • Improve pull for transit/bootstrap by @jhawk28 in #6353
  • Remove internal partners from root block creation by @zhangchiqing in #6472
  • Add stats about account format versions to util's checkpoint-collect-stats command by @fxamacker in #6882
  • Add support for account format v2 in util check-storage by @fxamacker in #6977
  • Add --exclude-randombeaconhistory to diff-states cmd in util program by @fxamacker in #6993
  • Add check-storage command to util program to check storage health of execution state by @fxamacker in #6364
  • Move and export some migration funcs to util package for reuse by @fxamacker in #6363
  • Add support for checking EVM registers in check-storage command by @fxamacker in #6430
  • Update checkpoint-collect-stats command to support payloads file and state commitment by @fxamacker in #6478
  • Collect and report more stats in checkpoint-collect-stats command by @fxamacker in #6482
  • Refactor the remote debugger and add support for using the execution data API by @turbolent in #6929
  • Allow using interface-converted type-requirement in fields by @SupunS in #6341
  • Add test for storage capability migration determinism by @SupunS in #6375

CI/CD

Maintenance and Lifecycle Updates

Misc

New Contributors

Full Changelog: v0.37.26...v0.38.1

Check out latest releases or
releases around onflow/flow-go v0.38.1

Don't miss a new flow-go release

NewReleases is sending notifications on new releases.

Get notifications