sigp/lighthouse v4.4.111-exp

EXPERIMENTAL Tree States v4.4.111

on GitHub

Disclaimer

⚠️ You should not run this alpha release supporting mainnet validators ⚠️

If you are looking for the latest Lighthouse release please go to https://github.com/sigp/lighthouse/releases/latest.

Summary

This is an alpha release of upcoming changes to Lighthouse which improve disk usage and state management.

We are making this alpha release so that expert users may help us test these improvements. It is not backwards-compatible and not recommended for mainnet validators. Our current plan is to stabilise these changes shortly after the Deneb hard fork on mainnet.

For the adventurous, the main benefits are:

• Smaller disk footprint for archive nodes, <150GB total. Use checkpoint sync, and set the flag --reconstruct-historic-states.
• Smaller disk footprint for regular nodes, <30GB.
• Faster handling of re-orgs.
• Faster restarts.
• Less disk I/O.

We hope that this is useful for running block explorers and supporting other beacon chain analytics. We are using it internally at SigP to run some of our analytics.

Compared to the previous v4.2.990-exp release the main changes are:

• Updates to the freezer database schema. There are now less reads required per historic state, at the cost of greater disk usage. A mainnet archive node with the default configuration is now ~140GB. Historic state loads are around 5x faster in testing: 400ms instead of 2s in the worst case. A re-sync is required due to this change.
• Configurable storage hierarchy. Diffs can now be stored on a per-slot basis, or less frequently than every epoch. We recommend using the default configuration, and haven't done enough testing of other configurations to provide space/speed estimates. See --hierarchy-exponents in lighthouse bn --help.
• Holesky testnet support (from v4.4.1 upstream).
• No historic states stored by default (same as v4.4.1). Please use --reconstruct-historic-states to store historic states, even when syncing from genesis. For more information see the section of the v4.4.1 release notes on State Pruning.
• Lots of optimisations, particularly to epoch-processing, block-processing and the underlying state library (milhouse). This release should now be on-par or faster than stable Lighthouse when it comes to importing blocks.

This release is based on v4.4.1, and also includes the other features from v4.4.1 (and v4.3.0).

⚠️ Backwards Compatibility ⚠️

This release is not backwards compatible with stable Lighthouse or prior tree-states releases. It uses a different database schema (v24) for which no automatic upgrade or downgrade is implemented. We intend to implement an automatic upgrade process once the new schema is finalized. A re-sync might be required to run future versions of tree-states. This release's schema might be the final iteration.

Please only run this release if you are willing to re-sync now, and again in several weeks/months.

There are several breaking changes to CLI flags:

• The secret dummy flag --unsafe-and-dangerous-mode must be supplied. This is to prevent accidental upgrades to the release by users who don't know what they're getting themselves into. Unfortunately we found that far too many people accidentally ran previous tree-states releases on mainnet.
• The flags --slots-per-restore-point and --historic-state-cache-size are deprecated and no longer necessary due to the freezer database using hierarchical diffs.
• The flag --db-migration-period from the previous tree-states release is gone and has been replaced by --epochs-per-migration (same name as upstream).

Known Issues

Expect a few sharp edges. Some things you may run into:

• aarch64 (ARM) support is currently broken. As a result there are no binaries or Docker images for ARM.
• WARN Parent state is not advanced is logged excessively during sync. This is harmless, albeit annoying.

If you observe any panics, crashes or major performance regressions, please open an issue!

Building from source

Build the v4.4.111-exp tag.

Binaries

See pre-built binaries documentation.

A Docker image for x86_64 only is available on DockerHub as sigp/lighthouse:v4.4.111-exp.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

All Changes

• Release v4.4.111-exp (#4729)
• Restore custom image for Cross
• Bump schema version to v24
• Change default epochs-per-state-diff to 16
• Attempt to fix flaky test
• Disable ARM docker builds
• Merge remote-tracking branch 'origin/unstable' into tree-states
• Await listening address from libp2p in RPC tests setup (#4705)
• Tree states to support per-slot state diffs (#4652)
• Remove Node.js from release-tests CI job since we no longer use ganache (#4691)
• Feat: siren faq update (#4685)
• Fix some typos in "Advanced Networking" documentation (#4672)
• Minor revision to Lighthouse Book on validator-manager (#4638)
• Fix genesis state storage for genesis sync (#4589)
• Fix deadlock in finalization migration (#4576)
• Optimise mutations in single-pass epoch processing (#4573)
• Merge remote-tracking branch 'origin/unstable' into tree-states
• Single-pass epoch processing (#4483)
• Merge remote-tracking branch 'origin/tree-states' into tree-states
• Merge remote-tracking branch 'origin/unstable' into tree-states
• Implement activation queue cache
• Fix EpochCache handling in ef-tests (#4454)
• Fix EpochCache handling in ef-tests
• Tree states optimization using EpochCache (#4429)
• Fix db-migration-period default (#4441)

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v4.4.111-exp-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v4.4.111-exp-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v4.4.111-exp-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v4.4.111-exp-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v4.4.111-exp-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v4.4.111-exp-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v4.4.111-exp	sigp/lighthouse