sigp/lighthouse v4.6.0-rc.0

Concerto

on GitHub

Disclaimer

⚠️ You should not run this pre-release supporting mainnet validators. This pre-release is, however, required for Goerli validators ⚠️

If you are looking for the latest Lighthouse release please go to https://github.com/sigp/lighthouse/releases/latest.

Something went wrong with signing this particular release, but the signatures have now been fixed and the binaries are provided in .tar.gz format as usual.

Summary

This is a pre-release that enables the Deneb hard fork on Goerli (#5036). The Deneb hard fork will occur at epoch 231680 on Goerli, which is Jan 17th, 2024 06:32:00 AM UTC.

All Goerli users must update their nodes to v4.6.0-rc.0 by Jan 17th, 2024 06:32:00 AM UTC.

This release also includes a variety of new features and a few breaking changes. Please see the section on Breaking Changes below.

Networking Improvements

Some extensive changes have been made to the networking components in this release. We have focused on a number of performance and structural changes to the gossipsub protocol and discovery mechanism. Some of the main gossipsub changes are listed in issue (#4918). An overview of the primary changes are:

• Removed gossipsub flood publishing (#4383) - Aimed to reduce message propagation latency and lighten the memory footprint
• Upgraded libp2p to v0.53.* (#4935)
• Enabled concurrent discovery RPC requests
• Improved handling of slow peers at the gossipsub level
• Handled memory allocations in message transmission in gossipsub
• Improved prioritisation of messages in gossipsub

Standard block V3 endpoint

The new produce block V3 API has been standardized across clients (ethereum/beacon-APIs#339). This endpoint allows us to give validators more control over whether to prefer a local or builder payload during proposals. The validator client will not use this new endpoint by default, it must be enabled by adding --produce-block-v3 to the validator client. With the advent of V3, the following flag will no longer be supported:

--builder-profit-threshold

Users of mev-boost can use the -min-bid flag to achieve the same functionality. An equivalent but more expressive config will be possible in an upcoming Lighthouse release (#5039). This config will also be able to restore functionality of the following flag, which is also being deprecated:

--always-prefer-builder-payload

Per-validator configuration of whether to use an external builder is still possible via the builder_proposals field. This will be extended to allow each validator to specify at what profit threshold a builder payload should be used in an upcoming release (#5039).

The block V3 endpoint also allows validators working with multiple beacon nodes (e.g. via Vouch) to compare rewards between beacon nodes before selecting payloads. This is done by exposing the consensus rewards of the block, as well as the execution rewards of the payload directly to the validator.

On the block V2 endpoints, if a validator is using the blinded block flow and gets exceptionally unlucky, falling back to another beacon node at an inopportune time, it could miss its proposal. This is no longer possible with the V3 endpoint.

Execution layer payload selection input

Recent changes in the execution APIs have also provided a new should_override_builder field along with each local payload. Lighthouse will check if this is set to true and return a local payload if it is. This field allows the execution layer to monitor the mempool and look for signs of ongoing censorship from block builders. It can signal its suggestion to use a local payload to combat censorship to the consensus layer to select a local payload in these scenarios.

Validator broadcast

Validator clients can now be configured to publish messages to all connected beacon nodes using the --broadcast flag. This allows users to improve redundency when publishing messages related to validator duties. The flag can be configured with specific message types (attestations,sync-committe,blocks, subscriptions). It can also be configured to make sure subscriptions are not sent to all beacon nodes (--broadcast none). This is equivalent to the --disable-run-on-all flag. Thanks @uvizhe for the contribution!

More information is available in the Redundancy section of the Lighthouse book.

(see #4920)

Standard Liveness Endpoint

The validator client now uses the standard liveness endpoint for doppelganger protection (see #4853). This makes the Lighthouse VC's doppelganger protection compatible with any beacon node that implements the standard liveness endpoint.

Major Changes

• Deneb support, enabled on Goerli (#5036)
• Fix an OOM crash caused by a build-up of messages in gossipsub send queues (#4918)
• Other networking improvements (libp2p upgrade, disable flood publishing)
• Checkpoint sync required by default (#5038)
• New HTTP APIs
• Database fixes for Holesky nodes (#4820, #4985)
• Faster fork choice by default (#4971)
• Faster block production, decreasing the chance of being reorged (#4925)
• Validator client broadcast feature (#4920, https://lighthouse-book.sigmaprime.io/redundancy.html#broadcast-modes)
• More efficient subscriptions in the validator client, which reduces load on machines with large numbers of validators (#4806)
• Attestation simulator (#4880)
• Missed block tracking (#4731)
• Command to prune states from the database (#4835)
• Fixes to rewards APIs (#4807, #4877)

New HTTP APIs

The Lighthouse BN supports the following new HTTP APIs:

• The new, standard produce block V3 endpoint (ethereum/beacon-APIs#339, ethereum/beacon-APIs#358, ethereum/beacon-APIs#386)
• POST APIs for fetching validators & balances in bulk: https://github.com/sigp/lighthouse/pull/
• Add standard graffiti management APIs (#4951)
• The standard liveness endpoint for doppelganger protection (#4853)
• GET /eth/v1/beacon/blob_sidecars for Deneb (https://ethereum.github.io/beacon-APIs/#/Beacon/getBlobSidecars)

⚠️ Breaking Changes ⚠️

VC and BN Compatibility

The Lighthouse VC now uses the liveness endpoint from the standard API, rather than a custom Lightouse endpoint (#4853). This only affects the --enable-doppelganger-protection feature.

When Doppelganger protection is enabled, Lighthouse VCs from this version will not work with Lighthouse BNs prior to v4.4.1 (released Sept. 2023).

Database Migration v18

To support Deneb, this release includes an automatic upgrade from v17 to v18.

To downgrade, follow the instructions in the book for Database Migrations.

Checkpoint sync is required by default

Genesis sync will no longer work without the use of --allow-insecure-genesis-sync. Checkpoint sync should always be preferred to protect yourself from long-range attacks. Additionally, genesis sync is not compatible with data availability checks that are added in Deneb. On chains that have been running for less than ~2 weeks, genesis sync will still work without --allow-insecure-genesis-sync, so this change won't impact local testnets or short-lived testnets.

In order to support archive nodes via checkpoint sync, both of the following flags should be used: --reconstruct-historic-states and --genesis-backfill, along with any other flags normally used for database customization.

Removed flags

The following flags have been deprecated in prior releases and are being removed. Lighthouse will not start up if any are included in the startup command.

• --spec for lighthouse bn
• --eth1-endpoint for lighthouse bn
• --eth1-endpoints for lighthouse bn
• --beacon-node for lighthouse vc
• --server for lighthouse vc
• --delete-lockfiles for lighthouse vc
• --allow-unsynced for lighthouse vc
• --strict-fee-recipient for lighthouse vc
• --merge for lighthouse bn
• --count-unrealized for lighthouse bn
• --count-unrealized-full for lighthouse bn
• --http-disable-legacy-spec for lighthouse bn
• --minify for lighthouse account validator slashing-protection import/export

(see #4906)

Deprecated flags

The following flags should be removed from setups. If they are not removed, they will have no effect.

Deprecated: --builder-profit-threshold flag

Prefer using the -min-bid flag in mev-boost. An alternative flag in Lighthouse will be provided in an upcoming release (see #5039).

Deprecated: --always-prefer-builder-payload flag

An alternative flag in Lighthouse will be provided in an upcoming release (see #5039).

Deprecated: --disable-run-on-all flag

This flag has been superseded by the --broadcast flag. Use --broadcast none to disable broadcast to all nodes (see #4920).

Rust MSRV

The minimum supported Rust version has been udpated from 1.69.0 to 1.73.0

Users who compile from source (i.e., not Docker or pre-built binary users) will receive the following error if they are using an earlier version of Rust:

lighthouse v4.6.0-rc.0 (/home/sigp/lighthouse/lighthouse)` cannot be built because it requires rustc 1.73.0 or newer

Users can typically obtain the latest version of Rust by running rustup update.

Update Priority

This table provides priorities for which classes of users should update particular components.

See Update Priorities more information about this table.

The Beacon Node must be running v4.4.1 or later to support a VC running this release. We recommend updating both the VC and BN to the same release.

This release is high-priority for Goerli users. Goerli users must update both the VC and BN.

All Changes

• Bump versions
• Switch libp2p sigp gossipsub fork (#4999)
• Fix IncorrectAttestationSource error in attn. simulator (#5048)
• Fix block v3 reward encodings (#5049)
• remove needless Into (#5026)
• Disallow genesis sync outside blob pruning window (#5038)
• Runtime rpc request sizes (#4841)
• Use blocks v3 endpoint in the VC (#4813)
• create unified slashing cache (#5033)
• Add hint for solving CLI tests failure (#5041)
• update goerli config (#5036)
• Block v3 builder boost factor (#5035)
• Track shared memory in metrics (#5023)
• Bump zerocopy (#5024)
• Clean up blockv3 metadata and client (#5015)
• Information for network testing (#4961)
• suppress error on duplicate blobs (#4995)
• Fix incorrect blob queue metrics (#5014)
• Update dependencies to get rid of the yanked deps (#4994)
• Fix off-by-one bug in missed block detection (#5012)
• Add flag to disable warning logs for duplicate gossip messages (#5009)
• Convert a FullPayload to a BlindedPayload correctly (#5005)
• Add attestation simulator (#4880)
• Use the block header to compute the canonical_root (#5003)
• add forK_choice_read_lock as parameter (#4978)
• Update Rust version in lcli Dockerfile. (#5002)
• Remove bors.toml 🫡 (#5001)
• Add mergify merge queue configuration file (#4917)
• Disable flood publishing (#4383)
• Batch Verify RPC Blobs (#4934)
• Remove delayed lookups (#4992)
• Implement POST validators/validator_balances APIs (#4872)
• Update docs for v4.6.0 (#4982)
• upgrade libp2p to v0.53.* (#4935)
• Fix corrupted DB on networks where the first slot is skipped (Holesky) (#4985)
• Implement graffiti management API (#4951)
• CLI in Lighthouse Book (#4571)
• Small Improvements (#4980)
• Sidecar inclusion proof (#4900)
• EIP-3076 tests - complete database (#4958)
• Fix stuck linkcheck on CI (#4977)
• Enable progressive balances fast mode by default (#4971)
• Standard Liveness Endpoint (#4853)
• Clone state ahead of block production (#4925)
• Refactor & Fix Bugs in Payload Selection Logic (#4950)
• Add lcli mock-el (#4587)
• API for LightClientBootstrap, LightClientFinalityUpdate, LightClientOptimisticUpdate and light client events (#3954)
• Remove block-delay-ms (#4956)
• Remove MAX_BLOBS_PER_BLOCK from Holesky config. (#4955)
• Broadcast various requests as per #4684 (#4920)
• Merge pull request #4939 from eserilev/block-v3-general-tests
• Fix Rust beta compiler warnings (rustc 1.75.0-beta.1 (782883f60 2023-11-12)) (#4932)
• Remove legacy database migrations (#4919)
• Fix missed block logs (#4922)
• Fix for issue 4860 - Added in process_justification_and_finalization (#4877)
• Always use a separate database for blobs (#4892)
• Update local testnet doc and parameters (#4749)
• Delete deprecated cli flags (#4906)
• Verify KZG in Bulk During Block Sync (#4903)
• Add block-v3 SSZ tests (#4902)
• Add missed blocks to monitored validators (#4731)
• more logging improvements (#4885)
• Fix comment for blob sidecar observation pruning (#4893)
• Add commmand for pruning states (#4835)
• Block v3 endpoint (#4629)
• fix deneb sync bug (#4869)
• Activate clippy::manual_let_else lint (#4889)
• restore cargo vendor in test suite (#4886)
• Fix block backfill with genesis skip slots (#4820)
• fix docs about --builder (#4754)
• deneb related logging improvements (#4859)
• Add block roots heal logic in v18 schema migration. (#4875)
• don't make lcli on self-hosted runners (#4874)
• Upgrade to v1.4.0-beta.3 (#4862)
• Reduce calls to network channel (#4863)
• Enable BLS portable feature on all CI tests (#4868)
• Add make lint to development environment section in Book (#4866)
• #4512 inactivity calculation for Altair (#4807)
• fix typos (#4838)
• update libp2p (#4864)
• Fix: write post state in lcli skip-slots (#4843)
• chore: replace deprecated hub with gh for releases (#4839)
• Add CARGO_USE_GIT_CLI to the Dockerfile to work around an OOM bug during cross-compiling (#4828)
• collect bandwidth metrics per transport (#4805)
• Deneb review .github (CI cleanup) (#4696)
• Reduce nextest threads to 8 (#4846)
• Fix Rayon deadlock in test utils (#4837)
• Generalise compare_fields to work with iterators (#4823)
• Remove blob clones in KZG verification (#4852)
• Fix Homebrew link (#4822)
• remove crit! logging from ListenerClosed event on Ok() (#4821)
• Add vendor directory to .gitignore (#4819)
• Fix metric for total block production time (#4794)
• Pre-generate test blobs bundle to improve test time. (#4829)
• Fix broken Nethermind integration tests (#4836)
• Merge Deneb (#4054)
• Deneb pr updates 2 (#4851)
• Reduce attestation subscription spam from VC (#4806)
• Revise doc API section (#4798)
• Address Clippy 1.73 lints (#4809)
• Very minor own nitpicks (#4845)
• Use Deneb fork in generate_genesis_header
• Add blob_sidecar event to SSE (#4790)
• Remove serde derive references (#4830)
• Delete unused ssz_types file (#4824)
• Prevent Overflow LRU Cache from Exploding (#4801)
• Address Clippy 1.73 lints on Deneb branch (#4810)
• Merge pull request #4808 from jimmygchen/merge-unstable-to-deneb-20231005
• Use peeking_take_while in BlockReplayer (#4803)
• Exit aggregation step early if no validator is aggregator (#4774)
• fix(validator_client): return http 404 rather than 405 in http api (#4758)
• Remove deficit gossipsub scoring during topic transition (#4486)
• Optimise head block root API (#4799)
• Don't downscore peers on duplicate blocks (#4791)
• Removed old Teku mainnet bootnode ENRs (#4786)
• enforce non zero enr ports (#4776)
• PeerManager: move the check for banned peers from connection_established (#4569)
• Use only lighthouse types in the mock builder (#4793)
• Ultra Fast Super Slick CI (#4755)
• Allow libp2p to determine listening addresses (#4700)
• Clean bors.toml (#4795)
• add processing and processed caching to the DA checker (#4732)
• insert cached child at the front of a chain of parent lookups (#4780)
• Add serde(default) to max_per_epoch_activation_churn_limit in spec config so that VC is compatible to older BN versions. (#4783)
• reduce blob prune logging in forward sync (#4779)
• Merge pull request #4781 from jimmygchen/merge-unstable-to-deneb-20230926

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v4.6.0-rc.0-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v4.6.0-rc.0-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v4.6.0-rc.0-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v4.6.0-rc.0-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v4.6.0-rc.0-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
	aarch64	lighthouse-v4.6.0-rc.0-aarch64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v4.6.0-rc.0-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v4.6.0-rc.0-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v4.6.0-rc.0	sigp/lighthouse