Mainnet 21

This release includes the following major changes,

1. Active pacemaker
2. BFT change to support Permissionless AN

💥 Breaking Changes

Upgrade of Flow's consensus to the Jolteon protocol, a substantially improved HotStuff derivative

• [Consensus] Active Pacemaker, SafetyData interface changes by @durkmurder in #2153
• [Consensus] Active Pacemaker, TimeoutAggregator interface changes by @durkmurder in #2155
• [Consensus] Timeout objects cache by @durkmurder in #2184
• [Consensus] Active Pacemaker, PaceMaker interface changes by @durkmurder in #2172
• [Consensus] Active Pacemaker multi message BLS aggregation by @durkmurder in #2243
• [Consensus] Support block-scoped and epoch-scoped identity queries in HotStuff committee by @jordanschalm in #2328
• [Consensus] Block header change by @durkmurder in #2367
• [Consensus] Active Pacemaker signer indices by @durkmurder in #2494
• [Consensus] Active Pacemaker SafetyRules implementation by @durkmurder with contributions from @AlexHentschel in #2444
• [Consensus] SafetyRules extra test cases by @durkmurder in #2557
• [Consensus] Active Pacemaker: entities validation by @durkmurder in #2456
• [Consensus] Add method for timeout weight threshold to Committee by @jordanschalm in #2438
• [Consensus] Add Safety Threshold for EECC by @jordanschalm in #2498
• [Consensus] ActivePaceMaker implementation by @durkmurder in #2564
• [Consensus] Active Pacemaker: Timeout Collector implementation by @durkmurder in #2620
• [Consensus] Active Pacemaker: Forks cleanup by @durkmurder with contributions from @AlexHentschel in #2685
• [Consensus] Trigger Epoch Fallback Based on Epoch Commitment Deadline by @jordanschalm in #2625
• [Consensus] Active Pacemaker: TimeoutAggregator implementation by @durkmurder in #2740
• [Alternative Implementation] Update Committee for changed epoch fallback trigger condition by @jordanschalm in #2782
• [Consensus] Replace DecreasingPruningHeightError with BelowPrunedThresholdError by @durkmurder in #2754
• [Consensus] EventHandler implementation and consensus modules integration by @durkmurder in #2787
• [Consensus] EventLoop internal queues update by @durkmurder in #2883
• [Consensus] Update HotStuff Finalization Logic by @jordanschalm with contributions from @AlexHentschel in #2954
• [Consensus] Update disabled integration tests by @durkmurder in #2935
• [Consensus] Modify EpochLookup to handle epoch fallback trigger condition by @jordanschalm in #2927
• [Consensus] Consensus Compliance Engine updated to conform to Component interface by @jordanschalm with contributions from @AlexHentschel in #3121
• [Consensus] New logic for round duration calculation by @durkmurder in #3201
• [Consensus] Collectioncompliance, epochmgr Engines Implement component.Component by @jordanschalm in #3248
• [Consensus] Timeout objects rebroadcast by @durkmurder in #3277
• [Consensus] Update HotStuff Reader CLI Tool by @jordanschalm in #3363
• [Consensus] Front-Load HotStuff Verification Pt. 1: Update Compliance Engines by @jordanschalm with contributions from @AlexHentschel in #3294
• [Consensus] Front-Load HotStuff Verification Pt. 2: Remove MarkValid API by @jordanschalm in #3303
• [Consensus] Handle unsealed invalid service events by @jordanschalm in #2969
• [Consensus] MessageHub implementation by @durkmurder in #3357
• [Consensus] Add ParentView to flow.Header by @durkmurder in #3372
• [Consensus] Timeout rebroadcast de-duplication by @durkmurder in #3430
• [Consensus] Own vote & timeout message routing by @durkmurder and @AlexHentschel in #3429
• [Consensus] Follower engine message queue by @durkmurder in #3479
• [Consensus] Extract interface for compliance engine by @durkmurder in #3512
• [Consensus] Standalone startup of hotstuff participant by @durkmurder in #3513
• [Consensus] Update telemetry for EventHandler by @durkmurder in #3571
• [Consensus] Implementation of missing consensus metrics by @durkmurder in #3623
• [Consensus] TC aggregation multi-message verification fix by @durkmurder in #3698
• [Consensus] Fix bug in cluster block validation. by @jordanschalm in #3707
• [Consensus] Fix name ordering in metrics by @jordanschalm in #3710
• [Consensus] Re-enable AN test, increase test timeout, add logging, document outstanding issues by @jordanschalm in #3631
• [Consensus] Fix sync engine bug, improve observability by @jordanschalm in #3714

Execution node

• code gen parseRestrict wrappers by @pattyshack in #3398
• [Execution] remove duplicate blobservice on execution nodes by @peterargue in #3418
• Handle event encoding error gracefully (Port of 2562) by @m4ksio in #2565
• [Execution] Adding more metrics to the execution node by @ramtinms in #3380
• [Execution] Do not depend on string conversion in register set tracking by @SaveTheRbtz in #3529
• [Execution] Minor ingestion logging cleanup by @SaveTheRbtz in #3522
• [Execution] Randomize collection fetch order by @SaveTheRbtz in #3445
• [Execution] Use generic slice algorithms instead of interfaces by @SaveTheRbtz in #3542
• [Execution] Remove useless condition from builder by @SaveTheRbtz in #3659
• [Execution] Kill pagecache mass eviction code by @SaveTheRbtz in #3662
• [Execution] Break blobservice out to own component in en builder by @peterargue in #3613
• [Ledger] Upgrade WAL to support fdatasync(2) by @SaveTheRbtz in #3693
• [FVM] Log fees error by @janezpodhostnik in #3392
• [FVM][Execution] Adding metrics for computation effort vector by @Tonix517 in #3403
• [FVM] Change payer balance check error to a failure by @janezpodhostnik in #3679

Execution node refactoring - Prerequisite for concurrent transaction execution

• Refactor transaction invoker. by @pattyshack in #3423
• return TotalComputationUsed() as uint64 by @pattyshack in #3394
• Add Pause/Resume functionality to TransactionState by @pattyshack in #3435
• Tighten unexpected nested txn check + delay procedure result population by @pattyshack in #3437
• Switch parseRestricted to use span name instead of hardcoded strings by @pattyshack in #3443
• Clean up fvm mocks by @pattyshack in #3450
• Test ErrorsCollector against SplitError by @pattyshack in #3440
• Disable limits for all of tx seq num checker / tx verifier by @pattyshack in #3474
• Define meter constants explicitly + add missing metering by @pattyshack in #3481
• Prep work for converting transaction invoker into continuation passing style executor by @pattyshack in #3483
• Prevent cadence from directly accessing fvm internal state by @pattyshack in #3462
• Convert ScriptProcedure.Run into executor object. by @pattyshack in #3496
• Switch transaction invoker to directly use cadence txn executor by @pattyshack in #3493
• Flatten transaction processor loop. by @pattyshack in #3505
• Split bootstrap parameters into a separate struct. by @pattyshack in #3532
• Convert bootstrap procedure into executor by @pattyshack in #3550
• Move transaction procedure Run logic into transaction executor. by @pattyshack in #3556
• Clean up environment initialization by @pattyshack in #3600
• dedup executeCollection logic by @pattyshack in #3607
• Only override meter parameters for transaction/script body execution by @pattyshack in #3611
• Simplify adding transaction result to ComputationResult by @pattyshack in #3618
• Stop special casing system collection execution by @pattyshack in #3629
• Refactor computer result collection by @pattyshack in #3637
• Replace Procedure's Run method with NewExecutor by @pattyshack in #3666
• Cleanup ComputationResult by @pattyshack in #3652
• Improve result collector to return error instead of panic by @pattyshack in #3653
• [FVM] Threadsafe event encoder by @SaveTheRbtz in #3531

Execution node performance improvement - caching FVM settings

• Move non address programs tracking to environment. by @pattyshack in #3527
• Refactor generic derived data implementation. by @pattyshack in #3524
• Renamed programs to derived data by @pattyshack in #3516
• Add GetOrCompute to generic derived data implementation. by @pattyshack in #3563
• Enable nested transaction to use a different set of meter parameters by @pattyshack in #3576
• Refactor meter parameter setup logic. by @pattyshack in #3575
• Update derived data tests by @pattyshack in #3582
• Explicitly name the programs derived data table/field by @pattyshack in #3583
• Fix storage interaction limit setup by @pattyshack in #3588
• Meter param overrides cache skeleton. by @pattyshack in #3594
• Rename BlockDerivedData by @pattyshack in #3612
• Only override meter parameters for transaction/script body execution by @pattyshack in #3611
• Misc. no-op refactoring to prep for meter param override invalidation by @pattyshack in #3626
• Renamed fvm/programs to fvm/derived by @pattyshack in #3651
• Replace ledger migration tool's view with fvm/utils' SimpleView by @pattyshack in #3672
• Invalidate derived data when meter param overrides change. by @pattyshack in #3644

Execution node - Performance optimizations

• [FVM] Move inclusion effort to transaction by @janezpodhostnik in #3471
• [FVM] Use empty programs during bootstrapping by @janezpodhostnik in #3472
• [FVM] Adding generic OCC implementation by @Tonix517 in #3484
• [FVM] Making BlockPrograms internals use generic OCC code by @Tonix517 in #3494
• [FVM] Fail fast when payer cannot pay by @janezpodhostnik in #3473
• [FVM] Check account storage before deducting fees by @janezpodhostnik in #3634
• Parallelize/Pipeline spock signature generation by @pattyshack in #3696
• [Resolved Performance Problem] make requester's Force method non-blocking by @zhangchiqing in #3517

Execution node - spork performance optimization & checkpointing improvements

• make the initial copying of the root checkpoint to be concurrent by @zhangchiqing in #3584
• Remove read last trie by @zhangchiqing in #3400
• Only read and write checkpoint in v6 by @zhangchiqing in #3628
• Optimize state migration by @zhangchiqing in #3621
• Adding read-only mode and hash value caching for protocol Merkle trie by @ramtinms in #3411
• [State Extraction] no migration optimization by @zhangchiqing in #3466

Sync engine

• Update depenencies to use onflow/go-bitswap fork by @peterargue in #3402
• Align request handlers between common and collection sync engines by @peterargue in #3407
• Update rate limited blobstore to drop requests by @smnzhu in #3349

Crypto

• [Crypto] Identity BLS public keys by @tarakby in #3421
• Crypto: update low level Relic dependency by @tarakby in #3558

Access and Observer node

• [Access] Add Block Status to BlockResponse by @lolpuddle in #3477
• [Access] Remove unused builder options by @peterargue in #3620
• [Access] Get Execution Data by BlockID by @lolpuddle in #3320
• [Access] Fix LRU creation check by @SaveTheRbtz in #3660
• [Access] Add admin command to get execution data by @peterargue in #3655
• [Access] Add support for blocklist in admin server by @peterargue in #3694
• [Access] Fix public network metrics by @peterargue in #3716

Performance benchmarking

• [Benchmark] Rewrite worker stats by @SaveTheRbtz in #3551
• [Benchmark] Faster account creation by @SaveTheRbtz in #3567
• [Benchmark] Use smaller transfer size for benchmark transactions by @SaveTheRbtz in #3568
• [Benchmark] Notify on slow transactions by @SaveTheRbtz in #3566
• [Benchmark] Remove unused flags from CI by @SaveTheRbtz in #3569
• [Benchmark] Remove unused code by @SaveTheRbtz in #3565
• [Benchmark] Add protobuf definition stubs for macro benchmarks by @SaveTheRbtz in #3570
• [Benchmark] Speedup benchmark convergence by @SaveTheRbtz in #3574
• [Benchmark] Throttle on timeout by @SaveTheRbtz in #3578
• [Benchmark] Remove facades from the benchmark library by @SaveTheRbtz in #3599
• [Benchmark] Change BigQuery schema by @SaveTheRbtz in #3633
• [Profile] Add ability to manually trigger profiler by @SaveTheRbtz in #3520
• [Profiler] Add selected /debug/pprof/ endpoints to admin interface by @SaveTheRbtz in #3508
• [Profiler] Add admin commands to control profiling rates by @SaveTheRbtz in #3561
• [Profiler] Start profiler asynchronously by @peterargue in #3615

Protocol changes required for Cadence features

• Add AccountKeysCount to AccountKeyReader to match changes in Cadence's runtime Interface by @dreamsmasher in #3444
• remove redundant address validity checks in AccountKeyReader by @dreamsmasher in #3463

Byzantine Fault Tolerance [BFT]

• [BFT] add suspicious flag to a few new log messages by @peterargue in #3376
• [BFT] Improve network message validation by @peterargue in #3375
• [BFT] Add bitswap tracer by @smnzhu in #3390
• [BFT] Enables disconnection from disallowed peers by @yhassanzadeh13 in #3459
• [BFT] Tune logging and settings based on canary testing by @peterargue in #3589
• [BFT] Handle unverifiable input error in cluster compliance core by @jordanschalm in #3610
• [BFT] Performance Improvements for Cluster Compliance Engine by @jordanschalm in #3598
• [BFT] Update Epoch Collection Node Join/Leave Integration Test by @jordanschalm in #3625
• [BFT] Add consensus and sync-related message types without nil struct fields by @jordanschalm in #3523
• [BFT] Unskip Test TestCrosstalkPreventionOnNetworkKeyChange by @LLsq2 in #3446
• [BFT] Khalil/1716 unicast rate limiter by @kc1116 in #2840
• [BFT] Khalil/1817 Check node is supposed to receive messages on unicast channel by @kc1116 in #3490
• [BFT] Prevent broadcasting unicast only messages by @kc1116 in #3525
• [BFT edge case] Process service events over ordered seals by @jordanschalm in #3461

Networking

• [Networking] adds protocol id and spork id logging by @yhassanzadeh13 in #3431
• [Networking] Reduce debug-level log spam by @SaveTheRbtz in #3506
• [Networking] Bump libp2p v0.22.0 -> v0.23.3 by @SaveTheRbtz in #3521
• [Networking] Add ability to change libp2p's logging level on the fly by @SaveTheRbtz in #3526
• [Networking] Adds metrics to GossipSub router by @yhassanzadeh13 in #3501
• [Networking] Fixes DNS TTL default value by @yhassanzadeh13 in #3507
• [Networking] Wires in GossipSub observability by @yhassanzadeh13 in #3643
• [Networking][BFT Testing] abstracts GossipSub-related primitives by @yhassanzadeh13 in #3585
• [Networking] Implements metrics reporter for libp2p resource manager by @yhassanzadeh13 in #3658
• [Networking] Do not use resource limits in tests by @SaveTheRbtz in #3669
• [Networking] Implements configurable scaler for libp2p resource manager by @yhassanzadeh13 in #3724
• [Network] Restrict messages that can be sent via unicast by @kc1116 in #3378
• [Network] Stop sending unused message.Message fields by @peterargue in #3480
• [Network] Fix cbor decoder validation by @peterargue in #3452

BFT Testing

• [BFT Testing] Corruptible LibP2P Node by @kc1116 in #3276
• [BFT Testing][CI][Testing] create insecure module for forked libp2p-pubsub, CI and flaky test updates by @gomisha in #3420
• [BFT Testing] fixes ingress related issues by @yhassanzadeh13 in #3591
• [BFT Testing] Topic validator tests by @kc1116 in #3580
• [Networking][BFT Testing] abstracts GossipSub-related primitives by @yhassanzadeh13 in #3585

Testing

• [Testing] quarantine flaky test TestCrosstalkPreventionOnNetworkKeyChange in network/p2p/test/sporking_test.go by @gomisha in #3465
• [BFT Testing][CI][Testing] create insecure module for forked libp2p-pubsub, CI and flaky test updates by @gomisha in #3420
• [BN2][Testing] Dynamic generation of Helm values.yml by @gomisha in #3488
• [BN2][Testing] - Validate automation steps for Benchnet2 by @gomisha in #3587
• [Testing] Use localhost in unit tests to reduce firewall alerts by @peterargue in #3680
• [Testing] Add verbose envvar and set in CI by @peterargue in #3671
• [Testing] Fix races in network unittests by @peterargue in #3684
• [BN2][Testing] Run latest version of Benchnet2 automation code by @gomisha in #3705
• [BN2][Testing] Generate public-private key pair for Benchnet2 by @gomisha in #3718
• Shorten bench and make it optional by @janezpodhostnik in #3627
• [FVM] Bench test fixes by @janezpodhostnik in #3416
• [FVM] Fix bench test by @janezpodhostnik in #3704

Profiler/Telemetry changes

• Add vars for TPS values by @AndrewM-SDET in #3385

Util changes

• [Node Operations] add a new utility cmd for execution state stats by @ramtinms in #3491
• [Node Operations] Update error handling in Admin Server by @jordanschalm in #3391
• [Node Operations] Admin command for Network Blocklist by @jordanschalm in #3560
• [Node Operations] Admin command for dynamic config updates by @jordanschalm in #3369

Checkpointing

Rolling upgrade support

🛠 General Node Improvements

• [Common Provider Engine] Adds inbound queue by @yhassanzadeh13 in #3404
• [Bootstrap] Improve error handling in node scaffold by @peterargue in #3619
• [Engine] Require explicit capacity for Fifo queue by @peterargue in #363
• [Logging] Add log throttle for debug/trace level events by @peterargue in #3608
• [Metrics] Report finalized height and executed height on startup by @zhangchiqing in #3596
• [Model] Fix race condition in DeterministicSample by @SaveTheRbtz in #3559
• [Provider Engine] log actual entities in the response by @zhangchiqing in #3692
• [Provider Engine] Change info level log for provider engine by @zhangchiqing in #3688
• [Node-internal Events] Emit Protocol Events after database transaction by @jordanschalm with contributions from @AlexHentschel in #2994
• [Bugfix] fix heap draining bug in synchronization engine by @AlexHentschel in #3478
• [Code-complexity reduction] Use generics for flow.GetIDs, clean up mempool package by @jordanschalm in #3606
• [Maintenance and Tech Debt] Small improvements to error types and messages in block extension logic by @jordanschalm in #3577
• Update to Cadence v0.30.0 by @turbolent in #3711
• Update core contract version by @janezpodhostnik in #3713

Infra and operations

• Reconcile ChainID changes for Sandboxnet by @sjonpaulbrown in #3433
• Open metrics port on 8080 by @sjonpaulbrown in #3663
• [Streamline logging] log entity ids in debug level log by @zhangchiqing in #3572

New Contributors

• @dreamsmasher made their first Cadence contribution in #3444