onflow/flow-go temp_mainnet_23

Mainnet 23 [WIP]

on GitHub

This release includes the following major changes,

💥 Breaking Changes

BFT

• Pending blocks cache for follower engine by @durkmurder in #3965

BFT Testing

• Fix CI build for #3585 (abstracts GossipSub-related primitives) by @gomisha in #3592
• Fixes nil issue with router during tests by @yhassanzadeh13 in #3665
• Khalil/3590 libp2psig verification test by @kc1116 in #3689
• fixes CI failure on PR 3564 by @yhassanzadeh13 in #3794
• Gossipsub spam test Framework - IHAVE by @gomisha in #3564

Access

• Do not copy protobuf messages by @SaveTheRbtz in #3790
• Add BlockStatus to GetBlock REST API by @lolpuddle in #3805
• Fix panic converting execution data to protobuf messages by @peterargue in #3887
• Improve Access API error reporting for results from ENs by @peterargue in #3915
• Add REST API metrics middleware by @diaswrd in #3691
• Export MessageToTrieUpdate converter method by @peterargue in #3897
• Improve api error handling by @peterargue in #3988
• Set message received limits by @peterargue in #3857

Networking

• Revamps connection manager by @yhassanzadeh13 in #3852
• Refactors test scenarios to comply with libp2p upgrade by @yhassanzadeh13 in #3875
• Enables logging of peer ids together with node id upon connection establishment by @yhassanzadeh13 in #3898
• Bump libp2p v0.23.3 -> v0.24.2 by @SaveTheRbtz in #3701
• Logging of libp2p Resource Limits at Node Startup by @yhassanzadeh13 in #3904
• Topology Tracing and Monitoring for GossipSub by @yhassanzadeh13 in #3970
• Thread-safe and non-blocking notification distributor by @yhassanzadeh13 in #3911
• Introducing Visibility to the GossipSub Scoring Mechanism by @yhassanzadeh13 in #3986
• Refactor GossipSub topology logger with zeroLog dictionary feature by @yhassanzadeh13 in #4035
• Add protocol metrics label by @peterargue in #3761
• Fix inbound message metrics label by @peterargue in #3767
• Fixes race conditions by @yhassanzadeh13 in #3687

Benchmarking

• Prevent PID windup at start by @SaveTheRbtz in #3759
• Support multiple keys by @SaveTheRbtz in #3641
• Match log-level w/ production by @SaveTheRbtz in #3780

CI/CD

• Add github action to build boot-tools on demand by @peterargue in #3787
• Remove --ssh flag from docker build for util by @peterargue in #3798
• Update ci.yml retry reference nick-invision => nick-fields by @gomisha in #3919
• Add workflow to build individual docker images by @peterargue in #4065
• Only run jobs for selected roles by @peterargue in #4075
• Add retry mechanism to integration tests by @gomisha in #3989

Consensus

• Update README.md for consensus by @durkmurder in #3863
• Snapshot uses QC storage by @durkmurder in #3947
• Replacing MutableState with FollowerState and ParticipantState by @durkmurder in #3949
• Updated main consensus compliance engine to support batched API by @durkmurder in #4024
• Implementation of PendingTree by @durkmurder in #4003

Crypto

• KeyGen improvement by @tarakby in #3788
• Pairing edge case by @tarakby in #3977
• New utils/rand package for non-deterministic true randomness by @tarakby in #4062
• improve usafeRandom and update math/rand in FVM by @tarakby in #4067

Execution

• Refactor uploader into separate module by @peterargue in #3676
• clean up - remove unused code by @ramtinms in #3980
• removing unused codes part 2 by @ramtinms in #3984
• remove dead code (delta sync) by @ramtinms in #4036
• move script execution logic to query package by @ramtinms in #4034
• Adding collection result consumer type by @ramtinms in #4049

FVM

• Log execution intensities even if the transaction fails by @janezpodhostnik in #3755
• Smart/partial programs cache invalidation by @janezpodhostnik in #3706
• Refactor accounts set storage used by @janezpodhostnik in #3865
• Add restriction that program loading cannot write to registers by @janezpodhostnik in #3871
• Split meter into components by @janezpodhostnik in #3870
• Debug log cadence logs by @janezpodhostnik in #3899
• Change interaction metering by @janezpodhostnik in #3874

Storage

• Storage for Quorum Certificates by @durkmurder in #3928

Testing

• Parameterize docker registry url by @gomisha in #3998
• Update benchnet2 automation by @haroldsphinx in #4057
• Move admin tests to integration by @peterargue in #3917
• Flaky Test Monitor - run regular tests more frequently by @gomisha in #3741

Tools

• Add bootstrap command to generate grpc TLS keys by @peterargue in #3789

Misc

• Move weight check out of common provider engine by @peterargue in #3923
• Add cli flag for grpc max message size by @peterargue in #3855
• Reporting node info metrics by @durkmurder in #3800
• Herocache reports ejected entites by @durkmurder in #3961
• Pin benchstat version by @SaveTheRbtz in #3833

New Contributors

• @diaswrd made their first contribution in #3691
• @haroldsphinx made their first contribution in #4057

Unclassified (please classify these to one of the headings above ☝️ )

• Fix/simplify value store span attribute setup by @pattyshack in #3738
• Simplify setting transaction procedure result values by @pattyshack in #3740
• Improve verification test's error message by @pattyshack in #3752
• Simplify chunk verifier api by @pattyshack in #3753
• Refactor environment.Tracer into a separate module. by @pattyshack in #3749
• Upgrade core contracts by @janezpodhostnik in #3733
• Ensure the same runtime config is used by @turbolent in #3768
• Crypto: code improvement by @tarakby in #3760
• Update to cadence v0.31.1 by @SupunS in #3774
• Improve transaction verifier clarity by @pattyshack in #3771
• Set transaction procedure's trace span via context by @pattyshack in #3766
• Update to Cadence v0.31.2 by @turbolent in #3777
• Improve error code search. by @pattyshack in #3776
• handle limited size of queue for holding incorporated block in sealing logic by @AlexHentschel in #3734
• Simplify Start Entity Span. by @pattyshack in #3786
• AddObserver adds a node to a suite by @m4ksio in #3746
• Batch removal by @m4ksio in #2535
• Remove RecordSpanFromParent from general tracing api by @pattyshack in #3791
• Use consistent default weight by @jordanschalm in #3783
• Fix data race in DKG integration test by @jordanschalm in #3792
• fix storage by @smnzhu in #3064
• Define vm interface in fvm instead of computer by @pattyshack in #3796
• Pre-populate []transactions into collectionItem by @pattyshack in #3804
• suggested revisions for PR #3762 by @AlexHentschel in #3814
• refactor chunk object initialization into flow model by @pattyshack in #3811
• Change RegisterUpdates's return type by @pattyshack in #3797
• remove RegisterIDToKey & RegisterValuesToValues from execution state by @pattyshack in #3821
• Don't recompute txn id inside fvm by @pattyshack in #3819
• Simplify block computer tracing by @pattyshack in #3793
• Khalil/1855 Remove block listed peers by @kc1116 in #3762
• Remove dead code by @pattyshack in #3827
• Update view's register ids apis. by @pattyshack in #3820
• Convert service event as part of fvm execution by @pattyshack in #3829
• Auto update to onflow/cadence v0.31.3 by @SupunS in #3836
• Use consistent method for creating temp dirs by @jordanschalm in #3846
• remove MarshalJSON/UnmarshalJSON from delta by @pattyshack in #3847
• Remove Touch/Delete from view interface. by @pattyshack in #3842
• Refactor meter/interaction limits control by @pattyshack in #3845
• Disable limiting storage/computation during bootstrapping by @janezpodhostnik in #3848
• increase GRPC limit to 20MB by @koko1123 in #3635
• Improve computation result pipelining. by @pattyshack in #3840
• Avoid oppressive terminology by @turbolent in #3370
• Remove UpdatedAddresses from fvm State by @pattyshack in #3853
• Replace meter.StorageInteractionKey with flow.RegisterID by @pattyshack in #3866
• Khalil/1846 close conn rate limited peer by @kc1116 in #3799
• Use RegisterId instead of (owner, key)-tuple as key for accessing register values by @pattyshack in #3867
• Remove dead emulator code by @pattyshack in #3876
• Use RegisterID instead of (owner, key)-tuple for accessing accounts by @pattyshack in #3882
• s/runtime.Address/common.Address by @pattyshack in #3885
• Documentation on Sealing Segment (and some minor bug fixes) by @AlexHentschel in #3824
• Use RegisterID instead of (owner, key)-tuple for views by @pattyshack in #3883
• Initialize register ids with flow.NewRegsiterId instead of flow.RegisterID{...} by @pattyshack in #3890
• Add cached programs metrics by @janezpodhostnik in #3888
• Add programs chache hit/miss metrics by @janezpodhostnik in #3896
• Suggested revisions for PR 3863 by @AlexHentschel in #3905
• Ensure NewRegisterID initialize 8 byte owners by @pattyshack in #3900
• Merge feature/active-pacemaker to master by @jordanschalm in #3895
• Khalil/1844 topic validator skip decoding by @kc1116 in #3831
• Update core-contracts dependency (slot selection) by @jordanschalm in #3837
• Fix OBO and inconsistency in signer indices decoding used in Access API by @jordanschalm in #3906
• Simplify SaveExecutionResults arguments by @pattyshack in #3913
• Parallelize saving and uploading computation result. by @pattyshack in #3916
• Clean up access request's address usage by @pattyshack in #3907
• Fix printing trie hash by @zhangchiqing in #3926
• Rename variables and update comments for trie update function by @zhangchiqing in #3922
• Log invalid vote in bootstrap finalize by @zhangchiqing in #3832
• Add LicensingApproach.md by @Kay-Zee in #3908
• Change delta view to read from StorageSnapshot interface instead of readFunc by @pattyshack in #3933
• Clean up address usage (part 1 of many) by @pattyshack in #3912
• Ignore Inclusion field when verifying trie proof by @zhangchiqing in #3940
• Generate block execution data / id as part of result collection by @pattyshack in #3927
• Remove requirement that weight > 0 in EpochSetup validation by @jordanschalm in #3934
• Generate execution result as part of block computation by @pattyshack in #3930
• Remove dead code by @pattyshack in #3951
• Khalil/6441 Blocklist admin command functional test by @kc1116 in #3877
• Amortize derived data validation cost by @pattyshack in #3945
• Make ComputeBlock idempotent by @pattyshack in #3944
• Update to Cadence v0.31.5 by @turbolent in #3959
• Upgrade Cadence to latest version (v0.35.0) by @janezpodhostnik in #3884
• Move StorageSnapshot from execution/state/delta to fvm/state by @pattyshack in #3963
• Fix derived transaction to support out-of-ordered validation by @pattyshack in #3966
• change simple view to use delta view internally by @pattyshack in #3964
• Disable account freezing by @janezpodhostnik in #3982
• Cleanup account freezer address usage by @pattyshack in #3973
• Combine TransactionState & DerivedTransactionData into a single interface by @pattyshack in #3962
• Generate execution receipt as part of block computation by @pattyshack in #3956
• Clean system contracts address usage by @pattyshack in #3974
• Change View merging to operate on execution snapshot by @pattyshack in #3975
• Refactor get or load program by @janezpodhostnik in #3976
• Commit nested transaction on derived value compute error by @janezpodhostnik in #3981
• Clean up verification fixture's computation result usage by @pattyshack in #4000
• Khalil/6477 single pairwise connection by @kc1116 in #3902
• Epoch Height Boundary Index by @jordanschalm in #3903
• Start previous epoch cluster consensus components after epoch boundary by @jordanschalm in #3921
• Remove unused types from entity.go by @jordanschalm in #4009
• Remove Proofs/ConvertedServiceEvents from computation result by @pattyshack in #4001
• Replace RemoteView with delta.View by @pattyshack in #4008
• Refactor fvm/utils/cadenceValues.go by @pattyshack in #3992
• Refactor fvm procedure output into a struct by @pattyshack in #4012
• Beacon Key Storage Error Docs / Handling Improvements by @jordanschalm in #3950
• Update mockery and mockgen, regenerate mocks by @turbolent in #4025
• Clean up more address usages by @pattyshack in #3991
• Update to Cadence v0.36.0 by @dsainati1 in #4017
• Increase timeout for committee test by @jordanschalm in #4023
• Add explicit exception type by @jordanschalm in #3999
• Decouple transaction execution from result collection by @pattyshack in #4018
• suggested revisions for follower cache by @AlexHentschel in #4043
• Remove account freezing by @janezpodhostnik in #4010
• Change computation result to store ExecutionSnapshot instead of delta Interactions by @pattyshack in #4027
• Use GetOrCompute for programs by @janezpodhostnik in #4002
• Add RunV2 to fvm VirtualMachine by @pattyshack in #4026
• Remove SimpleView + fix account reporter data race by @pattyshack in #4033
• Remove StateCommitments from ComputationResult by @pattyshack in #4046
• Compute/report execution stats inside result collector by @pattyshack in #4041
• Rename some address variables by @pattyshack in #4045
• Replace ExecutionSnapshot interface with concrete implementation by @pattyshack in #4007
• Update flow go crypto version by @tarakby in #3441
• Remove unused/usafe Get/Set on Derived data by @janezpodhostnik in #4011
• Suggestions for PR 3965 (part 2) by @AlexHentschel in #4051
• make LedgerStorageSnapshot's readCache thread safe by @pattyshack in #4047
• Remove nil location check in getOrLoad program by @janezpodhostnik in #4031
• update vm.Run calls to vm.RunV2 (part 1 of many) by @pattyshack in #4048
• optimize CI tests by @tarakby in #4055
• Upgrade Cadence by @janezpodhostnik in #4069
• Cleanup codeowners by @peterargue in #4076
• Update fvm GetAccount to use StorageSnapshot as input by @pattyshack in #4064
• Update fvm signature test to use vm.RunV2 by @pattyshack in #4073
• Replace View with SnapshotTree as storage representation by @pattyshack in #4054
• Change program dependencies to cadence location by @janezpodhostnik in #4056
• Add a simplified script environment constructor for emulator by @pattyshack in #4083
• Update automation to fully handle prefixing by @sjonpaulbrown in #4088
• Track script/transaction procedure program depedencies by @janezpodhostnik in #4059
• Remove derived block/txn data from (most) tests by @pattyshack in #4066
• Remove double counting of imports by @janezpodhostnik in #4060
• Port #4063 to master by @jordanschalm in #4086
• Enable attachments on all networks but Mainnet by @turbolent in #4020
• Add metric for epoch transition height by @jordanschalm in #4079
• Update to Cadence v0.37.0 by @turbolent in #4087
• Khalil/1883-inbound single pairwise connection by @kc1116 in #4022
• Port account linking test PRs from v0.29 to master by @turbolent in #4097
• Replace contract update key by @janezpodhostnik in #4091
• Change panic to failure in programs by @janezpodhostnik in #4092
• Update accounts test to use vm.RunV2 by @pattyshack in #4071
• Update fvm blockcontext test to use vm.RunV2 by @pattyshack in #4080
• Switch non-fvm use cases to NewScriptEnvironmentFromStorageSnapshot by @pattyshack in #4090
• Update fvm fuzz test to use vm.RunV2 by @pattyshack in #4081
• Update fvm test to use vm.RunV2 by @pattyshack in #4082
• Move logical time into a separate package by @pattyshack in #4089
• Fix error predicate typo by @pattyshack in #4096
• Khalil/6474 Gossipsub RPC control message Spam protection: GRAFT & PRUNE by @gomisha in #4103
• Add step to print results of test monitor by @jordanschalm in #4095
• Rename State to ExecutionState by @pattyshack in #4110
• extend documentation of pending tree by @AlexHentschel in #4114
• rm ExecutionStateSyncRequest/ExecutionStateDelta by @pattyshack in #4120
• Add read leaf nodes from checkpoint file for Archive node use by @koko1123 in https://github.com/onflow/flow-go/pull/4040classify into one of the above headers)

Full Changelog: v0.29.17...temp_mainnet_23