celestiaorg/celestia-node v0.20.2

on GitHub

NOTE: v0.18 and v0.19 are skipped for mainnet

Hey everyone, this is quite a mammoth of a release, so please hang tight. We have two major changes packaged in this release:

• Shwap (protocol-breaking)
• Celestia v3 compatibility (API-breaking)

The migration period for this release is quite tight (a bit longer than 2 weeks), so we urge all node operators to upgrade to this version and re-sync against the network by December 2nd.

As this release is API-breaking, we also urge all integrations to migrate to this new version by December 2nd. (The list of API breaks can be found below).

Shwap

This release features Shwap(CIP-19), a protocol-breaking feature that brings massive optimizations to disk usage(16x) and data access latencies. The new protocol is not backwards compatible with the currently deployed protocol and software.

Both protocols will run side-by-side for about a two-week transition period, from November 20 to December 2. During this period, node operators should migrate to the new version. At the end of the period, posted historical and recent data will no longer be available on the old protocol.

Migration Notes

Migration requires resyncing the node from scratch, as it now contains a new storage format.

Migration Steps over the old node:

• Remove old data:
  • In home directory(default .celestia-<type>) => rm -rf inverted_index index data transients blocks
  • Keep keys directory to preserve peer and chain identity keys in place
• Config updates
  • If defaults were in use, remove the old config and re-init in the next step
  • If you want to preserve the old config - move BlockstoreCacheSize entry from [Share.EDSStoreParams] section to just [Share]and remove GCInterval, RoutingTableRefreshPeriod
• Re-init node for the Mocha p2p network
  • celestia init
• Start the node and monitor syncing
  • celestia start

If the old node has to be kept running - initiate and start the new version in a different directory:

• celestia init --node.store <new_path>
• celestia start --node.store <new_path>

Celestia V3 compatibility

This release contains an upgrade to the celestia-app version (to v3). v3 will only be live on the mainnet in mid-December, but this release contains the requirements for compatibility with the upgrade.

Notable API breaks include:

• simplification of the blob share and state modules via the deduplication of types (via [#3782](#3782))
• share module now takes height instead of the full ExtendedHeader (via [#3870](#3870))

Comprehensive list of API breaks:

• GetSharesByNamespace renamed to GetNamespaceData
• the share module returns shwap.NamespaceData for the GetNamespaceData method
• the share module now takes height (uint64) instead of the full ExtendedHeader for all methods
• both the share and blob modules now use the go-square v2 libshare.Namespace (from https://github.com/celestiaorg/go-square/tree/v2.0.0) instead of the old share.Namespace for all methods
• the state module now uses the go-square v2 libshare.Blob type (from https://github.com/celestiaorg/go-square/tree/v2.0.0) instead of the old one from the state module
• the blob package now embeds the new go-square v2 libshare.Blob type mentioned above in the blob.Blob struct instead of the old v1 type.

Miscellaneous

What to look out for in the following releases:

• the -experimental-pruning flag will soon be deprecated in favour of pruning becoming the default mode for all node types, requiring users to pass -archival to retain block history beyond the sampling window
• the API gateway will be deprecated

What's Changed

Features

• feat(shwap): Add eds store by @walldiss in #3545
• feat(shwap): Add namespace data shwap id by @walldiss in #3587
• feat(shwap): Add axis roots to Accessor by @walldiss in #3586
• feat(shwap): Add bufferisation for writing ods/Q1Q4 files by @walldiss in #3592
• feat(shwap/p2p): Bitswap Getter by @Wondertan in #3584
• feat(shwap): Shwap add readFrom Writeto methods to shwap ids by @walldiss in #3605
• feat(shwap): Integrate shrex into shwap by @walldiss in #3554
• feat(shwap): Shwap core integration by @walldiss in #3598
• feat(shwap): Shwap cascade and mock getters by @walldiss in #3599
• feat(shwap): Update availability to work with shwap by @walldiss in #3600
• feat(shwap): add shwap to blob module by @walldiss in #3613
• feat(shwap): update daser for shwap by @walldiss in #3614
• feat(shwap): add shwap store getter by @walldiss in #3617
• feat(shwap): Add caching to blockstore by @walldiss in #3615
• feat(shwap/bitswap): Shwap optimized Bitswap constructors by @Wondertan in #3536
• feat(shwap): Improve eds store cache refcounter forgot Close() protection by @walldiss in #3638
• feat(shwap): shwap nodebuilder and cleanup by @walldiss in #3637
• feat(shwap/ods_file): tail padding trim by @Wondertan in #3620
• feat(shwap): Add discovery version by @walldiss in #3616
• feat(cmd/cel-shed): shwap by @Wondertan in #3720
• feat(shwap/store): Add q4 trimming support in store by @walldiss in #3680
• feat(nodebuilder): Blockstore cache params by @walldiss in #3739
• feat(share/availability): Dont store q4 on archival by @walldiss in #3751
• feat(store/cache): add Has methods by @Wondertan in #3757
• feat(cel-shed/p2p): bootstrapper probe tool by @smuu in #3780
• feat!: Live, laugh, Shwap by @Wondertan in #3675
• feat(cmd): adding timeout flag by @sebasti810 in #3580
• feat(nodebuilder/da): implement go-da v0.6.1 changes by @tzdybal in #3750
• feat(modp2p): close peer connections after successful blocking by @notlelouch in #3838
• feat(shwap/bitswap): use new option for optimized Has check by @Wondertan in #3813
• feat(store): Test all put funcs by @walldiss in #3847
• feat(store): use use relative path for empty blocks symlinks by @walldiss in #3861
• feat(store): correct corrupted files on write by @walldiss in #3859
• feat(share/availability)!: Store success results for light availability by @walldiss in #3887
• feat(pruner/light): add light pruning for shwap by @walldiss in #3896

Fixes

• fix(shwap/bitswap): close acccessor in Blockstore by @Wondertan in #3535
• fix(shwap/store): ensure EDS is accessible during writes by @Wondertan in #3596
• fix(shwap): don't cache empty eds and close recent accessor by @walldiss in #3608
• fix(store/files): ODSQ4 Closing audit by @Wondertan in #3691
• fix(shwap/cahce): use local copy of axis in proofs cache by @walldiss in #3657
• fix(shwap): io.ReadFull for read helpers by @Wondertan in #3696
• fix(shwap): io.ReadFull for NamespaceDataID by @walldiss in #3697
• fix(shwap/bitswap): correct protocol id by @Wondertan in #3722
• fix(store/odsq4): fix order of opened flag stored by @walldiss in #3735
• fix(store/ods): fix full ods caching by @walldiss in #3736
• fix(core): fix flaky generateNonEmptyBlocks by @walldiss in #3740
• fix(blob): fix blobsub subscription cancellation test by @walldiss in #3746
• fix(store): use symlinks for empty blocks heighs by @walldiss in #3747
• fix(header): soft failure is only for not enough voting power by @cristaloleg in #3769
• fix(core/listener): wait listener to shutdown before exit by @walldiss in #3775
• fix(docker): set addr flags to 0.0.0.0 in entrypoint file by @MSevey in #3808
• fix(modshare): ensure Bitswap Getter is started by @Wondertan in #3789
• fix(shwap/store): avoid double cache check by @Wondertan in #3788
• fix(nodebuidlder/p2p) Use semver for discovery prefix by @walldiss in #3768
• fix(modshare): discovery leftover from #3777 by @Wondertan in #3781
• fix(blob): add fixes along with the fuzzers that discovered them by @odeke-em in #3732
• fix: remove godebug asynctimerchan=1 by @cristaloleg in #3790
• Fix start with chain id mismatch by @jpserrat in #3664
• fix(nodebuilder): flag parsing for --node.type needs lowercase string, list network aliases for --p2p.network by @renaynay in #3827
• fix(share/p2p/bitswap): lower bitswap log severity by @walldiss in #3831
• fix(store): check if link exists on write by @walldiss in #3844
• fix(eds/cache): Fix prealloc for proof nodes by @walldiss in #3860
• fix(store/cache): unstick cache metrics names by @walldiss in #3857
• fix(docs) update txResponse in docs by @walldiss in #3848
• fix(store/file): wait for q4 to be written before returning by @walldiss in #3865
• fix(ci,release): run on any new tag or trigger manually by @cristaloleg in #3855
• fix(nodebuilder/p2p): lint by @cristaloleg in #3875
• fix(share/ipld): use bits.Len64 instead of math.Log2 in newProofCollector by @notlelouch in #3867
• fix(ci): GH_TOKEN for upload-docs by @cristaloleg in #3877
• fix: wrong network error by @jcstein in #3885
• fix(pruner): fix find algo so that it does not pass a malformed range into GetRangeByHeight call by @renaynay in #3828
• fix(share/availability): propagate errors from light availability properly by @walldiss in #3888
• fix(ci): do not version_bump by @cristaloleg in #3882
• fix(shwap/shrex): enable recovery middleware in shrex server by @walldiss in #3897
• fix(DASer): Log sampling errors in DASer by @walldiss in #3889
• fix(availability): prevent parallel availability calls by @walldiss in #3883
• fix(core): change fill block mode by @cristaloleg in #3936
• hotfix(core): don't hold on Data by @Wondertan in #3926
• fix(p2p): disable quic by @walldiss in #3937
• fix(shwap/bitswap): tune bitswap params by @Wondertan in #3940
• fix(shwap/bitswap): Blockstore.GetSize: getting size with no compute by @Wondertan in #3894
• fix(shwap/bitswap): update boxo and config by @Wondertan in #3946
• fix(share/availability/light): Allow retries on ODS size 1 by @renaynay in #3944

Refactors

• refactor(shwap): rename share.root to axisRoots by @walldiss in #3583
• refactor(shwap): revert DataRoot -> DataHash by @Wondertan in #3585
• refactor(shwap): Rename remaining dataroot references by @walldiss in #3607
• refactor!(shwap/store): decouple Q1Q4 and ODS files by @Wondertan in #3643
• refactor(shwap): Remove new_eds folder by @walldiss in #3738
• refactor(share/eds): various accessor testing suite improvements by @walldiss in #3737
• refactor(store): write actual share size to ods by @walldiss in #3863
• refactoring!(share): migrating to go-square v2 types by @vgonkivs in #3782
• refactor(nodebuilder): add archival flag by @walldiss in #3868
• refactor(api): add gateway deprecation notice by @walldiss in #3869
• !refactor(api/share): use height in share module by @walldiss in #3870
• refactor(share/availability): simplify light availability by @walldiss in #3895
• !refactor: rename GetSharesByNamespace to GetNamespaceData by @cristaloleg in #3902
• refactor: remove blob size check by @rootulp in #3934
• refactor(share/availability): Move window and constants to share/availability pkg by @renaynay in #3906

Miscellaneous

• chore(shwap): various improvements by @Wondertan in #3609
• chore(shwap)!: Merge main into shwap by @walldiss in #3662
• perf(store/sync): avoid syncing files by @Wondertan in #3693
• chore(shwap/bitswap): prevent not found error logging by @Wondertan in #3754
• chore(shwap/bitswap): add getter tracing by @Wondertan in #3755
• chore(mocks): update generated mocks by @walldiss in #3767
• chore(deps): upgrade to app v2.2.0-arabica by @rootulp in #3772
• chore(shwap/discovery): prefix -> suffix by @Wondertan in #3778
• .github/workflows: docker build publish on anything with a version v by @renaynay in #3787
• test(nodebuilder): disable TestArchivalBlobSync by @cristaloleg in #3635
• chore(deps): Bump celestiaorg/.github from 0.4.3 to 0.4.5 by @dependabot in #3785
• ci: fix linter by @cristaloleg in #3797
• chore(ci): group updates by @cristaloleg in #3748
• chore(.github): Remove ryan from CODEOWNERS by @renaynay in #3794
• chore: bump p2p deps by @Wondertan in #3811
• chore(logs): increase verbosity for bitswap log by @Wondertan in #3818
• chore(go.mod): Bump celestia-app to v2.2.0 release in main by @renaynay in #3826
• chore: remove some TODOs by @cristaloleg in #3810
• chore: bump to v2.3.0 by @evan-forbes in #3854
• chore(ci): remove release tracking epic ci by @MSevey in #3866
• chore(go.mod): bump header by @renaynay in #3872
• chore(share): remove unused ShareWithProof by @cristaloleg in #3893
• chore: remove unused code by @walldiss in #3898
• test(swamp): cover v1 blobs in swamp tests by @vgonkivs in #3908
• ci(docker): bump version by @tty47 in #3914
• deps(go.mod): bump app to arabica tag by @renaynay in #3910
• chore: bump to celestia-app v3.0.0-mocha by @evan-forbes in #3920
• chore(deps): temporary fork boxo by @Wondertan in #3939
• chore(logs): cleanup logging by @Wondertan in #3945
• chore(go.mod): bump boxo fork by @Wondertan in #3948
• perf(shwap/bitswap): add session pools inside Getter by @Wondertan in #3947

New Contributors

• @odeke-em made their first contribution in #3732
• @sebasti810 made their first contribution in #3580
• @notlelouch made their first contribution in #3838

Full Changelog: v0.17.2...v0.20.2