v0.11.0

This release contains 4 months of work moving towards a production-ready implementation of Celestia's data availability network.

Notable changes

Most notable changes (outside of general fixes) were focused around storage performance and API/RPC improvements and stability.

More granular changes can be found in the notes of all release candidates leading up to this release.

Some notable changes can be found below:

API + RPC changes and improvements

introduction of the BlobModule
refactor of the rpc-cli
changes to the Getter interface
deprecation of write-level endpoints on the Gateway
elevation of permissions to minimum read-level for all endpoints on the JSON-RPC client
new Namespace type

Storage performance optimizations and changes

This release includes several fixes, optimizations and breaks to the EDSStore (and the DAGStore) so it is incompatible with v0.10.4.

Network-level changes

This release contains a protocol-breaking change: a breaking fix to shrex-nd to allow blobs > 1MB in size.

Default network will now be `celestia` mainnet

The node will now point to the celestia main network by default. To connect to mocha-4 or arabica-10, use the --p2p.network flag.

What's Changed

feat!: bump app v1.0.0-rc0 by @evan-forbes in #2253
refactor(misc): pretty print namespace.iD and peer.ID by @walldiss in #2255
test: delete Test_dataFromSharesBSR by @rootulp in #2271
fix(share/discovery): timeout FindPeers operation by @Wondertan in #2263
feat!(blob): blob module by @vgonkivs in #2273
chore(deps): bump go.uber.org/fx from 1.19.2 to 1.19.3 by @dependabot in #2199
chore(deps): bump golang.org/x/crypto from 0.7.0 to 0.9.0 by @dependabot in #2276
chore(deps): bump codecov/codecov-action from 3.1.3 to 3.1.4 by @dependabot in #2235
chore(deps): bump alpine from 3.17.3 to 3.18.0 by @dependabot in #2198
chore!: bump arabica testnet params by @Bidon15 in #2281
feat(nodebuilder): add ip resolving capabilities to core.ip by @sysrex in #2277
chore(share/ipld): pretify error by @Wondertan in #2284
feat(blob): add marshaling for blob and proof by @vgonkivs in #2286
feat(nodebuilder): Add jwt.Signer to Node struct for auth token access by @renaynay in #2295
chore(nodebuilder): decrease lifecycle timeout for LN by @Wondertan in #2290
fix!(blob/module): fix api by changing variadic params to a slice by @vgonkivs in #2298
fix(gateway): remove the GetByHeight hussle and rely on Module's logic by @Wondertan in #2302
fix(blob): use default min gas price from core by @distractedm1nd in #2306
feat(cmd/rpc): adding blob module to rpc cli and fixing namespace parsing by @distractedm1nd in #2308
fix(getter/cascade): cascade getter should return fast on context cancel by @walldiss in #2310
fix(nodebuilder): fix parsing url for core.ip by @walldiss in #2313
chore(deps): bump github.com/filecoin-project/go-jsonrpc from 0.1.9 to 0.3.1 by @dependabot in #2318
fix(das): limit amount of recent jobs by @walldiss in #2314
chore(state): extend error handling in core accessor by @vgonkivs in #2307
chore: bump celestia-app to v1.0.0-rc2 by @evan-forbes in #2305
chore: bump common docker ci version pipeline by @Bidon15 in #2330
!feat(api/gateway): return txdata from submitPFB if it is present by @walldiss in #2329
fix(cmd/rpc): plaintext parsing of data for blob.Submit and state.SubmitPayForBlob by @distractedm1nd in #2340
fix(dagstore): changing log level to WARN to shorten startup by @distractedm1nd in #2336
fix(share): left pad nid in NewNamespaceV0 instead of right padding by @distractedm1nd in #2341
fix(daser) don't count recent jobs in total sampled stats by @walldiss in #2342
fix(state): fixing panic when err != nil but response is nil by @distractedm1nd in #2339
fix!(state): use node blob type instead of app type to fix inconsistent unmarshalling by @distractedm1nd in #2338
chore: mocha-2 support by @evan-forbes in #2326
fix(nodebuilder): Fix dns lookup by @walldiss in #2332
fix(rpc/client): stop overwriting existing clients by @distractedm1nd in #2361
fix(state): JSON marshaling for sdk Address wrapper by @distractedm1nd in #2348
chore!(gateway): add deprecation notices by @distractedm1nd in #2360
chore(deps): bump github.com/pyroscope-io/client from 0.7.0 to 0.7.1 by @dependabot in #2319
fix(header): GetByHeight requests for syncer head +1 shouldnt error by @distractedm1nd in #2362
fix(share/eds): fix test flake by @Wondertan in #2364
chore(deps): bump mheap/github-action-required-labels from 4 to 5 by @dependabot in #2354
feat!(nodbuilder): making lifecycle timeouts configurable by @distractedm1nd in #2347
chore(deps): bump golangci/golangci-lint-action from 3.4.0 to 3.6.0 by @dependabot in #2353
fix(swamp): use synchronized map as container for nodes by @Wondertan in #2358
chore(deps): bump github.com/imdario/mergo from 0.3.15 to 0.3.16 by @dependabot in #2355
chore(deps): bump celestiaorg/.github from 0.1.1 to 0.2.0 by @dependabot in #2317
chore(nodebuilder/tests): Clean up sync_test tests by @renaynay in #2162
feat(share): collect absence proof if namespace is not present in merkle tree by @walldiss in #2242
deps: bump libp2p by @Wondertan in #2359
feat(node): implement node.AuthVerify and node.AuthNew by @distractedm1nd in #2370
fix: parseNamespace for <= 10 byte namespace IDs by @rootulp in #2325
add kind:testing label by @walldiss in #2375
feat(libs/pidstore): Implement PeerIDStore by @renaynay in #2274
Update ci_release.yml by @MSevey in #2384
feat(gateway): enabling CORS * on gateway by @distractedm1nd in #2373
share: introduce Namespace type by @Wondertan in #2367
feat(header): Check app version in eh validation and return err on mismatch by @renaynay in #2138
chore(share/getters): add unit test for ctxwithsplittimeout by @walldiss in #2374
chore: bump celestia-app rc4 by @evan-forbes in #2371
feat(share/getter) Add support for Non-inclusion proofs by @walldiss in #2256
fix(api/gateway): Handle err from tx properly to avoid panic by @renaynay in #2393
chore: bump celestia-app to v1.0.0-rc5 by @evan-forbes in #2395
deps: bump libp2p by @Wondertan in #2399
fix(libs/utils): Use valid ip4 address by @renaynay in #2394
feat: adding spans to shrex getter by @distractedm1nd in #2404
refactor!(share): integrate new Namespace type by @Wondertan in #2388
chore(go.mod): Bump nmt by @renaynay in #2405
tests(share/eds): add store basic benchmarks by @walldiss in #2407
chore!: bump arabica-9 by @jrmanes in #2417
feat: update Go base alpine - add branch in makefile by @jrmanes in #2414
doc(share/p2p/shrexnd): Improve accuracy of RequestND description by @musalbas in #2418
chore(deps): bump alpine from 3.18.0 to 3.18.2 by @dependabot in #2380
chore(nodebuilder/p2p)! Upgrade to mocha-3 by @sysrex in #2432
share/befp: add benchmarks for fraud proof creation and validation by @vgonkivs in #2400
deps: bump go-header by @Wondertan in #2436
chore: bump celestia-app to v1.0.0-rc8 by @evan-forbes in #2434
deps: bump go-header by @Wondertan in #2438
feat: use context in the common pipeline by @jrmanes in #2447
fix(header): temporary disable skipping verification by @Wondertan in #2456
share/befp: rework collecting and verifying the BEFP by @vgonkivs in #2408
chore: bump celestia app v1.0.0-rc9 by @evan-forbes in #2458
fix(lint): fix importing by @vgonkivs in #2467
cmd: rpc.go fix error parsing height message by @tuxcanfly in #2469
fix(nodebuilder): fix nil pointer exception and simplify synchronization by @Wondertan in #2466
docs: clarify usage of len(merkleRowRoots) by @rootulp in #2431
chore(share/p2p/discovery): demote disc peers log to debug by @renaynay in #2473
chore(deps): bump otel, go-header, and go-fraud by @walldiss in #2472
chore: move versioning outside the main package by @vgonkivs in #2468
refactor(share/availability): don't log error canceled operation by @walldiss in #2475
feat(nodebuilder): Invoke traces from fx by @vgonkivs in #2477
fix: remove root from traces by @vgonkivs in #2483
feat(share/byzantine): handle additional cases in befp validation by @vgonkivs in #2383
chore(deps): bump celestiaorg/.github from 0.2.0 to 0.2.2 by @dependabot in #2484
fix(nodebuilder/tests/swamp): Bump timeout by @renaynay in #2504
dep: update nmt version by @vgonkivs in #2509
feat: new metrics - deployment dashboard by @jrmanes in #2464
test(share/utils): fix no deadline test for ctxWithSplitTimeout by @walldiss in #2502
fix(share/availability): update availability ErrNotAvailable handling logic by @walldiss in #2445
chore(deps): bump cosmossdk.io/errors from 1.0.0-beta.7 to 1.0.0 by @dependabot in #2486
fix: check for namespace version 0 by @rootulp in #2514
chore(deps): bump github.com/spf13/cobra from 1.6.1 to 1.7.0 by @dependabot in #2487
test(share/p2p): fix namespace add operations in tests by @walldiss in #2422
fix(share/eds): remove Has() and Sync() badger calls in inverted index by @walldiss in #2518
fix(share/eds): turn off eds store dagstore gc by default by @walldiss in #2529
feat(share/shrex/nd): add swamp test for shrex nd by @walldiss in #2227
feat(share/eds): introduce List method for the Store by @Wondertan in #2494
fix(share/eds): print error for stuck register shard by @walldiss in #2516
chore(deps): bump celestia-app to rc10 by @cmwaters in #2508
chore: bump app v1.0.0 rc11 by @evan-forbes in #2527
fix(share/p2p): fix race in shrexeds test by @walldiss in #2534
feat(share/store): add eds store metrics by @walldiss in #2536
!feat(share/p2p/shrex-nd): rework shrex-nd to serve data in multiple messages by @walldiss in #2444
chore: bump libp2p by @Wondertan in #2533
feat(cmd): use rpc client instead of http.Request by @vgonkivs in #2521
feat(eds/store): cache proofs after first eds recompute on sampling by @walldiss in #2429
release!: Breaking release V0.11.0 rc9 by @walldiss in #2545
misc(metrics)!: split global attributes into two by @walldiss in #2546
fix(blob): ensure that the share sequence is not empty by @vgonkivs in #2547
fix(rpc): fix commands handling by @vgonkivs in #2515
fix(cmd): fix formatting during submit by @vgonkivs in #2551
fix(share/eds): don't use in-memory buffer for mount after shard recover by @walldiss in #2554
fix(header): Only bridge node should panic on data root mismatch in e… by @renaynay in #2558
fix(share/eds): dagstore shard restore reflection workaround by @walldiss in #2559
metrics(runtime): add runtime metrics by @walldiss in #2570
fix(nodebuilder): set default log level for bitswap-client by @walldiss in #2576
fix: TestShareAvailable_DisconnectedFullNodes by @distractedm1nd in #2560
chore(deps): bump ipfs networking deps by @walldiss in #2478
fix(share/discovery): discovery should try to connect to empty hosts peers by @walldiss in #2575
feat(eds/store): add eds blockstore cache metrics by @walldiss in #2567
chore: bump to rsmt2d v0.11.0 by @rootulp in #2580
fix(share/ipld): Remove proofs from leaked ctx by @walldiss in #2574
feat(nodebuilder/state): Provide stubbed state module if a core endpoint not provided by @renaynay in #2577
chore!: update celestia-app to v1.0.0-rc12 by @evan-forbes in #2581
chore: bump deps + go1.21 by @Wondertan in #2589
fix(share/eds): scale up inverted_index badger levels by @walldiss in #2591
feat(blob): improve gas estimation and track min gas price by @cmwaters in #2511
fix(cmd/blob): fix response by @vgonkivs in #2600
deps: bump dagstore by @walldiss in #2598
nodebuilder/core: Set default ports for RPC and GRPC by @renaynay in #2596
feat(cmd/node): add logs cmd to dynamically change the log level by @vgonkivs in #2601
chore(deps): bump dagstore by @walldiss in #2602
refactor: address namespace follow-ups from celestia-app v1.0.0-rc12 bump by @rootulp in #2597
feat(edssser): introduce EDS Store Stresser and cel-shed utility by @Wondertan in #2482
feat(share/p2p/discovery): Warning for discovery loop if haven't found wanted peers in 5 minutes by @nodersteam in #2573
fix(eds/blockstore): Puts on EDSStore Blockstore by @distractedm1nd in #2532
fix: fixing panic on GetShare with out of bounds indexes by @distractedm1nd in #2605
chore: bump go-header and go-fraud by @Wondertan in #2619
chore(nodebuilder/p2p)!: Bump arabica-10 by @jrmanes in #2639
chore(deps): bump golanci-lint workflow by @distractedm1nd in #2641
feat(share/eds) add dagstore shards status metric by @walldiss in #2642
refactor(share)!: have NamespacedShares fields use snake_case in JSON by @distractedm1nd in #2633
chore(deps): bump golangci/golangci-lint-action from 3.6.0 to 3.7.0 by @dependabot in #2586
chore(deps): bump alpine from 3.18.2 to 3.18.3 by @dependabot in #2564
chore(deps): bump cosmossdk.io/math from 1.1.1 to 1.1.2 by @dependabot in #2594
tests(swamp): befp testing by @vgonkivs in #2584
chore(deps): bump github.com/ipfs/go-ipld-cbor from 0.0.6 to 0.1.0 by @dependabot in #2622
chore(deps): bump github.com/ipfs/go-ipld-format from 0.5.0 to 0.6.0 by @dependabot in #2621
feat: local docker telemetry infra by @distractedm1nd in #2296
chore(deps): bump github.com/ipfs/go-block-format from 0.1.2 to 0.2.0 by @dependabot in #2623
chore!: bump celestia-app to v1.0.0-rc13 by @evan-forbes in #2646
fix(blob): typo, sumitter -> submitter by @ramin in #2648
chore(nodebuilder/tests): Clean up integration tests in p2p_test by @renaynay in #2166
refactor(blob): removing jsonProof by @distractedm1nd in #2652
feat(eds/store): remove corrupted blocks from store by @distractedm1nd in #2625
chore(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp from 0.39.0 to 0.40.0 by @dependabot in #2635
chore(deps): migrate to boxo by @Wondertan in #2595
chore(deps): bump github.com/ipfs/go-ipfs-ds-help from 1.1.0 to 1.1.1 by @dependabot in #2624
chore(go.mod): bump go-header by @renaynay in #2657
feat(nodebuilder/header): Bootstrap from previously seen peers by @renaynay in #2507
feat(modp2p): listen on WebTransport by default by @Wondertan in #2660
fix(share/eds): skip concurrent writes on edsstore Put by @walldiss in #2613
chore(lint/sort-imports): fix absolute path for sort-imports by @walldiss in #2640
chore: cleanup usage of share.Root(DAH) by @Wondertan in #2481
improvement!(blob/service): add options to submit by @vgonkivs in #2630
chore!: bump mocha chain with new bootstrappers and genesis hash by @Bidon15 in #2668
feat(headertest): Validate exact error types in header tests by @Manav-Aggarwal in #2644
chore: bump celestia-app to v1.0.0-rc14 by @evan-forbes in #2662
chore: bump libp2p by @Wondertan in #2676
fix(modshare): start peer manager after Syncer by @Wondertan in #2677
feat(cmd/blob): add fee and gasLimit flags by @vgonkivs in #2669
chore(nodebuilder/p2p): BlockTime from 15 sec --> 10 sec by @renaynay in #2692
refactor(nodebuilder/header): get rid of InitStore by @renaynay in #2678
chore: fix unknown in ghcr registry by @Bidon15 in #2693
chore: bump docker ci to v0.2.3 by @Bidon15 in #2696
chore(deps): Bump actions/checkout from 3 to 4 by @dependabot in #2690
chore(deps): Bump golang.org/x/crypto from 0.12.0 to 0.13.0 by @dependabot in #2688
chore(deps): Bump github.com/ipfs/go-block-format from 0.1.2 to 0.2.0 by @dependabot in #2682
chore(deps): Bump github.com/libp2p/go-libp2p-routing-helpers from 0.7.2 to 0.7.3 by @dependabot in #2687
chore(deps): Bump go.opentelemetry.io/contrib/instrumentation/runtime from 0.42.0 to 0.43.0 by @dependabot in #2685
chore(deps): Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp from 1.16.0 to 1.17.0 by @dependabot in #2683
Stale issue github workflow by @ramin in #2667
refactor(swamp): use RPC client instead of service pointers by @distractedm1nd in #2699
Goreleaser by @MSevey in #2661
fix(shrexeds): close stream by @Wondertan in #2705
feat(share/eds): Rework accessor cache by @walldiss in #2612
chore(deps): Bump actions/checkout from 3 to 4 by @dependabot in #2718
chore(deps): Bump go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp from 0.40.0 to 0.41.0 by @dependabot in #2713
chore(deps): Bump github.com/hashicorp/golang-lru/v2 from 2.0.5 to 2.0.6 by @dependabot in #2711
feat(share/store/cache): Split accessor cache into recent and blockstore by @walldiss in #2656
chore(deps): Bump mathieudutour/github-tag-action from 6.0 to 6.1 by @dependabot in #2721
chore(deps): Bump goreleaser/goreleaser-action from 4 to 5 by @dependabot in #2719
chore(deps): Bump celestiaorg/.github from 0.2.2 to 0.2.3 by @dependabot in #2720
chore(deps): Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp from 1.17.0 to 1.18.0 by @dependabot in #2714
chore(dependabot): add otel group to dependabot by @walldiss in #2722
chore(deps): Bump go.uber.org/zap from 1.25.0 to 1.26.0 by @dependabot in #2731
chore(deps): Bump go.opentelemetry.io/contrib/instrumentation/runtime from 0.43.0 to 0.44.0 by @dependabot in #2732
docs: Add share p2p documentation by @derrandz in #2008
feat(share/eds/cache): Add accessor cache refs counter by @walldiss in #2723
feat(share/eds/store): expose eds store params by @walldiss in #2724
deps!: bump nmt, breaking JSON RPC return types including nmt Proofs by @distractedm1nd in #2728
refactoring(full/availability): removing TeeGetter, storing via SharesAvailable by @distractedm1nd in #2726
chore(deps): Bump google.golang.org/grpc from 1.58.0 to 1.58.1 by @dependabot in #2740
chore(deps): Bump github.com/ipfs/boxo from 0.12.0 to 0.13.0 by @dependabot in #2734
fix(full/availability): short-circuiting SharesAvailable on empty DAH by @distractedm1nd in #2746
fix: improve ErrNoStateAccess message by @distractedm1nd in #2752
refactor(availability): merging cache.ShareAvailability and light.ShareAvailability by @distractedm1nd in #2741
refactor(share/discovery): decrease default Advertisement interval by @walldiss in #2758
chore!: bump core and app by @cmwaters in #2753
chore(deps): Bump github.com/ipfs/boxo from 0.13.0 to 0.13.1 by @dependabot in #2757
chore(deps): Bump google.golang.org/grpc from 1.58.1 to 1.58.2 by @dependabot in #2756
fix(share/p2p/peers): Prevent making pools for nil DataHash by @renaynay in #2761
ignore dependabot in issue-label-automation/automate-labels workflow by @ramin in #2763
test(share/eds/cache): benchmark eds cache eviction performance by @walldiss in #2778
chore(deps): fix dependabot otel grouping by @walldiss in #2795
chore(deps): Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace from 1.18.0 to 1.19.0 by @dependabot in #2784
chore: bump celestia-app, celestia-core, and the cosmos-sdk by @evan-forbes in #2801
refactoring!(rpc/cmd): rework rpc commands by @vgonkivs in #2812
chore(deps): Bump the otel group with 2 updates by @dependabot in #2800
chore(deps): Bump github.com/prometheus/client_golang from 1.16.0 to 1.17.0 by @dependabot in #2787
chore(deps): Bump github.com/hashicorp/golang-lru/v2 from 2.0.6 to 2.0.7 by @dependabot in #2785
chore(deps): Bump alpine from 3.18.3 to 3.18.4 by @dependabot in #2794
(chore) ensure all dependency group PRs get the default label by @ramin in #2798
deps: bump go-header by @renaynay in #2813
fix(share/peer-manager): use mutex for peer-manager pool cleanup by @walldiss in #2817
chore(deps): Bump go.opentelemetry.io/contrib/instrumentation/runtime from 0.44.0 to 0.45.0 by @dependabot in #2819
feat(discovery)!: discover peers by tag by @walldiss in #2730
chore(deps): Bump golang.org/x/crypto from 0.13.0 to 0.14.0 by @dependabot in #2821
fix(docs): fix GetVerifiedRange doc by @vgonkivs in #2826
chore(deps): Bump golang.org/x/sync from 0.3.0 to 0.4.0 by @dependabot in #2820
fix(rpc)!: various rpc breaks by @distractedm1nd in #2825
chore(share/ipld): obliterate expensive ipld getter tracing by @Wondertan in #2832
fix(share/p2p/peer-manager) fix race for hasPeerCh pointer read by @walldiss in #2835
chore: bump to the official v1.0.0 tag of celestia-app by @evan-forbes in #2831
fix(share/eds): fix races in cache tests by @walldiss in #2834
chore: Update docker-build-publish.yml by @MSevey in #2838
fix(modp2p): temporary disable quic by @Wondertan in #2837
chore: adding mainnet chain id and setting as default by @distractedm1nd in #2840
bug: fix duplicate docker build stages by @MSevey in #2845
fix(blob/service): fix getByCommitment by @vgonkivs in #2828
fix: removing chainid from default store name for mainnet by @distractedm1nd in #2841
deps!: Bumps go-header by @renaynay in #2844
misc(blob): Add debug logs for the GetAll method by @rise2semi in #2749
chore(deps): Bump tspascoal/get-user-teams-membership from 2 to 3 by @dependabot in #2827
chore(deps): Bump github.com/celestiaorg/celestia-app from 1.0.0 to 1.1.0 by @dependabot in #2850
fix(share/getters): short-circuit on empty root in ShrexGetter by @distractedm1nd in #2846

New Contributors

@musalbas made their first contribution in #2418
@tuxcanfly made their first contribution in #2469
@cmwaters made their first contribution in #2508
@nodersteam made their first contribution in #2573
@Manav-Aggarwal made their first contribution in #2644
@rise2semi made their first contribution in #2749

Full Changelog: 0.10.1...v0.11.0

celestiaorg/celestia-node v0.11.0 on GitHub