Restate v1.4.0
[NOTICE] Expect a short period of leader unavailability during rolling upgrades from v1.3.2 to v1.4.0 due to changes in cluster controller and gossip protocol. The availability is transient and the cluster will quickly fix all partition leaders once all cluster nodes have been upgraded.
Release Highlights
- Restate clusters will automatically perform partition rebalancing across configured nodes. This delivers even space and compute utilization across the board.
- A new gossip-based failure detector and partition leader placement sub-systems which deliver up to 10x faster detection of network partitions or node failures. Gossip is also used to disseminate partition leadership state for faster reaction to partition reconfigurations.
- The Replicated Loglet is the default in v1.4. Existing loglets backed by the Local Loglet provider will be automatically migrated on first run. Replicated Loglet delivers better performance even for single-node deployments and is the recommended provider in all cases. The Local Loglet continues to be available and automatic migration will not be performed if it is explicitly configured as the default provider. Downgrading to Restate v1.3 is safe regardless of which loglet provider is used.
- The previously deprecated local metadata backend for single-node deployments is removed in v1.4.0. The default is the replicated metadata server, nodes will perform automatic migration on start.
- Restate server role
http-ingress
is enabled by default, new setups will automatically have this role turned on. - Query engine optimizations and fixes that improve responsiveness and system reliability under severe load and slow storage backends.
- [experimental] A new feature-gated option to enable longer retention of journal and completed invocations. This feature is not compatible with
v1.3.x
and can be enabled withadmin.experimental-feature-force-journal-retention = "30min"
. Care should be taken when enabling this feature as it will increase the storage footprint and it might increase latency in some operations. - Default invoker concurrent invocation node defaults to
1000
instead of100
. - Zstd compression is promoted to default for the message fabric in restate clusters.
- Partition Processors now track the locally-persisted LSN automatically. This is now reported by
restatectl
as Durable LSN (from Persisted LSN). The configuration propertiesworker.storage.persist-lsn-interval
andworker.storage.persist-lsn-threshold
are no longer required and will be ignored. If you previously used these to tune single-node log trimming, you can safely remove them. admin.query-engine.pgsql-bind-address
has been deprecated and its default value was set to None (#3088)- Various previously deprecated configuration properties have been removed (
admin.log-trim-threshold
,metadata-store-client.address
)
Install
Pull the Docker images
docker pull docker.restate.dev/restatedev/restate:1.4.0
docker pull docker.restate.dev/restatedev/restate-cli:1.4.0
Install prebuilt binaries via Homebrew
brew install restatedev/tap/restate-server
brew install restatedev/tap/restate
brew install restatedev/tap/restatectl
Install prebuilt binaries into your npm project
npm install @restatedev/restate-server@1.4.0
npm install @restatedev/restate@1.4.0
npm install @restatedev/restatectl@1.4.0
Download binary archives
File | Platform | Checksum |
---|---|---|
restate-server-aarch64-apple-darwin.tar.xz | Apple Silicon macOS | checksum |
restate-server-x86_64-apple-darwin.tar.xz | Intel macOS | checksum |
restate-server-aarch64-unknown-linux-musl.tar.xz | ARM64 MUSL Linux | checksum |
restate-server-x86_64-unknown-linux-musl.tar.xz | x64 MUSL Linux | checksum |
restate-cli-aarch64-apple-darwin.tar.xz | Apple Silicon macOS | checksum |
restate-cli-x86_64-apple-darwin.tar.xz | Intel macOS | checksum |
restate-cli-aarch64-unknown-linux-musl.tar.xz | ARM64 MUSL Linux | checksum |
restate-cli-x86_64-unknown-linux-musl.tar.xz | x64 MUSL Linux | checksum |
restatectl-aarch64-apple-darwin.tar.xz | Apple Silicon macOS | checksum |
restatectl-x86_64-apple-darwin.tar.xz | Intel macOS | checksum |
restatectl-aarch64-unknown-linux-musl.tar.xz | ARM64 MUSL Linux | checksum |
restatectl-x86_64-unknown-linux-musl.tar.xz | x64 MUSL Linux | checksum |
What's Changed
- [Core] New global metadata synchronization infrastructure by @AhmedSoliman in #3159
- [Bifrost] Improves appender behaviour when detecting seals by @AhmedSoliman in #3162
- [Core] Tuning failover config defaults by @AhmedSoliman in #3165
- Don't fail cloud login if we cannot open the browser by @tillrohrmann in #3180
- feat: add metadata-server raft log trim threshold option by @lsytj0413 in #3171
- Fix issue with incorrect command count by @slinkydeveloper in #3185
- Introduction of Message Fabric V2 by @AhmedSoliman in #3183
- [minor] Move partition processor rpc client to ingress-http crate by @AhmedSoliman in #3191
- [Core] Introducing network swimlanes by @AhmedSoliman in #3193
- [Core] Read-modify-write for global metadata by @AhmedSoliman in #3164
- Deprecation notice for old protocol by @slinkydeveloper in #3189
- Introduce ReadIsolation for Storage transactions by @tillrohrmann in #3198
- Use global metadata wrapper when writing metadata for provisioning the cluster by @tillrohrmann in #3174
- Add support for protocol version Bilrost (v2) by @muhamadazmy in #3190
- Using Bilrost for GetNodeStatus messages by @muhamadazmy in #3172
- Add support for removing/adding metadata server nodes by @tillrohrmann in #3138
- Invocation epoch by @slinkydeveloper in #2904
- Efficiently pack metadata versions in network structs by @AhmedSoliman in #3196
- [Core] Enable failure detector to start earlier by @AhmedSoliman in #3199
- [mechanical] Rename BaseRole into FailureDetector by @AhmedSoliman in #3200
- Bilrost only wire codec macro by @muhamadazmy in #3206
- fix(1585): enable query parallelism with 1 by @lsytj0413 in #3203
- [Core] Fix occasional shutdown lock-up due to incomplete connections being managed tasks by @AhmedSoliman in #3209
- [Core] WireEncode accepts a reference instead of an owned value by @AhmedSoliman in #3214
- introduce BilrostAs derive macro by @muhamadazmy in #3208
- Fix building docker by @slinkydeveloper in #3218
- Move
PartitionProcessorRpcClient
inrestate_types
, and rename itInvocationClient
. by @slinkydeveloper in #3211 - [invoker] Stop retrying when the invocation is not there anymore. by @slinkydeveloper in #3221
- [Admin API] Add deployment sdk version and min/max protocol versions. by @slinkydeveloper in #3220
- [Core] Support for sent notification for unary messages by @AhmedSoliman in #3213
- [Core] New LazyConnection with background auto-reconnect by @AhmedSoliman in #3215
- LogServer Bilrost (v2) message types by @muhamadazmy in #3176
- [Restate UI] Update to v0.0.66 by @nikrooz in #3231
- [Restate UI] Update to v0.0.67 by @nikrooz in #3232
- Upgrade tokio and metrics by @AhmedSoliman in #3234
- Leave member state when removing yourself from Raft configuration by @tillrohrmann in #3207
- Bilrost fast encoding + more common helper types by @muhamadazmy in #3222
- fix(1074): Do not panic in CLI example command if interrupted by @lsytj0413 in #3202
- Support zone spread in helm chart by @jackkleeman in #3225
- [Core] Introducing global ClusterState view by @AhmedSoliman in #3226
- [minor] Misc cleanups by @AhmedSoliman in #3229
- [FailureDetector] Gossip Configuration and Network Messages by @AhmedSoliman in #3228
- [FailureDetector] Gossip-based failure detection system by @AhmedSoliman in #3230
- [minor] Improve PartitionTable naming and style by @AhmedSoliman in #3243
- Removes SuspectNode from old ClusterState by @AhmedSoliman in #3241
- [Restate UI] Update to v0.0.68 by @nikrooz in #3245
- Add event-driven persisted LSN tracking support to PartitionStoreManager by @pcholakov in #3210
- [org] Splitting metadata-server and overhaul of metadata-related crates by @AhmedSoliman in #3253
- fix(3205): upload test logs when failed by @lsytj0413 in #3236
- [FailureDetector] PartitionReplicaSetStates in FailureDetector by @AhmedSoliman in #3244
- Introduce WorkerState by @tillrohrmann in #3262
- [Restate UI] Update to v0.0.69 by @nikrooz in #3269
- [Restate UI] Update to v0.0.70 by @nikrooz in #3270
- Enable concurrent PartitionStore snapshot creation by @pcholakov in #3271
- Support
#[net_serde(skip)]
attribute by @muhamadazmy in #3242 - Harden three_logserver_append_after_seal_concurrent by @tillrohrmann in #3266
- Set RESTATE_NODE_NAME in helm charts by @tillrohrmann in #3273
- Run SequencerAppender tasks in default runtime by @tillrohrmann in #3279
- Fix first_attempt detection when updating the Storage- and WorkerState by @tillrohrmann in #3281
- Improve trace logging of various components by @tillrohrmann in #3282
- Only transition node's MetadataServerState from Member to Standby if node was member before by @tillrohrmann in #3276
- [core] Enable protocol-version gating when sending messages by @AhmedSoliman in #3286
- Integrate restate_core::cluster_state::ClusterState into the ClusterController by @tillrohrmann in #3268
- WAL: Custom Encoding for
Envelope
by @muhamadazmy in #3261 - Batch bifrost watchdog trim operations by @jackkleeman in #3264
- [Restate UI] Update to v0.0.72 by @nikrooz in #3288
- [Restate UI] Update to v0.0.73 by @nikrooz in #3289
- Metadata sync messages to use normal connection abstractions by @AhmedSoliman in #3287
- Move some long-running ClusterContoller select arms to background tasks by @pcholakov in #3272
- [invoker] Validate when user incorrectly provides key by @slinkydeveloper in #3248
- EntryLite data model to avoid sending entry payloads around when rendering the invocation timeline by @slinkydeveloper in #3299
- Snapshot errors cleanup by @pcholakov in #3254
- chore: add hach clippy with each feature by @lsytj0413 in #3166
- Enable missing feature needed by restate-encoding by @muhamadazmy in #3304
- BalancedSpreadSelector initial implementation by @AhmedSoliman in #3283
- [Restate UI] Update to v0.0.74 by @nikrooz in #3310
- [Restate UI] Update to v0.0.75 by @nikrooz in #3312
- Extend EpochMetadata to support current and next PartitionConfiguration by @tillrohrmann in #3235
- Flag to always retain journal by @slinkydeveloper in #3296
- Add
sys_invocation.scheduled_start_at
by @slinkydeveloper in #3308 - Add just docker-local-fedora as temporary solution to iterate fast locally on e2e tests by @slinkydeveloper in #3316
- [Restate UI] Update to v0.0.76 by @nikrooz in #3317
- Remove invoker concurrency limit by @muhamadazmy in #3305
- bilrost 1013 by @mumbleskates in #3302
- Let scheduler populate the PartitionReplicaSetStates by @tillrohrmann in #3315
- Bump MAX_SERVICE_DISCOVERY_PROTOCOL_VERSION to V3 by @slinkydeveloper in #3320
- [FailureDetector] Seed FD with cluster-state from peer on startup by @AhmedSoliman in #3313
- [chore] Set the features for tonic used in grpc crates by @tillrohrmann in #3314
- [Bifrost] Fixes and improvements to check-seal/find-tail and read-stream by @AhmedSoliman in #3307
- [Bifrost] Self-initiated replicated loglet reconfiguration by @AhmedSoliman in #3309
- [Admin] Remove log controller as it's not needed anymore by @AhmedSoliman in #3318
- Improve restatectl_smoke_test resilience by @AhmedSoliman in #3326
- [Bifrost] Auto-improvement of logs on preferred writer nodes by @AhmedSoliman in #3325
- Properly handle PartitionReplication::Everywhere by @AhmedSoliman in #3327
- Prefer worker nodes that are caught up when selecting leaders by @AhmedSoliman in #3328
- Remove periodic find-tail config key by @AhmedSoliman in #3329
- [Gossip] Nodes delay their own suspect->alive transition slightly by @AhmedSoliman in #3331
- [Worker] Notify bifrost of being preferred writer only after successful leadership by @AhmedSoliman in #3332
- Rename Persisted LSN to Durable LSN by @pcholakov in #3277
- Fix: restate invocations cancel should not consider completed invocat… by @pcholakov in #3334
- Fix local-loglet params generation by @AhmedSoliman in #3335
- Update to latest rust-rocksdb 0.41.0 + Checkpoint flush fix by @pcholakov in #3338
- [EntryLite] Add result to GetEagerStateLite by @slinkydeveloper in #3343
- Report the correct planned mode based on the last observed leader epoch by @tillrohrmann in #3340
- [Admin] Ensure balanced replica-set selection for partitions by @AhmedSoliman in #3330
- [FailureDetector] Exchange partition leadership state by @AhmedSoliman in #3339
- [Admin] More graceful polls of epoch metadata by @AhmedSoliman in #3342
- [Metadata] Minor improvements to trace logging by @AhmedSoliman in #3344
- [Worker] minor metrics changes by @AhmedSoliman in #3345
- [restatectl] Support --replication by @AhmedSoliman in #3346
- Provide configuration knob in the deployment manifest to
enable_lazy_state
by @slinkydeveloper in #3319 - Improvements to ClusterTables by @AhmedSoliman in #3351
- bilrost 0.1013 is released 😌 by @mumbleskates in #3358
- Update PartitionRouting to rely in PartitionReplicaSetStates and ClusterState by @tillrohrmann in #3353
- Adds support for partition_replica_set cluster table by @AhmedSoliman in #3356
- [restatectl] nodes list shows StorageState and WorkerState by @pcholakov in #3354
- Split ClusterController reponsibilities into separate tasks by @tillrohrmann in #3357
- Add restate_version in ServiceInvocation by @slinkydeveloper in #3224
- Add stale (wrong epoch) journal read check in invoker. by @slinkydeveloper in #3362
- Use InvocationClient in admin API, introduce new API endpoints for cancel/kill/purge by @slinkydeveloper in #3227
- fix(3239): migrate replicated loglet message to bilrost by @lsytj0413 in #3274
- Switch to jiff for parsing for human-friendly durations by @pcholakov in #2917
- [restatectl] Add json output support to sql subcommand by @pcholakov in #3367
- [CLI] Add a
state patch
subcommand by @pcholakov in #3255 - Implement a custom encoder for 3rd party types by @muhamadazmy in #3359
- fix panic when receiving ControlProcessors from older version by @pcholakov in #3366
- Changes to the ingress private feature by @slinkydeveloper in #3374
- Reduce the size of some service-client enums by @jackkleeman in #3375
- Document debuginfod by @jackkleeman in #3376
- Clarify definition of application platform service in additional usage grant by @tillrohrmann in #3373
- [Restate UI] Update to v0.0.77 by @nikrooz in #3379
- Reducing Clones in journal_v2 encode path by @AhmedSoliman in #3368
- [chore] Rust toolchain update to 1.87.0 by @AhmedSoliman in #3361
- [cleanup] remove unused proxy types by @muhamadazmy in #3377
- Re-design of the
Schema
data structure by @slinkydeveloper in #3311 - Replace snapshot implicit log ids with partition table lookup by @pcholakov in #3349
- Update object_store crate to 0.12.2 by @pcholakov in #3383
- Fix bug in cleaner with overflows by @slinkydeveloper in #3378
- [Bifrost] Find tail will seal if the sequencer is gone or long dead by @AhmedSoliman in #3385
- Return errors when datafusion runs out of memory by @jackkleeman in #3381
- Store Transient errors in journal events by @slinkydeveloper in #3301
- Remove deprecated Local metadata server and config options by @pcholakov in #3350
- Ensure local df scanner is always used by @jackkleeman in #3393
- fix(3088): disable pgsql endpoint by default by @lsytj0413 in #3094
- [minor] PartitionRouting to pin task-center for more portability by @AhmedSoliman in #3397
- SequencerStatus refactor by @muhamadazmy in #3391
- Set the initial current partition configuration version to be equal to the EpochMetadata's version by @tillrohrmann in #3386
- flexbuffers backward compatibility for replicated_loglet messages by @muhamadazmy in #3390
- [datafusion] reduce default sizes by @igalshilman in #3398
- [tracing] improve http client errors by @muhamadazmy in #3394
- [bilrost] migrate remote scanner types to bilrost by @muhamadazmy in #3396
- Support pushed-down limits in datafusion by @jackkleeman in #3399
- fix: restore PartitionProcessorRpcError::Busy variant by @pcholakov in #3400
- Improve snapshot download error messages by @pcholakov in #3382
- Don't try and load region from AWS sdk if its been provided by @jackkleeman in #3403
- Simplify lambda restatectl command and add it to cli by @jackkleeman in #3267
- Enable http2 adaptive window and upgrade h2 by @AhmedSoliman in #3406
- [StateMachine] Semantic version barrier support by @AhmedSoliman in #3401
- Use normal hash join, just ensure that the big table is second by @jackkleeman in #3402
- [Restate UI] Update to v0.0.78 by @nikrooz in #3408
- Make datafusion configurable at runtime by @jackkleeman in #3409
- Configurable HTTP2 inital max send streams by @muhamadazmy in #3411
- [CI] pin install-action nextest to 0.9.98 by @AhmedSoliman in #3413
- Use ReplicatedLoglet by default by @pcholakov in #3410
- [chore] Key dependencies update by @AhmedSoliman in #3412
- Always re-resolve and re-connect initial raft addresses when using them by @jackkleeman in #3407
- [datafusion] Update to 48.0 by @igalshilman in #3419
- Set default grpc compression to Zstd by @muhamadazmy in #3414
- Remove deprecated configuration options by @muhamadazmy in #3416
- Cluster fingerprint in network and snapshots (passive support) by @AhmedSoliman in #3415
- Print version notice on restate-server releases by @AhmedSoliman in #3420
- [Release Prep] Setting min supported to be 1.3.0 by @AhmedSoliman in #3421
- fix deprecated into_path in dump-log command by @AhmedSoliman in #3423
- Set default concurrent invocation limit to 1000 by @muhamadazmy in #3422
- Fix
append_time
of theInputCommand
by @slinkydeveloper in #3430 - [RocksDB] Introducing background iterators for data fusion by @AhmedSoliman in #3428
- [DataFusion] Move VO status table to use background iterator by @AhmedSoliman in #3433
- [DataFusion] Move journal and idempotency tables to background iterators by @AhmedSoliman in #3429
- [DataFusion] Move promise and inbox tables to background iterators by @AhmedSoliman in #3431
- [DataFusion] Move invocation_status table to use background iterators by @AhmedSoliman in #3432
- Set histogram quantiles to [50, 90, 99, and 100%] by @muhamadazmy in #3425
- Disable debug assertions in release builds by @tillrohrmann in #3387
- fix: make clippy works by @lsytj0413 in #3437
- [restatectl] compacting partition list/logs tables by @AhmedSoliman in #3436
- Show
metadata-server remove-node
errors by @muhamadazmy in #3434 - [datafusion] Decouples the physical/logical partition scanners by @igalshilman in #3427
- fix(3157): remove deprecated options by @lsytj0413 in #3169
- [Telemetry] Sets default for histogram cleanup to 3 minutes by @AhmedSoliman in #3440
- [Telemetry] Cleaning up low-signal and unused metrics by @AhmedSoliman in #3442
- Enable http-ingress role by default by @AhmedSoliman in #3443
- [helm] Add namespace to metadata of remaining resources by @avilaton in #3336
- Set version to v1.4.0 prior to release by @AhmedSoliman in #3445
- Always seal local metadata store on startup by @pcholakov in #3444
New Contributors
- @mumbleskates made their first contribution in #3302
- @avilaton made their first contribution in #3336
Full Changelog: v1.3.2...v1.4.0