Dragonfly v2.0.0

We have been waiting three years to bump the major version.

While this specific release does not add new major features by itself, it marks a significant milestone in our maturity, performance, and production readiness.

Dragonfly v2.0.0 expands Redis and Valkey compatibility with Valkey 9 RDB loading, GEOSEARCHSTORE, and improved replication backlog controls. Performance work reduces connection-processing and memory-accounting overhead while adding tighter controls for defragmentation and buffer retention. Tiered storage, cluster migration, streams, compressed lists, and RESP3 behavior received substantial correctness improvements. This release also strengthens RESTORE validation against malformed or incompatible payloads.

Highlights

Added Valkey 9 RDB support and GEOSEARCHSTORE. (#8251, #7984)
Reduced connection-accounting, parsing, buffering, and RDB serialization costs. (#8246, #8225, #8140, #8005)
Hardened RESTORE, cluster migration, tiered storage, streams, RESP3, and compressed-list handling. (#8260, #8240, #8026, #8015, #8085, #8014)

Commands

Added GEOSEARCHSTORE with radius or box queries, sorting, limits, distance scores, ACL enforcement, and journaling. (#7984)
Added MEMORY DEFRAGMENT-SEGMENTS [threshold] to reclaim sparsely used DashTable pages and report page usage. (#7995)

Performance

Refreshes Memcached parser memory usage once per batch and skips no-op pipeline squash refreshes. (#8246)
Skips reply-batch setup when no pipeline replies are in flight. (#8224)
Refreshes connection-memory telemetry once per parse cycle instead of after every pipelined command. (#8225)
Restores fast allocation-tracker rejection outside configured ranges and after trackers are cleared. (#8207)
Adds an opt-in shared RESP V2 read buffer per proactor, disabled by default. [Projected future reduction from about 800 MB to under 1 MB for 25K connections on 32 CPUs; not realized by this commit.] (#8145)
Adds configurable burst and cooldown limits for background defragmentation. [Example configuration limits defragmentation to approximately 1% of one CPU core.] (#8115)
Releases oversized RDB serializer buffers after flushing. [Retained capacity is capped at 4 MiB by default; previously a 512 MiB object could leave 512 MiB allocated.] (#8140)
Shrinks client input buffers after low usage and adds configuration and metrics. [Reduces the default maximum by 50%, from 64 KiB to 32 KiB.] (#8005)

Replication

Loads Valkey 9 RDB hashes containing persistent and expiring fields. (#8251)
Retains replication backlog entries by age and memory budget, deprecating the fixed-length option. (#8039)

Bug fixes

RESTORE rejects duplicate hash fields in HASH and HASH_LISTPACK payloads. (#8260)
SLOWLOG GET now returns 10 entries by default while preserving explicit counts and -1. (#8248)
Makes connection memory introspection constant time by caching mutable state usage. [Reduces connection memory accounting from O(n) to O(1).] (#8245)
Detects RESP commands after empty inline lines, fixing redis-cli --pipe sentinel timeouts. (#8244)
Retries cluster migration finalization when pausing clients times out. (#8240)
Decodes RedisShake RDB type 20 as a set listpack regardless of the footer version. (#8227)
Rejects RESTORE payloads whose declared lengths exceed the remaining input, preventing remote out-of-memory crashes. (#8222)
Defers container deletion during paused yielding RM iterations to prevent use-after-free. (#8210)
Prevents SORT LIMIT range overflow near 32-bit limits. (#8212)
Returns the Redis-compatible <index>: no such index FT.INFO error. (#8199)
Makes SORT ... STORE delete the destination and return 0 when the source is missing. (#8190)
Releases unused cached ParsedCommand entries in the V2 pipeline. (#8192)
Reports multi-key XREADGROUP errors in command argument order. (#8186)
Preserves command-family and ACL metadata when command aliases are cloned. (#8175)
Validates every XREADGROUP stream before mutating consumer-group state. (#8158)
Uses a startup ACL snapshot throughout Lua script execution. (#8116)
Bounds incremental expired-key scanning by physical bucket, time, and item quotas. (#8129)
Recomputes invalid stream memory deltas instead of wrapping cached values. [Prevents reported size from inflating from a few GiB to terabytes.] (#8066)
Makes used_memory_peak_rss report peak resident memory. (#8143)
Excludes commands parsed before V2 traffic logging starts. (#8139)
Records migration target OOM acknowledgements and removes transferred slots. (#8127)
Fixes Apple Clang 16 ARM64 crashes during float32-to-bfloat16 conversion. (#8123)
Logs V2 traffic before dispatch without suspension or duplicate retry records. (#8096)
Delivers queued Pub/Sub messages before V2 UNSUBSCRIBE confirmations. (#8075)
Uses header-aware accounting and a fixed cutoff for tiered small-bin reclamation. (#8081)
Returns null arrays for empty ZMPOP, BZMPOP, LMPOP, and BLMPOP results. (#8092)
Honors ZADD GT and ZADD LT for skiplist-encoded sorted sets. (#8091)
Correctly encodes RESP3 null replies for blocking stream, list, and sorted-set commands. (#8085)
Rejects FT.CREATE definitions without a SCHEMA containing at least one field. (#8054)
Returns the required RESP3 map shape when a blocked XREAD wakes. (#8064)
Preserves offloaded hashes, updates, and Memcached flags during serialization. (#8026)
Switches TLS listener contexts atomically and corrects cross-thread tls_bytes accounting. (#8043)
Completes tiered SET slot and database accounting before journaling and stashing adjustments. (#8050)
Always exports expired_keys_total and evicted_keys_total, including zero values. (#8052)
Attributes tiered list-node operations to the correct cluster slot. (#8046)
Makes XGROUP MKSTREAM create a stream only when the key is absent. (#8057)
Uploads offloaded values before RENAME or MOVE transfers tiered keys. (#8004)
Journals FLUSHSLOTS inside its global transaction with distinct transaction IDs. (#8032)
Wakes blocked XREADGROUP clients with WRONGTYPE when a stream becomes another data type. (#8038)
Replicates stored cross-shard GEORADIUS results as explicit DEL and ZADD operations. (#8030)
Frees unfinished chunked values on their owning shards after incomplete RDB loads. (#8037)
Keeps search-indexed hashes resident and materializes existing offloaded hashes when needed. (#8018)
Wakes blocked XREADGROUP clients with NOGROUP when their stream expires. (#8031)
Prevents underfilled tiered bins from repeatedly entering defragmentation. (#8024)
Prevents Lua async-call detection from exhausting fiber stacks or performing unsafe rewrites. (#7974)
Removes QList compression savings when compressed interior nodes are erased. (#8023)
Restores the final QList node’s compressed state after partial reads. (#8010)
Wakes blocked XREADGROUP clients with NOGROUP or WRONGTYPE after stream deletion or recreation. (#8015)
Preserves serial semantics for connection-state commands in squashed pipelines. (#8016)
Corrects compressed-list memory tracking and unsigned underflow. [Corrects approximately 13x over-reporting on a Celery-like workload.] (#8014)
Copies only compressed data into replacement buffers during quicklist defragmentation. [Regression allocation fell from 32,112 bytes to 8,496 bytes.] (#8011)
Validates cuckoo filter capacity and expansion bounds while adding configurable defaults. (#7965)

Huge thanks to all the contributors! ❤️

New Contributors

@HighlightCode made their first contribution in #8017
@Shikha-code36 made their first contribution in #7974
@avy252004 made their first contribution in #8037
@PixiBixi made their first contribution in #8052
@marttp made their first contribution in #8138
@tjovicic made their first contribution in #8171
@denis-samatov made their first contribution in #8199
@samber made their first contribution in #8283
@jmealo made their first contribution in #8289

What's Changed

feat(server): add per-slot tiered_bytes to DFLYCLUSTER GETSLOTINFO by @vyavdoshenko in #7983
chore(helm-chart): update to v1.40.0 by @github-actions[bot] in #7991
feat: reduce min_capacity for OAHTable by @BorysTheDev in #8003
chore: cuckoo compatibility fixes and missing flags by @kostasrim in #7965
CI: do not print successful gtest output by @abhijat in #8000
fix(qlist): copy only the compressed blob when defragmenting a compressed node by @romange in #8011
fix(qlist): fix malloc_size_ accounting for compressed nodes by @romange in #8014
fix: preserve connection state across squashed pipelines (Fixes #8006) by @glevkovich in #8016
fix: wake blocked XREADGROUP when its stream is deleted by @BorysTheDev in #8015
feat: use OAHSet by default by @BorysTheDev in #8013
fix(core): make QList read paths footprint-neutral by @romange in #8010
CI: wait for release sync before running package-install tests by @abhijat in #7999
CI: install python3 in container before running script by @abhijat in #8021
chore(helm-chart): update to v1.40.1 by @github-actions[bot] in #8020
fix: wake blocked XREADGROUP on FLUSHDB by @BorysTheDev in #8019
fix(qlist): release compression stats when DelNode drops a compressed node by @romange in #8023
docs(ko-KR): fix typos in Korean README by @HighlightCode in #8017
fix(CI): Avoid re-signing existing released RPMs by @abhijat in #8022
feat(facade): reclaim (shrink) connection buffers by @glevkovich in #8005
fix(test_client_kill): wait for client kill cleanup by @glevkovich in #8025
fix(scripting): avoid std::regex recursion SIGABRT in DetectPossibleAsyncCalls by @Shikha-code36 in #7974
feat(core): Table segment defragmentation by @abhijat in #7979
fix(tiering): small-bins defrag livelock on underfilled bins by @kostasrim in #8024
fix: wake blocked XREADGROUP when its stream expires by @BorysTheDev in #8031
fix(test): cover V2 idle IoBuf reclamation by @glevkovich in #8036
fix: unblock stream if flush slots by @BorysTheDev in #8035
fix(search): keep indexed hashes resident under experimental hash offloading by @vyavdoshenko in #8018
Fix/rdb finishload cross shard free by @avy252004 in #8037
fix(geo): avoid replica crash on cross-shard GEORADIUS/GEORADIUSBYMEM… by @Shikha-code36 in #8030
chore: increase load for test_replicaof_reject_on_load by @kostasrim in #8044
fix: GAT 0 journals PERSIST instead of PEXPIREAT 0 by @vyavdoshenko in #8041
fix: the blocked XREADGROUP returns WRONGTYPE for overwrite commands by @BorysTheDev in #8038
fix(cluster): journal FLUSHSLOTS from inside the flush transaction by @vyavdoshenko in #8032
feat(server): Add defrag table segments command by @abhijat in #7995
fix(config): make notify_keyspace_events configuration correct by @vyavdoshenko in #8042
chore: Make ListNodeTieringTest backing file path pid-unique by @mkaruza in #8040
build(deps): update packaging requirement from >=26.2 to >=26.3 in /tests/dragonfly by @dependabot[bot] in #8045
perf(build): use minimal debug info in CI by @glevkovich in #8049
build(deps): bump the actions group across 1 directory with 5 updates by @dependabot[bot] in #7910
fix(tiering): Generic upload routine by @dranikpg in #8004
fix: add validation hop for multi-stream blocking XREADGROUP by @BorysTheDev in #8047
fix(test): Stabilize flaky SerializerBaseTest.DelayedEvicted by @mkaruza in #8048
fix(stream): Only create stream group when key is missing and MKSTREAM is set by @mkaruza in #8057
fix(tiering): Attribute list-node offload/load bytes to the right slot by @mkaruza in #8046
refactor: improve stream tests by @BorysTheDev in #8059
fix(metrics): always emit expired_keys_total and evicted_keys_total, including zero by @PixiBixi in #8052
fix(server): Fix memory accounting order on SET+tiered storage by @abhijat in #8050
tests: Fix some errors seen on valkey search path by @abhijat in #8060
fix(tls): atomic context switch and correct tls_bytes accounting by @vyavdoshenko in #8043
fix(tiering): preserve offloaded hashes during serialization by @vyavdoshenko in #8026
fix(stream): Reply with RESP3 map for XREAD BLOCK by @mkaruza in #8064
chore(server): Prefetch + cid caching for speedup by @dranikpg in #8033
fix(search): reject FT.CREATE missing the SCHEMA keyword by @Shikha-code36 in #8054
chore: remove dead test_replication_onmove_flow by @kostasrim in #8078
fix(server): emit expired keyspace events for already-past expirations by @vyavdoshenko in #8065
fix: SET with a past expiration honors NX/XX/GET and journals the delete by @vyavdoshenko in #8082
fix: test_cron_snapshot_failed_saving by @BorysTheDev in #8083
fix(tests): Sample lag continuously in test_replication_info by @mkaruza in #8084
fix(tests): extend pause wait (Fixes #8079) by @glevkovich in #8093
fix(facade): Handle RESP3 null array response by @mkaruza in #8085
fix(tests): fix test_pubsub_pipeline_starvation test by @glevkovich in #8094
fix(zset): honor GT/LT flags for skiplist-encoded sorted sets by @Shikha-code36 in #8091
fix: reply with a null array for empty ZMPOP/BZMPOP/LMPOP/BLMPOP by @vyavdoshenko in #8092
fix(ci): run cache reaper hourly by @glevkovich in #8097
test: un-skip graceful shutdown test by asserting the guaranteed durability contract by @vyavdoshenko in #8086
fix: null reply shapes for blocking moves, LPOP COUNT, ZRANK WITHSCORE, EXEC and friends by @vyavdoshenko in #8098
fix(chart): update vulnerable Go modules by @glevkovich in #8100
fix(ci): check out default branch for cache-reaper by @glevkovich in #8101
fix(tiering): tight small bin fragmentation cutoff by @kostasrim in #8081
fix(server): lower table growth margin default by @romange in #8080
feat: add time limitation for replication backlog by @BorysTheDev in #8039
fix(pubsub): preserve V2 message ordering by @glevkovich in #8075
fix: reply NOSCRIPT for a sha that only has SCRIPT FLAGS set by @vyavdoshenko in #8108
server: Add warning to defragment commands about time based CPU use by @abhijat in #8107
fix(bloom): reject SBF filters the rdb loader would refuse to load back by @vyavdoshenko in #8113
fix: release the global shard lock in the mode it was taken by @vyavdoshenko in #8111
fix: handle bad_alloc in squashed MULTI callback and surface hop status by @vyavdoshenko in #8114
fix(facade): log V2 traffic before dispatch by @glevkovich in #8096
fix: test_slot_migration_oom_replica_rollback by @kostasrim in #8117
test(facade): exclude debug traffic tests on epoll by @glevkovich in #8124
chore: split python cluster tests by @kostasrim in #8118
fix: apple clang16 crash on FloatToBf16 by @kostasrim in #8123
fix(cluster): delete slots when OOM is discovered via ACK by @kostasrim in #8127
chore: disable test_debug_traffic_v2_parse_in_proactor_does_not_preempt by @kostasrim in #8133
fix(tests): use FLUSHALL SYNC in test_rss_oom_ratio to stop RSS-drop … by @Shikha-code36 in #8135
fix: validate COUNT in ZMPOP/BZMPOP/LMPOP/BLMPOP and fix large-count overflow by @vyavdoshenko in #8130
refactor: improve OAHTable::GetRandomMember by @BorysTheDev in #8134
fix(traffic): preserve V2 parse-time logging (fixes #8120) by @glevkovich in #8139
server: Fix rss peak metric by @abhijat in #8143
server: Fix stream size memory accounting by @abhijat in #8066
fix: re-enable test_big_huge_streaming_restart by @kostasrim in #8141
chore(server): Release serializer buffer capacity over limit by @abhijat in #8140
feat(server): throttle defrag task CPU duty cycle by @kostasrim in #8115
fix: don't abort on in-flight OnChange serialization when unregistering snapshot listener by @vyavdoshenko in #8146
docs(readme): add Thai translation by @marttp in #8138
fix(server): Bound DeleteExpiredStep traversal with TraverseBySegment… by @dranikpg in #8129
fix: release delayed-entry bucket latches on tiered read error and snapshot cancellation by @vyavdoshenko in #8150
docs: add container pull count badges to the READMEs by @romange in #8154
fix(acl): script context acl and enable flaky tests by @kostasrim in #8116
feat(facade): share the RESP V2 read buffer by @glevkovich in #8145
test(fuzz): repair seed corpus so every seed executes and harden coverage guard by @vyavdoshenko in #8099
tests: Change assertion to eventual instead of count-based by @abhijat in #8156
fix(tiering): Use TraverseBySegmentOrder and export more metrics/configs by @dranikpg in #8077
tests: enable fixed Valkey stream TCL tests and add the pubsub suite by @vyavdoshenko in #8159
fix: test_big_huge_streaming_restart by @kostasrim in #8160
fix: OAH benchmark allocator init by @BorysTheDev in #8161
tests: Skip cache reclamation for v2 io loop by @abhijat in #8162
fix(search): reject FT.CREATE NUMERIC BLOCKSIZE 0 (divide-by-zero) by @vyavdoshenko in #8164
fix(stream): Validate all streams before mutating group state in XREADGROUP by @mkaruza in #8158
fix(stream): XPENDING with an unknown consumer returns an empty list by @vyavdoshenko in #8168
chore: add debug logs for test_replicaof_reject_on_load by @kostasrim in #8170
fix(list): set plain-node size from value length in QList::Replace by @vyavdoshenko in #8172
fix(tests): wait for loading to finish before first write in test_shu… by @Shikha-code36 in #8174
server: Initialize cmd family during clone for alias by @abhijat in #8175
fix(rdb): keep consumer-group entries_read when loading v2/v3 streams by @vyavdoshenko in #8176
fix(set): do not abort SRANDMEMBER on a zero-member set by @vyavdoshenko in #8177
fix(geo): COUNT without ANY returns the N closest members by @vyavdoshenko in #8178
fix(rdb): reject zero-member SET/HASH on RESTORE and skip empty keys on load by @vyavdoshenko in #8179
fix(generic): TTL of a logically expired key returns -2 instead of aborting by @vyavdoshenko in #8181
feat: improve ascii pack unpack operations performance by @BorysTheDev in #8169
fix(rdb): reject odd-entry HASH/ZSET listpack on RESTORE by @vyavdoshenko in #8185
fix(stream): Report XREADGROUP errors in argument order, not shard order by @mkaruza in #8186
fix(replication): bound RESP client parser against unbounded allocation by @vyavdoshenko in #8187
fix(server): keep the .rdb extension in the last-save path by @vyavdoshenko in #8184
fix(facade): reclaim V2 pipeline cache by @glevkovich in #8192
fix(server): register SORT's STORE destination wherever the option appears by @vyavdoshenko in #8191
chore: disable epoll test_replicaof_reject_on_load by @kostasrim in #8194
feat(geo): add GEOSEARCHSTORE command by @rounaknandanwar in #7984
fix(server): Fix journal omit breaking on stash bucket by @dranikpg in #8157
fix(generic): SORT STORE with a missing source deletes the destination and replies 0 by @vyavdoshenko in #8190
feat(acl): Add JWT validation as an alternative to password AUTH by @tjovicic in #8171
refactor: remove redis_parser from rdb_load_context by @BorysTheDev in #8198
refactor: improve OAHMap performance by @BorysTheDev in #8201
test(acl): retry GEOSEARCHSTORE while the ACL grant propagates by @vyavdoshenko in #8203
chore(server): remove DEBUG COMPRESSION, HuffmanCheckTask, and deprecate --huffman_table by @romange in #8205
fix: FT.INFO missing-index compatibility by @denis-samatov in #8199
core: Fix small tracker bugs by @abhijat in #8207
fix(generic): require every EXPIRE flag to hold instead of any one of them by @vyavdoshenko in #8189
fix(sort): stop SORT LIMIT from overflowing the entries range by @vyavdoshenko in #8212
fix(facade): cap inline requests at 64KB and stop eager bulk-length allocation by @vyavdoshenko in #8215
fix(facade): Reject unbounded memcached keys by @dranikpg in #8213
fix(generic): stop RM from freeing a container under a yielding iteration by @vyavdoshenko in #8210
feat(ci): add manual regression tests controls by @glevkovich in #8208
fix(debug): do not flush when the snapshot cannot be loaded by @vyavdoshenko in #8216
fix(facade): limit queue wakeup to V1 in EnqueueParsedCommand by @glevkovich in #8223
fix(rdb): bound RESTORE allocations to the remaining input to prevent a remote OOM by @vyavdoshenko in #8222
perf(facade): batch parser memory refreshes by @glevkovich in #8225
fix(stream): stop XREAD/XREADGROUP inside a script from aborting the server by @vyavdoshenko in #8228
perf(facade): skip empty reply batch setup by @glevkovich in #8224
fix: restore shard_repl_backlog_len for compatibility by @BorysTheDev in #8209
fix: compatibility RDB_TYPE_SET_LISTPACK with RedisShake by @BorysTheDev in #8227
fix(cluster): don't crash when migration finalize pause times out by @Shikha-code36 in #8240
fix(protocol): detect RESP commands after empty inline lines by @romange in #8244
fix(server): make connection memory accounting constant time by @romange in #8245
fix(ci): disable GoogleTests by default by @glevkovich in #8229
fix(deb): Change export syntax for DEB_BUILD_OPTIONS by @moredure in #8214
fix(generic): run RM as a no-key transaction by @vyavdoshenko in #8218
test(acl): cover expired JWT authentication by @tjovicic in #8221
fix(server): default SLOWLOG GET to 10 entries by @cuishuang in #8248
feat: add rdb valkey support by @BorysTheDev in #8251
chore: add docs/replication.md by @kostasrim in #7997
chore: log error if migration does not finalize by @kostasrim in #8258
server: Introduce cmd memory scope by @abhijat in #8151
fix(set): restore StringSet as the default SET implementation for 2.0 by @vyavdoshenko in #8257
perf(facade): simplify connection memory refreshes by @romange in #8246
fix(rdb): reject a hash with duplicate fields on RESTORE by @vyavdoshenko in #8260
server: Turn off scope based memory tracking by @abhijat in #8263
fix(server): free hll sds when PFADD rejects a corrupt sparse HLL by @foobar in #8262
fix: allocate sorted map pointers on data heap by @kostasrim in #8256
fix: replace broken awesome-prometheus-alerts link by @samber in #8283
fix(script): restore the caller's database after a script that runs SELECT by @vyavdoshenko in #8280
fix(replication): unblock blocked clients when the node becomes a replica by @vyavdoshenko in #8284
fix(multi): keep collecting after a queue-time error so EXEC aborts the whole transaction by @vyavdoshenko in #8281
ci: enable Epoll regression tests with RESP V2 by @glevkovich in #8265
chore(tests): cleanup and simplify code, use helpers where possible by @abhijat in #8294
fix(server): correct CL.THROTTLE rounding and batch its reply by @jmealo in #8289

Full Changelog: v1.40.0...v2.0.0

dragonflydb/dragonfly v2.0.0 on GitHub