Dragonfly v1.37.0

Important stability fixes and changes:

• Significant memory reduction around json documents #6511
• Vector search memory usage is 40% less with hash map documents: #6436 #6437
• Support of SORT command - including BY and GET options.
• Several bug fixes around streams: #6506 #6492 #6532
• Added support for DIGEST and DELEX commands: #6328

What's Changed

• feat(search): Use stateless allocator for strings by @abhijat in #6285
• chore: Refactor squashed command execution to avoid stub ConnectionContext by @romange in #6312
• refactor: add checks to investigate the race condition in slots migration by @BorysTheDev in #6331
• fix(helm): correct security context field placement and disable strict schema validation by @Copilot in #6213
• fix(tests): Use promtool for metrics sanity check by @abhijat in #6318
• fix(search): support parenthesized star in FT.SEARCH queries by @vyavdoshenko in #6343
• fix: helm push issue by @romange in #6339
• chore: move ReplyBuilder calls in generic_family.cc to new API by @romange in #6305
• Add checks for expiration inconsistency by @BorysTheDev in #6346
• feat(hscan): add NOVALUES flag to omit values by @glevkovich in #6325
• chore: refactor SortGeneric function by @romange in #6347
• feat: add pagination to cross-shard search by @BorysTheDev in #6352
• refactor: address comments by @BorysTheDev in #6350
• fix(set): SPOP random member even if it contains null bytes by @mkaruza in #6361
• fix(server): Set error type when returning -NOGROUP error by @mkaruza in #6358
• chore: improvements around sorting algorithm by @romange in #6360
• facade: adopt RespSrvParser in server connections by @romange in #6356
• feat(server): Support squashing of global eval with single shard by @dranikpg in #6321
• fix: memory calculation for OAHSet by @BorysTheDev in #6359
• fix: regtest failure by @romange in #6367
• test: use hiredis for reply_builder_test by @BorysTheDev in #6362
• ci: use OIDC for AWS authentication instead of access keys by @AdiCloudbit-DevOps in #6158
• refactor: improve OAHSet::Find() performance by @BorysTheDev in #6373
• chore(server): Simplify InvokeCmd by @dranikpg in #6363
• chore(cmake): Update jsoncons to 1.5.0 by @abhijat in #6349
• chore: encapsulate tx object within CommandContext by @romange in #6384
• fix(server): support "sort by" option by @romange in #6370
• refactor(acl): from facade to server by @kostasrim in #6340
• chore: add copilot env setup steps by @romange in #6387
• fix(ci): add AWS env fallbacks for OIDC in composite actions by @AdiCloudbit-DevOps in #6385
• chore: move CommandContext cid to private scope by @romange in #6386
• chore(server): Updates around DispatchMC/DispatchCommand by @dranikpg in #6351
• fix: compatibility with cpp20 by @BorysTheDev in #6395
• chore: Update CommandContext c'tor by @romange in #6393
• feat(server): Add GET option support to SORT command by @romange in #6388
• chore: update AGENTS.md by @romange in #6392
• Switch to C++20 by @Copilot in #6379
• chore: Clean up DispatchCommand a bit more. by @romange in #6397
• fix(core): Remove direct access to RObjWrapper by @dranikpg in #6140
• fix: copilot-setup-steps.yml by @romange in #6405
• Implement DIGEST and DELEX commands (Redis 8.4.0 compatible) by @Copilot in #6328
• chore: tx clean-ups by @romange in #6407
• chore: remove quicklist from the codebase by @romange in #6415
• chore(core): CompactObj is_key field, separate encoding by @dranikpg in #6253
• chore: wrap list operations within ListWrapper by @romange in #6417
• fix: s3 snapshot saving by @BorysTheDev in #6400
• chore(facade): Refactor IoLoop to new interface by @dranikpg in #6383
• chore: clarify IterateList interface by @romange in #6425
• chore: show slave_repl_offset on replicas when master link is down by @kostasrim in #6353
• fix: reloading vector index by @BorysTheDev in #6422
• chore(search): Simplify SFINAE usage with concepts in defragment-map by @abhijat in #6428
• fix: static assert on macos by @vyavdoshenko in #6430
• docs: add async-tiering.md by @romange in #6427
• chore(core): Move expire flag to key by @dranikpg in #6420
• chore: add ListPack class that wraps c listpack by @romange in #6432
• chore: factor out ContainerEntry and QList::Entry into a common class by @romange in #6426
• refactor: InvokeCmd by @kostasrim in #6399
• fix: partial sync from same source master with different shard sizes by @kostasrim in #6381
• feat: add hnsw metadata into index serialization by @BorysTheDev in #6429
• fix: QList::iterator semantics by @romange in #6439
• tests: Check if tx is initialized before accessing its state by @abhijat in #6446
• chore: support listpack with lists by @romange in #6424
• Use async tx for ioloopv2/set by @dranikpg in #6433
• feat(server): add protected mode to bind localhost when no password is set by @rafaelperoco in #6442
• tests: Replace pytest with gh workflow by @abhijat in #6453
• CI: remove faulty condition by @abhijat in #6458
• chore(server): Try to simplify build by @dranikpg in #6455
• chore: create smalls lists as listpacks by @romange in #6450
• chore(tiering): polish type system for better clarity and safety by @romange in #6465
• chore: unify auth errors under the same type by @romange in #6449
• fix(server): ignore SIGHUP to prevent termination on session closing by @boomballa in #6463
• chore(tiering): refactor stashing to determine serialization upfront by @romange in #6470
• Add index serialization by @BorysTheDev in #6461
• docs: add code review instructions for Copilot agents by @romange in #6471
• feat(hnswlib): Support using pointer to vector data by @mkaruza in #6436
• chore(tiering): refactor Read/Modify to free functions by @romange in #6469
• design: stateful async handlers by @dranikpg in #6462
• chore: better naming for compression functions in qlist.cc by @romange in #6474
• feat(search): Build vector index by using data pointer for HSET objects by @mkaruza in #6437
• refactor(facade): decouple pipeline traffic from administrative command queue (phase 1) by @glevkovich in #6423
• chore: bump helio to fix TLS socket crash by @glevkovich in #6480
• feat(search): Omit key defrag is vector data is used as reference by @mkaruza in #6454
• chore: sync rax and stream code with Valkey by @romange in #6473
• feat(cluster): add range syntax support to DFLYCLUSTER GETSLOTINFO by @vyavdoshenko in #6485
• chore: add tiering params and stats to qlist by @romange in #6476
• feat(json): Adds a ref-counted blob by @abhijat in #6456
• fix(server): Fix concurrent access to now_streamed_ by @dranikpg in #6486
• chore: add more invariant checks for streams (#6459) by @romange in #6488
• fix(search): GEO units shouldn't be tokens by @mkaruza in #6489
• fix: prevent connected_clients counter leak on replica connection migration by @vyavdoshenko in #6491
• feat(json): Adds a reference counted string class by @abhijat in #6497
• fix(tiering): implement qlist symmetric cooloff and fix memory handling by @romange in #6496
• chore: Allow JustReply() in cmd-support by @dranikpg in #6478
• fix(server): Async index builder, part 1 by @dranikpg in #6504
• feat: add shrink to OAHSet by @BorysTheDev in #6513
• feat: send index definition for every thread by @BorysTheDev in #6505
• fix(cmd): Ensure arg slice lifetime by @dranikpg in #6514
• fix(hset): delete key if empty after HEXPIRE by @glevkovich in #6503
• chore: Add MinIO support for S3 snapshot tests by @vyavdoshenko in #6501
• fix: xreadgroup is a write command by @kostasrim in #6492
• fix: add missing memory tracking on streams by @kostasrim in #6506
• fix: xreadgroup read_history should not stick by @kostasrim in #6523
• feat(search): Async index building (part 2) by @dranikpg in #6517
• refactor(facade): separate redis data/control paths (phase 2, final) by @glevkovich in #6477
• feat: add serialization for shard DocId mapping by @BorysTheDev in #6519
• refactor: simplify OAHSet code by @BorysTheDev in #6522
• fix: macOS build by @vyavdoshenko in #6526
• feat(hnswlib): Use __builtin_prefetch by @mkaruza in #6500
• fix: QUIT pipeline flush by @vyavdoshenko in #6518
• fix: allow lazy expiry during global EVAL transactions by @vyavdoshenko in #6524
• tests: filter up BullMQ failed tests. by @vyavdoshenko in #6533
• fix(tiering): Handle string that are encoded as LONG LONG objects by @mkaruza in #6530
• feat(core/search): Range Tree builder by @dranikpg in #6520
• fix: always replicate new consumers in xreadgroup by @kostasrim in #6532
• fix(facade): fix migration stats accounting crash by @glevkovich in #6538
• fix(search): Fix search test after async index construction by @dranikpg in #6541
• feat(json): Use reference counted strings for json keys by @abhijat in #6511
• fix: search and dfly lib circular dep by @kostasrim in #6543
• fix(search): Avoid second traversal without vector fields by @dranikpg in #6539
• chore: reduce includes by @dranikpg in #6464
• fix(search): Handle missing fields with GeoIndex by @mkaruza in #6550
• feat(server): Plug in RangeTree::Builder by @dranikpg in #6537
• fix: hnsw global serialization and add flag for hnsw save\load by @BorysTheDev in #6553
• json: Add test for pool shrinking by @abhijat in #6544
• Remove redundant includes to reduce compilation time by @Copilot in #6548
• fix(search): fix more tests by @dranikpg in #6547
• test: search with replication by @BorysTheDev in #6563
• feat(hnsw): Use cosine distance by @mkaruza in #6559
• chore: supress maint_notifications client errors by @romange in #6569
• Reduce compilation time by removing unnecessary header includes by @Copilot in #6562
• Migrate INCR/DECR/INCRBY/DECRBY command family to AsyncCmd architecture by @glevkovich in #6568
• feat(search): Handle GEO index fields by @mkaruza in #6566
• fix(search): Missing vector field should not be zero vector by @mkaruza in #6565
• refactor(string): migrate SET, APPEND, PREPEND to async SimpleContext by @glevkovich in #6580
• core/server: Compute memory size of json object using jsoncons API by @abhijat in #6572
• fix: validate interleaved-keys argument count by stored step by @vyavdoshenko in #6574
• build: add optional ASAN support for release builds by @vyavdoshenko in #6590
• refactor: store hnsw metadata in separate AUX field by @BorysTheDev in #6587
• chore(server): Move vector index construction to builder by @dranikpg in #6571
• chore: get rid of deprecated DFLY_PASSWORD env checks by @romange in #6586
• fix: reject GAT command sent via Redis protocol instead of crashing by @vyavdoshenko in #6592
• fix: handle MEMORY USAGE without key argument instead of crashing by @vyavdoshenko in #6597
• server: Change metric type to counter to match redis exporter by @abhijat in #6581
• fix: reject SCRIPT FLAGS with invalid SHA length instead of crashing by @vyavdoshenko in #6603
• feat(server): add CLIENT INFO command by @romange in #6576
• fix(tls): ub in HandleRequests after first socket read by @kostasrim in #6594
• Consolidate SlowLog tests in server_family_test by @Copilot in #6585
• Refactor server/common.h - Phase 1: Extract common types and statistics by @Copilot in #6582
• test: wait for async indexing completion in test_replicate_all_index_types by @Copilot in #6593
• fix(hnsw): Avoid possible HSET object memory misalignment by @mkaruza in #6577
• chore: simplify snapshotting serialization callback by @romange in #6595
• server: Record values in histogram atomically by @abhijat in #6583
• Refactor server/common.h - Phase 2: Extract execution state types by @Copilot in #6605
• Extract RespExprBuilder to shared test utils and migrate test_utils to RESPParser with RESP3 support by @Copilot in #6588
• fix: mimalloc page reclamation is too slow by @romange in #6575
• feat: add HNSW index deserialization by @BorysTheDev in #6606
• Refactor server/common.h - Phase 3: Extract synchronization primitives by @Copilot in #6612
• Refactor bitops/stream updater storage by @romange in #6613
• fix: make ChangeMonitor idempotent to prevent double-add in MULTI/EXEC by @vyavdoshenko in #6615
• test: Improve AFL++ fuzzing infrastructure by @vyavdoshenko in #6618
• fix: crash on EVALSHA with empty sha by @vyavdoshenko in #6620
• fix: string corruption during rdb load by @romange in #6621
• make migration more stable by @BorysTheDev in #6616
• chore(regtests): replication missmatching dbnum by @ashotland in #6607
• fix(server): fix OpenSSL shutdown race with mimalloc by @glevkovich in #6626
• fix(fuzz): fix false-green CI and simplify fuzzing workflow by @vyavdoshenko in #6632
• fix(server): guard ReadProcStats with linux for macOS compatibility by @vyavdoshenko in #6633
• fix: do not overrun stack for asan enabled builds by @romange in #6635
• fix(tiering): Fix replication serialization order by @dranikpg in #6610
• fix(memcache): reject gat/gats without key instead of crashing by @vyavdoshenko in #6639
• feat(server): Extensive slowlog stats for exec/eval by @dranikpg in #6584
• fix(facade): Fix missing DFLY_USE_SSL defines by @dranikpg in #6636
• fix(memcache): cap store value size with max_bulk_len by @vyavdoshenko in #6644
• chore: move simple types to common_types.h and reduce tx_base.h header includes by 52.9% by @Copilot in #6641
• fix(facade): Disable v2 loop for redis by @dranikpg in #6643
• test: Add memcache protocol fuzzing support by @vyavdoshenko in #6638
• fix(rdb): fix chunked SBF loading for filters >= 4 GiB by @vyavdoshenko in #6649
• chore: journal clean-ups by @romange in #6622
• fix: test_redis_om by @BorysTheDev in #6655
• fix(tiering): Reduce load on tiering test by @dranikpg in #6650
• chore: add logs to rdb_load by @kostasrim in #6642
• Load hnsw index by @BorysTheDev in #6647
• fix(facade): use WriteRef for large values in meta get response by @vyavdoshenko in #6657
• refactor: include what you use by @Copilot in #6651
• skip tiering test by @dranikpg in #6658
• fix(server): Wait for index load on replication by @dranikpg in #6645
• refactor: include what you use by @BorysTheDev in #6660
• feat(tiering): SSD tiering support during slot migration by @mkaruza in #6542
• refactor: remove extra includes from headers by @BorysTheDev in #6663
• chore: introduce heavy tests by @Copilot in #6659
• chore: Mark cluster tiering test as heavy by @mkaruza in #6671
• fix: backward compatibility for replication regarding HNSW index seri… by @BorysTheDev in #6664

Huge thanks to all the contributors! ❤️

New Contributors

• @AdiCloudbit-DevOps made their first contribution in #6158
• @rafaelperoco made their first contribution in #6442

Full Changelog: v1.36.0...v1.37.0