Dragonfly v1.32.0

Important stability fixes and changes:

• Lots of performance improvements around Search #5341#5327 #5350 #5545 #5533
• Add optional latency percentile stats #5372 and #5428
• Update mimalloc version to 2.24
• Added BZMPOP and BLMPOP commands #5436 and #5370
• Added CONFIG REWRITE command #5471

What's Changed

• test(search): add unified benchmarks for prefix/suffix/infix search by @vyavdoshenko in #5319
• chore: get rid of quicklist encoding by @romange in #5282
• feat: add flag pause_wait_timeout by @BorysTheDev in #5328
• test: enable test_multidim_knn python test by @vyavdoshenko in #5329
• test: make random determenistic in test_multidim_knn python test by @vyavdoshenko in #5335
• fix: data race when doing a snapshot in cache mode by @romange in #5336
• chore(search_family): Minor performance improvements to GetAllResults methods. FIRST PR by @BagritsevichStepan in #5341
• fix: JSON.DEL to return 0 for non-existing keys instead of error by @vyavdoshenko in #5343
• chore: improve test_full_sync test by @romange in #5340
• chore: Refactor CompactObject encoding by @dranikpg in #5342
• feat: Added Sort_RO Command. by @H4R5H1T-007 in #5332
• chore: avoid deduplicating docs ids when not needed by @romange in #5344
• chore: add hdr_histogram to the project by @romange in #5330
• chore(tiering_test): tune memory settings to be below 6GB by @romange in #5345
• chore: suffix search by @dranikpg in #5327
• test: Vector operations benchmark was added by @vyavdoshenko in #5353
• feat(replication): Aggregate data until threshold before replication by @mkaruza in #5308
• SimSIMD integration for Vector operations by @vyavdoshenko in #5348
• feat (server) : introduce no point in time replication by @adiholden in #5103
• chore: move 3rd party libraries into a separate cmake file by @romange in #5362
• fix(pytest): Increase allowed diff in test_partial_replication_on_same_source_master by @mkaruza in #5363
• fix(replication): reduce ERROR logging during normal shutdown by @vyavdoshenko in #5366
• chore: remove unneeded mimalloc-specific code by @romange in #5367
• chore: add mimalloc2 third_party library by @romange in #5364
• fix(replication): reduce ERROR to WARNING for connection issues during sync by @vyavdoshenko in #5369
• feat(search): Suffix/Infix search for Tags by @dranikpg in #5350
• fix(zset): expand parsing of rank arguments beyond int32 by @romange in #5377
• test: replication_test.py::test_bug_in_json_memory_tracking enabled by @vyavdoshenko in #5375
• chore: Disable test_replication_all cache mode by @mkaruza in #5385
• chore: Make fakeredis part of the core CI checks by @romange in #5380
• fix(zset_family): Return early if range is expected to be zero by @mkaruza in #5386
• fix: JSON.STRLEN behavior for non-existent keys by @vyavdoshenko in #5389
• chore: add psync metrics for replica by @kostasrim in #5365
• fix(server): fix serialization on move with big values by @adiholden in #5390
• feat(pytest): Set and log random seed used in pytest by @mkaruza in #5391
• chore: remove zset/t_zset files by @romange in #5387
• feat: add support for BLMPOP command by @EricHayter in #5370
• feat(pytest): Initialize random seed for each test by @mkaruza in #5395
• fix(values): fix typos in Helm README and values file by @adimoragbloch in #5400
• fix: no set/hmap expiration during serialization by @BorysTheDev in #5349
• feat(search_family): Add SortedVector<pair<DocId, double>> by @BagritsevichStepan in #5405
• feat: add snapshot-id field into dfs files by @BorysTheDev in #5399
• chore: Enable point_in_time_snapshot by default by @mkaruza in #5403
• chore: Increase BLMPopBlocking sleep duration by @mkaruza in #5408
• chore: skip flaky test_pipeline_cache_size by @kostasrim in #5410
• chore: introduce class scope fixtures by @romange in #5409
• fix(pytest): Increase REPLTAKOVER time in test_take_over_seeder by @mkaruza in #5415
• server_family: Expose command latency stats as part of info by @abhijat in #5372
• fix(search): Cut off results before serializing by @dranikpg in #5412
• test: Add Dragonfly search benchmark tests by @vyavdoshenko in #5406
• feat(search_family): Introduce RangeTree by @BagritsevichStepan in #5414
• fix: search_benchmark_utils on python 3.12 by @romange in #5422
• feat(search_family): Add RangeTree to NumericIndex by @BagritsevichStepan in #5416
• chore: switch to mimalloc2 by @romange in #5368
• feat(search_family): Add optimized merging logic for one and two blocks in RangeResult by @BagritsevichStepan in #5417
• chore(range_tree): Simplify RangeTree code by @BagritsevichStepan in #5418
• Feat config management by @boomballa in #5388
• Lua library Integer overflow can cause the DragonFly crash. (CVE-2020-14147) by @KIMDONGYEON00 in #5421
• chore(search): LIMIT=0 Fast path and remove LOAD parameter from FT.SEARCH by @dranikpg in #5419
• server_family: Support resetting latency histogram by @abhijat in #5428
• test: search benchmark updated to use class fixture for multiple tests by @vyavdoshenko in #5425
• chore: harden checks in _mi_page_malloc_zero by @romange in #5427
• fix(pytest): Wait replication to be in sync in test_stream_approximate_trimming by @mkaruza in #5430
• chore: avoid squashing when squashing_current_reply_size crosses limit by @kostasrim in #4924
• fix: acl log missing return after reply by @kostasrim in #5454
• chore: add acl help by @kostasrim in #5456
• chore: allow initializing a separate huffman table for string values by @romange in #5470
• feat(search_family): Introduce SingleBlockRangeResult and TwoBlocksRangeResult by @BagritsevichStepan in #5424
• chore: add tls_caching flags by @kostasrim in #5485
• chore: add more logs during replication by @BorysTheDev in #5489
• chore(set_family): Fast path SPOP with small count by @dranikpg in #5491
• fix: add a sleep to relax slot migration process by @romange in #5478
• chore: add tls_socket accept error metric by @romange in #5492
• chore: add io_read_cnt metric by @romange in #5494
• chore: add tls handshake statistics by @kostasrim in #5493
• feat: add more auxillary strings to the snapshot by @romange in #5411
• chore: log frequency for IsReplySizeOverLimit by @kostasrim in #5501
• fix: adjust BLMPOP command options and implementation for proper journaling by @EricHayter in #5503
• chore: add more metrics by @romange in #5504
• fix: allow eviction of zero-allocation entries in cache mode by @vyavdoshenko in #5500
• server: Make memory defrag blocking, accept optional threshold by @abhijat in #5498
• feat: add debug segments command by @romange in #5505
• fix: prevent idle disconnects for replica flow connections by @romange in #5506
• chore(search): Simplify return field logic by @dranikpg in #5486
• fix: lua_undeclared_keys_shas updated for additional sha by @vyavdoshenko in #5511
• chore: replace size() > 0 with !empty() by @kostasrim in #5499
• chore: add additional runtime configuration knobs for connections by @romange in #5520
• fix: reduce cardinality of the huffman histogram by @romange in #5518
• fix: enable keep alive on replica side of replication connections by @romange in #5519
• chore: flush after serializing big string by @kostasrim in #5401
• refactor: debug populate stream generation by @BorysTheDev in #5407
• chore(search_family): Simplify logic for numeric indexes. FIRST PR by @BagritsevichStepan in #5522
• chore: skip test_replication_onmove_flow by @kostasrim in #5526
• fix: Prevent INFO command deadlock during S3 save operations by @vyavdoshenko in #5528
• chore: measure p99.9 in the replayer by @romange in #5530
• chore: -time-limit option was added to record traffic tool by @vyavdoshenko in #5532
• chore: add p999 metric for realtime tables as well by @romange in #5531
• chore: fix monitor preemption in atomic section by @kostasrim in #5523
• feat(search_family): Add BLOCKSIZE option for FT.CREATE by @BagritsevichStepan in #5423
• fix(search_test): Add kMaxRangeBlockSize to the SeveralNumericAndTagIndexes benchmark by @BagritsevichStepan in #5541
• feat(search_family): Speed up merging for numeric indexes by @BagritsevichStepan in #5533
• chore: lower mem usage of test_big_strings by @kostasrim in #5540
• chore(tiering): Push error handling one more level up by @dranikpg in #5351
• chore: Add cmake build options to exclude code #1 by @dranikpg in #5536
• feat: add RESP3 support for JSON.TYPE and JSON numeric operations by @vyavdoshenko in #5517
• fix(search): Fix nullable sort out of bounds by @dranikpg in #5539
• core,server: collect page usage stats during defragmentation task by @abhijat in #5502
• chore: Add more stats to CLIENT LIST by @dranikpg in #5553
• chore: Refactor JournalReader::ReadString by @mkaruza in #5550
• chore(server): Some memory code updates by @dranikpg in #5379
• fix: FT.CREATE first-query initialization penalty (slowdown) by @vyavdoshenko in #5473
• feat: allow migrating client connections via CLIENT MIGRATE by @romange in #5551
• fix(acl): preemptive TlsSocket::Shutdown in atomic section by @kostasrim in #5525
• feat(config): Implement CONFIG REWRITE command by @boomballa in #5471
• feat(traffic_replay): Add compare option by @BagritsevichStepan in #5555
• feat(cluster): migrate thread on same shard config by @dranikpg in #5554
• chore: use stringmatchlen in GlobMatcher by @kostasrim in #5547
• chore: excessive logging in dragonfly connection by @kostasrim in #5558
• feat(search): Add IP (Inner Product) distance metric support for vector similarity by @vyavdoshenko in #5559
• fix(tiering): Fix bug in TryGrow by @dranikpg in #5563
• fix: Allow CLUSTER commands during LOADING state by @vyavdoshenko in #5566
• server: Check result of hdr_init by @abhijat in #5568
• chore: pull helio by @romange in #5557
• fix: Add enhanced error logging and TCP diagnostics for JournalStreamer disconnects by @vyavdoshenko in #5572
• Feat: Added BZMPOP Command by @H4R5H1T-007 in #5436
• chore: time sensitive sscan by @kostasrim in #5527
• chore: do not shutdown master on takeover in cluster mode by @kostasrim in #5544
• feat: remove crc calculation for migration by @BorysTheDev in #5577
• chore: Don't use point_in_time_replication = False in test_replication_all by @mkaruza in #5574
• fix(server): Make max_memory_flag atomic by @dranikpg in #5576
• fix(tiering): Fix rdb load backpressure by @dranikpg in #5578
• docker: Remove cmake dir from dockerfile which no longer exists, add patches dir by @abhijat in #5575
• feat(tiering): Error propagation to client (Part 3) by @dranikpg in #5573
• chore: pull helio by @dranikpg in #5567
• fix: temporary set ignore_crc=false during serialization for backward compatibility by @BorysTheDev in #5583
• feat(tiering): Error propagation to client (Part 2) by @dranikpg in #5565
• chore: do not enter loading state during partial sync by @kostasrim in #5579
• chore: Refactor CmdSerializer by @mkaruza in #5584
• chore(cluster): Decommit memory only under flag by @dranikpg in #5588
• chore: update helio to fix build by @dranikpg in #5593
• feat: Add LinearSearchMap by @BagritsevichStepan in #5591
• core,server: report defrag metrics by @abhijat in #5560
• fix: streamer.cc build on macos by @vyavdoshenko in #5594
• docs(build): fix missing -D flag in minimal debug build command by @guozhihao-224 in #5600
• chore: update daily builds notifications by @vyavdoshenko in #5599
• feat: introduce a CPU budget parameter to control slot migration pacing by @romange in #5598
• chore: extend DEBUG TOPK/KEYS to show total samples count by @romange in #5605
• feat(search_family): Speed up merging of index results by @BagritsevichStepan in #5545
• fix(tiering): Fix MacOs build by @dranikpg in #5602
• chore: add support for value length distribution sampling by @romange in #5606
• feat: fix lua memory deallocation and lua_mem_gc_threshold=0 disable lua force GC calls by @BorysTheDev in #5587

Huge thanks to all the contributors! ❤️

New Contributors

• @EricHayter made their first contribution in #5370
• @adimoragbloch made their first contribution in #5400
• @boomballa made their first contribution in #5388
• @guozhihao-224 made their first contribution in #5600

Full Changelog: v1.31.0...v1.32.0