Dragonfly v1.15.0

After more than a month, we are releasing a new version with even more efficient transactional algorithms, improved lua compatibility, better memory management and lots of stability fixes. For Mac enthusiasts - the project now builds natively on MacOs 💻

Some prominent changes include:

• Lua redis.log(..) is recognized (#2476)
• Lua can optionally return doubles as longs when replying to RESP2 clients using --lua_resp2_legacy_float flag (#2672)
• Added backlog option (#2456)

What's Changed

• chore: improve benchmark for bpptree and zsl tree by @romange in #2450
• feat(transaction): Independent out of order execution by @dranikpg in #2426
• fix(RestoreStreamer): Fix a few glitches by @chakaz in #2452
• feat(cluster): add tx execution in cluster_shard_migration by @BorysTheDev in #2385
• feat(server): Add backlog flag for listen() by @chakaz in #2456
• feat: allow reading offloaded strings without loading to the store by @romange in #2449
• refactor: return OpResult in DbSlice::AddOrFind instead of throwing std::bad_alloc by @kostasrim in #2427
• fix(transaction): Improve ACTIVE flags management by @dranikpg in #2458
• test(cluster): Enable seeder to work against a Dragonfly cluster by @chakaz in #2462
• Update README.md to include dragonfly cloud link by @worldsoup in #2469
• fix: resolves the crash with MGET and duplicate keys by @romange in #2471
• fix: fixes for v1.14.0 by @dranikpg in #2473
• feat(lua): add no-op redis.log command by @kostasrim in #2476
• fix: reduce test_pipeline_batching_while_migrating flakiness by @dranikpg in #2475
• chore: switch json object to pmr allocator by @romange in #2483
• chore(transaction): Simplify PollExecution() by @dranikpg in #2457
• chore: fix clang warnings by @romange in #2485
• chore(transaction): Launder copied keys in multi transactions by @dranikpg in #2478
• chore: lock keys when going through fast-path execution by @romange in #2491
• fix(server): mget crash on same key get by @adiholden in #2474
• chore: Introduce LockKey for LockTable by @romange in #2463
• fix(cluster): fix slot filtration to RestoreStreamer by @BorysTheDev in #2477
• feat(lua): add missing no-op redis.replicate_commands() by @kostasrim in #2472
• chore: remove batch reply statistics by @romange in #2490
• Fix redis->DF replication tests by @romange in #2495
• chore: introduce TaskQueue for executing shard local tasks by @romange in #2498
• fix(cluster): fix incorrect version checking and resource double free by @BorysTheDev in #2499
• fix(server): Fix crash in RENAME by @chakaz in #2503
• test: Disable flaky test test_policy_based_eviction_propagation by @chakaz in #2505
• feat(server): Memory tracker by @chakaz in #2501
• fix(server): update post updater iterator in tiering by @adiholden in #2497
• chore(transaction): Use PhasedBarrier for easier synchronization by @dranikpg in #2455
• feat(server): Implement SCRIPT FLUSH command by @highpon in #2493
• fix: do not crash with inconsistent watch queue by @romange in #2515
• Fix: awakened_keys inconcistency by @romange in #2517
• feat(cluster): add migration finalization by @BorysTheDev in #2507
• fix: fix BLOCKING/REVERSE_MAPPING flags for some commands by @romange in #2516
• fix: handle null length strings in stored commands by @dranikpg in #2518
• feat(server): Add MEMORY TRACK ADDRESS command by @chakaz in #2513
• feat(server): Implement CONFIG HELP command by @highpon in #2510
• fix(transaction): Add special barrier for blocking tx by @dranikpg in #2512
• fix(zset): ignore cancelled op status in BZPopMinMax by @andydunstall in #2527
• fix: fix script replication by @dranikpg in #2531
• chore: remove redis sorted set implementation by @romange in #2522
• fix: add emoji when regression tests timeout by @kostasrim in #2543
• fix(cluster): fix #2532 tests redis client close errors by @BorysTheDev in #2542
• bug: DFLY_PASSWORD environment variable deprecation by @mprimeaux in #2539
• fix: wrongly printing timedout emoji on test failures by @kostasrim in #2546
• feat: move lint test chart to seperate action and trigger it on reg tests by @kostasrim in #2536
• Update helio by @romange in #2538
• fix(server): Do not yield in journal if not allowed by @chakaz in #2540
• fix: fix interpreter acquisition with MULTI by @dranikpg in #2549
• fix(tests): increase interpreter test load, lower notice limit, include 2553 by @dranikpg in #2554
• feat: introduce a skeleton for jsonpath parser by @romange in #2521
• fix(pytest): Disable flaky interpreter test monitoring by @dranikpg in #2558
• feat(cluster): add slots blocking when we finalize migration by @BorysTheDev in #2484
• fix(server): Init tx time for all multi/lua transactions by @chakaz in #2562
• chore: enable ContendExpire by @romange in #2563
• fix(server): do not print warning on replica stop/ takeover by @adiholden in #2560
• chore: parse a simple jsonpath by @romange in #2566
• feat(pytest): Gen2 seeder, part 1 by @dranikpg in #2556
• fix: make sure SCRIPT FLUSH concludes by @romange in #2565
• chore: Remove manual memory management from stats by @dranikpg in #2550
• fix: asan errors on unit tests by @kostasrim in #2564
• chore(transaction): Simplify armed state by @dranikpg in #2508
• test(memory): Unaccounted memory test + add DEBUG POPULATE TYPE <type> by @chakaz in #2561
• chore: Recognize exit status in regression tests by @romange in #2571
• chore(server): replica not accumulate multi commands untill exec by @adiholden in #2557
• chore: use helio MPSCIntrusiveQueue queue by @romange in #2573
• chore: provide plumbing for jsonpath error propagation by @romange in #2567
• test(cluster-migration): Fix some bugs and add cluster migration fuzzy tests by @chakaz in #2572
• fix: do not migrate during connection close by @romange in #2570
• chore: fix fiber types in the codebase by @romange in #2574
• feat(release): Also bundle dragonfly-debug builds by @Pothulapati in #2424
• feat: jsonpath supports index/wildcard expressions by @romange in #2578
• fix(tiering tests): introduce wait until tiering entries num EQ/GT by @adiholden in #2559
• chore(pytest): Refactor snapshot test by @dranikpg in #2583
• fix(test): Apply pytest filter to regression tests as well by @chakaz in #2589
• feat: add descent operator to jsonpath parser by @romange in #2588
• test(cluster): Fix and uncomment counter sanity checks by @chakaz in #2591
• feat(tiering): add background offload step by @adiholden in #2504
• fix(test): Unflake GenericFamilyTest.Time*Keys by @chakaz in #2592
• chore: add jsonpath evaluate by @romange in #2586
• fix(test): Use less memory for STRING and HASH memory tests by @chakaz in #2593
• chore(transaction): Copy poll flags by @dranikpg in #2596
• feat: support descent path operator by @romange in #2594
• feat: Add dbfilename to config registry by @romange in #2598
• wire json::Path into json_family by @romange in #2587
• chore: refactor DfsItem to allow mutability semantics by @romange in #2602
• chore: cleanup of redis implementation of sets/hashmaps by @romange in #2605
• chore: remove ImportRObj routine and reduce reliance on object.c by @romange in #2607
• chore: consolidate json code under core/json by @romange in #2609
• chore(transaction): Untie scheduling from multi status by @dranikpg in #2590
• chore(transaction): Add debug stats for fail printing by @dranikpg in #2600
• Add mutate method to jsonpath by @romange in #2606
• add loadtest benchmarking program by @romange in #2603
• chore: get rid of object.c and robj* in cc code by @romange in #2610
• chore: clean up redis code by @romange in #2614
• chore: allow aggregate functions for jsonpath by @romange in #2611
• fix: fix cluster test by @dranikpg in #2612
• fix(rdb_load): EnsureRead(min) requesting more bytes than min by @kostasrim in #2604
• fix(server): saving is not a server state by @adiholden in #2613
• fix(server_family): GetMetrics should show commands in lowercase by @lsvmello in #2601
• feat(cluster): #2448 add new DFLYMIGRATE ACK cmd by @BorysTheDev in #2582
• test: Run slow tests in regression runs (again:) by @chakaz in #2616
• fix(bug): crash when starting replica while saving by @adiholden in #2618
• test: Run slow tests in regression runs (again^2) by @chakaz in #2623
• feat(cluster_mgr): Add support for migrate action by @chakaz in #2626
• feat(pytest): More types for seeder by @dranikpg in #2577
• fix: loading state error type to be compatible with redis by @kostasrim in #2629
• fix(cluster_mgr): Disable CPU affinity by @chakaz in #2632
• fix: provide more context in case we have duplicate keys by @romange in #2619
• chore: doc_accessors now parse with json::Path as well by @romange in #2615
• fix: crash when using nonexistent global variable in lua by @thweetkomputer in #2627
• feat(regTests): upload only failed test logs on ci and clean up logging by @kostasrim in #2547
• refactor: remove goto statements from replica.cc by @kostasrim in #2633
• fix(bug): server crash info memory while saving by @adiholden in #2637
• fix(streamer): Do not yield from the Traverse callback. by @chakaz in #2638
• chore: fix build for MacOs by @romange in #2635
• fix: avoid crashing during the connection traversal by @romange in #2651
• a test with flat buffers by @romange in #2520
• refactor: create one type for slots set #2459 by @BorysTheDev in #2645
• fix: data race in save_stages_controller by @kostasrim in #2647
• fix: fix zmalloc_size on macos by @andydunstall in #2646
• feat(connection): Support pipelining with Memcached by @dranikpg in #2648
• chore: JSON.GET now works with our own jsonpath by @romange in #2653
• chore: fully switch to dual parsing with json::Path by @romange in #2654
• chore: Move CallFromScript log to level 2 by @dranikpg in #2656
• chore: switch to self-built flatbuffers by @romange in #2657
• feat: add current_fork_perc in info all command by @kostasrim in #2640
• chore: Del and NUMINCRBY use json::Path by @romange in #2655
• chore: move jsoncons path code into dedicated files by @romange in #2666
• fix: replace localhost with 127.0.0.1 in cluster tests by @dranikpg in #2665
• chore: fuly cover json_family API with json::Path parsing by @romange in #2663
• feat(generic_family): implement RANDOMKEY command by @lsvmello in #2639
• fix(server): client pause work while blocking commands run by @adiholden in #2584
• feat(cluster_mgr): Add support for remote Dragonfly servers by @chakaz in #2671
• fix: optionally return float as integer from lua script by @romange in #2672
• refactor(cluster): #2652 initiate migration process from CONFIG cmd by @BorysTheDev in #2667
• chore: update helio dependency by @romange in #2674
• fix(snapshot): Fix flag access by @dranikpg in #2676
• chore: more fixes for macos by @romange in #2677
• feat(server): introduce oom_capacity_ratio flag by @adiholden in #2678
• chore: add oom stats to /metrics by @romange in #2680

New Contributors

• @mprimeaux made their first contribution in #2539
• @thweetkomputer made their first contribution in #2627

Huge thanks to all the contributors! ❤️

🇮🇱 🇺🇦

Full Changelog: v1.14.0...v1.15.0