Dragonfly v1.11.0

🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱
Our beloved country is currently facing difficult times, but we will win, for hope and progress are on our side.
🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱 🇮🇱

Important features and changes:

• Starting from v1.11.0 we support the BullMQ framework. For more information see BullMQ post and our docs.
• Lots of ACL/Auth fixes. Specifically (IMPORTANT): we changed the default http user to be "default" instead of "user" to be consistent
  with Dragonfly ACL rules. We also removed the requirement for authenticated access to /metrics http page even if ACL are enabled.
• STREAMS and ACL APIs are complete! Please let us know if you encounter any issues.
• Lots of performance improvements for Sorted Sets. Now Dragonfly uses our own b+tree implementation for sorted sets by default instead
  of the old Redis skiplist implementation. We improved some memory and performance bottlenecks with listpack encoding as well.
• Fix logrotate rules for our debian package.

What's Changed

• fix: Add missing encodings to strEncoding by @royjacobson in #1798
• fix: add maxmemory_policy item to info memory section by @romange in #1802
• Update Dockerfile.alpine-dev by @dranikpg in #1803
• chore: add macos daily build by @romange in #1795
• feat(search): sized vectors by @dranikpg in #1788
• fix: alpine weekly pipeline by @romange in #1811
• feat (hset): Support arguments (count, withvalues) in HRANDFIELD by @theyueli in #1804
• Remove ICU library by @dranikpg in #1812
• bug(server): multi cleanup on error by @adiholden in #1813
• chore(AclFamily): Move some of the regressions tests to unit and add by @kostasrim in #1816
• feat(stream): add support for xpending command by @Abhra303 in #1806
• chore(AclFamily): remove unused using decl from acl_family_test by @kostasrim in #1817
• feat(stream): add support for xclaim command by @Abhra303 in #1372
• docs: update README.zh-CN.md by @yancz2000 in #1772
• chore: partially revert adjust batching behavior to reduce network by @kostasrim in #1821
• feat(search): HNSW by @dranikpg in #1799
• fix: possible segfault bug by @romange in #1822
• fix(aws): initialise aws in a proactor by @andydunstall in #1818
• fix: Reply within async_dispatch block in squashed pipeline by @dranikpg in #1825
• Revert "chore: partially revert adjust batching behavior to reduce network" by @dranikpg in #1824
• feat(AclFamily): load/store aclfile by @kostasrim in #1820
• feat(snapshot): add snapshot storage by @andydunstall in #1827
• fix: regression tests for acl family test by @kostasrim in #1826
• fix(regression-tests): correct path for acl files by @kostasrim in #1829
• feat(CI): Configure sccache to work with GCS by @royjacobson in #1815
• Revert "feat(CI): Configure sccache to work with GCS" by @royjacobson in #1833
• skip test_multidim_knn by @adiholden in #1834
• fix(search-parser): fix AND, OR, NOT parsing issue by @Equilibrium2702 in #1838
• feat(snapshot): move load path to snapshot storage by @andydunstall in #1830
• fix(AclFamily): move acl test files to pytest by @kostasrim in #1843
• Add build_rpm script and rpm spec by @romange in #1831
• Support 'databases' alias for CONFIG GET by @romange in #1849
• chore(regression): send chat notificaiton only on main branch by @adiholden in #1847
• Fix a potential crash in OnBreakCb by @romange in #1850
• fix(tls): User friendly logging of OpenSSL errors by @royjacobson in #1851
• opt(lua): Coordinate single-shard Lua evals in remote thread by @chakaz in #1845
• feat(debug): Adds a DEBUG STACKTRACE subcommand by @royjacobson in #1848
• feat(server): Getting flags from the environment variables by @aidarbek in #1810
• feat(snapshot): add support for snapshot recovery from S3 by @andydunstall in #1839
• fix: dfly_main small typo on unix_socket flag by @kostasrim in #1853
• chore: enable search lib on Apple by @romange in #1854
• feat(tests): unset test DFLY env variables by @andydunstall in #1856
• feat(snapshot): fix build on macos by @andydunstall in #1861
• fix(ServerFamily): proper initialization of FileSnapshotStorage when replicaof flag is used by @kostasrim in #1858
• feat(AclFamily): add acl commands by @kostasrim in #1844
• Run unit tests in macos build by @romange in #1859
• fix: fix json.arrappend not allowing passing JSON objects by @UkuLoskit in #1867
• chore: build rpm package for the release by @romange in #1866
• fix(Auth): authenticate with any password for default user by @kostasrim in #1868
• tests: choose open ports randomally by @royjacobson in #1569
• chore: Fix regression test after #1569 by @royjacobson in #1872
• chore: Add a context manager to DflyInstance so we don't forget to close them. by @royjacobson in #1873
• feat(replication): Use a ring buffer with messages to serve replication. by @royjacobson in #1835
• fix: weekly build by @romange in #1871
• feat(AclFamily): add acl log by @kostasrim in #1865
• feat: Span-all no-key transactional commands by @dranikpg in #1864
• fix: Disable Int<> calls in cmd_arg_parser_test for MacOS by @dranikpg in #1880
• feat(AclFamily): add acl users command by @kostasrim in #1882
• feat(AclFamily): add acl cat command by @kostasrim in #1883
• feat(AclFamily): add acl getuser command by @kostasrim in #1884
• feat(AclFamily): add acl dryrun command by @kostasrim in #1894
• fix(regression-tests): increase waiting delay for df startup by @kostasrim in #1893
• fix: improve performance of listpack sorted sets by @romange in #1885
• chore: optimize BPTreeNode::BSearch by @romange in #1891
• chore: Introduce basic deadlock detector for tests by @romange in #1896
• fix: make blocking_controller_test being built on macos by @romange in #1903
• fix: Better error handling in search algorithm by @dranikpg in #1881
• bug(server): zadd wrong insert when non unique members by @adiholden in #1906
• run regression tests on CI by @adiholden in #1874
• fix(server): Flip tcp_nodelay to on by default by @royjacobson in #1910
• bug(server): global command stalls on server load with pipeline mode by @adiholden in #1909
• fix: avoid calling 'sink_->Write()' with an empty buffer by @romange in #1912
• doc(server): Copy important note from transactions doc to the code by @chakaz in #1913
• fix: Correctly persist vector indices by @dranikpg in #1905
• opt(server): Call reserve() with correct argument by @chakaz in #1914
• opt(lua): Avoid separate hops for lock & unlock on single shard execution by @chakaz in #1900
• chore: remove redundant Find calls when inserting into DenseSet by @romange in #1917
• fix: Actually print fibers from all the threads by @romange in #1918
• chore(regtest): Update redis dependency by @dranikpg in #1915
• chore: implement ParseDouble using fast_float by @romange in #1920
• fix(regtests): filter client setinfo from rewrite log by @dranikpg in #1921
• bug(streams): fixing xreadgroup's behavior to be compatible with Redis by @theyueli in #1908
• fix(regests): Fix port detection with gdb by @dranikpg in #1922
• feat(json): support JSON.GET pretty print options by @iko1 in #1832
• feat(search): return scores by @dranikpg in #1870
• chore: Avoid allocating unique_members arrays by @romange in #1911
• fix(dragonfly_test): failing bug207 on build opt by @kostasrim in #1919
• feat(build): fix lua on mac arm64 by @andydunstall in #1930
• chore: performance improvements around zset by @romange in #1928
• fix: macos segfault loading snapshot by @andydunstall in #1931
• feat(replication): First iteration on partial sync. by @royjacobson in #1836
• fix: set the key index within the json.debug command by @iko1 in #1932
• chore: flip default sorted set implementation to bptree by @romange in #1933
• chore: improve doubles encoding for listpacks by @romange in #1938
• feat(search): Early limit cutoff by @dranikpg in #1943
• feat(streams): support XACK command by @theyueli in #1923
• chore: Change the DNS log message to something that makes sense externally by @royjacobson in #1947
• fix(regtests): Bind to 127.0.0.1 and not localhost by @royjacobson in #1937
• bug(streams): entries_read is not updated on xreadgroup command by @theyueli in #1946
• fix: test_cancel_replication_immediately by @kostasrim in #1942
• fix: small typos on flags by @kostasrim in #1953
• fix(AclFamily): do not allow to delete default user by @kostasrim in #1954
• chore: Small refactor of DflyInstance by @royjacobson in #1951
• feat(streams): support entries_read and lag for XINFO GROUPS by @theyueli in #1952
• fix(regtests): Limit infinite loop in replication_test by @royjacobson in #1961
• Add a feature flag for partial_sync by @royjacobson in #1957
• chore: preallocate lua arrays by @romange in #1964
• fix(regTests): adjust network disconnect small buffer to pass on by @kostasrim in #1970
• feat(cmd): add restricted commands flag by @andydunstall in #1967
• chore: rename owner to conn by @romange in #1973
• chore: adjust verbosity level 1 not be over-verbose in prod by @romange in #1974
• feat(search): Basic sorting by @dranikpg in #1941
• fix(regtests): Colored per-instance log by @dranikpg in #1971
• chore: Print useful info when printing DflyInstance objects by @romange in #1977
• chore(regression): test bptree on regression pytests by @adiholden in #1963
• fix(RegTests): fix replication test_network_disconnect_active_stream by @kostasrim in #1980
• chore(dfly_main): refactor to use RAII by @kostasrim in #1981
• chore(Http): change authed username to default and skip auth on metrics by @kostasrim in #1950
• fix(regtests): Reduce load on redis replication test by @dranikpg in #1984
• fix: typos by @omahs in #1986
• fix(server): Fix async lua by @dranikpg in #1982
• chore: DEBUG SHARDS by @dranikpg in #1987
• fix(Listener): skip auth when requirepass is empty on http by @kostasrim in #1985
• feat(streams): support LASTID option for XCLAIM command. by @theyueli in #1968
• Docs: fix typos and grammar by @masterujjval in #1983
• fix(regTests): adjust small network buff test by @kostasrim in #1991
• feat(search): Plug in polymorphic allocator by @dranikpg in #1819
• fix(regTests): test_cancel_replication_immediately races by @kostasrim in #1990
• feat(aws): add s3 awsv2 by @andydunstall in #1929
• fix(server): Add additional metrics by @dranikpg in #1975
• fix: fix eval in multi interpreter borrow by @dranikpg in #1999
• fix: fix queue backpressure access by @dranikpg in #2000
• feat: add support for XINFO STREAM stream_name by @rounaknandanwar in #1533
• update(requirepass): behave as redis requirepass by @kostasrim in #1996
• feat(streams): support XAUTOCLAIM command by @theyueli in #1969
• fix: make --help flag work for development builds by @romange in #2004
• fix: logrotate for dragonfly logs by @romange in #1972
• fix(acl/admin_port): skip acl validation on admin port by @kostasrim in #1997
• opt(server): If unspecified, use num_shards == proactor_threads by @chakaz in #1998
• feat(aws): s3 snapshot error handling by @andydunstall in #2002
• chore: add balls and bins simulator by @romange in #2001

New Contributors

• @yancz2000 made their first contribution in #1772
• @Equilibrium2702 made their first contribution in #1838
• @aidarbek made their first contribution in #1810
• @UkuLoskit made their first contribution in #1867
• @omahs made their first contribution in #1986
• @masterujjval made their first contribution in #1983

Huge thanks to all the contributors! ❤️

Slava Ukraini ! 🇺🇦
Glory to Israel ! 🇮🇱

Full Changelog: v1.9.0...v1.11.0