Dragonfly v1.36.0
Important stability fixes and changes:
- Support Global Vector index #6103
- Support unicode tags in #6127
- django-cacheops compatibility fixes: #6123 #6129
- better memory defragmentation for json objects: #6240 #6276
What's Changed
- actions: Add a volume mount to container by @abhijat in #6048
- chore: remove RecvProvided from dragonfly connection by @kostasrim in #6050
- fix(rdb_load): Store streamed items per rdb lodaer by @dranikpg in #6049
- fix(search): Result set cutoff by @dranikpg in #5906
- fix: remove duplicate expired/evicted metric header by @romange in #6058
- fix(test): ssl test to work with pytest 3.12 by @romange in #6059
- feat(hset): Load tiered values in read-only by @dranikpg in #5996
- chore(tiering): Add throttling metric + more tests by @dranikpg in #6040
- fix: macOS build by @vyavdoshenko in #6065
- server: Parse JSON on destination shard by @abhijat in #6061
- feat: add CSS option for FT.CREATE by @BorysTheDev in #6057
- chore(facade): Handle Socket::Recv() returning 0 explicitly by @romange in #6066
- fix: Remove index docs on eviction and cluster migration by @mkaruza in #6062
- fix(tests): Disable huge val streaming by @dranikpg in #6074
- fix(replication): Add detailed error message for REPLTAKEOVER failures by @vyavdoshenko in #6071
- fix(tiering): Fix upload strategy by @dranikpg in #6067
- fix(replication): use correct replica pointer in takeover catchup loop by @vyavdoshenko in #6068
- fix(replication): add socket timeout to REPLTAKEOVER command by @vyavdoshenko in #6070
- fix(takeover): pick correct replica to reconcile slots by @kostasrim in #6034
- test: fix flaky test_takeover_bug_wrong_replica_checked_in_logs by @vyavdoshenko in #6075
- feat(helm): adds common label support to the helm chart by @gaganhr94 in #6042
- Revert "feat(helm): adds common label support to the helm chart (#6042)" by @romange in #6077
- fix(server): Move LpGetView to core target by @dranikpg in #6076
- feat: adds common label support to the helm chart by @gaganhr94 in #6080
- docs(build-from-source): update LTO instructions to use USE_MOLD option by @glevkovich in #6081
- Update build-from-source.md by @dranikpg in #6087
- feat: add ftsearch result parsing for css by @BorysTheDev in #6090
- chore: pull the latest helio by @romange in #6096
- fix: suppress BUSYGROUP warning logs by @vyavdoshenko in #6095
- feat(tiering): Basic metrics by @dranikpg in #6086
- chore: change the resolution of reply latency to measurement to ns by @romange in #6092
- fix: test_reply_count regression by @romange in #6099
- chore(hooks): add commit-msg hook to enforce Signed-off-by line in commit messages by @glevkovich in #6082
- fix(server): handle INFO command with multiple sections correctly by @glevkovich in #6093
- chore: pull helio by @romange in #6100
- fix: smashing stack on Huffman test by @vyavdoshenko in #6102
- tools: Add scripts and supporting files to generate os packge repo by @abhijat in #5923
- chore(deps): bump the actions group across 1 directory with 4 updates by @dependabot[bot] in #6107
- chore(core): Refactoring to prepare for using stateless allocator by @abhijat in #6111
- tools: Download ARM deb package, disable fedora ARM test by @abhijat in #6110
- fix: replica of self by @kostasrim in #6097
- chore: relax pytest requirements by @romange in #6113
- feat(JSON): Use stateless allocator by @abhijat in #6060
- fix(dfly_bench): fix parsing for memcache by @romange in #6116
- chore: update vector-bench README by @romange in #6117
- chore: add mi_malloc memory test by @romange in #6118
- docs: add copilot-instructions.md for AI coding agent onboarding by @Copilot in #6112
- fix: Handle django-cache script so it could run in dragonfly by @romange in #6123
- chore: replace PMR_NS with StatelessAllocator in dense_set by @romange in #6121
- chore: implement background self-identification of huffman encoding by @romange in #6106
- chore: gcc14 for fedora 30 by @vyavdoshenko in #6124
- chore: expose more information when takeover fails on master. by @romange in #6130
- feat(search): Global HNSW vector index by @mkaruza in #6103
- fix(search): Support unicode for tags by @dranikpg in #6127
- chore: use our own vector for hnsw distance computation by @romange in #6132
- chore: more fixes for django-cacheops compatibility by @romange in #6129
- chore: refactor hnsw index into separate files by @romange in #6134
- chore: move HierarchicalNSW class to our codebase by @romange in #6136
- feat(tiering): Serialize hashes by @dranikpg in #6015
- fix(stream): Allow XINFO in script by @dranikpg in #6128
- fix(core): Remove async delete flag by @dranikpg in #6139
- feat: implement DenseSet::Shrink by @vyavdoshenko in #6094
- chore(core): Move page_usage related code to its own library by @abhijat in #6146
- chore: quietly ignore LATENCY HISTOGRAM command by @romange in #6151
- fix(tiering): missing comma in dashboard json by @goochs in #6150
- chore: add a tool that creates charts based on memtier metrics data by @romange in #6154
- chore: add latency graph to local dashboard by @romange in #6153
- fix: incorrect stalled heartbeat warning by @romange in #6155
- feat(lua): add legacy-float flag for Lua 5.1 number compatibility by @vyavdoshenko in #6149
- chore(dfly_bench): properly handle 0 from Recv by @romange in #6159
- chore: minor cleanups around the eval flow by @romange in #6152
- tests: Use AWS public docker registry by @abhijat in #6161
- fix(search): Fix numeric tree splits by @dranikpg in #6143
- fix(tiering): Small test fixes by @dranikpg in #6133
- chore: introduce common directory by @romange in #6170
- chore: introduce ParsedArguments class by @romange in #6169
- chore: rename WRITE to JOURNALED by @romange in #6172
- chore: fix data corruption bug by @romange in #6174
- chore: add ability for ParsedArgs to reference BackedArguments by @romange in #6171
- chore: use BackedArguments for implementing a pipeline message by @romange in #6167
- feat(search): Defragment search indices by @abhijat in #6144
- chore(core): Refactor SmallString by @dranikpg in #6138
- chore: clean up sanitizers checks by @romange in #6173
- feat(fuzz): improve fuzzing effectiveness, better timeouts, and comprehensive seeds, reproduction tooling by @vyavdoshenko in #6166
- chore: use BackedArguments for memcache_parser by @romange in #6176
- CI: download docker for correct architecture by @abhijat in #6180
- fix(search): Fix numeric tree finalization by @dranikpg in #6188
- chore: improve backed_args by allowing random access to its arguments by @romange in #6189
- Add coordinator connection reusing by @BorysTheDev in #6179
- chore(server): Don't decode size for empty blob by @mkaruza in #6194
- chore: hold memcache value in BackedArguments by @romange in #6193
- chore(core): Remove some unused CompactObj code by @dranikpg in #6141
- feat(search): Simple multi-reader multi-writer mutex for hnsw index by @mkaruza in #6156
- chore: asynchronous IO for connection fiber by @kostasrim in #6069
- fix: macOS test by @vyavdoshenko in #6195
- Integrate hiredis by @BorysTheDev in #6198
- fix(search): Use thread lock for hnsw multi reader/multi writer mutex by @mkaruza in #6205
- chore: to move some of the command-specific state into CommandContext by @romange in #6199
- hset_family: Preserve ttl on increment command by @abhijat in #6186
- Pass CommandContext as a mutable pointer to all the commands. by @romange in #6206
- fix: macOS compilation error by @vyavdoshenko in #6208
- chore(rax): add move ctor to rax tree, cleanup defragmap by @abhijat in #6178
- docs: add antigravity agent instructions by @romange in #6209
- chore: reduce boilerplate by removing redundant function declarations by @romange in #6207
- chore: move some SendError calls to CommandContext by @romange in #6211
- chore(server): polish CommandContext data structure by @romange in #6214
- fix: regression tests by @romange in #6212
- feat: add WITHSORTKEYS option to FT.SEARCH by @BorysTheDev in #6210
- chore(tiering): Optimize serialization by @dranikpg in #6164
- chore: Introduce ParsedCommand by @romange in #6219
- fix(search): remove invalid DCHECK in FT.INFO for concurrent index operations by @vyavdoshenko in #6204
- fix(server): Fix EXPIRE bugs, support non-exclusive options by @dranikpg in #6203
- chore: pass ParsedCommand to DispatchMC by @romange in #6221
- fix(server): Rename with tiering by @dranikpg in #6217
- feat(search): Yield when rebuilding index by @mkaruza in #6224
- fix: load search index from old snapshots with TAG SORTABLE SEPARATOR order by @vyavdoshenko in #6220
- chore: switch dev docker to GCC 14 by @vyavdoshenko in #6225
- fix: fix build when search is off by @BorysTheDev in #6229
- chore: further clean ups around ParsedCommand. by @romange in #6223
- docs: create unified AI agent instruction system with AGENTS.md by @vyavdoshenko in #6231
- fix(core): Account for size change on json object defragment by @abhijat in #6053
- feat: create a wrapper for hiredis by @BorysTheDev in #6233
- chore: further replace SendError calls with CommandContext by @romange in #6236
- chore: simplify memcache pipelining flow by @romange in #6230
- chore: reduce reply builder interface by @romange in #6237
- chore: swap SendError calls in json_family by @romange in #6238
- feat: add open addressing hash set by @BorysTheDev in #4705
- fix: macOS build by @vyavdoshenko in #6239
- chore: reduce compilation dependencies between server and facade by @romange in #6242
- chore: switch SendError calls to CommandContext by @romange in #6245
- chore: minor fixes around async execution by @romange in #6241
- chore: Split CompactObj into Key&Value by @dranikpg in #6218
- chore(core): remove unused num_used_buckets_ from DenseSet by @vyavdoshenko in #6250
- fix: cross-shard search tests by @BorysTheDev in #6244
- chore: refactor ParsedCommand implementation into a separate file by @romange in #6251
- fix: reset obj_malloc_used_ in DenseSet::Clear by @vyavdoshenko in #6254
- chore: simplify memcache_parser_test by @romange in #6259
- chore(server): Remove more family header definitions by @dranikpg in #6255
- chore: MemcacheParser now consumes all the input. by @romange in #6260
- chore: an issue of destruction of in-flight parsed commands by @romange in #6258
- chore: switch SendError calls in zset_family by @romange in #6248
- feat(json): Improve json defragmentation by @abhijat in #6240
- chore: align parse errors with deferred replies by @romange in #6261
- chore(server): Split tiering into own cmake target by @dranikpg in #6252
- chore(server): Remove some manual acl by @dranikpg in #6256
- chore(common): Include cstdint to fix build error on newer fedora versions by @abhijat in #6268
- chore(server): Multi consistency test by @dranikpg in #6215
- chore: move memcache reply rendering into MCRender by @romange in #6267
- chore: reduce compilation dependencies between facade and server by @romange in #6269
- chore(deps): bump the actions group with 4 updates by @dependabot[bot] in #6222
- chore: get rid of command specific flags in MCReplyBuilder by @romange in #6273
- feat(transaction): Blocking unlock multi by @dranikpg in #6246
- fix: macOS build by @vyavdoshenko in #6275
- Add alloc size to OAHSet by @BorysTheDev in #6272
- chore: switch SendError calls to CommandContext API by @romange in #6270
- chore: implement async dispatching for memcache SET command by @romange in #6247
- chore(helio): bump for improved TLS EOF handling by @glevkovich in #6274
- chore(server): Unite family headers by @dranikpg in #6266
- chore(server): Use cycle based quota for defragmentation by @abhijat in #6262
- chore: swap ParsedEntry::CmdData with BackedArguments by @romange in #6278
- chore(json): Use new json defragmentation by @abhijat in #6276
- fix: loading of a snapshot from a cloud folder by @romange in #6280
- chore: fix build error by @romange in #6289
- chore: simplify GAT command arguments by @romange in #6288
- chore: combine GAT and MGET code together by @romange in #6290
- fix: allow Cm_Cache_Backend_Redis Lua scripts by @vyavdoshenko in #6292
- fix: CONFIG GET returns numeric bytes for memory configs by @vyavdoshenko in #6291
- chore: remove Stored and Deleted interfaces from ParsedCommand by @romange in #6287
- fix: macOS build by @vyavdoshenko in #6293
- chore: change DispatchCommand interface by @romange in #6283
- chore: implement support for deferred mget by @romange in #6286
- chore: update .gitignore and clarify TRAFFIC help by @glevkovich in #6294
- chore: optimize experimental_io_loop_v2 flow by @romange in #6295
- feat(build): rename cxx_test to helio_cxx_test by @glevkovich in #6297
- chore(server): Remove list family header file by @dranikpg in #6296
- Sorting for cluster search by @BorysTheDev in #6301
- feat(server): Allow SELECT inside global/non-atomic exec/eval by @dranikpg in #6302
- chore: introduce RespSrvParser by @romange in #6300
- chore: move more reply calls to CommandContext API by @romange in #6303
- fix: Handle expired hash fields during RENAME without crashing by @vyavdoshenko in #6306
- chore: refresh release settings for dragonfly by @romange in #6304
- chore: fix release action by @romange in #6308
- test(geo): relax fp checks for Helio Haswell upgrade by @glevkovich in #6307
- Fix fiber stack overflow and improve JSON safety by @romange in #6316
- CI: sync valkey tests by @abhijat in #6319
- chore: Add debug information to investigate memcache regtest failure. by @romange in #6322
- fix: handle control messages during SquashPipeline iteration by @vyavdoshenko in #6313
- chore(defrag): Add defragmentation of search indices by @abhijat in #6281
- CI: Extract common step for syncing valkey tests into action by @abhijat in #6333
- tests: make valkey_search integration tests optional by @vyavdoshenko in #6332
- chore: Reused parsed messages for memcache workloads by @romange in #6324
- fix: memcache parser CRLF handling by @romange in #6330
Huge thanks to all the contributors! ❤️
New Contributors
- @gaganhr94 made their first contribution in #6042
- @glevkovich made their first contribution in #6081
- @Copilot made their first contribution in #6112
- @goochs made their first contribution in #6150
Full Changelog: v1.35.0...v1.36.0