milvus-io/milvus v2.0.0-pre-ga

milvus-2.0.0-PreGA

on GitHub

v2.0.0-PreGA

Release date: 2021-12-31

Compatibility

Milvus 2.0.0-PreGA is the preview release of Milvus 2.0.0-GA. It now supports entity deletion by primary key and data compaction to purge deleted data. We also introduce a load balancing mechanism into Milvus to distribute the memory usage of each query node evenly. Some critical issues are fixed in this release, including cleanup of dropped collection data, wrong distance calculation of Jaccard distance, and several bugs that cause system hang and memory leakage.

It should be noted that Milvus 2.0.0-PreGA is NOT compatible with previous versions of Milvus 2.0.0 because of some changes made to data codec format and RocksMQ data format.

Features

• Deleting entity: Milvus now supports deleting entities through primary keys. Whereas Milvus relies on append-only storage, it only supports logical deletion, id est, Milvus inserts a deletion mark on the entities to cover actual data so that no search or query will return the marked entities. Therefore, it should be noted that overusing deletion may cause search performance plummeting and storage usage surging. See Delete entities for more instruction.

• Compaction: Compaction mechanism purges the deleted or expired entities in binlogs to save storage space. It is a background task that is triggered by data coord and executed by data node.

• Automatic Loadbalance/Handoff #9481：Loadbalance mechanism distributes segments evenly across query nodes to balance the memory usage of the cluster. It can be triggered either automatically or by users. Handoff mechanism refers to that, when a growing segment is sealed, query node waits until the segment is built with index by index node and then loads the segment into memory for search or query.

Improvements

• #12199 Parallelizes executions between segments to improve the search performance.
• #11373 Allows batch consumption of messages in RocksMQ internal loop to improve the system efficiency.
• #11665 Postpones the execution of handoff until index creation is completed.

Bug fixes

• Data is not cleared on etcd, Pulsar, and MinIO when a collection is dropped:
  • #12191 Clears the metadata of the dropped segment on etcd.
  • #11554 Adds garbage collector for data coord.
  • #11552 Completes procedure of dropping collection in data node.
  • #12227 Removes all index when dropping collection.
  • #11436 Changes the default retentionSizeInMB to 8192 (8GB).
• #11901 Wrong distances calculation caused by properties of different metric types.
• #12511 Wrong similarity correlation caused by properties of different metric types.
• #12225 RocksMQ produce hang when do search repeatedly
• #12255 RocksMQ server does not close when standalone exits.
• #12281 Error when dropping alias.
• #11769 Update serviceableTime incorrectly.
• #11325 Panic when reducing search results.
• #11248 Parameter guarantee_timestamp is not working.

Other Enhancements

• #12351 Changes proxy default RPC transfer limitation.
• #12055 Reduces memory cost when loading from MinIO.
• #12248 Supports more deployment metrics.
• #11247 Adds getNodeInfoByID and getSegmentInfoByNode function for cluster.
• #11181 Refactors segment allocate policy on query coord.