milvus-io/milvus v2.2.0

milvus-2.2.0

on GitHub

V2.2.0

Release date: 18 November, 2022

Compatibility

Milvus 2.2.0 introduces many new features including support for Disk-based approximate nearest neighbor (ANN) algorithm, bulk insertion of entities from files, and role-based access control (RBAC) for an improved security. In addition, this major release also ushers in a new era for vector search with enhanced stability, faster search speed, and more flexible scalability.

Since metadata storage is refined and API usage is normalize, Milvus 2.2 is not fully compatible with earlier releases. Read these guides to learn how to safely upgrade from Milvus 2.1.x to 2.2.0.

Features

• Support for bulk insertion of entities from files
  Milvus now offers a new set of bulk insertion APIs to make data insertion more efficient. You can now upload entities in a Json file directly to Milvus. See Insert Entities from Files for details.

• Query Result Pagination
  To avoid massive search and query results returned in a single RPC, Milvus now supports configuring offset and filtering results with keywords in searches and queries. See Search and Query for details.

• Role-based access control (RBAC)
  Like other traditional databases, Milvus now supports RBAC so that you can manages users, roles and privileges. See Enable RBAC for details.

• Quota limitation
  Quota is a new mechanism that protects the system from OOM and crash under a burst of traffic. By imposing quota limitations, you can limit ingestion rate, search rate, etc. See Quota and Limitation Configurations for details.

• Time to live (TTL) at a collection level
  In prior releases, we only support configuring TTL at a cluster level. Milvus 2.2.0 now supports configuring collection TTL when you create or modify a collection. After setting TTL for a collection, the entities in this collection automatically expires after the specified period of time. See Create a collection or Modify a collection for details.

• Support for Disk-based approximate nearest neighbor search (ANNS) indexes (Beta)
  Traditionally, you need to load the entire index into memory before search. Now with DiskANN, an SSD-resident and Vamana graph-based ANNS algorithm, you can directly search on large-scale datasets and save up to 10 times the memory.

• Data backup (Beta)
  Thanks to the contribution from Zilliz, Milvus 2.2.0 now provides milvus-backup to back up and restore data. The tool can be used either in a command line or an API server for data security.

Bug fixes and stability

• Implements query coord V2, which handles all channel/segment allocation in a fully event-driven and asynchronous mode. Query coord V2 address all issues of stuck searches and accelerates failure recovery.
• Root coord and index coord are refactored for more elegant handling of errors and better task scheduling.
• Fixes the issue of invalid RocksMQ retention mechanism when Milvus Standalone restarts.
• Meta storage format in etcd is refactored. With the new compression mechanism, etcd kv size is reduced by 10 times and the issues of etcd memory and space are solved.
• Fixes a couple of memory issues when entities are continuously inserted or deleted.

Improvements

• Performance
  • Fixes performance bottleneck to that Milvus can fully utilize all cores when CPU is more than 8 cores.
  • Dramatically improves the search throughput and reduce the latency.
  • Decreases load speed by processing load in parallel.
• Observability
  • Changes all log levels to info by default.
  • Added collection-level latency metrics for search, query, insertion, and deletion.
• Debug tool
  • BirdWatcher, the debug tool for Milvus, is further optimized as it can now connect to Milvus meta storage and inspect the part of the internal status of the Milvus system.

Others

• Index and load
  • A collection can only be loaded with an index created on it.
  • Indexes cannot be created after a collection is loaded.
  • A loaded collection must be released before dropping the index created on this collection.
• Flush
  • Flush API, which forces a seal on a growing segment and syncs the segment to object storage, is now exposed to users. Calling flush() frequently may affect search performance as too many small segments are created.
  • No auto-flush is triggered by any SDK APIs such as num_entities(), create_index(), etc.
• Time Travel
  • In Milvus 2.2, Time Travel is disabled by default to save disk usage. To enable Time Travel, configure the parameter common.retentionDuration manually.