weaviate/weaviate v1.11.0

v1.11.0 - Reliability fixes, Open AI key at runtime

on GitHub

Changes

Breaking Changes

none

New Features

• Open AI module: provide API key at query time by @antas-marcin in #1817
  For untrusted server environments, it is not advised to store the third-party API key along with the setup. Instead it should be provided at runtime which is now possible in this release.

Fixes

• Improve way objects are counted on disk #1811
  Prior to this release a meta { count } in Aggregate would require reading every object from disk. With this change the net additions of each segment are calculated when initializing the segment and count only has to sum up each segments' values which is orders of magnitude faster

• Internally version index/shard changes more precisely #1833
  This release introduces a new internal shard/index versioning system that will allow introducing breaking changes in a non-breaking fashion. For example, new indexes created with v1.11.0 will store the keys of the Map type in the LSM store in an always-sorted fashion for additional performance (see below). Indexes built prior to v1.11.0 will still work with this version as they will simply be sorted at read-time.

• Delete using filter leaves objects searchable by @antas-marcin in #1845 #1836
  This release fixes an issue where duplicate IDs across classes led to issues on delete. Since DELETE /v1/objects/{id} would previously only delete the first object found, there could be a situation where the specified UUID still existed on another class. With this fix, every object with the specified ID - regardless of class - will be deleted. As part of this investigation we have found out that the current DELETE API is suboptimal: It does not take the class name as a parameter and therefore prevents classes from acting as real namespaces. We will deprecate this API in a future release and will add an alternative as part of the same release.

• HNSW index fails if the initial insert has doc id > 24999 by @antas-marcin in #1851 #1848
  With the introduction of importing objects without a vector and then adding a vector later in one of the previous releases, a new buggy situation was created: When more than 25,000 objects without a vector were imported, the next import with a vector would fail as the initial size of the HNSW index was smaller than the doc id and there was no check for the initial insert. This release fixes this.

• Fix segfault by copying memory by @etiennedi in #1843 #1837
  Prior to this release, it was possible to run into a SEGFAULT when importing objects and querying the inverted index concurrently. The cause for this was memory that was shared longer than a lock was held in the LSM store. Therefore an LSM compaction could remove old segments while the memory of that segment was still in use, thus leading to a SEGFAULT. Since it is not reasonable to hold such a lock for excessively long times, this was fixed by copying the respective memory on read which effectively makes it read-only and thus thread-safe.

• Fix create/update timestamp issues in GraphQL and REST by @parkerduckworth in #1847 #1844
  There were two issues: (1) the create timestamp would be overwritten on an update, (2) neither the create, nor the update timestamp could be retrieved using GraphQL. This release fixes both.

• Fix LSM net additions count and add WAL threshold by @parkerduckworth in #1855 #1830
  Prior to this release very frequent object updates, such as during a POST /v1/batch/references request would led to very frequent flushing of the memtable leading to a lot of small segments. Initializing and merging those unnecessarily small segments cost a lot of time later on during large imports. This was caused by the memtable size assuming each write was an addition. Thus any replaced value would count into the flush size counter leading to a premature flush. This release fixes this behavior by considering the net additions of a write. In addition, a new threshold is introduced to make sure that update-only requests do not lead to excessively large WALs which don't increase the memtable size counter.

• Improve how inverted index is stored on disk #1832
  For performance reasons keys of the LSM store type Map will now be stored in an always-sorted fashion. This allows for faster merging and scoring at runtime. This change only affects new indices built from this version on and is non-breaking to older versions due to #1833.

• Sort memtable KV pairs on read by @etiennedi in #1853 #1852
  Fixes an unreleased performance regression that would have been introduced by #1832

• Fix nil-pointer in segment cursor by @etiennedi in #1859 #1850
  Prior to this fix, it was possible to run into errors when listing objects while importing them. The cause for this bug was an incorrectly placed lock that was obtained slightly too late, thus allowing for the possibility of a race to occur. This fixes this by placing the lock correctly.

• Fix typos/documentation

  • Fix deprecation log field typo by @parkerduckworth in #1823
  • Restore build tags that travisbot removed by @parkerduckworth in #1857
  • Fixed typo in error message by @illagrenan in #1818

New Contributors

• @illagrenan made their first contribution in #1818
• @parkerduckworth made their first contribution in #1823

Full Changelog: v1.10.1...v1.10.2