vaticle/typedb 2.2.0

TypeDB 2.2.0

on GitHub

Install & Run: http://docs.vaticle.com/docs/running-typedb/install-and-run

New Features

Bugs Fixed

• Reduce traversal and resolver caches memory overhead
  The type resolver and traversal caches with up to 10k elements can each occupy a lot of native memory (non-heap), via the MP solver objects. The type resolver traversals are no longer cached and traversal cache is limited to 500 plans.

• Address RocksDB iterator assertion error due to no keys
  When RocksDB -dev is enabled (ie with its own assertions), tests (notably BasicTest) fail when re-opening a RocksDB database. This is caused by syncing our key generator, which relies on a rocks iterator's seekForPrev() call. We get an invalid iterator back when there are 0 keys in RocksDB, which is only a RocksDB error when development JNI is in use. We solve this by adding an isValid() check after doing seekForPrev()

• Delete and Insert queries are validated once rather than per answer
  We refactor Deleter, Inserter and Updater in order to validate the queries once, before execution, rather than per-answer.

• Enable reasoner to handle reflexive relations
  We enable reasoner to handle reflexive relations where previously it would throw an internal error.

• RPC prefetch size must be least 1
  To avoid potential stalls (when latency is 0/disabled in the future), RPC prefetch size must be least 1 according to the user configuration. This PR adds the required validation.

• Ensure that the RPC prefetch size is at least 1
  Ensure that the minimum prefetch size is at least 1, so that the user cannot end up in a situation that streams back 0 answers per res_part request.

• Hugely reduce transaction memory usage
  To enable transactions to traverse and operate over millions of vertices and edges, we cannot cache every vertex and edge read from the storage layer. This PR disables caching vertex (and implicitly edge) caching, allowing traversals of millions of vertices without OOM. Concepts automatically convert their inner read vertex to a write vertex (which is cached) whenever a write is submitted onto a concept.

Code Refactors

• Allow tests to include anonymous identifiers in filter
  For the purpose of testing reasoner, we need to be able to return answers to anonymous variables, for queries constructed using backend TypeQL patterns. This is almost doable, except that our filters only allow named identifiers. This change allows us to pass anonymous identifiers in the filter, and calling the executeTraversal/executeReasoner methods directly, which are now public.

• Streamline statistics counting and fix ACID with no RocksDB writes
  As of #6306, Attribute PUT operations do a read before doing a write to rocksDB. We can harness this to only add a attribute PUT count job if the attribute is actually new, dramatically reducing the number of background count operations the statistics thread has to perform.
  This leads to an ACID bug where doing true attribute PUT leads to no actual writes on RocksDB, and therefore no storage version range against which to check for concurrent commits. We fix this by always writing at least one key to RocksDB per write transaction.

• Type casting errors include type label
  As identified in #6248 and in several community messages, we clean up UX that is exposed when a user mis-uses a type as a different meta type (eg. uses an entity in place of an attribute). We now include the type label in the error message to allow tracking down the error much more easily.

Other Improvements