Install & Run: http://docs.vaticle.com/docs/running-typedb/install-and-run
New Features
- Implement transaction timeouts
We implement transaction timeouts (outined in #6487), to prevent users from accidentally leaving transactions open indefinitely and leaking memory as a result. The transaction timeout is configurable via the usual transaction/session options.
- Validate encoding version and compress encoding ranges
On the rare occasion we change the data format or break backward compatibility, we should prevent users from copy-pasting their databases across incompatible server versions. We now write a data encoding version into the schema database on creation, and validate it against the server's data encoding version on open. Users will receive an exception and the server will refuse to start when given an incompatible database.
non backward-compatible change
Bugs Fixed
- Remove all default string encodes/decodes
Although #6475 encoded string attributes using UTF-8, we hadn't extended it to encoded labels and rule bodies, etc. This meant that using platform defaults via Java could corrupt labels and rules when moving across operating systems.
We now specify UTF-8 for all string operations.
- Correctly apply multiple negations with the same variables
The parsing of a TypeQL query into server-side Disjunctions, Conjunctions, and Negations, we relied on the equality function of negations, which is still ill-defined (consequence of #6115). As a result, negations were lost when negations contain the same variables and were put in a set. Identified in #6471.
- Validate and encode strings as UTF-8
We use UTF8 encoding when converting a String to byte arrays. We also test that all unicode characters are correctly encoded and decoded by the database, throwing invalid strings (ie. not encodable) as an exception.
- removes warning for deprecated bottle :unneeded
Code Refactors
- Partition storage using RocksDB column families
We can improve the efficiency of our storage layer by taking advantage of partitioned RocksDB column families. Using these, we can introduce different caching, bloom filter, and memtable configurations for each part of the data. This reduces overall memory requirements and helps TypeDB scale to huge amounts of data. At scale, this change improves random lookup performance, plus memory requirements. We also delete redundant data structures on disk, reducing disk storage requirements by at least 10%, up to much higher propertions in attribute-heavy, sparely connected datasets.
- Configure RocksDB to minimise memory usage
We extensively reconfigure RocksDB to minimise memory usage, and allow the user to configure caches that directly affect performance.
We can approximate the memory usage of RocksDB with the following formula:
storage_memory = 0.5gb*#CPU + 2*(cache in gb) + 2gb
Note that JVM memory is additional to this. If the JVM uses 4gb, and there are 4 cores, and 1gb of cache, we expect a minimum of 10gb ram to be made available.
- Clarify transaction consistency error messages
Consistency violations throw error messages that could either be interpreted as a consistency error within the same transaction or across transactions. We clarify errors which are caused by transaction-transaction conflicts.
- YAML Configuration
In order to facilitate our long term goals of having a single configuration system for the entire database, including logging, we convert the preexisting typedb.properties and logback.xml into a Yaml single configuration file. This will also allow us to replicate configurations across an entire cluster in the future. In the process we also fix the bug where we were unable to configure log directories to be outside of the server distribution (#6370, #6372).
The top-level API is almost the same as before, please use the help command to get an overview:
./typedb server --help
Note: all YAML configurations can be overriden or extended with command line args, with . for nesting:
./typedb server --log.output.file.directory=/var/log/typedb
will override the persisted log directory set in the configuration.
The medium term goal is to eliminate the user-configurable typedb.yml file that comes with the server and ask the user to update configurations directly via console (#6461).
Other Improvements
-
bump VERSION
-
update dependencies to use tags
-
Upgrade grpc to 1.43.0 and netty to 4.1.61.Final
We silence false-positive grpc stream errors by upgrading grpc and netty. This propagates work started by vaticle/dependencies#339.
-
Update @vaticle_dependencies
-
Update build jobs to use Ubuntu 21.04
We've updated the CI to use the most up-to-date Factory job image which is based on Ubuntu 21.04.
-
Use update @vaticle_typedb_common
-
update @vaticle_dependencies to latest
-
cleanup Config Unexpected Value error message
-
fix = and space separated cli arg parsing
-
Upgrade @vaticle_dependencies
Upgrade @vaticle_dependencies to include latest fixes.
- Upgrade RocksDB to 6.26.1
Upgrade RocksDB to 6.26.1, the latest version.
- Update release notes script
We've updated @vaticle_dependencies, bringing in the new release notes creation script.
-
Update package structure docs
-
Remove Meetup shield, because link is broken