vaticle/typedb 1.5.8

Grakn Core 1.5.8

on GitHub

Install & Run: http://dev.grakn.ai/docs/running-grakn/install-and-run

New Features

• Always infer types in queries.
  Previously we were only inferring types (roles, relation types) in queries when the reasoning was set to on. Now all queries, regardless of the reasoning setting, shall benefit from the type inference.

Bugs Fixed

• Add extra validation for copying attribute values in rules.
  Previously we were silently allowing to commit rules which copy attribute values with incompatible (different) data types. This PR adds an extra validation step to prevent that.

• Update isaExecutor behaviour to handle all types of concepts.
  Continuation of #5145. Previously when executing ISA properties, we were giving attributes special consideration. Now we treat all concept types equally - we allow for upcasting while throwing an exception when a downcast is attempted. Additionally, if a concept already exists, we do nothing instead of throwing an exception.

• Make sure when and then blocks are parsed correctly.
  Make sure that when and then blocks are parsed correctly so that rules can be defined via the API without issues.

• Simplify the architecture by removing the 'graknsystem' keyspace.
  We have removed the graknsystem keyspace. It is not needed and the removal leads to a simpler architecture and faster load time.

• Upgrade Jackson-Databind to address security concern and fix SonarCloud bugs.
  Resolve security issue with Jackson-Databind by upgrading to latest version and fix minor bugs flagged by SonarCloud

• Refactor and optimise resolution plan.
  To simplify and correct the behaviour of resolution planning: remove the recursion and make sure the ordering ensures atom connectedness if possible.

• Ensure rule states are created lazily.
  Previously the rule state iterator was built from a stream containing a stateful intermediate operation - sorting, which requires processing of all elements. As a result the rule states were constructed eagerly. This PR makes sure that they are created lazily.

• Refactor parsing of ontological atoms.
  To make sure predicate variables of ontological atoms are always returned which ensures correctness of results - otherwise Gremlin throws an IllegalArgumentException due to failure of finding the variable.

• Use persisted values when materialising attributes.
  Previously when materialising attributes (inserting inferred attributes) we were using the value provided in the query. It was possible to have a mismatch between the provided value and the persisted value of the corresponding attribute. As a result, it was possible to miss an already existing attribute and proceed with the insertion resulting in an attribute duplicate. This PR fixes this issue by ensuring we are always following the PUT behaviour with a persisted attribute data type.

• Remove grakn.properties GlobalOffline Configuration Options.
  After some experimentation, we realised that the two options cache.db-cache-clean-wait and cache.db-cache-time were not having any effect. This is because they fall under GLOBAL_OFFLINE configuration options (see janusgraph configuration reference https://docs.janusgraph.org/0.2.0/config-ref.html), and require a different approach for being set compared to maskable or local options.
  The accepted current solution is that we fully remove the GLOBAL_OFFLINE options from the grakn.properties file until firstly, we have a good understanding of why these options are GLOBAL_OFFLINE and are therefore persisted to the backend and propagated to all nodes on bootup. Secondly, once we have this understanding, a full fix may end up re-exposing the db-cache-time etc. in a roboust way.

• Look for non-trivial instance loops when determining whether reiteration is needed.
  When trying to determine whether a resolution reiteration is needed, we check for cycles in the type dependency graph and later in the relation instances. Previously when looking for cycles among relations, we were looking at direct neighbours only. This would lead finishing the processing too early in some cases. Now we look for non-trivial (transitive) cycles.

• Do not print success when failing at loading file.
  We were previously printing "Successful Commit:" even when exceptions were thrown while loading data from file using console, i.e. grakn console -f ./test.cql.
  With this PR we make sure the successful message is not printed anymore and instead we only print the exception message.

• Include subtypes when determining fruitless rules.
  Previously in the RuleCache we were trying to assess whether a type has instances. To do that we were looking whether the type has instances and whether an instance could be created via a rule. When looking at possible rules to do that, we were only looking at rules of the specific type omitting it's subtypes. This PR fixes that.

• Moves version instructions to command usage of grakn server.
  version is no longer listed in the command usage instructions of grakn. Instead, it's now displayed as one of the grakn server commands.

• Fix comparisons of variable value predicates.
  Previously when making equivalence checks for VariableValuePredicates we weren't taking into account the underlying Operation object. This PR makes sure we include the operation in both the hashing and equality checks.

• Remove some of the sonarcloud bugs.
  Address some of the bugs highlighted by sonarcloud, especially those concerning possible nulls, use of resources and use of Random.

• Make sure all inferred concept dependants are persisted when inserting.
  When we are inserting, to ensure all required inferred concepts are marked for persistence, this includes situations:

  • multiply-nested relations where children role players are inferred
  • implicit relations of attributes
  • when inserted relations are non-inferred but role players are possibly inferred

Code Refactors

• Convert attribute values to persisted data type.
  We are parsing attributes as they are - we do not take into account the validity or compatibility of the provided value with respect to the underlying attribute data type. This is fine as the QueryParser's job is to just parse patterns and it doesn't have any information about the specific data model. However, during query processing this potential lack of agreement between datatypes leads to problems, especially when reasoning (equivalence checks, rule applicability checks). As now all query processing happens via reasoning facilities, it makes sense to introduce a value conversion when we define atoms. In this way all attribute atoms always are defined in terms of the defined persisted data type. This PR accomplishes that.

• Refactor keyspace service to allow extension.
  In order to support authenticated operations in KeyspaceService in enterprise version, it should be refactored.

• Make resolution logs not output redundant info.
  Previously some of the resolution states would produce string outputs with repeated information. In this PR we make sure no redundancies are present.

• Track inferred concepts separately to use them for determining persistence on commit.
  Previously we were using the full conceptCache to determine which inferred concepts to delete or persist on commit. As the concept cache can grow substantially, extra filtering for inferred concepts introduces a lot of extra overhead. In this PR, we introduce separate tracking for inferred concepts in the TransactionCache so that we can fetch them easily on commit.

• split valueExecutor's operations into separate files.
  Make the use of ValueExecutor's operations easier and more straight-forward to follow. Previously the single ValueExecutor file contained 13 different (mostly inner) classes. Even with correct indentation, it makes it hard to follow and make any changes.

• Remove redundant answer projection.
  Remove unnecessary answer projection and distinct stream operations. These are redundant and introduce overhead for larger numbers of answers.

• ResolutionState refactor.
  To simplify and flatten the structure of ResolutionStates. The aim is to make further more advanced optimisations easier to kickstart and carry out.

Other Improvements

• Make Bazel output less verbose logs.
  We've made Bazel logs to be quieter so that we can focus more on logs that matters.

• Implements code quality analysis powered by sonarcloud.io.
  We have integrated sonarcloud.io with the intention of increasing the visibility of source code health (fixes #4187).
  The code analysis step is ran in the build CI job and the result can be viewed at https://sonarcloud.io/dashboard?id=graknlabs-grakn

• Enable Error Prone.
  Enable Google's Error Prone (https://errorprone.info/index) to enable static analysis of code