vaticle/typedb 1.8.0

Grakn Core 1.8.0

on GitHub

Install & Run: http://dev.grakn.ai/docs/running-grakn/install-and-run

Highlights:

• New delete syntax
• Server gRPC backpressure
• Remove Implicit Attribute Ownerships
• New Query Options, including batch_size and explain (required to be set to true for explanations to be available)

Note: the DB folder for 1.8 is not backwards compatible with prior versions

New Features

• Introduce Explanation Cache and Transaction Query Explain Option.
  When the user requests it, we save answers and the associated Explanation into a specialised cache. Transaction query requests now have an Options associated, which have a new option: explain=true/false. By default, the server uses false.
  We no longer have to painstakingly "reconstruct" explanations when the user asks for them. The cache only expands to contain many entries as the user explores explanation trees, only caching the top level Answer - Explanation (top level contains full Explanation tree, so may be large).

• Merge changes from 1.7.2. - gRPC server backpressure
  We made some changes in 1.7.2 to add GRPC back pressure in order to address some client issues with server overloading. Without back-pressure, excessive async client requests (possible in the new client releases) could cause the server to become overloaded and run out of memory, taking down the entire server. Since Grakn currently does not have automatic restart or recovery features, this is considered a serious issue so this fix was patched into 1.7.2 during the 1.8 development cycle and now needs to be merged into 1.8

• Remove implicit types and attribute relations.
  Implenting #5614, and invalidates the backwards compatibility of the DB data directory!
  We split attribute ownerships from being full Grakn relations, using new storage structures: a simple edge for attribute key/ownerships, and in the schema has and key are new edge types. A major implication is that queries and reasoning over generic relations will no longer include attribute ownerships.

• **Collect match clause before execute delete, Non-Lazy deletions **
  To avoid modifying iterators while traversing, we collect all answers from the match clause, closing the iterator, and perform deletions on all the found concept maps

• ** Deletes accept statements instead of variables **
  match...; delete...; queries now accept full graql syntax in the delete clause.

This means we now:

• can finally remove role players from relations using: match $r (arole: $x) isa somerelation; delete $r (arole: $x)
• We don't require via to remove attribute ownerships anymore, instead write: match $x has attribute $a; delete $x has attribute $a;
• delete an instance using an isa clause: match $x isa person; delete $x isa person;

• Replaced the term 'DataType' with 'ValueType'.
  We are replacing the term DataType in AttributeType with ValueType). Previously, an AttributeType has a DataType. And an Attribute has Value. The type of Value is defined by DataType. We can see the inconsistency there. Now, AttributeType has ValueType, and Attribute has Value. The type of Value is ValueType. So it's all consistent.

Bugs Fixed

• Graceful session error on server restart.
  When opening a session, restarting the server, and continuing to use the session, an null pointer/unknown exception is thrown. This PR improves the error messages reported to the user in cases where the session ID is no longer found on the server.

• Fix too eager query completeness acking for queries with negation.
  Fix an occasional reasoning completeness issue encountered when combining negation with other rules.
  We track query completeness by explicitly acknowledging that specific queries were exhaustively explored once all iterations were completed in ResolutionIterator. To do that we were using the subGoals - queries that were visited during query execution. This approach is however not correct when we deal with queries with negation. Each negation query has its own ResolutionIterator however the subgoals are injected from the iterator that triggered the negation query. As a result, if the negation query does an exhaustive search which happens when no answers can be found - it can acknowledge completion of queries that were injected but not fully completed. Due to this eager acknowledgment, queries that were actually not exhaustively explored will be omitted during reasoning. We fix this issue in this PR by explicitly tracking queries explored exhaustively within a specific ResolutionIterator and using them insted of subGoals to ack their completion once all iterations have finished

• Fix test-deployment-rpm due to missing python3 dependencies.
  We have fixed test-deployment-rpm caused by missing python3 dependencies.

• Fix test-assembly-windows-zip by adding python3 to path.
  Fix test-assembly-windows-zip by adding python3 to path

• Re-enable copying attribute values across types.
  Allow copying an attribute value across types, using a rule. This was enabled in the past, but at some point between 1.6.2 and 1.7.1, a refactor accidentally caused this feature to be lost. This PR re-adds it by relaxing overly-strict constraints in AttributeSemanticProcessor, and adds a test for this case.

• Deletion - fix ordering bug.
  In some complex deletion cases, we must delete attribute ownerships and role playing first, followed by relations instances and other instances. We applied an incorrect tie breaker, resulting in deleting relations before attribute ownerships, resulting in vertex deleted exceptions from Janus. We also enforce deleting attribute ownerships before role players, as role player deletion could lead to cascading deletions.

• Apply correct tx type for incoming RPC transaction requests.
  Previous PR had accidentally always returned a WRITE transaction regardless of what the client had requested. This PR adds another test for this situation that tests it via the RPC interface to Grakn Core, and resolves the issue.

• Fix issue with too eager subgoal propagation for queries with negation.
  Negation queries are short circuited when one contradiction is found from the negation block. However, we mark negation sub-queries as fully explored by adding the sub-query to the visited subgoals that is shared globally. We then re-visit the same state at another point while exploring the negation query. However, we are missing answers, since the cache will only contain a single answer to the visited sub-query, not all of them. This means we can miss other solutions to the negation block. This is the underlying problem of #5697.

Code Refactors

• Fixed implementation of Thing.keys(types).
  There was a bug in ThingImpl.keys(AttributeType<?>... attributeTypes), where it would always return all instances of all key types, even when the user provided a subset of the key types in the argument. Additionally, the implementation can be made more readable and efficient, which could have contributed to why the bug existed in the first place.

• Refactor test packages.
  Multiple test currently reach across package boundaries to utilise the GraknTestServer or the test-integration/logback.xml. We can fix this architecture and resolve #5043 at the same time.

Other Improvements

• Enable Graql Unit Tests in CI and remove bad merge directories.
  Fix Graql unit tests, and add them to CI so they are executed going forward.
  Also, remove a copy of directories that probably resulted from a merge process that were duplicate and outside of the new test package structure (previously, had integration tests outside, now inside test/.

• Discord badge added.
  Slack badge changed to discord badge

• Bump console dep to be 1.8 compatible.
  We bump the console dependency to a newer version that is compatible with client-java 1.8 and 1.8.

• Include protocol datetime instead of date.
  We update the protocol to include new usage of datetime instead of date, which is reflected here. vaticle/typedb-protocol#34 .

• Add platform-specific instructions for installing Bazelisk.
  In #5732 we added instructions on how to build Core using bazelisk. This PR elaborates on them by providing platform-specific instructions on installing Bazelisk itself.

• Update README to mention Bazelisk.
  Ensure whoever wants to build Grakn Core from sources uses the correct Bazel version.

• Bump Graql Dependency and use DateTime.
  Re-synchronise dependencies to use master of different repositories. This change includes using datetime instead of date for the Date attribute type.

• Support for Graql Delete and validate keys on deletion of ownership.
  Delete BDD scenarios are implemented in vaticle/typedb-behaviour#38 - this PR implements the new steps required to execute the new scenarios. We also find and fix a bug where validation was not performed for keys when attributes were "un-owned".

• Update circle ci file for incorrect behavior steps.
  Fix the CI file that ran the transaction behaviour tests twice and not the undefine behaviour tests at all.