vaticle/typedb 2.0.0-alpha-8

Grakn Core 2.0.0-alpha-8

on GitHub

Install & Run: http://dev.docs.grakn.ai/docs/running-grakn/install-and-run

New in alpha-8: Grakn's shiny, brand-new rule engine is now enabled! It provides significantly more scalable and performant reasoning with a more consistent API and fewer edge cases - however, as always, please do submit bug reports for any bugs found!

See the updated documentation (http://dev.docs.grakn.ai/docs/schema/rules) to learn about rules.

Note: for this release, reasoning is only available in read transactions.

New Features

• Enable reasoner with synchronous iterators
  Enable reasoning by default, and add the set of passing reasoning tests to CI. To do this, we revert reasoning to use synchronous (iterator-based) traversals. This PR signals that reasoner is now ready to start being tested!

• Allow key generator serialisation to support cluster key synchronisation
  Allow key generator serialisation so that key generator values can be put into cluster's log entries to support cluster key synchronisation on leader election.

• Use self-hosted Bazel cache
  Speed up CI by using self-hosted remote Bazel cache

• Propagate resolver exceptions to query
  This PR connects exceptions thrown during reasoner resolution to the match query to report errors to the user. During resolution, it is possible a resolver generates or receives an exception in the event loop, which should lead to an error in the query being reported to the user, which must be asynchronously propagated through the resolvers back to the user's iterator.

• Reasoner handles nested disjunctions inside negations
  We make it possible for reasoner to process disjunctions that are nested inside negations.

• Resolution tracing
  We introduce a visual reasoner tracing tool that displays the internal actions of the reasoning engine. It maps out the messages passed between the actors of the message passing system that implements reasoning in Grakn. This has been built for internal use, but is exposed for use by anyone using an option on transactions.

• Retrievable resolvers use parallelised async traversal
  To speed up resolution, we parallelise and hand off traversal work to the Traversal engine on a separate thread pool.

• Split rule resolution into Conclusion and Condition steps
  To make best use of all the information known at resolution time, we make a large resolution optimisation: to ensure known concepts arriving in a rule then are not lost, we traverse the rule then and pass possible expanded answers found to the when of the rule. To facilitate this, we split the old RuleResolver into Condition and Conclusion resolvers separately.

• Traversals and reasoner return concepts for anonymous variables too
  To enable reasoner to bind concepts to anonymous variables in conjunctions, we make a sweeping change that traversal engine and reasoner both return anonymous variables as well. Consequently, in Grakn now ConceptMaps contain Identifer rather than References, and can identify anonymous variables.

• Native reasoner disjunctions, negations, limit and offset
  To efficiently and properly implement query-level negation, we must push negation into the reasoning resolvers. This PR implements reasoner-based negations and disjunctions which are required to support all top-level query constructs, along with limit and offset.

• Inject filters into reasoner resolvers
  To correctly deduplicate answers within a conjunction, we filter answers in the Root resolver before adding them to the deduplication set. We also convert filter to be set type, rather than list type.

• Port AlphaEquivalence tests from 1.x
  Port 1.x tests dealing with alpha-equivalence: AtomicEquivalenceIT, AtomicQueryEquivalenceIT for improved coverage.

• Reasoner integration
  Provide the final integration between components (along with the necessary fixes), such that the reasoner runs end-to-end for basic reasoning queries.

• Prioritise connected resolution plans
  The reasoner planner should always prioritise using connected resolvables before disconnected ones - otherwise we can amplify the number of messages sent massively. This PR modifies the resolution planner to prioritise the most connected resolvables first.

• Reasoner planning
  We add a rudimentary planner for the order of resolution by the reasoner. A query is split into resolvable parts: those that can be retrieved those that can be concluded. These resolvables can have inter-dependencies that need to be observed for correctness, which the planner accounts for. Within these bounds the planner uses basic heuristics to choose the plan, for example: resolving the concludable that can trigger the fewest rules first.

• Implement retrievable resolver
  Retrievables are composed of elements of a conjunction that can be answered directly by retrieval from the database. We implement the actor that processes requests and responses for a given Retrievable.

• Rule index: fast rule lookup and prevent deleting types in rules
  Create a Rule Index, which indexes Types to Rules they are either 1) mentioned in, or 2) they they may be concluded by. 1) allows us to prevent undefining types that are utilised in rules. 2) allows us to find rules that are relevant to a reasoning query, without relying on scanning every rule.

• Refactor Concludables to achieve proper unification
  We refactor the structure of a concludable to hold one of four specific structures. This makes the task of unification with Rule Conclusions more succinct and straightforward. This new structure also properly accommodates the unification of value comparison cases, some of which were difficult to accommodate.

Bugs Fixed

• Fix possible segfault when closing session
  We fixed a possible segfault when closing a session while a db operation is running.

• Fix distribution runner in common
  Fix distribution runner in common repository.

• Refactor resolver's async traversal
  Refactor and simplify the control flow of a Retrievable resolver's asynchronous traversal execution.

• Include console jars in assembled distribution
  Fix the assembly distribution that causes assembled distribution doesn't have console in it.

• Reduce Enum memory overhead by hardcoding lookups
  Calling Enum.values() allocates a new byte array for each call. This leads to high memory overheads when converting bytes to our Encoding objects. To mitigate this, we create static indexing lists based on the byte value of a key, where possible, and elsewhere fall back to if-elseif blocks.

• Propagate session options down to transactions
  Previously, options set at the Session level were not being propagated down to Transactions. Now, they are correctly propagated.

• Refactor partial answer flow to fix missing answers
  We fix several misbehaviours with an architectural cleanup of how we handle answer states. There were three issues all leading to missing answers. These are caused by incorrect termination criteria for reasoning, leading to premature termination.

• Fix partial answer flow and exact value comparison
  We fix issues in reasoner regarding: partial answer de-duplication; exact equality comparison between two variables by value; and correctness of passing partial answers onwards.

• Allow unifiers to impose concept identity requirements
  Rule materialisation, and subsequent reverse unification can create answers that do not necessarily match concepts that were found before. To align the concepts found by a rule, with concepts found by the parent resolver of the rule, we introduce Unifier.Requirements.Instance, which are generated at unification time, and re-applied at un-unification time.

• Fix materialisation from rule traversal answers
  Rules that perform traversals to find answers, should also materialise them as the then of the rule, and pass the result upstream. This PR fixes the previous incorrect behaviour

• Reasoner resolvers handle old exhausted messages correctly
  To prevent a class of bugs to do with reiteration (which are difficult to test), we make sure we don't let previous iterations' Exhausted event loop messages pollute more advanced iteration states. One visible consequence is when a reasoning query reports it is finished, and we close resources, we no longer see exceptions due to ongoing resolver work.

• Event loop catches all errors
  The event loop threads should not die when errors nor exceptions are thrown. This PR allows the event loop to catch any throwable, and propagate the error to the responsible actor.

• Type resolution correctly handles explicitly labelled roles
  TypeResolver can now handle TypeVariable within a query. We no longer use the Variable method addTo(). This will fix a bug where explicitly labelled roleTypes were not dealt with properly. The test have also been improved, so that it checks which resolvedTypes are added to Anonymous and Labels Variables as well as Named Variables.

Code Refactors

• Resolution answer state equality depends on applicable resolver
  To ensure that equality functions in AnswerState can distinguish between two answers that come from content-identical paths but with different resolvers (eg. two resolvers that produce the same answer, but have different queries or resolver instances), we update the equality functions to include the correct resolver that extends any answer state.

• Prevent open iterators from crashing RocksDB when tx closes
  It's currently possible to crash RocksDB by closing open iterators that a class is currently using. Guarding against this requires synchronisation between state changes in RocksIterator, and guarding between closing storage and opening new iterators in RocksStorage.

• Resolution and reiteration test
  Reasoner resolution relies on a message-passing actor model internally. In order to test this we have integration tests that check that the messages sent and received are as expected. We upgrade these tests (which were mocked) to be compatible with the now functional reasoner. We also create a thorough test case for reasoner reiteration, which is required in order to run Nth passes over materialised answers in order to find the full and correct set of answers for a query.

• Run reasoning exclusively when rules are present
  To avoid having to deduplicate answers from traversal engine and reasoner, we choose to either only execute a query in the traversal engine, or only in reasoner. The switch for this is whether there are any rules present. Further, the root reasoning resolver only creates child resolvers if any rules are applicable to the query it contains.

• Disable derivation recording in Resolution
  As we are not working towards enabling explanations in 2.0.0, we are disable code that mostly supports collecting explanations, but is out of date by about 3 months. We will go back and enable the ignored code paths when fleshing out Explanations fully.

• RolePlayer Unification Without Explicit Indexing

  • Reimplementation of Unify() in Concludable.Relation without using explicit indexing.
  • Make the code more readable by removing all indexing

Other Improvements

• Fix build error in RocksStorage
  We fixed a build error in RocksStorage.

• Upgrade common
  We upgraded the common repo version.

• Update protocol (for Grakn Cluster)
  We updated the protobuf definitions for use by Grakn Cluster.

• Migrate and update reasoning behaviour tests from 1.8
  We implement the required steps to run reasoning BDD (called "resolution" features for now). We also bring in and update all the test and BUILD files required to execute reasoning BDD scenarios.

• Outline unimplemented rule validation requirements
  Add comment on rule validation procedure to be implemented in detail.

Please refer to full release notes of 2.0.0-alpha to see the changes in 2.0.0.