Features

• #3765: Add support for entities with roles and grouping of entities in Rasa NLU.

  You can now define a role and/or group label in addition to the entity type for entities.
  Use the role label if an entity can play different roles in your assistant.
  For example, a city can be a destination or a departure city.
  The group label can be used to group multiple entities together.
  For example, you could group different pizza orders, so that you know what toppings goes with which pizza and
  what size which pizza has.
  For more details see entities-roles-groups.

  To fill slots from entities with a specific role/group, you need to either use forms or use a custom action.
  We updated the tracker method get_latest_entity_values to take an optional role/group label.
  If you want to use a form, you can add the specific role/group label of interest to the slot mapping function
  from_entity (see forms).

  Composite entities are currently just supported by the `diet-classifier` and `CRFEntityExtractor`.

• #5465: Update training data format for NLU to support entities with a role or group label.

  You can now specify synonyms, roles, and groups of entities using the following data format:
  Markdown:

  [LA]{"entity": "location", "role": "city", "group": "CA", "value": "Los Angeles"}
  

  JSON:

  "entities": [
      {
          "start": 10,
          "end": 12,
          "value": "Los Angeles",
          "entity": "location",
          "role": "city",
          "group": "CA",
      }
  ]
  

  The markdown format [LA](location:Los Angeles) is deprecated. To update your training data file just
  execute the following command on the terminal of your choice:
  sed -i .deprecated -E 's/\[(.*)\]\((.*):(.*)\)/\[\1\]\{"entity": "\2", "value": "\3"\}/g' nlu.md

  For more information about the new data format see training-data-format.

Improvements

• #2224: Suppressed pika logs when establishing the connection. These log messages
  mostly happened when Rasa X and RabbitMQ were started at the same time. Since RabbitMQ
  can take a few seconds to initialize, Rasa X has to re-try until the connection is
  established.
  In case you suspect a different problem (such as failing authentication) you can
  re-enable the pika logs by setting the log level to DEBUG. To run Rasa Open
  Source in debug mode, use the --debug flag. To run Rasa X in debug mode, set the
  environment variable DEBUG_MODE to true.

• #3419: Include the source filename of a story in the failed stories

  Include the source filename of a story in the failed stories to make it easier to identify the file which contains the failed story.

• #5544: Add confusion matrix and "confused_with" to response selection evaluation

  If you are using ResponseSelectors, they now produce similiar outputs during NLU evaluation. Misclassfied responses are listed in a "confused_with" attribute in the evaluation report. Similiarily, a confusion matrix of all responses is plotted.

• #5578: Added socketio to the compatible channels for reminders-and-external-events.

• #5595: Update POST /model/train endpoint to accept retrieval action responses
  at the responses key of the JSON payload.

• #5627: All Rasa Open Source images are now using Python 3.7 instead of Python 3.6.

• #5635: Update dependencies based on the dependabot check.

• #5636: Add dropout between FFNN and DenseForSparse layers in DIETClassifier,
  ResponseSelector and EmbeddingIntentClassifier controlled by use_dense_input_dropout config parameter.

• #5646: DIETClassifier only counts as extractor in rasa test if it was actually trained for entity recognition.

• #5669: Remove regularization gradient for variables that don't have prediction gradient.

• #5672: Raise a warning in CRFEntityExtractor and DIETClassifier if entities are not correctly annotated in the
  training data, e.g. their start and end values do not match any start and end values of tokens.

• #5690: Add full_retrieval_intent property to ResponseSelector rankings

• #5717: Change default values for hyper-parameters in EmbeddingIntentClassifier and DIETClassifier

  Use scale_loss=False in DIETClassifier. Reduce the number of dense dimensions for sparse features of text from 512 to 256 in EmbeddingIntentClassifier.

Bugfixes

• #5230: Fixed issue where posting to certain callback channel URLs would return a 500 error on successful posts due to invalid response format.

• #5475: One word can just have one entity label.

  If you are using, for example, ConveRTTokenizer words can be split into multiple tokens.
  Our entity extractors assign entity labels per token. So, it might happen, that a word, that was split into two tokens,
  got assigned two different entity labels. This is now fixed. One word can just have one entity label at a time.

• #5509: An entity label should always cover a complete word.

  If you are using, for example, ConveRTTokenizer words can be split into multiple tokens.
  Our entity extractors assign entity labels per token. So, it might happen, that just a part of a word has
  an entity label. This is now fixed. An entity label always covers a complete word.

• #5574: Fixed an issue that happened when metadata is passed in a new session.

  Now the metadata is correctly passed to the ActionSessionStart.

• #5672: Updated Python dependency ruamel.yaml to >=0.16. We recommend to use at least
  0.16.10 due to the security issue
  CVE-2019-20478 which is present in
  in prior versions.

Miscellaneous internal changes

• #5556, #5587, #5614, #5631, #5633