Summary

This release includes two minor changes:

1. Fixed schema for Microsoft Defender Device Process events - this should allow it to work with the Process Tree visualization
2. Single-sign-on in Azure ML notebooks using MSI

Fixes and New Features

MDE Process Tree

The schema originally deployed for generating process tree data from Microsoft Defender for Endpoint data was incorrect - we were using an internal schema that did not match the data retrieved via the Defender APIs.
This has now been fixed along with some related items:

• DateTime fields returned in the Defender data are automatically converted to pandas Timestamps (previously they were left as strings)
• The process tree and other visualizations will automatically convert required timestamp columns to datetime format - this is useful where data is retrieved from an API or file where the required datetime columns are stored as date strings

Single Sign-On for notebooks in Azure ML (in Microsoft Sentinel)

When you sign on an Azure ML workspace, Managed Identity (MSI) credentials are automatically enabled on the Azure Computes used to run the notebooks. The MSI credentials use the user principal of the user that has logged on (not the machine identity). This allows processes on the Computes to use these credentials to authorize to other Azure resources.
The Log Analytics team made some changes to allow the Managed Identity credentials from Azure ML to authorize to Log Analytics/Microsoft Sentinel data API.

With this release, connecting to Microsoft Sentinel will automatically try to use the MSI credentials. Existing authentication flow options such as Azure CLI and Device Code authentication are still available.

To use MSI-based SSO:

qry_prov = QueryProvider("AzureSentinel")   # we will change this to MicrosoftSentinel soon!
qry_prov.connect(WorkspaceConfig())

To force specific authentication flavors, you can set your preferences explicitly in the msticpconfig.yaml. You can also do this at the command line.

To specify an explicit set of authentication types to use (they are tried in the order specified):

qry_prov.connect(WorkspaceConfig(), mp_az_auth=["cli", "interactive"])

To skip integrated Azure authentication altogether and use interactive Device Code authentication:

qry_prov.connect(WorkspaceConfig(), mp_az_auth=False)
# or
qry_prov.connect(WorkspaceConfig(), mp_az_auth=["interactive"])

Detailed changes

4fdf286@Ianhelle/mde proctree fixes 2021 12 16 (#239)* Changes:

• Removed checking of DataFamily in data_query_reader.py
• Changed default behavior of kql_driver.py to use Azure authentication
• Change security_base.py to use entities.OSFamily rather than query_defns.DataFamily
• Added ensure_df_datetimes function to auto-convert specified columns from string to datetime
• Added unit test test_query_defns.py
• Added ensure_df_datetime to timeline.py
• Added ensure_df_datetime to timeline_duration.py
• Added ensure_df_datetime to proc_tree_build_mde.py
• Added ensure_df_datetime to proc_tree_build_winlx.py
• Changed mpconfig_defaults.yaml to include "msi"
• Switching "Microsoft Sentinel" to "Azure Sentinel" in UI/messages.
• Updated black version in .pre-commit-config.yaml
• Using returned schema in mdatp_driver.py to auto-format datetimes to pandas timestamp
• Returning full response from query_with_results in odata_driver.py
• Some fixes in convert_mde_schema_to_internal - field naming and using tz-aware Unknown time value
• Moved the proc tree schema code from proc_tree_builder.py to proc_tree_schema.py
• Add mocked az_connect to test_kql_driver.py tests - since the change to defaulting to AZ auth in the kql_driver means that tests fail.
• Added test case for MDE public data - mde_proc_pub.pkl
• Adding tooltips to settings editor buttons
• also changing button text for simple setting editor to "Update" (from "Save")
• changing order of checks in base64unpack to let you specify UTF-16 decoding
• updating version to 1.5.1