Highlights

We've added comprehensive support for Azure sovereign clouds (China, Germany and US Government).
To set the appropriate cloud open MpConfigEdit and choose the Azure tab.

Read more about this here

There are also additions and improvements to the visualizations.

Matrix Plot is an interaction plot letting you visualize the interactions between two sets of entities.
A typical use would be seeing connections between source and destination IP Addresses.

Read more about this here

Another visualization change is a major overhaul of the ProcessTree visualization. This now accepts generic data
sets (not specifically Azure Sentinel) and has special support for Microsoft Defender for Endpoint logs.
The data format should be auto-detected.

Finally, we've also collected our visualization methods into a single pandas accessor - mp_plot.
Using this accessor you can plot any of the visualizations from a pandas dataframe:

df.mp_plot.timeline()
df.mp_plot.process_tree()

New Features

889112d@Matrix plot visualization (#197)* Matrix plot visualization
Includes

• vis/matrix_plot.py - main plotting code
• MatrixPlot.ipynb - notebook
• visualization/MatrixPlot.rst - documentation
• tests/vis/test_matrix_plot.py - unit test
• mp_pandas_plot - generic pandas extension "mp_plot" that includes all
  visualizations

d425c77@Process tree updates to handle more generic data (#178)*

• Tidying up some of the code in timeline.py
• Moving the pd accessors to timeline_pd_accessor.py
• Updating notebook EventTimeline.ipynb and documentation EventTimeline.rst
• Unit tests for timeline controls in test_timeline.py
• Removed line to slim down columns - need to keep original data columns because they may be
  referenced in source_columns, etc.
• Process tree updates for MDE process data
• Updating docs, adding pid_fmt parameter
• Adding explanation of schema to NB and doc
• Adding updated notebook
• Expanded the schema table to show all types.

a00741d@Adding support for multiple Azure Clouds (#193)
Co-authored-by: Pete Bryan pebryan@microsoft.com

• Adding cloud options to settings, kql_driver and keyvault
• Updated help text and combined CESimpleSettings class.
• Azure auth multicloud support
• Replacing hard-coded URIs with msrestazure enums
• Modularize multicloud support
• Refactoring code to use AzureCloudConfig instead of reading directly from config
  • azure_auth.py - uses AzureCloudConfig, added credential_scopes option SubscriptionClient init, spelling corrections
  • azure_auth_core.py - create AzureCloudConfig, and _AzCachedConnect classes to read cloud config values and cache tokens
  • cred_wrapper.py - minor formatting
  • keyvault_settings.py - use AzureCloudConfig
  • ce_common.py - uses AzureCloudConfig in get_def_tenant_id function
  • azure_data.py - uses AzureCloudConfig, added credential_scopes option to client class inits, spelling corrections
  • azure_sentinel.py - uses AzureCloudConfig. added functions to set default subscription and workspace, changed _build_paths to method
  • kql_driver.py - uses AzureCloudConfig, added load_mode == Silent, added mp_az_auth to control using Azure auth instead of native Kqlmagic auth,
  • resource_graph_driver.py - uses AzureCloudConfig, added credential_scopes option SubscriptionClient init
  • test_azure_auth_core - Added test for AzureCloudConfig class
  • test_resource_graph_driver.py, test_azure_data.py, test_azure_sentinel.py - added real credential classes to test
  • msticpyconfig-test.yaml, msticpyconfig.yaml - added Azure sections to test config.
• Update azure-mgmt-subscription and azure_mgmt-resource versions

Fixes

d81801d@Ianhelle/tests and tidy 2021 09 01 (#200)*

• Added separate unit test for nbwidgets to cover cases missed in notebook.
• Removing deprecated code from utility.py
• Removing unneeded function from mordor_driver.py
• Sourcery code refactor in entity.py
• Deprecating create_ip_record function in ip_utils.py
• Split wsconfig tests out of test_pkg_config.py
• Adding test_wsconfig.py. Added new test for couple of utility methods
• Removing redundant import from test_nbwidgets.py
• Moved lots of test files into sub-package folders corresponding to tested modules
• Merged a few tests into single files - test_entity.py and test_timeline.py
• Re-enabling IP_stack/geoip notebook test in test_geoip.py
• Updating use of tqdm to remove deprecated tqdm_notebook in utility.py
• Adding some debugging output to geoip GeoLite class.
• Updating test_geoip.py to use predictable msticpyconfig-test.yaml
• Updating test to restore env var values after test in test_pkg_config.py
• Fixing aliases element in kql query records
• Handle aliases as list or string in help

• Updates from PR review

f58e35c@Changing behavior so that vtlookupv3.py does not throw an exception when an item is not found. (#199)

• This is especially annoying when doing multiple look-ups and one item is not found.
• Minor changes to VTLookupV3.ipynb - to use get_object API from the vtlookupv3 library

d425c77@Process tree updates to handle more generic data (#178)

• Fixing bug in customjs and adding mde test files
• Update NOTICE.txt
• Fixing progress widget to use correct visibility settings.
• Changing user_config to be more defensive against bad settings.
• Error in query_store passing list instead of strings to MsticpyUserConfigError exception

• Adding a bit more text/warning to check the browser.

Co-authored-by: Pete Bryan pebryan@microsoft.com
ff87ebb@cred-scan suppression (#196)