Highlights

The highlights of this release (which is really 1.2.0 with some last-minute fixes) are: two new data providers for Azure Resource Graph and Sumologic and a DataViewer control for pandas dataframes.

Azure Resource Graph provider

The Azure Resource graph
provider lets you query Azure resources using KQL queries. This works much like our other
query providers and gives you a lot more flexibility in navigating around Azure resources
than the current AzureData provider.

Explore more in the Resource Graph Driver notebook and the online docs

Many thanks to Ryan Cobb @rcobb-scwx for creating and contributing this.

Sumologic provider

Sumo Logic Inc. , is a cloud-based machine data analytics service
focusing on security, operations and BI use cases. This provider allows you to connect to
and query your data from MSTICPy via their Search API
Explore more in the Sumologic Data Connector notebook

Many thanks to Julien (@juju4) for building and contributing this.

DataViewer

The data viewer uses the [Bokeh DataTable control[(https://docs.bokeh.org/en/latest/docs/user_guide/interaction/widgets.html#datatable)
to display and browse through data in a pandas DataFrame. It lets you sort by column, choose which columns to display and filter by multiple columns. It keeps a synchronized copy of the DataFrame with column choice and filtering applied, so that you can always access the data as it appears in the control.

Explore more in the DataViewer notebook
and the Dataview online docs

New Features and Updates

ab6eb73 - (#164) DataViewer control

• DataViewer control - Bokeh based data table control
• Documentation in DataViewer.rst
• Notebook in DataViewer.ipynb

3ab31ad- Azure Resource Graph Data Provider (#167)

• Co-authored-by: Ryan Cobb rcobb@secureworks.com
• Co-authored-by: Pete Bryan peter.bryan@microsoft.com

2a33d61- Sumologic Data Provider (#165)

• Co-authored-by: juju4 juju4@users.noreply.github.com
• Co-authored-by: Ian Hellen ianhelle@microsoft.com

4c4b8ca- Ianhelle/getting started fixes 2021 06 04 (#170)

• Updates to init_notebook in nb_init.py
  • Moved most of the logic of the Azure Sentinel notebooks nb_check script into msticpy - azure_ml_tools
  • Will now create msticpyconfig.yaml if it doesn't exist
  • Output message from init_notebook when msticpyconfig.yaml is created in nbinit.py

8214c90 - Minor fixes to VT TI Provider and TIBrowser (#173)

• Adding QueryTime instance to Dataproviders in data-data_providers.py. Let's you specify time range (or use default 1 day range) for all queries from provider
• Added Browshot to mpconfig_defaults.yaml so that it can be edited in MP settings editor

ab6eb73 (#164)

• added recent blog articles and video to a renamed resources page for RTD - blob_articles.

Fixes

8214c90 - Minor fixes to VT TI Provider and TIBrowser (#173)

• Updating VT TI Provider for better severity accuracy with additional "detected_" types.

• Changing ti_browser to display "information" results if no "warning" or "high" items in dataset.

• Updated compound_ctrls.py and mp_config_file.py to work when azure-keyvault-secrets not installed

• Update to QueryTime to fix default max time when "day" is used for units.

e6ea232- Pebryan/2021 6 7 grey noise updates (#171)

• Update GreyNoise naming and added custom UA

4c4b8ca- Ianhelle/getting started fixes 2021 06 04 (#170)

• Fixing a query error in kql_sent_azuresentinel.yaml
• Fixing an issue with contain naming when creating pivot functions
• Fixing link in TIProviders.rst
• Fixing mypy error in azure_auth_core
• Adding check for KV settings before trying to initialize secrets client in provider_settings.py
• Adding function to search for file in utility.py
• mp_config_edit.py:
  • Renaming "Save File" button to "Save Settings"
  • Add config_filepath parameter to specify file to save when settings not imported from a file
  • Changing item update button to "Update" rather than Save in comp_edit.py
  • Strip extraneous quotes from string "default" value in mp_config_control.py
  • Corrected open params to "w" rather than "w+" (write append)
• Added "browse" as alias for browse_queries in data_providers.py.
• Fixed misreporting of DataFrame column name in exception when column parameter does not match a column in the input data. in pivot_register.py
• Adding azure_ml_tools.py (from nb_check.py script)
• Added "severities='all'" parameter option. Added friendly warning (rather than exception) when no data is present in the input (either empty DF or nothing above the default severities - in ti_browser.py
• Fixing additional areas breaking in timeline plot module if you don't supply "source_columns" parameter.
• Fixing mypy warning in domain_utils.py
• Fixed allowing geolite to create a database folder even if parent folders don't exist.
• Adding "col" and "column" aliases for lookup_iocs in tilookup.py
• Changed score logic for IBM XForce - prevent misreporting of score==1 as warning.
• Updating version in _version.py
• Added test_azure_ml_tools.py
• unit_test_lib updates:
  • Added change_directory context manager (with lock) to allow individual test cases to change directory
  • Fixed TEST_DATA_PATH to always return absolute path
• Fixing pylint warnings about the lack of use of context managers in mordor_driver.py, base64unpack.py, morph_charts.py
• Pylint warnings suppression about lowercase enums in tilookup.py
• Fixing version number format problem in azure_ml_tools
• Fixing warning that wasn't processing an f-string in tilookup.py