Release Summary

There are several cool new features in this release:

Microsoft Sentinel Incident exploration
RiskIQ Threat Intellligence provider and pivot functions
Microsoft Defender 365 API/Data provider support
Preview Azure Data Explorer/Kusto data provider support
Microsoft graph provider now supports sovereign clouds
ProcessTree visualization support for Microsoft Defender data in Microsoft Sentinel

Microsoft Sentinel Incident Exploration

0037dd6@Merge pull request #211 from microsoft/pebryan/2021-8-18_IncidentTriage

Provides new features to support Incident Triage scenarios (as included in the new Microsoft Sentinel Incident Triage notebook Azure/Azure-Sentinel-Notebooks#120).

Entity Graph Visualization

This new visualization allows for the graphing of an investigation including any alerts, incidents or entities associated with that investigation.
The graph can be created, updated and visualized either as a graph or with a graph and a timeline of the graph event times.

from msticpy.vis.entity_graph_tools import EntityGraph
graph = EntityGraph(incident)
graph.plot()

The Entity Graph can also be created from a dataframe of events, either by passing in the dataframe or as a Pandas accessor.

df.mp_plot.incident_graph(timeline=True)

Microsoft Sentinel Incident API support

RiskIQ PassiveTotal TI provider and Pivot provider

9353d9f@RiskIQ threat intel provider & pivots (#175)

Delivers a new "RiskIQ" threat intel provider, enabling access to RiskIQ Summary Card and Reputation
datasets for IPs and hostnames with the same syntax and response format used with other TI Providers.

ti_lookup = TILookup()
lookup_result = ti_lookup.lookup_ioc(observable="137.184.153.130", providers="RiskIQ")
ti_lookup.result_to_df(lookup_result).T

Introduces a new mechanism for registering MSTICPy Pivot Functions provided by threat intel providers,
then uses that code to register capabilities for nearly all RiskIQ API endpoints on hostnames, FQDN and
IPAddress entities. These pivot functions enable easy integration of RiskIQ-provided intelligence in
existing pivot pipelines, and makes it easy to build new pipelines leveraging unique RiskIQ datasets.

(
    Dns.RiskIQ.resolutions("jquery.su")
    .query("recordtype=='A'")
    .mp_pivot.run(Dns.RiskIQ.reputation, column='resolve', join='left')
    .mp_pivot.run(IpAddress.RiskIQ.resolutions, column='resolve', join='left')
)

Preview support for Kusto (Azure Data Explorer)

98864aa@Ianhelle/kusto and md365 support 2021 11 01 (#225)

This release contains a preview (i.e. not fully baked) of support for Kusto cluster queryingand Microsoft Defender 365 data providers.
The Kusto/Azure Data Explorer support is based on the existing Microsoft Sentinel provider.

kusto_provider = QueryProvider("Kusto")
kusto_provider.connect(
    "azure_data-Explorer://code;cluster='my_cluster';database='my_db';alias='my_alias'"
)

Some more documentation to follow about configuration in msticpyconfig and
how to build template queries for Kusto/Azure Data explorer.

Support for Microsoft 365 Defender data providers

This is slight expansion of our current support for MS Defender for endpoint. Using the new "M365D" provider name
will cause the provider to use the new combined Microsoft Defender APIs, giving you consolidated access
to data from multiple Defender services.

To create a provider for Microsoft 365 Defender:

m365d_provider = QueryProvider("M365D")
m365d_provider.connect()

All template queries for that were available for MDE are also loaded for M365D.

Using Microsoft Defender for Endpoint is still supported. To use only
the Microsoft Defender for Endpoint APIs use "MDE".

You can specify your connection parameters in msticpconfig.yaml

DataProviders:
  # ...
  MicrosoftDefender:
    Args:
      ClientId: 66b9818a-26cd-4584-8eb0-7f7a499242aa
      ClientSecret:
        KeyVault:
      TenantId: 8360dd21-0294-4240-9128-89611f415c53

This configuration expects to find the client secret in your configured Key Vault with the name
"DataProviders-MicrosoftDefender-Args-ClientSecret" - the msticpy config editor will create this for you.
You can store the secret in your config file but, obviously, we would not recommend this.

You can also specify configuration/secret values in the call to m365d_provider.connect().
For more information see the Defender provider MSTICPy documentation

Other improvements

The Graph provider now supports sovereign clouds as well as the Azure global cloud.
You can configure settings for MS Defender and MS Graph via the MSTICPy settings editor.
If you have configured settings for these you can now connect with a simplified
syntax, with no parameters required.

graph_prov = QueryProvider("SecurityGraph")
graph_prov .connect()

The schema for the configuration is the same format as the Defender format described above.

DataProviders:
  # ...
  MicrosoftGraph:
    Args:
      ClientId: 66b9818a-26cd-4584-8eb0-7f7a499242aa
      ClientSecret:
        KeyVault:
      TenantId: 8360dd21-0294-4240-9128-89611f415c53

Process Tree visualization supports MS Sentinel schema for MDE process data

The schema of the DeviceProcessEvents table in Microsoft Sentinel is different
from the data queried directly from the MS Defender advance hunting API. This
update allows you to query MS Defender process data from Azure Sentinel and
display in the Process Tree visualization.

New CI Pipelines using GitHub Actions and Azure Pipelines

The old Azure DevOps pipeline has been decommissioned and replaced by two new ones - GitHub actions and an improved version of the Azure DevOps pipeline.

The main advantage of the GitHub actions pipeline is that the results
of tests and code checks are visible to external contributors.

However, there are some component governance and security checks that
we are required to run by Microsoft but which are not currently available
in GitHub actions. We've opted to run both in parallel.

The new Azure DevOps pipeline is more sophisticated and comprehensive than
the previous one. It allows us to run builds and tests on Windows, Linux and
Mac platforms and to cover Python versions 3.6 and 3.8.
(Note: we will likely switch to 3.8 and 3.10 in the near future)

Other changes and fixes