github microsoft/msticpy v2.5.0
on GitHub

latest releases: v2.13.1, v2.13.0, v2.12.0...
16 months ago

Summary of main changes

  • New MS Sentinel and Azure Kusto drivers/data providers - these include support for multi-threaded parallel queries, proxies and user-defined query timeouts.
  • Extensibility model for MSTICPy - you can create private data providers, TI and Context providers and load them into MSTICPy alongside the built-in providers.
  • MS Sentinel repo query download - add current detection and hunting queries from the Sentinel repo as Sentinel queries runnable from MSTICPy/notebooks
  • OSQuery data provider - makes it easy to import OS Query logs to dataframes to do additional processing/analysis on them.
  • Panel tabulator now supported as default data viewer (a million times better than the one we built!)

More details on these changes below

Sentinel and Kusto provider new drivers

This change adds replacement drivers for the MSSentinel and Kusto data providers.
In place of Kqlmagic, these drivers use the azure-kusto-data and azure-monitor-query SDKs, respectively.

Currently these drivers are enabled alongside the existing versions - in a future version we will make these the defaults for Sentinel and Kusto.

Some of the main changes with these new versions:

  • They use the provider names MSSentinel_New and Kusto_New when creating a QueryProvider instance.
  • Both drivers support setting proxies for firewall-protected networks
  • Both drivers support custom configuration of the server timeout via a timeout parameter
  • Both drivers use integrated Azure authentication by default and support the auth_types and tenant_id parameters used elsewhere
    in MSTICPy
  • Both drivers support threaded execution for parallelizing queries (across multiple workspaces/clusters or split by time) - this functionality, however, will be exposed in v2.6.0 via a separate feature.
  • The MSSentinel_New driver allows you to execute the same query across multiple workspaces in parallel and returns the results as a combined dataframe.
  • Some of the previous parameters have been deprecated:
    • mp_az_auth is replaced by auth_types (the former still works but will be removed in a future release).
    • mp_az_auth_tenant_id is replaced by tenant_id (the former is not supported in the new providers).

Note: in order to use these new versions you must have the azure-kusto-data and/or azure-monitor-query Python packages
installed. You can install these using pip install msticpy[azure] or install them separately using pip.

For more details on how to use these providers, see:

Changes specific to the MS Sentinel provider

Connecting to multiple workspaces allows you to run queries across these workspaces and return the combined results as a single Pandas DataFrame. The workspaces must use common authentication credentials and should have the same data schema. 

# use workspace names if these workspaces are configured in msticpyconfig.yaml
qry_prov.connect(workspaces=["Default", "MyOtherWorkspace"])

# or use a list of workspace IDs
qry_prov.connect(workspaces=["e6b4bc15-119b-45a2-8f3d-c39ed384ed37", "b17e0e5a...."])

# run query against connected workspaces
qry_prov.SecurityAlert.list_alerts()

Changes specific to the Kusto provider

  • The settings format has changed (although the existing format is still supported albeit with some limited functionality).
    See the Kusto provider documentation for details.
  • In the earlier implementation of driver you can specify a new cluster to connect to in when executing a query. This is no longer supported.
    Once the provider is connected to a cluster it will only execute queries against that cluster. (You can however, call the connect() function to
    connect the provider to a new cluster before running the query.)
  • Filtering pre-defined queries by cluster. If you have MSTICPy query definitions for the Kusto provider, these will all be attached as methods
    of the QueryProvider, when it is created. However, as soon as you connect to a specific cluster, the queries will be filtered down to show
    only the queries that are intended to run on that cluster.
  • New APIs (exposed via the query_provider):
    • get_database_names() - return list of databases for the connected cluster
    • get_database_schema() - return table schema for a database in the cluster
    • configured_clusters() - return a list of clusters configured in msticpyconfig.yaml
    • set_cluster() - switch connected to cluster to a different one (you can use the connect method to do this, which also lets you specify
      additional connection parameters).

Extend MSTICPy with Data provider, TI provider and Context provider plugins

This adds the ability to "side-load" data providers, TI providers and context providers. If you have a data/TI/context source that you want to use in MSTICPy you can write a provider (deriving from one of the base provider classes) and tell MSTICPy where to load it from.

In a future release we'll build on this framework to let you install plugins from external packages and provide some cookie-cutter templates to generate skelton provider classes.

Writing a TI provider or Context provider (partial example) 

    class TIProviderHttpTest(HttpTIProvider):
        """Custom IT provider TI HTTP."""

        PROVIDER_NAME = "MyTIProvider"
        _BASE_URL = "https://api.service.com"
        _QUERIES = _QUERIES = {
            "ipv4": APILookupParams(path="/api/v1/indicators/IPv4/{observable}/general"),
            "ipv6": APILookupParams(path="/api/v1/indicators/IPv6/{observable}/general"),

Telling MSTICPy to load the plugins

Load on demand 

    import msticpy as mp

    mp.load_plugins(plugin_paths="/my_modules")

    # or multiple paths
    mp.load_plugins(
        plugin_paths=["./my_modules", "./my_other_modules"]
    )

Or specify in msticpyconfig.yaml 

        ...
        Custom:
            - "testdata"
    PluginFolders:
        - tests/testdata/plugins
    Azure:
        ...

See the new Extending Msticpy section in our docs.
If you want to contribute any of the drivers you write, also check out the new Development section in the MSTICPy docs.

OS Query Provider

Great contribution from @juju4 here (with a bit of collaboration with @ianhelle).
Create a MSTICPy QueryProvider with the data environment name "OSQueryLogs" and load forensic logs from OSQuery. 

# specify one or more paths to folders where the dumped JSON OSQuery logs can be found
qry_prov = mp.QueryProvider("OSQueryLogs", data_paths=["~/logs1", "~/logs2"])
qry_prov.connect()
qry_prov.list_queries()
['osquery.acpi_tables',
'osquery.device_nodes',
'osquery.dns_resolvers',
'osquery.events',
'osquery.fim',
'osquery.last',
'osquery.listening_ports',
'osquery.logged_in_users',
'osquery.mounts',
'osquery.open_sockets',
...

Each event type is available as a separate function that returns a pandas DataFrame with the combined events from the logs for that type 

qry_prov.osquery.processes()

Downloading Sentinel Detection and Hunting queries for the Sentinel Query Provider

We haven't finished documenting this or integrating it fully, so will leave the full announcement of this until the next release. If you want to play around with it look at the following module: 

from msticpy.data.drivers.sentinel_query_reader import download_and_write_sentinel_queries

download_and_write_sentinel_queries(
    query_type="Hunting",    # or "Detections"
    yaml_output_folder="./sentinel_hunting",
)
qry_prov = mp.QueryProvider("Sentinel_New", query_paths=["./sentinel_hunting"])

Since there are lots of queries, the import might take a little while in its current form.

Panel Tabulator now available as a DataViewer control.

HoloViz Panel is a powerful Bokeh-based data exploration & web app framework for Python. It has an immense amount of functionality that you can read about at the Panel documentation site. You need to have panel installed for the Tabulator-based viewer to run (pip install panel).

Unfortunately, the documentation for our Tabulator view never made it into this release but most of the functionality should be obvious from the UI. There are some useful load-time parameters that you can use at startup for things like:

  • selecting an initial column set.
  • adding columns to a per-row expando pane - useful for viewing long column values such as command-line.

We also kept the column chooser widget from the previous data viewer so that you can interactively select which columns to display. The Tabulator MSTICPy initialization parameters are documented in the code.

Most of the Tabulator init parameters are also passed through to the underlying control - which give you an immense amount of control over the viewer. These are described in the Panel Tabulator documentation

Big thanks to our contributors in this release!

@juju4
@jannieli
@ianhelle
@Tatsuya-hasegawa
@FlorianBracq
@danielyates2
@petebryan
@ashwin-patil

What's Changed PR Reference

New Contributors

Full Changelog: v2.4.0...v2.5.0

Check out latest releases or
releases around microsoft/msticpy v2.5.0

Don't miss a new msticpy release

NewReleases is sending notifications on new releases.

Get notifications