microsoft/msticpy v2.2.3

on GitHub

Some new data-related features in this release.

• Support for the new (still in preview at time of writing) Dynamic Summaries feature of MS Sentinel
• Added ability to create and use "ad-hoc" parameterized queries for data providers
• Simple search mechanism for finding queries
• Support for JSON queries for CyberReason

Support for Microsoft Sentinel Dynamic Summaries

Dynamic Summaries are a Sentinel feature that allow you to persist results of
query jobs in a summarized/serialized form. This might be useful for keeping
results of daily watch jobs, for example. We will be using it in MSTICPy notebooks
to publish more complex result sets from automated notebook runs.

MSTICPy operations available include:

• Retrieve list of current dynamic Summaries
• Retrieve a full dynamic summary
• Create a dynamic summary
• Delete a dynamic summary
• Update an existing dynamic summary

Examples:

# list dynamic summaries
sentinel.list_dynamic_summaries()

# create a dynamic summary in Sentinel
sentinel.connect()
sentinel.create_dynamic_summary(
    name="My_XYZ_Summary",
    description="Summarizing the running of the XYZ job.",
    data=summary_df,
    tactics=["discovery", "exploitation"],
    techniques=["T1064", "T1286"],
    search_key="host.domain.dom",
)

The MSTICPy support also includes a DynamicSummary class that lets you
manipulate dynamic summary objects more easily

  # can also import the class directly
  # from msticpy.context.azure.sentinel_dynamic import DynamicSummary
  # dyn_summary = DynamicSummary(....)
  # This example shows using the "factory" method - new_dynamic_summary
  dyn_summary = sentinel.new_dynamic_summary(
      summary_name="My new summary",
      summary_description="Description of summary",
      source_info={"TI Records": "misc"},
      summary_items=ti_summary_df,
  )
  # Add the local summary object to add to the Sentinel dynamic summaries.
  sentinel.create_dynamic_summary(dyn_summary)

# Retrieve a dynamic summary from Sentinel
dyn_summary = sentinel.get_dynamic_summary(
      summary_id="cea27320-829c-4654-bbf0-b14367483418"
)
# the return value is a DynamicSummary object
dyn_summary

  DynamicSummary(id=cea27320-829c-4654-bbf0-b14367483418, name=test2, items=0)

By default get_dynamic_summary returns the header data for the summary.

The next example shows how you can also fetch full data for the dynamic
summary (by adding summary_items=True). From the returned object,
you can convert the summary items to a pandas DataFrame.

Note: fetching summary items is done via the Sentinel QueryProvider
since the APIs do not support retrieving these.

    dyn_summary = sentinel.get_dynamic_summary(
        summary_id="cea27320-829c-4654-bbf0-b14367483418",
        summary_items=True
    )

dyn_summary.to_df()

You can also create dynamic summaries from a DataFrame and append
DataFrame records to an existing dynamic summary.

Read the full documentation in MSTICPy Sentinel Dynamic Summaries doc

New QueryProvider API to dynamically add a parameterized query.

MSTICPy has always supported the ability to run ad hoc text queries for different providers
and return the results as a DataFrame. Using a static query string like this is quick and easy
if you only want to run a query once but what if you want to re-run with different time
range or host name? A lot of tedious editing or string search/replace!

Adding a full query template to MSTICPy, on the other hand, is overkill for this kind of thing.
Dynamic parameterized queries are especially suited for notebooks - you can create an
in-line parameterized query and have it update with the new parameters every time
you run the notebook.

To use dynamic queries - define the query with parameter placeholders (delimited
with curly braces "{" and "}"), then create parameter objects (these handle any special
formatting for datetimes, lists, etc.).
You add the list of parameter objects along with the replaceable parameter values
when you run the query, as shown below.

# intialize a query provider
qry_prov = mp.QueryProvider("MSSentinel")

# define a query
query = """
SecurityEvent
| where EventID == {event_id}
| where TimeGenerated between (datetime({start}) .. datetime({end}))
| where Computer has "{host_name}"
"""
# define the query parameters
qp_host = qry_prov.Param("host_name", "str", "Name of Host")
qp_start = qry_prov.Param("start", "datetime")
qp_end = qry_prov.Param("end", "datetime")
qp_evt = qry_prov.Param("event_id", "int", None, 4688)

# add the query
qry_prov.add_custom_query(
    name="get_host_events",
    query=query,
    family="Custom",
    parameters=[qp_host, qp_start, qp_end, qp_evt]
)

# query is now available as
qry_prov.Custom.get_host_events(host_name="MyPC"....)

See Dynamically Adding Queries in MSTICPy Docs

QueryProvider - Query Search

As the number of queries for some providers grows, it has become more difficult to quickly
find the right query. We've implemented a simple search capability that lets you search
over the names or properties of queries. It takes four parameters:

• search - search terms to look for in the
  query name, description, parameter names, table and query text.
• table - search terms to match on the target table of the query.
  (note: not all queries have the table parameter defined in their metadata)
• param - search terms to match on a parameter name
• case - boolean to force case-sensitive matching (default is case-sensitive).

The first three parameters can be a simple string or an iterable (e.g. list, tuple)
of search terms. The search terms are treated as regular expressions. This
means that a the search terms are treated as substrings (if no other
regular expression syntax is included).

Find all queries that have the term "syslog" in their properties

    qry_prov.search("syslog")
    # equivalent to qry_prov.search(search="syslog")

    ['LinuxSyslog.all_syslog',
    'LinuxSyslog.cron_activity',
    'LinuxSyslog.list_account_logon_failures',
    ...

See Search queries in MSTICPY Docs

Support for JSON queries in Data Providers

@FlorianBracq has updated the CyberReason data provider so that it supports JSON queries. The
mechanism that we used for KQL and SQL queries breaks JSON since it is a simple string substitution.
Other data sources that use JSON queries include Elastic - we are planning to leverage the same
mechanism to support parameterized Elastic queries in a future release.
Thanks @FlorianBracq!

What Else has Changed?

• Kql query formatting by @FlorianBracq in #595
• Fix minor linting issues in main by @petebryan in #604
• Updated M365D and MDE data connectors with correct scopes when using delegated auth. by @petebryan in #580
• Ianhelle/remove extranous nb 2022 11 28 by @ianhelle in #588
• Enable native JSON support for Data Providers + move Cybereason driver to native JSON by @FlorianBracq in #584
• Adding query search to data_providers.py by @ianhelle in #587
• Fix typo by @FlorianBracq in #606
• Ianhelle/mypy cache 2023 01 17 by @ianhelle in #608
• Added API to QueryProvider to add a custom query at runtime by @ianhelle in #586
• Bump sphinx from 5.3.0 to 6.1.3 by @dependabot in #605
• Bump httpx from 0.23.0 to 0.23.3 by @dependabot in #607
• Dynamic Summaries Sentinel API and DynamicSummary class. by @ianhelle in #593
• Update sentinel_analytics.py list_alert_rules API version. by @pensivepaddle in #592

Full Changelog: v2.2.0...v2.2.3