github microsoft/msticpy v1.6.1
v1.6.1 - Sentinel APIs, Folium, VirusTotal, ProcessTree, and more

latest releases: v2.13.1, v2.13.0, v2.12.0...
2 years ago

In case you're wondering - we never released 1.6.0 to PyPI so this is the official 1.6.0 release.

Summary

Adding Clustering, SubGrouping, Geohash decoding and Layering to FoliumMap class (#227)

The FoliumMap class now supports layering and subgrouping. This was contributed by @tj-senserva
(from Senserva)

Adding New Features to Sentinel APIs (#280)

Sentinel APIs has had signficant rework and expansion to cover new public APIs. These include Analytics and Watchlists as
well as Incidents and Bookmarks from earlier releases. Each API now includes modification
operations such as Create, Delete, Update (not all types support all types of modification).

Note: the module and class names drop the "Azure" prefix following the
renaming of Azure Sentinel to Microsoft Sentinel

VirusTotal FileBehavior, FileBrowser and Pivot functions (#260)

The VirusTotal V3 module has been expanded to add additional operations and data visualization:

  • VT FileBehavior class lets you download and view file detonation data.
  • VT ObjectBrowser lets you download and view basic attributes of a file object
  • Pivot functions - these expose the VT3 Relationship APIs allowing you to pivot between
    related IPs, URLs, Files and Domains.

Added instance handling to Dataproviders (#218)

Several providers now support instances - e.g. Splunk, Sumologic can support multiple
configuration entries in msticpyconfig.yaml with the use of an instance suffix, e.g.
Spunk-Env1, Splunk-Evn2.
This change also makes it possible to combine these instances into a single aggregate
data provider and run the same query across the combined instances. This also
works for multiple Microsoft Sentinel workspaces.

Process tree visualization updates to make schema-agnostic (#256)

The process tree visualization code previous had several dependencies on Sentinel or MDE
schema specifics. The code should now accept process event data sets from many sources.

Adding Sysmon ProcSchema (#267) - A schema mapping for Sysmon data was contribute by @nbareil

Build pipeline fixes for external forks (#270)

Previously builds triggered from external forks would fail for a variety of reasons
(no access to pipeline secrets, MS-Internal-only build actions). This is now fixed so that
external contributors should expect a clean build.

Added default caching option for mordor driver (#257)

The OTRF Security Datasets driver and browser now cache security data set and MITRE ATT&CK
data for quicker startup. By default, the cache files are stored in $HOME/.msticpy/mordor
or the location specified in the Dataproviders/Mordor section of the msticpyconfig.

What's Changed

  • Ianhelle/proc tree fixes 2022 01 03 by @ianhelle in #256
  • Bump sphinx from 4.2.0 to 4.3.2 by @dependabot in #241
  • Adding Clustering, SubGrouping, Geohash decoding and Layering to FoliumMap class by @tj-senserva in #227
  • 20220105 hackathon blob storage conn string by @lucky-luk3 in #262
  • Added instance handling to Dataproviders by @ianhelle in #218
  • Added default caching option for mordor driver by @ianhelle in #257
  • Ianhelle/virustotalv3 additions 2021 10 22 by @ianhelle in #260
  • Initial attempt at masking failing tests for external forks. by @ianhelle in #270
  • Adding Sysmon Process Create's ProcSchema by @nbareil in #267
  • Removing validate configuration for external forks by @ianhelle in #273
  • Update AzureData to use correct Azure subscription management API by @liamkirton in #269
  • Added details to README on upgrading msticpy by @danielc-evans in #274
  • Updated IPython elements and requirements by @petebryan in #278
  • Fixes and improvements to KqlDriver and KustoDriver Azure auth. by @liamkirton in #277
  • Miscellaneous fixes for VT, Auth and other items by @ianhelle in #281
  • Adding New Features to Sentinel APIs by @petebryan in #280
  • Correcting error introduced in nbinit refactoring by @ianhelle in #286

New Contributors

Additional feature details

7fa1598@

  • Add ability to filter queries by substring in list_queries in data_providers.py
  • Adding coordinates property to GeoLocation entity geo_location.py
  • Simplify verbose option and refactoring complex method in init_notebook
    c0a32a0@Merge pull request #277 from liamkirton/liamkirton/kql-kusto-driver-auth
  • Fixes and improvements to KqlDriver and KustoDriver Azure auth.
    ae6bcd7@Updated IPython elements and requirements (#278)* Updated IPython elements and requirements
    79910e2@Merge pull request #274 from danielc-evans/updated-readmeAdded details to README on upgrading msticpy
    7038e28@Changed type hints in enum_parse in utility.py
  • Added autoload of Pivot and vt_pivot (if VT config detected)
  • Moved some common methods to IPyDisplayMixin class in nbwidgets.py
  • Added extra parameter for VT Private API definition to mpconfig_defaults.yaml. Also changed defaults for some providers to Primary=False
  • Updates to test_mp_config_controls.py to deal with new VT parameter
  • Add fixed vtobject_browser.py + unit test in test_vtlookupv3.py
    8308849@Added the ability to connect to a blob storage using the connection string. Container name parameter changed to optional.

Additional fix details

7fa1598@Miscellaneous fixes for VT, Auth and other items (#281)* Updates to Timeseries analysis and plotting modules

  • Avoiding exception when cmdline is NaN or non-string in process_tree.py
  • Removing "Authenticating to Azure" output since it fires on every query.
  • Removing credential caching in azure_auth_core.py

f730ba2@Moved geoip GeoLiteLookup to check for and update DB before first query (rather than in init)
Added full copy of GeoLite2-City.mmdb to test data
Linting fixes in mordor_driver.py and test-test_mordor_driver.py
Cherry picked commit for python-package and azure-pipelines yamls

569eb75@Forcing install of pandas>=1.30 in pytest block
857b08e@Saved mde_proc_pub.pkl as pickle protocol 4 to work in Python 3.6
70b5c55@Adding updated notice file.
4599d6e@Fixed use of hard-code global endpoint in azure_blob_storage.py

cad8256@Added create incident and create rule features
70b5c55@Adding upgrade to pandas to so that it uses latest version and avoids pickle compatability issue with test data pickled with pandas > 1.2.5Also added higher version constraint to requirements-dev.txt and conda-reqs-dev.txt
Adding updated notice file.

4599d6e@Added instance handling to Dataproviders (#218)

  • Fixed use of hard-code global endpoint in azure_blob_storage.py
  • Adding better handling of response data in kql_driver.py
  • Removing default start and end parameters from queries. We should rely on the builtin querytimes control
  • Adding issue tracker URL and additional classifiers to setup.py
  • Updating import_analyzer.py to include resolved and unresolved paths for determining whether a package belongs to the std library. (these paths may be different if some of them are links)
    0b9c81a@Bump sphinx from 4.2.0 to 4.3.2 (#241)

Full Changelog: v.1.5.2...v1.6.1

Don't miss a new msticpy release

NewReleases is sending notifications on new releases.