Overview

We've finally decided that we're ready to put a 1.0.0 release number
on MSTICPy and move out of the Beta world.

There are three new features in this release and one major update.

• Flexible dependencies with pip extras
• Settings management and auto-load of components
• An experimental SQL to KQL translator
• Significant updates to the pivoting library.

New Features

MSTICPY Dependencies - Implementing Extras #128

We've split MSTICPy dependencies into "extras". This means that the
base install of MSTICPy does not install every dependency. Previously,
many rarely-used dependencies were installed regardless of whether
they were going to be used or not.
This is couple with a refactoring of the code so that, should you try
to use functionality that does not have a dependency installed, an informative
exception message is displayed telling you which extra you need to install.

• Implemented pip "extras" for msticpy install - drastically reduces
  install time for core msticpy.
• Refactored many modules to emit informative warning if user tries to load functionality
  that requires a different "extra"
• Refactored unit tests to work with missing extras.
• Added pyperclip to pkg dependencies exceptions.
• Added extras documentation to Installing documentation
• Added requirements-all.txt - that will always install all dependencies.
• Added pre-commit hook to generate requirements-all.txt

MSTICPY config settings management #136

The MSTICPy configuration file has grown to be quite complex. We've tried
to address this by creating some interactive tools to let you create and edit
settings using a simple GUI and creating a notebook that walks you through
creating your settings file for the first time.

MSTICPy itself has a number of initialization/loading steps that you need to carry out
before starting to use it in a notebook. The other part of this feature
is the ability to specify, in settings, what to load at initialization.
Components that can be auto-loaded include: DataProviders, TI Providers,
Notebooklets and Pivot functions. These are specified in the msticpyconfig.yaml
(you can edit these settings with the Settings editor) and auto-loaded
when you run init_notebook() at the start of your notebook.

• Flexible UI for configuring MSTICPy settings
• User environment configuration for notebooks - lets you specify (in
  settings), which providers/modules, etc. that should be loaded automatically.
• Added minimal output from nbinit to show imported modules
• Added check_version in init_notebook function to indicate if
  a new version of MSTICPy has been released.
• Added a function to retrieve and show current KV secrets
• Updated MSTICPy Configuration documentation
• Added MSTICPy Settings Editor documenation
• Added MSTICPy Settings notebook
• Added documentation diagram
  Config editor, auto-load, dependencies and Sql2Kql translator

SQL to KQL Translator #132

• Core feature implemented as msticpy.data.sql_to_kql using moz_sql_parser
• Support for limited SparkSQL extensions
• SQL TO KQL Conversion Documentation
• SQLToKql Notebook

Updated Features

Pivot Functions #131

The pivot functionality has been updated with several usability and
documentation improvements. Notable features include:

• Pivot browser, letting you browse/search for pivot functions.
• Persistent Pipelines - you can save pipelined pivot operations in a
  YAML file and run them on input DataFrames

Full list of changes

• Made AzureSentinel and MDE the preferred names for LogAnalytics and MDE drivers.
• Added pivot_browser UI - pivot_browser.py
• Added ability to read pipeline definitions from yaml files - pivot_pipeline.py
• Adding pivot.tee_exec pipeline function - in pivot_pd_accessor.py
• Add ability to add arbitrary/ad hoc functions as pivots - in pivot.py
• Exposing get_timespan function in Pivot class as public function - in pivot.py.
• Added DNS entity to several pivot functions - mp_pivot_reg.yaml
• Fixed some queries for more consistency.
• Pivot data query functions now prefixed with table name.
• Added ability for pivot functions to return raw output.
• Add joins for pivot data queries in pivot_data_queries.py
• Add "print" query debug parameter in data_providers.py
• Add find_entity function in entities init.py
• Add "pivots" attribute (an alias for get_pivot_list) in entity.py
• Add ability to set timespan more flexibly. Calling set_timespan() no longer resets the timespan.
• Add PivotBrowser method to Pivot class - in pivot.py
• Switched engine to "Python" for pd.read_csv in pivot_magic_core.py to handle more formatting types.
• Add positional parameters to pipeline step and cleaned up code in pivot_pipeline.py
• Updated PivotFunctions documentation
• Updated PivotFunctions Notebook
• Added PivotFunctions-Introduction notebook

Miscellaneous Updates and Additions

• added SplunkProvider documentation
• Updated README and package summary section of MSTICPy to align with current features

Fixes

• Updated formatting for new black version
• Remove unneeded code from keyvault_client.py
• Fixed pivot_register_reader to skip classes that cannot be instantiated (e.g. IPStack if user doesn't have API key)
• Additions/corrections to Installing.rst
• Correction to FoliumMap.ipynb - removing dropna from read_csv in FoliumMap notebook
• Adding vt, vt_graph to Sphinx mock list
• Fixed some problems and renamed module locations in notebooks and RST docs.
• Some corrections to documentation in AzureSentinel and DataAcquisition docs.
• some fixes to tests for test_pkg_imports and import_analyzer.py
• fix to config2kv.py to correct some problems
• fix for ipwidgets warning about deprecated on_submit() method
• multiple fixes for typos and duplicate section names in: DataProviders.rst, UploadData.rst, PivotFunctions.rst
• fixed issue in nbinit.py where extra_imports were being lost.
• fix for QueryTime in nbwidgets.py - exception if user types invalid value into date field.
• fixed several issues in test_mp_release.cmd with messed up folders/current folder.
• Bandit warning on use of random.randint()
• Removing test "secret" from MPSettingsEditor.ipynb triggering credscan warning