dyvenia/viadot v0.2.3

Viadot 0.2.3

on GitHub

Added

• Added a test for SupermetricsToADLS flow
• Added a test for AzureDataLakeList task
• Added PR template for new PRs
• Added a write_to_json util task to the SupermetricsToADLS flow. This task dumps the input expectations dict to the local filesystem as is required by Great Expectations.
  This allows the user to simply pass a dict with their expectations and not worry about the project structure required by Great Expectations
• Added Shapely and imagehash dependencies required for full visions functionality (installing visions[all] breaks the build)
• Added more parameters to control CSV parsing in the ADLSGen1ToAzureSQLNew flow
• Added keep_output parameter to the RunGreatExpectationsValidation task to control Great Expectations output to the filesystem
• Added keep_validation_output parameter and cleanup_validation_clutter task to the SupermetricsToADLS flow to control Great Expectations output to the filesystem

Changed

• Modified RunGreatExpectationsValidation task to use the built in support for evaluation parameters added in Prefect v0.15.3
• Modified SupermetricsToADLS and ADLSGen1ToAzureSQLNew flows to align with this recipe for reading the expectation suite JSON
  The suite now has to be loaded before flow initialization in the flow's python file and passed as an argument to the flow's constructor.
• Modified RunGreatExpectationsValidation's expectations_path parameter to point to the directory containing the expectation suites instead of the
  Great Expectations project directory, which was confusing. The project directory is now only used internally and not exposed to the user
• Changed the logging of docs URL for RunGreatExpectationsValidation task to use GE's recipe from the docs

Removed

• Removed SupermetricsToAzureSQLv2 and SupermetricsToAzureSQLv3 flows
• Removed geopy dependency