great-expectations/great_expectations 0.8.0

on GitHub

Version 0.8.0 is a significant update to Great Expectations, with many improvements focused on configurability and usability. See the migrating versions guide for more details on specific changes, which include several breaking changes to configs and APIs.

Highlights include:

1. Validation Operators and Actions. Validation operators make it easy to integrate GE into a variety of pipeline runners. They offer one-line integration that emphasizes configurability. See the validation operators and actions feature guide for more information.

   • The DataContext get_batch method no longer treats expectation_suite_name or batch_kwargs as optional; they must be explicitly specified.
   • The top-level GE validate method allows more options for specifying the specific data_asset class to use.

2. First-class support for plugins in a DataContext, with several features that make it easier to configure and
   maintain DataContexts across common deployment patterns.

   • Environments: A DataContext can now manage environment_and_secrets more easily thanks to more dynamic and flexible variable substitution.
   • Stores: A new internal abstraction for DataContexts, stores_reference, make extending GE easier by consolidating logic for reading and writing resources from a database, local, or cloud storage.
   • Types: Utilities configured in a DataContext are now referenced using class_name and module_name throughout the DataContext configuration, making it easier to extend or supplement pre-built resources. For now, the "type" parameter is still supported but expect it to be removed in a future release.

3. Partitioners: Batch Kwargs are clarified and enhanced to help easily reference well-known chunks of data using a partition_id. Batch ID and Batch Fingerprint help round out support for enhanced metadata around data assets that GE validates. See batch_identifiers for more information. The GlobReaderGenerator, QueryGenerator, S3Generator, SubdirReaderGenerator, and TableGenerator all support partition_id for easily accessing data assets.

4. Other Improvements:

   • We're beginning a long process of some under-the-covers refactors designed to make GE more maintainable as we begin adding additional features.
   • Restructured documentation: our docs have a new structure and have been reorganized to provide space for more easily adding and accessing reference material. Stay tuned for additional detail.
   • The command build-documentation has been renamed build-docs and now by default opens the Data Docs in the users' browser.