fivetran/great_expectations 1.23.1

on GitHub

Highlights

• Spark now evaluates each regex independently with match_on="all" — On Spark, ExpectColumnValuesToMatchRegexList with match_on="all" now checks every regex separately against each column value, so patterns anchored at different positions (such as ^A and [0-9]{3}$) both match a value that satisfies them. This matches the behavior already seen on Pandas and SQL. (#12198)

  gxe.ExpectColumnValuesToMatchRegexList(
      column="id",
      regex_list=["^A", "[0-9]{3}$"],
      match_on="all",
  )

• Each Validator reports results for its own Batch when a datasource is reused — Validators built on the same datasource no longer borrow one another's Batch. Running two validation definitions on threads, or creating two validators from one datasource on a single thread, now evaluates and reports each validator's own data, with the correct batch_id, batch_spec and batch_definition on the result. This fixes a long-standing latent bug made reachable by #12148 in the 1.23.0 release. (#12211)

  validator_a = context.get_validator(batch_request=request_a)
  validator_b = context.get_validator(batch_request=request_b)
  # validator_a still validates request_a's batch
  result = validator_a.expect_table_row_count_to_equal(value=3)

• A Spark schema saved in great_expectations.yml reloads correctly — A persisted spark_schema is now read back through StructType.fromJson, so reopening a File Data Context round-trips the schema instead of failing inside PySpark. Values that are not an accepted schema form now raise a validation error naming the field and the accepted types. (#12200)

  context = gx.get_context(mode="file")
  asset = context.data_sources.get("spark_ds").get_asset("my_asset")
  assert asset.spark_schema is not None

• GX config files are read and written as UTF-8 regardless of locale — config_variables.yml and great_expectations.yml, and the .gitignore read while scaffolding a project, are now opened with an explicit UTF-8 encoding. Projects containing non-ASCII values or comments can be created and reloaded on hosts with a non-UTF-8 locale, and a project YAML file that is not valid UTF-8 now raises an error naming the file instead of a bare decode error. (#12182, #12204)

• A gallery-wide test tier for data sources — A new gallery support tier asserts a measured test result across the entire shipped expectation library — one case per registered expectation, each pairing a passing and a failing configuration — and nine data sources (pandas in-memory and filesystem CSV, SQLite, MySQL, PostgreSQL, Trino, BigQuery, Databricks and Redshift) now declare it after being measured against the full gallery. (#12150)

Changes

Features

• Adds a gallery support tier that asserts a passing test result over the whole shipped expectation library, with nine data sources declaring it, a required CI lane per member, membership and coverage guards, and a measurement mode for evaluating new candidate backends. (#12150)

Bug fixes

• ExpectColumnValuesToMatchRegexList with match_on="all" on Spark now evaluates each regex independently, so patterns anchored at different positions no longer fail on values that satisfy them all. (#12198)
• A Validator now keeps the identity of the Batches it loaded even when its execution engine is shared, so concurrent validations and multiple validators on one datasource each evaluate and report their own Batch; a configuration naming a batch the engine does not hold now raises instead of silently validating the most recently loaded batch. (#12211)
• The remaining config file reads and writes in the serializable data context are pinned to UTF-8, so scaffolding a project with a non-ASCII .gitignore and reading or updating great_expectations.yml work under a non-UTF-8 locale; an unreadable project YAML now raises an error naming the file. (#12204)
• A spark_schema persisted in great_expectations.yml is loaded back through StructType.fromJson so the schema round-trips when the context is reopened, and an unsupported value raises a validation error naming the field and the accepted types. (#12200)
• The Spark test-connection test, which covers behavior when PySpark is unavailable, is now skipped when PySpark is installed so it no longer fails for contributors with PySpark in their environment. (#12199)
• config_variables.yml is now read and written as UTF-8, so saving and reloading a configuration variable containing non-ASCII characters works on hosts with a non-UTF-8 locale. (#12182)

Docs

• The changelog now carries a deprecation timeline table listing every deprecated item, the version that deprecated it, and the version that removes it. (#12224)
• Documentation fixes: the credential-configuration pages now point at gx/uncommitted/config_variables.yml, a mistagged code fence highlights again, a misspelled snippet name is corrected, and doubled words and spelling errors across the core docs, ADRs, gallery docs and contrib READMEs are fixed. (#12177)
• Oracle is now documented on the connection-string reference (oracle+oracledb://...?service_name=...), the compatibility reference with its tested database version, and the data source method reference via add_sql. (#12167)

Contributors

Thanks to @lakshayxi (first contribution), @feiiiiii5 (first contribution), @ptimizeroracle (first contribution), @alibro005 (first contribution), @yigitcan-ozturk, @toyeshhm (first contribution), @nanjeshramesh, @p-mandale (first contribution).