unionai-oss/pandera v0.32.0 on GitHub

⭐️ Highlights: Add support for `narwhals` backend

In this release, we're adding support for a narwhals backend that supports the polars, ibis and pyspark-sql schema APIs. To enable it, install the narwhals extra with the support framework of your choice:

pip install 'pandera[narwhals,polars]'   # Polars
pip install 'pandera[narwhals,ibis]'     # Ibis
pip install 'pandera[narwhals,pyspark]'  # PySpark SQL

Then enable with an environment variable:

export PANDERA_USE_NARWHALS_BACKEND=True

Or configure programmatically:

import pandera.polars as pa 
# import pandera.pyspark as pa  # 👈 for pyspark schemas
# import pandera.ibis as pa  # 👈 for ibis schemas

# CONFIG.use_narwhals_backend is read here — not at import time above
pa.config.set_config(use_narwhals_backend=True)

schema = pa.DataFrameSchema({"name": pa.Column(str)})  # narwhals backends registered
schema.validate(df)

See the docs for more information.

What's Changed

fix: Ibis validation fails checks if column all nulls by @cosmicBboy in #2296
fix: StringDtype validation fails for empty DataFrames by @cosmicBboy in #2295
Clean up duplicate code and extra files from #2296 by @deepyaman in #2298
Pyspark: Added unique_values_eq builtin check by @aditypan in #2300
feat(narwhals): implement the new unifying backend by @deepyaman in #2223
refactor(narwhals): make backend opt-in via config by @deepyaman in #2314
refactor(narwhals): add per-backend test structure by @deepyaman in #2315
[Narwhals] add docs, some code cleanup by @cosmicBboy in #2312
fix(polars): handle PydanticModel dtype in DataFrameModel.empty() by @Dev-iL in #2313
fix: ensure index fields have persistent metadata by @WPDOrdina in #2217
refactor(polars): preserve input frame kind in coerce_dtype by @Dev-iL in #2317
chore: delete extraneous isort hook for pre-commit by @deepyaman in #2320
test(ibis): enable decimal datatype test after fix by @deepyaman in #2321
fix(tests): drop deprecated axis=None from DataFrame.sum call (#2309) by @terryzm1 in #2311
docs: clarify required extra for data synthesis strategies by @RedZapdos123 in #2304
chore: add Time as a cross-engine semantic dtype by @deepyaman in #2323
refactor: consolidate backend-specific test suites by @deepyaman in #2318
Bugfix/2319 by @Concrete-Slab in #2324
feat(strategies): aggregate check constraints to skip filter chains by @cosmicBboy in #2322
Improvement/ enable raising schema error in custom check by @Jarek-Rolski in #2340
fix(polars): stringify nested dtypes in lazy failure-case formatter by @lukew-cogapp in #2341
docs: clarify optional missing columns by @remsky in #2338
fix: handle SchemaErrors in check_types Union fallback (#2325) by @SAY-5 in #2326
Fix mypy type hinting for Polars DataFrameModel field annotations by @RedZapdos123 in #2303
Fixed issue where Pandera cannot handle metadata through Annotated types by @NGHades in #2111
fix narwhals incompatibility by @cosmicBboy in #2342
refactor: pydantic/polars cleanup and benchmark coverage by @Dev-iL in #2357
Clarify ignore_na null handling in checks docs by @vanta722 in #2355
fix(polars): avoid coercing missing required columns by @remsky in #2346
feat(pyspark): register Narwhals backend for Spark by @deepyaman in #2339
Address comments from full Narwhals backend review by @deepyaman in #2360
docs(pydantic): clarify v2 string coercion semantics by @RedZapdos123 in #2348
update narwhals docs by @cosmicBboy in #2365
update config, add set_config by @cosmicBboy in #2366
Harden CI dependency installation by @RedZapdos123 in #2369
fix(extensions): reject extra positional args in register_check_method (#480) by @jbbqqf in #2347
fix: handle mixed-timezone pandas datetime coercion by @RedZapdos123 in #2350
fix(pandas): keep ArrowString resolution stable by @RedZapdos123 in #2353
feat: allow rename_columns to work as long as final columns are unique by @AchillesTurtle in #2373
fix(polars): don't error on optional regex column with no match by @gaoflow in #2374
fix: reload built-in check dispatcher by its own name so a custom name= works (#2042) by @vineethsaivs in #2371
fix(polars): report concrete column name in regex coercion errors (#2363) by @gaoflow in #2376
feat: enable runtime narwhals backend toggling via set_config by @cosmicBboy in #2375
fix(pandas): preserve drop_invalid_rows in schema IO by @RedZapdos123 in #2354
docs: note regex engine differences between pandas and polars backends (#2349) by @cycsmail in #2378

New Contributors

@WPDOrdina made their first contribution in #2217
@Concrete-Slab made their first contribution in #2324
@lukew-cogapp made their first contribution in #2341
@remsky made their first contribution in #2338
@SAY-5 made their first contribution in #2326
@NGHades made their first contribution in #2111
@vanta722 made their first contribution in #2355
@jbbqqf made their first contribution in #2347
@AchillesTurtle made their first contribution in #2373
@gaoflow made their first contribution in #2374
@vineethsaivs made their first contribution in #2371
@cycsmail made their first contribution in #2378

Full Changelog: v0.31.1...v0.32.0

unionai-oss/pandera v0.32.0 on GitHub

⭐️ Highlights: Add support for narwhals backend

What's Changed

New Contributors

unionai-oss/pandera v0.32.0
on GitHub

⭐️ Highlights: Add support for `narwhals` backend