⭐️ Highlights

Pandera now supports DataFusion schemas

import pyarrow
from datafusion import SessionContext

import pandera.datafusion as pa

schema = pa.DataFrameSchema(
    {
        "state": pa.Column(str),
        "city": pa.Column(str),
        "price": pa.Column(int, pa.Check.in_range(5, 20)),
    }
)

ctx = SessionContext()
df = ctx.from_arrow(
    pyarrow.table(
        {
            "state": ["FL", "FL", "CA", "CA"],
            "city": ["Orlando", "Miami", "Los Angeles", "San Francisco"],
            "price": [8, 12, 10, 16],
        }
    )
)

schema.validate(df)

And of course the class-based dataframe models as well:

class Schema(pa.DataFrameModel):
    state: str
    city: str
    price: int = pa.Field(in_range={"min_value": 5, "max_value": 20})


Schema.validate(df)

What's Changed

• ci: fix path-filter quantifier so docs-only PRs skip unit tests by @cosmicbboy in #2545
• fix: propagate nested Column drop_invalid_rows without mutation by @cosmicbboy in #2551
• fix(io): read parametrized polars dtypes back from yaml/json by @aminmiral in #2523
• fix(polars,ibis): ignore nulls in unique_values_eq by @MohammadHijjawi97 in #2538
• spec: pluggable data synthesis backends (realistic data generation) by @cosmicbboy in #2557
• docs(spec): apply review fixes to pluggable data synthesis proposal by @cosmicbboy in #2561
• feat(tensordict): nested keys, element-level check errors, field custom checks, and @check_types integration by @cosmicbboy in #2560
• fix(io): serialize MultiIndex options so they survive yaml/json/script round trips by @owgreen-dev in #2504
• feat(datafusion): add datafusion.DataFrame schema validation via the narwhals-datafusion plugin by @s5dsn-eqee in #2540
• fix(pandas): carry Column default through schema serialization by @feiiiiii5 in #2528
• fix(decorators): forward validation options to the positional obj_getter selector by @feiiiiii5 in #2499
• fix(checks): reject conflicting length arguments in Check.str_length by @yang00333 in #2564
• fix(narwhals): resolve six datafusion xfails in the shared backend by @cosmicbboy in #2562
• test(polars): reset only polars entries in the shared backend registry by @feiiiiii5 in #2502
• docs(tensordict): make failure-case example self-contained so docs build passes by @cosmicbboy in #2569
• fix(polars): coerce to Decimal without a Float64 round trip by @DG47 in #2574
• fix(pandas): carry report_duplicates and index default through serialization by @aminmiral in #2573
• docs(contributing): point the Sphinx RST link at the current primer by @Yi-111-a in #2571
• fix(io): serialize set check statistics such as unique_values_eq by @LuisFigueroaG in #2591
• fix(tests): suppress large_base_example health check in hypothesis profiles by @cosmicbboy in #2593
• fix(dask): validate Series indexes by @ska2704 in #2596
• fix(polars): report a schema error for a non-decimal column by @atharva-mashalkar in #2595
• fix(engines): resolve parametrized generic Decimal dtypes by @DG47 in #2592
• fix(pandas): drop columns without deprecated pandas inplace param by @cosmicbboy in #2600
• fix(xarray): propagate coordinate required metadata by @antoninkriz in #2488
• fix(io): format dataframe-level checks in to_script by @chuenchen309 in #2418
• fix(polars): exclude explicit columns from regex alias selectors (#2343) by @capyBearista in #2441
• docs: advertise DataFusion support in the docs top-level banner by @cosmicbboy in #2604
• Add explicit stacklevel to warnings and enforce with ruff B028 by @feiiiiii5 in #2500
• fix(datafusion): fix warning lint and backend test matrix by @Numroid in #2606
• fix(pandas): don't repeat duplicate-label columns when adding missing columns by @raams95 in #2601
• docs(api): drop the stray dataframe_schema param from to_json_schema by @VenishPaneliya in #2597
• fix: only patch numpy 2 aliases for pyspark < 4 (#2134) by @adilKhan72 in #2590
• ci: don't run the full test matrix for non-CI files under .github/ by @agarg21 in #2589
• chore: configure Ruff as VS Code formatter by @BALOGUN-DAVID in #2588
• docs(contributing): fix broken Dataframe schemas examples link by @kun654 in #2587

New Contributors

• @aminmiral made their first contribution in #2523
• @MohammadHijjawi97 made their first contribution in #2538
• @owgreen-dev made their first contribution in #2504
• @s5dsn-eqee made their first contribution in #2540
• @yang00333 made their first contribution in #2564
• @DG47 made their first contribution in #2574
• @Yi-111-a made their first contribution in #2571
• @LuisFigueroaG made their first contribution in #2591
• @ska2704 made their first contribution in #2596
• @atharva-mashalkar made their first contribution in #2595
• @antoninkriz made their first contribution in #2488
• @chuenchen309 made their first contribution in #2418
• @capyBearista made their first contribution in #2441
• @Numroid made their first contribution in #2606
• @raams95 made their first contribution in #2601
• @adilKhan72 made their first contribution in #2590
• @agarg21 made their first contribution in #2589
• @BALOGUN-DAVID made their first contribution in #2588
• @kun654 made their first contribution in #2587

Full Changelog: v0.34.1...v0.35.0