apache/iceberg-python pyiceberg-0.6.0

PyIceberg 0.6.0

on GitHub

What's Changed

• Python: Migrate from iceberg to iceberg-python by @Fokko in #3
• Build: Bump duckdb from 0.8.1 to 0.9.0 by @dependabot in #4
• Build: Bump mkdocs-section-index from 0.3.7 to 0.3.8 by @dependabot in #5
• Build: Bump mkdocstrings-python from 1.7.0 to 1.7.1 by @dependabot in #6
• Build: Bump pydantic from 2.3.0 to 2.4.2 by @dependabot in #7
• Build: Bump psycopg2-binary from 2.9.7 to 2.9.8 by @dependabot in #8
• Build: Bump moto from 4.2.4 to 4.2.5 by @dependabot in #9
• Build: Bump mkdocs-material from 9.4.1 to 9.4.2 by @dependabot in #10
• Build: Bump rich from 13.5.3 to 13.6.0 by @dependabot in #11
• Build: Bump typing-extensions from 4.7.1 to 4.8.0 by @dependabot in #12
• Build: Bump griffe from 0.36.2 to 0.36.4 by @dependabot in #13
• Build: Bump urllib3 from 1.26.16 to 1.26.17 by @dependabot in #36
• Update how to release by @Fokko in #34
• pydantic exclude 2.4.0, 2.4.1 by @syun64 in #38
• Add logic to generate a new snapshot-id by @Fokko in #37
• Fix the TableIdentifier by @Fokko in #44
• Convert the Logical to Physical map to a visitor by @Fokko in #43
• Build: Bump mkdocstrings-python from 1.7.1 to 1.7.2 by @dependabot in #52
• Build: Bump fastavro from 1.8.3 to 1.8.4 by @dependabot in #51
• Build: Bump pypa/cibuildwheel from 2.16.0 to 2.16.2 by @dependabot in #47
• Build: Bump psycopg2-binary from 2.9.8 to 2.9.9 by @dependabot in #49
• Build: Bump coverage from 7.3.1 to 7.3.2 by @dependabot in #50
• Build: Bump cython from 3.0.2 to 3.0.3 by @dependabot in #48
• Docs: Fix repo name and url by @manuzhang in #54
• Run integration tests with Iceberg 1.4.0 by @Fokko in #56
• Add logic for table format-version updates by @Fokko in #55
• Disable merge-commit and enforce linear history by @Fokko in #57
• Construct a writer tree by @Fokko in #40
• Add method and property around sequence-numbers by @Fokko in #60
• Fix column rename doc example to reflect correct API by @cabhishek in #59
• Expression: Part of the expression is ignored when multiple and/or expressions are specified by @amogh-jahagirdar in #65
• Fix Iceberg to Avro Schema Conversion: Fixed, Decimal, UUID by @HonahX in #53
• allow override env-variables in load_catalog by @bdilday in #45
• Make next_sequence_number private by @Fokko in #62
• Check for empty responses by @Fokko in #69
• Fix Arrow fixed type by @Fokko in #70
• Bump version to 0.5.1 by @Fokko in #68
• Add spec_id back to data file by @puchengy in #63
• Build: Bump ray from 2.7.0 to 2.7.1 by @dependabot in #77
• Build: Bump griffe from 0.36.4 to 0.36.5 by @dependabot in #76
• Build: Bump mypy-boto3-glue from 1.28.36 to 1.28.63 by @dependabot in #75
• Build: Bump mkdocstrings-python from 1.7.2 to 1.7.3 by @dependabot in #74
• Build: Bump moto from 4.2.5 to 4.2.6 by @dependabot in #73
• Remove python working directory by @Fokko in #71
• Don't fail on warning when releasing by @Fokko in #80
• Remove example since it is deprecated by @Fokko in #79
• Build: Bump urllib3 from 1.26.17 to 1.26.18 by @dependabot in #84
• Doc: Fix "Verifying Checksums" script in verify-release.md by @HonahX in #82
• Make to_arrow function capable of handling parquet files with sanitized name due to Avro restirction by @puchengy in #83
• Require full expression parse match by @danielcweeks in #88
• Fix NotStartsWith negation by @danielcweeks in #92
• Fix some broken commands and URLs in the docs by @hussein-awala in #89
• Update like statements to reflect sql behaviors by @danielcweeks in #91
• Fix equality of bound expressions by @Fokko in #95
• Build: Bump mkdocs-material from 9.4.2 to 9.4.6 by @dependabot in #100
• Build: Bump pytest-mock from 3.11.1 to 3.12.0 by @dependabot in #99
• Build: Bump sqlalchemy from 2.0.21 to 2.0.22 by @dependabot in #98
• Build: Bump griffe from 0.36.5 to 0.36.7 by @dependabot in #97
• Build: Bump adlfs from 2023.9.0 to 2023.10.0 by @dependabot in #96
• Replace old %-formatted by f-strings by @hussein-awala in #93
• Fix literal predicate equality check by @danielcweeks in #94
• Fix the nullability of snapshot-id on AssertRefSnapshotId by @Fokko in #103
• Build: Bump werkzeug from 2.3.7 to 3.0.1 by @dependabot in #105
• Api docs refactor by @mobley-trent in #106
• Fixed typos by @whisk in #108
• Build: Bump duckdb from 0.9.0 to 0.9.1 by @dependabot in #114
• Build: Bump pre-commit from 3.4.0 to 3.5.0 by @dependabot in #113
• Build: Bump mkdocs-material from 9.4.6 to 9.4.7 by @dependabot in #111
• Build: Bump pytest from 7.4.2 to 7.4.3 by @dependabot in #112
• Build: Bump moto from 4.2.6 to 4.2.7 by @dependabot in #110
• fix: partition evaluator thread safety by @skellys in #115
• Run dependabot daily by @Fokko in #66
• Build: Bump griffe from 0.36.7 to 0.36.9 by @dependabot in #118
• Build: Bump cython from 3.0.3 to 3.0.5 by @dependabot in #122
• Build: Bump sqlalchemy from 2.0.22 to 2.0.23 by @dependabot in #125
• Build: Bump zstandard from 0.21.0 to 0.22.0 by @dependabot in #120
• Build: Bump fastavro from 1.8.4 to 1.9.0 by @dependabot in #119
• Refactor Arrow schema conversion by @Fokko in #117
• Build: Bump pyarrow from 13.0.0 to 14.0.0 by @Fokko in #126
• Build: Bump mkdocs-material-extensions from 1.2 to 1.3 by @dependabot in #128
• Add flake8-pie to ruff by @Fokko in #86
• Update pre-commit by @Fokko in #85
• Bump version to 0.6.0 by @Fokko in #72
• Build: Bump mypy-boto3-glue from 1.28.63 to 1.28.77 by @dependabot in #130
• Catch warning in PyLint tests by @Fokko in #33
• Build: Bump mkdocs-material from 9.4.7 to 9.4.8 by @dependabot in #131
• Fix Github Pages path by @Fokko in #133
• Build: Bump pyarrow from 14.0.0 to 14.0.1 by @dependabot in #136
• Add list-refs cli command by @amogh-jahagirdar in #137
• Docs: Add section on pandas by @Fokko in #138
• Build: Bump mkdocstrings-python from 1.7.3 to 1.7.4 by @dependabot in #142
• Build: Bump moto from 4.2.7 to 4.2.8 by @dependabot in #143
• Build: Bump griffe from 0.36.9 to 0.38.0 by @dependabot in #144
• Build: Bump aiohttp from 3.8.5 to 3.8.6 by @dependabot in #149
• Build: Bump mypy-boto3-glue from 1.28.77 to 1.29.0 by @dependabot in #148
• Build: Bump mkdocstrings from 0.23.0 to 0.24.0 by @dependabot in #147
• Build: Bump duckdb from 0.9.1 to 0.9.2 by @dependabot in #146
• Build: Bump mypy-boto3-glue from 1.29.0 to 1.29.2 by @dependabot in #154
• Build: Bump moto from 4.2.8 to 4.2.9 by @dependabot in #153
• Build: Bump rich from 13.6.0 to 13.7.0 by @dependabot in #151
• Bump Poetry to 1.7.1 by @Fokko in #158
• Build: Bump mkdocstrings-python from 1.7.4 to 1.7.5 by @dependabot in #157
• Build: Bump mkdocs-material from 9.4.8 to 9.4.10 by @dependabot in #155
• Build: Bump mkdocs-material-extensions from 1.3 to 1.3.1 by @dependabot in #160
• Build: Bump moto from 4.2.9 to 4.2.10 by @dependabot in #164
• Build: Bump mkdocs-material from 9.4.10 to 9.4.12 by @dependabot in #163
• Support catalog name in table identifier during load, rename, drop, and purge by @pdames in #150
• Build: Bump cython from 3.0.5 to 3.0.6 by @dependabot in #167
• Build: Bump mkdocs-material from 9.4.12 to 9.4.14 by @dependabot in #166
• Python: Clarify docs on environment variables by @Fokko in #165
• Build: Bump cryptography from 41.0.4 to 41.0.6 by @dependabot in #169
• CI: Bump Spark to 3.4.2 by @Fokko in #177
• Fix downgrading of schema by @Fokko in #176
• Build: Bump mypy-boto3-glue from 1.29.2 to 1.33.5 by @dependabot in #172
• Build: Bump pydantic from 2.4.2 to 2.5.2 by @dependabot in #159
• Update table metadata by @HonahX in #139
• Build: Bump moto from 4.2.10 to 4.2.11 by @dependabot in #180
• Use Pydantic's model_copy for model modification when updating table metadata by @HonahX in #182
• fix(bucket transform): bad comparison for truthy value by @jayceslesar in #187
• Build: Bump actions/setup-python from 4 to 5 by @dependabot in #191
• Build: Bump fastavro from 1.9.0 to 1.9.1 by @dependabot in #190
• Build: Bump griffe from 0.38.0 to 0.38.1 by @dependabot in #189
• Automatically create the tables for the SqlCatalog by @cosmastech in #186
• maint(transforms): replace type() calls with isinstance() by @jayceslesar in #188
• Add Snapshot logic and Summary generation by @Fokko in #61
• Build: Bump mkdocs-material from 9.4.14 to 9.5.1 by @dependabot in #197
• Build: Bump actions/stale from 8.0.0 to 9.0.0 by @dependabot in #195
• feat(docs): example of multiple catalogs defined in .pyiceberg.yaml by @jayceslesar in #194
• Table Requirements Validation by @HonahX in #200
• Add SQLite support by @Fokko in #178
• Build: Bump mkdocs-material from 9.5.1 to 9.5.2 by @dependabot in #203
• Build: Bump typing-extensions from 4.8.0 to 4.9.0 by @dependabot in #204
• fix snapshot schema issue: wrong usage of walrus operator by @tonyps1223 in #211
• Build: Bump mypy-boto3-glue from 1.33.5 to 1.34.0 by @dependabot in #213
• Build: Bump actions/upload-artifact from 3 to 4 by @dependabot in #215
• Build: Bump coverage from 7.3.2 to 7.3.3 by @dependabot in #214
• Build: Bump pyarrow from 14.0.1 to 14.0.2 by @dependabot in #225
• Build: Bump moto from 4.2.11 to 4.2.12 by @dependabot in #224
• Add name-mapping by @Fokko in #212
• Moto server implementation for pytest by @sebpretzer in #223
• Make connect_timeout configurable in IO by @jqin61 in #218
• Build: Bump aiohttp from 3.8.6 to 3.9.0 by @dependabot in #168
• Build: Bump cython from 3.0.6 to 3.0.7 by @dependabot in #228
• Build: Bump coverage from 7.3.3 to 7.3.4 by @dependabot in #231
• Build: Bump fastavro from 1.9.1 to 1.9.2 by @dependabot in #236
• Build: Bump mypy-boto3-glue from 1.34.0 to 1.34.7 by @dependabot in #238
• Build: Bump pydantic from 2.5.2 to 2.5.3 by @dependabot in #237
• Build: Bump mkdocs-material from 9.5.2 to 9.5.3 by @dependabot in #239
• Build: Bump sqlalchemy from 2.0.23 to 2.0.24 by @dependabot in #244
• Build: Bump coverage from 7.3.4 to 7.4.0 by @dependabot in #243
• Glue catalog commit table by @HonahX in #140
• Build: Bump pytest from 7.4.3 to 7.4.4 by @dependabot in #248
• Allow filtering on newly added columns by @Fokko in #246
• Build: Bump sqlalchemy from 2.0.24 to 2.0.25 by @dependabot in #250
• falsy value of zero by @MehulBatra in #249
• Correct schema behavior by @Fokko in #247
• Replace black by Ruff Formatter by @hussein-awala in #127
• Fix lint tests failed in main by @hussein-awala in #253
• Build: Bump moto from 4.2.12 to 4.2.13 by @dependabot in #257
• Build: Bump mkdocstrings-python from 1.7.5 to 1.8.0 by @dependabot in #256
• Build: Bump fastavro from 1.9.2 to 1.9.3 by @dependabot in #258
• [Bug fix] Ensure string env var s3.connect-timeout is cast to float by @syun64 in #259
• Update feature support documentation by @ndrluis in #261
• Build: Bump cython from 3.0.7 to 3.0.8 by @dependabot in #260
• Build: Bump jinja2 from 3.1.2 to 3.1.3 in /mkdocs by @dependabot in #263
• Build: Bump jinja2 from 3.1.2 to 3.1.3 by @dependabot in #264
• Arrow: Set field-id with prefix by @Fokko in #227
• Rest Catalog Support for a Separate OAuth Server URI by @syun64 in #233
• Build: Bump mkdocs-material from 9.5.3 to 9.5.4 by @dependabot in #267
• Table Metadata Update: Support SetPropertiesUpdate and RemovePropertiesUpdate by @HonahX in #266
• Build: Bump griffe from 0.38.1 to 0.39.0 by @dependabot in #272
• Add SqlCatalog _commit_table support by @syun64 in #265
• Add Hive integration tests by @Fokko in #207
• [Bug Fix] TruncateTransform for falsey values by @syun64 in #276
• Add small test on concurrent changes by @Fokko in #273
• Write support by @Fokko in #41
• Implement pre-existing session support for dynamodb catalog by @waifairer in #104
• Fix the CI by @Fokko in #279
• [Bug Fix] all TimeTransforms for falsey values by @syun64 in #280
• Apply Name mapping by @syun64 in #219
• Build: Bump griffe from 0.39.0 to 0.39.1 by @dependabot in #282
• Fix moto server port conflict by @kevinjqliu in #292
• implement hive catalog _commit_table by @kevinjqliu in #294
• Build: Bump pyarrow from 14.0.2 to 15.0.0 by @dependabot in #295
• Add install extra for sqlite by @kevinjqliu in #297
• Lock Ray on <2.8.0 by @Fokko in #298
• Set Glue Table Information when creating/updating tables by @mgmarino in #288
• Remove redundant API call to Glue by @geruh in #300
• Build: Bump mkdocs-material from 9.5.4 to 9.5.5 by @dependabot in #302
• Bump Pyspark to 3.5.0 by @HonahX in #303
• Arrow: Don't copy the list/map when not needed by @Fokko in #252
• Add UnionByName functionality by @Fokko in #296
• Build: Bump pypa/cibuildwheel from 2.16.2 to 2.16.3 by @dependabot in #309
• Build: Bump boto3 from 1.34.22 to 1.34.27 by @dependabot in #308
• Build: Bump coverage from 7.4.0 to 7.4.1 by @dependabot in #307
• Check the types when writing by @Fokko in #313
• Fix writing to local filesystem by @kevinjqliu in #301
• docs: Remove feature support page by @Fokko in #316
• create_table with a PyArrow Schema by @syun64 in #305
• docs: Add community page by @Fokko in #315
• Build: Bump aiohttp from 3.9.1 to 3.9.2 by @dependabot in #323
• Build: Bump adlfs from 2023.12.0 to 2024.1.0 by @dependabot in #320
• Build: Bump pydantic from 2.5.3 to 2.6.0 by @dependabot in #317
• Build: Bump mkdocs-material from 9.5.5 to 9.5.6 by @dependabot in #318
• Build: Bump pypa/cibuildwheel from 2.16.3 to 2.16.4 by @dependabot in #322
• Small getting started guide on writes by @Fokko in #311
• Use .read_bytes() instead by @Fokko in #325
• REST: Set fresh IDs on create-table by @Fokko in #327
• Add PyArrow for the Cython decoder tests by @Fokko in #329
• Remove MotoServer from the conftest.py scope by @Fokko in #331
• Build: Bump griffe from 0.39.1 to 0.40.0 by @dependabot in #333
• The name needs to be unique per run by @Fokko in #332
• Allow running the Avro decoder tests without Arrow by @Fokko in #335
• Build: Bump pypa/cibuildwheel from 2.16.4 to 2.16.5 by @dependabot in #343
• Build: Bump mypy-boto3-glue from 1.34.7 to 1.34.32 by @dependabot in #342
• Create namespace through the CLI by @hendrikmakait in #336
• Remove sort-order constraint by @Fokko in #349
• Bug fix for writing empty df or null only columns by @syun64 in #350
• Small simplification in the name-mapping by @Fokko in #351
• Get Started: Add sqlcatalog and local fs warehouse by @kevinjqliu in #361
• SqlCatalog, default create_engine echo to False by @kevinjqliu in #360
• fix postgres catalog initialization when tables do not exist by @rahij in #356
• Use write.parquet.compression-{codec,level} by @jonashaag in #358
• Bug Fix: Allow GlueCatalog to create table with TimestampzType by @HonahX in #366
• docs: Document Parquet write options by @Fokko in #364
• Build: Bump moto from 4.2.13 to 5.0.1 by @hussein-awala in #373
• Build: Bump mkdocs-material from 9.5.6 to 9.5.7 by @dependabot in #369
• Build: Bump mypy-boto3-glue from 1.34.32 to 1.34.35 by @dependabot in #371
• Build: Bump adlfs from 2024.1.0 to 2024.2.0 by @dependabot in #372
• Add Daft examples and code into PyIceberg docs and Table by @jaychia in #355
• Bug Fix: Handle null vals from later row_groups in StatsAggregator by @syun64 in #379
• Do not set Parquet version by @jonashaag in #377
• release: Bump the moto version by @Fokko in #380
• Add syun64 as collaborator by @Fokko in #383
• Build: Bump ray from 2.7.2 to 2.9.2 by @dependabot in #384
• Build: Bump boto3 from 1.34.27 to 1.34.34 by @dependabot in #385
• Build: Bump pydantic from 2.6.0 to 2.6.1 by @dependabot in #386
• Arrow: Support large-string by @Fokko in #382
• Add tests and fixes for Daft integration by @jaychia in #381
• Build: Bump mkdocs-material from 9.5.7 to 9.5.8 by @dependabot in #395
• Arrow: Support int8 and int16 types by @Fokko in #391
• Bug Fix: Rest Catalog update partition spec and sort order when fresh schema is created by @syun64 in #392
• Centralized table properties management by @HonahX in #388
• Allow setting {row-group,page} limit by @Fokko in #390
• Build: Bump griffe from 0.40.0 to 0.40.1 by @dependabot in #400
• Build: Bump getdaft from 0.2.12 to 0.2.13 by @dependabot in #399
• Downgrade to the previous artifact upload action by @Fokko in #396
• Hive locking by @danielcweeks in #405
• Arrow: Support Large Binary when using to_arrow by @castedice in #409
• Use the correct name of extra that adds GCS support by @vaultah in #412
• Fix setting V1 format version for Non-REST catalogs by @amogh-jahagirdar in #411
• Add Thrift and Hive to NOTICE by @Fokko in #410
• Build: Bump mkdocs-material from 9.5.8 to 9.5.9 by @dependabot in #418
• Build: Bump sqlalchemy from 2.0.25 to 2.0.26 by @dependabot in #417
• Add test to capture Duckdb behavior by @Fokko in #389
• Update LICENSE and NOTICE by @Fokko in #413
• docs: Add section on fetching the remote by @Fokko in #421
• Chore: build wheels for all macos versions including arm macos-14 by @hussein-awala in #416
• Build: Bump fastavro from 1.9.3 to 1.9.4 by @dependabot in #427
• Build: Bump getdaft from 0.2.13 to 0.2.14 by @dependabot in #426
• Build: Bump duckdb from 0.9.2 to 0.10.0 by @dependabot in #425
• Build: Bump sqlalchemy from 2.0.26 to 2.0.27 by @dependabot in #424
• Set version in path instead of OS by @Fokko in #422
• Fix environment variable parsing by @Fokko in #423

New Contributors

• @syun64 made their first contribution in #38
• @manuzhang made their first contribution in #54
• @cabhishek made their first contribution in #59
• @amogh-jahagirdar made their first contribution in #65
• @HonahX made their first contribution in #53
• @bdilday made their first contribution in #45
• @puchengy made their first contribution in #63
• @danielcweeks made their first contribution in #88
• @hussein-awala made their first contribution in #89
• @mobley-trent made their first contribution in #106
• @whisk made their first contribution in #108
• @skellys made their first contribution in #115
• @pdames made their first contribution in #150
• @jayceslesar made their first contribution in #187
• @cosmastech made their first contribution in #186
• @tonyps1223 made their first contribution in #211
• @sebpretzer made their first contribution in #223
• @jqin61 made their first contribution in #218
• @MehulBatra made their first contribution in #249
• @ndrluis made their first contribution in #261
• @waifairer made their first contribution in #104
• @kevinjqliu made their first contribution in #292
• @mgmarino made their first contribution in #288
• @geruh made their first contribution in #300
• @hendrikmakait made their first contribution in #336
• @rahij made their first contribution in #356
• @jonashaag made their first contribution in #358
• @jaychia made their first contribution in #355
• @castedice made their first contribution in #409
• @vaultah made their first contribution in #412

Full Changelog: pyiceberg-0.5.1...pyiceberg-0.6.0