apache/beam v2.51.0

Beam 2.51.0 release

on GitHub

We are happy to present the new 2.51.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.

For more information on changes in 2.51.0, check out the detailed release notes.

New Features / Improvements

• In Python, RunInference now supports loading many models in the same transform using a KeyedModelHandler (#27628).
• In Python, the VertexAIModelHandlerJSON now supports passing in inference_args. These will be passed through to the Vertex endpoint as parameters.
• Added support to run mypy on user pipelines (#27906)

Breaking Changes

• Removed fastjson library dependency for Beam SQL. Table property is changed to be based on jackson ObjectNode (Java) (#24154).
• Removed TensorFlow from Beam Python container images PR. If you have been negatively affected by this change, please comment on #20605.
• Removed the parameter t reflect.Type from parquetio.Write. The element type is derived from the input PCollection (Go) (#28490)
• Refactor BeamSqlSeekableTable.setUp adding a parameter joinSubsetType. #28283

Bugfixes

• Fixed exception chaining issue in GCS connector (Python) (#26769).
• Fixed streaming inserts exception handling, GoogleAPICallErrors are now retried according to retry strategy and routed to failed rows where appropriate rather than causing a pipeline error (Python) (#21080).
• Fixed a bug in Python SDK's cross-language Bigtable sink that mishandled records that don't have an explicit timestamp set: #28632.

Security Fixes

• Python containers updated, fixing CVE-2021-30474, CVE-2021-30475, CVE-2021-30473, CVE-2020-36133, CVE-2020-36131, CVE-2020-36130, and CVE-2020-36135
• Used go 1.21.1 to build, fixing CVE-2023-39320

Known Issues

• Python pipelines using BigQuery Storage Read API must pin fastavro dependency to 1.8.3
  or earlier: #28811

List of Contributors

According to git shortlog, the following people contributed to the 2.50.0 release. Thank you to all contributors!

Adam Whitmore

Ahmed Abualsaud

Ahmet Altay

Aleksandr Dudko

Alexey Romanenko

Anand Inguva

Andrey Devyatkin

Arvind Ram

Arwin Tio

BjornPrime

Bruno Volpato

Bulat

Celeste Zeng

Chamikara Jayalath

Clay Johnson

Damon

Danny McCormick

David Cavazos

Dip Patel

Hai Joey Tran

Hao Xu

Haruka Abe

Jack Dingilian

Jack McCluskey

Jeff Kinard

Jeffrey Kinard

Joey Tran

Johanna Öjeling

Julien Tournay

Kenneth Knowles

Kerry Donny-Clark

Mattie Fu

Melissa Pashniak

Michel Davit

Moritz Mack

Pranav Bhandari

Rebecca Szper

Reeba Qureshi

Reuven Lax

Ritesh Ghorse

Robert Bradshaw

Robert Burke

Ruwann

Ryan Tam

Sam Rohde

Sereana Seim

Svetak Sundhar

Tim Grein

Udi Meiri

Valentyn Tymofieiev

Vitaly Terentyev

Vlado Djerek

Xinyu Liu

Yi Hu

Zbynek Konecny

Zechen Jiang

bzablocki

caneff

dependabot[bot]

gDuperran

gabry.wu

johnjcasey

kberezin-nshl

kennknowles

liferoad

lostluck

magicgoody

martin trieu

mosche

olalamichelle

tvalentyn

xqhu

Łukasz Spyra