apache/beam v2.34.0

Beam 2.34.0 release

on GitHub

We are happy to present the new 2.34.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.

For more information on changes in 2.34.0, check out the detailed release
notes.

Highlights

• The Beam Java API for Calcite SqlTransform is no longer experimental (BEAM-12680).
• Python's ParDo (Map, FlatMap, etc.) transforms now suport a with_exception_handling option for easily ignoring bad records and implementing the dead letter pattern.

I/Os

• ReadFromBigQuery and ReadAllFromBigQuery now run queries with BATCH priority by default. The query_priority parameter is introduced to the same transforms to allow configuring the query priority (Python) (BEAM-12913).
• [EXPERIMENTAL] Support for BigQuery Storage Read API added to ReadFromBigQuery. The newly introduced method parameter can be set as DIRECT_READ to use the Storage Read API. The default is EXPORT which invokes a BigQuery export request. (Python) (BEAM-10917).
• [EXPERIMENTAL] Added use_native_datetime parameter to ReadFromBigQuery to configure the return type of DATETIME fields when using ReadFromBigQuery. This parameter can only be used when method = DIRECT_READ(Python) (BEAM-10917).

New Features / Improvements

• Upgrade to Calcite 1.26.0 (BEAM-9379).
• Added a new dataframe extra to the Python SDK that tracks pandas versions
  we've verified compatibility with. We now recommend installing Beam with pip install apache-beam[dataframe] when you intend to use the DataFrame API
  (BEAM-12906).
• Add an example of deploying Python Apache Beam job with Spark Cluster

Breaking Changes

• SQL Rows are no longer flattened (BEAM-5505).
• [Go SDK] beam.TryCrossLanguage's signature now matches beam.CrossLanguage. Like other Try functions it returns an error instead of panicking. (BEAM-9918).
• BEAM-12925 was fixed. It used to silently pass incorrect null data read from JdbcIO. Pipelines affected by this will now start throwing failures instead of silently passing incorrect data.

Bugfixes

• Fixed error while writing multiple DeferredFrames to csv (Python) (BEAM-12701).
• Fixed error when importing the DataFrame API with pandas 1.0.x installed (BEAM-12945).
• Fixed top.SmallestPerKey implementation in the Go SDK (BEAM-12946).

List of Contributors

According to git shortlog, the following people contributed to the 2.34.0 release. Thank you to all contributors!

Ahmet Altay,
Aizhamal Nurmamat kyzy,
Alex Amato,
Alexander Chermenin,
Alexey Romanenko,
AlikRodriguez,
Andrew Pilloud,
Andy Xu,
Ankur Goenka,
Aydar Farrakhov,
Aydar Zainutdinov,
Aydar Zaynutdinov,
AydarZaynutdinov,
Benjamin Gonzalez,
BenWhitehead,
Brachi Packter,
Brian Hulette,
Bu Sun Kim,
Chamikara Jayalath,
Chris Gray,
Chuck Yang,
Chun Yang,
Claire McGinty,
comet,
Daniel Collins,
Daniel Oliveira,
Daniel Thevessen,
daria.malkova,
David Cavazos,
David Huntsperger,
Dmytro Kozhevin,
dpcollins-google,
Eduardo Sánchez López,
Elias Djurfeldt,
emily,
Emily Ye,
Enis Sert,
Etienne Chauchot,
Fernando Morales,
Heejong Lee,
Ihor Indyk,
Ismaël Mejía,
Israel Herraiz,
Jack McCluskey,
Jonathan Hourany,
Judah Rand,
Kenneth Knowles,
KevinGG,
Ke Wu,
kileys,
Kyle Weaver,
Luke Cwik,
masahitojp,
MiguelAnzoWizeline,
Minbo Bae,
Niels Basjes,
Ning Kang,
Pablo Estrada,
pareshsarafmdb,
Paul Féraud,
Piotr Szczepanik,
Reuven Lax,
Ritesh Ghorse,
R. Miles McCain,
Robert Bradshaw,
Robert Burke,
Rogan Morrow,
Ruwan Lambrichts,
rvballada,
Ryan Thompson,
Sam Rohde,
Sam Whittle,
Ștefan Istrate,
Steve Niemitz,
Thomas Li Fredriksen,
Tomo Suzuki,
tvalentyn,
Udi Meiri,
Vachan,
Valentyn Tymofieiev,
Vincent Marquez,
WinsonT,
Yichi Zhang,
Yifan Mai,
Yilei "Dolee" Yang,
zhoufek