apache/beam v2.36.0

Beam 2.36.0 release

on GitHub

We are happy to present the new 2.36.0 release of Apache Beam.
This release includes both improvements and new functionality.
See the download page for this release.

For more information on changes in 2.36.0, check out the detailed release
notes.

I/Os

• Support for stopReadTime on KafkaIO SDF (Java).(BEAM-13171).

New Features / Improvements

• Added support for cloudpickle as a pickling library for Python SDK (BEAM-8123). To use cloudpickle, set pipeline option: --pickler_lib=cloudpickle
• Added option to specify triggering frequency when streaming to BigQuery (Python) (BEAM-12865).
• Added option to enable caching uploaded artifacts across job runs for Python Dataflow jobs (BEAM-13459). To enable, set pipeline option: --enable_artifact_caching, this will be enabled by default in a future release.

Breaking Changes

• Updated the jedis from 3.x to 4.x to Java RedisIO. If you are using RedisIO and using jedis directly, please refer to this page to update it. (BEAM-12092).
• Datatype of timestamp fields in SqsMessage for AWS IOs for SDK v2 was changed from String to long, visibility of all fields was fixed from package private to public BEAM-13638.
• Properly check output timestamps on elements output from DoFns, timers, and onWindowExpiration in Java BEAM-12931.
• Fixed a bug with DeferredDataFrame.xs when used with a non-tuple key
  (BEAM-13421).

Known Issues

• Users may encounter an unexpected java.lang.ArithmeticException when outputting a timestamp
  for an element further than allowedSkew from an allowed DoFN skew set to a value more than
  Integer.MAX_VALUE.
• See a full list of open issues that affect this version.

List of Contributors

According to git shortlog, the following people contributed to the 2.36.0 release. Thank you to all contributors!

Ada Wong
Ahmet Altay
Alexander
Alexander Dahl
Alexandr Zhuravlev
Alexey Romanenko
AlikRodriguez
Anand Inguva
Andrew Pilloud
Andy Ye
Arkadiusz Gasiński
Artur Khanin
Arun Pandian
Aydar Farrakhov
Aydar Zainutdinov
AydarZaynutdinov
Benjamin Gonzalez
Brian Hulette
Chamikara Jayalath
Daniel Collins
Daniel Oliveira
Daniel Thevessen
Daniela Martín
David Hinkes
David Huntsperger
Emily Ye
Etienne Chauchot
Evan Galpin
Heejong Lee
Ilya
Ilya Kozyrev
In-Ho Yi
Jack McCluskey
Janek Bevendorff
Jarek Potiuk
Ke Wu
KevinGG
Kyle Hersey
Kyle Weaver
Luís Bianchin
Luke Cwik
Masato Nakamura
Matthias Baetens
Mehdi Drissi
Melissa Pashniak
Michel Davit
Miguel Hernandez
MiguelAnzoWizeline
Milena Bukal
Moritz Mack
Mostafa Aghajani
Nathan J Mehl
Niel Markwick
Ning Kang
Pablo Estrada
Pavel Avilov
Quentin Sommer
Reuben van Ammers
Reuven Lax
Ritesh Ghorse
Robert Bradshaw
Robert Burke
Ryan Thompson
Sam Whittle
Sayat
Sergei Lebedev
Sergey Kalinin
Steve Niemitz
Talat Uyarer
Thiago Nunes
Tianyang Hu
Tim Robertson
Valentyn Tymofieiev
Vitaly Ivanov
Yichi Zhang
Yiru Tang
Yu Feng
Yu ISHIKAWA
Zachary Houfek
blais
daria-malkova
daria.malkova
darshan-sj
dpcollins-google
emily
ewianda
johnjcasey
kileys
lam206
laraschmidt
mosche
msbukal@google.com
tvalentyn