We are happy to present the new 2.62.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.
For more information on changes in 2.62.0, check out the detailed release notes.
New Features / Improvements
- Added support for stateful processing in Spark Runner for streaming pipelines. Timer functionality is not yet supported and will be implemented in a future release (#33237).
- The datetime module is now available for use in jinja templatization for yaml.
- Improved batch performance of SparkRunner's GroupByKey (#20943).
- Support OnWindowExpiration in Prism (#32211).
- This enables initial Java GroupIntoBatches support.
- Support OrderedListState in Prism (#32929).
I/Os
- gcs-connector config options can be set via GcsOptions (Java) (#32769).
- [Managed Iceberg] Support partitioning by time (year, month, day, hour) for types
date,time,timestamp, andtimestamp(tz)(#32939) - Upgraded the default version of Hadoop dependencies to 3.4.1. Hadoop 2.10.2 is still supported (Java) (#33011).
- [BigQueryIO] Create managed BigLake tables dynamically (#33125)
Breaking Changes
- Upgraded ZetaSQL to 2024.11.1 (#32902). Java11+ is now needed if Beam's ZetaSQL component is used.
Bugfixes
- Fixed EventTimeTimer ordering in Prism. (#32222).
- [Managed Iceberg] Fixed a bug where DataFile metadata was assigned incorrect partition values (#33549).
Security Fixes
- Fixed (CVE-2024-47561)[https://www.cve.org/CVERecord?id=CVE-2024-47561] (Java) by upgrading Avro version to 1.11.4
For the most up to date list of known issues, see https://github.com/apache/beam/blob/master/CHANGES.md
List of Contributors
According to git shortlog, the following people contributed to the 2.62.0 release. Thank you to all contributors!
Ahmed Abualsaud,
Ahmet Altay,
Alex Merose,
Andrew Crites,
Arnout Engelen,
Attila Doroszlai,
Bartosz Zablocki,
Chamikara Jayalath,
Claire McGinty,
Claude van der Merwe,
Damon Douglas,
Danny McCormick,
Gabija Balvociute,
Hai Joey Tran,
Hakampreet Singh Pandher,
Ian Sullivan,
Jack McCluskey,
Jan Lukavský,
Jeff Kinard,
Jeffrey Kinard,
Laura Detmer,
Kenneth Knowles,
Martin Trieu,
Mattie Fu,
Michel Davit,
Naireen Hussain,
Nick Anikin,
Radosław Stankiewicz,
Ravi Magham,
Reeba Qureshi,
Robert Bradshaw,
Robert Burke,
Rohit Sinha,
S. Veyrié,
Sam Whittle,
Shingo Furuyama,
Shunping Huang,
Svetak Sundhar,
Valentyn Tymofieiev,
Vlado Djerek,
XQ Hu,
Yi Hu,
twosom