We are happy to present the new 2.32.0 release of Apache Beam. This release includes both improvements and new functionality.
See the download page for this release.
For more information on changes in 2.32.0, check out the
Highlights
- The Beam DataFrame
API is no
longer experimental! We've spent the time since the 2.26.0 preview
announcement
implementing the most frequently used pandas operations
(BEAM-9547), improving
documentation
and error messages,
adding
examples,
integrating DataFrames with interactive
Beam,
and of course finding and fixing
bugs.
Leaving experimental just means that we now have high confidence in the API
and recommend its use for production workloads. We will continue to improve
the API, guided by your
feedback.
I/Os
- Support for X source added (Java/Python) (BEAM-X).
- Added ability to use JdbcIO.Write.withResults without statement and preparedStatementSetter. (BEAM-12511)
- Added ability to register URI schemes to use the S3 protocol via FileIO. (BEAM-12435).
- Respect number of shards set in SnowflakeWrite batch mode. (BEAM-12715)
- Java SDK: Update Google Cloud Healthcare IO connectors from using v1beta1 to using the GA version.
New Features / Improvements
- Add support to convert Beam Schema to Avro Schema for JDBC LogicalTypes:
VARCHAR,NVARCHAR,LONGVARCHAR,LONGNVARCHAR,DATE,TIME
(Java)(BEAM-12385). - Reading from JDBC source by partitions (Java) (BEAM-12456).
- PubsubIO can now write to a dead-letter topic after a parsing error (Java)(BEAM-12474).
- New append-only option for Elasticsearch sink (Java) BEAM-12601
Breaking Changes
- ListShards (with DescribeStreamSummary) is used instead of DescribeStream to list shards in Kinesis streams. Due to this change, as mentioned in AWS documentation, for fine-grained IAM policies it is required to update them to allow calls to ListShards and DescribeStreamSummary APIs. For more information, see Controlling Access to Amazon Kinesis Data Streams (BEAM-12225).
Deprecations
Known Issues
- Fixed race condition in RabbitMqIO causing duplicate acks (Java) (BEAM-6516))
List of Contributors
According to git shortlog, the following people contributed to the 2.32.0 release. Thank you to all contributors!
Ahmet Altay, Ajo Thomas, Alex Amato, Alexey Romanenko, Alex Koay, allenpradeep, Anant Damle, Andrew Pilloud, Ankur Goenka, Ashwin Ramaswami, Benjamin Gonzalez, BenWhitehead, Blake Williams, Boyuan Zhang, Brian Hulette, Chamikara Jayalath, Daniel Oliveira, Daniel Thevessen, daria-malkova, David Cavazos, David Huntsperger, dennisylyung, Dennis Yung, dmkozh, egalpin, emily, Esun Kim, Gabriel Melo de Paula, Harch Vardhan, Heejong Lee, heidimhurst, hoshimura, Iñigo San Jose Visiers, Ismaël Mejía, Jack McCluskey, Jan Lukavský, Justin King, Kenneth Knowles, KevinGG, Ke Wu, kileys, Kyle Weaver, Luke Cwik, Maksym Skorupskyi, masahitojp, Matthew Ouyang, Matthias Baetens, Matt Rudary, MiguelAnzoWizeline, Miguel Hernandez, Nikita Petunin, Ning Ding, Ning Kang, odidev, Pablo Estrada, Pascal Gillet, rafal.ochyra, raphael.sanamyan, Reuven Lax, Robert Bradshaw, Robert Burke, roger-mike, Ryan McDowell, Sam Rohde, Sam Whittle, Siyuan Chen, Teng Qiu, Tianzi Cai, Tobias Hermann, Tomo Suzuki, tvalentyn, Tyson Hamilton, Udi Meiri, Valentyn Tymofieiev, Vitaly Terentyev, Yichi Zhang, Yifan Mai, yoshiki.obata, Yu Feng, YuqiHuai, yzhang559, Zachary Houfek, zhoufek