Release Week of 2023-07-04
Note: This release is being rolled out. It may not be in your region yet.
⚠️ In Beam version 2.48.0, when you use Pub/Sub sinks with batch jobs, your pipeline might incorrectly receive the error SizeLimitExceededException for any messages above 100 bytes. For more information, see the Apache Beam GitHub issue #27000. This bug affected the following templates: JdbcToPubsub, TextToPubsub, StreamingDataGenerator. We have taken precautions to revert the templates to a known good version, but let us know if there are any related issues.
Improvements
[Spanner Change Streams] shard id is read from session files instead of being hardcoded
[Datastream to Spanner] Add transformationContextFilePath parameter for sharded transfers
[Datastream to Spanner] Allow reprocessing severe errors through a DLQ
[Utility] WindowedFilenamePolicyOptions defaults to 0 for numShards and lets the runner decide the runner if kept at 0
[BigQuery and Secret Manager] Version pins removed. This may impact the version selected during builds.
[Pub/Sub to Avro] Added numShards parameter
[Proto Support] Increase version to 3.21.12 to align with Beam
Bug Fixes
[Security] Upgrade os-maven-plugin to 1.7.1
[Pub/Sub to Avro] Fixed display name
Contributors
- abacn
- bvolpato
- pranavbhandari24
- Internal Contributors