Release Week of 2022-11-28
Note: This release is in the process of rolling out. It may not be in your region yet.
Improvements
- [All templates] Introduce metadata annotations.
- [DataStreamToBigQuery] Expose mergeConcurrency option and re-throw error on merge statement fail.
- Trigger Java PR workflow when any XML is changed + files are deleted.
- [Classic templates] Support JSONB arrays.
- [PubSubCdcToBigQuery] Support maxStreamingBatchSize parameter
- [DatastreamToSpanner] Add changes for the new HarbourBridge session file with tableID and columnID support.
- [All templates] Upgrade Beam version to 2.43.
- [Classic templates] Prepare plugin infra to test classic templates + Create BulkCompressionIT
- [Integration Tests] Do not make artifactBucket mandatory (only if bucketName not provided for ITs)
- [DataStreamToSpanner] Change default values for dlqRetryMinutes and dlqMaxRetryCount params.
- [Integration Tests] Avoid Joiner conflict, and improve plugin staging speed
- [Flex templates] Plain text logging for Flex Templates unit tests
- [Integration Tests] Improve plugin bucket parameter requirements
- [SpannerChangeStreamsTemplates] Simplify the code of setting experiments for spanner change streams to BigQuery and spanner change streams to GCS templates.
- [Integration Tests] Create MongoDB Resource Manager
- [Integration Tests] Create MongoDBToBigQueryIntegrationTest
- [Integration Tests] Add TestContainers framework
- [Integration Tests] Create PubsubAvroToBigQueryIT + prepare profile to run integration tests together
- [MongoDBToBigQuery] Create udf for MongoDB BigQuery Templates
- [Security] Update hadoop version affected by CVE-2022-25168
- Improve Templates Plugin instructions
Bug Fixes
- [Classic templates] WindowedFilenamePolicy's dayPattern defaults to dd instead of DD
- [JDBC Templates] Do not log unencrypted values/keys to the console
Contributors
@bvolpato
@oleg-semenov
@Polber
@pranavbhandari24
@theshanbhag