PeerDB-io/peerdb v0.35.0

on GitHub

Breaking changes

• We removed the PEERDB_CLICKHOUSE_NORMALIZATION_PARTS setting which historically allow us to chunk our normalization query into parts to address OOM issues for tables with very high volume of updates per batch during CDC. This setting is not enabled by default, and was used sparingly by us. We discovered that if used in conjunction with PEERDB_CLICKHOUSE_PARALLEL_NORMALIZE, could trigger a race condition that results in data loss during replication. The setting that is more useful for addressing OOMs is PEERDB_S3_BYTES_PER_AVRO_FILE, which determines the staging Avro file size limit.

New features

• We added support for Amazon DocumentDB 5.0 by testing its change stream API and removing validation checks that were blocking DocumentDB peer setup.

Changelog

• ui: update dependencies by @serprex in #3638
• fix: trigger GetSourceVariant only during ScheduledTasks by @heavycrystal in #3639
• feat: upgrade docker-compose.yml stable image tags by @github-actions[bot] in #3636
• skip unsupported validations for documentdb by @jgao54 in #3635
• make required check generic by @jgao54 in #3643
• fix flow test complete check by @jgao54 in #3652
• chore(deps): update github/codeql-action digest to 0499de3 by @renovate[bot] in #3648
• chore(deps): update node.js to f36fed0 by @renovate[bot] in #3649
• chore(deps): update temporalio/auto-setup:1.29 docker digest to 5b3502a by @renovate[bot] in #3650
• PEERDB_LOG_LEVEL by @serprex in #3647
• improve normalize logic readability by @jgao54 in #3640
• Dev build for PeerDB UI by @ilidemi in #3645
• Speed up dev nexus build by @ilidemi in #3644
• dynamic required job v2 by @jgao54 in #3653
• Add PG18 to CI by @ilidemi in #3641
• [nexus] remove peer_connections by @heavycrystal in #3651
• deprecate PEERDB_CLICKHOUSE_NORMALIZATION_PARTS by @jgao54 in #3654
• [nexus] base64 decode SSH private key before use by @heavycrystal in #3612
• dynamic required job v3 by @jgao54 in #3656
• [postgres] QuoteIdentifier slotName during StartReplication by @heavycrystal in #3658
• Fully remove PEERDB_CLICKHOUSE_INITIAL_LOAD_PARTS_PER_PARTITION by @jgao54 in #3659
• [nexus] create features, disable everything for dev by @heavycrystal in #3646
• PeerDB UI: Better handling of charts and isolate dev changes by @Amogh-Bharadwaj in #3660

Full Changelog: v0.34.8...v0.35.0