What's Changed
In this version of PeerDB, we restructured our flow architecture resulting in improved reliability (ex: drop mirror) and performance. We added support for a brand new connector in Clickhouse! On the UI side, we now have upgraded validation checks across the board, along with cutting-edge features such as pausing and editing mirrors.
This release sees a major investment in the alerting and monitoring systems, with support for email and slack alerts. We also added support for SSH connections to PostgreSQL peers. When it comes to the core ETL logic, we've solidified our data type handling, memory and storage management, logging and testing.
- 🔄 Split cdc_flow into cdc_flow / sync_flow by @serprex in #1365
- 🛠️ Clickhouse cdc by @pankaj-peerdb in #1096
- 📧 feat(alerting): add email sender by @iamKunalGupta in #1433
- ⏸️ Pause and resume mirror buttons, along with state reflection by @heavycrystal in #1133
- ⚙️ Add Tables Feature: Added dynamic table addition to existing props signal by @heavycrystal in #1106
- 🚀 feat: add telemetry/alerts via sns by @iamKunalGupta in #1411
- 📦 Setup partitioning and clustering for raw table by @iskakaushik in #915
- 🗝️ SSH Support: Use SSHWrappedPool when querying the Postgres Peer by @iskakaushik in #1148
- ⚙️ Custom threshold support for slack alerts by @heavycrystal in #1277
- 🔄 Normalize concurrently with sync flows by @serprex in #893
- 💾 Spill to disk based on flow-worker memory usage by @heavycrystal in #1231
- 🔄 Update max batch size on signal by @iskakaushik in #910
- 🔄 drop_flow: retry until both source/destination succeed by @serprex in #1201
- ⚙️ Adds idletimeout to flow config, ui and temporal signal by @Amogh-Bharadwaj in #952
- ⚙️ Added capability for BQ CDC across datasets by @heavycrystal in #904
- 💾 Support dynamic numeric with defaults by @Amogh-Bharadwaj in #1194
- 🖥️ UI: Edit Page and Refactor Edit Mirror Route by @Amogh-Bharadwaj in #1156
- 🔄 Retry when WAL segment has not been found by @iskakaushik in #930
- 🗺️ HStore and Geospatial for Postgres by @Amogh-Bharadwaj in #1091
- ⚙️ Dynamically add new tables to CDC mirrors by @heavycrystal in #1084
- 🔄 cdc_flow: listen for shutdown request while sync flow in progress by @serprex in #1103
- 🔄 Go 1.22 by @serprex in #1219
- 🖥️ Clickhouse UI by @pankaj-peerdb in #1022
- 🛠️ Validate Mirror: PostgreSQL Checks by @Amogh-Bharadwaj in #1110
- ⚙️ Support specifying host key for ssh config by @serprex in #1125
- 🗑️ drop_flow: drop destination/source concurrently by @serprex in #1101
- 📦 BQ Peer: Support project dot dataset by @Amogh-Bharadwaj in #1073
- 🛠️ Validate peer: permission check for snowflake by @Amogh-Bharadwaj in #1126
- 🗑️ Removing deprecated params from QRep by @heavycrystal in #1154
- 💓 HeartbeatRoutine: use explicit ticker by @serprex in #1157
- 🛠️ Validate peer: check bigquery permissions by @Amogh-Bharadwaj in #1119
- 🖥️ UI: Show peer configuration on clicking peer by @Amogh-Bharadwaj in #1168
- 🔄 Always set application_name when connecting to postgres by @serprex in #1169
- 🖥️ UI: Resync button by @Amogh-Bharadwaj in #1178
- 🖥️ UI: Line chart for slot growth by @Amogh-Bharadwaj in #1184
- 🗑️ Remove configurable postgres metadata database by @serprex in #1189
- 🛠️ Snowflake: stop storing metadata on warehouse; store in catalog by @serprex in #1179
- 🛠️ BigQuery: stop storing metadata on warehouse; store in catalog by @serprex in #1191
- 🔄 Clickhouse cdc data types by @pankaj-peerdb in #1210
- 📝 Better CDC error logging by @iskakaushik in #1275
- 🛠️ Connectors: build on a single GetConnector using generic GetConnectorAs function by @serprex in #1281
Full Changelog: v0.10.2...v0.11.0