What's Changed
In this version of PeerDB, we improved the way in which pull rows from Postgres, and we also track the WAL extensively. This release includes support for Temporal Cloud. Of course, it comes with exciting new features for PeerDB mirror such as Mirror Resync, major UI upgrades, and heavy focus on development experience.
- 🔄 Postgres cdc: update mirror lsn_offset when wal processing raises consumedXLogPos by @serprex in #823
- ❄️ Snowflake: Run merges in parallel during normalize flow by @iskakaushik in #662
- 🔄 Replication should now work from PG 16 Standbys by @saisrirampur in #858
- 🚨 Basic alerting, refactored to use slack-go instead by @heavycrystal in #866
- 💼 Xmin rep by @serprex in #747
- ✨ Support Temporal Cloud by @Amogh-Bharadwaj in #692
- 🎯 Central slot collecting function for UI and monitoring by @Amogh-Bharadwaj in #771
- 💻 S3 Peer UI by @Amogh-Bharadwaj in #668
- 📋 Column Exclusion by @serprex in #601
- 🔄 full table resync for Snowflake QRep by @heavycrystal in #617
- 🔁 CDC/QRep full resync support for BigQuery by @heavycrystal in #639
- 🔄 schema changes for QRep BigQuery by @heavycrystal in #633
- 🔧 PG,BQ,SF CDC: PeerDB Columns by @Amogh-Bharadwaj in #845
- 📄 added metadata_schema option for SF and PG peers by @heavycrystal in #560
- 🔄 [eventhubs] Add more logs and retries by @iskakaushik in #622
- 🔁 RESYNC MIRROR for QRep Snowflake mirrors by @heavycrystal in #618
- 💻 UI for BigQuery Peer by @Amogh-Bharadwaj in #620
- ⏸️ PAUSE MIRROR support by @heavycrystal in #605
- 📝 Add screen to view mirror activity by @pankaj-peerdb in #62
- 🔄 Parallel Snowflake tests by @iskakaushik in #640
- 🛠️ Allow configurable column names for soft-delete and synced-at by @iskakaushik in #653
- 🖥️ Mirror Overview UI by @Amogh-Bharadwaj in #664
- 🔄 handling soft deleted rows during resync by @heavycrystal in #676
- ⏱️ Runs SendWalHeartbeat in parallel by @Amogh-Bharadwaj in #675
- 🗄️ Use catalog as external store by @Amogh-Bharadwaj in #680
- 🔍 [cdc] Flush at the beginning of CDC until consumedXLogPos by @iskakaushik in #688
- 🔄 SyncedAt Column for QRep by @Amogh-Bharadwaj in #854
- 📊 New Graph UI by @Amogh-Bharadwaj in #684
- 🌳 Heirarchical UI For CDC Table Picker And Refactoring by @Amogh-Bharadwaj in #700
- 🔒 UI auth by @serprex in #699
- 💓 WalHeartBeat across peers by @Amogh-Bharadwaj in #708
- 📚 Reads cert and key as base64 for Temporal Cloud by @Amogh-Bharadwaj in #725
- 🔄 support mixed case table names pg->pg by @iskakaushik in #722
- 🔄 Get task queues from a function based on deployment UID variable by @Amogh-Bharadwaj in #729
- 💻 Support for mirror name filter by @Amogh-Bharadwaj in #731
- 🔢 Use number of records synced as decider for CDC tests by @Amogh-Bharadwaj in #738
- 📝 add better logging for cdc flow by @iskakaushik in #740
- 🗄️ filtering system schemas from all tables and all schemas queries by @heavycrystal in #741
- 🏷️ added Dropdown to filter peers by type by @heavycrystal in #742
- 🛡️ Better peer checks by @Amogh-Bharadwaj in #572
- 🔄 add an endpoint to expose the PeerDB version from Flow API by @iskakaushik in #749
- 📊 Metrics For XMIN by @Amogh-Bharadwaj in #762
- ➕ Added spill to disk using Pebble for CDC records by @heavycrystal in #760
- 📊 Improve: slot size monitoring by @Amogh-Bharadwaj in #782
- 🛠️ Address GH code scanning by @serprex in #773
- ⚡ Make tests faster by @iskakaushik in #783
- 🔄 Track wal_status in slot info by @Amogh-Bharadwaj in #790
- 🔄 Update go modules by @serprex in #787
- 🚫 Avoid having multiple catalog connection pools by @serprex in #793
- 🔑 Allow the ability to connect to Postgres via an SSH tunnel by @iskakaushik in #800
- 🔄 Make sync batch size dynamic by @iskakaushik in #806
- ➕ Adding IF NOT EXISTS for pg and bq by @heavycrystal in #808
- 🛠️ BigQuery: Avro loader improvements by @iskakaushik in #810
- ⏱️ Changing idle timeout to 60 seconds by @heavycrystal in #791
- 🔄 Migrate to Go 1.21's native slog by @Amogh-Bharadwaj in #764
- 🔍 BigQuery: add wait for table by @iskakaushik in #817
- 🔧 Move configuration params to a central place by @iskakaushik in #818
- 🖥️ UI: Advanced section with pull batch size by @Amogh-Bharadwaj in #822
- 🖥️ Improve loading, mirrors page by @pankaj-peerdb in #841
- 📜 logs schema deltas to catalog as soon as they are read by @heavycrystal in #842
- 🔄 Update go dependencies: CVE-2023-48795 by @serprex in #850
- 🛠️ Only run nexus/flow/ui CI when PR affects their directory by @serprex in #848
- 🔧 Remove direct dependency on pkcs1/pkcs8 by @serprex in #853
- 🔄 Refactor replica identity type and primary key column retrieval in Postgres by @iskakaushik in #860
Full Changelog: v0.9.2...v0.10.0