github toluaina/pgsync 8.0.0

one hour ago

8.0.0

Child changes always find their document

  • A change to a child row now finds its root document from the change itself, or with a database query. It no longer has to wait
    until the search engine can see the root.
  • A child that moves to a new parent now updates both the old and the new document.
  • Composite foreign keys now use all their columns, not only the first one.
  • Many changes to the same row now give one update instead of many.

MySQL and MariaDB

  • MariaDB now resumes the binlog from the checkpoint. Before, every start read the binlog from the beginning. In testing, a
    restart went from over 3 minutes to about a second.
  • No more repeated documents or repeated nested items.
  • binlog_row_image=MINIMAL and NOBLOB are now supported.
  • _meta now holds the keys of every table, so child updates and deletes find their document.

Terminal monitor

  • New --monitor (-m) flag gives a full screen view of events, throughput, slot lag and errors.
  • Install it with pip install 'pgsync[monitor]'.
  • Ctrl+C now stops at once.

Cleaner terminal output

  • New key=value status line, written only when something changes. Set LOG_FORMAT=legacy to keep the old one.
  • The schema tree at startup is clearer: aligned columns with the table, the relationship and the through tables. Set
    SHOW_TREE=false to hide it.

Correctness under load

  • Transaction id wraparound is handled. All xmin comparisons use 64 bit ids.
  • Checkpoints no longer skip transactions that were still open during a pull.
  • WAL changes are now indexed and consumed under one lock with the same bound, so another consumer cannot discard changes before
    they are indexed.
  • Polling mode now works on read only databases and hot standbys, and saves its checkpoint. Before, it rescanned everything at
    every interval.
  • A malformed item in the Redis queue no longer blocks the queue.

Other improvements

  • PostgreSQL 19 in Docker Compose and CI.
  • Python 3.15 support.
  • Docker images now use a newer base with the Debian security updates, with no critical vulnerabilities.
  • The Docker Hub build script now publishes for both amd64 and arm64.
  • Connection URLs now keep spaces and special characters in usernames and passwords.
  • Routed documents are now cleaned up even without the old routing value.
  • POLLING and WAL can no longer both be set.

Upgrading from 7.x

  • Trigger mode: the trigger function now sends the foreign keys of a deleted row. To install it, stop the producer, let the
    consumer empty the queue, run bootstrap, then start pgsync. Bootstrap deletes the checkpoint file, and if REDIS_CHECKPOINT is not
    set the next start does a full sync. If you keep the old trigger, deletes still work through a search, as before.
  • The status line format has changed. Set LOG_FORMAT=legacy if you parse the old one.
  • Removed: Payload.foreign_key_constraint, upgrade_function, the demo app, and the prefix and leaf arguments of Node.display().

Notes:

  • The restart time ("over 3 minutes to about a second") comes from my local MariaDB test: 193 seconds before and 1 second after.
    It depends on the size of the binlog, so remove the number if you prefer.
  • Docker Hub: commit 8fb5934 changed the build script, but issue #640 says the published image still has no amd64 build. Keep
    that line only if you push a new image with this release.
  • Version: pgsync/init.py is already 8.0.0. The Unreleased heading in CHANGELOG.rst needs the version and the date.

Don't miss a new pgsync release

NewReleases is sending notifications on new releases.