Keeper.sh v2.7.0
This includes a pretty significant set of changes that should improve reliability and performance pretty significantly - particularly with halted syncs requiring a restart. This should no longer be an issue as I've hardened the idempotency of the syncing engine.
Sync Engine Rewrite
- This includes a complete rewrite of the sync engine to eliminate the immense database backpressure that would occur, and lock up syncing.
- Replaced the legacy four-job sync infrastructure with two decoupled jobs, the
ingest-sourcesjobs, and thepush-destinationsjob. Each does the respective action on an interval, with the push interval depending on your plan. - All event states changes are now computed in memory and flushed in a single database transaction, which should alleviate connection pool backpressure.
- Operations now occur in chronological order by event start time, and deletes are processed before additions to prevent partial syncs leaving your calendar in a temporary duplicate state.
- Added a shared Redis-backed sliding window rate limitation (600 requests/min.) for Google, which is shared between runtimes. Reduced to 500 requests per minute to give it a little headroom.
- Added a per-calendar sync lock which ensures that at most one active sync per calendar can exist, with graceful wind-down on supersession. Partial results are flushed, and not discarded which should make for a more graceful handoff.
- Syncs now gracefully wind down on timeout (5-minutes) instead of being absolutely smashed by the
Promise#race. - Mappings are always flushed for successfully pushed events.
- Google Batch API should now make for much faster sync times for Google. Before, I had implemented sequential and atomic requests in a
for of awaitwhich meant that, if resolution of the request took the average ~1.5s, syncing 1,000 events would take ~25 minutes. 1,000 events should now take ~2 minutes. - Hardened the sync status on the dashboard page.
Code Cleanliness
- This includes a pretty significant code cleanup, in which 18 packages were consolidated down into 7 core packages.
- Implemented path aliasing across the board.
- Adjusted the Dockerfiles, CI, s6 overlay scripts, etc., to accom., the new paths.
- Fixed turbo cache inputs to include
package.json.