Fixed: a backup could hang at the end instead of finishing (#394)
Reported on Windows with a long exclusion list: the backup ran, borg finished, and the job then sat indefinitely with no output and no error until it was cancelled by hand.
Two unbounded waits in the catalog streaming pipe, both at the point the job appeared to stall:
- When the pipe took longer than expected to close, the error handler read the process's output before stopping it. A process that is still running never closes that output, so the read never returned — the timeout was bounded, but what it fell through to was not.
- The pipe was opened with its output going to OS pipes that nothing read while the backup was running. Those hold about 64KB; once one filled, the connection stopped accepting data, the agent's write blocked, and with it the loop reading borg's output — so borg blocked too and the whole backup froze. This is the same failure that caused the prune stall fixed in 2.71.0, in a different place.
Both are now bounded: the process is stopped before anything is read from it, and its output no longer goes through a pipe that can fill. The wait for the catalog to finish writing is also more generous, since a large catalog completing normally should not be mistaken for a hang.
If a backup still hangs after upgrading, the agent log is what identifies where — it records when the catalog pipe opens and when it closes, and the gap between those two lines says whether this was the cause. Details of what to send are in the issue.
Thanks to @ArbiterGR for the detailed reproduction, including the working and failing exclusion lists side by side — that comparison is what made the failure locatable.
Changed: a failing pre-backup plugin no longer cancels the backup
Previously, any pre-backup plugin that failed aborted the entire job: no further plugins ran, borg never started, and nothing was archived. A control-panel or database dump failing on a handful of items therefore cost the client every backup of everything else — including filesystem data that had nothing to do with the plugin.
Plugin failures are now recorded and the run continues. The remaining plugins execute, the archive is written, and the job reports completed with warnings with the failure in its log, so it still surfaces rather than passing as a clean success.
Each plugin configuration gains "Abort the backup if this plugin fails", off by default. Turn it on where an archive without that plugin's output would be worthless — a database dump, typically. Existing configurations keep the new, non-aborting behaviour.
Database dumps that failed are also excluded from the restore options offered on completion, so a dump that wrote nothing can no longer be selected for restore.
Groundwork for future push notifications
Server-side plumbing and schema for a push notification capability that is still being built. It is disabled by default, nothing is configured, and nothing is sent. There is no user-facing change in this release.
Agent
Agent version 2.77.0, matching the server. Agents that accept server-driven updates will pick it up on their next check-in; the fix above only takes effect once an agent is running this version.