nfrastack/db-backup 5.0.0

on GitHub

What started out as a basic MySQL backup container back in early 2017, each major release has brought significant improvements to its capabilities. Nine years and millions of deployments later, this is a complete rewrite that has been in the works for the past 15 months. Hundreds of human written hours have been invested into getting things to where we are today.

Version 5 throws out near everything short of wisdom that I had built up over the years, hundreds of lines of fragile scripting calling third party tools of each respective database engines clients, compression and encryption tools, and whatever else I needed to glue together, replacing it all with a single binary called dbb. This opens up a so many opportunities for availability, reliability, features, updates.

The dbb binary today is under 25MB, which while it operates on its own, the most common deployment would be via a container, which takes the size from the version 4.x.x image variants from 2.0GB -> under 100MB - crazy! This single binary operates on multiple architectures, operates just as good within a container as it does outside, running as a systemd service, being declared as a NixOS module, or via your own command line for one shot reliable backups using a consistent configuration syntax. Very little remains of the previous versions codebase, except for the features.

There is also a website with documentation outlining all the features, configuration options, paired with reference guides and recipes to quickly get up and running.

What we are today (5.0.0):

Platforms

• Thanks to the portability of go, the tool is encapsulated inside of a single binary. Even better, is that we can support multiple architectures (amd64 and aarch64 are prebuilt), and also support other operating systems other than Linux. Linux, MacOS, FreeBSD, OpenBSD, and Windows all come with the exact same functionality with small variances in default path values.

Config

• YAML configuration with named profiles, instance wide defaults, multi file includes, designed to keep secrets out of your files
• Live progress display showing bytes transferred, rate and elapsed time
• Structured logging (text or json) with per job overrides to console/journal, file, or both
• Dry-run mode for prune, archive and restore operations

Backup

• Full backups across MySQL, MariaDB, PostgreSQL, MongoDB, MSSQL, Redis, SQLite3, InfluxDB and CouchDB database hosts using native drivers
• Incremental and differential backup strategies allow for faster alonbg with reduced storage costs
• Schema-only (no data) dumps, table level include/exclude filters, and split per database output when backing up an entire database engine instance
• Compression via zstd, gzip, bzip2 or xz with configurable compression levels, parallel threading, and rsync resumable output
• Checksums (md5/sha1) recorded alongside backups in a sidecar file
• Encryption at rest with AGE, GPG/OpenPGP or OpenSSL AES-256 using either recipient/public+private keys or passphrases

Storage

• Filesystem, Azure Blob, Google Cloud, S3 and compatible, WebDAV support
• Archiving hot backups to a secondary cold storage automatically after each job
• Custom file naming functionality
• Link pointers to quickly find most recent dump in a sea of hundreds of giles
• Write as specific user/group with custom permissions

Scheduling

• Cron expressions, fixed intervals, clock times (HH:MM), begin offsets (+MM)
• Natural language schedules (every 10 minutes, daily at 3pm)
• Ability to schedule a job multiple times per day, while utilizing day of week/month filters,
• Blackout windows to pause schedules during quiet periods

Restore

• Restore any backup regardless of strategy, compression, of encryption from the CLI, interactively, or driven by config profiles
• Support for automatic restores

Retention

• Keep the last N backups or prune by time window
• Grandfather-Father-Son tiers (keep N hourly, daily, weekly, monthly, yearly)

Maintenance

• Perform operations like check tables, analyze, optimize, vacuum, reindex, compact, memory purge after a successful job backup or independently
• Verify validity of files via tool based on data provided in JSON sidecar

Customization

• Pre/post shell hooks with metadata passed via environment variables per operation (DBB_JOB, DBB_TYPE, DBB_HOST, DBB_STATUS)
• Connectivity checks with retry before each backup attempt
• CI Scripting friendly with consistent exit codes and error output
• Automatic Version checking (opt-out) and opt-in anonymous usage statistics to guide roadmap
• So much more..

Upgrading from 4.x

See the upgrade guide for the full breakdown. TLDR:

• If you used the common environment vars (DB01_TYPE, DB01_HOST, etc.) your setup should work without changes
• Many environment variables have been deprecated such as GZ_RSYNCABLE, INFLUX_VERSION, and MYSQL_TLS....
• Filename patterns changed - old files may not be recognized by the new pruning logic, remove your old <5.x.x backups manually
• Checksums are now in a sidecar JSON file instead of separate .md5/.sha1 files
• Pre/Post Hooks no longer rely on positional arguments and export environment variables for your usage, they will need to be rewritten.
• Default encryption is now AGE, not GPG.

Removed

• Anything relying on a third party CLI tool is gone because the binary handles it natively.
• If you utilized the EXTRA_OPTS or similar variables, those customizations no longer work.
• Notifications and Zabbix metrics were half baked and not brought forward.

Licensing

This release changes how db-backup is licensed. It uses an open core model:

• Community Edition is free and open source under AGPL-3.0-or-later. Full backups of every engine, Filesystem/S3/WebDAV storage, AGE encryption, cron scheduling, basic retention, restore, hooks. Almost all features as what <5.x was, just better.
• Supporter Edition unlocks incremental/differential backups, Azure/GCS storage, GPG/OpenSSL encryption for backups, natural language scheduling, day filters, GFS retention tiers, archive to secondary storage, restore profiles and advanced maintenance operations. Licensed under the Nfrastack Supporter License v1 (NSLv1).
• Enterprise adds a response SLA, invoicing/PO billing and a negotiated enterprise license agreement.

The container is still free and open source under the MIT License, yet the included binary utilizes the above licensing.

See the editions page for the full comparison and pricing.

What tomorrow looks like (>> 5.1.x)

While the roadmap is not yet formalized, some ideas include:

• additional engines support
• dashboard
• control plane
• reimagined alerts
• restore enhancements
• metrics exporting
• backing up to multiple storage destinations concurrently
• further refined logic for job operations
• ??

Thanks

To everyone who filed issues, tested betas, sponsored the work, or contributed code over the years to the previous base, I'll be reaching out to you with a supporter license shortly.

If you have any feature ideas, collaboration opportunities, or need implementation support please do not hesitate to reach out to hello@nfrastack.com.

If you are using this in production, please consider picking up a Supporter license, and if using in your workplace, talk to those in charge of procurement and get setup on an Enterprise agreement. I'm doing whatever takes to put food on my table in challenging times.